From pfournier at loups.net Wed Oct 2 13:40:42 2002 From: pfournier at loups.net (Patrice Fournier) Date: Wed, 2 Oct 2002 08:40:42 -0400 Subject: [SA-exim] Score check Message-ID: <1033562442.3d9ae94a285d9@www.courrier.sabius.net> Hi, I believe the score checks should use '>=' rather than '>' between the score and the configured value to be consistent with the config file (and common expectation) where those variables are explained as: 'SA score when you start ...' -- Patrice Fournier pfournier@loups.net From pfournier at loups.net Fri Oct 11 11:11:36 2002 From: pfournier at loups.net (Patrice Fournier) Date: Fri, 11 Oct 2002 06:11:36 -0400 Subject: [SA-exim] Suggestion for spamd timeout Message-ID: <1034331096.3da6a3d83db8b@www.courrier.sabius.net> Hi, When sa-exim accepts a mail because SA took too much time to scan the message, there should be a possibility to add a header to the accepted message. I sometimes get questions from users wondering why a spam message has no SpamAssassin headers when this happends to a spam message. The header would explain what happened. Thanks, -- Patrice Fournier pfournier@loups.net From marc at merlins.org Sat Oct 12 06:32:20 2002 From: marc at merlins.org (Marc MERLIN) Date: Fri, 11 Oct 2002 22:32:20 -0700 Subject: [SA-exim] Score check In-Reply-To: <1034331096.3da6a3d83db8b@www.courrier.sabius.net> <1033562442.3d9ae94a285d9@www.courrier.sabius.net> References: <1034331096.3da6a3d83db8b@www.courrier.sabius.net> <1033562442.3d9ae94a285d9@www.courrier.sabius.net> Message-ID: <20021012053220.GO14183@merlins.org> On Wed, Oct 02, 2002 at 08:40:42AM -0400, Patrice Fournier wrote: > Hi, > > I believe the score checks should use '>=' rather than '>' between the > score and the configured value to be consistent with the config file (and > common expectation) where those variables are explained as: 'SA score when > you start ...' Right, I'll do that, thanks. > -- On Fri, Oct 11, 2002 at 06:11:36AM -0400, Patrice Fournier wrote: > Hi, > > When sa-exim accepts a mail because SA took too much time to scan the > message, there should be a possibility to add a header to the accepted > message. I sometimes get questions from users wondering why a spam message > has no SpamAssassin headers when this happends to a spam message. The > header would explain what happened. Let me guess, it happened thursday afternoon/evening, razor wasn't answering. It sounds like a good idea, I'll add that in the next release (that's been overdue for a while now; _maybe_ this weekend) I'll also have to look at SA and find out why the alarm in there failed. razor should not hang forever... Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc_news at merlins.org Mon Oct 14 20:09:50 2002 From: marc_news at merlins.org (Marc MERLIN) Date: Mon, 14 Oct 2002 12:09:50 -0700 Subject: [SA-exim] Announce: New SA-Exim with Teergrub support Message-ID: <20021014190950.GA18416@merlins.org> [I don't usually spam the exim-users list for announcements, but considering that teergrub support is rather cool, I thought I'd make an exception this time. Please followup on the sa-exim list unless it's exim specific] Thanks for your patience to those who were waiting and had submitted ideas, it's been a whole 3 months for me to find the time to release this version (I unfortunately do not get to admin exim as part of my job anymore, although I'm hoping to change that) For the full changelog, scroll down to the changelog section of http://marc.merlins.org/linux/exim/sa.html Note that the default dir is now /etc/exim and spams are saved in a new/ subdir. I've also added a quick section on better integration in exim4.conf You can download from: http://marc.merlins.org/linux/exim/files/ http://sourceforge.net/projects/sa-exim/ So, what's teergrub? Details can be found here: http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html but basically, you this is what you get in the SMTP session: ---------------------------------------------------------------------------- data 354 Enter message, ending with "." on a line by itself (...) body SEE_FOR_YOURSELF /See (?:for|it) yourself\b/i describe SEE_FOR_YOURSELF See for yourself body ORDER_NOW /\border (?:now|soon|fast|quickly|while)\b/i describe ORDER_NOW Encourages you to waste no time in ordering From rabe at RWTH-Aachen.DE Mon Oct 14 20:38:38 2002 From: rabe at RWTH-Aachen.DE (Ralf G. R. Bergs) Date: Mon, 14 Oct 2002 21:38:38 +0200 Subject: [SA-exim] Re: [Exim] Announce: New SA-Exim with Teergrub support In-Reply-To: <20021014190950.GA18416@merlins.org> Message-ID: On Mon, 14 Oct 2002 12:09:50 -0700, Marc MERLIN wrote: >Note that the default dir is now /etc/exim and spams are saved in a new/ >subdir. You're NOT telling us that you are saving DATA to /etc/exim/new/, are you??? I hope that you don't store data in THAT place because that's a no-no. Such data belongs somewhere under the /var hierarchy. Apart from that your post was somehow "broken." :-) PS. Are you the guy who wrote the sa-exim scan_local dldopen patch? Great work. :-) -- L I N U X .~. The Choice /V\ of a GNU /( )\ Generation ^^-^^ From marc at merlins.org Mon Oct 14 20:50:42 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 14 Oct 2002 12:50:42 -0700 Subject: [SA-exim] Re: [Exim] Announce: New SA-Exim with Teergrub support In-Reply-To: References: <20021014190950.GA18416@merlins.org> Message-ID: <20021014195042.GD18416@merlins.org> On Mon, Oct 14, 2002 at 09:38:38PM +0200, Ralf G. R. Bergs wrote: > On Mon, 14 Oct 2002 12:09:50 -0700, Marc MERLIN wrote: > > >Note that the default dir is now /etc/exim and spams are saved in a new/ > >subdir. > > You're NOT telling us that you are saving DATA to /etc/exim/new/, are > you??? I hope that you don't store data in THAT place because that's a > no-no. Such data belongs somewhere under the /var hierarchy. Correct. If you have sa-exim configured to save some mails, it used to save them, say, in /var/spool/exim/SApermreject/ Now, it would save them in /var/spool/exim/SApermreject/new/ > Apart from that your post was somehow "broken." :-) Do you care to elaborate? > PS. Are you the guy who wrote the sa-exim scan_local dldopen patch? Great > work. :-) I wrote sa-exim, but not the dldopen patch (that was # David Woodhouse) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc_news at merlins.org Mon Oct 14 20:55:19 2002 From: marc_news at merlins.org (Marc MERLIN) Date: Mon, 14 Oct 2002 12:55:19 -0700 Subject: [SA-exim] Re: [Exim] Announce: New SA-Exim with Teergrub_e_ support In-Reply-To: <20021014190950.GA18416@merlins.org> References: <20021014190950.GA18416@merlins.org> Message-ID: <20021014195519.GF18416@merlins.org> On Mon, Oct 14, 2002 at 12:09:50PM -0700, Marc MERLIN wrote: > [I don't usually spam the exim-users list for announcements, but considering > that teergrub support is rather cool, I thought I'd make an exception this Ok, it's actually spelled teergrube (thanks to Alan J. Flavell for pointing that out). I'll fix the next version to have the correct spelling. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From ay at linpro.no Sun Oct 20 17:26:39 2002 From: ay at linpro.no (Audun Ytterdal) Date: Sun, 20 Oct 2002 18:26:39 +0200 Subject: [SA-exim] SA-scan compared to exiscan Message-ID: Hi. I've been using sa-exim on a customers site for a month or two now and it works flawlessly. Excellent work! The customer now wants virusscanning as well, and I looked into the exiscan-patch (http://duncanthrax.net/exiscan/), installed it on another customers site who wanted only virusscannning. To me, sa-exim and exiscan seems to be doing the same thing, except exiscan also does virusscanning, and sa-exim has more fancy smtp-reject options like teergrube'ing..... Could someone please tell me more about the difference between them? -- Audun http://audun.ytterdal.net From lists at timj.co.uk Sun Oct 20 22:22:18 2002 From: lists at timj.co.uk (Tim Jackson) Date: Sun, 20 Oct 2002 22:22:18 +0100 Subject: [SA-exim] SA-scan compared to exiscan In-Reply-To: References: Message-ID: <20021020222218.2e5f4c6e.lists@timj.co.uk> Hi Audun, on Sun, 20 Oct 2002 18:26:39 +0200 you wrote: > To me, sa-exim and exiscan seems to be doing the same thing, > except exiscan also does virusscanning, and sa-exim has more fancy > smtp-reject options like teergrube'ing..... > Could someone please tell me more about the difference between them? You've got it about right. Exiscan does a number of jobs, including passing messages off to virus scanners, spam scanners and scanning in other ways. It's fairly configurable, in a fairly high-level sense. SA-Exim does one thing, does it very well, and offers a lot of options to do it including granular control over exactly what happens. For that reason, I've chosen to use both Exiscan and SA-Exim together - using Exiscan for virus scanning only, and letting SA-Exim handle spam scanning. This works fine, because Exiscan patches Exim directly (effectively duplicating the local_scan() functionality) and leaves the local_scan() free for other stuff (such as SA-Exim). Tim From marc at merlins.org Sun Oct 20 23:10:11 2002 From: marc at merlins.org (Marc MERLIN) Date: Sun, 20 Oct 2002 15:10:11 -0700 Subject: [SA-exim] SA-scan compared to exiscan In-Reply-To: <20021020222218.2e5f4c6e.lists@timj.co.uk> References: <20021020222218.2e5f4c6e.lists@timj.co.uk> Message-ID: <20021020221011.GA6199@merlins.org> [Ccing Tom, exiscan author, so that he can correct me if needed] On Sun, Oct 20, 2002 at 06:26:39PM +0200, Audun Ytterdal wrote: > I've been using sa-exim on a customers site for a month or two now and > it works flawlessly. Excellent work! Thanks. (teergrube needs to be ironed out a bit, I'm working on that) > To me, sa-exim and exiscan seems to be doing the same thing, > except exiscan also does virusscanning, and sa-exim has more fancy > smtp-reject options like teergrube'ing..... I don't know exiscan in details but from what I understand, it does virus scanning, and has the option of passing on the mail to sa-exim. Unless I'm misinformed, exiscan does not do spam checking. (looking at the web page) Well, it looks like Tom has actually reimplemented some of sa-exim's features into sa-exim, so it used to let you pass on a mail to sa-exim, but now does something similar internally. It's not as complete, but it comes reasonably close. (there are futher options of what headers to include or not, but if you don't want some, they'd be trivial to delete from system_filter anyway) If you need virus scanning, you'd definitely want to go with exiscan since I provide none in sa-exim, virii are not a problem for me, I just reject anything executable and don't run windows anyway... If you just want to deal with spam in very configurable ways, then sa-exim would be a better choice. On Sun, Oct 20, 2002 at 10:22:18PM +0100, Tim Jackson wrote: > For that reason, I've chosen to use both Exiscan and SA-Exim together - > using Exiscan for virus scanning only, and letting SA-Exim handle spam > scanning. This works fine, because Exiscan patches Exim directly > (effectively duplicating the local_scan() functionality) and leaves the > local_scan() free for other stuff (such as SA-Exim). Interesting. Are you talking about exiscan 3? I thought exiscan 4 went in local_scan, but that Tom gave you the option to daisy chain sa-exim? Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Mon Oct 21 00:03:17 2002 From: marc at merlins.org (Marc MERLIN) Date: Sun, 20 Oct 2002 16:03:17 -0700 Subject: [SA-exim] Re: SA-Exim timeout In-Reply-To: <20021020233218.38b5ce5c.tim@timj.co.uk> References: <20021020233218.38b5ce5c.tim@timj.co.uk> Message-ID: <20021020230317.GD6199@merlins.org> On Sun, Oct 20, 2002 at 11:32:18PM +0100, Tim Jackson wrote: > Marc, [Answering on the list, because this may affect more than just you] > Just a quickie before I go to bed - any quick ideas about how exim might > time SA-Exim out, when I've got it set to explicitly time spamd out after > 4 minutes? I've just had another quick look through the code and I can't > figure out where it might have stalled. Right. I've noticed that too, and honestly, I'm not too sure, it's not supposed to happen. I'm pretty sure I had tested that when I wrote the timeout feature. When do you get the timemout? Are you trying to stall the sender or is spamd just taking too long to answer? If the C coders can look at: http://marc.merlins.org/linux/exim/files/sa-exim-cvs/sa-exim.c In both teergrube and and stallsender, I do /* Exim might want to stop us if we run for too long, but that's * exactly what we're trying to do, so let's override that */ alarm(0); Otherwise, all the code also gets signal(SIGALRM, alarm_handler); alarm (SAtimeout); If the SA alarm quicks in, you should get this in your log: log_write(0, LOG_MAIN | LOG_REJECT, "SA: spamd took more than %d secs to run, accepting message", SAtimeout); I know this code works at least sometimes because I am getting this message in my exim logs from time to time. So the question is when does it not work? Is it since you switched to exim 4.10 or a more recent sa-exim? How about other people? Are you seeing this too? Are you just doing rejection/tagging or also talling the sender on high spam scores? > I've just noticed today (since I upgraded to 2.1, and did various other > hacks including adding exiscan) that one odd message this evening got > timed out *by exim* (not internally within SA-Exim) when it was in > SA-Exim: > 2002-10-20 22:02:46 183N8i-00086l-00 local_scan() function timed out - > message temporarily rejected Right. I'm seeing this on my teergrube code, although not when I tried to reproduce it by sending a spam myself over telnet. Mmhh... > Haven't tried the teergrubeing yet :) Well, it works, except for the fact that I use printf to output the SMTP response, and that if I'm inside an SSL connection, I output cleartext and break the SSL connection :-) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From lists at timj.co.uk Mon Oct 21 10:04:44 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 21 Oct 2002 10:04:44 +0100 Subject: [SA-exim] Re: SA-Exim timeout In-Reply-To: <20021020230317.GD6199@merlins.org> References: <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> Message-ID: <20021021100444.1b86915a.lists@timj.co.uk> Hi Marc, on Sun, 20 Oct 2002 16:03:17 -0700 you wrote: > When do you get the timemout? Are you trying to stall the sender or is > spamd just taking too long to answer? I'm not trying to stall the sender, no. The only thing I have enabled at the moment is permanent rejection over a threshold. I'm not using tempreject, senderstall, teergrube or anything like that. So it's either spamd taking too long to answer or something else. The fact it's just happened since I upgraded to SA-E2.1 does suggest it's not actually spamd's fault though, especially since this is on a fairly fast and lightly-loaded machine, where checks should never time out. > So the question is when does it not work? > Is it since you switched to exim 4.10 or a more recent sa-exim? It's happened to me since I switched to SA Exim 2.1 (from 2.0.1). I've just grepped my logs and it definitely never happened before yesterday. I don't think it's Exim, because I've been running 4.10 for months. Having said that, my current troubles seem to be with one particular message, so that might suggest there's something in the message which is causing spamd to choke. It's not something to do with this is it? Newly introduced with 2.1, and I don't quite follow it: (from sa-exim.c) /* How much message body you want to feed to SA. We'll feed spamc a slightly larger message than it it's willing to process so that spamc decides that it's not going to process the message vs spamc getting a truncated message */ I'm going to enable SA-E debugging and hope the trouble message gets retried shortly and I can give some more clues. Tim From lists at timj.co.uk Mon Oct 21 10:17:25 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 21 Oct 2002 10:17:25 +0100 Subject: [SA-exim] Re: SA-Exim timeout In-Reply-To: <20021020230317.GD6199@merlins.org> References: <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> Message-ID: <20021021101725.12bfa5e2.lists@timj.co.uk> Hi Marc, on Sun, 20 Oct 2002 16:03:17 -0700 you wrote: > So the question is when does it not work? > Is it since you switched to exim 4.10 or a more recent sa-exim? Following on from my last e-mail, I should just mention that yesterday I not only switched to SA-E 2.1, but also upgraded SA from 2.42 to 2.43. However, I don't think it's caused by that, because I put SA-E 2.1 on another machine the other day, which has still got SA2.42 on it, and that's having a similar problem. But I thought I should mention it. Tim From lists at timj.co.uk Mon Oct 21 10:43:33 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 21 Oct 2002 10:43:33 +0100 Subject: [SA-exim] Re: SA-Exim timeout In-Reply-To: <20021020230317.GD6199@merlins.org> References: <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> Message-ID: <20021021104333.2fd00345.lists@timj.co.uk> Hi Marc, on Sun, 20 Oct 2002 16:03:17 -0700 you wrote: > So the question is when does it not work? Wahey. I just managed to turn on debugging (damn: only level 5, which misses a few bits) and be watching my logs as the problem mail got retried. It's very strange - it seems like SA-Exim is not setting the timeout for some reason. Here's a previous message which worked OK: SA: Debug enabled, reading config from file /my/config/file SA: config read SAspamcpath = /my/spamc SA: config read SAEximRunCond = ${fairly_normal_condition} SA: config read SAEximRejCond = ${simple_condition} SA: config read SAmaxarchivebody = 20971520 SA: config read SAerrmaxarchivebody = 1073741824 SA: config read SAtimeout = 200 SA: config read SAtimeoutsave = /my/SAtimeoutsave SA: config read SAtimeoutSavCond = 1 SA: config read SAerrorsave = /my/SAerrorsave SA: config read SAerrorSavCond = 1 SA: config read SAtemprejectonerror = 0 SA: config read SAteergrubtime = 900 SA: config read SAteergrubSavCond = 1 SA: config read SAteergrubsave = /my/SAteergrub SA: config read SAteergruboverwrite = 1 SA: config read SAstallsendertime = 900 SA: config read SAstallsenderSavCond = 1 SA: config read SAstallsendersave = /my/SAstallsender SA: config read SAstallsenderoverwrite = 1 SA: config read SAdevnullSavCond = 1 SA: config read SAdevnullsave = /my/SAdevnull SA: config read SApermreject = 14.000000 SA: config read SApermrejectSavCond = 1 SA: config read SApermrejectsave = /my/SApermreject SA: config read SAtemprejectSavCond = 1 SA: config read SAtemprejectsave = /my/SAtempreject SA: config read SAtemprejectoverwrite = 1 SA: config read SAspamacceptsave = /my/SAspamaccept SA: config read SAspamacceptSavCond = 0 SA: config read SAnotspamsave = /mySAnotspam SA: config read SAnotspamSavCond = 0 SA: SAEximRunCond expand returned: '1' SA: check succeeded, running spamc SA: Setting timeout of 200 secs before reading from spamc SA: Read from X-Spam-Status: hits=-81.8 required=6.0 SA: savemail condition expand returned: '0' SA: savemail condition expanded to false, not saving message to disk SA: score hits=-81.8 required=6.0 (scanned in 1/1 secs) Now, here's the delivery that failed. Everything was identical until: 10:18:48 ... 10:18:48 183Yho-0004Ju-00 SA: check succeeded, running spamc 10:23:48 183Yho-0004Ju-00 local_scan() function timed out - message temporarily rejected Notice the lack of a "Setting timeout..." message here. Weird, eh? SAtimeout was definitely read from the config file in both cases (well, according to the debug stuff) During the 5 minutes, there was a spamc process sitting around supposedly processing the message, but it didn't seem to be doing much (it wasn't using any CPU time to speak of). The message did have a large attachment, though. I think there are thus two separate problems here: 1. something that is causing SA-E not to set alarm() 2. something that is causing SA to not scan this particular message properly (maybe because of its size? I wonder if it may be something like the fact that SA-E has cut the message off half way through an attachment or something? Is spamc's cutoff handling more intelligent or does it simply cut off after 'x' bytes?) Obviously, 2) is causing 1) to show up. I'm more worried about 1), because if that's sorted, then a problem message like this would have just got accepted (with my settings, anyway). Luckily I caught a copy of the offending message body while it was in the spool directory, so now I'm going to play around some more. Tim From lists at timj.co.uk Mon Oct 21 12:20:40 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 21 Oct 2002 12:20:40 +0100 Subject: [SA-exim] SA-Exim timeout In-Reply-To: <20021020230317.GD6199@merlins.org> References: <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> Message-ID: <20021021122040.7a5cfcbb.lists@timj.co.uk> Hi all, OK, I think the timeout we've been discussing is a bug in SA-E, with SA-E getting caught in a loop. Basically, if I'm right, 'ret' is being re-used and overwritten when we don't want it to be. Here's a preliminary patch that seems to fix it (warning: not at all properly tested!). --- sa-exim-2.1/sa-exim.c.orig Mon Oct 14 04:27:57 2002 +++ sa-exim-2.1/sa-exim.c Mon Oct 21 11:58:52 2002 @@ -312,6 +312,7 @@ { #warning you shouldn''t worry about the "might be clobbered by longjmp", see source int ret; + int ret_read; int pid; int writefd[2]; int readfd[2]; @@ -618,12 +619,12 @@ * than SA is going to process, but let's send at least one byte more for * spamc to do the size cutoff, not us */ chunk=(samaxbody+1 / sizeof(buffera)); - while ((ret=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > 0) + while ((ret_read=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > 0) { ret=write(writefd[1], buffer, ret); CHECKERR(ret,"body write",__LINE__); } - CHECKERR(ret, "read body", __LINE__ - 4); + CHECKERR(ret_read, "read body", __LINE__ - 4); close(writefd[1]); if (SAEximDebug > 5) @@ -813,6 +814,7 @@ afterscan=time(NULL); scantime=afterscan-beforescan; + ret=0; wait(&ret); if (ret) { From lists at timj.co.uk Mon Oct 21 13:46:56 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 21 Oct 2002 13:46:56 +0100 Subject: [SA-exim] Chunking problems Message-ID: <20021021134656.42fc83ed.lists@timj.co.uk> Hello, While I've been looking into the timeout issues, I think I've spotted a problem with the chunking of data when sending to spamc. For example, around line 620 (depending on whether you've applied my previous patch or not): chunk=(samaxbody+1 / sizeof(buffera)); where chunk is an int. I'm assuming that the intended result here is that chunk contains 15. (although actually, I think it should be 16 to avoid missing the last block) But, if I add a bit of debugging logging: log_write(0, LOG_MAIN, "SA: samaxbody %d, buffera-size %d, chunks %d", samaxbody+1,sizeof(buffera),chunk); I get in my Exim log: SA: samaxbody 256001, buffera-size 16384, chunks 256000 !!! This doesn't stop it working (since the return value from read() will be zero once everything's read) but ought to be fixed. I think the relevant calculation should be something like: chunk=((samaxbody+1) / sizeof(buffera))+1; (Also, why is sizeof(buffera)=16384 and not 4096 as it seems to be when declared?) Interestingly, when saving chunks to file (around line 215), the divisor used is "sizeof(buffera)-1": chunk=(SAmaxarchivebody / (sizeof(buffera)-1))+1; Why? Which is right? (I don't think it's critical, but it would be nice to be consistent :) Incidentally, in the code it's mentioned about talking to spamd directly rather than via spamc. Exiscan implements this, so if you want a test implementation to copy, it's there :) Here's a diff combining my "timeout" and "chunking" fixes (care: a couple of lines might have been wrapped) --- sa-exim-2.1/sa-exim.c.orig Mon Oct 14 04:27:57 2002 +++ sa-exim-2.1/sa-exim.c Mon Oct 21 13:41:06 2002 @@ -312,6 +312,7 @@ { #warning you shouldn''t worry about the "might be clobbered by longjmp", see source int ret; + int ret_read; int pid; int writefd[2]; int readfd[2]; @@ -617,13 +618,17 @@ /* We're now feeding the body to SA, but let's not send much more body data * than SA is going to process, but let's send at least one byte more for * spamc to do the size cutoff, not us */ - chunk=(samaxbody+1 / sizeof(buffera)); - while ((ret=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > 0) + chunk=((samaxbody+1) / sizeof(buffera))+1; + while ((ret_read=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > 0) { + if (SAEximDebug > 10) + { + log_write(0, LOG_MAIN, "SA: Sending body chunk %d to spamc", chunk); + } ret=write(writefd[1], buffer, ret); CHECKERR(ret,"body write",__LINE__); } - CHECKERR(ret, "read body", __LINE__ - 4); + CHECKERR(ret_read, "read body", __LINE__ - 4); close(writefd[1]); if (SAEximDebug > 5) @@ -813,6 +818,7 @@ afterscan=time(NULL); scantime=afterscan-beforescan; + ret=0; wait(&ret); if (ret) { Tim From marc at merlins.org Mon Oct 21 19:14:12 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 21 Oct 2002 11:14:12 -0700 Subject: [SA-exim] Re: SA-Exim timeout In-Reply-To: <20021021134656.42fc83ed.lists@timj.co.uk> <20021021122040.7a5cfcbb.lists@timj.co.uk> <20021021104333.2fd00345.lists@timj.co.uk> <20021021101725.12bfa5e2.lists@timj.co.uk> <20021021100444.1b86915a.lists@timj.co.uk> References: <20021021122040.7a5cfcbb.lists@timj.co.uk> <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> <20021021104333.2fd00345.lists@timj.co.uk> <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> <20021021101725.12bfa5e2.lists@timj.co.uk> <20021020233218.38b5ce5c.tim@timj.co.uk> <20021020230317.GD6199@merlins.org> <20021021100444.1b86915a.lists@timj.co.uk> Message-ID: <20021021181412.GG26406@merlins.org> On Mon, Oct 21, 2002 at 10:04:44AM +0100, Tim Jackson wrote: > Hi Marc, on Sun, 20 Oct 2002 16:03:17 -0700 you wrote: Hi, I appreciate your looking into this, especially while I was sleeping :-) > > When do you get the timemout? Are you trying to stall the sender or is > > spamd just taking too long to answer? > > I'm not trying to stall the sender, no. The only thing I have enabled at > the moment is permanent rejection over a threshold. I'm not using Ok. I was looking into this plus teergrube on my side, which didn't really help :-) > tempreject, senderstall, teergrube or anything like that. So it's either > spamd taking too long to answer or something else. The fact it's just > happened since I upgraded to SA-E2.1 does suggest it's not actually Ok, so I did screw something up in the last release while trying to fix that other report about sa-exim sometimes feeding a truncated message to spamc. That's very good to know, thanks. > I'm going to enable SA-E debugging and hope the trouble message gets > retried shortly and I can give some more clues. I have a secondary MX where I disabled sa-exim for now so that I can capture such a message and re-feed it at will. On Mon, Oct 21, 2002 at 10:17:25AM +0100, Tim Jackson wrote: > Following on from my last e-mail, I should just mention that yesterday I > not only switched to SA-E 2.1, but also upgraded SA from 2.42 to 2.43. > However, I don't think it's caused by that, because I put SA-E 2.1 on > another machine the other day, which has still got SA2.42 on it, and > that's having a similar problem. But I thought I should mention it. Right, but we know it's not that. On Mon, Oct 21, 2002 at 10:43:33AM +0100, Tim Jackson wrote: > 10:18:48 ... > 10:18:48 183Yho-0004Ju-00 SA: check succeeded, running spamc > 10:23:48 183Yho-0004Ju-00 local_scan() function timed out - message > temporarily rejected > > Notice the lack of a "Setting timeout..." message here. Weird, eh? Ok, that's _very_ good to know. > 1. something that is causing SA-E not to set alarm() It never gets there. > 2. something that is causing SA to not scan this particular message > properly (maybe because of its size? I wonder if it may be something > like the fact that SA-E has cut the message off half way through an > attachment or something? Is spamc's cutoff handling more intelligent or > does it simply cut off after 'x' bytes?) That's what I tried to fix. SA-Exim will send slighlty more than samaxbody to spamc so that spamc gets the option of not feeding the spam to spamd, and just returning "not scanned" I think I really should just see how big the file is and not even feed it to spamc if it's too big, but in the meantime, the current code should stll work, and has worked so far. On Mon, Oct 21, 2002 at 12:20:40PM +0100, Tim Jackson wrote: > OK, I think the timeout we've been discussing is a bug in SA-E, with SA-E > getting caught in a loop. Basically, if I'm right, 'ret' is being re-used > and overwritten when we don't want it to be. Ok, I don't get this. > --- sa-exim-2.1/sa-exim.c.orig Mon Oct 14 04:27:57 2002 > +++ sa-exim-2.1/sa-exim.c Mon Oct 21 11:58:52 2002 > @@ -312,6 +312,7 @@ > { > #warning you shouldn''t worry about the "might be clobbered by longjmp", > see source > int ret; > + int ret_read; > int pid; > int writefd[2]; > int readfd[2]; > @@ -618,12 +619,12 @@ > * than SA is going to process, but let's send at least one byte more > for > * spamc to do the size cutoff, not us */ > chunk=(samaxbody+1 / sizeof(buffera)); > - while ((ret=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > 0) > + while ((ret_read=read(fd, buffer, sizeof(buffera))) > 0 && chunk-- > > 0) > { > ret=write(writefd[1], buffer, ret); > CHECKERR(ret,"body write",__LINE__); > } > - CHECKERR(ret, "read body", __LINE__ - 4); > + CHECKERR(ret_read, "read body", __LINE__ - 4); Granted, it's monday morning, and I was hacking late (on something else :-) last night, but where's the problem with the old code? If ret is 0, we don't enter while, if it's negative we display the error. Inside the loop, ret should be >=0 or it is a write error. Sure, that value is rechecked when you exit the loop, but that doesn't matter, it's still >=0 or it would have tripped the first CHECKERR. > @@ -813,6 +814,7 @@ > afterscan=time(NULL); > scantime=afterscan-beforescan; > > + ret=0; > wait(&ret); This was for testing, right? (it shouldn't do anything) On Mon, Oct 21, 2002 at 01:46:56PM +0100, Tim Jackson wrote: > chunk=(samaxbody+1 / sizeof(buffera)); > > where chunk is an int. I'm assuming that the intended result here is that > chunk contains 15. (although actually, I think it should be 16 to avoid > missing the last block) Right. I make it a bit bigger on purpose. Too big is better than missing the end of the body :-) > But, if I add a bit of debugging logging: > > log_write(0, LOG_MAIN, "SA: samaxbody %d, buffera-size %d, chunks %d", > samaxbody+1,sizeof(buffera),chunk); > > I get in my Exim log: > > SA: samaxbody 256001, buffera-size 16384, chunks 256000 I don't understand why chunks is 256000 here. > This doesn't stop it working (since the return value from read() will be > zero once everything's read) but ought to be fixed. I think the relevant > calculation should be something like: > > chunk=((samaxbody+1) / sizeof(buffera))+1; I switched it to, which is what I think it should be. chunk=(samaxbody / sizeof(buffera))+1; > (Also, why is sizeof(buffera)=16384 and not 4096 as it seems to be when > declared?) buffera-size=16384 is a compiler issue, it allocates 4 bytes for a char, but since I read sizeof(buffera), I don't make any assumption on the size of buffera. In other words, it's unexpected, but it's ok. > Interestingly, when saving chunks to file (around line 215), the divisor > used is "sizeof(buffera)-1": Right. I touched the code over too many months to remember that :-) > chunk=(SAmaxarchivebody / (sizeof(buffera)-1))+1; > Why? Which is right? (I don't think it's critical, but it would be nice to > be consistent :) I've changed both to be like above (without the -1, which was an initial worry about getting one more byte written for the trailing null (not needed)) > Incidentally, in the code it's mentioned about talking to spamd directly > rather than via spamc. Exiscan implements this, so if you want a test > implementation to copy, it's there :) I might do that, although I kind of like not having to worry about the network protocol, possible changes, and so forth. Forking spamc is easier IMO Basically I'm thinking that the problem was the missing the +1 at the end of chunk as you suggested and I wasn't sending the whole body to spamc, but I'm not quite sure why it would cause the code to stop there and not continue to log_write(0, LOG_MAIN, "SA: fed spam to spamc, reading result"); and setting the SA timeout. Either way, I've added more debug status output to see where it stops should it happen again See CVS for the current version I started running 5mn ago :-) http://sourceforge.net/cvs/?group_id=56124 http://marc.merlins.org/linux/exim/files/sa-exim-cvs/ Thanks for your copious feedback Cheers, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Tue Oct 22 10:23:10 2002 From: marc at merlins.org (Marc MERLIN) Date: Tue, 22 Oct 2002 02:23:10 -0700 Subject: [SA-exim] Re: SA-Exim timeout -> use SA 2.1 cvs, not 2.1 Message-ID: <20021022092310.GR26406@merlins.org> On Mon, Oct 21, 2002 at 10:03:46PM +0100, Tim Jackson wrote: > Hi Marc, > > OK, I give up. write() seems to be stalling on the chunk that would Well, I took over while you were sleeping :-) and fixed the problem with what I should have done in the first place (See below) > "overflow" the spamc size limit, but at this point I don't have the > knowledge or C experience to know why. I'm going to write one less chunk > for now so that at least my system works :) Please do keep me up to date > with how the bugfix is developing, though, and what the solution turns out > to be. Honestly, I don't know why spamc stops reading from the pipe and hangs my process. By all accounts it shouldn't especially in the middle of a message body. I wouldn't be surprized if it's some subtle bug in spamc. Anyway, it doesn't matter, I fixed the problem by skipping the message altogether if it's bigger than what spamc will accept. Let me know if this works for you (it works for me), and I'll probably release a 2.1.1 or 2.2 this weekend with: - this fix - a fix I have yet to write on teergrube over SSL (not that it should ever happen :-) - Make SAmaxbody an config option and not a compile time value - Allow SA-Exim to pass an truncated body to spamc instead of just accepting big Emails. - Clean the body save to a file code now that I save fdstart BTW, if a real C programmer can answer this for me: /* Ok, tell me what's the better way of getting the size of the file an fd * points to */ fdstart=lseek(fd, 0, SEEK_CUR); CHECKERR(fdstart,"lseek SEEK_CUR",__LINE__); /* this is the body size plus a few bytes (exim msg ID) */ /* it should be 18 bytes, but I'll assume it could be more or less */ fdsize=lseek(fd, 0, SEEK_END); CHECKERR(fdsize,"lseek SEEK_END",__LINE__); /* Reset fd to the body start */ ret=(int) lseek(fd, fdstart, SEEK_SET); CHECKERR(ret,"lseek SEEK_SET",__LINE__); BTW, I did not use your ret=0 before wait(&ret), the man page makes it clear that it's a write only value and setting it to 0 beforehand should do nothing. > Would it not be a good idea (if possible) to implement some kind of alarm > around this point, as a safety measure in case this sort of thing happens > (either because of a bug or some other oddity)? I may move the timeout before I start writing to spamc I'll study that after I get some sleep. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From tom at duncanthrax.net Tue Oct 22 09:56:37 2002 From: tom at duncanthrax.net (Tom Kistner) Date: Tue, 22 Oct 2002 10:56:37 +0200 Subject: [SA-exim] SA-scan compared to exiscan References: <20021020222218.2e5f4c6e.lists@timj.co.uk> <20021020221011.GA6199@merlins.org> Message-ID: <3DB512C5.6030700@duncanthrax.net> Marc MERLIN wrote: > [Ccing Tom, exiscan author, so that he can correct me if needed] No corrections necessary :) I have added the antispam facility to exiscan since most people want to do both av and antispam. However, my goal is always to keep the configuration as simple as possible and the impact on the SMTP protocol as low as possible, so exiscan does not support as many options as sa-exim and I will also not go into things like tarpitting. > I thought exiscan 4 went in local_scan, but that Tom gave you the option > to daisy chain sa-exim? Yes. I moved exiscan out of local_scan completely, so you can use any other local_scan plugin in conjunction with exiscan. It is no problem to use both exiscan and sa-exim together, and I would recommend it to those who need a more sophisticated antispam configuration. :) regards, /tom -- Tom Kistner ICQ 1501527 dcanthrax@efnet http://duncanthrax.net From dman at dman.ddts.net Tue Oct 22 17:35:44 2002 From: dman at dman.ddts.net (Derrick 'dman' Hudson) Date: Tue, 22 Oct 2002 12:35:44 -0400 Subject: [SA-exim] Re: SA-Exim timeout -> use SA 2.1 cvs, not 2.1 In-Reply-To: <20021022092310.GR26406@merlins.org> References: <20021022092310.GR26406@merlins.org> Message-ID: <20021022163543.GA6342@dman.ddts.net> ---------------------- multipart/signed attachment On Tue, Oct 22, 2002 at 02:23:10AM -0700, Marc MERLIN wrote: | BTW, if a real C programmer can answer this for me: | /* Ok, tell me what's the better way of getting the size of the file = an fd | * points to */ You should stat it. Excerpts from stat(2) : int fstat(int filedes, struct stat *buf); These functions return information about the specified file. You= do not need any access rights to the file to get this information but = you need search rights to all directories named in the path leading to = the file. They all return a stat structure, which contains the following field= s: struct stat { dev_t st_dev; /* device */ ino_t st_ino; /* inode */ mode_t st_mode; /* protection */ nlink_t st_nlink; /* number of hard links */ uid_t st_uid; /* user ID of owner */ gid_t st_gid; /* group ID of owner */ dev_t st_rdev; /* device type (if inode devic= e) */ off_t st_size; /* total size, in bytes */ blksize_t st_blksize; /* blocksize for filesystem I/= O */ blkcnt_t st_blocks; /* number of blocks allocated = */ time_t st_atime; /* time of last access */ time_t st_mtime; /* time of last modification */ time_t st_ctime; /* time of last change */ }; -D --=20 If anyone would come after me, he must deny himself and take up his cross and follow me. For whoever wants to save his life will lose it, but whoever loses his life for me and for the gospel will save it. What good is it for a man to gain the whole world, yet forfeit his soul? Or what can a man give in exchange for his soul? Mark 8:34-37 =20 http://dman.ddts.net/~dman/ ---------------------- multipart/signed attachment A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available Url : http://lists.merlins.org/archives/sa-exim/attachments/20021022/2d6bb54a/attachment.bin ---------------------- multipart/signed attachment-- From marc at merlins.org Tue Oct 22 17:35:26 2002 From: marc at merlins.org (Marc MERLIN) Date: Tue, 22 Oct 2002 09:35:26 -0700 Subject: [SA-exim] Re: SA-Exim timeout -> use SA 2.1 cvs, not 2.1 In-Reply-To: <20021022163543.GA6342@dman.ddts.net> References: <20021022092310.GR26406@merlins.org> <20021022163543.GA6342@dman.ddts.net> Message-ID: <20021022163526.GV26406@merlins.org> ---------------------- multipart/signed attachment On Tue, Oct 22, 2002 at 12:35:44PM -0400, Derrick 'dman' Hudson wrote: > On Tue, Oct 22, 2002 at 02:23:10AM -0700, Marc MERLIN wrote: >=20 > | BTW, if a real C programmer can answer this for me: > | /* Ok, tell me what's the better way of getting the size of the fil= e an fd > | * points to */ >=20 > You should stat it. >=20 > Excerpts from stat(2) : =20 Duh! Of course (I new about stat and lstat and didn't bother to check if there as a stat for already opened file descriptors) Thanks Marc --=20 "A mouse is a device used to point at the xterm you want to type in" - A.S.= R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet coo= king=20 Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP= key ---------------------- multipart/signed attachment A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available Url : http://lists.merlins.org/archives/sa-exim/attachments/20021022/dbc3bf77/attachment.bin ---------------------- multipart/signed attachment-- From marc at merlins.org Wed Oct 23 04:01:39 2002 From: marc at merlins.org (Marc MERLIN) Date: Tue, 22 Oct 2002 20:01:39 -0700 Subject: [SA-exim] SA-scan compared to exiscan In-Reply-To: <3DB512C5.6030700@duncanthrax.net> References: <20021020222218.2e5f4c6e.lists@timj.co.uk> <20021020221011.GA6199@merlins.org> <3DB512C5.6030700@duncanthrax.net> Message-ID: <20021023030139.GT2740@merlins.org> On Tue, Oct 22, 2002 at 10:56:37AM +0200, Tom Kistner wrote: > Marc MERLIN wrote: > > >[Ccing Tom, exiscan author, so that he can correct me if needed] BTW, I added you in the 'can post' list so the list shouldn't send you a warning were you to post again. > No corrections necessary :) I have added the antispam facility to > exiscan since most people want to do both av and antispam. However, my > goal is always to keep the configuration as simple as possible and the > impact on the SMTP protocol as low as possible, so exiscan does not > support as many options as sa-exim and I will also not go into things > like tarpitting. That makes sense, thanks for the info. > >I thought exiscan 4 went in local_scan, but that Tom gave you the option > >to daisy chain sa-exim? > > Yes. I moved exiscan out of local_scan completely, so you can use any > other local_scan plugin in conjunction with exiscan. It is no problem to > use both exiscan and sa-exim together, and I would recommend it to those > who need a more sophisticated antispam configuration. :) Cool, thanks. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Thu Oct 24 18:15:50 2002 From: marc at merlins.org (Marc MERLIN) Date: Thu, 24 Oct 2002 10:15:50 -0700 Subject: [SA-exim] Re: SA-Exim In-Reply-To: References: <20021024134158.GD4423@merlins.org> Message-ID: <20021024171550.GG4423@merlins.org> On Thu, Oct 24, 2002 at 04:22:58PM +0100, Tim Jackson wrote: > So it looks like it's working for now. I'd still like to find a better > solution really, because from some quick tests I did, it seemed like spamc > actually didn't totally ignore messages larger than the limit, but did > still give them a score (presumably based on the part of the mail that is > under that limit). I could be wrong though, and having said that, the > machine I'm working on (SA2.43) seems to be ignoring messages greater than > the size: > > $ spamc -c 0/0 Right. Spamc will ignore a message that's too big and not scan half of it. The reason is that its mime decoding code fails if the whole message isn't fed to SA > I'm pretty sure at home it was getting a score though. But maybe I'm > confused. My code used to truncate the message to be right at the limit spamc used to accept. This was reported as a bug (rightfully so): I had intended to feed a partial message to spamc, but SA would give a score to perfectly good message because the mime decoding failed. I fixed that for SA 2.1 by having SA truncate the message to right under the spamc limit, but in return, we found out that spamc started hanging when we fed it a message that was bigger than it was willing to accept. So, when I release SA 2.2, I plan to offer: - not pass message bigger than samaxbody to spamc - truncate message to less than samaxbody (but you'll have to modify SA to not give a score for messages with no mime ending) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From brian at enchanter.net Mon Oct 28 15:59:27 2002 From: brian at enchanter.net (Brian Kendig) Date: Mon, 28 Oct 2002 10:59:27 -0500 Subject: [SA-exim] Timeout error on large messages Message-ID: <3C8BF9DE-EA8E-11D6-8FEB-003065546CF4@enchanter.net> I think this is the same problem that's been discussed here recently, but I went to *such* trouble to track it down, and I want to document it *someplace*... :-) When my Exim server receives a large message (like, with a big attachment), this appears in '/var/spool/exim/log/mainlog': 2002-10-28 09:53:49 H4P5DM-000A85-00 SA: SAEximRunCond expand returned: '1' 2002-10-28 09:53:49 H4P5DM-000A85-00 SA: check succeeded, running spamc And then exactly five minutes later it says: 2002-10-28 09:58:49 H4P5DM-000A85-00 local_scan() function timed out - message temporarily rejected Nothing is written to system.log or mail.log; spamd doesn't ever say anything about this large message. ps shows: perl /usr/bin/spamd -d -u nobody -D -L (I'm running spamd in debug and local mode) spamc -s 256000 (This process sits around for five minutes and doesn't seem to do anything) So I'm hoping this is the same exact problem as was discussed earlier. :-) ____ |\/| Brian Kendig Set your priorities right. \ /\ / ..__. brian at enchanter net No one ever said on his \/ \__\ _/ http://www.enchanter.net/ death bed, "Gee, if I'd \__ __ \_ Be insatiably curious. only spent more time at \____\___\ Ask "why" a lot. the office." From marc at merlins.org Mon Oct 28 16:03:30 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 28 Oct 2002 08:03:30 -0800 Subject: [SA-exim] Timeout error on large messages In-Reply-To: <3C8BF9DE-EA8E-11D6-8FEB-003065546CF4@enchanter.net> References: <3C8BF9DE-EA8E-11D6-8FEB-003065546CF4@enchanter.net> Message-ID: <20021028160330.GV14688@merlins.org> On Mon, Oct 28, 2002 at 10:59:27AM -0500, Brian Kendig wrote: > spamc -s 256000 > (This process sits around for five minutes and doesn't seem to do > anything) > > So I'm hoping this is the same exact problem as was discussed earlier. > :-) Right, it's the same problem. The problem actually seems to be with spamc hanging on input and hanging sa-exim, but the end result is the same. It's fixed in CVS and I was supposed to release the new version yesterday, except that I've been sick for the last 4 days, so it kind of hinders on my coding... If you get the version from CVS, it's fixed, and I really hope to release version 2.2 RSN. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From brian at enchanter.net Mon Oct 28 16:17:02 2002 From: brian at enchanter.net (Brian Kendig) Date: Mon, 28 Oct 2002 11:17:02 -0500 Subject: [SA-exim] SA-Exim checks *outgoing* mail too? Message-ID: So I noticed this in /var/spool/exim/log/mainlog just now as I sent a message to this mailing list... Is SA-Exim running all of my *outgoing* mail through SpamAssassin too? If so, then how do I turn this off? I don't ever want someone to receive email from me already (and incorrectly) marked as spam! H4P8F3-000AB4-00 SA: SAEximRunCond expand returned: '1' H4P8F3-000AB4-00 SA: check succeeded, running spamc H4P8F3-000AB4-00 SA: savemail condition expanded to false, not saving message to disk H4P8F3-000AB4-00 SA: score hits=-1.0 required=5.0 (scanned in 1/1 secs) H4P8F3-000AB4-00 <= brian@enchanter.net H=swing.enchanter.bogus (enchanter.net) [10.0.1.4] P=esmtp S=2010 id=3C8BF9DE-EA8E-11D6-8FEB-003065546CF4@enchanter.net H4P8F3-000AB4-00 => sa-exim@lists.merlins.org R=dnslookup T=remote_smtp H=mail1.merlins.org [216.200.201.205] X=TLSv1:DES-CBC3-SHA:168 H4P8F3-000AB4-00 Completed ____ |\/| Brian Kendig Set your priorities right. \ /\ / ..__. brian at enchanter net No one ever said on his \/ \__\ _/ http://www.enchanter.net/ death bed, "Gee, if I'd \__ __ \_ Be insatiably curious. only spent more time at \____\___\ Ask "why" a lot. the office." From marc at merlins.org Mon Oct 28 16:26:11 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 28 Oct 2002 08:26:11 -0800 Subject: [SA-exim] SA-Exim checks *outgoing* mail too? In-Reply-To: References: Message-ID: <20021028162611.GX14688@merlins.org> On Mon, Oct 28, 2002 at 11:17:02AM -0500, Brian Kendig wrote: > So I noticed this in /var/spool/exim/log/mainlog just now as I sent a > message to this mailing list... > > Is SA-Exim running all of my *outgoing* mail through SpamAssassin too? > > If so, then how do I turn this off? I don't ever want someone to > receive email from me already (and incorrectly) marked as spam! Look a spamassassin.conf # Exim configuration string to run before running SA against the message # You should not put double quotes around the expression # This decides whether SA gets run against the message or not. Messages will # not be rejected if the message had SA headers but weren't added by us # If you comment this out, SA will be disabled # Watch your logs, you will get errors and your messages will get temporarily # bounced if the expansion fails # Anything that doesn't expand to "" or "0" (without quotes) will be considered # true If you set the string to 1, it will be true without going through exim's # condition evaluator (and if you leave it unset, it will default to 0) SAEximRunCond: ${if and {{def:sender_host_address} {!Jeq {$sender_host_address}{127.0.0.1}} {!eq {$h_X-SA-Do-Not-Run:}{Yes}} } {1}{0}} You also want to look at my exim4.conf config if you haven't done so yet: http://marc.merlins.org/linux/exim/#conf The check_rcpt ACL has: warn message = X-SA-Do-Not-Rej: Yes local_parts = +nosarej:postmaster:abuse warn message = X-SA-Do-Not-Run: Yes hosts = +relay_from_hosts warn message = X-SA-Do-Not-Run: Yes authenticated = * Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From brian at enchanter.net Mon Oct 28 19:50:28 2002 From: brian at enchanter.net (Brian Kendig) Date: Mon, 28 Oct 2002 14:50:28 -0500 Subject: [SA-exim] Timeout error on large messages In-Reply-To: <20021028160330.GV14688@merlins.org> Message-ID: <82910804-EAAE-11D6-8FEB-003065546CF4@enchanter.net> Marc MERLIN wrote: > It's fixed in CVS and I was supposed to release the new > version > yesterday, except that I've been sick for the last 4 days, so it kind > of > hinders on my coding... You *rock.* :-) I hope you feel better soon! ____ |\/| Brian Kendig Set your priorities right. \ /\ / ..__. brian at enchanter net No one ever said on his \/ \__\ _/ http://www.enchanter.net/ death bed, "Gee, if I'd \__ __ \_ Be insatiably curious. only spent more time at \____\___\ Ask "why" a lot. the office." From brian at enchanter.net Mon Oct 28 20:45:35 2002 From: brian at enchanter.net (Brian Kendig) Date: Mon, 28 Oct 2002 15:45:35 -0500 Subject: [SA-exim] SA-Exim checks *outgoing* mail too? In-Reply-To: <20021028162611.GX14688@merlins.org> Message-ID: <353E7190-EAB6-11D6-8FEB-003065546CF4@enchanter.net> Marc MERLIN wrote: > Brian Kendig wrote: >> Is SA-Exim running all of my *outgoing* mail through SpamAssassin too? > > Look a spamassassin.conf > > SAEximRunCond: ${if and {{def:sender_host_address} {!Jeq > {$sender_host_address}{127.0.0.1}} {!eq {$h_X-SA-Do-Not-Run:}{Yes}} } > {1}{0}} So what this is saying is that SpamAssassin will be run if the sender host address is NOT localhost and if 'X-SA-Do-Not-Run: Yes' isn't found in the message, right? Can I specify wildcards here? My home network is all in the 10.0.x.x subnet, so can I say something like 'if the sender host address is not 10.0.*'? And what happens if a spammer gets wise to this and sends me spam with a 'X-SA-Do-Not-Run: Yes' header? ;-) > You also want to look at my exim4.conf config if you haven't done so > yet: > http://marc.merlins.org/linux/exim/#conf > > The check_rcpt ACL has: > warn message = X-SA-Do-Not-Rej: Yes > local_parts = +nosarej:postmaster:abuse > > warn message = X-SA-Do-Not-Run: Yes > hosts = +relay_from_hosts > > warn message = X-SA-Do-Not-Run: Yes > authenticated = * Thanks for the pointer -- I'm looking over them and figuring them out as I go. :) ____ |\/| Brian Kendig Set your priorities right. \ /\ / ..__. brian at enchanter net No one ever said on his \/ \__\ _/ http://www.enchanter.net/ death bed, "Gee, if I'd \__ __ \_ Be insatiably curious. only spent more time at \____\___\ Ask "why" a lot. the office." From brian at enchanter.net Mon Oct 28 21:23:46 2002 From: brian at enchanter.net (Brian Kendig) Date: Mon, 28 Oct 2002 16:23:46 -0500 Subject: [SA-exim] Customizing the rejection message? Message-ID: <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> Right now it seems that when SA-Exim rejects a message, it gives the mailer a message like this: hits=49.0 required=5.0 trigger=40.0 which some mailers will then display to the human being who tried to send the message. Is there any straightforward way to customize this message to something like: Your message was rejected because it looks like spam. If it really wasn't, please re-send it to notspam@mydomain.com. ____ |\/| Brian Kendig Set your priorities right. \ /\ / ..__. brian at enchanter net No one ever said on his \/ \__\ _/ http://www.enchanter.net/ death bed, "Gee, if I'd \__ __ \_ Be insatiably curious. only spent more time at \____\___\ Ask "why" a lot. the office." From lists at timj.co.uk Mon Oct 28 22:26:23 2002 From: lists at timj.co.uk (Tim Jackson) Date: Mon, 28 Oct 2002 22:26:23 +0000 Subject: [SA-exim] Customizing the rejection message? In-Reply-To: <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> References: <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> Message-ID: <20021028222623.761362e3.lists@timj.co.uk> Hi Brian, on Mon, 28 Oct 2002 16:23:46 -0500 you wrote: [rejection messages] > Is there any straightforward way to customize this message I had the same question as you when I first installed SA-E. The answer is that it's not a configuration option, but you'll find it's very easy to patch the source to customise the message (and others, such as teergrubeing etc.). I've mulled over whether to write a patch that would make the message a config file option (or beg Marc to do it :), but I personally decided against it on these grounds: 1. it's something you rarely need to change 2. it's very easy to patch the source anyway 3. with custom messages, you'd want to provide a selection of variables to be used in the message, so you'd then have to mess around with substitution 'variables' etc. Tim From marc at merlins.org Tue Oct 29 02:11:02 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 28 Oct 2002 18:11:02 -0800 Subject: [SA-exim] Customizing the rejection message? In-Reply-To: <20021028222623.761362e3.lists@timj.co.uk> <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> References: <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> <20021028222623.761362e3.lists@timj.co.uk> <8AE9BDD9-EABB-11D6-8FEB-003065546CF4@enchanter.net> Message-ID: <20021029021102.GQ14688@merlins.org> On Mon, Oct 28, 2002 at 04:23:46PM -0500, Brian Kendig wrote: > Right now it seems that when SA-Exim rejects a message, it gives the > mailer a message like this: > > hits=49.0 required=5.0 trigger=40.0 > > which some mailers will then display to the human being who tried to > send the message. Correct. > Is there any straightforward way to customize this message to something > like: Yes, you have to hack the code :-) On Mon, Oct 28, 2002 at 10:26:23PM +0000, Tim Jackson wrote: > I've mulled over whether to write a patch that would make the message a > config file option (or beg Marc to do it :), but I personally decided > against it on these grounds: > > 1. it's something you rarely need to change > > 2. it's very easy to patch the source anyway Those have been my arguments for not worying abou this so far :-) > 3. with custom messages, you'd want to provide a selection of variables to > be used in the message, so you'd then have to mess around with > substitution 'variables' etc. Exactly. It seemed like work, and it's a lot easier for everyone to edit the source appropriately :-) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Tue Oct 29 02:18:17 2002 From: marc at merlins.org (Marc MERLIN) Date: Mon, 28 Oct 2002 18:18:17 -0800 Subject: [SA-exim] SA-Exim checks *outgoing* mail too? In-Reply-To: <353E7190-EAB6-11D6-8FEB-003065546CF4@enchanter.net> References: <20021028162611.GX14688@merlins.org> <353E7190-EAB6-11D6-8FEB-003065546CF4@enchanter.net> Message-ID: <20021029021817.GS14688@merlins.org> On Mon, Oct 28, 2002 at 03:45:35PM -0500, Brian Kendig wrote: > Marc MERLIN wrote: > >Brian Kendig wrote: > >>Is SA-Exim running all of my *outgoing* mail through SpamAssassin too? > > > >Look a spamassassin.conf > > > >SAEximRunCond: ${if and {{def:sender_host_address} {!Jeq > >{$sender_host_address}{127.0.0.1}} {!eq {$h_X-SA-Do-Not-Run:}{Yes}} } > >{1}{0}} > > So what this is saying is that SpamAssassin will be run if the sender > host address is NOT localhost and if 'X-SA-Do-Not-Run: Yes' isn't found > in the message, right? Right. > Can I specify wildcards here? My home network is all in the 10.0.x.x > subnet, so can I say something like 'if the sender host address is not > 10.0.*'? See the exim spec, there is something to test if an IP is in a range (you wouldn't say 10.0.*, you'd say 10.0.0.0/16) > And what happens if a spammer gets wise to this and sends me spam with > a 'X-SA-Do-Not-Run: Yes' header? ;-) You rename the header :-) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Tue Oct 29 16:17:37 2002 From: marc at merlins.org (Marc MERLIN) Date: Tue, 29 Oct 2002 08:17:37 -0800 Subject: [SA-exim] SA-Exim 2.2 released Message-ID: <20021029161737.GA1269@merlins.org> Ok, so as you probably read, SA-Exim 2.1 had a bug with big messages when they were passed to spamc and spamc would hang or stop reading (not sure which one) I should have released 2.2 with that fix (available in CVS soon after) but I didn't really want to release a half assed fix. After a bunch of hours, I think I have something that is much better. * 2002/10/28 - v2.2 + Fixed a bug that affected all mails bigger than what spamc would accept. Doh! (it not clear why, but spamc would hang and stop reading after it had been fed more than it was willing to accept) + Added more debugging code to help track the above problem + Depending on SATruncBodyCond will now either not pass a message that's too big to spamc, or will optionally truncate it first + Now strips any X-SA-Exim-* headers already present in the message before scanning it + Added new X-SA-Exim-Rcpt-To: header (see privacy section in README) after a suggestion from Brian Kendig + Teergrube is now spelled correctly (note that the option names in spamassassin.conf changed as a result) + Teergrubing has been re-implemented to detect that the other side went away so that exim doesn't stay around for nothing. + Added SAteergrubecond so that you don't teergrube your neighbours + Small cleanups Please see the README, there is actually stuff to read there now :-) It hasn't run for days on my mail server yet (just 10H or so, I do need to replace SA-Exim 2.1 _now_), but no problems so far, and I had more time to test it this time. As always, it's here: http://marc.merlins.org/linux/exim/sa.html http://sourceforge.net/projects/sa-exim/ (RSN) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From marc at merlins.org Wed Oct 30 06:14:53 2002 From: marc at merlins.org (Marc MERLIN) Date: Tue, 29 Oct 2002 22:14:53 -0800 Subject: [SA-exim] Teergrubing, how long are you holding off spammers? Message-ID: <20021030061453.GA14688@merlins.org> For those who are upgrading/planning to upgrade to SA-Exim 2.2 and willing to use teergrubing, I'd be curious to know how long you can hold spammers off before the disconnect (this is the first version of SA-Exim that can detect that the other side went way and will stop spewing crap into the void :-) So far, I found that apparently both sendmail and postfix will wait during the full 15 minutes that I teergrube them (well, I don't anymore since those are known places who just happen to relay spam through mailing lists, so I've configured SA-Exim not to teergrube them but just reject the message) That said, I just caught my first direct connection from a real spammer ----- Forwarded message from root ----- Subject: magic.merlins.org 2002/10/29 22:02 system check This mail is sent by logcheck. If you do not want to receive it any more, please modify the configuration files in /etc/logcheck or deinstall logcheck. Unusual System Events =-=-=-=-=-=-=-=-=-=-= (...) 2002-10-29 21:53:47 186llf-0003iE-00 SA: SAteergrubecond expand returned: '1' 2002-10-29 21:53:47 186llf-0003iE-00 SA: Writing message to /var/spool/exim/SAteergrube/new/E186llf-0003iE-00@mail1.merlins.org 2002-10-29 21:53:47 186llf-0003iE-00 SA: local_scan will teergrube the sender for 900 secs: hits=26.6 required=7.0 trigger=25.0 (scanned in 1/1 secs). From (host=NULL [218.54.77.151]) for marc@merlins.org 2002-10-29 21:55:27 186llf-0003iE-00 SA: Interrupting Teergrube, remote side closed the connection after about 100 secs 2002-10-29 21:55:27 186llf-0003iE-00 temporarily rejected by local_scan(): Remote side closed the connection after about 100 secs of teergrube (hits=26.6 required=7.0 trigger=25.0) ----- End forwarded message ----- 100 seconds instead of the usual 1-5 secs that they usually spend to send a spam. That's slowed them down by a factor of more than 20, not bad :-) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From ay at linpro.no Wed Oct 30 13:08:34 2002 From: ay at linpro.no (Audun Ytterdal) Date: Wed, 30 Oct 2002 14:08:34 +0100 Subject: [SA-exim] Re: SA-Exim 2.2 released References: <20021029161737.GA1269@merlins.org> Message-ID: Marc MERLIN writes: > As always, it's here: > http://marc.merlins.org/linux/exim/sa.html > http://sourceforge.net/projects/sa-exim/ (RSN) It would be nice to have a single patch to exim like exiscan has. Now you have to build it outside of exim with references to the source. -- Audun From marc at merlins.org Wed Oct 30 16:08:58 2002 From: marc at merlins.org (Marc MERLIN) Date: Wed, 30 Oct 2002 08:08:58 -0800 Subject: [SA-exim] Re: SA-Exim 2.2 released In-Reply-To: References: <20021029161737.GA1269@merlins.org> Message-ID: <20021030160857.GB14688@merlins.org> On Wed, Oct 30, 2002 at 02:08:34PM +0100, Audun Ytterdal wrote: > > As always, it's here: > > http://marc.merlins.org/linux/exim/sa.html > > http://sourceforge.net/projects/sa-exim/ (RSN) > > It would be nice to have a single patch to exim like exiscan has. > Now you have to build it outside of exim with references to the > source. You don't have to actually. You get to pick whether you just want to copy the local_scan function in your exim source and rebuild exim every time you upgrade sa-exim, or whether you want to patch exim with the dlopen patch once, and then rebuild and upgrade sa-exim independantly from exim (as explained in the INSTALL file). If you rebuild sa-exim independantly from exim, then yes, you do need the exim source tree, afterall sa-exim does have to use the exim API. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key From ay at linpro.no Wed Oct 30 16:27:59 2002 From: ay at linpro.no (Audun Ytterdal) Date: Wed, 30 Oct 2002 17:27:59 +0100 Subject: [SA-exim] Re: SA-Exim 2.2 released In-Reply-To: <20021030160857.GB14688@merlins.org> (Marc MERLIN's message of "Wed, 30 Oct 2002 08:08:58 -0800") References: <20021029161737.GA1269@merlins.org> <20021030160857.GB14688@merlins.org> Message-ID: Marc MERLIN writes: > On Wed, Oct 30, 2002 at 02:08:34PM +0100, Audun Ytterdal wrote: >> > As always, it's here: >> > http://marc.merlins.org/linux/exim/sa.html >> > http://sourceforge.net/projects/sa-exim/ (RSN) >> >> It would be nice to have a single patch to exim like exiscan has. >> Now you have to build it outside of exim with references to the >> source. > > You don't have to actually. > > You get to pick whether you just want to copy the local_scan function in > your exim source and rebuild exim every time you upgrade sa-exim, or > whether you want to patch exim with the dlopen patch once, and then > rebuild and upgrade sa-exim independantly from exim (as explained in the > INSTALL file). > If you rebuild sa-exim independantly from exim, then yes, you do need > the exim source tree, afterall sa-exim does have to use the exim API. Ups. I missed something in my explaination, I want the loadable module option, But if you are going to make a rpm or a debian dbs-type of package you want a single tar.gz file one or more patches. So I could just make a diff/patch of the local_scan replacement, but I want to include the dlopenpatch and a patch that inserts the two new source files (dummy and sa-exim.so) and modifes the buildprocess to include sa-exim when you build exim. This is very simple inside my head, but somewhat hard to write down in english. -- Audun From marc at merlins.org Wed Oct 30 17:01:13 2002 From: marc at merlins.org (Marc MERLIN) Date: Wed, 30 Oct 2002 09:01:13 -0800 Subject: [SA-exim] Re: SA-Exim 2.2 released In-Reply-To: References: <20021029161737.GA1269@merlins.org> <20021030160857.GB14688@merlins.org> Message-ID: <20021030170113.GC14688@merlins.org> On Wed, Oct 30, 2002 at 05:27:59PM +0100, Audun Ytterdal wrote: > Ups. I missed something in my explaination, I want the loadable module > option, But if you are going to make a rpm or a debian dbs-type of > package you want a single tar.gz file one or more patches. On debian, it's not really a problem, I already build a package and just put both files in the deb. For that matter, it wouldn't really be a problem with RPM either. > So I could just make a diff/patch of the local_scan replacement, but I > want to include the dlopenpatch and a patch that inserts the two new > source files (dummy and sa-exim.so) and modifes the buildprocess to > include sa-exim when you build exim. I think I understand what you want to do, and basically, it should really not be a problem, there is only one patch and the other files are separate so you can simply include them as entire files in the SRPM. Either way, this is really a packaing issue. If you have further issues, let's continue off list, it probably isn't of much interest to the rest of the list. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key