Splitting and procmail

Splitting and procmail

[Jonas Steverud]

I have mail-source set to '((directory :path "~/Mail" :suffix "")) and
is using procmail to filter my email.

A quick test with nnmail-split-methods gave that splitting is not used
when the source is directory. I used the following test:

:0:
* ^Subject: splittest 1
Splittest1

:0:
* ^Subject: splittest 2
Splittest2

(setq nnmail-split-methods
      (list
       '("Splittest2" "^Subject: splittest 1")
       '("Splittest1" "^Subject: splittest 2")
       '("Mainmailbox" "")
       ))

All mail turned up according to the Procmail-rules.

So, my question is now: Mail splitting cannot be used togeather with
procmail, right?

My idea was to use mail splitting (maybe the fancy one too) and have
procmail do the spam filtering. I.e all mail except for spam are
placed in Mainmailbox and all spam are placed in Spam. Mainmailbox
should be splitted but the Spam box should not.

Possible or are there a better solution?

I have fairly nice spam filtering in my procmail and I don't want to
rewrite it as splitting rules (takes too much time). I am also
planning to start using Spamassassin (the spam.el seems interesting, I
really like the idea with using BBDB as a whitelist since I consider
all mails from MSN, Hotmail, Yahoo et al as spam) so I seems to need
procmail however I do.

Or does someone have a script that converts a procmailrc to a
split rule(s)? ;-)


-- 
(          www.dtek.chalmers.se/~d4jonas/         !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Splitting and procmail

[Reiner Steib]

On Sun, Sep 08 2002, Jonas Steverud wrote:

> I have mail-source set to '((directory :path "~/Mail" :suffix "")) and
> is using procmail to filter my email.
>
> A quick test with nnmail-split-methods gave that splitting is not used
> when the source is directory. 

,----[ C-h v nnmail-resplit-incoming RET ]
| nnmail-resplit-incoming's value is nil
| 
| Documentation:
| *If non-nil, re-split incoming procmail sorted mail.
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Splitting and procmail

[Mark Triggs]

Jonas Steverud <d4jonas@dtek.chalmers.se> writes:

> I have mail-source set to '((directory :path "~/Mail" :suffix "")) and
> is using procmail to filter my email.

[...]

> So, my question is now: Mail splitting cannot be used togeather with
> procmail, right?
>
> My idea was to use mail splitting (maybe the fancy one too) and have
> procmail do the spam filtering. I.e all mail except for spam are
> placed in Mainmailbox and all spam are placed in Spam. Mainmailbox
> should be splitted but the Spam box should not.

It should be possible. I *think* that if you use the 'directory'
mail-source directive to grab your mail, it isn't run through the
splitting process, as each file in that directory is assumed to
correspond to a gnus group. So, if you want stuff to be split, you'll
need to use the 'file' directive instead.

In your case, you could probably just specify your mail-sources as
'((file :path "~/Mail/Mainmailbox" :suffix "")). In my config, I have
something like:

  (setq mail-sources
        '((directory :path "~/docs/mail/incoming" :suffix ".spool")
  	;; Check this one separately for splitting purposes.
  	(file :path "~/docs/mail/incoming/personal.spool")))

In this case, personal.spool is split via my splitting rules, and the
rest of the files in that directory are just put directly into their
corresponding groups.

Hope this helps a bit, anyway.

Mark

-- 
Mark Triggs
<mst@dishevelled.net>

Re: Splitting and procmail

[Jonas Steverud]

Mark Triggs <mst@dishevelled.net> writes:

[...]
> Hope this helps a bit, anyway.

Yep. I was thinking of something like this but I didn't want to
experiment since if I break anything, things will go terrible wrong.

Thanks.

-- 
(          www.dtek.chalmers.se/~d4jonas/         !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Splitting and procmail

[Harry Putnam]

Reiner Steib <4uce.02.r.steib@gmx.net> writes:

> ,----[ C-h v nnmail-resplit-incoming RET ]
> | nnmail-resplit-incoming's value is nil
> | 
> | Documentation:
> | *If non-nil, re-split incoming procmail sorted mail.
> `----

Reiner,
How does that actually work.  I see no documentation in manual with
`i' (index) or s (regex) searches

Say my mail has been split to 25 different files by procmail.

All in ~/spool/*.in  files
When I call `g' in gnus with nnmail-resplit-incoming set to t what
will actually happen?  Instead of moving these 25 files to there
similarly named nnml counterparts, all 25 files are resplit
accourding to whatever split I have inside gnus... What is this for?

Re: Splitting and procmail

[Harry Putnam]

Mark Triggs <mst@dishevelled.net> writes:

>> So, my question is now: Mail splitting cannot be used togeather with
>> procmail, right?

No wrong,  using them together can give twice the power and agility.

Here is how I do it.  I never knew about the variable Reiner
mentioned `nnmail-resplit-incoming' not sure what it does.  But we've
always been able to replit a group with `B r'.

I have procmailrc setup so that a final catchall group is listed
after all spam (or other) rules.  That way all spam etc is sorted,
then anything left is dumped into a file I call prinb.in
(preinbox.in)

At the end of procmailrc:
  :0
 prinb.in

Everything that wasn't spam or something else I sort with procmail,
is now in ~/spool/prinb.in

Now when I press `g' in gnus, it slurps prinb.in into nnml:prinb

I scan that group by eye, inside gnus, see if I want to take any
actions right then, if not press `M-P a (select all) B r
(respool). and the group nnml:prinb is split according to my nnmail-split
setup.  (I actually have a keyboard macro set to run the above
commands with a keycombo  C-x C-z).

I find that I recognize most stuff that needs attention on the first
visual scan, after splitting, anything I missed will be sent to the
appropriate group where I will notice the new message count.

But my mail only amounts to some 150 to 200 per day incoming.

No tricky stuff on mail-sources, just the normal type setup one uses
with procmail:

(setq mail-sources
      '((directory :path "/home/reader/spool/"      :suffix ".in"))) 

So to summarize:  What you want to do can all be done by setting a
catchall at the end of procmailrc, sort it after gnus slurps it with
nnmail-split rules.

Re: Splitting and procmail

[Scott A Crosby]

On Sun, 08 Sep 2002 10:40:45 +0200, Jonas Steverud <d4jonas@dtek.chalmers.se> writes:

> I have mail-source set to '((directory :path "~/Mail" :suffix "")) and
> is using procmail to filter my email.

I am using maildir because I don't trust anything else, especially
over NFS. I use Safe-Cat to do mail-injection.

(setq mail-sources '((maildir :path "~/Mesg/Spool/" :subdirs ("Inbox.Bulk" "Inbox.Spam" "Inbox.Rest"))))

I am NOT using procmail for any actual catagorization.. IE, procmail
dumps all mail into those 3 maildirs. Gnus reads all 3 of them. (The
reason for this is so that I can 'less +/.*Rest.*
~/Mesg/Spool/logfile' and put 'less' into tail-mode and look for
hilighted lines.

Procmail does run some analysis routines, right now, Spamassassin, and
in the future, maybe my bayes classifier or DNSRBL's. But gnus does
the actual splitting into seperate folders.

I do this because I don't really want to have two different levels and
two/three different types of mail catagorization rules. Better to put
them all in ONE system. So, I have all of my actual catagorization
rules in nnmail-split-fancy... Which seems to be a nice superset of
procmail anyways.

I may later add in extra spamassassin rules 'FLAG_FOO' with low
weights ~.000001 soley as flags for nnmail-split-fancy rules to match
on: ("X-Spam-Status" "FLAG_PROJECT_FOO" "projects.foo"). Or, I may
instead chose to do it via functional nnmail-split-fancy rules that
scan the body for regexps.

> My idea was to use mail splitting (maybe the fancy one too) and have
> procmail do the spam filtering. I.e all mail except for spam are
> placed in Mainmailbox and all spam are placed in Spam. Mainmailbox
> should be splitted but the Spam box should not.
> 
> Possible or are there a better solution?

One option is to use 'formail' within procmail to put a special header
on what you classify as spam and have the first split-fancy rule match
on that header and put matching emails into the spam folder.

(setq nnmail-split-fancy
      '(| 
         ("My-Procmail-Classification"  "Spam" "junk.spam")
         ("My-Procmail-Classification"  "Family" "family")
         "inbox"))

> I have fairly nice spam filtering in my procmail and I don't want to
> rewrite it as splitting rules (takes too much time). I am also

I'm curious.. What does your system look like? Why not rewrite it,
piecemeal, into fancy-split-mail rules? split-fancy is so much nicer
to use than procmail, its not funny. I put 5 pages of almost
ununderstandable procmail rules into barely a screen of split-fancy
rules.

Not to mention.. split-fancy-with-parent is a GODSEND. Whomever wrote
it.. THANK YOU![1]

> planning to start using Spamassassin (the spam.el seems interesting, I
> really like the idea with using BBDB as a whitelist since I consider
> all mails from MSN, Hotmail, Yahoo et al as spam) so I seems to need
> procmail however I do.

Spamassassin offers an automatic whitelist option where it looks at
the average score for email claiming to be from a particular user and
adjusts itself as suited. With a threshold of 5, its had only one
false-positive. (An orbitz confirmation email for some tickets I'd
just bought.)

> Or does someone have a script that converts a procmailrc to a
> split rule(s)? ;-)

Maybe you should write one? :)

Scott


[1] I do have one feature request.. It'd be nice if you could define a
list of regexps that it was NOT to return. Something like:

(defun my-split-fancy-with-parent ()
  "Do a split-with-parent, however, ignore the result if it wants to put it in sent-mesg. This way, we put followups in the same group, however, we never put followups into sent-mesg."

  (let ((my-split (nnmail-split-fancy-with-parent)))
    ;;(message "Doing-my-split-fancy-with-parent")
    ;;(message my-split)
    (if (or (null my-split) (string-match "sent-mesg" my-split))
	nil
      my-split)))

Re: Splitting and procmail

[Mark Triggs]

Harry Putnam <reader@newsguy.com> writes:

> Mark Triggs <mst@dishevelled.net> writes:
>
>>> So, my question is now: Mail splitting cannot be used togeather with
>>> procmail, right?
>

> No wrong, using them together can give twice the power and agility.

Just for the record, that comment was actually made by Jonas, not me.

[...]

> So to summarize:  What you want to do can all be done by setting a
> catchall at the end of procmailrc, sort it after gnus slurps it with
> nnmail-split rules.

That's a neat idea, thanks :o)

Mark

-- 
Mark Triggs
<mst@dishevelled.net>

Re: Splitting and procmail

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> All in ~/spool/*.in  files
> When I call `g' in gnus with nnmail-resplit-incoming set to t what
> will actually happen?  Instead of moving these 25 files to there
> similarly named nnml counterparts, all 25 files are resplit
> accourding to whatever split I have inside gnus...

Exactly.

> What is this for?

Good question.  It seems some people like it, for whatever reason.
It does allow you to use procmail to split personal/mailing list
mail.  Then you can run xbiff (say) on the personal file only.  And
you can use a `directory' mail-sources entry to get the messages into
Gnus, where they are split again.

(Of course, two `file' entries would also do.)

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting and procmail

[Kai Großjohann]

Scott A Crosby <scrosby@cs.rice.edu> writes:

> [1] I do have one feature request.. It'd be nice if you could define a
> list of regexps that it was NOT to return. Something like:
>
> (defun my-split-fancy-with-parent ()
>   "Do a split-with-parent, however, ignore the result if it wants to put it in sent-mesg. This way, we put followups in the same group, however, we never put followups into sent-mesg."

See the variable nnmail-split-fancy-with-parent-ignore-groups.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting and procmail

[Reiner Steib]

On Mon, Sep 09 2002, Harry Putnam wrote:

> Reiner Steib <4uce.02.r.steib@gmx.net> writes:
[...]
>> | nnmail-resplit-incoming's value is nil
> How does that actually work.

As you see (the value is nil), I don't use it. See Kai's answer.

> I see no documentation in manual with `i' (index) or s (regex)
> searches

It should be mentioned somewhere. Oops, it already is:

,----[ <info://gnus/Mail+Source+Specifiers> ]
| `directory'
| [...]
|      There is also the variable `nnmail-resplit-incoming', if you set
|      that to a non-nil value, then the normal splitting process is
|      applied to all the files from the directory.
`----

Maybe you don't read the most recent (Oort) info files. But indeed
there is no index entry.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Splitting and procmail

[Kai Großjohann]

Reiner Steib <4uce.02.r.steib@gmx.net> writes:

> Maybe you don't read the most recent (Oort) info files. But indeed
> there is no index entry.

Meep!  Wrong!  There is an index entry (now) 8-)

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting and procmail

[Reiner Steib]

[-- Attachment #1: Type: text/plain, Size: 127 bytes --]

On Mon, Sep 09 2002, Kai Großjohann wrote:

> Meep!  Wrong!  There is an index entry (now) 8-)

... and a typo too. ;-)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: typos near nnmail-resplit-incoming --]
[-- Type: text/x-patch, Size: 570 bytes --]

--- gnus.texi.~6.313.~	Mon Sep  9 15:36:45 2002
+++ gnus.texi	Mon Sep  9 15:48:18 2002
@@ -12566,7 +12566,7 @@
 @vindex nnmail-resplit-incoming
 By default, splitting is not performed on all incoming messages.  If
 you specify a @code{directory} entry for the variable
-@code{mail-sources} @pref{Mail Source Specifiers}, however, then
+@code{mail-sources} @pxref{Mail Source Specifiers}, however, then
 splitting does @emph{not} happen by default.  You can set the variable
 @code{nnmail-resplit-incoming} to a non-nil value to make splitting
 happen even in this case.

[-- Attachment #3: Type: text/plain, Size: 116 bytes --]


Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Splitting and procmail

[Kai Großjohann]

Reiner Steib <4uce.02.r.steib@gmx.net> writes:

> On Mon, Sep 09 2002, Kai Großjohann wrote:
>
>> Meep!  Wrong!  There is an index entry (now) 8-)
>
> ... and a typo too. ;-)

Thanks.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting and procmail

[Harry Putnam]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 441 bytes --]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Reiner Steib <4uce.02.r.steib@gmx.net> writes:
>
>> Maybe you don't read the most recent (Oort) info files. But indeed
>> there is no index entry.
>
> Meep!  Wrong!  There is an index entry (now) 8-)
>
> kai

Reiner Steib <4uce.02.r.steib@gmx.net> writes:

> On Mon, Sep 09 2002, Kai Großjohann wrote:

[...]

> ... and a typo too. ;-)

Man you guys are moving pretty fast...

Re: Splitting and procmail

[Jonas Steverud]

Scott A Crosby <scrosby@cs.rice.edu> writes:

> On Sun, 08 Sep 2002 10:40:45 +0200, Jonas Steverud <d4jonas@dtek.chalmers.se> writes:
[...]
>> My idea was to use mail splitting (maybe the fancy one too) and have
>> procmail do the spam filtering.
[...]
> One option is to use 'formail' within procmail to put a special header
> on what you classify as spam and have the first split-fancy rule match
> on that header and put matching emails into the spam folder.

I think I will have Spamassassin and formail mark the mails with a
X-Steverud-Procmailrc header and then split on that using (fancy?)
split and have the split take care of mailing lists etc.

I have done an analysis of my .procmailrc and found that the spam
filtering are probably the same as SA and those that are not can
easily be taken care of with a split rule or two.

Is it possible to have a split that checks the From and if a certain
regexp matches, a function is called that tests wheather the sender is a
member of BBDB and if not it is placed in "Spam", otherwise the next
rule is tested? I.e. use BBDB as a white list. (I consider all mails
from MSN, Hotmail, Yahoo etc. as Spam. Seldom fails.)

It is the middle part of the paragraph above, the test against the
BBDB, I cannot figure out.

>> I have fairly nice spam filtering in my procmail and I don't want to
>> rewrite it as splitting rules (takes too much time). I am also
>
> I'm curious.. What does your system look like? Why not rewrite it,
> piecemeal, into fancy-split-mail rules?

It takes some time to "port" it, my procmailrc is 561 lines including
blank lines and comments, but I found out that most of the spam
filters where probably covered by SA and the splitting of mailinglists
was not that hard to convert. That's why. :-)

Besides, I like to see how many mails I got from diffrent sources when
I log in (using mailstat) but I came up with a solution to that
problem too the other day. Not an issue anylonger.

I will probably do a test as soon as I got some time over in a week or two.

>> planning to start using Spamassassin (the spam.el seems interesting, I
>> really like the idea with using BBDB as a whitelist since I consider
>> all mails from MSN, Hotmail, Yahoo et al as spam) so I seems to need
>> procmail however I do.
>
> Spamassassin offers an automatic whitelist option where it looks at
> the average score for email claiming to be from a particular user and
> adjusts itself as suited.

I would like to use BBDB as a white list. I do not like to store
information in two diffrent places, they are going to get out of sync
sooner or later. It is easier to add new persons in one place - to the
BBDB, where they going to end up anyway - then in two places (even if
the SA white list is some sort of automaticly generated list). In this
case I do not want some Artificially Stupidity-generated list but a
list I control 100 % myself.

Thanks everyone that has responded.

-- 
(          www.dtek.chalmers.se/~d4jonas/         !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Splitting and procmail

[Jonas Steverud]

Jonas Steverud <d4jonas@dtek.chalmers.se> writes:

[...]
> Is it possible to have a split that checks the From and if a certain
> regexp matches, a function is called that tests wheather the sender is a
> member of BBDB and if not it is placed in "Spam", otherwise the next
> rule is tested?

After doing some searches in the archives and the 'Net as such, I've
come up with the following function. I haven't tested it yet, so
comments are more then welcome.

I hope my example in the documentation string is correct.

(defun js-gnus-split-if-not-in-bbdb (SPLIT)
  "If the sender is not within the BBDB, the function returns SPLIT,
i.e. it is split to that group. SPLIT is a `nnmail-split-fancy'
`split' construct - usually a string with a group name.

To use BBDB as a white list for spam:
  nnmail-split-fancy
  ...
    \(|
      ...
       \(any \"\\\\(hotmail\\\\|msn\\\\|aol\\\\|yahoo\\\\|excite\\\\)\\.\\\\(co\\.uk\\\\|com\\\\|org\\\\)\"
                   \(: js-gnus-split-if-not-in-bbdb \"Spam\"\)
           \)
      

Handy togeather with very harsh spam filtering regexp's, like all mail
from Hotmail, MSN, AOL et al. Place valid addresses in BBDB and all
other mails will go to the spam group.

Inspired by http://www.gnus.org/list-archives/ding/200204/msg00077.html.
"
    (let ((from (message-fetch-field "from")))
      (and from
	   (let ((comp (condition-case ()
			   (mail-extract-address-components from)
			 (error nil))))
	     (if (and comp (not (bbdb-search-simple (car comp) (cadr comp))))
		 SPLIT)))))



-- 
(          www.dtek.chalmers.se/~d4jonas/         !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Splitting and procmail

[Jonas Steverud]

Jonas Steverud <d4jonas@dtek.chalmers.se> writes:

> Jonas Steverud <d4jonas@dtek.chalmers.se> writes:

To continue with this monologue, I would just like to report that I've
just received two emails (of which one was a test sent by me from
another account) which both where correctly splitted according to my
nnmail-fancy-split.

I use Spamassassin in my .procmailrc, I have a three spam filtering
rules in my nnmail-fancy-split of which one uses the
js-gnus-split-if-not-in-bbdb I posted earlier, I also use
bbdb/gnus-split-method and I have some other tricks to take care of
some special cases.

All mail is delivered into one mailbox and I have set the mail-sources
to a "file" source.

I have now a better understanding of how Gnus splits and handles
mail. Sorry for my possible whimsical postings before.

Problems (seems to be) solved. Case closed. Thanks.

-- 
(          www.dtek.chalmers.se/~d4jonas/         !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )