spam.el: automatically resplitting ham in a spam group?

spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

I may have missed something obvious, but is there a way to
automatically resplit the ham that I find in a spam group?  Currently
they all go to a designated mail group and then just sit there until I
remember to resplit them by hand, which kind of works, but is not very
nice.

More details:

My mail splitting is handled entirely by Gnus.  For spam filtering, I
use Bogofilter, which is run by Gnus' spam.el during the splitting.
All spam candidates discovered by spam-split are stored in a spam
group named nnml+mail:mail.spam.candidates.  This candidates group has
no spam processor, but it has a spam destination for the really
hardcore spam group, nnml+mail:mail.spam.proven, which has its spam
process set to feed all spam to Bogofilter's magical spam analysis
algorithms.  My real mail groups also move their spam to
mail.spam.proven, and that is the only mail group with a spam exit
processor.  The idea is that spam is only fed to Bogofilter if I had a
chance to look at it, either by spotting it as a false negative in a
normal mailgroup, or by leaving its spam mark set while browsing for
false positives in mail.spam.candidates.  By having two spam groups, I
can avoid ever having to deal with the same spam twice, which is
important to me.  I also like having all my spam tidily collected in a
single group, hence mail.spam.proven instead of just expiring the
messages in whatever group they first appeared in.

I think all this is not unsimilar to some previously discussed
configurations.  Does this sound all right to you, or is this setup a
perverse hack of the delicate instrument that is spam.el? :-)

My problem is what to do with false positives in mail.spam.candidates.
When I mark them as ham and exit the candidates group, I would like
to have them (1) unregistered as spam (if needed), (2) registered as
ham, and (3) re-fed to the mail split mechanism as if the message had
just arrived in my inbox.  I understand that (1) will be solved
elegantly with the registry, and I am happy to wait for it.  I have
already solved (2) by setting spam-process-ham-in-spam-groups to t,
and setting up ham exit processors for my spam groups.  But I could
not find an easy solution for (3): I have even considered setting up
an nnmaildir group, and then adding its directory to my mail-sources,
but that practice is explicitly discouraged in the manual, so I
hesitate to do that.  Could someone help me with this?

For even more details, here is my config:

        (setq spam-use-bogofilter t)

	(setq gnus-spam-process-destinations
	      '(("mail\\.spam" nil)
		("mail\\." "nnml+mail:mail.spam.proven")
		("list\\." "nnml+mail:mail.spam.proven")))

        (setq spam-split-group "mail.spam.candidates")

        ;; Process ham in spam groups (to teach Bogofilter about false positives).
	(setq spam-process-ham-in-spam-groups t)

	;; Don't ignore the spam process destination of spam groups.
	(setq spam-move-spam-nonspam-groups-only nil)

        (spam-initialize)

        ;;; ...

	(defun lk-gnus-group-split-updated-hook ()
	  (setq nnmail-split-fancy
		(list '|
                      ;; Get rid of spam.
	              '("keywords" "CERN SpamKiller Note: [12][0-9]" "mail.spam.candidates")
	              '(: spam-split)
                      ;; Put followups into the same group their parents are in.
	              (: nnmail-split-fancy-with-parent)
                      ;; Handle the rest.
                      nnmail-split-fancy)))

	(add-hook 'gnus-group-split-updated-hook
		  'lk-gnus-group-split-updated-hook)

        (gnus-group-split-setup 'auto-update)

The relevant group parameters for nnml+mail:mail.spam.candidates are:

	(spam-contents gnus-group-spam-classification-spam)
	(spam-process
	 (gnus-group-ham-exit-processor-bogofilter))
	(spam-process-destination . "nnml+mail:mail.spam.proven")
	(ham-process-destination . "nnml+mail:mail.ham.proven")

The same for nnml+mail:mail.spam.proven:

	(spam-contents gnus-group-spam-classification-spam)
	(spam-process
	 (gnus-group-spam-exit-processor-bogofilter gnus-group-ham-exit-processor-bogofilter))
	(spam-process-destination)
 	(ham-process-destination . "nnml+mail:mail.ham.proven")

(nnml+mail:mail.ham.proven is just a normal mail group without spam or
ham classification.)

-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Wed, 19 Nov 2003, lorentey@elte.hu wrote:

> I may have missed something obvious, but is there a way to
> automatically resplit the ham that I find in a spam group?
> Currently they all go to a designated mail group and then just sit
> there until I remember to resplit them by hand, which kind of works,
> but is not very nice.

Short answer: it can be done.

> My mail splitting is handled entirely by Gnus.
[...]
> I think all this is not unsimilar to some previously discussed
> configurations.  Does this sound all right to you, or is this setup
> a perverse hack of the delicate instrument that is spam.el? :-)

Your setup is very similar to mine.  I'm not sure if it's optimal,
but it works well for me.  I'm pretty paranoid about missing
important mail, though.

> My problem is what to do with false positives in
> mail.spam.candidates.  When I mark them as ham and exit the
> candidates group, I would like to have them (1) unregistered as spam
> (if needed), (2) registered as ham, and (3) re-fed to the mail split
> mechanism as if the message had just arrived in my inbox.  I
> understand that (1) will be solved elegantly with the registry, and
> I am happy to wait for it.

Good.  I'm already able to remember the act of registering spam/ham,
now I need to have a way to undo that act.  It's coming.

> But I could not find an easy solution for (3): I have even
> considered setting up an nnmaildir group, and then adding its
> directory to my mail-sources, but that practice is explicitly
> discouraged in the manual, so I hesitate to do that.  Could someone
> help me with this?

You can set a ham-destination of "INBOX" or wherever your incoming
mail comes from.  That might help you (but make sure to tell spam.el
to mark ham unread before moving it to INBOX).

The problem with automatic resplitting is that it could send your
mail back to the spam folder.  I don't like that possibility.  I
could make a ham-destination option to use 'respool instead of a
group name, but just recently there was discussion of a nnimap bug
with respooling.  What if the respooling returns nil, do we leave the
ham message in the spam group?

By the way, recently I added multiple spam/ham-process-destinations.
So now from a spam group, you can send ham to "INBOX" and to a
"train_ham" group.  I find that very useful for training, since
"train_ham" will only contain the misclassified messages.  With your
setup, you may like it too.

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Jody Klymak]

Hello Ted,

Ted Zlatanov <tzz@lifelogs.com> writes:

> By the way, recently I added multiple spam/ham-process-destinations.
> So now from a spam group, you can send ham to "INBOX" and to a
> "train_ham" group.  I find that very useful for training, since
> "train_ham" will only contain the misclassified messages.  With your
> setup, you may like it too.

What is the rationale behind this?  Do you do the training on the mail
server?

I train locally using bogofilter, and have found simply training on
misclassified messages to work quite well.  ie. only train on spam in
my ham group and only on ham in my spam group.  So I have been curious
about these train_ham and train_spam groups.

Thanks,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Wed, 19 Nov 2003, jklymak@coas.oregonstate.edu wrote:

> Hello Ted,
> 
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> By the way, recently I added multiple
>> spam/ham-process-destinations.  So now from a spam group, you can
>> send ham to "INBOX" and to a "train_ham" group.  I find that very
>> useful for training, since "train_ham" will only contain the
>> misclassified messages.  With your setup, you may like it too.
> 
> What is the rationale behind this?  Do you do the training on the
> mail server?

Yes.

> I train locally using bogofilter, and have found simply training on
> misclassified messages to work quite well.  ie. only train on spam
> in my ham group and only on ham in my spam group.  So I have been
> curious about these train_ham and train_spam groups.

Correct, for local training bogofilter is fine.  I can't use it with
my ISP, though.

A change that's coming is that I'll make all the registration
functions batch-oriented - so instead of registering one message at a
time, all of them can be registered at once.  That should speed things
up enormously, and it should make remote training possible (you can
save the articles in a file and send it over via ssh or whatever
mechanism you like).

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Reiner Steib]

On Wed, Nov 19 2003, Jody Klymak wrote:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> By the way, recently I added multiple spam/ham-process-destinations.
>> So now from a spam group, you can send ham to "INBOX" and to a
>> "train_ham" group. [...]

Now it tested this and it works fine.  Thanks, Ted.  Sorry for not
confirming it earlier.

> What is the rationale behind this?  Do you do the training on the mail
> server?

I requested this feature.  See
<URL:http://thread.gmane.org/gmane.emacs.gnus.general/54439> or
<news:v91xt3kan7.fsf@marauder.physik.uni-ulm.de> for details why it's
useful for me.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: spam.el: automatically resplitting ham in a spam group?

[Kai Grossjohann]

nnimap knows how to split mail from a group into other groups, but it
is restricted to nnimap groups.  I wonder if that code can be
leveraged, somehow, to split from any group to any other group...

Maybe it's sufficient to hang a function off of getting news which
just enters the ham.proven group and performs resplitting there, for
all articles it finds?

But of course it would be nice if the spam package supported
resplitting as a ham process destination, or suchlike.  I would use
it, for sure ;-)

Kai

Re: spam.el: automatically resplitting ham in a spam group?

[Kai Grossjohann]

Kai Grossjohann <kai@emptydomain.de> writes:

> But of course it would be nice if the spam package supported
> resplitting as a ham process destination, or suchlike.  I would use
> it, for sure ;-)

Eek, I forget that I have a server-side setup where splitting is done
on the server side...  Things You Have To Go Through When You Want To
Read Mail With A Web Client...

Kai

Re: spam.el: automatically resplitting ham in a spam group?

[Jody Klymak]

Hi Reiner and Ted,

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> On Wed, Nov 19 2003, Jody Klymak wrote:
>> What is the rationale behind this?  Do you do the training on the mail
>> server?
>
> I requested this feature.  See
> <URL:http://thread.gmane.org/gmane.emacs.gnus.general/54439> or
> <news:v91xt3kan7.fsf@marauder.physik.uni-ulm.de> for details why
> it's useful for me.

Great, thanks for the clarification.  

I wonder if these different set-ups could be made a series of
tutorials on http://www.emacswiki.org?  I'll try and write up my
simple local-processing set up by this weekend.

Thanks,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el: automatically resplitting ham in a spam group?

[Josh Huber]

Kai Grossjohann <kai@emptydomain.de> writes:

> Eek, I forget that I have a server-side setup where splitting is
> done on the server side...  Things You Have To Go Through When You
> Want To Read Mail With A Web Client...

I have a similar setup, using exim to feed cyrus imap.  Clients are
Gnus and squirrelmail.  I've got a pretty complex sieve script, and
like the fact that all the filtering is done inside the imap server
during delivery.

Kai -- do you use spam.el?  I've been thinking about ways to get bayes
spam filtering working well with a configuration like this...any
recommendations?

The best would be a solution everyone with an account on my server
could use, if they wanted to.  I like the idea of using folders which
let people reprocess misclassified ham/spam.

-- 
Josh Huber

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Wed, 19 Nov 2003, jklymak@coas.oregonstate.edu wrote:

> I wonder if these different set-ups could be made a series of
> tutorials on http://www.emacswiki.org?  I'll try and write up my
> simple local-processing set up by this weekend.

That's fine.  I think some additions to the manual in that direction
would also be most welcome.

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Wed, 19 Nov 2003, kai@emptydomain.de wrote:

> Maybe it's sufficient to hang a function off of getting news which
> just enters the ham.proven group and performs resplitting there, for
> all articles it finds?

That would actually be very useful.  I need to write something like
it for cross-server splitting, where the gnus-registry tracks an
article to a group on another server.  The articles will go in a
"queue" group and then get moved to their final destinations.

> But of course it would be nice if the spam package supported
> resplitting as a ham process destination, or suchlike.  I would use
> it, for sure ;-)

I added it as a customization option to ham-process-destination.
I'll see if I can add the actual code for resplitting to spam.el today.

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Ted Zlatanov <tzz@lifelogs.com> writes:
> You can set a ham-destination of "INBOX" or wherever your incoming
> mail comes from.  That might help you (but make sure to tell spam.el
> to mark ham unread before moving it to INBOX).

My incoming mail comes from a Maildir, where it gets put by fetchmail.
I am hesitant about leaving my mail on a network server, so at the
moment I use IMAP only as a kind of fancy POP3.

I can not specify my incoming Maildir as a ham-destination, because
having the same Maildir serve as a mail source and a Gnus group is an
official Bad Practice(tm).  Of course, I could have a cron job moving
messages from the ham.proven group to my incoming Maildir.  That would
work, but it would be somewhat of a kludge.

> The problem with automatic resplitting is that it could send your
> mail back to the spam folder.  I don't like that possibility.

You are right, I have not thought about that.  Of course, in my setup
it would just mean that the mail gets back to the spam.candidates
group and I have to have a look at it again the next time I decide for
a little cherry-picking of false positives.  Sooner or later,
BogoFilter sees the message as ham so many times that its spamicity
level gets below the threshold.

> I could make a ham-destination option to use 'respool instead of a
> group name, but just recently there was discussion of a nnimap bug
> with respooling.  What if the respooling returns nil, do we leave
> the ham message in the spam group?

Hm.  What if spam-split would simply be disabled during the ham-exit
respool?

-- 
Lőrentey Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Wed, 19 Nov 2003, kai@emptydomain.de wrote:
>
>> Maybe it's sufficient to hang a function off of getting news which
>> just enters the ham.proven group and performs resplitting there, for
>> all articles it finds?
>
> That would actually be very useful.  I need to write something like
> it for cross-server splitting, where the gnus-registry tracks an
> article to a group on another server.  The articles will go in a
> "queue" group and then get moved to their final destinations.

This feature would work for me nicely, too.

>> But of course it would be nice if the spam package supported
>> resplitting as a ham process destination, or suchlike.  I would use
>> it, for sure ;-)
>
> I added it as a customization option to ham-process-destination.
> I'll see if I can add the actual code for resplitting to spam.el today.

Great, thank you!

-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Lőrentey Károly <lorentey@elte.hu> writes:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I could make a ham-destination option to use 'respool instead of a
>> group name, but just recently there was discussion of a nnimap bug
>> with respooling.  What if the respooling returns nil, do we leave
>> the ham message in the spam group?
>
> Hm.  What if spam-split would simply be disabled during the ham-exit
> respool?

Sorry, my mind was wandering about respooling back to the spam group.
When does respooling return nil?  Does it mean that the respooling
failed to categorize the message in any group?  I guess in my case,
leaving ham in the spam group is OK if it can not be meaningfully
resplit.

-- 
Lőrentey Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Thu, 20 Nov 2003, lorentey@elte.hu wrote:

>> I could make a ham-destination option to use 'respool instead of a
>> group name, but just recently there was discussion of a nnimap bug
>> with respooling.  What if the respooling returns nil, do we leave
>> the ham message in the spam group?
> 
> Hm.  What if spam-split would simply be disabled during the ham-exit
> respool?

That's not necessarily the only problem.  The real issue is that the
destination is being decided dynamically, so every function in the
splitting sequence would have to know that returning the name of the
current group is not OK - we don't want the ham to stay in the spam
group, no matter what.  I hope you see the problem now.

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Thu, 20 Nov 2003, lorentey@elte.hu wrote:

> Lőrentey Károly <lorentey@elte.hu> writes:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> I could make a ham-destination option to use 'respool instead of a
>>> group name, but just recently there was discussion of a nnimap bug
>>> with respooling.  What if the respooling returns nil, do we leave
>>> the ham message in the spam group?
>>
>> Hm.  What if spam-split would simply be disabled during the
>> ham-exit respool?
> 
> Sorry, my mind was wandering about respooling back to the spam
> group.  When does respooling return nil?  Does it mean that the
> respooling failed to categorize the message in any group?  I guess
> in my case, leaving ham in the spam group is OK if it can not be
> meaningfully resplit.

Respooling could return nil if the last element is spam-split or some
other function that can return nil.  I think returning nil as the last
split element is officially a Bad Thing, and spam-split does it
because anywhere but the last element it indicates a fall-through
(thus, ham gets through spam-split).

Anyhow, the big concern, which I just formulated this morning because
my brain works better then :) is that we resplit right back into the
spam group.  That's a very bad thing, because it will look to the
user like spam.el is not doing its job.

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Jake Colman]

>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:

   TZ> On Wed, 19 Nov 2003, jklymak@coas.oregonstate.edu wrote:
   >> I wonder if these different set-ups could be made a series of
   >> tutorials on http://www.emacswiki.org?  I'll try and write up my
   >> simple local-processing set up by this weekend.

   TZ> That's fine.  I think some additions to the manual in that direction
   TZ> would also be most welcome.

Moreover, I would say that having it in the manual is AT LEAST as important
as having it on the wiki.

-- 
Jake Colman                     

Principia Partners LLC                    Phone: (201) 209-2467
Harborside Financial Center                 Fax: (201) 946-0320
902 Plaza Two                          E-mail: colman@ppllc.com
Jersey City, NJ 07311                 www.principiapartners.com

Re: spam.el: automatically resplitting ham in a spam group?

[Jake Colman]

>>>>> "KG" == Kai Grossjohann <kai@emptydomain.de> writes:

   KG> Kai Grossjohann <kai@emptydomain.de> writes:
   >> But of course it would be nice if the spam package supported
   >> resplitting as a ham process destination, or suchlike.  I would use
   >> it, for sure ;-)

   KG> Eek, I forget that I have a server-side setup where splitting is done
   KG> on the server side...  Things You Have To Go Through When You Want To
   KG> Read Mail With A Web Client...

But for server-side splitting you must be using something other than
XEmacs/Gnus, right?

-- 
Jake Colman                     

Principia Partners LLC                    Phone: (201) 209-2467
Harborside Financial Center                 Fax: (201) 946-0320
902 Plaza Two                          E-mail: colman@ppllc.com
Jersey City, NJ 07311                 www.principiapartners.com

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Thu, 20 Nov 2003, lorentey@elte.hu wrote:
>
>> Lőrentey Károly <lorentey@elte.hu> writes:
>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>>> I could make a ham-destination option to use 'respool instead of a
>>>> group name, but just recently there was discussion of a nnimap bug
>>>> with respooling.  What if the respooling returns nil, do we leave
>>>> the ham message in the spam group?
>>>
>>> Hm.  What if spam-split would simply be disabled during the
>>> ham-exit respool?
>> 
>> Sorry, my mind was wandering about respooling back to the spam
>> group.  When does respooling return nil?  Does it mean that the
>> respooling failed to categorize the message in any group?  I guess
>> in my case, leaving ham in the spam group is OK if it can not be
>> meaningfully resplit.
>
> Respooling could return nil if the last element is spam-split or some
> other function that can return nil.  I think returning nil as the last
> split element is officially a Bad Thing, and spam-split does it
> because anywhere but the last element it indicates a fall-through
> (thus, ham gets through spam-split).

I see.  But I don't think it is normal that spam-split (or some other
nil-returning function) would be at the end of the split
specification.

> Anyhow, the big concern, which I just formulated this morning because
> my brain works better then :) is that we resplit right back into the
> spam group.  That's a very bad thing, because it will look to the
> user like spam.el is not doing its job.

IMHO most of this (except infrequent the nil case) would be eliminated
by disabling spam-split.

-- 
Lőrentey Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Thu, 20 Nov 2003, lorentey@elte.hu wrote:

> I see.  But I don't think it is normal that spam-split (or some
> other nil-returning function) would be at the end of the split
> specification.

That's never stopped anyone :)  Let me give you an example:
gnus-use-ifile is guaranteed to return a group name.  So the user
makes spam-split the last entry in the split rules.  Eventually the
user decides to turn spam-use-ifile off.  It's not a farfetched
scenario at all.

> IMHO most of this (except infrequent the nil case) would be
> eliminated by disabling spam-split.

Well, try it with the current respool behavior and let's see how it
works.  It may be that all this worrying was unnecessary.

Thanks
Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Thu, 20 Nov 2003, lorentey@elte.hu wrote:
>
>> I see.  But I don't think it is normal that spam-split (or some
>> other nil-returning function) would be at the end of the split
>> specification.
>
> That's never stopped anyone :)

You are right, of course. :-)

> Let me give you an example: gnus-use-ifile is guaranteed to return a
> group name.  So the user makes spam-split the last entry in the
> split rules.  Eventually the user decides to turn spam-use-ifile
> off.  It's not a farfetched scenario at all.

But if I understand it well, no message would be lost; they would just
reappear in the current group, and I think these users can easily fix
their configuration, or simply ignore the new respooling feature.
Adding a note to the documentation along the lines of 'to prevent
unexpected results, if you plan to use this feature, make sure that
your split rules are set up so that the last rule never returns nil'
would be enough, I guess.

(By the way: is it 'respool' or 'resplit'?  I don't know the correct
terminology.)

>> IMHO most of this (except infrequent the nil case) would be
>> eliminated by disabling spam-split.
>
> Well, try it with the current respool behavior and let's see how it
> works.  It may be that all this worrying was unnecessary.

Thank you for implementing this!  I appreciate your work.

I have played with it for a while, and after adding the
no-spam-split-during-respool thingy, I think it does exactly what I
need.

If you're interested, here is my change: (it's quite ugly, but works)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: spam.el diff --]
[-- Type: text/x-patch, Size: 4201 bytes --]

Index: spam.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam.el,v
retrieving revision 6.133
diff -u -r6.133 spam.el
--- spam.el	20 Nov 2003 14:52:28 -0000	6.133
+++ spam.el	20 Nov 2003 18:16:33 -0000
@@ -116,6 +116,15 @@
   :type 'boolean
   :group 'spam)
 
+(defcustom spam-disable-spam-split-during-ham-respool nil
+  "Whether spam-split should be ignored while resplitting ham in
+a process destination.  This is useful to prevent ham from ending
+up in the same spam group after the resplit.  Don't set this if
+you have spam-split as the last rule in your split
+configuration."
+  :type 'boolean
+  :group 'spam)
+
 (defcustom spam-whitelist (expand-file-name "whitelist" spam-directory)
   "The location of the whitelist.
 The file format is one regular expression per line.
@@ -623,7 +632,8 @@
 	  (gnus-summary-set-process-mark article))
 
 	(if respool			   ; respooling is with a "fake" group
-	    (gnus-summary-respool-article nil respool-method)
+	    (let ((spam-split-disabled spam-disable-spam-split-during-ham-respool))
+	      (gnus-summary-respool-article nil respool-method))
 	  (if (or (not backend-supports-deletions) ; else, we are not respooling
 		  (> (length groups) 1))
 	      (progn			; if copying, copy and set deletep
@@ -763,6 +773,9 @@
 "The spam-list-of-statistical-checks list contains all the mail
 splitters that need to have the full message body available.")
 
+(defvar spam-split-disabled nil
+  "If non-nil, spam-split is disabled, and always returns nil.")
+
 ;;;TODO: modify to invoke self with each specific check if invoked without specific checks
 (defun spam-split (&rest specific-checks)
   "Split this message into the `spam' group if it is spam.
@@ -773,35 +786,36 @@
 
 See the Info node `(gnus)Fancy Mail Splitting' for more details."
   (interactive)
-  (let ((spam-split-group-choice spam-split-group))
-    (dolist (check specific-checks)
-      (when (stringp check)
-	(setq spam-split-group-choice check)
-	(setq specific-checks (delq check specific-checks))))
-
-    (let ((spam-split-group spam-split-group-choice))
-      (save-excursion
-	(save-restriction
-	  (dolist (check spam-list-of-statistical-checks)
-	    (when (and (symbolp check) (symbol-value check))
-	      (widen)
-	      (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
-			    (symbol-name check))
-	      (return)))
-	  ;;   (progn (widen) (debug (buffer-string)))
-	  (let ((list-of-checks spam-list-of-checks)
-		decision)
-	    (while (and list-of-checks (not decision))
-	      (let ((pair (pop list-of-checks)))
-		(when (and (symbol-value (car pair))
-			   (or (null specific-checks)
-			       (memq (car pair) specific-checks)))
-		  (gnus-message 5 "spam-split: calling the %s function" 
-				(symbol-name (cdr pair)))
-		  (setq decision (funcall (cdr pair))))))
-	    (if (eq decision t)
-		nil
-	      decision)))))))
+  (unless spam-split-disabled
+    (let ((spam-split-group-choice spam-split-group))
+      (dolist (check specific-checks)
+	(when (stringp check)
+	  (setq spam-split-group-choice check)
+	  (setq specific-checks (delq check specific-checks))))
+      
+      (let ((spam-split-group spam-split-group-choice))
+	(save-excursion
+	  (save-restriction
+	    (dolist (check spam-list-of-statistical-checks)
+	      (when (and (symbolp check) (symbol-value check))
+		(widen)
+		(gnus-message 8 "spam-split: widening the buffer (%s requires it)"
+			      (symbol-name check))
+		(return)))
+	    ;;   (progn (widen) (debug (buffer-string)))
+	    (let ((list-of-checks spam-list-of-checks)
+		  decision)
+	      (while (and list-of-checks (not decision))
+		(let ((pair (pop list-of-checks)))
+		  (when (and (symbol-value (car pair))
+			     (or (null specific-checks)
+				 (memq (car pair) specific-checks)))
+		    (gnus-message 5 "spam-split: calling the %s function" 
+				  (symbol-name (cdr pair)))
+		    (setq decision (funcall (cdr pair))))))
+	      (if (eq decision t)
+		  nil
+		decision))))))))
 
 ;;; log a ham- or spam-processor invocation to the registry
 (defun spam-log-processing-to-registry (id type classification check group)

[-- Attachment #3: Type: text/plain, Size: 15 bytes --]


-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Ted Zlatanov <tzz@lifelogs.com> writes:
> That's not necessarily the only problem.  The real issue is that the
> destination is being decided dynamically, so every function in the
> splitting sequence would have to know that returning the name of the
> current group is not OK - we don't want the ham to stay in the spam
> group, no matter what.

Actually, in border cases, putting the ham right back to the spam
group is exactly what I would expect the respooling mechanism to do.

For example, with the following split rule, if I set the ham process
destination of mail.spam.silly-people to 'respool, I expect that all
ham in that group to reappear there after the resplit, each and every
time I exit that group:

	("from" "lorentey@.*elte\\.hu" "mail.spam.silly-people")

I expect the same from non-spam splitting functions.  I know, it's not
very useful, but at least it's not confusing either.

(Actually, the respool already puts ham back in a spam group sometimes
when I manually respool the articles in my mail.ham.proven group.  In
my case, disabling spam-split during the respool takes care of this
from now on, but I have not been bothered by this effect at all.)

-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Kai Grossjohann]

Josh Huber <huber+news@alum.wpi.edu> writes:

> Kai -- do you use spam.el?  I've been thinking about ways to get bayes
> spam filtering working well with a configuration like this...any
> recommendations?

Yes.  I just tell spam.el to move spams to INBOX.makespam and hams to
INBOX.makeham, and then the server-side config does the rest.

Incoming mail is passed through Bogofilter before invoking
cyrdeliver.  (I should be using LMTP but haven't figured that out,
yet.)

There are cron jobs which connect to the imap server using an
administrative account (it's a Perl script using Mail::IMAPClient).
The cron jobs train Bogofilter on user.jrl.makespam and
user.jrl.makeham.  The spam is then deleted, and the ham moved back
into the user's inbox.  (I have a separate database for each user.)

I could show the script if you like, but it's really ugly.

Kai

Re: spam.el: automatically resplitting ham in a spam group?

[Josh Huber]

Kai Grossjohann <kai@emptydomain.de> writes:

> Yes.  I just tell spam.el to move spams to INBOX.makespam and hams
> to INBOX.makeham, and then the server-side config does the rest.

Sounds nice!

> Incoming mail is passed through Bogofilter before invoking
> cyrdeliver.  (I should be using LMTP but haven't figured that out,
> yet.)

I was thinking of using the exiscan extensions inside exim to tag
messages with a header, and then users could create a sieve rule to
filter based on that header.  I guess that's about the same as you're
doing.

> There are cron jobs which connect to the imap server using an
> administrative account (it's a Perl script using Mail::IMAPClient).
> The cron jobs train Bogofilter on user.jrl.makespam and
> user.jrl.makeham.  The spam is then deleted, and the ham moved back
> into the user's inbox.  (I have a separate database for each user.)

This sounds like a nice solution.  Based on the bogofilter score, do
you sometimes reject messages?  I was thinking it would be nice if
this were tied into the Exim data acl so I could reject messages early
on, but I guess that's not a requirement.

> I could show the script if you like, but it's really ugly.

Could you send it to me privately, just so I can get a good idea of
how this is done?  I promise I won't tell anyone how ugly it is :)

-- 
Josh Huber

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Thu, 20 Nov 2003, lorentey@elte.hu wrote:

> But if I understand it well, no message would be lost; they would
> just reappear in the current group, and I think these users can
> easily fix their configuration, or simply ignore the new respooling
> feature.  Adding a note to the documentation along the lines of 'to
> prevent unexpected results, if you plan to use this feature, make
> sure that your split rules are set up so that the last rule never
> returns nil' would be enough, I guess.

Well, don't forget that nnimap is buggy with respooling - I always
see nil as the respool result, no matter what.  I will leave things
the way they are for now, and fix any obvious bugs but generally I'll
leave further work in this direction until No Gnus.

> (By the way: is it 'respool' or 'resplit'?  I don't know the correct
> terminology.)

The respooling should invoke the splitting functions, thus
resplitting the article :)

> I have played with it for a while, and after adding the
> no-spam-split-during-respool thingy, I think it does exactly what I
> need.
> 
> If you're interested, here is my change: (it's quite ugly, but
> works)

I didn't see this until I had done quite a bit of work on spam.el
in a different direction, so I'm afraid your patch won't work
cleanly.  Can you check your patch against the mega-patch I'll be
posting later tonight and see if there are any conflicts?  Thanks!

Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

Sorry it took me so long to get back to this.

Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you're interested, here is my change: (it's quite ugly, but
>> works)
>
> I didn't see this until I had done quite a bit of work on spam.el
> in a different direction, so I'm afraid your patch won't work
> cleanly.  Can you check your patch against the mega-patch I'll be
> posting later tonight and see if there are any conflicts?  Thanks!

By all means!  It seems there were no conflicts.  Here is a new patch
against the current CVS:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Optionally disable spam-split during ham respool. --]
[-- Type: text/x-patch, Size: 4313 bytes --]

Index: spam.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam.el,v
retrieving revision 6.139
diff -u -r6.139 spam.el
--- spam.el	7 Dec 2003 15:39:13 -0000	6.139
+++ spam.el	9 Dec 2003 01:58:06 -0000
@@ -119,6 +119,15 @@
   :type 'boolean
   :group 'spam)
 
+(defcustom spam-disable-spam-split-during-ham-respool nil
+  "Whether spam-split should be ignored while resplitting ham in
+a process destination.  This is useful to prevent ham from ending
+up in the same spam group after the resplit.  Don't set this to t
+if you have spam-split as the last rule in your split
+configuration."
+  :type 'boolean
+  :group 'spam)
+
 (defcustom spam-whitelist (expand-file-name "whitelist" spam-directory)
   "The location of the whitelist.
 The file format is one regular expression per line.
@@ -696,8 +705,11 @@
 	    (gnus-summary-mark-article article gnus-unread-mark))
 	  (gnus-summary-set-process-mark article))
 
-	(if respool		   ; respooling is with a "fake" group
-	    (gnus-summary-respool-article nil respool-method)
+	(if respool			   ; respooling is with a "fake" group
+	    (let ((spam-split-disabled
+		   (or spam-split-disabled
+		       spam-disable-spam-split-during-ham-respool)))
+	      (gnus-summary-respool-article nil respool-method))
 	  (if (or (not backend-supports-deletions) ; else, we are not respooling
 		  (> (length groups) 1))
 	      (progn		    ; if copying, copy and set deletep
@@ -826,6 +838,9 @@
   "The spam-list-of-statistical-checks list contains all the mail
 splitters that need to have the full message body available.")
 
+(defvar spam-split-disabled nil
+  "If non-nil, spam-split is disabled, and always returns nil.")
+
 ;;;TODO: modify to invoke self with each check if invoked without specifics
 (defun spam-split (&rest specific-checks)
   "Split this message into the `spam' group if it is spam.
@@ -836,35 +851,36 @@
 
 See the Info node `(gnus)Fancy Mail Splitting' for more details."
   (interactive)
-  (let ((spam-split-group-choice spam-split-group))
-    (dolist (check specific-checks)
-      (when (stringp check)
-	(setq spam-split-group-choice check)
-	(setq specific-checks (delq check specific-checks))))
-
-    (let ((spam-split-group spam-split-group-choice))
-      (save-excursion
-	(save-restriction
-	  (dolist (check spam-list-of-statistical-checks)
-	    (when (and (symbolp check) (symbol-value check))
-	      (widen)
-	      (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
-			    (symbol-name check))
-	      (return)))
-	  ;;   (progn (widen) (debug (buffer-string)))
-	  (let ((list-of-checks spam-list-of-checks)
-		decision)
-	    (while (and list-of-checks (not decision))
-	      (let ((pair (pop list-of-checks)))
-		(when (and (symbol-value (car pair))
-			   (or (null specific-checks)
-			       (memq (car pair) specific-checks)))
-		  (gnus-message 5 "spam-split: calling the %s function" 
-				(symbol-name (cdr pair)))
-		  (setq decision (funcall (cdr pair))))))
-	    (if (eq decision t)
-		nil
-	      decision)))))))
+  (unless spam-split-disabled
+    (let ((spam-split-group-choice spam-split-group))
+      (dolist (check specific-checks)
+	(when (stringp check)
+	  (setq spam-split-group-choice check)
+	  (setq specific-checks (delq check specific-checks))))
+      
+      (let ((spam-split-group spam-split-group-choice))
+	(save-excursion
+	  (save-restriction
+	    (dolist (check spam-list-of-statistical-checks)
+	      (when (and (symbolp check) (symbol-value check))
+		(widen)
+		(gnus-message 8 "spam-split: widening the buffer (%s requires it)"
+			      (symbol-name check))
+		(return)))
+	    ;;   (progn (widen) (debug (buffer-string)))
+	    (let ((list-of-checks spam-list-of-checks)
+		  decision)
+	      (while (and list-of-checks (not decision))
+		(let ((pair (pop list-of-checks)))
+		  (when (and (symbol-value (car pair))
+			     (or (null specific-checks)
+				 (memq (car pair) specific-checks)))
+		    (gnus-message 5 "spam-split: calling the %s function" 
+				  (symbol-name (cdr pair)))
+		    (setq decision (funcall (cdr pair))))))
+	      (if (eq decision t)
+		  nil
+		decision))))))))
 
 (defvar spam-registration-functions
   ;; first the ham register, second the spam register function

[-- Attachment #3: Type: text/plain, Size: 15 bytes --]


-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Tue, 09 Dec 2003, lorentey@elte.hu wrote:

> Sorry it took me so long to get back to this.

> By all means!  It seems there were no conflicts.  Here is a new
> patch against the current CVS:
[...]

Can you just confirm you have FSF papers?  This is a fairly large
commit.  If I've asked before, please excuse my absent-mindedness.

Thanks
Ted

Re: spam.el: automatically resplitting ham in a spam group?

[Lőrentey Károly]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Tue, 09 Dec 2003, lorentey@elte.hu wrote:
>> By all means!  It seems there were no conflicts.  Here is a new
>> patch against the current CVS:
>
> Can you just confirm you have FSF papers?  This is a fairly large
> commit.  If I've asked before, please excuse my absent-mindedness.

Yes, I have an assignment paper at FSF covering my past & future Emacs changes.

-- 
Károly

Re: spam.el: automatically resplitting ham in a spam group?

[Ted Zlatanov]

On Tue, 09 Dec 2003, lorentey@elte.hu wrote:

> Yes, I have an assignment paper at FSF covering my past & future
> Emacs changes.

Patch committed, thank you.

Ted