Splitting on mail contents? (using spam-stat.el)

Splitting on mail contents? (using spam-stat.el)

[Oystein Viggen]

Hi

I've been playing with Alex Schroeder's spam-stat.el, getting quite good
results on the dry run tests, and would like to try using it in my split
rules.  The problem is that I can't figure out how to get at the
complete mail and not just the headers from a function in
nnmail-split-methods.

I don't use fancy splitting, and gnus-original-article-buffer, which is
used in spam-stat-split-fancy seems not to be updated during splitting,
and will just contain the last article viewed.  Will I need to do some
very fancy widening in my function, or is there a standard way?

I'd like to have something like this:

(setq nnmail-split-methods 
  '(("mail.spam-stat" my-spam-stat-non-fancy-split)
    ;; other rules here
    ("mail.misc" "")))

My-spam-stat-non-fancy-split should run spam-stat-score-buffer on the
complete mail, both the headers and the contents.  Can this be done?

Oystein
-- 
When in doubt: Think again.

Re: Splitting on mail contents? (using spam-stat.el)

[Kai Großjohann]

Oystein Viggen <oysteivi@tihlde.org> writes:

> (setq nnmail-split-methods 
>   '(("mail.spam-stat" my-spam-stat-non-fancy-split)
>     ;; other rules here
>     ("mail.misc" "")))

Regular splitting can't run functions, I think.  I suggest that you
switch to fancy splitting.  I find that even cases where regular
splitting is sufficient can benefit from fancy splitting, because the
fancy rules are easier to write.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting on mail contents? (using spam-stat.el)

[Oystein Viggen]

* [Kai Großjohann] 

> Regular splitting can't run functions, I think.

You're wrong there.  From C-h v nnmail-split-methods:
,----
| The second element can also be a function.  In that case, it will be
| called narrowed to the headers with the first element of the rule as
| the argument.  It should return a non-nil value if it thinks that the
| mail belongs in that group.
`----

I've tested this, and the function is called as specified.

> I suggest that you switch to fancy splitting.

Maybe I should, but can functions called from fancy splitting easily
access the whole article?  It seems I would want to use (: FUNCTION),
but according to the documentation, this function will be called "in the
buffer containing the message headers".

I see that spam-stat-split-fancy uses gnus-original-article-buffer.
Does nnmail-split-fancy perhaps store the article in this buffer when
splitting?  I'll have to investigate that.

My nnmail-split-methods is about 100 lines long, and uses a few home
grown hacks in addition, so I don't want to do the big job of converting
unless I'm pretty sure something will be gained by it :)

Oystein
-- 
Ebg13 arire tbrf bhg bs fglyr..

Re: Splitting on mail contents? (using spam-stat.el)

[Kai Großjohann]

Oystein Viggen <oysteivi@tihlde.org> writes:

> * [Kai Großjohann] 
>
>> Regular splitting can't run functions, I think.
>
> You're wrong there.  From C-h v nnmail-split-methods:
> ,----
> | The second element can also be a function.  In that case, it will be
> | called narrowed to the headers with the first element of the rule as
> | the argument.  It should return a non-nil value if it thinks that the
> | mail belongs in that group.
> `----
>
> I've tested this, and the function is called as specified.

OK, sorry, then.

>> I suggest that you switch to fancy splitting.
>
> Maybe I should, but can functions called from fancy splitting easily
> access the whole article?  It seems I would want to use (: FUNCTION),
> but according to the documentation, this function will be called "in the
> buffer containing the message headers".
>
> I see that spam-stat-split-fancy uses gnus-original-article-buffer.
> Does nnmail-split-fancy perhaps store the article in this buffer when
> splitting?  I'll have to investigate that.

Well, I remember that Gnus uses two different buffers for splitting
and for `B r'.  Beware!

> My nnmail-split-methods is about 100 lines long, and uses a few home
> grown hacks in addition, so I don't want to do the big job of converting
> unless I'm pretty sure something will be gained by it :)

Maybe not...

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting on mail contents? (using spam-stat.el)

[Oystein Viggen]

* [Kai Großjohann] 

> Well, I remember that Gnus uses two different buffers for splitting
> and for `B r'.  Beware!

Ah, thanks for the warning.  I'll go read some more code now  :)

Oystein
-- 
When in doubt: Recompile.

Re: Splitting on mail contents? (using spam-stat.el)

[Jeremy H. Brown]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> Well, I remember that Gnus uses two different buffers for splitting
> and for `B r'.  Beware!

I use advice to put a uniform name on the entire article buffer for
fancy-splits in nnmail.  Below is a (simplified, untested) version of
my advice; using it, the article-buffer is named the-article-buffer in
your split and any functions you may call from it.

This works with 5.8.8/5.9; I have no idea how it does with oort.

Jeremy


(defadvice nnmail-article-group (around name-article-buffer (&rest args))
  "name the article buffer for use in fancy splits"
  (let ((the-article-buffer (current-buffer)))
    ad-do-it))

(ad-activate-regexp "name-article-buffer")

Re: Splitting on mail contents? (using spam-stat.el)

[Oystein Viggen]

* [Jeremy H. Brown] 

> This works with 5.8.8/5.9; I have no idea how it does with oort.

This seems to work in Oort, too.  Thanks a lot!

Oystein
-- 
When in doubt: Recompile.

finding the article buffer when splitting

[Alex Schroeder]

jhbrown@ai.mit.edu (Jeremy H. Brown) writes:

> (defadvice nnmail-article-group (around name-article-buffer (&rest args))
>   "name the article buffer for use in fancy splits"
>   (let ((the-article-buffer (current-buffer)))
>     ad-do-it))
>
> (ad-activate-regexp "name-article-buffer")

Is anything like that going to be in Gnus?  This seems to be really
easy to implement.  Otherwise, we might consider adding the following
macro:

(defmacro with-article-buffer (&rest body)
  "Make the article buffer current while executing BODY.
This is useful when splitting mail with `nnmail-split-fancy'.
The article buffer is stored either in `gnus-original-article-buffer'
or `nnmail-article-buffer'."
  `(save-excursion
     (cond ((and (boundp 'gnus-original-article-buffer)
		 (gnus-buffer-live-p gnus-original-article-buffer))
	    (set-buffer gnus-original-article-buffer)
	    ,@body)
	   ((and (boundp 'nnmail-article-buffer)
		 (gnus-buffer-live-p nnmail-article-buffer))
	    (set-buffer nnmail-article-buffer)
	    ,@body)
	   (t
	    (error "There is no article buffer available.")))))

Alex.
-- 
I will be in Istanbul next week.

Re: finding the article buffer when splitting

[Oystein Viggen]

* [Alex Schroeder] 

> Is anything like that going to be in Gnus?  This seems to be really
> easy to implement.  Otherwise, we might consider adding the following
> macro:

Nice idea, but one objection:

In my Gnus, gnus-original-article-buffer seems to always contain the
last article viewed, and not the one currently spooling.  This makes the
" *Original Article*" buffer kind of useless for splitting incoming
mail.

On a smallish test, I didn't seem to get the "using bogus group" problem
when using your with-article-buffer macro, however.

I'm currently using Jeremy's defadvice combined with a slightly modified
spam-stat-split-fancy (set-buffer the-article-buffer instead of
gnus-original-article-buffer), which sorts correctly but gives a high
bogus ratio.  (but it works nicely when marking everything in the bogus
group and respooling)

Oystein
-- 
Ebg13 arire tbrf bhg bs fglyr..

Re: finding the article buffer when splitting

[Alex Schroeder]

Oystein Viggen <oysteivi@tihlde.org> writes:

> In my Gnus, gnus-original-article-buffer seems to always contain the
> last article viewed, and not the one currently spooling.  This makes the
> " *Original Article*" buffer kind of useless for splitting incoming
> mail.

Hm, I think in the summary buffer:

B t runs the command gnus-summary-respool-trace
   which is an interactive compiled Lisp function in `gnus-sum'.

which calls gnus-summary-respool-query
which uses gnus-original-article-buffer.

:)

> I'm currently using Jeremy's defadvice combined with a slightly modified
> spam-stat-split-fancy (set-buffer the-article-buffer instead of
> gnus-original-article-buffer), which sorts correctly but gives a high
> bogus ratio.  (but it works nicely when marking everything in the bogus
> group and respooling)

Hehe.  I got rid of the bogus problem now, so I think you're best of
using my latest copy of spam-stat.el.  I used Jeremy's advice with the
exact same change you listed.  If you just want to adapt that, you
need to change the call (buffer-size) to (- (point-max) (point-min))
-- because buffer-size does not take narrowing into account.  The
resulting numbers caused errors, which resulted in bogus group
splitting.

Alex.
-- 
I will be in Istanbul next week.

Re: finding the article buffer when splitting

[Oystein Viggen]

* [Alex Schroeder] 

> Hehe.  I got rid of the bogus problem now, so I think you're best of
> using my latest copy of spam-stat.el. 

Great, I'll give it a whirl.  The latest version is 0.2.1, right?
(That's the newest one I could find in gnu.emacs.sources).

Oystein
-- 
If it ain't broke, don't break it.

Re: finding the article buffer when splitting

[Jeremy H. Brown]

Alex Schroeder <alex@emacswiki.org> writes:

> jhbrown@ai.mit.edu (Jeremy H. Brown) writes:
> 
> > (defadvice nnmail-article-group (around name-article-buffer (&rest args))
> >   "name the article buffer for use in fancy splits"
> >   (let ((the-article-buffer (current-buffer)))
> >     ad-do-it))
> >
> > (ad-activate-regexp "name-article-buffer")
> 
> Is anything like that going to be in Gnus?  This seems to be really
> easy to implement.  Otherwise, we might consider adding the following
> macro:
> 
[snip]

Having thought about this more, I think what we really want is not a
variable gnus-the-article-buffer, but a function
(gnus-current-article-buffer) which returns a buffer containing the
current article.  We want a function because if you're using the imap
backend, normal splitting functions don't download the whole article,
just the headers; this function can download the article body if
necessary, and otherwise just immediately return the buffer with the
body in it.

Jeremy

Re: finding the article buffer when splitting

[Alex Schroeder]

jhbrown@ai.mit.edu (Jeremy H. Brown) writes:

> Having thought about this more, I think what we really want is not a
> variable gnus-the-article-buffer, but a function
> (gnus-current-article-buffer) which returns a buffer containing the
> current article.  We want a function because if you're using the
> imap backend, normal splitting functions don't download the whole
> article, just the headers; this function can download the article
> body if necessary, and otherwise just immediately return the buffer
> with the body in it.

Hm, my spam-stat.el still has some problems.  I use the following
macro, and now found message that was considered spam when splitting,
and not-spam when respooling.  To me, that suggests that something in
this article buffer business is still malfunctioning.

Alex.


(defmacro with-article-buffer (&rest body)
    "Make the article buffer current while executing BODY.
This is useful when splitting mail with `nnmail-split-fancy'.
The article buffer is stored either in `gnus-original-article-buffer'
or `nnmail-article-buffer'."
    `(save-excursion
       (cond ((and (boundp 'gnus-original-article-buffer)
		   (gnus-buffer-live-p gnus-original-article-buffer))
	      (set-buffer gnus-original-article-buffer)
	      ,@body)
	     ((and (boundp 'nnmail-article-buffer)
		   (gnus-buffer-live-p nnmail-article-buffer))
	      (set-buffer nnmail-article-buffer)
	      ,@body)
	     (t
	      (error "There is no article buffer available.")))))

Re: finding the article buffer when splitting

[Oystein Viggen]

* [Alex Schroeder] 

> Hm, my spam-stat.el still has some problems.  I use the following
> macro, and now found message that was considered spam when splitting,
> and not-spam when respooling.  To me, that suggests that something in
> this article buffer business is still malfunctioning.

I believe this is because your code still prefers to use
gnus-original-article-buffer, which is not updated when spooling new
mail.  View a spam before you spool in some new mail from your mailbox,
and I think you will find all your new mail is considered spam.  
(Note: Bt and Br do not count as spooling in new mail - Send yourself
some test mails instead).

There are a few possible workarounds:
1. Kill gnus-original-article-buffer from gnus-get-new-news-hook
2. Check for nnmail-article-buffer before gnus-original-article-buffer
in your code.
3. Use the defadvice posted by Jeremy H. Brown in
< news:uv6r8g9n5id.fsf@suspiria.ai.mit.edu > and change
spam-stat-split-fancy to look in the-article-buffer.
(This last one is what I use).

spam-stat.el is now my only spam filter.  It Just Works. :)

Oystein
-- 
If it ain't broke, don't break it.

Re: Splitting on mail contents? (using spam-stat.el)

[Alex Schroeder]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> I see that spam-stat-split-fancy uses gnus-original-article-buffer.
>> Does nnmail-split-fancy perhaps store the article in this buffer when
>> splitting?  I'll have to investigate that.
>
> Well, I remember that Gnus uses two different buffers for splitting
> and for `B r'.  Beware!

Yeah, I am still trying to find a bug in my code that sends about 1%
of all mail into the bogus group.  The problem is that I cannot debug
this (because it only happens in 1% of the cases, and errors are
probably caught using condition-case).  Just today I switched to the
following definition, hoping to finally find the bug. 

(defun spam-stat-split-fancy ()
  "Return the name of the spam group if the current mail is spam.
Use this function on `nnmail-split-fancy'.  If you are interested in
the raw data used for the last run of `spam-stat-score-buffer',
check the variable `spam-stat-score-data'."
  (condition-case var
      (save-excursion
	(set-buffer (if (boundp 'gnus-original-article-buffer)
			gnus-original-article-buffer
		      (message "Original article buffer not found.")
		      (current-buffer)))
	(goto-char (point-min))
	(if (> (spam-stat-score-buffer) 0.9)
	    (progn
	      (message "Spam-stat says this is spam!")
	      spam-stat-split-fancy-spam-group)
	  (message "Spam-stat says this is no spam.")))
    (error (message "Spam-stat found an error: %S" var))))

Alex.

Re: Splitting on mail contents? (using spam-stat.el)

[Alex Schroeder]

Alex Schroeder <alex@emacswiki.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>>> I see that spam-stat-split-fancy uses gnus-original-article-buffer.
>>> Does nnmail-split-fancy perhaps store the article in this buffer when
>>> splitting?  I'll have to investigate that.
>>
>> Well, I remember that Gnus uses two different buffers for splitting
>> and for `B r'.  Beware!
>
> Yeah, I am still trying to find a bug in my code that sends about 1%
> of all mail into the bogus group.

You were totally right (once again), Kai!

Here is from my *Messages* buffer.

Wrote /home/alex/Mail/mail/emacs/devel/5431
Spam-stat found an error: (error "No buffer named  *Original Article*")
Creating mail directory /home/alex/Mail/Spam-stat found an error: (error "No buffer named  *Original Article*")/
Wrote /home/alex/Mail/Spam-stat found an error: (error "No buffer named  *Original Article*")/1
Wrote /home/alex/Mail/mail/spam/1043

Now what would be the correct way of dealing with this problem?  Would
to real solution not be to change Gnus such that only one buffer is
used, and advertize it in the manual?

Alex.

Re: Splitting on mail contents? (using spam-stat.el)

[Simon Josefsson]

Alex Schroeder <alex@emacswiki.org> writes:

> Now what would be the correct way of dealing with this problem?  Would
> to real solution not be to change Gnus such that only one buffer is
> used, and advertize it in the manual?

Yes.  The buffer name should probably be stored in a variable.  Only
problem would be that people are already using the two hard coded
buffer names... having two variables and keeping the two buffer names
seems like a possibility but perhaps it is not that important?  Having
one buffer name would be better.

Re: Splitting on mail contents? (using spam-stat.el)

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Yes.  The buffer name should probably be stored in a variable.  Only
> problem would be that people are already using the two hard coded
> buffer names... having two variables and keeping the two buffer names
> seems like a possibility but perhaps it is not that important?  Having
> one buffer name would be better.

The variable could be updated to contain the right value at the right
times...

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting on mail contents? (using spam-stat.el)

[Kai Großjohann]

Alex Schroeder <alex@emacswiki.org> writes:

> Now what would be the correct way of dealing with this problem?  Would
> to real solution not be to change Gnus such that only one buffer is
> used, and advertize it in the manual?

Pretty please!

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Splitting on mail contents? (using spam-stat.el)

[Paul Jarc]

Oystein Viggen <oysteivi@tihlde.org> wrote:
> The problem is that I can't figure out how to get at the
> complete mail and not just the headers from a function in
> nnmail-split-methods.

Have you tried (save-restriction (widen) (your-code-here))?  I'm
guessing the buffer contains the whole article, but is narrowed to
just the headers.


paul