What's wrong with my spam.el settings now?

What's wrong with my spam.el settings now?

[Björn Lindström]

After upgrading to No Gnus (CVS of 20050719) my spam group is behaving
differently. Spam is split to nnml:spam properly, and if I enter that
group and leave, the new spam messages will still have it spam mark, and
when I leave again, it will be processed again.

What has changed that I need to compensate for?

Here are the relevant parts of my configuration:

;; Spam
(require 'spam)
(spam-initialize)

;; Prevent multiple registration of articles.
(require 'gnus-registry)
(gnus-registry-initialize)
(setq spam-log-to-registry t)

;; Use bogofilter for spam processing.
(setq spam-use-bogofilter t)

;; Put incoming spam in "spam".
(setq spam-split-group "spam")

;; Set spam and ham groups.
(setq gnus-spam-newsgroup-contents
      '(("^nnml:spam" gnus-group-spam-classification-spam)
        ("^nnml:.*" gnus-group-spam-classification-ham)))

;; Train the filter.
(setq spam-process-ham-in-spam-groups t
      gnus-spam-process-newsgroups
      '(("^nntp\\+news\\.gmane\\.org:"
         ((spam spam-use-gmane)))
        ("^nnml:.*"
         ((spam spam-use-bogofilter)))))

;; Move processed spam.
(setq spam-move-spam-nonspam-groups-only t)
(setq spam-mark-only-unseen-as-spam t)
(setq gnus-spam-process-destinations
      '(("^nnml:.*" "nnml:spam")))

;; Move processed ham.
(setq spam-mark-ham-unread-before-move-from-spam-group t)
(setq gnus-ham-process-destinations
      '(("^nnml:spam" "nnml:reclassify")))

;; Splitting
(setq '(|
         (: spam-split)
         ...))

Re: What's wrong with my spam.el settings now?

[Ted Zlatanov]

On Sun, 07 Aug 2005, bkhl@elektrubadur.se wrote:

> After upgrading to No Gnus (CVS of 20050719) my spam group is behaving
> differently. Spam is split to nnml:spam properly, and if I enter that
> group and leave, the new spam messages will still have it spam mark, and
> when I leave again, it will be processed again.

Hi Bjorn,

first of all, sorry for the very late response to you and all the
other people with spam.el questions (especially Reiner Steib).  I was
unable to read ding or contribute to Gnus until now.

I see from your configuration that spam-mark-only-unseen-as-spam is
set.  You should only get messages marked as spam once, when you enter
the group and they are unseen.  After that, they will be unread but
not unseen, so they shouldn't be marked as spam again.  To confirm,
when you enter the second time, is there a dot (.) to the left of the
spam-marked article in the summary line?  If yes, Gnus is not saving
the marks of the articles, that's why you see them marked as spam
again.

If no, set gnus-verbose to 10 and send me the output in *Messages*
when you enter and exit the group twice.  Let me know if you need more
help.

Thanks!
Ted


> What has changed that I need to compensate for?
> 
> Here are the relevant parts of my configuration:
> 
> ;; Spam
> (require 'spam)
> (spam-initialize)
> 
> ;; Prevent multiple registration of articles.
> (require 'gnus-registry)
> (gnus-registry-initialize)
> (setq spam-log-to-registry t)
> 
> ;; Use bogofilter for spam processing.
> (setq spam-use-bogofilter t)
> 
> ;; Put incoming spam in "spam".
> (setq spam-split-group "spam")
> 
> ;; Set spam and ham groups.
> (setq gnus-spam-newsgroup-contents
>       '(("^nnml:spam" gnus-group-spam-classification-spam)
>         ("^nnml:.*" gnus-group-spam-classification-ham)))
> 
> ;; Train the filter.
> (setq spam-process-ham-in-spam-groups t
>       gnus-spam-process-newsgroups
>       '(("^nntp\\+news\\.gmane\\.org:"
>          ((spam spam-use-gmane)))
>         ("^nnml:.*"
>          ((spam spam-use-bogofilter)))))
> 
> ;; Move processed spam.
> (setq spam-move-spam-nonspam-groups-only t)
  > (setq spam-mark-only-unseen-as-spam t)
> (setq gnus-spam-process-destinations
>       '(("^nnml:.*" "nnml:spam")))
> 
> ;; Move processed ham.
> (setq spam-mark-ham-unread-before-move-from-spam-group t)
> (setq gnus-ham-process-destinations
>       '(("^nnml:spam" "nnml:reclassify")))
> 
> ;; Splitting
> (setq '(|
>          (: spam-split)
>          ...))

Re: What's wrong with my spam.el settings now?

[Björn Lindström]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Sun, 07 Aug 2005, bkhl@elektrubadur.se wrote:
>
>> After upgrading to No Gnus (CVS of 20050719) my spam group is
>> behaving differently. Spam is split to nnml:spam properly, and if I
>> enter that group and leave, the new spam messages will still have it
>> spam mark, and when I leave again, it will be processed again.
>
> first of all, sorry for the very late response to you and all the
> other people with spam.el questions (especially Reiner Steib).  I was
> unable to read ding or contribute to Gnus until now.

No problem. This isn't the first time I'm asking questions about spam.el
in here, so I know you always get around to writing a good response. :-)

> I see from your configuration that spam-mark-only-unseen-as-spam is
> set.  You should only get messages marked as spam once, when you enter
> the group and they are unseen.  After that, they will be unread but
> not unseen, so they shouldn't be marked as spam again.  To confirm,
> when you enter the second time, is there a dot (.) to the left of the
> spam-marked article in the summary line?  If yes, Gnus is not saving
> the marks of the articles, that's why you see them marked as spam
> again.

Yes. Here's a typical line:

$. [   2: <som999.@yahoo.com> ] ¢ÒÂÊè§à¤Ã×èͧàÊÕ§µÔ´Ã¶Â¹µì

> If no, set gnus-verbose to 10 and send me the output in *Messages*
> when you enter and exit the group twice.  Let me know if you need more
> help.

Here it is. To make the output more manageable, I opened the group twice
with M-1 RET. What do you make of this?

Retrieving newsgroup: nnml:spam...
Fetching headers for nnml:spam...done
Scoring...done
Sorting articles...done
Generating summary...done
Registry: Registering article 11220 with group nnml:spam
Marking unseen articles as spam
No more unread articles
Exiting summary buffer and applying spam rules
1 messages were saved the trouble of unregistering and then registering
Registering 1 specific articles as spam using backend spam-use-move
Marking spam article 11220 expirable before move
Moving article 11220 to group nnml:spam
Moving to nnml:spam: (11220)...
Wrote /home/bkhl/Mail/spam/11221
Registry: article <E1EHl5t-0004Ae-8C@lucien.dreaming> going from nnml:spam to nnml:spam
1 spam messages were registered by backend spam-use-move.
Expiring articles...
Registry: article <E1EHl5t-0004Ae-8C@lucien.dreaming> going from nnml:spam to nil
Expiring articles...done
No more unread newsgroups
Retrieving newsgroup: nnml:spam...
Fetching headers for nnml:spam...done
Scoring...done
Sorting articles...done
Generating summary...done
Registry: Registering article 11221 with group nnml:spam
Marking unseen articles as spam
No more unread articles
Exiting summary buffer and applying spam rules
1 messages were saved the trouble of unregistering and then registering
Registering 1 specific articles as spam using backend spam-use-move
Marking spam article 11221 expirable before move
Moving article 11221 to group nnml:spam
Moving to nnml:spam: (11221)...
Wrote /home/bkhl/Mail/spam/11222
Registry: article <E1EHl5t-0004Ae-8C@lucien.dreaming> going from nnml:spam to nnml:spam
1 spam messages were registered by backend spam-use-move.
Expiring articles...
Registry: article <E1EHl5t-0004Ae-8C@lucien.dreaming> going from nnml:spam to nil
Expiring articles...done

Re: What's wrong with my spam.el settings now?

[Ted Zlatanov]

On Tue, 20 Sep 2005, bkhl@elektrubadur.se wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:

>> I see from your configuration that spam-mark-only-unseen-as-spam is
>> set.  You should only get messages marked as spam once, when you enter
>> the group and they are unseen.  After that, they will be unread but
>> not unseen, so they shouldn't be marked as spam again.  To confirm,
>> when you enter the second time, is there a dot (.) to the left of the
>> spam-marked article in the summary line?  If yes, Gnus is not saving
>> the marks of the articles, that's why you see them marked as spam
>> again.
> 
> Yes. Here's a typical line:
> 
> $. [   2: <som999.@yahoo.com> ] ¢ÒÂÊè§à¤Ã×èͧàÊÕ§µÔ´Ã¶Â¹µì

If you see this on the second entry on an article that was already in
the group on the first entry, try another experiment: mark an article
with `!' in the summary buffer, then exit.  If the ! mark persists,
then the unread/unseen state is not preserved correctly, and we need
to bug the maintainer of the nnml backend.  If the ! mark does not
persist, the marks are not preserved, and we'll still bug the
maintainer :)

Thanks for your patience.

Ted

Re: What's wrong with my spam.el settings now?

[Björn Lindström]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> If you see this on the second entry on an article that was already in
> the group on the first entry, try another experiment: mark an article
> with `!' in the summary buffer, then exit.  If the ! mark persists,
> then the unread/unseen state is not preserved correctly, and we need
> to bug the maintainer of the nnml backend.

That's what happens. Who's that then?

Re: What's wrong with my spam.el settings now?

[Ted Zlatanov]

On Wed, 21 Sep 2005, bkhl@elektrubadur.se wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
> 
>> If you see this on the second entry on an article that was already in
>> the group on the first entry, try another experiment: mark an article
>> with `!' in the summary buffer, then exit.  If the ! mark persists,
>> then the unread/unseen state is not preserved correctly, and we need
>> to bug the maintainer of the nnml backend.
> 
> That's what happens. Who's that then?

nnml.el says:

;; Authors: Didier Verna <didier@xemacs.org> (adding compaction)
;;      Simon Josefsson <simon@josefsson.org> (adding MARKS)
;;      Lars Magne Ingebrigtsen <larsi@gnus.org>
;;      Masanobu UMEDA <umerin@flab.flab.fujitsu.junet>

I would guess that Simon would be a great person to contact, since
this seems marks-related.

Ted

Re: What's wrong with my spam.el settings now?

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Wed, 21 Sep 2005, bkhl@elektrubadur.se wrote:
>
>> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>> 
>>> If you see this on the second entry on an article that was already in
>>> the group on the first entry, try another experiment: mark an article
>>> with `!' in the summary buffer, then exit.  If the ! mark persists,
>>> then the unread/unseen state is not preserved correctly, and we need
>>> to bug the maintainer of the nnml backend.
>> 
>> That's what happens. Who's that then?
>
> nnml.el says:
>
> ;; Authors: Didier Verna <didier@xemacs.org> (adding compaction)
> ;;      Simon Josefsson <simon@josefsson.org> (adding MARKS)
> ;;      Lars Magne Ingebrigtsen <larsi@gnus.org>
> ;;      Masanobu UMEDA <umerin@flab.flab.fujitsu.junet>
>
> I would guess that Simon would be a great person to contact, since
> this seems marks-related.

How is spam.el setting the spam mark?  In particular, is it using the
gnus-request-set-mark interface?  That interface must be called with
the flag update whenever any code that alter message flags without
going through the summary buffer variables (which are turned into
gnus-request-set-mark calls when exiting the group).

I suspect what is happening is that spam.el set the spam flag without
calling that function, and then the next time you enter the group, the
nnml marks code will revert the flag to what is stored under
~/News/nnml/ because it believes those marks to be more recent.

If the spam mark should never be synced across Gnus installations,
then it should be added to gnus-article-unpropagated-mark-lists, then
the nnml code will not revert the spam flag.  However, it seems like
the spam mark should be synchronized between Gnus installations,
should it not?  Then the spam.el code has to call the g-r-s-m
interface when updating the marks.

I didn't actually read any code to answer this, and my memory of all
this may be failing.

Re: What's wrong with my spam.el settings now?

[Ted Zlatanov]

On Wed, 21 Sep 2005, jas@extundo.com wrote:

> How is spam.el setting the spam mark?  In particular, is it using the
> gnus-request-set-mark interface?  That interface must be called with
> the flag update whenever any code that alter message flags without
> going through the summary buffer variables (which are turned into
> gnus-request-set-mark calls when exiting the group).

I do it with gnus-summary-mark-article:

5 matches for "mark-article" in buffer: spam.el
   1325:	    (gnus-summary-mark-article article gnus-spam-mark))
   1387:      (gnus-summary-mark-article article gnus-expirable-mark)))
   1427:	    (gnus-summary-mark-article article gnus-unread-mark))
   1432:	    (gnus-summary-mark-article article gnus-expirable-mark))
   1708:		 (gnus-summary-mark-article article gnus-spam-mark))

which worked fine so far.

If that's wrong, what's the right way to do it (if you can point me to
code that does the right thing, I'll make sure to implement mine the
same way)?

> If the spam mark should never be synced across Gnus installations,
> then it should be added to gnus-article-unpropagated-mark-lists, then
> the nnml code will not revert the spam flag.  However, it seems like
> the spam mark should be synchronized between Gnus installations,
> should it not?  Then the spam.el code has to call the g-r-s-m
> interface when updating the marks.

The spam mark is ephemeral, it only matters for the duration of a
summary visit.  When you exit, spam-marked articles get processed and
expired.  I did this intentionally when I started writing the spam.el
logic.

Ted

Re: What's wrong with my spam.el settings now?

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Wed, 21 Sep 2005, jas@extundo.com wrote:
>
>> How is spam.el setting the spam mark?  In particular, is it using the
>> gnus-request-set-mark interface?  That interface must be called with
>> the flag update whenever any code that alter message flags without
>> going through the summary buffer variables (which are turned into
>> gnus-request-set-mark calls when exiting the group).
>
> I do it with gnus-summary-mark-article:
>
> 5 matches for "mark-article" in buffer: spam.el
>    1325:	    (gnus-summary-mark-article article gnus-spam-mark))
>    1387:      (gnus-summary-mark-article article gnus-expirable-mark)))
>    1427:	    (gnus-summary-mark-article article gnus-unread-mark))
>    1432:	    (gnus-summary-mark-article article gnus-expirable-mark))
>    1708:		 (gnus-summary-mark-article article gnus-spam-mark))
>
> which worked fine so far.
>
> If that's wrong, what's the right way to do it (if you can point me to
> code that does the right thing, I'll make sure to implement mine the
> same way)?
>
>> If the spam mark should never be synced across Gnus installations,
>> then it should be added to gnus-article-unpropagated-mark-lists, then
>> the nnml code will not revert the spam flag.  However, it seems like
>> the spam mark should be synchronized between Gnus installations,
>> should it not?  Then the spam.el code has to call the g-r-s-m
>> interface when updating the marks.
>
> The spam mark is ephemeral, it only matters for the duration of a
> summary visit.  When you exit, spam-marked articles get processed and
> expired.  I did this intentionally when I started writing the spam.el
> logic.

Ok, this all looks good.  I misunderstood the original report.  The
problem seem to be with the seen/unseen mark, not the spam mark.
Björn, how are you quitting the group?  Are you exiting it with 'q' to
save the marks?  If you type G E on the group before entering, and
save the list of articles associated with 'seen', then enter the group
and read a few articles and quit the group using 'q', and press G E
again, do the 'seen' list look the same?

What is your general setup?  Is your ~/News/marks directory read-only
somehow?

I don't recall any reports of seen/unseen troubles like this at all.

Re: What's wrong with my spam.el settings now?

[Björn Lindström]

Simon Josefsson <jas@extundo.com> writes:

> Björn, how are you quitting the group? Are you exiting it with 'q' to
> save the marks?

Yes, that's what I did in the test I demonstrated earlier.

> If you type G E on the group before entering, and save the list of
> articles associated with 'seen', then enter the group and read a few
> articles and quit the group using 'q', and press G E again, do the
> 'seen' list look the same?

They change a little bit.

If I do GE on the spam group, it can look like this:

 ((seen [...]
        (11842 . 11843)))

If I enter the group with M-1 RET, I see one article with a spam mark. I
then leave the group with q, and the last element of the seen list has
changed so that:

 ((seen [...]
        (11842 . 11844)))

> What is your general setup?  Is your ~/News/marks directory read-only
> somehow?

Nope. It's just as Gnus created it, on a local file system.

Re: What's wrong with my spam.el settings now?

[Simon Josefsson]

bkhl@elektrubadur.se (Björn Lindström) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Björn, how are you quitting the group? Are you exiting it with 'q' to
>> save the marks?
>
> Yes, that's what I did in the test I demonstrated earlier.
>
>> If you type G E on the group before entering, and save the list of
>> articles associated with 'seen', then enter the group and read a few
>> articles and quit the group using 'q', and press G E again, do the
>> 'seen' list look the same?
>
> They change a little bit.
>
> If I do GE on the spam group, it can look like this:
>
>  ((seen [...]
>         (11842 . 11843)))
>
> If I enter the group with M-1 RET, I see one article with a spam mark. I
> then leave the group with q, and the last element of the seen list has
> changed so that:
>
>  ((seen [...]
>         (11842 . 11844)))

This information and the information earlier in the thread doesn't add
up for me.  Earlier I interpreted the problem that seen marks were
visible on articles the second time you entered a group with that
article.  Did I misunderstand that?  Is that still the problem?  If
the seen list is updated as above, then the summary buffer shouldn't
have a "unseen" mark on the 11844 article upon entering the group the
second time.  Did it?

I'm not sure how this problem relate to the spam mark, but presumably
it is because spam.el use the seen mark to decide which articles to
mark as spam?

Re: What's wrong with my spam.el settings now?

[Björn Lindström]

Simon Josefsson <jas@extundo.com> writes:

> This information and the information earlier in the thread doesn't add
> up for me.  Earlier I interpreted the problem that seen marks were
> visible on articles the second time you entered a group with that
> article.  Did I misunderstand that?  Is that still the problem?  If
> the seen list is updated as above, then the summary buffer shouldn't
> have a "unseen" mark on the 11844 article upon entering the group the
> second time.  Did it?

It keeps the unseen dot, even if I repeatedly enter and exit the group.

Re: What's wrong with my spam.el settings now?

[Ted Zlatanov]

On Fri, 30 Sep 2005, jas@extundo.com wrote:

> This information and the information earlier in the thread doesn't add
> up for me.  Earlier I interpreted the problem that seen marks were
> visible on articles the second time you entered a group with that
> article.  Did I misunderstand that?  Is that still the problem?  If
> the seen list is updated as above, then the summary buffer shouldn't
> have a "unseen" mark on the 11844 article upon entering the group the
> second time.  Did it?
> 
> I'm not sure how this problem relate to the spam mark, but presumably
> it is because spam.el use the seen mark to decide which articles to
> mark as spam?

In spam groups, spam.el will spam-mark unseen articles only (unless
spam-mark-only-unseen-as-spam is nil, which does not apply here
because Bjorn reports the unseen mark is returning).  So the problem
seems to be the unseen mark's return, not the spam.el code.

Ted