Fancy splitting getting headers from included text

Fancy splitting getting headers from included text

[David Z Maze]

Occasionally I get a forwarded message.  When nnmail-split-fancy looks
at the message, it seems to get its headers from the body of the
message, so the message will get sorted into the mail group
corresponding to the list it was forwarded from, possibly in addition
to the group it was forwarded too.  I haven't looked in detail at
whether this involves MIME-forwarded messages or other mechanisms, and
whether the normal headers are involved; I can if it would help track
down the problem.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Fancy splitting getting headers from included text

[Niklas Morberg]

David Z Maze <dmaze@MIT.EDU> writes:

> Occasionally I get a forwarded message.  When nnmail-split-fancy looks
> at the message, it seems to get its headers from the body of the
> message, so the message will get sorted into the mail group
> corresponding to the list it was forwarded from, possibly in addition
> to the group it was forwarded too.  I haven't looked in detail at
> whether this involves MIME-forwarded messages or other mechanisms, and
> whether the normal headers are involved; I can if it would help track
> down the problem.

It sounds like the same problem I am having, see
<URL:http://article.gmane.org/gmane.emacs.gnus.general/50831>.

I assumed it had something to do with nnimap, but your problem
report indicates all mail backends might be affected.

Niklas

Re: Fancy splitting getting headers from included text

[Lars Magne Ingebrigtsen]

David Z Maze <dmaze@MIT.EDU> writes:

> Occasionally I get a forwarded message.  When nnmail-split-fancy looks
> at the message, it seems to get its headers from the body of the
> message,

It shouldn't -- the buffer should be limited to the headers only.
(Unless somebody changed that while I wasn't looking.  If so, they
should change it back again.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Fancy splitting getting headers from included text

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It shouldn't -- the buffer should be limited to the headers only.
> (Unless somebody changed that while I wasn't looking.  If so, they
> should change it back again.  :-)

Some people needed the body for spam processing.  There is a variable
for this.  Hmm...  Ah.  nnimap-split-download-body, perhaps?
-- 
A preposition is not a good thing to end a sentence with.

Re: Fancy splitting getting headers from included text

[Simon Josefsson]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> It shouldn't -- the buffer should be limited to the headers only.
>> (Unless somebody changed that while I wasn't looking.  If so, they
>> should change it back again.  :-)
>
> Some people needed the body for spam processing.  There is a variable
> for this.  Hmm...  Ah.  nnimap-split-download-body, perhaps?

But it is off by default, and it is easy to see if the body is
incorrectly downloaded anyway in *imap-log*.

I think it was reported to happen for nnmail too.  Hm.  Those who are
seeing this, are you using spam.el and downloading bodies too?  Could
you edebug nnmail-article-group to see if nnmail-split-fancy is
somehow invoked without narrowed headers?  I'm not sure where the spam
code is invoked, but if it does a (widen) it should probably do it
inside a save-restriction.  Does this patch help?

--- spam.el.~6.87.~	Thu Mar 20 00:10:30 2003
+++ spam.el	Sun Mar 30 14:08:05 2003
@@ -595,7 +595,7 @@
 
 See the Info node `(gnus)Fancy Mail Splitting' for more details."
   (interactive)
-
+  (save-excursion
   (dolist (check spam-list-of-statistical-checks)
     (when (symbol-value check)
       (widen)
@@ -612,7 +612,7 @@
 	  (setq decision (funcall (cdr pair))))))
     (if (eq decision t)
 	nil
-      decision)))
+	decision))))
 
 (defun spam-setup-widening ()
   (dolist (check spam-list-of-statistical-checks)

Re: Fancy splitting getting headers from included text

[Ted Zlatanov]

On Sun, 30 Mar 2003, jas@extundo.com wrote:
> I'm not sure where the spam code is invoked, but if it does a
> (widen) it should probably do it inside a save-restriction.  Does
> this patch help?

[ ... patch omitted ... ]

This is a good idea anyhow, I should have done it from the start.
Patch committed, thanks.

Ted

Re: Fancy splitting getting headers from included text

[David Z Maze]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> I think it was reported to happen for nnmail too.  Hm.  Those who are
> seeing this, are you using spam.el and downloading bodies too?

Using spam.el, yes.  Relevant parts of my .gnus file:

(setq nnmail-split-fancy
      '(|
        (: spam-split)
        (to "reuse-\\(\\w*\\)@mit\\.edu" "mail.lists.reuse.\\1")
        (any "dmaze@.*mit\\.edu" "mail.misc.personal")
        "mail.misc.impersonal"))
(setq nnmail-split-methods 'nnmail-split-fancy)
(setq mail-sources '((imap :server "po10.mit.edu" :password ""
                           :stream kerberos4 :authentication kerberos4
                           :predicate "UNDELETED")))
(setq spam-split-group "mail.misc.spam")

Given this set of rules, the attached message trips the bug: it got
sorted into mail.lists.reuse.ask, where reuse-ask is only mentioned in
the body of the message.  It should go in mail.lists.reuse.housing,
since that's what's in the headers.


[-- Attachment #2: reuse-misfiled --]
[-- Type: message/rfc822, Size: 2047 bytes --]

From: <reuse-request@MIT.EDU>
To: <reuse-housing@mit.edu>
Subject: apartment wanted (fwd)
Date: Mon, 14 Apr 2003 11:36:20 -0400 (EDT)
Message-ID: <Pine.GSO.4.33L.0304141136170.9597-100000@scrubbing-bubbles.mit.edu>




---------- Forwarded message ----------
Date: Mon, 14 Apr 2003 11:31:20 -0400
From: Laura Redi <redi@MIT.EDU>
To: reuse-ask@mit.edu
Subject: apartment wanted

Hi,
I'm looking for a studio or 1 BR apartment available June 1 or possibly
earlier. If you know of anyone who plans to be moving out of his/her
place please let me know.
Thanks,
Laura

--
Laura Redi
Ph.D. Candidate
Speech and Hearing Bioscience and Technology
Harvard-MIT Health Sciences Technology Program

redi@mit.edu
www.mit.edu/~redi
W: 617-253-5957







[-- Attachment #3: Type: text/plain, Size: 423 bytes --]


> Could you edebug nnmail-article-group to see if nnmail-split-fancy
> is somehow invoked without narrowed headers?

It doesn't get tripped if I 'B q' having started Gnus, only while mail
is being slurped in.  This makes it a little harder to debug.  :-/

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Fancy splitting getting headers from included text

[Simon Josefsson]

David Z Maze <dmaze@MIT.EDU> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> I think it was reported to happen for nnmail too.  Hm.  Those who are
>> seeing this, are you using spam.el and downloading bodies too?
>
> Using spam.el, yes.  Relevant parts of my .gnus file:

And it isn't fixed in CVS?  I think a fix related to this went into
spam.el recently.

Re: Fancy splitting getting headers from included text

[David Z Maze]

Simon Josefsson <jas@extundo.com> writes:

> David Z Maze <dmaze@MIT.EDU> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> I think it was reported to happen for nnmail too.  Hm.  Those who are
>>> seeing this, are you using spam.el and downloading bodies too?
>>
>> Using spam.el, yes.  Relevant parts of my .gnus file:
>
> And it isn't fixed in CVS?  I think a fix related to this went into
> spam.el recently.

Yup, still there with current CVS.  I even see the proposed
save-excursion in spam-split in spam.el.

(And I tried to follow it a little, and my brain hurt.  I'm not used
to elisp code having that sort of control flow.  (return) isn't in my
normal elisp vocabulary, particularly when it does something *other*
than return from the current function...it *does* look like the
save-excursion there should wrap the potential call to (widen),
though.  I have spam-use-stat set to t, so the code path that calls
(widen) does get taken.)

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Fancy splitting getting headers from included text

[Ted Zlatanov]

On Mon, 14 Apr 2003, dmaze@MIT.EDU wrote:
> Yup, still there with current CVS.  I even see the proposed
> save-excursion in spam-split in spam.el.
> 
> (And I tried to follow it a little, and my brain hurt.  I'm not used
> to elisp code having that sort of control flow.  (return) isn't in
> my normal elisp vocabulary, particularly when it does something
> *other* than return from the current function...it *does* look like
> the save-excursion there should wrap the potential call to (widen),
> though.  I have spam-use-stat set to t, so the code path that calls
> (widen) does get taken.)

I think the code you mean is this:

  (save-excursion
  (dolist (check spam-list-of-statistical-checks)
    (when (symbol-value check)
      (widen)
      (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
		    (symbol-name check))
      (return)))
...)

The reason for (return) is that I like dolist, and (return) is what
you're supposed to use to exit a dolist loop early.

Are you and Simon saying the code also needs a save-restriction inside
the save-excursion?  If that's the case, try this (improperly
indented) version of spam-split and see how it works:

(defun spam-split ()
  "Split this message into the `spam' group if it is spam.
This function can be used as an entry in `nnmail-split-fancy', for
example like this: (: spam-split)

See the Info node `(gnus)Fancy Mail Splitting' for more details."
  (interactive)
  (save-excursion
  (save-restriction
  (dolist (check spam-list-of-statistical-checks)
    (when (symbol-value check)
      (widen)
      (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
		    (symbol-name check))
      (return)))
;;   (progn (widen) (debug (buffer-string)))
  (let ((list-of-checks spam-list-of-checks)
	decision)
    (while (and list-of-checks (not decision))
      (let ((pair (pop list-of-checks)))
	(when (symbol-value (car pair))
	  (gnus-message 5 "spam-split: calling the %s function" (symbol-name (cdr pair)))
	  (setq decision (funcall (cdr pair))))))
    (if (eq decision t)
	nil
	decision)))))


Ted

Re: Fancy splitting getting headers from included text

[David Z Maze]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Are you and Simon saying the code also needs a save-restriction inside
> the save-excursion?  If that's the case, try this (improperly
> indented) version of spam-split and see how it works:

Aah, that seems to have helped.

> I think the code you mean is this:
>
>   (save-excursion
>   (dolist (check spam-list-of-statistical-checks)
>     (when (symbol-value check)
>       (widen)
>       (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
> 		    (symbol-name check))
>       (return)))
> ...)

Yeah, I might write something more like

(if (memq nil
      (mapcar 'not
        (mapcar 'symbol-value spam-list-of-statistical-checks)))
  (widen))

Though that eliminates the possibility of the gnus-message.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Fancy splitting getting headers from included text

[Ted Zlatanov]

On Tue, 15 Apr 2003, dmaze@MIT.EDU wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Are you and Simon saying the code also needs a save-restriction
>> inside the save-excursion?  If that's the case, try this
>> (improperly indented) version of spam-split and see how it works:
> 
> Aah, that seems to have helped.

OK, fix committed to CVS.  I hope it works for everyone.

>> I think the code you mean is this:
>>
>>   (save-excursion
>>   (dolist (check spam-list-of-statistical-checks)
>>     (when (symbol-value check)
>>       (widen) (gnus-message 8 "spam-split: widening the buffer (%s requires it)"
>> 		    (symbol-name check))
>>       (return)))
>> ...)
> 
> Yeah, I might write something more like
> 
> (if (memq nil
>       (mapcar 'not
>         (mapcar 'symbol-value spam-list-of-statistical-checks)))
>   (widen))
> 
> Though that eliminates the possibility of the gnus-message.

Ah, but mapcar, just like dolist, goes through the whole list, and I
wanted to optimize :) Looking back I see it's a pretty useless
optimization, but I was trying to learn Lisp and how to return from
forms at the time.

Ted

Re: Fancy splitting getting headers from included text

[Niklas Morberg]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 15 Apr 2003, dmaze@MIT.EDU wrote:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> Are you and Simon saying the code also needs a save-restriction
>>> inside the save-excursion?  If that's the case, try this
>>> (improperly indented) version of spam-split and see how it works:
>> 
>> Aah, that seems to have helped.
>
> OK, fix committed to CVS.  I hope it works for everyone.

I still see the problem here with CVS from 2003-05-06. Is it
working for everybody else?

Niklas

Re: Fancy splitting getting headers from included text

[Niklas Morberg]

Niklas Morberg <niklas@axis.com> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> On Tue, 15 Apr 2003, dmaze@MIT.EDU wrote:
>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> 
>>>> Are you and Simon saying the code also needs a save-restriction
>>>> inside the save-excursion?  If that's the case, try this
>>>> (improperly indented) version of spam-split and see how it works:
>>> 
>>> Aah, that seems to have helped.
>>
>> OK, fix committed to CVS.  I hope it works for everyone.
>
> I still see the problem here with CVS from 2003-05-06. Is it
> working for everybody else?

CVS from 2003-06-05. Still broken here. Where should I look
to debug the problem?

Niklas

Re: Fancy splitting getting headers from included text

[Niklas Morberg]

Niklas Morberg <niklas@axis.com> writes:

> CVS from 2003-06-05. Still broken here. Where should I look
> to debug the problem?

The fact that nnimap-split-fancy does _not_ use the *nnimap
[server]* buffer took a while to understand. It turns out
it uses the *nntpd* buffer instead.

Anyway, what I've found is that ietf-drums-narrow-to-header
never works on my system. nnimap-split-articles calls
mail-narrow-to-head (which is an alias to i-d-n-t-h), but
the *nntpd* buffer is never narrowed...

(defun ietf-drums-narrow-to-header ()
  "Narrow to the header section in the current buffer."
  (narrow-to-region
   (goto-char (point-min))
   (if (re-search-forward "^\r?$" nil 1)
       (match-beginning 0)
     (point-max)))
  (goto-char (point-min)))

I'm stumped. The regexp in i-d-n-t-h looks fine. If I go to
the *nntpd* buffer and manually do `M-x re-search-forward
[RET] ^\r?$' it will move point to the end of headers. Yet
the function i-d-n-t-h fails to narrow the buffer to
headers.

I have no idea on how to move forward on this. Help?

Niklas

Re: Fancy splitting getting headers from included text

[Niklas Morberg]

Niklas Morberg <niklas@axis.com> writes:

> I'm stumped. The regexp in i-d-n-t-h looks fine. If I go to
> the *nntpd* buffer and manually do `M-x re-search-forward
> [RET] ^\r?$' it will move point to the end of headers. Yet
> the function i-d-n-t-h fails to narrow the buffer to
> headers.

And if I go an article buffer in gnus and do `M-:
(ietf-drums-narrow-to-header) [RET]' the buffer is properly
narrowed. But it does not work with the *nntpd* buffer...

Niklas

[PATCH] Re: Fancy splitting getting headers from included text

[Niklas Morberg]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Niklas Morberg <niklas@axis.com> writes:

> Niklas Morberg <niklas@axis.com> writes:
>
>> I'm stumped. The regexp in i-d-n-t-h looks fine. If I go to
>> the *nntpd* buffer and manually do `M-x re-search-forward
>> [RET] ^\r?$' it will move point to the end of headers. Yet
>> the function i-d-n-t-h fails to narrow the buffer to
>> headers.
>
> And if I go an article buffer in gnus and do `M-:
> (ietf-drums-narrow-to-header) [RET]' the buffer is properly
> narrowed. But it does not work with the *nntpd* buffer...

Clarification: if I manually go to the *nntpd* buffer and
do `M-: (ietf-drums-narrow-to-header) [RET]' it will get
narrowed properly. But it doesn't work when called from
nnimap-split-articles.

Could it be that nnimap-split-articles is using the wrong
buffer? Hm. It seems that it is. The following patch makes
splitting work as expected (not taking body text into
account).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Narrow proper buffer. --]
[-- Type: text/x-patch, Size: 677 bytes --]

cd c:/cygwin/usr/share/site-lisp/
diff -u -b "c:/cygwin/usr/share/site-lisp/nnimap.el~" "c:/cygwin/usr/share/site-lisp/nnimap.el"
--- c:/cygwin/usr/share/site-lisp/nnimap.el~	2003-06-05 09:53:38.000000000 +0200
+++ c:/cygwin/usr/share/site-lisp/nnimap.el	2003-06-13 11:44:28.000000000 +0200
@@ -1271,7 +1271,7 @@
 			    nnimap-split-download-body-default
 			  nnimap-split-download-body)
 			(and (nnimap-request-article article)
-			     (mail-narrow-to-head))
+			     (with-current-buffer nntp-server-buffer (mail-narrow-to-head)))
 		      (nnimap-request-head article))
 		;; copy article to right group(s)
 		(setq removeorig nil)

Diff finished at Fri Jun 13 11:48:11

[-- Attachment #3: Type: text/plain, Size: 207 bytes --]


I have nnimap as my primary select method, maybe the above
patch will break stuff for people who have it as their
secondary select method? All I know is that with this patch
splitting works for me.

Niklas

Re: [PATCH] Re: Fancy splitting getting headers from included text

[Kai Großjohann]

Niklas Morberg <niklas.morberg@axis.com> writes:

> Could it be that nnimap-split-articles is using the wrong
> buffer? Hm. It seems that it is. The following patch makes
> splitting work as expected (not taking body text into
> account).

Very nice.  Committed.  (But not tested.)
-- 
This line is not blank.