widening in nnml split does not work

widening in nnml split does not work

[Ted Zlatanov]

Why does the following not work in a fancy splitting function on nnml
incoming messages?  I'm trying this code in the spam-split function:

  (save-restriction
    (widen)
    (let ((b (buffer-string)))
      (save-excursion
	(set-buffer (get-buffer-create "splitting log"))
	(insert b))))

It works just fine for nnimap with nnimap-split-download-body set to
t, so there's something different about nnml splitting.  I tried
tracing the source, but it's taking me a long time and I keep going
down wrong paths.  So if anyone could help, I'd appreciate it.

Thanks
Ted

Re: widening in nnml split does not work

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It works just fine for nnimap with nnimap-split-download-body set to
> t, so there's something different about nnml splitting.

When you get mail from /var/mail/jrl, say, then the whole thing is
put in a buffer.  Then nnmail-process-unix-mail-format narrows to the
headers of the first msg, calls splitting, processes that msg, then
moves on to the next msg.

So the (widen) will actually give you all the messages in a buffer,
instead of just the first.

I think.
-- 
Ambibibentists unite!

Re: widening in nnml split does not work

[Ted Zlatanov]

On Sat, 25 Jan 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> It works just fine for nnimap with nnimap-split-download-body set
>> to t, so there's something different about nnml splitting.
> 
> When you get mail from /var/mail/jrl, say, then the whole thing is
> put in a buffer.  Then nnmail-process-unix-mail-format narrows to
> the headers of the first msg, calls splitting, processes that msg,
> then moves on to the next msg.
> 
> So the (widen) will actually give you all the messages in a buffer,
> instead of just the first.

OK, but it doesn't.  (widen) just gives me the headers of the
message.

This is incorrect behavior according to the manual, so I'd say it's a
legitimate bug.  Yay.

Thanks
Ted

Re: widening in nnml split does not work

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK, but it doesn't.  (widen) just gives me the headers of the
> message.
>
> This is incorrect behavior according to the manual, so I'd say it's a
> legitimate bug.  Yay.

Well, I've now looked at the code.  It seems it's intentional, and
not easy to fix.  Here is how it goes:

nnmail-process-unix-mail-format calls nnmail-check-duplication on
each message.  That function calls nnmail-article-group, which copies
the headers to a temp buffer and does some MIMEy decoding on them
before doing the splitting dance.

So I'm afraid that maybe the manual needs to be fixed, rather than
the code.

Or nnmail-article-group could be augmented to optionally do some work
on the body, too.

Hm.  Sounds like real work.
-- 
Ambibibentists unite!

Re: widening in nnml split does not work

[Simon Josefsson]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> OK, but it doesn't.  (widen) just gives me the headers of the
>> message.
>>
>> This is incorrect behavior according to the manual, so I'd say it's a
>> legitimate bug.  Yay.
>
> Well, I've now looked at the code.  It seems it's intentional, and
> not easy to fix.  Here is how it goes:
>
> nnmail-process-unix-mail-format calls nnmail-check-duplication on
> each message.  That function calls nnmail-article-group, which copies
> the headers to a temp buffer and does some MIMEy decoding on them
> before doing the splitting dance.
>
> So I'm afraid that maybe the manual needs to be fixed, rather than
> the code.
>
> Or nnmail-article-group could be augmented to optionally do some work
> on the body, too.
>
> Hm.  Sounds like real work.

Augmenting n-a-g doesn't sound too hard.  Wouldn't something like the
following work?

--- nnmail.el.~6.57.~	Sun Jan 19 11:31:14 2003
+++ nnmail.el	Sat Jan 25 15:47:38 2003
@@ -998,8 +998,7 @@
 FUNC will be called with the group name to determine the article number."
   (let ((methods (or nnmail-split-methods '(("bogus" ""))))
 	(obuf (current-buffer))
-	(beg (point-min))
-	end group-art method grp)
+	group-art method grp)
     (if (and (sequencep methods)
 	     (= (length methods) 1))
 	;; If there is only just one group to put everything in, we
@@ -1008,13 +1007,12 @@
 	      (list (cons (caar methods) (funcall func (caar methods)))))
       ;; We do actual comparison.
       (save-excursion
-	;; Find headers.
-	(goto-char beg)
-	(setq end (if (search-forward "\n\n" nil t) (point) (point-max)))
+	;; Copy the article into the work buffer.
 	(set-buffer nntp-server-buffer)
 	(erase-buffer)
-	;; Copy the headers into the work buffer.
-	(insert-buffer-substring obuf beg end)
+	(insert-buffer obuf)
+	;; Narrow to headers.
+	(mail-narrow-to-head)
 	;; Decode MIME headers and charsets.
 	(when nnmail-mail-splitting-decodes
 	  (let ((mail-parse-charset nnmail-mail-splitting-charset))
@@ -1115,7 +1113,8 @@
 	  (let (elem)
 	    (while (setq elem (car (memq 'junk group-art)))
 	      (setq group-art (delq elem group-art)))
-	    (nreverse group-art)))))))
+	    (nreverse group-art))))
+      (widen))))
 
 (defun nnmail-insert-lines ()
   "Insert how many lines there are in the body of the mail.

Re: widening in nnml split does not work

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Augmenting n-a-g doesn't sound too hard.  Wouldn't something like the
> following work?

Interesting.  So the article headers are deMIMEified whereas the body
is left as-is?  Yeah, that sounds useful.  And a (widen) in the
splitting code would access the whole message.

Yes.  How about installing it to have our guinea pigs test it? ;-)
-- 
Ambibibentists unite!

Re: widening in nnml split does not work

[Ted Zlatanov]

On Sat, 25 Jan 2003, kai.grossjohann@uni-duisburg.de wrote:
> Simon Josefsson <jas@extundo.com> writes:
> 
>> Augmenting n-a-g doesn't sound too hard.  Wouldn't something like
>> the following work?
> 
> Interesting.  So the article headers are deMIMEified whereas the
> body is left as-is?  Yeah, that sounds useful.  And a (widen) in the
> splitting code would access the whole message.
> 
> Yes.  How about installing it to have our guinea pigs test it? ;-)

Seconded, guinea pigs standing by...

Ted

Re: widening in nnml split does not work

[Simon Josefsson]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Augmenting n-a-g doesn't sound too hard.  Wouldn't something like the
>> following work?
>
> Interesting.  So the article headers are deMIMEified whereas the body
> is left as-is?  Yeah, that sounds useful.  And a (widen) in the
> splitting code would access the whole message.

Yup.  Hm.  Would splitting code want the body deMIMEified?

I guess some text analysis splitters would want to base64 decoding to
happen of course.  I wonder how much deMIMEification is wanted.
Really, only the splitter functions can know this (some splitters
might want raw MIME, some decoded base64 but not other MIME-decoding,
some might want everything MIME related to be removed) so it should
perform it itself, or Gnus should provide a framework for deciding
what should be done for each splitter function.  Anyway, until that
happens, copying the raw body is probably mostly harmless.

> Yes.  How about installing it to have our guinea pigs test it? ;-)

I installed it.

Re: widening in nnml split does not work

[Ted Zlatanov]

On Sat, 25 Jan 2003, jas@extundo.com wrote:
> Yup.  Hm.  Would splitting code want the body deMIMEified?
> 
> I guess some text analysis splitters would want to base64 decoding
> to happen of course.  I wonder how much deMIMEification is wanted.
> Really, only the splitter functions can know this (some splitters
> might want raw MIME, some decoded base64 but not other
> MIME-decoding, some might want everything MIME related to be
> removed) so it should perform it itself, or Gnus should provide a
> framework for deciding what should be done for each splitter
> function.  Anyway, until that happens, copying the raw body is
> probably mostly harmless.

Agreed, let's leave the raw body until someone requests more.  I don't
believe in unnnecessary work on Saturday mornings :)

> I installed it.

How about the other backends?  spam-split can only be invoked by
nnmail-split-fancy and nnimap-split-fancy; besides nnimap and nnmail,
are there any other backends I need to worry about to make spam-split
work universally?

Ted

Re: widening in nnml split does not work

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Yup.  Hm.  Would splitting code want the body deMIMEified?

Good question.  Hm.  My hunch is that it would be useful to discard
all useless MIME parts, such as the non-text parts.  And the rest
should be properly decoded.  (But text/html should not be rendered.)

Whee.  Getting a little bit complicated, these rules.  What do people
think?

A variable nnmail-split-ignored-content-types which contains a list
of strings along the lines of "image/*" or "application/ms-word"
appears to be sensible.

But it's not clear to me how to specify that text/html should not be
rendered.  Hm.  Maybe for all types, it should just do qp or base64
decoding and that's it; no further rendering.
-- 
Ambibibentists unite!

Re: widening in nnml split does not work

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> How about the other backends?  spam-split can only be invoked by
> nnmail-split-fancy and nnimap-split-fancy; besides nnimap and nnmail,
> are there any other backends I need to worry about to make spam-split
> work universally?

Well, nnmaildir is a mail backend which presumably supports
splitting.  But it is not based on nnmail.

I still haven't tried nnmaildir.  Bad Kai!
-- 
Ambibibentists unite!

Re: widening in nnml split does not work

[Ted Zlatanov]

On Sat, 25 Jan 2003, kai.grossjohann@uni-duisburg.de wrote:
> Well, nnmaildir is a mail backend which presumably supports
> splitting.  But it is not based on nnmail.

I see.  There's no nnmaildir-split-fancy either, since the incoming
mail is already split usually with maildir.  So I won't worry about it
until users ask for fancy splitting for nnmaildir.

Ted