carriage-returns and nnimap

carriage-returns and nnimap

[Andrew Cohen]

I had noticed that gnus-registry-split-fancy-with-parent had stopped
working with "sender" splitting. I finally decided to track this down.

Looking at the registry I noticed that the "sender" and "subject" fields
(and some others) had a spurious space at the end of them (which is
wrong), while an old copy of the registry (from months ago) has no such
spaces (which is right). I can't pinpoint exactly when this changed so
I'm not sure of the exact cause.

Tracing this a bit I found that 'gnus-get-newsgroup-headers translates
\r into spaces, so I suspect that this is where the extra spaces are
coming from. I added a call to 'nnheader-remove-cr-followed-by-lf in
'gnus-get-newsgroup-headers just before the \r->space translation and it
seems to fix the problem. 

I didn't want to just push this in case I've missed something. The whole
carriage-return thing in nnimap confuses me.

Regards,
Andy


diff --git a/lisp/gnus-sum.el b/lisp/gnus-sum.el
index ceaa014..dbd25ca 100644
--- a/lisp/gnus-sum.el
+++ b/lisp/gnus-sum.el
@@ -6262,6 +6262,7 @@ The resulting hash table is returned, or nil if no Xrefs were found."
                               gnus-newsgroup-ignored-charsets)))
     (with-current-buffer nntp-server-buffer
       ;; Translate all TAB characters into SPACE characters.
+      (nnheader-remove-cr-followed-by-lf)
       (subst-char-in-region (point-min) (point-max) ?\t ?  t)
       (subst-char-in-region (point-min) (point-max) ?\r ?  t)
       (ietf-drums-unfold-fws)

Re: carriage-returns and nnimap

[Ted Zlatanov]

On Fri, 17 Dec 2010 07:58:29 -0500 Andrew Cohen <cohen@andy.bu.edu> wrote: 

AC> I had noticed that gnus-registry-split-fancy-with-parent had stopped
AC> working with "sender" splitting. I finally decided to track this down.

AC> Looking at the registry I noticed that the "sender" and "subject" fields
AC> (and some others) had a spurious space at the end of them (which is
AC> wrong), while an old copy of the registry (from months ago) has no such
AC> spaces (which is right). I can't pinpoint exactly when this changed so
AC> I'm not sure of the exact cause.

AC> Tracing this a bit I found that 'gnus-get-newsgroup-headers translates
AC> \r into spaces, so I suspect that this is where the extra spaces are
AC> coming from. I added a call to 'nnheader-remove-cr-followed-by-lf in
AC> 'gnus-get-newsgroup-headers just before the \r->space translation and it
AC> seems to fix the problem. 

AC> I didn't want to just push this in case I've missed something. The whole
AC> carriage-return thing in nnimap confuses me.

I hadn't noticed, thanks for catching that.

I think your fix is correct, but if necessary I can always adjust
gnus-registry.el alone to trim those extra CRs.

Ted

Re: carriage-returns and nnimap

[Andrew Cohen]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

    Ted> On Fri, 17 Dec 2010 07:58:29 -0500 Andrew Cohen <cohen@andy.bu.edu> wrote:

[...]


    AC> I didn't want to just push this in case I've missed
    AC> something. The whole carriage-return thing in nnimap confuses
    AC> me.

    Ted> I hadn't noticed, thanks for catching that.

    Ted> I think your fix is correct, but if necessary I can always
    Ted> adjust gnus-registry.el alone to trim those extra CRs.

I'm becoming more confident that the header parsing routine should
remove them. I wonder why we have two nearly (but not quite) identical
parsing routines. One is inside 'gnus-get-newsgroup-headers, while the
other is 'nnheader-parse-naked-head. The latter one includes the CR
removal, so I suspect the former should as well.

I'll push this shortly unless I hear something.

Best,
Andy

Re: carriage-returns and nnimap

[Ted Zlatanov]

On Fri, 17 Dec 2010 09:18:17 -0500 Andrew Cohen <cohen@andy.bu.edu> wrote: 

AC> I wonder why we have two nearly (but not quite) identical parsing
AC> routines. One is inside 'gnus-get-newsgroup-headers, while the other
AC> is 'nnheader-parse-naked-head. The latter one includes the CR
AC> removal, so I suspect the former should as well.

I don't know why, but we should get rid of one of the two, probably the
former.

Ted

Re: carriage-returns and nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> Tracing this a bit I found that 'gnus-get-newsgroup-headers translates
> \r into spaces, so I suspect that this is where the extra spaces are
> coming from. I added a call to 'nnheader-remove-cr-followed-by-lf in
> 'gnus-get-newsgroup-headers just before the \r->space translation and it
> seems to fix the problem. 

There should be no \r\n in the buffers when Gnus gets them.  The backend
should do \r\n decoding if the transport it uses uses \r\n...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: carriage-returns and nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I'm becoming more confident that the header parsing routine should
> remove them. I wonder why we have two nearly (but not quite) identical
> parsing routines. One is inside 'gnus-get-newsgroup-headers, while the
> other is 'nnheader-parse-naked-head. The latter one includes the CR
> removal, so I suspect the former should as well.

The former is a Gnus function which does threading etc etc while
parsing.  The latter is a general convenience function used by any
backends that need to do ad-hoc parsing for some reason or other.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: carriage-returns and nnimap

[Andrew Cohen]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    Lars> Andrew Cohen <cohen@andy.bu.edu> writes:
    >> Tracing this a bit I found that 'gnus-get-newsgroup-headers
    >> translates \r into spaces, so I suspect that this is where the
    >> extra spaces are coming from. I added a call to
    >> 'nnheader-remove-cr-followed-by-lf in 'gnus-get-newsgroup-headers
    >> just before the \r->space translation and it seems to fix the
    >> problem.

    Lars> There should be no \r\n in the buffers when Gnus gets them.
    Lars> The backend should do \r\n decoding if the transport it uses
    Lars> uses \r\n...

OK, so this should be a change in 'nnimap-retrieve-headers? Maybe added
to 'nnimap-transform-headers?

Re: carriage-returns and nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> OK, so this should be a change in 'nnimap-retrieve-headers? Maybe added
> to 'nnimap-transform-headers?

Yup.  I've now done so.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen