agent articles get \201 inserted

agent articles get \201 inserted

[Steinar Bang]

I run nnimap with agent.

I also use the following setting to cache articles in the agent when
reading them:
 ;; Store article in agent when reading it while online
 (add-hook 'gnus-select-article-hook 'gnus-agent-fetch-selected-article)

In the last week or so I've seen a lot of articles with strange \201
characters.  It looks like they are inserted when the agent saves them.

The articles I've looked at today are sent with charset=iso-8859-1.
In the Maildir on the dovecot server they are shown with just the single
iso-8859-1 character (eg. å).  In the article buffer of Gnus they are
shown as \201å.  And in the agent cache the characters are \201\345.

Is there something fishy in the charset the agent believes it has when
it saves the articles.

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> In the last week or so I've seen a lot of articles with strange \201
> characters.  It looks like they are inserted when the agent saves them.

> The articles I've looked at today are sent with charset=iso-8859-1.
> In the Maildir on the dovecot server they are shown with just the single
> iso-8859-1 character (eg. å).  In the article buffer of Gnus they are
> shown as \201å.  And in the agent cache the characters are \201\345.

> Is there something fishy in the charset the agent believes it has when
> it saves the articles.

This actually happens with nnimap, even without using the agent.  It
looks like ISO-8859-1 is misinterpreted as UTF-8 somehow.  UTF-8 articles
survive intact.

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Steinar Bang <sb@dod.no>:

>> In the last week or so I've seen a lot of articles with strange \201
>> characters.  It looks like they are inserted when the agent saves
>> them.

>> The articles I've looked at today are sent with charset=iso-8859-1.
>> In the Maildir on the dovecot server they are shown with just the
>> single iso-8859-1 character (eg. å).  In the article buffer of Gnus
>> they are shown as \201å.  And in the agent cache the characters are
>> \201\345.

>> Is there something fishy in the charset the agent believes it has
>> when it saves the articles.

> This actually happens with nnimap, even without using the agent.  It
> looks like ISO-8859-1 is misinterpreted as UTF-8 somehow.  UTF-8
> articles survive intact.

Actually... this seems to be an emacs22 issue.  Happens with emacs22 on
Ubuntu 10.10 and on CentOS5, but not in emacs23 on debian testing.

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Actually... this seems to be an emacs22 issue.  Happens with emacs22 on
> Ubuntu 10.10 and on CentOS5, but not in emacs23 on debian testing.

Hm.  The nnimap connection buffer seems to have all the right
incantations (`mm-disable-multibyte', `binary' coding systems) for this
to work correctly...

I don't seem to have Emacs 22 available on my machines, so this is kinda
difficult for me to debug.

This bug sounds familiar to me, but I don't remember what the underlying
issue was in Emacs 22 any more.  Anybody remember?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Actually... this seems to be an emacs22 issue.  Happens with emacs22 on
>> Ubuntu 10.10 and on CentOS5, but not in emacs23 on debian testing.
[snip!]
> This bug sounds familiar to me, but I don't remember what the underlying
> issue was in Emacs 22 any more.  Anybody remember?

Digging through some archives:
 http://thread.gmane.org/gmane.emacs.gnus.user/7984

This was nnml, not nnimap. But I used to have that
standard-display-european setting:

 r1030 | sb | 2008-04-28 12:15:40 +0200 (Mon, 28 Apr 2008) | 7 lines

 Changed to work with GNU emacs 22:
  - remove the semi-deprecated use of standard-display-european
  - remove the use of iso-accents-mode, which doesn't seem to
    be there anymore (pity, since it's in my fingers)

(hm... removed that setting for emacs22, actually..?)

No conclusions on that thread, unfortunately...:-/

Back in 2009, Simon had a similar problem when sending
 http://thread.gmane.org/gmane.emacs.gnus.general/67826/focus=24020

The thread mentions a change in nnimap-demule that was reverted in
2004, and broke emacs23 display of email in 2008.  But nnimap-demule
doesn't exist anymore, so that's another dead end.

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:
>> Steinar Bang <sb@dod.no> writes:
>>> Actually... this seems to be an emacs22 issue.  Happens with emacs22 on
>>> Ubuntu 10.10 and on CentOS5, but not in emacs23 on debian testing.
> [snip!]
>> This bug sounds familiar to me, but I don't remember what the underlying
>> issue was in Emacs 22 any more.  Anybody remember?

I decided to bisect the issue.  Between each good or bad, I did
 (cd ~/git/gnus; make clean; make; cd lisp; make tags)
and started a fresh emacs on the results.

It turns out that the issue has been there for a while (and also that
the bisect result isn't very useful... see below...:-) ).

The reason I haven't noticed it, is that I get it on some articles.  And
one of the offending articles have:
 Content-transfer-encoding: 8BIT
 Content-type: text/plain; charset=UTF-8

Could it be CTE 8BIT that is the culprit?  There was something about
that in the most recent thread (the 2008 thread from this group).

Articles that display correctly, are
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable

and (curiously):
 Content-transfer-encoding: 8BIT
 Content-type: text/html; charset=utf-8

(I'm not sure what I use to render HTML.  But should it matter, if the
problem is getting 8BIT stuff from nnimap?)


Anyway... the "offending" commit is this one:

sb@aa1steinar:~/git/gnus$ git bisect good
df6ac45b952171e7e709c1b0bf23cc7346429f81 is the first bad commit
commit df6ac45b952171e7e709c1b0bf23cc7346429f81
Author: Lars Magne Ingebrigtsen <larsi@quimbies.gnus.org>
Date:   Fri Sep 17 18:18:39 2010 +0200

    Reimplement nnimap, and do tweaks to the rest of the code to support that.
    
    * gnus-int.el (gnus-finish-retrieve-group-infos)
    (gnus-retrieve-group-data-early): New functions.
    
    * gnus-range.el (gnus-range-nconcat): New function.
    
    * gnus-start.el (gnus-get-unread-articles): Support early retrieval of
    data.
    (gnus-read-active-for-groups): Support finishing the early retrieval of
    data.
    
    * gnus-sum.el (gnus-summary-move-article): Pass the move-to group name
    if the move is internal, so that nnimap can do fast internal moves.
    
    * gnus.el (gnus-article-special-mark-lists): Add uid/active tuples, for
    nnimap usage.
    
    * nnimap.el: Rewritten.
    
    * nnmail.el (nnmail-inhibit-default-split-group): New internal variable
    to allow the mail splitting to not return a default group.  This is
    useful for nnimap, which will leave unmatched mail in the inbox.
    
    * utf7.el (utf7-encode): Autoload.

:040000 040000 09a1ec556b1e7bd509b0d7ccaa718be697b6c6cd b9ca8dee8837ae5b13e9c23fece66471ebb0b5c3 M	lisp

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Could it be CTE 8BIT that is the culprit?  There was something about
> that in the most recent thread (the 2008 thread from this group).
>
> Articles that display correctly, are
>  Content-Type: text/plain; charset=ISO-8859-1
>  Content-Transfer-Encoding: quoted-printable
>
> and (curiously):
>  Content-transfer-encoding: 8BIT
>  Content-type: text/html; charset=utf-8
>
> (I'm not sure what I use to render HTML.  But should it matter, if the
> problem is getting 8BIT stuff from nnimap?)

So, when getting CTE: 8BIT articles from nnimap (on Emacs 22), the Agent
versions of the articles end up with \201 in front of them?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> So, when getting CTE: 8BIT articles from nnimap (on Emacs 22), the Agent
> versions of the articles end up with \201 in front of them?

Not quite: when you have CTE: 8BIT on text/plain (not in text/html.  At
least not in one sample article I have), gnus shows nnimap articles with
\201 in front of non-ASCII chars.

The agent saving them with \201 in front of non-ASCII characters, is
just an effect of the article buffer having \201 in the first place.

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

>> So, when getting CTE: 8BIT articles from nnimap (on Emacs 22), the Agent
>> versions of the articles end up with \201 in front of them?
>
> Not quite: when you have CTE: 8BIT on text/plain (not in text/html.  At
> least not in one sample article I have), gnus shows nnimap articles with
> \201 in front of non-ASCII chars.

So this is without the Agent, even?  That should be even easier to
reproduce...

But it's just on Emacs 22, right?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> So this is without the Agent, even? 

Yep.

> That should be even easier to reproduce...

> But it's just on Emacs 22, right?

Correct.  (unless it shows up on some XEmacsen as well.  I haven't
tested)

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

>> But it's just on Emacs 22, right?
>
> Correct.  (unless it shows up on some XEmacsen as well.  I haven't
> tested)

Right.  I'll fire up an Emacs 22 and see if I can reproduce it.

So this is the test message.  It has éight-bit characters, and it's
hopefully with CTE: 8bit.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So this is the test message.  It has éight-bit characters, and it's
> hopefully with CTE: 8bit.

This is driving me crazy.

If I look at the character "ø" in the nntp server buffer, I get:

248 (#o370, #xf8, ?\370)

If I look at it in the nnimap buffer, I get:

248 (#o370, #xf8)

Er.  WTF?!

And if I do a `copy-buffer' to a different buffer, I get the correct
thing copied over from the nntp buffer, but not from the nnimap buffer.

Both buffers are opened with (mm-disable-multibyte), and the connections
are both binary...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: agent articles get \201 inserted

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> This is driving me crazy.

Is it possible to find out what the old nnimap backend did?  There
should be some commits related to this?

Re: agent articles get \201 inserted

[Lars Magne Ingebrigtsen]

FOUND IT!!!1!1!

Man, that was painful.  The solution was to use
`nntp-insert-buffer-substring'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/