Automatic decoding of charsets

Automatic decoding of charsets

[Julien Cubizolles]

Since switching from Gnus 5.13 to Nognus 0.18, charsets are often not
decoded in the articles I receive: accented characters are displayed as
"\351" for "é" for example. W-M-c fixes it (gnus-article-decode-charset)
but I'd like to automate the process. I tried :
,----
| (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
`----
to no avail.

What could I do ? And what did change in Gnus to cause this ?


Julien.

Re: Automatic decoding of charsets

[Adam Sjøgren]

On Mon, 26 Sep 2011 11:10:36 +0200, Julien wrote:

> Since switching from Gnus 5.13 to Nognus 0.18, charsets are often not
> decoded in the articles I receive: accented characters are displayed as
> "\351" for "é" for example. W-M-c fixes it (gnus-article-decode-charset)
> but I'd like to automate the process. I tried :
> ,----
> | (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
                    ^
                   If that uppercase A isn't just a typo in your
article, I think that might be why your workaround didn't work.

> What could I do ? And what did change in Gnus to cause this ?

If you could provide an example of an article where it happens, other
people could try and reproduce the problem, and that might shed some
light on it.


  Best regards,

    Adam

-- 
 "But after all, who is? At least in our case - no one        Adam Sjøgren
  is such a great musician. But we are stubborn."        asjo@koldfront.dk

Re: Automatic decoding of charsets

[Julien Cubizolles]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

asjo@koldfront.dk (Adam Sjøgren) writes:

> On Mon, 26 Sep 2011 11:10:36 +0200, Julien wrote:
>
>> Since switching from Gnus 5.13 to Nognus 0.18, charsets are often not
>> decoded in the articles I receive: accented characters are displayed as
>> "\351" for "é" for example. W-M-c fixes it (gnus-article-decode-charset)
>> but I'd like to automate the process. I tried :
>> ,----
>> | (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
>                     ^
>                    If that uppercase A isn't just a typo in your
> article, I think that might be why your workaround didn't work.

It wasn't a typo... I had copied the hook name from a web page. The hook
works now with 
,----
| (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
`----
 but it's not a satisfying solution : I noticed that I loose the ability
 to click on buttons for attachments, the fontifying of citations and so
 on... 

>> What could I do ? And what did change in Gnus to cause this ?
>
> If you could provide an example of an article where it happens, other
> people could try and reproduce the problem, and that might shed some
> light on it.

Here is an example of such an article, attached as a text file (I
removed some headers). Let me know if some other headers are needed and
if there is a better way to attach the file.

Best Regards,

Julien.

[-- Attachment #2: test2.mail --]
[-- Type: application/octet-stream, Size: 2009 bytes --]

MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_13167084834e7b6083a32ff_=_"


--_=_swift_v4_13167084834e7b6083a32ff_=_ Content-type: text/plain Content-transfer-encoding: quoted-printable



=20 =C2=A0 =20 Les inscriptions au 59=C2=B0 co= ngr=C3=A8s des professeurs de physique et de chimie organis=C3=A9 par = l'UdPPC sont ouvertes et le paiement par carte bancaire est op=C3=A9ra= tionnel !  =20 A vos clics !  =20 http://montpellier2011.udp= pc.asso.fr/ =20 Christophe REY =20 Micheline Izbicki


--_=_swift_v4_13167084834e7b6083a32ff_=_ Content-type: text/html Content-transfer-encoding: quoted-printable

<p style=3D"text-align: left;"><img src=3D"http://udp-bup.org/images/log= o.png" alt=3D"logo udppc" width=3D"201" height=3D"55" /></p> <p st= yle=3D"text-align: left;">&nbsp;</p> <p><span style=3D"font-size: sm= all;">Les inscriptions au 59&deg; congr&egrave;s des professeurs de phys= ique et de chimie organis&eacute; par l'UdPPC sont ouvertes et le paieme= nt par carte bancaire est op&eacute;rationnel !</span></p> <p style=3D= "text-align: center;"><span style=3D"font-size: small;">A vos clics
!</spa=
n></p>
<p style=3D"text-align: center;"><span style=3D"font-size: sm= all;"><span style=3D"font-family: Times New Roman','serif'; color: red; f= ont-size: 12pt;"><a title=3D"http://udppc.asso.fr/" href=3D"http://montp= ellier2011.udppc.asso.fr/"><span style=3D"font-family: Comic Sans MS'; co= lor: red; font-size: 10pt;">http://montpellier2011.udppc.asso.fr/</span>= </a></span></span></p> <p style=3D"text-align: center;"><span styl= e=3D"color: #000000; font-size: small;"><span style=3D"font-family:
Tim= es New Roman','serif'; font-size: 12pt;">Christophe REY</span></span></p= > <p style=3D"text-align: center;"><span style=3D"font-size: small;= "><span style=3D"font-family: Times New Roman','serif'; color: red; font-= size: 12pt;"><span style=3D"color: #000000;">Micheline Izbicki</span></spa=
n></span></p>

--_=_swift_v4_13167084834e7b6083a32ff_=_--

Re: Automatic decoding of charsets

[Lars Magne Ingebrigtsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Here is an example of such an article, attached as a text file (I
> removed some headers).

Could you include the entire article?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Automatic decoding of charsets

[Adam Sjøgren]

On Mon, 26 Sep 2011 15:12:06 +0200, Julien wrote:

> It wasn't a typo... I had copied the hook name from a web page. The hook
> works now with 
> ,----
> | (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
> `----

In your first article you wrote that the exact same command didn't work
- what changed?

(I wouldn't expect it to work, since gnus-Article-prepare-hook doesn't
exist in my Emacs, while gnus-article-prepare-hook does - but super if
it works.)

>> If you could provide an example of an article where it happens, other
>> people could try and reproduce the problem, and that might shed some
>> light on it.

> Here is an example of such an article, attached as a text file (I
> removed some headers). Let me know if some other headers are needed and
> if there is a better way to attach the file.

I think the chance of someone figuring out the problem is a lot higher
if you provide a complete article with which the problem can be easily
reproduced.

(If the file can be saved and the problem demonstrated by going G f
<filename> in Gnus, for instance).

I couldn't with the cut down test2.mail you included.


  Best regards,

    Adam

-- 
 "Sometimes it pays to stay in bed on Monday, rather          Adam Sjøgren
  than spending the rest of the week debugging           asjo@koldfront.dk
  Monday's code."

Re: Automatic decoding of charsets

[Julien Cubizolles]

[-- Attachment #1: Type: text/plain, Size: 284 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> Here is an example of such an article, attached as a text file (I
>> removed some headers).
>
> Could you include the entire article?

Here it is, I just removed the addresses.


[-- Attachment #2: test_mail_with_prefetch_option --]
[-- Type: application/octet-stream, Size: 2934 bytes --]

Return-Path: udpml@sirtem.fr
Received: from zimbra63-e11.priv.proxad.net (LHLO
 zimbra63-e11.priv.proxad.net) (172.20.243.213) by
 zimbra63-e11.priv.proxad.net with LMTP; Thu, 22 Sep 2011 18:10:29 +0200
 (CEST)
Received: from rinage.sirtem.fr (mx26-g26.priv.proxad.net [172.20.243.96])
	by zimbra63-e11.priv.proxad.net (Postfix) with ESMTP id 783062A5522
	for <j.cubizolles@free.fr>; Thu, 22 Sep 2011 18:10:29 +0200 (CEST)
Received: from rinage.sirtem.fr ([195.101.104.1])
	by mx1-g20.free.fr (MXproxy) for j.cubizolles@free.fr;
	Thu, 22 Sep 2011 18:10:29 +0200 (CEST)
X-ProXaD-SC: state=HAM score=10
Received: from [127.0.0.1] (localhost [127.0.0.1])
	by rinage.sirtem.fr (8.14.4/8.14.4) with ESMTP id p8MGF15g021180
	for <j.cubizolles@free.fr>; Thu, 22 Sep 2011 18:21:23 +0200
Message-ID: <1316708483.4e7b6083a1dbc@swift.generated>
Date: Thu, 22 Sep 2011 18:21:23 +0200
Subject: =?utf-8?Q?inscription=5Fcongr=C3=A8s?=
From: Liste UdPPC nationale 
Reply-To: 
To: j.cubizolles@free.fr
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_13167084834e7b6083a32ff_=_"


--_=_swift_v4_13167084834e7b6083a32ff_=_
Content-type: text/plain
Content-transfer-encoding: quoted-printable



=20
=C2=A0
=20
Les inscriptions au 59=C2=B0 co=
ngr=C3=A8s des professeurs de physique et de chimie
organis=C3=A9 par =
l'UdPPC sont ouvertes et le paiement par carte bancaire est
op=C3=A9ra=
tionnel !
=20
A vos clics !
=20
http://montpellier2011.udp=
pc.asso.fr/
=20

=20



--_=_swift_v4_13167084834e7b6083a32ff_=_
Content-type: text/html
Content-transfer-encoding: quoted-printable

<p style=3D"text-align: left;"><img src=3D"http://udp-bup.org/images/log=
o.png" alt=3D"logo udppc" width=3D"201" height=3D"55" /></p>
<p st=
yle=3D"text-align: left;">&nbsp;</p>
<p><span style=3D"font-size: sm=
all;">Les inscriptions au 59&deg; congr&egrave;s des professeurs de phys=
ique et de chimie organis&eacute; par l'UdPPC sont ouvertes et le paieme=
nt par carte bancaire est op&eacute;rationnel !</span></p>
<p style=3D=
"text-align: center;"><span style=3D"font-size: small;">A vos clics !</spa=
n></p>
<p style=3D"text-align: center;"><span style=3D"font-size: sm=
all;"><span style=3D"font-family: 'Times New Roman','serif'; color: red; f=
ont-size: 12pt;"><a title=3D"http://udppc.asso.fr/" href=3D"http://montp=
ellier2011.udppc.asso.fr/"><span style=3D"font-family: 'Comic Sans MS'; co=
lor: red; font-size: 10pt;">http://montpellier2011.udppc.asso.fr/</span>=
</a></span></span></p>
<p style=3D"text-align: center;"><span styl=
e=3D"color: #000000; font-size: small;"><span style=3D"font-family: 'Tim=
es New Roman','serif'; font-size: 12pt;"></span></span></p=
>
<p style=3D"text-align: center;"><span style=3D"font-size: small;=
"><span style=3D"font-family: 'Times New Roman','serif'; color: red; font-=
size: 12pt;"><span style=3D"color: #000000;"></span></spa=
n></span></p>

--_=_swift_v4_13167084834e7b6083a32ff_=_--

[-- Attachment #3: Type: text/plain, Size: 67 bytes --]


And here is the file correctly read without the prefetch option.


[-- Attachment #4: test_mail_without_prefetch_option --]
[-- Type: application/octet-stream, Size: 3083 bytes --]

Return-Path: udpml@sirtem.fr
Received: from zimbra63-e11.priv.proxad.net (LHLO
 zimbra63-e11.priv.proxad.net) (172.20.243.213) by
 zimbra63-e11.priv.proxad.net with LMTP; Thu, 22 Sep 2011 18:10:29 +0200
 (CEST)
Received: from rinage.sirtem.fr (mx26-g26.priv.proxad.net [172.20.243.96])
	by zimbra63-e11.priv.proxad.net (Postfix) with ESMTP id 783062A5522
	for <j.cubizolles@free.fr>; Thu, 22 Sep 2011 18:10:29 +0200 (CEST)
Received: from rinage.sirtem.fr ([195.101.104.1])
	by mx1-g20.free.fr (MXproxy) for j.cubizolles@free.fr;
	Thu, 22 Sep 2011 18:10:29 +0200 (CEST)
X-ProXaD-SC: state=HAM score=10
Received: from [127.0.0.1] (localhost [127.0.0.1])
	by rinage.sirtem.fr (8.14.4/8.14.4) with ESMTP id p8MGF15g021180
	for <j.cubizolles@free.fr>; Thu, 22 Sep 2011 18:21:23 +0200
Message-ID: <1316708483.4e7b6083a1dbc@swift.generated>
Date: Thu, 22 Sep 2011 18:21:23 +0200
Subject: =?utf-8?Q?inscription=5Fcongr=C3=A8s?=
From: Liste UdPPC nationale <udpml@sirtem.fr>
Reply-To: Jean-Marie Biau <jm.biau@udppc.asso.fr>
To: j.cubizolles@free.fr
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_13167084834e7b6083a32ff_=_"


--_=_swift_v4_13167084834e7b6083a32ff_=_
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable



=20
=C2=A0
=20
Les inscriptions au 59=C2=B0 co=
ngr=C3=A8s des professeurs de physique et de chimie
organis=C3=A9 par =
l'UdPPC sont ouvertes et le paiement par carte bancaire est
op=C3=A9ra=
tionnel !
=20
A vos clics !
=20
http://montpellier2011.udp=
pc.asso.fr/
=20
Christophe REY
=20
Micheline Izbicki


--_=_swift_v4_13167084834e7b6083a32ff_=_
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<p style=3D"text-align: left;"><img src=3D"http://udp-bup.org/images/log=
o.png" alt=3D"logo udppc" width=3D"201" height=3D"55" /></p>
<p st=
yle=3D"text-align: left;">&nbsp;</p>
<p><span style=3D"font-size: sm=
all;">Les inscriptions au 59&deg; congr&egrave;s des professeurs de phys=
ique et de chimie organis&eacute; par l'UdPPC sont ouvertes et le paieme=
nt par carte bancaire est op&eacute;rationnel !</span></p>
<p style=3D=
"text-align: center;"><span style=3D"font-size: small;">A vos clics !</spa=
n></p>
<p style=3D"text-align: center;"><span style=3D"font-size: sm=
all;"><span style=3D"font-family: 'Times New Roman','serif'; color: red; f=
ont-size: 12pt;"><a title=3D"http://udppc.asso.fr/" href=3D"http://montp=
ellier2011.udppc.asso.fr/"><span style=3D"font-family: 'Comic Sans MS'; co=
lor: red; font-size: 10pt;">http://montpellier2011.udppc.asso.fr/</span>=
</a></span></span></p>
<p style=3D"text-align: center;"><span styl=
e=3D"color: #000000; font-size: small;"><span style=3D"font-family: 'Tim=
es New Roman','serif'; font-size: 12pt;">Christophe REY</span></span></p=
>
<p style=3D"text-align: center;"><span style=3D"font-size: small;=
"><span style=3D"font-family: 'Times New Roman','serif'; color: red; font-=
size: 12pt;"><span style=3D"color: #000000;">Micheline Izbicki</span></spa=
n></span></p>

--_=_swift_v4_13167084834e7b6083a32ff_=_--


[-- Attachment #5: Type: text/plain, Size: 9 bytes --]


Julien.

Re: Automatic decoding of charsets

[Julien Cubizolles]

asjo@koldfront.dk (Adam Sjøgren) writes:

> On Mon, 26 Sep 2011 15:12:06 +0200, Julien wrote:
>
>> It wasn't a typo... I had copied the hook name from a web page. The hook
>> works now with 
>> ,----
>> | (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
>> `----
>
> In your first article you wrote that the exact same command didn't work
> - what changed?
>
> (I wouldn't expect it to work, since gnus-Article-prepare-hook doesn't
> exist in my Emacs, while gnus-article-prepare-hook does - but super if
> it works.)

My bad, the last one was indeed a typo. So:
,----
| (add-hook 'gnus-Article-prepare-hook 'gnus-article-decode-charset)
`----
doesn't work but:
,----
| (add-hook 'gnus-article-prepare-hook 'gnus-article-decode-charset)
`----
does as you had guessed.

>>> If you could provide an example of an article where it happens, other
>>> people could try and reproduce the problem, and that might shed some
>>> light on it.

I joined the complete file in another reply to Lars.

Thanks for your help,

Julien.

Re: Automatic decoding of charsets

[Lars Magne Ingebrigtsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Here it is, I just removed the addresses.

The difference is this:

-Content-type: text/plain
-Content-transfer-encoding: quoted-printable
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable

That is, the charset parameter goes AWOL if fetching partial articles,
which is interesting.

Could you switch on partial fetching again, and then select the
article.  After selecting it, go to the " *nnimap..." (note leading
space) buffer and send the contents of the buffer to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/