[9fans] char encoding problem

[9fans] char encoding problem

[Mathieu Lonjaret]

Hello 9fans,

I have a problem with one specific e-mail in p9p acme Mail: only some
part of it is displayed (~1/2) when I open it (right click) in acme
Mail. According to mutt, the encoding is iso-8859-15 and I can read it
fine there.
There does not seem to be anything special in the first line which is
"not there" in acme Mail; some accented characters but not different
from other ones which appear sooner in the mail.

As turjo advised me on #plan9, I tried
'9p read mail/mbox/n/raw'
in an acme win. That outputed the whole message, however the accented
chars were replaced by runes like that one: �
I also fetched the latest p9p (20080711); same behaviour.

Any idea about what's going on please?

Cheers,
Mathieu.

Re: [9fans] char encoding problem

[andrey mirtchovski]

you can try using tcs to convert the email to utf, but the man page
doesn't say whether tcs speaks 8859-15, but the p9p program says it
does (tcs -lv):

http://swtch.com/plan9port/man/man1/tcs.html

On Fri, Jul 11, 2008 at 9:55 AM, Mathieu Lonjaret <lejatorn@gmail.com> wrote:
> Hello 9fans,
>
> I have a problem with one specific e-mail in p9p acme Mail: only some
> part of it is displayed (~1/2) when I open it (right click) in acme
> Mail. According to mutt, the encoding is iso-8859-15 and I can read it
> fine there.
> There does not seem to be anything special in the first line which is
> "not there" in acme Mail; some accented characters but not different
> from other ones which appear sooner in the mail.
>
> As turjo advised me on #plan9, I tried
> '9p read mail/mbox/n/raw'
> in an acme win. That outputed the whole message, however the accented
> chars were replaced by runes like that one: �
> I also fetched the latest p9p (20080711); same behaviour.
>
> Any idea about what's going on please?
>
> Cheers,
> Mathieu.
>
>
>
>

Re: [9fans] char encoding problem

[Mathieu Lonjaret]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

Yes,
9p read mail/mbox/40/raw | tcs -f 8859-15
seems to be working fine, thanks.

Then any idea why acme Mail has a problem with this message?

Cheers,
Mathieu.

[-- Attachment #2: Type: message/rfc822, Size: 4723 bytes --]

From: "andrey mirtchovski" <mirtchovski@gmail.com>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@9fans.net>
Subject: Re: [9fans] char encoding problem
Date: Fri, 11 Jul 2008 09:56:40 -0600
Message-ID: <14ec7b180807110856r30c90fa5h18ddc2fb9dacf2b1@mail.gmail.com>

you can try using tcs to convert the email to utf, but the man page
doesn't say whether tcs speaks 8859-15, but the p9p program says it
does (tcs -lv):

http://swtch.com/plan9port/man/man1/tcs.html

On Fri, Jul 11, 2008 at 9:55 AM, Mathieu Lonjaret <lejatorn@gmail.com> wrote:
> Hello 9fans,
>
> I have a problem with one specific e-mail in p9p acme Mail: only some
> part of it is displayed (~1/2) when I open it (right click) in acme
> Mail. According to mutt, the encoding is iso-8859-15 and I can read it
> fine there.
> There does not seem to be anything special in the first line which is
> "not there" in acme Mail; some accented characters but not different
> from other ones which appear sooner in the mail.
>
> As turjo advised me on #plan9, I tried
> '9p read mail/mbox/n/raw'
> in an acme win. That outputed the whole message, however the accented
> chars were replaced by runes like that one: �
> I also fetched the latest p9p (20080711); same behaviour.
>
> Any idea about what's going on please?
>
> Cheers,
> Mathieu.
>
>
>
>

Re: [9fans] char encoding problem

[erik quanstrom]

> Yes,
> 9p read mail/mbox/40/raw | tcs -f 8859-15
> seems to be working fine, thanks.
>
> Then any idea why acme Mail has a problem with this message?

likely the mime character set string is different
from what tcs expects.

- erik

Re: [9fans] char encoding problem

[Russ Cox]

> I have a problem with one specific e-mail in p9p acme Mail: only some
> part of it is displayed (~1/2) when I open it (right click) in acme
> Mail. According to mutt, the encoding is iso-8859-15 and I can read it
> fine there.
> There does not seem to be anything special in the first line which is
> "not there" in acme Mail; some accented characters but not different
> from other ones which appear sooner in the mail.

acme couldn't care less.
mailfs is converting the message to utf-8
(or not) according to the content headers.

9p read mail/mbox/40/raw |grep -i '^content'
might tell something useful,
like what character set the message
is claiming to be in.

russ

Re: [9fans] char encoding problem

[Mathieu Lonjaret]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

% 9p read mail/mbox/40/raw |grep -i '^content'
Content-Type: text/plain; charset=
Content-Transfer-Encoding: 8bit

Does it mean that the character set is in iso-8859-15 but the sender
of the message somehow set it wrongly in the headers to be iso-8859-1?
Hence mailfs is failing at trying to convert it from iso-8859-1 to utf-8
because it trusts what's written in the headers?

Thanks,
Mathieu.

[-- Attachment #2: Type: message/rfc822, Size: 3617 bytes --]

From: "Russ Cox" <rsc@swtch.com>
To: 9fans@9fans.net
Subject: Re: [9fans] char encoding problem
Date: Mon, 14 Jul 2008 12:31:56 -0400
Message-ID: <20080714163031.DDC181E8C45@holo.morphisms.net>

> I have a problem with one specific e-mail in p9p acme Mail: only some
> part of it is displayed (~1/2) when I open it (right click) in acme
> Mail. According to mutt, the encoding is iso-8859-15 and I can read it
> fine there.
> There does not seem to be anything special in the first line which is
> "not there" in acme Mail; some accented characters but not different
> from other ones which appear sooner in the mail.

acme couldn't care less.
mailfs is converting the message to utf-8
(or not) according to the content headers.

9p read mail/mbox/40/raw |grep -i '^content'
might tell something useful,
like what character set the message
is claiming to be in.

russ

Re: [9fans] char encoding problem

[erik quanstrom]

> % 9p read mail/mbox/40/raw |grep -i '^content'
> Content-Type: text/plain; charset=
> Content-Transfer-Encoding: 8bit
>
> Does it mean that the character set is in iso-8859-15 but the sender
> of the message somehow set it wrongly in the headers to be iso-8859-1?
> Hence mailfs is failing at trying to convert it from iso-8859-1 to utf-8
> because it trusts what's written in the headers?

it's hard to tell from this.  the charset appears missing
entirely.  it may be that upas has misparsed the header.

you may need to read all of mail/mbox/40/rawheader
to find if the charset was really set to something.

if it is misparsed, i'd like to fix the problem.  you could
send me the raw header offline.

- erik