(Re-)coding problems forwarding calendar attachment

(Re-)coding problems forwarding calendar attachment

[Bjørn Mork]

I have problems forwarding calendar attachments using a current CVS
Gnus.

I don't really know if this posting will survive since I'm posting it
using Gnus, but I'm trying to force it to 8bit latin1 (knowing that
there are really utf-8 characters embedded within this article).

[Uh-oh, I am getting the magic question of desctruction (see below) when
posting this, so I'm pretty sure it will be screwed up. Trying
anyway. You may have to filter the result...]


This is how the original Microsoft generated blob looks like:


------_=_NextPart_001_01C7E4B1.D35AB8EB
Content-Type: text/calendar; method=REQUEST; name="meeting.ics"
Content-Transfer-Encoding: 8bit
Content-class: urn:content-classes:calendarmessage

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft CDO for Microsoft Exchange
VERSION:2.0
BEGIN:VTIMEZONE
TZID:(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna
X-MICROSOFT-CDO-TZID:4
BEGIN:STANDARD
DTSTART:16010101T030000
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=10;BYDAY=-1SU
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T020000
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20070822T114442Z
DTSTART;TZID="(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T14000
 0
SUMMARY:Test med Ø og Å
UID:{AF8C0874-B1EB-413A-87D4-1C56EC0B94B9}
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN="Mork Bjø
 rn (Networks)":MAILTO:bmork@example.net
ORGANIZER;CN="Mork Bjørn (IS  Nordic)":MAILTO:Bjorn.Mork@example.com
LOCATION:Bø
DTEND;TZID="(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T143000
DESCRIPTION:\NPrøver ut vCalendar tegnsett....\N\N\NBjørn\N
SEQUENCE:1
PRIORITY:5
CLASS:
CREATED:20070822T114440Z
LAST-MODIFIED:20070822T114440Z
STATUS:CONFIRMED
TRANSP:OPAQUE
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-OWNERAPPTID:-1
X-MICROSOFT-CDO-ATTENDEE-CRITICAL-CHANGE:20070822T114440Z
X-MICROSOFT-CDO-OWNER-CRITICAL-CHANGE:20070822T114442Z
END:VEVENT
END:VCALENDAR

------_=_NextPart_001_01C7E4B1.D35AB8EB--



So they are using
 - 8bit encoding,
 - mime-type text/calendar, 
 - default charset, 
 - vCalendar version 2.0

This all looks good to me.  According to RFC2445, the default charset
for an iCalendar object is UTF-8, and the text within the object looks
like valid UTF-8 to me.

If I try to forward a mail like this, Gnus will recode the attachment
using it's own defaults:

 - quoted-printable encoding,
 - mime-type text/calendar, 
 - charset utf-8, 
 - vCalendar version 2.0

But when sendting it I get the annoying question 

 "Non-printable characters found.  Continue sending? (d, r, i, e, ?)"

and even if I answer "i" to ignore, the contents
is changed resulting in invalid UTF-8:

--==-=-=
Content-Type: text/calendar; charset=utf-8; name=meeting.ics
Content-Transfer-Encoding: quoted-printable

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft CDO for Microsoft Exchange
VERSION:2.0
BEGIN:VTIMEZONE
TZID:(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna
X-MICROSOFT-CDO-TZID:4
BEGIN:STANDARD
DTSTART:16010101T030000
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
RRULE:FREQ=3DYEARLY;WKST=3DMO;INTERVAL=3D1;BYMONTH=3D10;BYDAY=3D-1SU
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T020000
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
RRULE:FREQ=3DYEARLY;WKST=3DMO;INTERVAL=3D1;BYMONTH=3D3;BYDAY=3D-1SU
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20070822T114442Z
DTSTART;TZID=3D"(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T140=
00
 0
SUMMARY:Test med =C3=83=C2=98 og =C3=83=C2=85
UID:{AF8C0874-B1EB-413A-87D4-1C56EC0B94B9}
ATTENDEE;ROLE=3DREQ-PARTICIPANT;PARTSTAT=3DNEEDS-ACTION;RSVP=3DTRUE;CN=3D"M=
ork Bj=C3=83=C2=B8
 rn (Networks)":MAILTO:bmork@example.net
ORGANIZER;CN=3D"Mork Bj=C3=83=C2=B8rn (IS  Nordic)":MAILTO:Bjorn.Mork@examp=
le.com
LOCATION:B=C3=83=C2=B8
DTEND;TZID=3D"(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T143000
DESCRIPTION:\NPr=C3=83=C2=B8ver ut vCalendar tegnsett....\N\N\NBj=C3=83=C2=
=B8rn\N
SEQUENCE:1
PRIORITY:5
CLASS:
CREATED:20070822T114440Z
LAST-MODIFIED:20070822T114440Z
STATUS:CONFIRMED
TRANSP:OPAQUE
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-OWNERAPPTID:-1
X-MICROSOFT-CDO-ATTENDEE-CRITICAL-CHANGE:20070822T114440Z
X-MICROSOFT-CDO-OWNER-CRITICAL-CHANGE:20070822T114442Z
END:VEVENT
END:VCALENDAR

--==-=-=--



If I try to forward a raw copy of the mail with the attachment (C-u 2
C-c C-f), I get the mime-headers unchanged as expected. But I get the
same annoying question and the contents is still changed adding a "Ã" in
front of the UTF-8 multibyte characters resulting in invalid UTF-8:

------_=_NextPart_001_01C7E4B1.D35AB8EB
Content-Type: text/calendar; method=REQUEST; name="meeting.ics"
Content-Transfer-Encoding: 8bit
Content-class: urn:content-classes:calendarmessage

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft CDO for Microsoft Exchange
VERSION:2.0
BEGIN:VTIMEZONE
TZID:(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna
X-MICROSOFT-CDO-TZID:4
BEGIN:STANDARD
DTSTART:16010101T030000
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=10;BYDAY=-1SU
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T020000
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20070822T114442Z
DTSTART;TZID="(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T14000
 0
SUMMARY:Test med Ø og Å
UID:{AF8C0874-B1EB-413A-87D4-1C56EC0B94B9}
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN="Mork Bjø
 rn (Networks)":MAILTO:bmork@example.net
ORGANIZER;CN="Mork Bjørn (IS  Nordic)":MAILTO:Bjorn.Mork@example.com
LOCATION:Bø
DTEND;TZID="(GMT+01.00) Berlin/Stockholm/Rome/Bern/Vienna":20070822T143000
DESCRIPTION:\NPrøver ut vCalendar tegnsett....\N\N\NBjørn\N
SEQUENCE:1
PRIORITY:5
CLASS:
CREATED:20070822T114440Z
LAST-MODIFIED:20070822T114440Z
STATUS:CONFIRMED
TRANSP:OPAQUE
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-OWNERAPPTID:-1
X-MICROSOFT-CDO-ATTENDEE-CRITICAL-CHANGE:20070822T114440Z
X-MICROSOFT-CDO-OWNER-CRITICAL-CHANGE:20070822T114442Z
END:VEVENT
END:VCALENDAR

------_=_NextPart_001_01C7E4B1.D35AB8EB--




Bjørn
-- 
Ban the war on freedom!

Re: (Re-)coding problems forwarding calendar attachment

[Katsumi Yamaoka]

>>>>> Bjørn Mork wrote:

> I have problems forwarding calendar attachments using a current CVS
> Gnus.

> I don't really know if this posting will survive since I'm posting it
> using Gnus, but I'm trying to force it to 8bit latin1 (knowing that
> there are really utf-8 characters embedded within this article).

Maybe it survived, if the changes that I made in the CVS trunk
solve your problem. ;-)

> This is how the original Microsoft generated blob looks like:

> ------_=_NextPart_001_01C7E4B1.D35AB8EB
> Content-Type: text/calendar; method=REQUEST; name="meeting.ics"
> Content-Transfer-Encoding: 8bit
> Content-class: urn:content-classes:calendarmessage

[...]

> So they are using
>  - 8bit encoding,
>  - mime-type text/calendar,
>  - default charset,
>  - vCalendar version 2.0

> This all looks good to me.  According to RFC2445, the default charset
> for an iCalendar object is UTF-8, and the text within the object looks
> like valid UTF-8 to me.

I've modified Gnus so as to use UTF-8 by default for decoding
text/calender parts.

> If I try to forward a mail like this, Gnus will recode the attachment
> using it's own defaults:

>  - quoted-printable encoding,
>  - mime-type text/calendar,
>  - charset utf-8,
>  - vCalendar version 2.0

> But when sendting it I get the annoying question

>  "Non-printable characters found.  Continue sending? (d, r, i, e, ?)"

> and even if I answer "i" to ignore, the contents
> is changed resulting in invalid UTF-8:

It is because Gnus doesn't decode the text/calendar part since
Gnus doesn't know the default charset for it, and inserts raw
UTF-8 data in the message buffer.

[...]

> If I try to forward a raw copy of the mail with the attachment (C-u 2
> C-c C-f), I get the mime-headers unchanged as expected.

(You don't need `C-u' in Gnus buffers; just type `2 C-c C-f'.)

Now you can simply use `C-c C-f' and Gnus will not make such a
question.  Even if you have to use `2 C-c C-f' in order not to
re-encode the text/calendar part, you will not be prompted for a
question now.  Or is it better that Gnus always uses the raw
form for text/calendar parts regardless of the prefix argument?

> But I get the same annoying question and the contents is still
> changed adding a "Ã" in front of the UTF-8 multibyte characters
> resulting in invalid UTF-8:

This is due to a bug in mml.el.

Re: (Re-)coding problems forwarding calendar attachment

[Bjørn Mork]

Katsumi Yamaoka <yamaoka@jpl.org> writes:
>>>>>> Bjørn Mork wrote:
>
>> This all looks good to me.  According to RFC2445, the default charset
>> for an iCalendar object is UTF-8, and the text within the object looks
>> like valid UTF-8 to me.
>
> I've modified Gnus so as to use UTF-8 by default for decoding
> text/calender parts.

Probably good, but may cause problems with pre-RFC (version 1.0)
text/calendar parts.  Let's hope everybody using those were sending
strict US-ASCII (possibly using QP to encode other charsets)

> It is because Gnus doesn't decode the text/calendar part since
> Gnus doesn't know the default charset for it, and inserts raw
> UTF-8 data in the message buffer.

I don't get the question anymore. Thanks.

But it still doesn't work.  The attachment is encoded as
quoted-printable and gets this header:

 Content-Type: text/calendar; charset=iso-8859-1; name=meeting.ics
 Content-Transfer-Encoding: quoted-printable

But it is not converted from utf-8, so that 'æ' is coded as '=C3=A6',
'å' as '=C3=A5' etc. 

Everything would have been fine if Gnus had just used

 Content-Type: text/calendar; charset=utf-8; name=meeting.ics
 Content-Transfer-Encoding: quoted-printable


Any idea where this confusion comes from?



Bjørn
-- 
I don't want to hear about your prostitute.

Re: (Re-)coding problems forwarding calendar attachment

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

>>>>> Bjørn Mork wrote:
> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> I've modified Gnus so as to use UTF-8 by default for decoding
>> text/calender parts.

> Probably good, but may cause problems with pre-RFC (version 1.0)
> text/calendar parts.  Let's hope everybody using those were sending
> strict US-ASCII (possibly using QP to encode other charsets)

Well, if the text/calendar part is encoded by arbitrary charset
other than UTF-8 but has no charset spec in the Content-Type
header, there will be no way to detect it automatically.  Though,
we can guess it and apply it through the `K C' command.

> But it still doesn't work.  The attachment is encoded as
> quoted-printable and gets this header:

>  Content-Type: text/calendar; charset=iso-8859-1; name=meeting.ics
>  Content-Transfer-Encoding: quoted-printable

> But it is not converted from utf-8, so that 'æ' is coded as '=C3=A6',
> 'å' as '=C3=A5' etc.

> Everything would have been fine if Gnus had just used

>  Content-Type: text/calendar; charset=utf-8; name=meeting.ics
>  Content-Transfer-Encoding: quoted-printable

> Any idea where this confusion comes from?

I probably have understood what is a prerequisite when one hands
iCalendar files to others.  The text/calendar part that has been
encoded by the iso-8859-1 charset is indeed a valid MIME message.
However, when a recipient saves the part in a file and passes it
to the calendar program, or passes it to the program through the
pipe, the charset is unlikely to be notified to the program.
Therefor, there needs to be an implicit agreement.  That is just
UTF-8, isn't it?

I've changed Gnus' encoder so that UTF-8 is most preferred when
encoding text/calendar parts.  Actually, now Gnus will never use
the charset other than UTF-8.

It is not a matter that the Content-Transfer-Encoding is QP or
base64, is it?  Gnus may use base64 as the case may be.  If it
has to always be QP, I have a plan:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1071 bytes --]

--- mml.el~	2007-08-24 08:17:16 +0000
+++ mml.el	2007-08-24 10:18:56 +0000
@@ -535,15 +535,18 @@
 			;; insert a "; format=flowed" string unless the
 			;; user has already specified it.
 			(setq flowed (null (assq 'format cont)))))
-		    ;; Prefer `utf-8' for text/calendar parts.
+		    ;; Prefer `utf-8' and QP for text/calendar parts.
 		    (if (or charset
 			    (not (string= type "text/calendar")))
-			(setq charset (mm-encode-body charset))
+			(setq charset (mm-encode-body charset)
+			      encoding (mm-body-encoding
+					charset (cdr (assq 'encoding cont))))
 		      (let ((mm-coding-system-priorities
 			     (cons 'utf-8 mm-coding-system-priorities)))
-			(setq charset (mm-encode-body))))
-		    (setq encoding (mm-body-encoding
-				    charset (cdr (assq 'encoding cont))))))
+			(setq charset (mm-encode-body)
+			      encoding (mm-body-encoding
+					charset (or (cdr (assq 'encoding cont))
+						    'quoted-printable)))))))
 		  (setq coded (buffer-string)))
 		(mml-insert-mime-headers cont type charset encoding flowed)
 		(insert "\n")

[-- Attachment #3: Type: text/plain, Size: 71 bytes --]

P.S.
I will be inactive in the net for a week because of the holidays.