Re: More liberal MIME decoding (unencoded question marks in encoded words)

[Katsumi Yamaoka <yamaoka@jpl.org>]

I've installed the new ones in the Gnus trunk.  Decoding bad Q
encoding is enabled by default.

>>>>> Reiner Steib wrote:
> On Mon, Nov 26 2007, Katsumi Yamaoka wrote:

>> Would we be able to make complete test cases?
>>
>> (rfc2047-decode-string "=?ISO-8859-1?Q??foo?=")
>> "?foo"

[...]

> Do you see the other examples often in the wild?

No, I've never seen such ones at all, though I always examine
raw data when decoding fails.  What I saw were mainly broken B
encoding (99.9% of Japanese MIME messages use B encoding).

> If not, I'd rather not make the decode too liberal.

I thought it's not going too far since it doesn't support encoded
words folded into two or more lines.  In reality, there's the
reason I didn't make it support newlines in encoded words.  Because
the regexp pattern for Q encoding is ambiguous in a sense, if it
supports newlines, it might lead re-search to get stuck with an
encoded word that is not terminated with "?=".

FYI:

> +\\(B\\?[+/0-9A-Za-z]*=*\

This pattern is restricted into only the characters that B
encoding uses, since the base64 decoder doesn't work with data
containing other characters.

> +\\|Q\\?\\(?:\\?+[ -<>@-~]\\)?\\(?:[ ->@-~]+\\?+[ -<>@-~]\\)*[ ->@-~]*\\?*\
> +\\)\\?="))

This pattern is similar to:

"Q\\?\\(\\?+[^\n=?]\\)?\\([^\n?]+\\?+[^\n=?]\\)*[^\n?]*\\?*"
     <--------1-------><----------2,3----------><--4--><-5->

1. After "Q?", allow "?"s that follow a character other than "=".
2. Allow "=" after "Q?"; it isn't regarded as the terminator.
3. In the middle of an encoded word, allow "?"s that follow a
   character other than "=".
4. Allow any characters other than "?" in the middle of an
   encoded word.
5. At the end, allow "?"s.

> And we probably should have an option to toggle strict/loose
> decoding.

I've introduced the `rfc2047-allow-irregular-q-encoded-words'
option.  I wish that it is tested widely, so I've set the default
value to t.  But it might have to be nil when it is imported into
the stable branch.  Now there are two regexps; one is
`rfc2047-encoded-word-regexp' for strict decoding, the other is
`rfc2047-encoded-word-regexp-loose'.

> BTW, another problem is that we "double encode"
> (`rfc2047-encode-encoded-words') such subjects:

ELISP> (rfc2047-decode-string "=?ISO-8859-1?Q?foo??=")
> "=?ISO-8859-1?Q?foo??="
ELISP> (rfc2047-encode-string "=?ISO-8859-1?Q?foo??=")
> "=?us-ascii?Q?=3D=3FISO-8859-1=3FQ=3Ffoo=3F=3F=3D?="

> AFAICS, Gnus (`rfc2047-encodable-p'?) simply looks for "=?".

[...]

> ..., i.e. shouldn't we use "=\\?.+\\?[qb]\\?.+\\?=" (or similar)
> instead of "=?"?

I agree with you.  I've made `rfc2047-encodable-p' use
`rfc2047-encoded-word-regexp' instead of "=?".  It will be hard
to be found out even if this change causes another trouble, though.

Regards,