Re: Gnus: UTF-8 and compatibility with other MUAs

[Simon Josefsson <jas@extundo.com>]

Oliver Scholz <alkibiades@gmx.de> writes:

> [...]
>> UTF-16?  It's not even a well define encoding scheme, two files may
>> contain the exact same Unicode code points, but may differ in a binary
>> comparison, due to byte ordering.  
>
> That's what the byte order mark is for.

But it doesn't solve the problem. 'cmp' still says the files are
different.  UTF-8 had a similar problem (overlong encodings) but that
has been fixed, UTF-16 and UTF-32 can't be.

>> And concatenating two UTF-16 strings from different sources requires
>> knowledge about the encoding. And surrogate pairs complicate matters
>> as well.
>
> Why do you think that surrogate pairs complicate matters? There can't
> be any confusion whether an arbitrary 16 bit value is part a surrogate
> pair or not; and if it is, whether it is the higher surrogate or the
> lower one.

One way to realize it is to compare UTF-16 with either UTF-8 or
UTF-32.  The surrogate pair construction make UTF-16 contain the
disadvantage of both UTF-8 and UTF-32, but none of their advantage.

The disadvantage with UTF-8 is that you don't know where a code value
ends within the encoded data without knowledge of UTF-8, and the
disadvantage with UTF-32 is that it wastes space since most data fit
in 16 bits or less.

If normal computers was 16 bit, I could understand the trade-off, but
with 32 bit (or more) machines you can remove one of the disadvantages
by choosing either UTF-8 or UTF-32 instead of UTF-16.

> As for concatenating I'd say this depends on whether the tools are
> able to deal with it.

Right, and many tools assume that if you receive two binary blobs A
and B which are said to contain text, you can form the concatenation
of the text by concatenating the binary blobs as A||B.  This is a
reasonable assumption, and it works for most encoding schemes,
including UTF-8.  It doesn't work for UTF-16 or UTF-32.

My preference is to use UTF-8 when data is stored or transfered, and
only use UTF-32 internally because applications may need to compare
data against Unicode code points.  If I must use Unicode at all, that
is.