Re: [9fans] UTF-8 criticism?

["Joel Salomon" <salomo3@cooper.edu>]

Joel Salomon said:
> As an aside, the way I've understood the Unicode standard (4.0), 21 bit
> characters can be encoded in 1, 2, 3, or 4 bytes in UTF-8 and if text is
> internally represented by int32, some out-of-band information (like EOF,
> or bad UTF (but preserving the original bytes)) can be carried along.
>

And here's where the out-of-band encoding might come in useful:

rog@vitanuova.com said:
> you do have to be a bit careful with utf-8, as many possible byte
> sequences map down to the same rune (error), so if you
> do your comparisons too early, you run the risk of inconsistency.
>
> for instance, you can exploit this (at least, i *think* this is the
> cause) to create a file that can never be removed on ken's fileserver:
<snip>

but if "error" becomes 0x80000000 & XX, where XX is the original (bad, or
out-of-place) byte, we never lose the ability to retrieve/delete the file.
This would be an extension to Unicode, possibly a dangerous one, but maybe
worth considering.

--Joel