Re: [9fans] I don't understand utf8 (it seems)

[erik quanstrom <quanstro@quanstro.net>]

On Mon Jan  5 13:48:47 PST 2015, steve@quintile.net wrote:
> I am trying to parse a stream from a tcp connection.
> 
> I think the data is utf8, here is a sample
> 
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
> 
> which when I print it I get:
> 
>      -       e  s  k       r  o  z  h  l  a  s           
>            ^          ^
>         missing    missing
> 
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
> 
> 	unicode c8 fd
> 	È
> 	ý
> 
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?

latin1 has this property that if you embed the byte in a rune-sized chunk, then it's
a valid Rune.  but latin1 is invalid utf-8.

  the reason that unicode(1) failed to meet expectations, is that the desire was 
to convert the supposed utf-8 0xc8 to a codepoint, but what unicode did was
convert the codepoint 0xc8 into utf-8.

- erik