Re: [9fans] I don't understand utf8 (it seems)

[Antons Suspans <antox@ml.lv>]

On Mon, Jan 05, 2015 at 09:52:12PM +0000, Steve Simon wrote:
> I am trying to parse a stream from a tcp connection.
> 
> I think the data is utf8, here is a sample
> 
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
> 
> which when I print it I get:
> 
>      -       e  s  k       r  o  z  h  l  a  s           
>            ^          ^
>         missing    missing
> 
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
> 
> 	unicode c8 fd
> 	È
> 	ý
> 
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
> 
> thanks in advance of useful suggestions ☺
> 
> -Steve
> 
> 

Those might be ISO-8859-1 octets.
The first 256 codepoints of Unicode are those of ISO-8859-1.

% unicode -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | xd
- right away produces 18 octets representing 16 Unicode codepoints (listed as args).

% ascii -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | tcs -f 8859-1 | xd
- yields 16 octets which are then treated as ones encoding 16 codepoints in ISO-8859-1
and transformed to 18 octets in UTF-8 (representing those same 16 codepoints).

Hope this helps,

-- 
Antons