[9fans] I don't understand utf8 (it seems)

[9fans] I don't understand utf8 (it seems)

[Steve Simon]

I am trying to parse a stream from a tcp connection.

I think the data is utf8, here is a sample

	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73

which when I print it I get:

     -       e  s  k       r  o  z  h  l  a  s           
           ^          ^
        missing    missing

there are two missing characters. Ok, bad UTF8 perhaps?
but when I try unicode(1) I see:

	unicode c8 fd
	È
	ý

Is this 8 bit runes? (!)
Is there a name for such a thing?
Is this common?
Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?

thanks in advance of useful suggestions ☺

-Steve

Re: [9fans] I don't understand utf8 (it seems)

[erik quanstrom]

On Mon Jan  5 13:48:47 PST 2015, steve@quintile.net wrote:
> I am trying to parse a stream from a tcp connection.
> 
> I think the data is utf8, here is a sample
> 
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
> 
> which when I print it I get:
> 
>      -       e  s  k       r  o  z  h  l  a  s           
>            ^          ^
>         missing    missing
> 
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
> 
> 	unicode c8 fd
> 	È
> 	ý
> 
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?

latin1 has this property that if you embed the byte in a rune-sized chunk, then it's
a valid Rune.  but latin1 is invalid utf-8.

  the reason that unicode(1) failed to meet expectations, is that the desire was 
to convert the supposed utf-8 0xc8 to a codepoint, but what unicode did was
convert the codepoint 0xc8 into utf-8.

- erik

Re: [9fans] I don't understand utf8 (it seems)

[Quintile]

ok, I understand, what I thought was UTF8 is in fact latin1.

thanks. that makes sense.

-Steve





> On 5 Jan 2015, at 22:05, erik quanstrom <quanstro@quanstro.net> wrote:
> 
>> On Mon Jan  5 13:48:47 PST 2015, steve@quintile.net wrote:
>> I am trying to parse a stream from a tcp connection.
>> 
>> I think the data is utf8, here is a sample
>> 
>>     20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>> 
>> which when I print it I get:
>> 
>>     -       e  s  k       r  o  z  h  l  a  s           
>>           ^          ^
>>        missing    missing
>> 
>> there are two missing characters. Ok, bad UTF8 perhaps?
>> but when I try unicode(1) I see:
>> 
>>    unicode c8 fd
>>    È
>>    ý
>> 
>> Is this 8 bit runes? (!)
>> Is there a name for such a thing?
>> Is this common?
>> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
> 
> latin1 has this property that if you embed the byte in a rune-sized chunk, then it's
> a valid Rune.  but latin1 is invalid utf-8.
> 
>  the reason that unicode(1) failed to meet expectations, is that the desire was 
> to convert the supposed utf-8 0xc8 to a codepoint, but what unicode did was
> convert the codepoint 0xc8 into utf-8.
> 
> - erik

Re: [9fans] I don't understand utf8 (it seems)

[Antons Suspans]

On Mon, Jan 05, 2015 at 09:52:12PM +0000, Steve Simon wrote:
> I am trying to parse a stream from a tcp connection.
> 
> I think the data is utf8, here is a sample
> 
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
> 
> which when I print it I get:
> 
>      -       e  s  k       r  o  z  h  l  a  s           
>            ^          ^
>         missing    missing
> 
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
> 
> 	unicode c8 fd
> 	È
> 	ý
> 
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
> 
> thanks in advance of useful suggestions ☺
> 
> -Steve
> 
> 

Those might be ISO-8859-1 octets.
The first 256 codepoints of Unicode are those of ISO-8859-1.

% unicode -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | xd
- right away produces 18 octets representing 16 Unicode codepoints (listed as args).

% ascii -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | tcs -f 8859-1 | xd
- yields 16 octets which are then treated as ones encoding 16 codepoints in ISO-8859-1
and transformed to 18 octets in UTF-8 (representing those same 16 codepoints).

Hope this helps,

-- 
Antons

Re: [9fans] I don't understand utf8 (it seems)

[Bakul Shah]

On Mon, 05 Jan 2015 21:52:12 GMT "Steve Simon" <steve@quintile.net> wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
>      -       e  s  k       r  o  z  h  l  a  s          =20
>            ^          ^
>         missing    missing
>
> there are two missing characters. Ok, bad UTF8 perhaps?

According to http://www.isthisthingon.org/unicode/index.php
this is an invalid UTF8 hex code.

Re: [9fans] I don't understand utf8 (it seems)

[Matěj Cepl]

On 2015-01-05, 21:52 GMT, Steve Simon wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 	 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
>      -       e  s  k       r  o  z  h  l  a  s           
>            ^          ^
>         missing    missing

I don't know whether it helps, but I would bet (being a native 
Czech) that this is actually a name of the Czech radio, which is

    Český rozhlas

Not sure if it helps.

Matěj

Re: [9fans] I don't understand utf8 (it seems)

[Quintile]

indeed, Czech jazz, excellent station.

I have restarted on my (plan9) internet radio... 😄

-Steve





> On 6 Jan 2015, at 19:57, Matěj Cepl <mcepl@cepl.eu> wrote:
> 
>> On 2015-01-05, 21:52 GMT, Steve Simon wrote:
>> I am trying to parse a stream from a tcp connection.
>> 
>> I think the data is utf8, here is a sample
>> 
>>     20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>> 
>> which when I print it I get:
>> 
>>     -       e  s  k       r  o  z  h  l  a  s           
>>           ^          ^
>>        missing    missing
> 
> I don't know whether it helps, but I would bet (being a native 
> Czech) that this is actually a name of the Czech radio, which is
> 
>    Český rozhlas
> 
> Not sure if it helps.
> 
> Matěj
> 

Re: [9fans] I don't understand utf8 (it seems)

[a.f.e.belinfante]

This seems to suggest that you are not dealing with latin1, but with latin2.
Adapting what Antons wrote:

% ascii -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | tcs -f 8859-2
- Český rozhlas

Axel.


On 06 Jan 2015, at 23:09 , Quintile <steve@quintile.net> wrote:

> indeed, Czech jazz, excellent station.
> 
> I have restarted on my (plan9) internet radio... 😄
> 
> -Steve
> 
> 
>> On 6 Jan 2015, at 19:57, Matěj Cepl <mcepl@cepl.eu> wrote:
>> 
>>> On 2015-01-05, 21:52 GMT, Steve Simon wrote:
>>> I am trying to parse a stream from a tcp connection.
>>> 
>>> I think the data is utf8, here is a sample
>>> 
>>>    20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>>> 
>>> which when I print it I get:
>>> 
>>>    -       e  s  k       r  o  z  h  l  a  s           
>>>          ^          ^
>>>       missing    missing
>> 
>> I don't know whether it helps, but I would bet (being a native 
>> Czech) that this is actually a name of the Czech radio, which is
>> 
>>   Český rozhlas
>> 
>> Not sure if it helps.
>> 
>> Matěj