* [9fans] I don't understand utf8 (it seems)
@ 2015-01-05 21:52 Steve Simon
2015-01-05 22:05 ` erik quanstrom
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Steve Simon @ 2015-01-05 21:52 UTC (permalink / raw)
To: 9fans
I am trying to parse a stream from a tcp connection.
I think the data is utf8, here is a sample
20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
which when I print it I get:
- e s k r o z h l a s
^ ^
missing missing
there are two missing characters. Ok, bad UTF8 perhaps?
but when I try unicode(1) I see:
unicode c8 fd
È
ý
Is this 8 bit runes? (!)
Is there a name for such a thing?
Is this common?
Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
thanks in advance of useful suggestions ☺
-Steve
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-05 21:52 [9fans] I don't understand utf8 (it seems) Steve Simon
@ 2015-01-05 22:05 ` erik quanstrom
2015-01-05 22:27 ` Quintile
2015-01-05 22:15 ` Antons Suspans
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: erik quanstrom @ 2015-01-05 22:05 UTC (permalink / raw)
To: 9fans
On Mon Jan 5 13:48:47 PST 2015, steve@quintile.net wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
> - e s k r o z h l a s
> ^ ^
> missing missing
>
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
>
> unicode c8 fd
> È
> ý
>
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
latin1 has this property that if you embed the byte in a rune-sized chunk, then it's
a valid Rune. but latin1 is invalid utf-8.
the reason that unicode(1) failed to meet expectations, is that the desire was
to convert the supposed utf-8 0xc8 to a codepoint, but what unicode did was
convert the codepoint 0xc8 into utf-8.
- erik
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-05 21:52 [9fans] I don't understand utf8 (it seems) Steve Simon
2015-01-05 22:05 ` erik quanstrom
@ 2015-01-05 22:15 ` Antons Suspans
2015-01-05 22:31 ` Bakul Shah
2015-01-06 19:57 ` Matěj Cepl
3 siblings, 0 replies; 8+ messages in thread
From: Antons Suspans @ 2015-01-05 22:15 UTC (permalink / raw)
To: Fans of the OS Plan 9 from Bell Labs
On Mon, Jan 05, 2015 at 09:52:12PM +0000, Steve Simon wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
> - e s k r o z h l a s
> ^ ^
> missing missing
>
> there are two missing characters. Ok, bad UTF8 perhaps?
> but when I try unicode(1) I see:
>
> unicode c8 fd
> È
> ý
>
> Is this 8 bit runes? (!)
> Is there a name for such a thing?
> Is this common?
> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
>
> thanks in advance of useful suggestions ☺
>
> -Steve
>
>
Those might be ISO-8859-1 octets.
The first 256 codepoints of Unicode are those of ISO-8859-1.
% unicode -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | xd
- right away produces 18 octets representing 16 Unicode codepoints (listed as args).
% ascii -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | tcs -f 8859-1 | xd
- yields 16 octets which are then treated as ones encoding 16 codepoints in ISO-8859-1
and transformed to 18 octets in UTF-8 (representing those same 16 codepoints).
Hope this helps,
--
Antons
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-05 22:05 ` erik quanstrom
@ 2015-01-05 22:27 ` Quintile
0 siblings, 0 replies; 8+ messages in thread
From: Quintile @ 2015-01-05 22:27 UTC (permalink / raw)
To: Fans of the OS Plan 9 from Bell Labs
ok, I understand, what I thought was UTF8 is in fact latin1.
thanks. that makes sense.
-Steve
> On 5 Jan 2015, at 22:05, erik quanstrom <quanstro@quanstro.net> wrote:
>
>> On Mon Jan 5 13:48:47 PST 2015, steve@quintile.net wrote:
>> I am trying to parse a stream from a tcp connection.
>>
>> I think the data is utf8, here is a sample
>>
>> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>>
>> which when I print it I get:
>>
>> - e s k r o z h l a s
>> ^ ^
>> missing missing
>>
>> there are two missing characters. Ok, bad UTF8 perhaps?
>> but when I try unicode(1) I see:
>>
>> unicode c8 fd
>> È
>> ý
>>
>> Is this 8 bit runes? (!)
>> Is there a name for such a thing?
>> Is this common?
>> Is it just MS code pages but the >0x7f values happen (designed to) to map onto the same letters as utf8?
>
> latin1 has this property that if you embed the byte in a rune-sized chunk, then it's
> a valid Rune. but latin1 is invalid utf-8.
>
> the reason that unicode(1) failed to meet expectations, is that the desire was
> to convert the supposed utf-8 0xc8 to a codepoint, but what unicode did was
> convert the codepoint 0xc8 into utf-8.
>
> - erik
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-05 21:52 [9fans] I don't understand utf8 (it seems) Steve Simon
2015-01-05 22:05 ` erik quanstrom
2015-01-05 22:15 ` Antons Suspans
@ 2015-01-05 22:31 ` Bakul Shah
2015-01-06 19:57 ` Matěj Cepl
3 siblings, 0 replies; 8+ messages in thread
From: Bakul Shah @ 2015-01-05 22:31 UTC (permalink / raw)
To: Fans of the OS Plan 9 from Bell Labs
On Mon, 05 Jan 2015 21:52:12 GMT "Steve Simon" <steve@quintile.net> wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
> - e s k r o z h l a s =20
> ^ ^
> missing missing
>
> there are two missing characters. Ok, bad UTF8 perhaps?
According to http://www.isthisthingon.org/unicode/index.php
this is an invalid UTF8 hex code.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-05 21:52 [9fans] I don't understand utf8 (it seems) Steve Simon
` (2 preceding siblings ...)
2015-01-05 22:31 ` Bakul Shah
@ 2015-01-06 19:57 ` Matěj Cepl
2015-01-06 22:09 ` Quintile
3 siblings, 1 reply; 8+ messages in thread
From: Matěj Cepl @ 2015-01-06 19:57 UTC (permalink / raw)
To: 9fans
On 2015-01-05, 21:52 GMT, Steve Simon wrote:
> I am trying to parse a stream from a tcp connection.
>
> I think the data is utf8, here is a sample
>
> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>
> which when I print it I get:
>
> - e s k r o z h l a s
> ^ ^
> missing missing
I don't know whether it helps, but I would bet (being a native
Czech) that this is actually a name of the Czech radio, which is
Český rozhlas
Not sure if it helps.
Matěj
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-06 19:57 ` Matěj Cepl
@ 2015-01-06 22:09 ` Quintile
2015-01-07 9:43 ` a.f.e.belinfante
0 siblings, 1 reply; 8+ messages in thread
From: Quintile @ 2015-01-06 22:09 UTC (permalink / raw)
To: Fans of the OS Plan 9 from Bell Labs
indeed, Czech jazz, excellent station.
I have restarted on my (plan9) internet radio... 😄
-Steve
> On 6 Jan 2015, at 19:57, Matěj Cepl <mcepl@cepl.eu> wrote:
>
>> On 2015-01-05, 21:52 GMT, Steve Simon wrote:
>> I am trying to parse a stream from a tcp connection.
>>
>> I think the data is utf8, here is a sample
>>
>> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>>
>> which when I print it I get:
>>
>> - e s k r o z h l a s
>> ^ ^
>> missing missing
>
> I don't know whether it helps, but I would bet (being a native
> Czech) that this is actually a name of the Czech radio, which is
>
> Český rozhlas
>
> Not sure if it helps.
>
> Matěj
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] I don't understand utf8 (it seems)
2015-01-06 22:09 ` Quintile
@ 2015-01-07 9:43 ` a.f.e.belinfante
0 siblings, 0 replies; 8+ messages in thread
From: a.f.e.belinfante @ 2015-01-07 9:43 UTC (permalink / raw)
To: 9fans
This seems to suggest that you are not dealing with latin1, but with latin2.
Adapting what Antons wrote:
% ascii -t 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73 | tcs -f 8859-2
- Český rozhlas
Axel.
On 06 Jan 2015, at 23:09 , Quintile <steve@quintile.net> wrote:
> indeed, Czech jazz, excellent station.
>
> I have restarted on my (plan9) internet radio... 😄
>
> -Steve
>
>
>> On 6 Jan 2015, at 19:57, Matěj Cepl <mcepl@cepl.eu> wrote:
>>
>>> On 2015-01-05, 21:52 GMT, Steve Simon wrote:
>>> I am trying to parse a stream from a tcp connection.
>>>
>>> I think the data is utf8, here is a sample
>>>
>>> 20 2d 20 c8 65 73 6b fd 20 72 6f 7a 68 6c 61 73
>>>
>>> which when I print it I get:
>>>
>>> - e s k r o z h l a s
>>> ^ ^
>>> missing missing
>>
>> I don't know whether it helps, but I would bet (being a native
>> Czech) that this is actually a name of the Czech radio, which is
>>
>> Český rozhlas
>>
>> Not sure if it helps.
>>
>> Matěj
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-01-07 9:43 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-05 21:52 [9fans] I don't understand utf8 (it seems) Steve Simon
2015-01-05 22:05 ` erik quanstrom
2015-01-05 22:27 ` Quintile
2015-01-05 22:15 ` Antons Suspans
2015-01-05 22:31 ` Bakul Shah
2015-01-06 19:57 ` Matěj Cepl
2015-01-06 22:09 ` Quintile
2015-01-07 9:43 ` a.f.e.belinfante
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).