Re: [9fans] Octets regexp

[tlaronde@polynum.com]

On Thu, May 02, 2013 at 08:48:06AM -0400, erik quanstrom wrote:
> > Regexp(6) handles "characters" that are runes.
>
> perhaps the man page is misleading.  rune in this context means utf-8.
> see regexp(2).  all the functions take char*s.

But the source files deal with runes...

>
> one of the points of plan 9 was to standardize on one character set,
> utf-8.  imho, localization and character set aren't related unless one
> is dealing with 8859-x overlays or some other character set insufficient
> to represent the range of languages.
>

Localization (as "handled" in POSIX for example) is a mess. So the Plan9
solution, with still octets (UTF-8) makes far more sense, since it
allows to extend, for the user, the "characters" that can be used in
naming computer objects, but this is just for nicknames: the system
still speaks C/9P.

So it is better, except perhaps for one thing: for me, the system
"speaks" C or even, obviously, "Plan9" (well: 9P). It does not have
to speak french, hebrew, etc. or even english! So it takes or gives
bytes, and this is good.  But the UTF-8 encoding is the main convention
for user interface, but can it be unset? I mean, can one use a
"raw" window, putting uninterpreted bytes, and rendering bytes (with
a special "ASCII" font with whether ASCII + "0xdd" glyphes or whatever,
using fonts to do what is done with vis(1) on Unices or od(1)/xd(1))
and do not impose the assumption that the octet strings is UTF-8? Can
one make a file entering bytes---i.e. binary values that yield
incorrect UTF-8 sequences?

This is a reflexion made to me by a developer who can use, when
needed, regexp (ed(1) or sed(1)) on an Unix where they still deal
with "char" (bytes) to search for a string of bytes in a binary.

And after some thought, I don't see an obvious reason why the regexp
could not be used with bytes strings (so UTF-8 is OK) without trying to
match runes (since not every bytes string is a correct UTF-8 sequence).

Corollary: I don't know if there is an UTF-8 sequence that can tell:
stop interpreting as UTF-8, takes "as is" (except every incorrect
sequence, problem being to come back from there: if everything is OK "as
is", what can be interpreted as: "stops raw, restart
UTF-8"---solution: this is on user level, not low level, and this is in
the shell explicitely delimiting chunks, like "'" is the only delimiter,
and every embedded "'" has to be "escaped" by doubling it).
--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C