[9fans] Lex, Yacc, Unicode Plane 1

[9fans] Lex, Yacc, Unicode Plane 1

[Karljurgen Feuerherm]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

Hello,
 
A colleague put me on to Plan9, some of whose online documentation I
have read with interest, in particular the "Hello World" discussion as
it relates to Unicode/UTF-8.
 
I'm one of the authors of the Cuneiform proposal now encoded under
Unicode (see block U+12000), and I'm interesting in lex/yacc-like
parsing of Unicode input to produce (among other things) Cuneiform
output.
 
I realize some of the documentation was written long ago... so I'm
unclear as to whether or not (or how easily) Plan9 (and specifically its
lex/yacc software, etc.) handles such things? (this sparked by the
references to four hex digits etc.)
 
Many thanks if you can point me in the right direction :) (or to an
alternative solution, if need be!)
 
Best
 
K
 
Karljürgen G. Feuerherm, PhD
Department of Archaeology and Classical Studies
Wilfrid Laurier University
75 University Avenue West
Waterloo, Ontario N2L 3C5
Tel. (519) 884-1970 x3193
Fax (519) 883-0991 (ATTN Arch. & Classics)

[-- Attachment #2: HTML --]
[-- Type: text/html, Size: 1430 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

> A colleague put me on to Plan9, some of whose online documentation I
> have read with interest, in particular the "Hello World" discussion as
> it relates to Unicode/UTF-8.
>
> I'm one of the authors of the Cuneiform proposal now encoded under
> Unicode (see block U+12000), and I'm interesting in lex/yacc-like
> parsing of Unicode input to produce (among other things) Cuneiform
> output.
>
> I realize some of the documentation was written long ago... so I'm
> unclear as to whether or not (or how easily) Plan9 (and specifically its
> lex/yacc software, etc.) handles such things? (this sparked by the
> references to four hex digits etc.)

that's interesting stuff.

lex(1) is generally not used, and doesn't support
unicode.  yacc(1) does a fine job with unicode.
though, to be fair, most of that job falls on the
lexer.  however this is not hard to do by hand. there
are many good examples in the distribution.
the bio(2) buffered io library provides a Bgetrune
function, which is generally what is desired.

(i have some patches, partially stolen from russ,
that should support extended plane runes at the
cost of double the storage.)

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[geoff]

I've extended old code using lex to accept utf by massaging the input
stream, before lex sees it, to parse utf and encode non-ascii Runes
into '\33' (escape) followed by 4 hex digits.  A simple lex rule then
decodes for the benefit of yacc.

This encodes:

/*
 * lex can't cope with character sets wider than 8 bits, so convert
 * s to runes and encode non-ascii runes as <esc><hex><hex><hex><hex>.
 * result is malloced.
 */
char *
utf2lex(char *s)
{
	int nb, bytes;
	Rune r;
	char *news, *p, *ds;

	/* pass 1: count bytes needed by the converted string; watch for UTF */
	for (p = s, nb = 0; *p != '\0'; p += bytes, nb++) {
		bytes = chartorune(&r, p);
		if (bytes > 1)
			nb += 4;
	}
	news = malloc(nb+1);
	if (news != 0) {
		/* pass 2: convert s into new string */
		news[nb] = '\0';
		for (p = s, ds = news; *p != '\0'; p += bytes) {
			bytes = chartorune(&r, p);
			if (bytes == 1)
				*ds++ = r;
			else
				ds += sprint(ds, "\33%.4ux", (int)r);
		}
	}
	return news;
}

and this lex code decodes:

%{
char *lex2rune(Rune *rp, char *s);
char *estrdup(char *);

static Rune inrune;
%}
E	\33
%%
{E}....			{
			yylval.charp = estrdup(lex2rune(&inrune, yytext+1));
			return inrune;
			}
%%
char *
lex2rune(Rune *rp, char *s)
{
	static char utf[UTFmax+1];

	*rp = strtoul(s, 0, 16);
	utf[runetochar(utf, rp)] = '\0';
	return utf;
}

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Karljurgen Feuerherm]

[-- Attachment #1: Type: text/plain, Size: 2048 bytes --]

Thanks, Geoff, and Erik.
 
However... (with my 5 minute intro to Runes courtesy of Hello World
doc...) we're still talking BMP, right?
 
(I programmed in B back in the day... i.e. 1980-ish and due to a career
shift have been out of things for a while, so forgive my potential
obtuseness as I gradually reintegrate...!)
 
This reminds me of what I read here: http://www.w3.org/2005/03/23-lex-U

 
K
 
Karljürgen G. Feuerherm, PhD
Department of Archaeology and Classical Studies
Wilfrid Laurier University
75 University Avenue West
Waterloo, Ontario N2L 3C5
Tel. (519) 884-1970 x3193
Fax (519) 883-0991 (ATTN Arch. & Classics)

>>> <geoff@plan9.bell-labs.com> 28/01/2010 3:46:27 pm >>>
I've extended old code using lex to accept utf by massaging the input
stream, before lex sees it, to parse utf and encode non-ascii Runes
into '\33' (escape) followed by 4 hex digits. A simple lex rule then
decodes for the benefit of yacc.

This encodes:

/*
* lex can't cope with character sets wider than 8 bits, so convert
* s to runes and encode non-ascii runes as <esc><hex><hex><hex><hex>.
* result is malloced.
*/
char *
utf2lex(char *s)
{
int nb, bytes;
Rune r;
char *news, *p, *ds;

/* pass 1: count bytes needed by the converted string; watch for UTF
*/
for (p = s, nb = 0; *p != '\0'; p += bytes, nb++) {
bytes = chartorune(&r, p);
if (bytes > 1)
nb += 4;
}
news = malloc(nb+1);
if (news != 0) {
/* pass 2: convert s into new string */
news[nb] = '\0';
for (p = s, ds = news; *p != '\0'; p += bytes) {
bytes = chartorune(&r, p);
if (bytes == 1)
*ds++ = r;
else
ds += sprint(ds, "\33%.4ux", (int)r);
}
}
return news;
}

and this lex code decodes:

%{
char *lex2rune(Rune *rp, char *s);
char *estrdup(char *);

static Rune inrune;
%}
E\33
%%
{E}....{
yylval.charp = estrdup(lex2rune(&inrune, yytext+1));
return inrune;
}
%%
char *
lex2rune(Rune *rp, char *s)
{
static char utf[UTFmax+1];

*rp = strtoul(s, 0, 16);
utf[runetochar(utf, rp)] = '\0';
return utf;
}




[-- Attachment #2: HTML --]
[-- Type: text/html, Size: 2677 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[geoff]

Yes, we only support the 16-bit runes of Unicode plane 0.  That really
should be enough space, except for bungling by the Unicode Consortium.

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Karljurgen Feuerherm]

[-- Attachment #1: Type: text/plain, Size: 653 bytes --]

Well.... having worked with the Unicode Consortium, I know there's a little more to it than that... :)
 
But it's ok. If I have to write a preprocessor to make it work on Plan9, I might as well stay with the Unix system currently available to me and write a preprocessor for that.
 
I suppose I could use the PUA and write a postprocessor....
 
Anyhow, thanks for the info. Now, at least, I know more clearly what my options are.
 
K

>>> <geoff@plan9.bell-labs.com> 28/01/2010 4:20:41 pm >>>
Yes, we only support the 16-bit runes of Unicode plane 0. That really
should be enough space, except for bungling by the Unicode Consortium.




[-- Attachment #2: HTML --]
[-- Type: text/html, Size: 1027 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[ron minnich]

On Thu, Jan 28, 2010 at 1:51 PM, Karljurgen Feuerherm <kfeuerherm@wlu.ca> wrote:
> Well.... having worked with the Unicode Consortium, I know there's a little
> more to it than that... :)


I'm curious because I don't know much about all this stuff, I'm just
grateful I can live in the low 7 bits ... what more is there?

thanks

ron

Re: [9fans] Lex, Yacc, Unicode Plane 1

[hiro]

extinct languages, mahjong tiles,...

On Thu, Jan 28, 2010 at 11:07 PM, ron minnich <rminnich@gmail.com> wrote:
> On Thu, Jan 28, 2010 at 1:51 PM, Karljurgen Feuerherm <kfeuerherm@wlu.ca> wrote:
>> Well.... having worked with the Unicode Consortium, I know there's a little
>> more to it than that... :)
>
>
> I'm curious because I don't know much about all this stuff, I'm just
> grateful I can live in the low 7 bits ... what more is there?
>
> thanks
>
> ron
>
>

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Karljurgen Feuerherm]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

There's a lot more than that! See http://www.unicode.org/charts/ 
 
K
 
>>> hiro <23hiro@googlemail.com> 28/01/2010 5:19:58 pm >>>
extinct languages, mahjong tiles,...

On Thu, Jan 28, 2010 at 11:07 PM, ron minnich < rminnich@gmail.com > wrote:
> On Thu, Jan 28, 2010 at 1:51 PM, Karljurgen Feuerherm < kfeuerherm@wlu.ca > wrote:
>> Well.... having worked with the Unicode Consortium, I know there's a little
>> more to it than that... :)
>
>
> I'm curious because I don't know much about all this stuff, I'm just
> grateful I can live in the low 7 bits ... what more is there?
>
> thanks
>
> ron
>
>



[-- Attachment #2: HTML --]
[-- Type: text/html, Size: 1162 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

On Thu Jan 28 17:09:45 EST 2010, rminnich@gmail.com wrote:
> On Thu, Jan 28, 2010 at 1:51 PM, Karljurgen Feuerherm <kfeuerherm@wlu.ca> wrote:
> > Well.... having worked with the Unicode Consortium, I know there's a little
> > more to it than that... :)
>
>
> I'm curious because I don't know much about all this stuff, I'm just
> grateful I can live in the low 7 bits ... what more is there? ☺

there, fixed that for ya!

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Federico G. Benavento]

as erik mentioned earlier, plan9port has 32bit runes...

On Thu, Jan 28, 2010 at 8:56 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> On Thu Jan 28 17:09:45 EST 2010, rminnich@gmail.com wrote:
>> On Thu, Jan 28, 2010 at 1:51 PM, Karljurgen Feuerherm <kfeuerherm@wlu.ca> wrote:
>> > Well.... having worked with the Unicode Consortium, I know there's a little
>> > more to it than that... :)
>>
>>
>> I'm curious because I don't know much about all this stuff, I'm just
>> grateful I can live in the low 7 bits ... what more is there? ☺
>
> there, fixed that for ya!
>
> - erik
>
>



-- 
Federico G. Benavento

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

On Thu Jan 28 16:22:58 EST 2010, geoff@plan9.bell-labs.com wrote:
> Yes, we only support the 16-bit runes of Unicode plane 0.  That really
> should be enough space, except for bungling by the Unicode Consortium.

good point.

at this point only ~21829 codepoints are assigned, depending
on your definition of assigned.  and i agree that the unicode
consortium has taken a number decisions that make life difficult
(unnecessary combiners and font-encodings for math characters
are my pet peeves) .

but now that the decision has been made, i think it makes sense
to adapt, or at least put ourselves in the position to adapt.
such a principled stance doesn't help someone who needs
codepoints outside the basic plane.

otherwise we become the 64000 characters ought to
be enough for everyone guys.

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Karljurgen Feuerherm]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

thanks, erik. that's exactly right--we may or may not agree with Unicode decisions, but the fact is some of poor slobs are stuck in the higher planes whether we like it or not, so are looking for help...
 
erik, should we pursue the plan9port question offlist? it is going to take me time to integrate info, no need to bog the list down in what is likely to be simple for most folk...
 
(apologies if i'm not catching on as quickly as one might like...)
 
K

>>> erik quanstrom <quanstro@quanstro.net> 28/01/2010 6:42:45 pm >>>
On Thu Jan 28 16:22:58 EST 2010, geoff@plan9.bell-labs.com wrote:
> Yes, we only support the 16-bit runes of Unicode plane 0. That really
> should be enough space, except for bungling by the Unicode Consortium.

good point.

at this point only ~21829 codepoints are assigned, depending
on your definition of assigned. and i agree that the unicode
consortium has taken a number decisions that make life difficult
(unnecessary combiners and font-encodings for math characters
are my pet peeves) .

but now that the decision has been made, i think it makes sense
to adapt, or at least put ourselves in the position to adapt.
such a principled stance doesn't help someone who needs
codepoints outside the basic plane.

otherwise we become the 64000 characters ought to
be enough for everyone guys.

- erik



[-- Attachment #2: HTML --]
[-- Type: text/html, Size: 1812 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Rob Pike]

On Fri, Jan 29, 2010 at 8:20 AM,  <geoff@plan9.bell-labs.com> wrote:
> Yes, we only support the 16-bit runes of Unicode plane 0.

Really?  They're 32 bits in plan9port and, although there are a few
things that need to be patched, we know what they are.

Rune should be 32 bits by now.

-rob

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

> Really?  They're 32 bits in plan9port and, although there are a few
> things that need to be patched, we know what they are.
>
> Rune should be 32 bits by now.

there is a patch for plan 9.  actually 2 that enable one to set
UTFmax = 4 and Runemax = 0x10ffff:

/n/sources/patch/saved/runesize /n/sources/patch/saved/runesize2

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Russ Cox]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Specifically:

http://code.swtch.com/plan9port/changeset/3095
http://code.swtch.com/plan9port/changeset/3102
http://code.swtch.com/plan9port/changeset/3103
http://code.swtch.com/plan9port/changeset/3104
http://code.swtch.com/plan9port/changeset/3110
http://code.swtch.com/plan9port/changeset/3121

That should cover the bulk of the Plan 9 libraries
and commands but omits the kernel.

Russ

[-- Attachment #2: Type: text/html, Size: 915 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

> Specifically:
>
> http://code.swtch.com/plan9port/changeset/3095
> http://code.swtch.com/plan9port/changeset/3102
> http://code.swtch.com/plan9port/changeset/3103
> http://code.swtch.com/plan9port/changeset/3104
> http://code.swtch.com/plan9port/changeset/3110
> http://code.swtch.com/plan9port/changeset/3121
>
> That should cover the bulk of the Plan 9 libraries
> and commands but omits the kernel.

plan 9 port rejects fonts with characters at codepoints
outside the basic plane.

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Russ Cox]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

On Thu, Jan 28, 2010 at 4:42 PM, erik quanstrom <quanstro@quanstro.net>wrote:

> > Specifically:
> >
> > http://code.swtch.com/plan9port/changeset/3095
> > http://code.swtch.com/plan9port/changeset/3102
> > http://code.swtch.com/plan9port/changeset/3103
> > http://code.swtch.com/plan9port/changeset/3104
> > http://code.swtch.com/plan9port/changeset/3110
> > http://code.swtch.com/plan9port/changeset/3121
> >
> > That should cover the bulk of the Plan 9 libraries
> > and commands but omits the kernel.
>
> plan 9 port rejects fonts with characters at codepoints
> outside the basic plane.
>

http://code.swtch.com/plan9port/changeset/3140

Russ

[-- Attachment #2: Type: text/html, Size: 1567 bytes --]

Re: [9fans] Lex, Yacc, Unicode Plane 1

[erik quanstrom]

if you're doing this in plan 9, bootstrapping the compiler
is a bit of a pain.  this could save some hassle:
/n/sources/contrib/quanstro/8c-32bitrune

these are the patches it turns out were missing
/n/sources/patch/cc-32bitrune
/n/sources/patch/sed-32bitrune
/n/sources/patch/ed-32bitrune
/n/sources/patch/libdraw-32bitrune
/n/sources/patch/sambufsz

there doesn't appear to be a convention for entering 32-bit runes
in p9p yet, as in compose + X + hhhh.  i propose by silly extension
compose + Y + hhhhhh.

i've got my system working to the point where i can type
compose + Y01d510 with the clarisr font and get a
fraktur m on the screen.  nothing like a useless demo.

one thing i really love about plan 9 is the ability to make
big changes like this without having as step 1: boil the
oceans.

- erik

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Justin Jackson]

>one thing i really love about plan 9 is the ability to make
>big changes like this without having as step 1: boil the
>oceans.

That's the quote of the week. Love it.

Re: [9fans] Lex, Yacc, Unicode Plane 1

[Ethan Grammatikidis]

On 29 Jan 2010, at 6:18 am, Justin Jackson wrote:

>> one thing i really love about plan 9 is the ability to make
>> big changes like this without having as step 1: boil the
>> oceans.
>
> That's the quote of the week. Love it.
>

I keep saying I like namespaces and file interfaces, but really this
is _the_ reason I use Plan 9. :D


--
http://xkcd.com/676/

Ethan Grammatikidis
eekee57@fastmail.fm