[9fans] Replacements for lex

[9fans] Replacements for lex

[Joel Salomon]

Has anyone had success with lexer-generators other than lex under Plan
9?  I'm taking a compilers class this semester and I'd like to do as
much work under Plan 9 as possible.

I looked around online and found re2c, which looked interesting except
it's written in C++.

For languages sufficiently like C, is regexp(2) suitable?

--Joel

Re: [9fans] Replacements for lex

[Dan Cross]

On Fri, Jan 19, 2007 at 01:58:06PM -0500, Joel Salomon wrote:
> Has anyone had success with lexer-generators other than lex under Plan
> 9?  I'm taking a compilers class this semester and I'd like to do as
> much work under Plan 9 as possible.
>
> I looked around online and found re2c, which looked interesting except
> it's written in C++.
>
> For languages sufficiently like C, is regexp(2) suitable?

I think that most people roll their own lexical analyzers under Plan 9.
That's typically not too hard to do, though.

	- Dan C.

Re: [9fans] Replacements for lex

[Joel Salomon]

> I think that most people roll their own lexical analyzers under Plan 9.
> That's typically not too hard to do, though.

That's likely what I'll do for the final project (probably yet another
C complier, but I might try my hand with [a subset of] D), but the
professor has said he'll want us to learn to use
mechanically-generated scanners and compare the results.

I expect the lexer in /sys/src/cmd/cc/ will be a good example.

--Joel

Re: [9fans] Replacements for lex

[C H Forsyth]

>I expect the lexer in /sys/src/cmd/cc/ will be a good example.

it's a little more elaborate than some (not gcc!) because it takes on limited C preprocessing
duties as well.  you might also observe that the author enjoys his gotos, but since it is essentially
a finite state machine that seems fair enough, and furthermore he's allowed to take advantage of his
`licence to GOTO' awarded after a killer early paper. (Since that was about automata, it should probably
be an Earley paper, except that it's slightly earlier than that.)  anyway, he meant no harm.

Re: [9fans] Replacements for lex

[erik quanstrom]

i think the ken c lexer is a very good example.  it shows how
goto, well applied, can be a very useful tool for an ad hoc problem.
it also shows that in practice lexers and parsers are seldom as
separate as they appear in textbooks.

on the other hand, rc has a very hairy lexer. hoc has a small, elegant one.
i believe the _practice of programming_ goes into detail about it.

- erik

On Fri Jan 19 15:37:42 EST 2007, forsyth@vitanuova.com wrote:
> >I expect the lexer in /sys/src/cmd/cc/ will be a good example.
>
> it's a little more elaborate than some (not gcc!) because it takes on limited C preprocessing
> duties as well.  you might also observe that the author enjoys his gotos, but since it is essentially
> a finite state machine that seems fair enough, and furthermore he's allowed to take advantage of his
> `licence to GOTO' awarded after a killer early paper. (Since that was about automata, it should probably
> be an Earley paper, except that it's slightly earlier than that.)  anyway, he meant no harm.

Re: [9fans] Replacements for lex

[Taj Khattra]

> you might also observe that the author enjoys his gotos

If you want to go somewhere, goto is the best way to get there. - ken

:)

Re: [9fans] Replacements for lex

[Brantley Coile]

>> you might also observe that the author enjoys his gotos
>
> If you want to go somewhere, goto is the best way to get there. - ken
>
> :)

Goto Statements Considered Harmful
		-- Niklaus Wirth

(Dijkstra wrote the letter, but Wirth gave it its title.)

There's a really good scanner in Wirth's compiler book.

http://www.oberon.ethz.ch/WirthPubl/CBEAll.pdf

Re: [9fans] Replacements for lex

[William Josephson]

On Fri, Jan 19, 2007 at 02:26:47PM -0500, Joel Salomon wrote:
> >I think that most people roll their own lexical analyzers under Plan 9.
> >That's typically not too hard to do, though.
>
> That's likely what I'll do for the final project (probably yet another
> C complier, but I might try my hand with [a subset of] D), but the

You'll learn a lot more if you don't try anything like C
let alone C++ or D.  For a semester-length project, you'll
learn far more if you work on a much smaller language --
ideally domain specific, in my opinion -- and really see it
through in detail.

Re: [9fans] Replacements for lex

[Bruce Ellis]

I taught a course in advanced compiler construction where the
students were required to write a lisp interpreter in C and then
write a rudimentry C compiler in their lisp.

The "bits you don't have to do" included the c lex function which
was presented to them in lisp as a (c:token) function.  I've written
a lot of lex-ers and it was just the same code again (in c) which
stopped the students from spending all of their time trying to do
the same in lisp.  This enabled them to concentrate on the more
interesting tasks (I also supplied the lisp alloc and gc code).

It was a fun course and I hope things were learnt.

For most stuff something like lex is fine.  Have a try at getting
<integer/float constant> reg-exp useable and I'll type in a lex-er
faster than you can debug the reg-exps (a chore i decided was
a waste of time for the course).

brucee

On 1/20/07, William Josephson <jkw@eecs.harvard.edu> wrote:
> On Fri, Jan 19, 2007 at 02:26:47PM -0500, Joel Salomon wrote:
> > >I think that most people roll their own lexical analyzers under Plan 9.
> > >That's typically not too hard to do, though.
> >
> > That's likely what I'll do for the final project (probably yet another
> > C complier, but I might try my hand with [a subset of] D), but the
>
> You'll learn a lot more if you don't try anything like C
> let alone C++ or D.  For a semester-length project, you'll
> learn far more if you work on a much smaller language --
> ideally domain specific, in my opinion -- and really see it
> through in detail.
>

Re: [9fans] Replacements for lex

[Brantley Coile]

My suggestion is just to write scanners by hand.  I've written dozens
and they take only a few minutes.  While a half hour seems long
compared to the speed of an automaticly generated scanner, the time
difference over the total project is quite small.  And besides.  You
want to learn how to write compilers.


> Has anyone had success with lexer-generators other than lex under Plan
> 9?  I'm taking a compilers class this semester and I'd like to do as
> much work under Plan 9 as possible.
>
> I looked around online and found re2c, which looked interesting except
> it's written in C++.
>
> For languages sufficiently like C, is regexp(2) suitable?
>
> --Joel

Re: [9fans] Replacements for lex

[erik quanstrom]

take a look at the lexing in the c compiler -- /sys/src/cmd/cc/lex.c.

- erik

Re: [9fans] Replacements for lex

[Federico Benavento]

yeah, great example :)

On 1/19/07, erik quanstrom <quanstro@coraid.com> wrote:
> take a look at the lexing in the c compiler -- /sys/src/cmd/cc/lex.c.
>
> - erik
>


--
Federico G. Benavento

Re: [9fans] Replacements for lex

[William Josephson]

On Fri, Jan 19, 2007 at 01:58:06PM -0500, Joel Salomon wrote:
> For languages sufficiently like C, is regexp(2) suitable?

For something that simple, you might as well just hand write it,
particularly if you're going to have to support somethig typedef-like.