lexer library

lexer library

[skaller]

I am interested in building a dynamically extensible lexer.. can
anyone recommend any libraries or tools that would help?

I could use Dypgen (extensible GLR parser) for this, but it is 
probably too slow.

A system using combinators (rather than strings) would be preferred,
and Unicode support would be nice too. Ideally the lexer automaton
could be saved with Marshal, in which case the time to build the
automaton isn't important.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] lexer library

[Jon Harrop]

On Saturday 18 August 2007 19:39:52 skaller wrote:
> I am interested in building a dynamically extensible lexer.. can
> anyone recommend any libraries or tools that would help?
>
> I could use Dypgen (extensible GLR parser) for this, but it is
> probably too slow.
>
> A system using combinators (rather than strings) would be preferred,
> and Unicode support would be nice too. Ideally the lexer automaton
> could be saved with Marshal, in which case the time to build the
> automaton isn't important.

The new camlp4 looks like it supports this but I'm damned if I can figure out 
how you use it. There is a beautiful tutorial on the Wiki that fizzles out 
just when it gets exciting.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] lexer library

[Nicolas Pouillard]

Excerpts from Jon Harrop's message of Sun Aug 19 00:34:44 +0200 2007:
> On Saturday 18 August 2007 19:39:52 skaller wrote:
> > I am interested in building a dynamically extensible lexer.. can
> > anyone recommend any libraries or tools that would help?
> >
> > I could use Dypgen (extensible GLR parser) for this, but it is
> > probably too slow.
> >
> > A system using combinators (rather than strings) would be preferred,
> > and Unicode support would be nice too. Ideally the lexer automaton
> > could be saved with Marshal, in which case the time to build the
> > automaton isn't important.
> 
> The new camlp4 looks like it supports this but I'm damned if I can figure out 
> how you use it. There is a beautiful tutorial on the Wiki that fizzles out 
> just when it gets exciting.
> 

Not  really,  with  the new camlp4 you can change the lexer and the token type
(thanks  to  the  system  module).  One can also dynamically add lexer filters
that  transforms  the  token stream. However the provided default lexer is not
extensible (it's an ocamllex one).

-- 
Nicolas Pouillard aka Ertai

Re: [Caml-list] lexer library

[Joel Reymont]

On Aug 19, 2007, at 11:13 AM, Nicolas Pouillard wrote:

> Not  really,  with  the new camlp4 you can change the lexer and the  
> token type (thanks  to  the  system  module).  One can also  
> dynamically add lexer filters that  transforms  the  token stream.

Are there any examples of this?

	Thanks, Joel

--
http://wagerlabs.com