ocamlyacc/lex reentrancy

[skaller <skaller@maxtal.com.au>]

At present, ocamllex cannot be used reentrantly by the client,
since there is nowhere to put auxilliary data.

The obvious place to attach this data is the lexbuf;
(say by using get and set functions):
this would be transparent (allow existing lexers to work
without source code modification). Also, no changes to the
lexer would be required (only the lexbuf).

At present, ocamlyacc parsers accept a lexer and a lexbuf.
In principle, this is the wrong interface: the lexer and
parser should be decoupled; the parser should only
require a function which acts as a token source.

However, there is also no place for auxilliary data 
to be stored while parsing. If the lexbuf is extended to
provide access to client data, the current interface would
support supplying that data to client code processing
non-terminals on reduction.

This suggests that each start symbol should generate two
parsers: one using the old interface, and a new interface
passing a function:

	'a -> token

The old interface can call the new interface, after fetching
the client data from the lexbuf.

These changes together would seem to allow all existing
ocamllex/ocamlyacc sources to continue to work unmodified,
while allowing new codes for both ocamllex and ocamlyacc
to be written which are reentrant, and also allow alternate
sources of tokens to be used with ocamlyacc.

One minor problem: the type 'a could not be infered for the lexbuf,
if client code never used it? Is there a good way to modify
ocamlyacc/lex, so existing code works, but which also
supports supplying auxilliary data, so that both
the lexer and parser can be reentered?

-- 
John Skaller, mailto:skaller@maxtal.com.au
10/1 Toxteth Rd Glebe NSW 2037 Australia
homepage: http://www.maxtal.com.au/~skaller
voice: 61-2-9660-0850