Custom lexer in Camlp4

Custom lexer in Camlp4

[Jeremy Yallop]

Is there a way to replace the default lexer in Camlp4?  I'd like to use 
(an extended version of) one of the OCaml grammars, but with a different 
lexer.

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: [Caml-list] Custom lexer in Camlp4

[Jeremy Yallop]

Nicolas Pouillard wrote:
> Excerpts from Jeremy Yallop's message of Tue Jun 24 15:03:19 +0200 2008:
>> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
>> (an extended version of) one of the OCaml grammars, but with a different 
>> lexer.
> 
> You can rebuild (I mean functor application), your own camlp4 by giving
> another Lexer module. Basically you will have to copy/paste the short
> Camlp4.PreCast module (or a part of), by substituting Camlp4.Struct.Lexer
> by your Lexer implementation. That's fairly heavy but works.

Ok, I've built a slightly modified clone of PreCast.  What's the best 
way to persuade Camlp4 to use it?  The original PreCast seems fairly 
hardwired, e.g. in the Register module.

Thanks for your help.

Jeremy.

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: [Caml-list] Custom lexer in Camlp4

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Excerpts from Jeremy Yallop's message of Tue Jun 24 15:03:19 +0200 2008:
> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
> (an extended version of) one of the OCaml grammars, but with a different 
> lexer.

You can rebuild (I mean functor application), your own camlp4 by giving
another Lexer module. Basically you will have to copy/paste the short
Camlp4.PreCast module (or a part of), by substituting Camlp4.Struct.Lexer
by your Lexer implementation. That's fairly heavy but works.

Regards,

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [Caml-list] Custom lexer in Camlp4

[Richard Jones]

On Tue, Jun 24, 2008 at 03:30:23PM +0100, Jeremy Yallop wrote:
> Ok, I've built a slightly modified clone of PreCast.  What's the best 
> way to persuade Camlp4 to use it?  The original PreCast seems fairly 
> hardwired, e.g. in the Register module.

If you get an answer to this, please post it.  I would love to be able
to add new integer literals to the language (0UL and so on).

On a related note I had a crazy brainwave that we could use camlp4 to
use 'ordinary' operators in an overloaded context.  Something like:

  INT64 (2L * n / 3L)

The INT64(expr) macro would inspect the AST of expr and change + ->
Int64.add etc.  Of course one can do this using the pa_openin macro
and a module which overrides (+) etc, but maybe this is better?  I was
going to try modifying parts of virt-df to use this syntax to see if
it would be beneficial.

After writing virt-df which uses mixed int64, int32, int and int63[*]
types I'm starting to come around to Jon's opinion that some limited
overloading or type classes or whatever would be worthwhile.

Rich.

[*] int63 is my own type: It turns into an efficient int on 64 bit
platforms, and is emulated on (now rare) 32 bit platforms.  It is
necessary for virt-df because we want ints which can comfortably hold
the size (in bytes / sectors / blocks / etc) of a block device.  You
don't really need a 64 bit int for this, but you do need something
which is bigger than 32 bits.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Custom lexer in Camlp4

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Excerpts from Jeremy Yallop's message of Tue Jun 24 16:30:23 +0200 2008:
> Nicolas Pouillard wrote:
> > Excerpts from Jeremy Yallop's message of Tue Jun 24 15:03:19 +0200 2008:
> >> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
> >> (an extended version of) one of the OCaml grammars, but with a different 
> >> lexer.
> > 
> > You can rebuild (I mean functor application), your own camlp4 by giving
> > another Lexer module. Basically you will have to copy/paste the short
> > Camlp4.PreCast module (or a part of), by substituting Camlp4.Struct.Lexer
> > by your Lexer implementation. That's fairly heavy but works.
> 
> Ok, I've built a slightly modified clone of PreCast.  What's the best 
> way to persuade Camlp4 to use it?  The original PreCast seems fairly 
> hardwired, e.g. in the Register module.

Hum, you can't you have to build your own camlp4 driver :(

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 240 bytes --]

Re: [Caml-list] Custom lexer in Camlp4

[Andrej Bauer]

On a related note: how hard would it be to convince camlp4 to translate
the Coq-like context notation

  expr % M

to the "open in" construct

 open M in expr ?

Since % is already an infix operator, this shouldn't be too hard.

Andrej

Re: [Caml-list] Custom lexer in Camlp4

[Hezekiah M. Carty]

On Tue, Jun 24, 2008 at 11:16 AM, Richard Jones <rich@annexia.org> wrote:
> On Tue, Jun 24, 2008 at 03:30:23PM +0100, Jeremy Yallop wrote:
> On a related note I had a crazy brainwave that we could use camlp4 to
> use 'ordinary' operators in an overloaded context.  Something like:
>
>  INT64 (2L * n / 3L)
>
> The INT64(expr) macro would inspect the AST of expr and change + ->
> Int64.add etc.  Of course one can do this using the pa_openin macro
> and a module which overrides (+) etc, but maybe this is better?  I was
> going to try modifying parts of virt-df to use this syntax to see if
> it would be beneficial.

I toyed around with a simple extension to the pa_float example on the
camlp4 wiki.  The modified version is here:

http://0ok.org/code/ocaml/pa_float.ml

With this, Float.(1 + 2) becomes (1.0 +. 2.0).  However, if you used
Foo.(a + b), the result would be (Foo.add a b).  In the Int64 case,
Int64.(1L + 2L) becomes (Int64.add 1L 2L).  Same for (-) -> .sub, ( *
) -> .mul, and (/) -> .div.

This is the result of some chatting on IRC and toying around with the
pa_float example, so I don't know how write or wrong this
implementation is.

Another approach I tried after further IRC discussion (thanks to
bluestorm and David Teller!) is a twist on pa_openin:

http://0ok.org/code/ocaml/pa_context.ml

In this case, Foo.(1 + 2) would use the operator Foo.(+) rather than a
pre-defined transform of (+) -> Foo.add.  This might work well if
Num-like ( +/ ) operators were added to modules, since it would allow
something along the lines of:

Int64.(1L +/ some_int64_value +/ of_int 2 -/ of_float some_float_value)

to work properly.  (+/) may not be very pretty, but it doesn't
conflict with (+) or (+.).  David Teller may be doing something like
this with the Batteries project.

I think Zheng Li released an extension similar to pa_context, but I am
not sure how similar or different it is in function.

Sadly camlp4 3.10+ does not play well with the toplevel, so these are
difficult to work with interactively.  I do use pa_context in several
places though, and it seems to work well.

Hez

-- 
Hezekiah M. Carty
Graduate Research Assistant
University of Maryland
Department of Atmospheric and Oceanic Science

Re: [Caml-list] Custom lexer in Camlp4

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Excerpts from Hezekiah M. Carty's message of Tue Jun 24 17:43:52 +0200 2008:
> On Tue, Jun 24, 2008 at 11:16 AM, Richard Jones <rich@annexia.org> wrote:
> > On Tue, Jun 24, 2008 at 03:30:23PM +0100, Jeremy Yallop wrote:
> > On a related note I had a crazy brainwave that we could use camlp4 to
> > use 'ordinary' operators in an overloaded context.  Something like:
> >
> >  INT64 (2L * n / 3L)
> >
> > The INT64(expr) macro would inspect the AST of expr and change + ->
> > Int64.add etc.  Of course one can do this using the pa_openin macro
> > and a module which overrides (+) etc, but maybe this is better?  I was
> > going to try modifying parts of virt-df to use this syntax to see if
> > it would be beneficial.
> 
> I toyed around with a simple extension to the pa_float example on the
> camlp4 wiki.  The modified version is here:

[...]

> Sadly camlp4 3.10+ does not play well with the toplevel, so these are
> difficult to work with interactively.  I do use pa_context in several
> places though, and it seems to work well.

Can you elaborate on that?

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [Caml-list] Custom lexer in Camlp4

[Hezekiah M. Carty]

I forgot to include the relevant external links in my prior email.
They are included below.

On Tue, Jun 24, 2008 at 11:43 AM, Hezekiah M. Carty
<hcarty@atmos.umd.edu> wrote:
> I toyed around with a simple extension to the pa_float example on the
> camlp4 wiki.  The modified version is here:
>
> http://0ok.org/code/ocaml/pa_float.ml

The original pa_float is on the camlp4 wiki:

http://brion.inria.fr/gallium/index.php/Pa_float

> I think Zheng Li released an extension similar to pa_context, but I am
> not sure how similar or different it is in function.

Zheng Li's extension is called pa_scope, and can be found here:

http://www.pps.jussieu.fr/~li/software/index.html#pa_scope

I included the camlp4 + toplevel bug information in a separate email.

Hez

Re: [Caml-list] Custom lexer in Camlp4

[Pietro Abate]

Hi,

a while ago I posted an example to replace the camlp4 lexer with a
custom one. Nobody replied to my post. Maybe it was just plain wrong,
but maybe it is worth a look. 

http://groups.google.com/group/fa.caml/browse_thread/thread/e51ee5ef1d837d36/0bd94b4d2e824536?lnk=gst&q=camlp4+and+lexers#

p


On Tue, Jun 24, 2008 at 02:03:19PM +0100, Jeremy Yallop wrote:
> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
> (an extended version of) one of the OCaml grammars, but with a different 
> lexer.
> 
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Custom lexer in Camlp4

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

You might want to take a look at an OSP project that is going on this summer
that is aiming to implement something rather similar to what you describe:

http://osp.janestcapital.com/files/delimited-overloading.pdf

y

On Tue, Jun 24, 2008 at 11:16 AM, Richard Jones <rich@annexia.org> wrote:

> On Tue, Jun 24, 2008 at 03:30:23PM +0100, Jeremy Yallop wrote:
> > Ok, I've built a slightly modified clone of PreCast.  What's the best
> > way to persuade Camlp4 to use it?  The original PreCast seems fairly
> > hardwired, e.g. in the Register module.
>
> If you get an answer to this, please post it.  I would love to be able
> to add new integer literals to the language (0UL and so on).
>
> On a related note I had a crazy brainwave that we could use camlp4 to
> use 'ordinary' operators in an overloaded context.  Something like:
>
>  INT64 (2L * n / 3L)
>
> The INT64(expr) macro would inspect the AST of expr and change + ->
> Int64.add etc.  Of course one can do this using the pa_openin macro
> and a module which overrides (+) etc, but maybe this is better?  I was
> going to try modifying parts of virt-df to use this syntax to see if
> it would be beneficial.
>
> After writing virt-df which uses mixed int64, int32, int and int63[*]
> types I'm starting to come around to Jon's opinion that some limited
> overloading or type classes or whatever would be worthwhile.
>
> Rich.
>
> [*] int63 is my own type: It turns into an efficient int on 64 bit
> platforms, and is emulated on (now rare) 32 bit platforms.  It is
> necessary for virt-df because we want ints which can comfortably hold
> the size (in bytes / sectors / blocks / etc) of a block device.  You
> don't really need a 64 bit int for this, but you do need something
> which is bigger than 32 bits.
>
> --
> Richard Jones
> Red Hat
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2967 bytes --]

Re: [Caml-list] Custom lexer in Camlp4

[Richard Jones]

On Tue, Jun 24, 2008 at 12:59:00PM -0400, Yaron Minsky wrote:
> You might want to take a look at an OSP project that is going on this summer
> that is aiming to implement something rather similar to what you describe:
> 
> http://osp.janestcapital.com/files/delimited-overloading.pdf

Yes, when can I use it :-)

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Custom lexer in Camlp4

[Christophe TROESTLER]

On Wed, 25 Jun 2008 09:41:31 +0100, Richard Jones wrote:
> 
> On Tue, Jun 24, 2008 at 12:59:00PM -0400, Yaron Minsky wrote:
> > You might want to take a look at an OSP project that is going on this summer
> > that is aiming to implement something rather similar to what you describe:
> > 
> > http://osp.janestcapital.com/files/delimited-overloading.pdf
> 
> Yes, when can I use it :-)

For the use you described, expect a preversion to be available in 2-3 weeks.

Cheers,
ChriS

Re: [Caml-list] Custom lexer in Camlp4

[Jeremy Yallop]

Nicolas Pouillard wrote:
> Excerpts from Jeremy Yallop's message of Tue Jun 24 16:30:23 +0200 2008:
>> Nicolas Pouillard wrote:
>>> Excerpts from Jeremy Yallop's message of Tue Jun 24 15:03:19 +0200 2008:
>>>> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
>>>> (an extended version of) one of the OCaml grammars, but with a different 
>>>> lexer.
>>> You can rebuild (I mean functor application), your own camlp4 by giving
>>> another Lexer module. Basically you will have to copy/paste the short
>>> Camlp4.PreCast module (or a part of), by substituting Camlp4.Struct.Lexer
>>> by your Lexer implementation. That's fairly heavy but works.
>> Ok, I've built a slightly modified clone of PreCast.  What's the best 
>> way to persuade Camlp4 to use it?  The original PreCast seems fairly 
>> hardwired, e.g. in the Register module.
> 
> Hum, you can't you have to build your own camlp4 driver :(

Thanks, Nicolas.  That's useful to know.

The solution I've adopted for now is to change the default quotation 
expander to a function that just slots in my lexer before calling the 
original expander.  This means that I can use my custom lexical syntax 
inside the << >> delimiters and I can use those delimiters either around 
the whole program or around a particular term.  I think this'll be fine 
for my needs, although it would be nice to be able to customize the 
lexer that's used for regular code as well.

Jeremy

Re: [Caml-list] Custom lexer in Camlp4

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]

Excerpts from Jeremy Yallop's message of Wed Jun 25 13:53:02 +0200 2008:
> Nicolas Pouillard wrote:
> > Excerpts from Jeremy Yallop's message of Tue Jun 24 16:30:23 +0200 2008:
> >> Nicolas Pouillard wrote:
> >>> Excerpts from Jeremy Yallop's message of Tue Jun 24 15:03:19 +0200 2008:
> >>>> Is there a way to replace the default lexer in Camlp4?  I'd like to use 
> >>>> (an extended version of) one of the OCaml grammars, but with a different 
> >>>> lexer.
> >>> You can rebuild (I mean functor application), your own camlp4 by giving
> >>> another Lexer module. Basically you will have to copy/paste the short
> >>> Camlp4.PreCast module (or a part of), by substituting Camlp4.Struct.Lexer
> >>> by your Lexer implementation. That's fairly heavy but works.
> >> Ok, I've built a slightly modified clone of PreCast.  What's the best 
> >> way to persuade Camlp4 to use it?  The original PreCast seems fairly 
> >> hardwired, e.g. in the Register module.
> > 
> > Hum, you can't you have to build your own camlp4 driver :(
> 
> Thanks, Nicolas.  That's useful to know.
> 
> The solution I've adopted for now is to change the default quotation 
> expander to a function that just slots in my lexer before calling the 
> original expander.  This means that I can use my custom lexical syntax 
> inside the << >> delimiters and I can use those delimiters either around 
> the whole program or around a particular term.  I think this'll be fine 
> for my needs, although it would be nice to be able to customize the 
> lexer that's used for regular code as well.

You've  followed  a  reasonable  approach,  where one use quotations to embed a
"foreign"  language  in  the  sense that it even doesn't share the same lexical
conventions.

However your right that changing the lexer should be doable.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]