[Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[沈胜宇]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

Dear all:

I am using ocamllex to write a lexer, the language I am analyzing have "include" mechanism that refer to another file.

So how can I open the refered file and switch to it in ocamllex?

Shen

[-- Attachment #2: Type: text/html, Size: 223 bytes --]

RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[Soegtrop, Michael]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

Dear Shen,

the technique I typically use is to define a preprocessor, which sits in between the lexer and the parser. It provides the same interface to the parser as a lexer and takes your lexer at the other end. Your lexer also includes preprocessor tokens. The preprocessor usually just forwards tokens from the lexer to the parser, but it intercepts preprocessor tokens and then creates e.g. a new instance of the lexer to read from an include file or plays back token from a define. When the preprocessor receives an EOF token from an include file, it closes the include lexer and switches back to the main lexer. So typically the preprocessor maintains a stack of lexers. The advantage compared to messing around inside of the lexer is that it is clearer and more easily reused.

Best regards,

Michael


From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On Behalf Of ???
Sent: Wednesday, November 25, 2015 3:04 PM
To: caml-list
Subject: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

Dear all:

I am using ocamllex to write a lexer, the language I am analyzing have "include" mechanism that refer to another file.

So how can I open the refered file and switch to it in ocamllex?

Shen
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 5266 bytes --]

RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[Soegtrop, Michael]

[-- Attachment #1: Type: text/plain, Size: 3258 bytes --]

Desr Shen,

I have a working parser for a subset of analog Verilog in Menhir. I haven’t worked with ocamlyacc but compared to C yacc or bison, Menhir is quite user friendly, e.g. the capability to explain conflicts. This gave me sufficient insight to slightly change the productions (not the grammar!) such that it is acceptable to Menhir. Unfortunately I cannot share my parser, but if you name the specific production rules you have issues with, I might be able to help.

Best regards,

Michael


From: 沈胜宇 [mailto:syshen@nudt.edu.cn]
Sent: Wednesday, December 09, 2015 2:41 AM
To: Soegtrop, Michael
Cc: caml-list
Subject: Re: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?


I have resolved it, and I have a further question:

I am developing a parser with ocamllex and ocamyacc for verilog language.

But I find the BNF include some rules that beyond the LA(1), so how to result this ?

Shen


-----原始邮件-----
发件人: "Soegtrop, Michael" <michael.soegtrop@intel.com<mailto:michael.soegtrop@intel.com>>
发送时间: 2015-11-25 22:24:57 (星期三)
收件人: ??? <syshen@nudt.edu.cn<mailto:syshen@nudt.edu.cn>>, caml-list <caml-list@inria.fr<mailto:caml-list@inria.fr>>
抄送:
主题: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?
Dear Shen,

the technique I typically use is to define a preprocessor, which sits in between the lexer and the parser. It provides the same interface to the parser as a lexer and takes your lexer at the other end. Your lexer also includes preprocessor tokens. The preprocessor usually just forwards tokens from the lexer to the parser, but it intercepts preprocessor tokens and then creates e.g. a new instance of the lexer to read from an include file or plays back token from a define. When the preprocessor receives an EOF token from an include file, it closes the include lexer and switches back to the main lexer. So typically the preprocessor maintains a stack of lexers. The advantage compared to messing around inside of the lexer is that it is clearer and more easily reused.

Best regards,

Michael


From: caml-list-request@inria.fr<mailto:caml-list-request@inria.fr> [mailto:caml-list-request@inria.fr<mailto:caml-list-request@inria.fr>] On Behalf Of ???
Sent: Wednesday, November 25, 2015 3:04 PM
To: caml-list
Subject: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

Dear all:

I am using ocamllex to write a lexer, the language I am analyzing have "include" mechanism that refer to another file.

So how can I open the refered file and switch to it in ocamllex?

Shen

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de<http://www.intel.de>
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 10814 bytes --]

RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[Soegtrop, Michael]

[-- Attachment #1: Type: text/plain, Size: 6562 bytes --]

Dear Shen,

I guess we first need to discuss your BNF syntax. As far as I understand { x } means 0..n repeats of x and [ x ] means 0 or 1 copy of x. But what does [ [ constant_expression ] ] or { [ expression ] } mean? Also not sure what you mean by […].

Verilog A is a bit different here (no constant_expression in identifiers):

identifier ::=
IDENTIFIER [ { . IDENTIFIER } ]
NOTE: The period in identifier may not be preceded or followed by a space.

Also, as you can see, Verilog A syntax specifies that identifiers may not contain spaces, so you either need to handle spaces in the parser or hierarchical identifiers in the lexer. I did the latter, since handling spaces in the parser is no fun. As a result I don’t have any conflicts around this. Does your Verilog variant have the same restriction regarding spaces and identifiers? If so, I would check if you can move the whole hierarchical_identifier to the lexer level. If you have to move space handling to the parser, your syntax will get very verbose.

There are typically ways to make parser generators accept grammars for common computer languages (maybe except for Fortran). But if a grammar is translated 1:1 from BNF to a parser generator you usually get some issues. There are many ways to express the same grammar with different BNF and some are more suitable for parser generators than others. So usually you need to restructure the BNF a bit. As I mentioned, I found the “explain” function of Menhir quite useful to understand what the issue is.

Best regards,

Michael


From: 沈胜宇 [mailto:syshen@nudt.edu.cn]
Sent: Wednesday, December 09, 2015 10:07 AM
To: Soegtrop, Michael
Cc: caml-list
Subject: Re: RE: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

Dear Michael

 one of the problem is with :

hierarchical_identifier ::= { identifier [ [ constant_expression ] ] . } identifier

you can notice that hierarchical_identifier end without [...],
but some times hierarchical_identifier can followed by [...], like the following rule:

primary ::=
number
| hierarchical_identifier [ { [ expression ] } [ range_expression ] ]
..

In such case, the parser will run ahead to consume all the following [...], can complain about no identifier can be found in the first rule.

while in other case, the hierarchical_identifier do not followed by [...]

So does this means that I need something stronger that LA(1) that can look further ahead, but I think such thing may not exist becasue of complexity.




-----原始邮件-----
发件人: "Soegtrop, Michael" <michael.soegtrop@intel.com<mailto:michael.soegtrop@intel.com>>
发送时间: 2015-12-09 16:42:33 (星期三)
收件人: "syshen@nudt.edu.cn<mailto:syshen@nudt.edu.cn>" <syshen@nudt.edu.cn<mailto:syshen@nudt.edu.cn>>
抄送: caml-list <caml-list@inria.fr<mailto:caml-list@inria.fr>>
主题: RE: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?
Desr Shen,

I have a working parser for a subset of analog Verilog in Menhir. I haven’t worked with ocamlyacc but compared to C yacc or bison, Menhir is quite user friendly, e.g. the capability to explain conflicts. This gave me sufficient insight to slightly change the productions (not the grammar!) such that it is acceptable to Menhir. Unfortunately I cannot share my parser, but if you name the specific production rules you have issues with, I might be able to help.

Best regards,

Michael


From: 沈胜宇 [mailto:syshen@nudt.edu.cn<mailto:syshen@nudt.edu.cn>]
Sent: Wednesday, December 09, 2015 2:41 AM
To: Soegtrop, Michael
Cc: caml-list
Subject: Re: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?


I have resolved it, and I have a further question:

I am developing a parser with ocamllex and ocamyacc for verilog language.

But I find the BNF include some rules that beyond the LA(1), so how to result this ?

Shen

-----原始邮件-----
发件人: "Soegtrop, Michael" <michael.soegtrop@intel.com<mailto:michael.soegtrop@intel.com>>
发送时间: 2015-11-25 22:24:57 (星期三)
收件人: ??? <syshen@nudt.edu.cn<mailto:syshen@nudt.edu.cn>>, caml-list <caml-list@inria.fr<mailto:caml-list@inria.fr>>
抄送:
主题: RE: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?
Dear Shen,

the technique I typically use is to define a preprocessor, which sits in between the lexer and the parser. It provides the same interface to the parser as a lexer and takes your lexer at the other end. Your lexer also includes preprocessor tokens. The preprocessor usually just forwards tokens from the lexer to the parser, but it intercepts preprocessor tokens and then creates e.g. a new instance of the lexer to read from an include file or plays back token from a define. When the preprocessor receives an EOF token from an include file, it closes the include lexer and switches back to the main lexer. So typically the preprocessor maintains a stack of lexers. The advantage compared to messing around inside of the lexer is that it is clearer and more easily reused.

Best regards,

Michael


From: caml-list-request@inria.fr<mailto:caml-list-request@inria.fr> [mailto:caml-list-request@inria.fr<mailto:caml-list-request@inria.fr>] On Behalf Of ???
Sent: Wednesday, November 25, 2015 3:04 PM
To: caml-list
Subject: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

Dear all:

I am using ocamllex to write a lexer, the language I am analyzing have "include" mechanism that refer to another file.

So how can I open the refered file and switch to it in ocamllex?

Shen

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de<http://www.intel.de>
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de<http://www.intel.de>
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 19022 bytes --]

Re: [Caml-list] [OCAML]:: how to switch buffer in ocamllex?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

Am Mittwoch, den 25.11.2015, 22:04 +0800 schrieb 沈胜宇:
> Dear all:
> 
> I am using ocamllex to write a lexer, the language I am analyzing have
> "include" mechanism that refer to another file.
> 
> So how can I open the refered file and switch to it in ocamllex?

Just open it with Lexing.from_file, and recursively call the lexer from
itself.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]