From: Pierre Weis <Pierre.Weis@inria.fr>
To: opons@aenegada.inria.fr (Olivier Pons)
Cc: caml-list@inria.fr
Subject: Re: another question on lexer function
Date: Fri, 8 Nov 1996 14:21:55 +0100 (MET) [thread overview]
Message-ID: <199611081321.OAA28852@pauillac.inria.fr> (raw)
In-Reply-To: <199611071007.LAA25922@aenegada.inria.fr> from Olivier Pons at "Nov 7, 96 11:07:04 am"
> is it forbidden to use the ";;" token in a make_lexer function ?
You cannot use an arbitrary sequence of characters as a declared
keyword: the sequences you write must be recognized as an identifier
by the ``next_token'' function inside make_lexer, (or be a single
non-alphanumeric character). The next_token function recognizes two
kinds of identifiers: regular ones (roughly speaking sequences of
alphanumeric characters), and special ones or symbols (sequences of
non alphanumeric characters, such as ++). Once an identifier is
recognized, it is compared with the list of declared keywords: if
found in the list, a Kwd token is emitted, otherwise an Ident token is
returned.
Since the sequence ;; is recognized as two single characters by
next_token (same treatment as for parens, brackets or commas). But
the rule for single non-alphanumeric characters is that they must have
been declared as keywords, or it is a lexical error. Since ; has not
been declared as a keyword of the lexer, an error occurs.
Now, if you want to deal with a token ``;;'', you may:
1) Declare ";" as a keyword, then interpret to successive `;' tokens as a
";;" in your grammar rule.
2) Adapt the genlex module to your specific needs. I strongly
recommend this solution, in particular if you want to understand
stream parsing, or if you will use the lexer in an intensive way.
Best regards,
Pierre Weis
INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis
prev parent reply other threads:[~1996-11-08 13:51 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
1996-11-07 10:07 Olivier Pons
1996-11-08 13:21 ` Pierre Weis [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=199611081321.OAA28852@pauillac.inria.fr \
--to=pierre.weis@inria.fr \
--cc=caml-list@inria.fr \
--cc=opons@aenegada.inria.fr \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).