Dear Shen,

 

the technique I typically use is to define a preprocessor, which sits in between the lexer and the parser. It provides the same interface to the parser as a lexer and takes your lexer at the other end. Your lexer also includes preprocessor tokens. The preprocessor usually just forwards tokens from the lexer to the parser, but it intercepts preprocessor tokens and then creates e.g. a new instance of the lexer to read from an include file or plays back token from a define. When the preprocessor receives an EOF token from an include file, it closes the include lexer and switches back to the main lexer. So typically the preprocessor maintains a stack of lexers. The advantage compared to messing around inside of the lexer is that it is clearer and more easily reused.

 

Best regards,

 

Michael