\scantokens in luametatex

\scantokens in luametatex

[Marcel Fabian Krüger]

Hi,

in the current luametatex upload, \scantokens (and also \scantextokens)
behave odd: They seem to act like \detokenize, except that spaces get
catcode other instead of catcode space:

\starttext
\edef\abc{\scantokens{\relax}}
\abc
\edef\abc{\scantextokens{\relax}}
\abc
\stoptext

used to result in an empty output file but with luametatex, it  writes \relax \relax to the output.

\starttext
\def\showsecond#1{\show}
\edef\abc{\scantokens{a b}}
\expandafter\showsecond\abc
\edef\abc{\detokenize{a b}}
\expandafter\showsecond\abc
\stoptext

prints

> the character  
> blank space  

Best regard,
Marcel Krüger
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: \scantokens in luametatex

[Hans Hagen]

On 8/13/2020 2:31 PM, Marcel Fabian Krüger wrote:

> in the current luametatex upload, \scantokens (and also \scantextokens)
> behave odd: They seem to act like \detokenize, except that spaces get
> catcode other instead of catcode space:
> 
> \starttext
> \edef\abc{\scantokens{\relax}}
> \abc
> \edef\abc{\scantextokens{\relax}}
> \abc
> \stoptext
> 
> used to result in an empty output file but with luametatex, it  writes \relax \relax to the output.
i'll look at it (i probably forget to set the catcodetable)

here is the generalized case:

     \def\foo{!}

     \edef\abc{\tokenized catcodetable \catcodetable {a\relax \foo}} \abc
     \edef\abc{\tokenized catcodetable -1            {a\relax \foo}} \abc

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: \scantokens in luametatex

[Marcel Fabian Krüger]

Thank you for fixing this!

On Thu, Aug 13, 2020 at 02:31:51PM +0200, Marcel Fabian Krüger wrote:
> Hi,
> 
> in the current luametatex upload, \scantokens (and also \scantextokens)
> behave odd: They seem to act like \detokenize, except that spaces get
> catcode other instead of catcode space:
> 
> \starttext
> \edef\abc{\scantokens{\relax}}
> \abc
> \edef\abc{\scantextokens{\relax}}
> \abc
> \stoptext
> 
> used to result in an empty output file but with luametatex, it  writes \relax \relax to the output.
> 
> \starttext
> \def\showsecond#1{\show}
> \edef\abc{\scantokens{a b}}
> \expandafter\showsecond\abc
> \edef\abc{\detokenize{a b}}
> \expandafter\showsecond\abc
> \stoptext
> 
> prints
> 
> > the character  
> > blank space  
> 
> Best regard,
> Marcel Krüger
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________