Re: LMTX MkIV difference in expansion

[Hans Hagen <j.hagen@xs4all.nl>]

On 11/19/2020 9:41 PM, Wolfgang Schuster wrote:
> Rik Kabel schrieb am 19.11.2020 um 21:20:
>> Another LMTX/MkIV difference, this time with expansion:
>>
>>     \define\Align{yes}
>>     \starttext
>>     \startalignment[\Align]
>>          This works with MkIV but fails with LMTX, complaining: {\tt
>>          tex error on line 3 in file G:/expand.mkvi: The file ended
>>          when scanning an argument.}
>>     \blank
>>         It works in both when \tex{def} or \tex{defineexpandable} is
>>         used instead of \tex{define}.
>>     \blank
>>         What changed?
>>     \stopalignment
>>     \stoptext
>>
>> It may well be that I have been abusing some laxity in MkIV and that 
>> LMTX is a bit stricter in what it accepts, but I would like to know if 
>> this is an expected difference.
> 
> You have to wait for Hans to get an answer but here is a minimal example.
> 
> \starttext
> 
> \protected\def\testparameter{test}
> %\def\testparameter{test}
> 
> \def\test[#1]%
>    {\expandafter\let\expandafter\testargumentlist\csname#1\endcsname}
> 
> \test[\testparameter]
> 
> \stoptext
Often arguments to commands like \startsomething[xx] let the xx end up 
in some \(if)csname expansion. A protected (\unexpanded in context 
speak) macro doesn't expand inside for instance an \edef (or comparable 
expandable situation). Now, from that it makes perfect sense to also not 
let it expand inside a \csname or \ifcsname. One reason is that when it 
does expand, you can get a pretty wild (nested) sequence of nested 
expansions and one can be pretty sure that we then don't have a proper 
csname. This is why in luatex we have a catch for running wild csname 
checking.

The original \ifcsname test was inherited from etex. The \protected 
feature also comes from etex. But \csname is a tex natural.
In pdftex (and luatex) a protected macro inside an \(if)csname does 
expand which to makes no sense and smells like a bug. Or maybe it was 
tricky to catch (the implementation of protected a bit of a hack).

In luametatex protected macros are native and in the process I also 
decided to *not* expand them in a \(if)csname where I expect (as said) 
protected macros to behave like in an edef. I nice side effect is that 
running wild no longer happens (but we still catch it) which can save 
quite some useless backup token list construction (needed because tex 
has to push back stuff in order to be able to report an error).

So, when you still don't understand it (which I can understand) I'm sure 
Wolfgang can explain it better now.

\starttext

           \def\foo{foo}
\protected\def\oof{oof}

\csname foo\endcsname
\csname oof\endcsname
\csname \foo\endcsname

% error in luametatex, ok in pdftex/luatex:

% \csname \oof\endcsname

\ifcsname  foo\endcsname yes\else nop\fi
\ifcsname  oof\endcsname yes\else nop\fi
\ifcsname \foo\endcsname yes\else nop\fi

% nop in luametatex (error intercepted), yes in pdftex/luatex

\ifcsname \oof\endcsname yes\else nop\fi

\stoptext

Now, one can argue that if I consider it a but in the other engines, why 
I don't argue that it should be solved. Well, there is too much legacy 
code already that might use it as feature so it will not change. But in 
luametatex we can 'fix' these things. (We also use the csname in a 
rather predictable way in context so i don't expect issues in the core.)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________