another error message in LMTX

another error message in LMTX

[Pablo Rodriguez]

Hans,

using latest from yesterday, I get the following error:

 tex error       > tex error on line 22 in file cont-yes.mkiv: Valid
 keyword expected, likely 'minus'

 <lua output> mt
     xrunjit
 <to be read again>
     \relax
 <macro> \buff_verbatim_type_buffer_indeed ...rrentregime
 }tab{\typingparameter \c!tab }method{\p_buff_option
 }escape{\typingparameter \c!escape }nature{display}\relax
     \dostoptagged \endofverbatimlines \dostoptagged \stoppacked
     \typingparameter \c!after

 [...]

 You started a keyword but it seems to be an invalid one. The first
 character(s) might give you a clue. You might want to quit unwanted
 lookahead with \relax.

This comes from a book with > 300pp. It compiles perfectly fine with
MkIV, so the typo might be in LMTX itself.

I wonder whether it would be possible to add the invalid keyword (and
related command) to the error message. This would mean something like:

 tex error       > tex error on line 22 in file cont-yes.mkiv: Valid
 keyword expected, got "invalid one" (related to "command").

This would help a lot to figure out where to start searching for clues
for a MWE.

Many thanks for your help,

Pablo
--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: another error message in LMTX

[Hans Hagen]

On 11/7/2020 12:41 PM, Pablo Rodriguez wrote:
> Hans,
> 
> using latest from yesterday, I get the following error:
> 
>   tex error       > tex error on line 22 in file cont-yes.mkiv: Valid
>   keyword expected, likely 'minus'
> 
>   <lua output> mt
>       xrunjit
>   <to be read again>
>       \relax
>   <macro> \buff_verbatim_type_buffer_indeed ...rrentregime
>   }tab{\typingparameter \c!tab }method{\p_buff_option
>   }escape{\typingparameter \c!escape }nature{display}\relax
>       \dostoptagged \endofverbatimlines \dostoptagged \stoppacked
>       \typingparameter \c!after
> 
>   [...]
> 
>   You started a keyword but it seems to be an invalid one. The first
>   character(s) might give you a clue. You might want to quit unwanted
>   lookahead with \relax.
> 
> This comes from a book with > 300pp. It compiles perfectly fine with
> MkIV, so the typo might be in LMTX itself.
> 
> I wonder whether it would be possible to add the invalid keyword (and
> related command) to the error message. This would mean something like:
> 
>   tex error       > tex error on line 22 in file cont-yes.mkiv: Valid
>   keyword expected, got "invalid one" (related to "command").
> 
> This would help a lot to figure out where to start searching for clues
> for a MWE.
well, the clue is there: you have

you have a \hskip or \vskip someplace that is not properly terminated by 
\relax so tex keeps scanning for optional plus|minus keywords and in 
your case runs into "mtxrunjit"

in luametatex we have a slightly different keyword scanner

- in traditional tex it will just push back the 'non valid' keyword and 
continue

- but that actually obscures a potentential error when the \hskip is 
burried deep: maybe you are typesetting "minus" as is and then it *is* a 
valid keyword and probably an error wil be that no valid dimension is seen

- so, you'rve just been lucky so far

so, this new keyword scanner is more picky in the sense that when it 
notices a potential keywors start it will complain earlier which might 
actually catch potential errors ealier (in this case all follow up words 
that start with a 'p' or 'm'

another change in scanning is that in many cases order no longer 
matters; for instance in scanning skips it's a 'plus' spec followed by a 
'minus' spec but in rules it can be mized 'width', 'height', 'depths'
  in any order and repetitive

in luametatex (unless there is good reason) the order doesn't matter, 
and often repetition is ok (so, multiple plus and minus in a skip and in 
any order where the last one counts)

this also permits for instance an order like:

     \dontleavehmode
     \hbox to 20pt xoffset  3cm{1\hss}
     \hbox to 20pt             {\hss2\hss}
     \hbox to 20pt xoffset -3cm{\hss3}

where previously the 'to' spec always came last and multiple 'to' are 
okay (offsets etc are a lmtx feature, and it means many more keywords 
have to be scanned, which was one reason for cooking up a better 
mechanism, even if right from the start that was already rather optimized).

now, in principle this brings the possibility to support partial 
keywords, so 'xo' 'xoff' etc (in fact i only need to change a few lines 
of code) which can be handy when for instance one passes many 
attributes) but first i want the main machinery to be stable

actually, this was already there for a while; the good news is that all 
this doesn't harm performance (we actually gain some here, not that it's 
normally noticeable)

so: more flexibility, better performance, earlier interception but of 
course the later might demand users to fix code (i actually had to fix 
context in one spot for this lookahead issue).

(I'll add \relaxes to the after handling but that just obscures the issue.)

Hans

----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: another error message in LMTX

[Pablo Rodriguez]

On 11/7/20 1:29 PM, Hans Hagen wrote:
> [...]

Many thanks for your in-depth explanation, Hans.

This sample triggers the error:

    \appendtoks
       \unexpanded\def\doverbatimspace{\char32\hskip\zeropoint}%
    \to \everyenableelements

    \setuptagging[state=start]

    \starttext
    \starttyping
    mtxrunjit --generate
    \stoptyping

    \startitemize[n]
    \startitem a\stopitem
    \stopitemize
    \stoptext

I have no idea why.

Many thanks for your help,

Pablo


> well, the clue is there: you have
>
> you have a \hskip or \vskip someplace that is not properly terminated by
> \relax so tex keeps scanning for optional plus|minus keywords and in
> your case runs into "mtxrunjit"
>
> in luametatex we have a slightly different keyword scanner
>
> - in traditional tex it will just push back the 'non valid' keyword and
> continue
>
> - but that actually obscures a potentential error when the \hskip is
> burried deep: maybe you are typesetting "minus" as is and then it *is* a
> valid keyword and probably an error wil be that no valid dimension is seen
>
> - so, you'rve just been lucky so far
>
> so, this new keyword scanner is more picky in the sense that when it
> notices a potential keywors start it will complain earlier which might
> actually catch potential errors ealier (in this case all follow up words
> that start with a 'p' or 'm'
>
> another change in scanning is that in many cases order no longer
> matters; for instance in scanning skips it's a 'plus' spec followed by a
> 'minus' spec but in rules it can be mized 'width', 'height', 'depths'
>   in any order and repetitive
>
> in luametatex (unless there is good reason) the order doesn't matter,
> and often repetition is ok (so, multiple plus and minus in a skip and in
> any order where the last one counts)
>
> this also permits for instance an order like:
>
>      \dontleavehmode
>      \hbox to 20pt xoffset  3cm{1\hss}
>      \hbox to 20pt             {\hss2\hss}
>      \hbox to 20pt xoffset -3cm{\hss3}
>
> where previously the 'to' spec always came last and multiple 'to' are
> okay (offsets etc are a lmtx feature, and it means many more keywords
> have to be scanned, which was one reason for cooking up a better
> mechanism, even if right from the start that was already rather optimized).
>
> now, in principle this brings the possibility to support partial
> keywords, so 'xo' 'xoff' etc (in fact i only need to change a few lines
> of code) which can be handy when for instance one passes many
> attributes) but first i want the main machinery to be stable
>
> actually, this was already there for a while; the good news is that all
> this doesn't harm performance (we actually gain some here, not that it's
> normally noticeable)
>
> so: more flexibility, better performance, earlier interception but of
> course the later might demand users to fix code (i actually had to fix
> context in one spot for this lookahead issue).
>
> (I'll add \relaxes to the after handling but that just obscures the issue.)


--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: another error message in LMTX

[Hans Hagen]

On 11/7/2020 3:31 PM, Pablo Rodriguez wrote:
> On 11/7/20 1:29 PM, Hans Hagen wrote:
>> [...]
> 
> Many thanks for your in-depth explanation, Hans.
> 
> This sample triggers the error:
> 
>      \appendtoks
>         \unexpanded\def\doverbatimspace{\char32\hskip\zeropoint}%
>      \to \everyenableelements
> 
>      \setuptagging[state=start]
> 
>      \starttext
>      \starttyping
>      mtxrunjit --generate
>      \stoptyping
> 
>      \startitemize[n]
>      \startitem a\stopitem
>      \stopitemize
>      \stoptext
> 
> I have no idea why.

because \doverbatimspace is defined in a way that will look ahead

    \unexpanded\def\doverbatimspace{\char32\hskip\zeropoint\relax}

so

\starttyping
     plus 10pt  --generate
\stoptyping

probably makes the "plus 10pt" disappear


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________