Changed behavior of ^^ in csname

Changed behavior of ^^ in csname

[Marcel Fabian Krüger]

Hi,

I noticed that in the latest LMTX version, ^^ (with ^ of catcode 7)
is no longer interpreted inside csnames. Is this intentional?

(Example:

\starttext
\catcode\circumflexasciicode=\superscriptcatcode% Ensure correct catcodes

\show ^^41 % This still works and shows "> the letter A"

\show^^41 % This used to be an error "Undefined control sequence \showA",
          % now show "> The letter A"

\show\^^41 % Used to show "> \A=undefined", now shows "> \^=macro:->\enco_handle_accent {^}"
           % and "! Missing $ inserted"
\stoptext

)

Best regards,
Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Hans Hagen]

On 6/11/2020 4:06 AM, Marcel Fabian Krüger wrote:
> Hi,
> 
> I noticed that in the latest LMTX version, ^^ (with ^ of catcode 7)
> is no longer interpreted inside csnames. Is this intentional?
> 
> (Example:
> 
> \starttext
> \catcode\circumflexasciicode=\superscriptcatcode% Ensure correct catcodes
> 
> \show ^^41 % This still works and shows "> the letter A"
> 
> \show^^41 % This used to be an error "Undefined control sequence \showA",
>            % now show "> The letter A"
> 
> \show\^^41 % Used to show "> \A=undefined", now shows "> \^=macro:->\enco_handle_accent {^}"
>             % and "! Missing $ inserted"
> \stoptext
> 
> )
Indeed that is intentional because in lmtx we disable them (partially). 
You have to say:

\normalsupmarkmode\zerocount % we use 1 as default in lmtx

This is because in lmtx we have native prescript support, as in:

   $ x^{rt}_{rb}^^{lt}__{lb} $

so, double _ ^ can have a different meaning there.

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Marcel Fabian Krüger]

On Thu, Jun 11, 2020 at 10:07:34AM +0200, Hans Hagen wrote:
> On 6/11/2020 4:06 AM, Marcel Fabian Krüger wrote:
> > Hi,
> > 
> > I noticed that in the latest LMTX version, ^^ (with ^ of catcode 7)
> > is no longer interpreted inside csnames. Is this intentional?
> > 
> > (Example:
> > 
> > \starttext
> > \catcode\circumflexasciicode=\superscriptcatcode% Ensure correct catcodes
> > 
> > \show ^^41 % This still works and shows "> the letter A"
> > 
> > \show^^41 % This used to be an error "Undefined control sequence \showA",
> >            % now show "> The letter A"
> > 
> > \show\^^41 % Used to show "> \A=undefined", now shows "> \^=macro:->\enco_handle_accent {^}"
> >             % and "! Missing $ inserted"
> > \stoptext
> > 
> > )
> Indeed that is intentional because in lmtx we disable them (partially). You
> have to say:
> 
> \normalsupmarkmode\zerocount % we use 1 as default in lmtx

Hi,

shouldn't \supmarkmode=1 only disable these in math mode?
In any case, \normalsupmarkmode\zerocount doesn't make a difference
here:

\starttext
\normalsupmarkmode\zerocount
\catcode\circumflexasciicode=\superscriptcatcode% Ensure correct catcodes

\show ^^41 % This still works and shows "> the letter A"

\show^^41 % This used to be an error "Undefined control sequence \showA",
           % now show "> The letter A"

\show\^^41 % Used to show "> \A=undefined", now shows "> \^=macro:->\enco_handle_accent {^}"
            % and "! Missing $ inserted"
\stoptext

shows exactly the same behavior.


Best regards,
Marcel

> 
> This is because in lmtx we have native prescript support, as in:
> 
>   $ x^{rt}_{rb}^^{lt}__{lb} $
> 
> so, double _ ^ can have a different meaning there.
> 
> Hans
> 
> 
> -----------------------------------------------------------------
>                                           Hans Hagen | PRAGMA ADE
>               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Hans Hagen]

On 6/11/2020 12:24 PM, Marcel Fabian Krüger wrote:

> shows exactly the same behavior.
Looks like i don't catch the case where a cs is constructed, so i'll add 
some checking there too (it is tempting to remove all that ^^ stuff).

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Marcel Fabian Krüger]

On Thu, Jun 11, 2020 at 07:51:56PM +0200, Hans Hagen wrote:
> On 6/11/2020 12:24 PM, Marcel Fabian Krüger wrote:
> 
> > shows exactly the same behavior.
> Looks like i don't catch the case where a cs is constructed, so i'll add
> some checking there too (it is tempting to remove all that ^^ stuff).

Thanks. I think "the ^^ stuff" is actually quite useful, especially when
debugging international documents. Especially when the glyphs come from
a language you are not used to, it is quite nice to just have a hex
representation without needing pecial editors and ^^ is the only way to get
that without introducing actual changes to behavior.

I had another issue with the latest luametatex:

The following document:

\starttext
\def\mymacro#1 bc{}
{\mymacro a b bc}
\stoptext

used to work and lead to no output. But in the new luametatex version,
it complains about "! Argument of \mymacro has an extra }" as if it
doesn't see the " bc" argument delimiter.

Marcel

> 
> Hans
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Hans Hagen]

On 6/11/2020 10:47 PM, Marcel Fabian Krüger wrote:

> The following document:
> 
> \starttext
> \def\mymacro#1 bc{}
> {\mymacro a b bc}
> \stoptext
> 
> used to work and lead to no output. But in the new luametatex version,
> it complains about "! Argument of \mymacro has an extra }" as if it
> doesn't see the " bc" argument delimiter.
ah, i see, i forget to update a variable

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Changed behavior of ^^ in csname

[Marcel Fabian Krüger]

Hi,

thank you for fixing this in the latest version.
I found another problem with ^^, this time when followed by hexadecimal
digits:

\starttext
\normalsupmarkmode\zerocount
\catcode\circumflexasciicode=\superscriptcatcode% Ensure correct catcodes

\let\ß\relax
\show\ß % Shows "> \ß=\relax", no surprise here
\show\^^df % Should be the same, but actually shows "> \� %=undefined"
\show ß % Shows "> the letter ß"
\show ^^df % Should be the same, but actually shows "> the character "
           % and "luatex warning  > print: bad string offset"
\showthe\numexpr`\ß % Shows "> 223"
\showthe\numexpr`\^^df % Gives an error "! Improper alphabetic constant"

\stoptext


Best regards,
Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Yet another cornercase of ^^ handling in LuaMetaTeX

[Hans Hagen]

On 6/17/2020 12:10 PM, Marcel Fabian Krüger wrote:
> On Mon, Jun 15, 2020 at 04:16:06PM +0200, Hans Hagen wrote:
>>
>> ah, i removed some doee that i though was never seen; i'll add that bit
>> again
>>
>>> Beside that, I noticed a smaller weird thing when using LuaMetaTeX in ini
>>> mode without Context: The default mathcodes for digits and letters have
>>> their class and family values switched. So e.g. "A" has a default
>>> mathcode of 119537729=0x7200041, indicating class 1 and family 7,
>>> instead of the expected 31457345=0x1E00041, indicating class 7 and
>>> family 1. Of course this isn't really a problem because the format will
>>> normally assign new mathcodes anyway, but especially class 1 seems like
>>> a *very* unnatural default for normal letters.
>> i'll check it (digits get class 0) ... indeed these defaults nowadays never
>> kick in this way
> 
> Thanks. Is there any way to convert either a token number (as given e.g.
> by t.tok) or a numeric cs identifier (as given e.g. by t.tok-cs_token_flag)
> to a token uservalue?
> In LuaTeX I used to use `token.new(tok, 0)` for this, but LuaMetaTeX
> seems to do more checking here.

Indeed there is more checking. So, when you create a token you need to 
use a valid cmd code and a valid chr code.

\ctxlua{
     local t = token.new(string.byte("X"),token.command_id("letter"))
     tex.print(t)
}

The interface might evolve a bit more. Valid commands (at this moment) are:

\ctxlua{
     for k, v in ipairs(token.values("command")) do
         print(k,v)
     end
}

So, things like this should work

\dimen3456=1234pt

\dimen3467=
\ctxlua{
     local t = token.new(3456,token.command_id("register_dimen"))
   % local t = token.new("register_dimen",3456) % in upcoming version
     tex.sprint(t)
}

\the\dimen3467

As does:

\ctxlua{
     local t1 = token.new(string.byte("{"),token.command_id("letter"))
     local t2 = token.new(string.byte("}"),token.command_id("letter"))
     tex.print(t1)
     tex.print(t2)
}

Properties like cs_token_flag are not user level and have no meaning in 
the interface. In fact, the 'token-as-number' is just some 
representation with no intended meaning. Also, tokens that are 
problematic or can interfere badly might become inaccessible in future 
versions.

(I don't expect many users to use the low level token interface anyway 
but it's there as part of the package.)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Yet another cornercase of ^^ handling in LuaMetaTeX

[Marcel Fabian Krüger]

On Wed, Jun 17, 2020 at 03:43:49PM +0200, Hans Hagen wrote:
> On 6/17/2020 12:10 PM, Marcel Fabian Krüger wrote:
> > On Mon, Jun 15, 2020 at 04:16:06PM +0200, Hans Hagen wrote:
> > > 
> > > ah, i removed some doee that i though was never seen; i'll add that bit
> > > again
> > > 
> > > > Beside that, I noticed a smaller weird thing when using LuaMetaTeX in ini
> > > > mode without Context: The default mathcodes for digits and letters have
> > > > their class and family values switched. So e.g. "A" has a default
> > > > mathcode of 119537729=0x7200041, indicating class 1 and family 7,
> > > > instead of the expected 31457345=0x1E00041, indicating class 7 and
> > > > family 1. Of course this isn't really a problem because the format will
> > > > normally assign new mathcodes anyway, but especially class 1 seems like
> > > > a *very* unnatural default for normal letters.
> > > i'll check it (digits get class 0) ... indeed these defaults nowadays never
> > > kick in this way
> > 
> > Thanks. Is there any way to convert either a token number (as given e.g.
> > by t.tok) or a numeric cs identifier (as given e.g. by t.tok-cs_token_flag)
> > to a token uservalue?
> > In LuaTeX I used to use `token.new(tok, 0)` for this, but LuaMetaTeX
> > seems to do more checking here.
> 
> Indeed there is more checking. So, when you create a token you need to use a
> valid cmd code and a valid chr code.
> 
> \ctxlua{
>     local t = token.new(string.byte("X"),token.command_id("letter"))
>     tex.print(t)
> }
> 
> The interface might evolve a bit more. Valid commands (at this moment) are:
> 
> \ctxlua{
>     for k, v in ipairs(token.values("command")) do
>         print(k,v)
>     end
> }
> 
> So, things like this should work
> 
> \dimen3456=1234pt
> 
> \dimen3467=
> \ctxlua{
>     local t = token.new(3456,token.command_id("register_dimen"))
>   % local t = token.new("register_dimen",3456) % in upcoming version
>     tex.sprint(t)
> }
> 
> \the\dimen3467
> 
> As does:
> 
> \ctxlua{
>     local t1 = token.new(string.byte("{"),token.command_id("letter"))
>     local t2 = token.new(string.byte("}"),token.command_id("letter"))
>     tex.print(t1)
>     tex.print(t2)
> }
> 
> Properties like cs_token_flag are not user level and have no meaning in the
> interface. In fact, the 'token-as-number' is just some representation with
> no intended meaning. Also, tokens that are problematic or can interfere
> badly might become inaccessible in future versions.
> 
> (I don't expect many users to use the low level token interface anyway but
> it's there as part of the package.)
> 
> Hans
> 

How does this work with mark nodes? Their mark field has a table
containing triples of command, character and "cs" value. How can the
actual csname of a controlsequence be extracted from this in LuaMetaTeX?

Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Yet another cornercase of ^^ handling in LuaMetaTeX

[Hans Hagen]

On 6/17/2020 4:24 PM, Marcel Fabian Krüger wrote:

> How does this work with mark nodes? Their mark field has a table
> containing triples of command, character and "cs" value. How can the
> actual csname of a controlsequence be extracted from this in LuaMetaTeX?
at the token ends marks are stored as token lists but not directly 
accessible by a token itself as they get resolved during a split; a 
topmarks token is a cmd/chr combination that indicates that token 
(\topmarks) which then picks up a number (i noticed that i somehow lost 
picking up the number at the tex end but i'll fix that)

i'll add a tex.getmarks("top|bottom|...",<class number>) for whatveer 
use it might have and probably also backport that to luatex then

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________