Re: Yet another cornercase of ^^ handling in LuaMetaTeX

[Hans Hagen <j.hagen@xs4all.nl>]

On 6/17/2020 12:10 PM, Marcel Fabian Krüger wrote:
> On Mon, Jun 15, 2020 at 04:16:06PM +0200, Hans Hagen wrote:
>>
>> ah, i removed some doee that i though was never seen; i'll add that bit
>> again
>>
>>> Beside that, I noticed a smaller weird thing when using LuaMetaTeX in ini
>>> mode without Context: The default mathcodes for digits and letters have
>>> their class and family values switched. So e.g. "A" has a default
>>> mathcode of 119537729=0x7200041, indicating class 1 and family 7,
>>> instead of the expected 31457345=0x1E00041, indicating class 7 and
>>> family 1. Of course this isn't really a problem because the format will
>>> normally assign new mathcodes anyway, but especially class 1 seems like
>>> a *very* unnatural default for normal letters.
>> i'll check it (digits get class 0) ... indeed these defaults nowadays never
>> kick in this way
> 
> Thanks. Is there any way to convert either a token number (as given e.g.
> by t.tok) or a numeric cs identifier (as given e.g. by t.tok-cs_token_flag)
> to a token uservalue?
> In LuaTeX I used to use `token.new(tok, 0)` for this, but LuaMetaTeX
> seems to do more checking here.

Indeed there is more checking. So, when you create a token you need to 
use a valid cmd code and a valid chr code.

\ctxlua{
     local t = token.new(string.byte("X"),token.command_id("letter"))
     tex.print(t)
}

The interface might evolve a bit more. Valid commands (at this moment) are:

\ctxlua{
     for k, v in ipairs(token.values("command")) do
         print(k,v)
     end
}

So, things like this should work

\dimen3456=1234pt

\dimen3467=
\ctxlua{
     local t = token.new(3456,token.command_id("register_dimen"))
   % local t = token.new("register_dimen",3456) % in upcoming version
     tex.sprint(t)
}

\the\dimen3467

As does:

\ctxlua{
     local t1 = token.new(string.byte("{"),token.command_id("letter"))
     local t2 = token.new(string.byte("}"),token.command_id("letter"))
     tex.print(t1)
     tex.print(t2)
}

Properties like cs_token_flag are not user level and have no meaning in 
the interface. In fact, the 'token-as-number' is just some 
representation with no intended meaning. Also, tokens that are 
problematic or can interfere badly might become inaccessible in future 
versions.

(I don't expect many users to use the low level token interface anyway 
but it's there as part of the package.)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________