[NTG-context] Re: Quickly invoke a self-defined index sorting file?

[Hans Hagen <j.hagen@xs4all.nl>]

On 1/12/2025 9:58 AM, autumnus wrote:
> hi,
> 
> I defined an index file for sorting Chinese. (a bit large with almost 4MB)
> 
> https://github.com/Soanguy/ConTeXt-Chinese-Example/blob/master/sorting/sort-alpha.lua
> 
> But I have to pre-reference this file with input every time to enable it.
> How do I fuse these files with my local context system and use setup to enable it directly.
> 
> I want directly use
> %%%
> \setupregister[index][n=4,language=cn-alpha,]
> %%%
> 
> instead of
> %%%%
> \input sort-alpha.lua
> \setupregister[index][
>    n=4,
>    language={cn-alpha},]
> %%%

remove \startluacode and \stopluacode in that file and do this instead:

\registerctxluafile{sort-hanzi}{}

\starttext
     test
\stoptext

currently you first load that whole file in memory tokenized (i.e. 1 
byte becomes 8 bytes) which is fine (and fast) for reasonable size files 
but in your case it has to bump token memory

also, you don't really need the huge entries table because you're not 
going to split the index for every first character

maybe this

definitions["cn-hanzi"].entries = table.setmetatableindex(function(t,k)
     if utfbyte(k) < 1000 then
         return "latin"
     else
         return "chinese"
     end
end)

print(definitions["cn-hanzi"].entries['a'])
print(definitions["cn-hanzi"].entries['咗'])

but even then ... korean and japanese don't have that either so 
basically you only need the order (is that order defined somewhere in 
unicode?

> In addition to this, I found that the following notification appeared on the tex terminal,
> probably because there are too many characters in the index file (tens of thousands of characters).
> Can I avoid this notification?
> 
> tex memory      > bumping category 'token' succeeded, details: all=16000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=2000000 | min=2000000 | ptr=1999080 | set=10000000 | stp=1000000 | top=2000000
> tex memory      > bumping category 'token' succeeded, details: all=24000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=3000000 | min=2000000 | ptr=2999080 | set=10000000 | stp=1000000 | top=3000000
> tex memory      > bumping category 'token' succeeded, details: all=32000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=4000000 | min=2000000 | ptr=3999080 | set=10000000 | stp=1000000 | top=4000000
> tex memory      > bumping category 'token' succeeded, details: all=40000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=5000000 | min=2000000 | ptr=4999080 | set=10000000 | stp=1000000 | top=5000000
> tex memory      > bumping category 'token' succeeded, details: all=48000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=6000000 | min=2000000 | ptr=5999080 | set=10000000 | stp=1000000 | top=6000000
> tex memory      > bumping category 'token' succeeded, details: all=56000000 | ext=0 | ini=568315 | itm=8 | max=10000000 | mem=7000000 | min=2000000 | ptr=6999080 | set=10000000 | stp=1000000 | top=7000000
> 
> autumnus
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
> webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
> archive  : https://github.com/contextgarden/context
> wiki     : https://wiki.contextgarden.net
> ___________________________________________________________________________________


-- 

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________