czech sorting in MiIV

czech sorting in MiIV

[Tomáš Pustelník]

[-- Attachment #1.1: Type: text/plain, Size: 992 bytes --]

Hi,
is czech sorting in MkIV working? When I tried it with this code:

\mainlanguage[cz]
\enableregime [utf-8]

\starttext
\setupregister[index][language=cz]

\index{chobot}chobot 
\index{áro}áro 
\index{akát}akát 
\index{Áčko}Áčko 
\index{astma}astma 
\index{ámerika}ámerika 
\index{arábie}arábie 
\index{Árun}Árun 
\index{žába}žába 
\index{cibule}cibule 
\index{čára}čára 
\index{bomba}bomba 
\index{Auto}Auto
\index{arek}arek

\placeindex
\stoptext

ConTeXt sorted only letters without accent and rest of the words just put before index without any kind of sorting, also accented characters inside words are sorted wrong way. Is it just me or the sorting is really broken? Is there some text how to do sorting in MkIV so I can fix it, or I just need to follow rest of the sort-lan.lua to write sorting rules?

Also in MkII what commands \exportsortreduction and \exportsortshortcut do? I couldn't figured it out from source codes.

Thanks 

Tomas Pustelnik

[-- Attachment #1.2: Type: text/html, Size: 2239 bytes --]

[-- Attachment #2: Type: text/plain, Size: 486 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Hans Hagen]

On 15-5-2010 2:49, Tomáš Pustelník wrote:
> Hi,
> is czech sorting in MkIV working? When I tried it with this code:
>
> \mainlanguage[cz]
> \enableregime [utf-8]
>
> \starttext
> \setupregister[index][language=cz]
>
> \index{chobot}chobot
> \index{áro}áro
> \index{akát}akát
> \index{Áčko}Áčko
> \index{astma}astma
> \index{ámerika}ámerika
> \index{arábie}arábie
> \index{Árun}Árun
> \index{žába}žába
> \index{cibule}cibule
> \index{čára}čára
> \index{bomba}bomba
> \index{Auto}Auto
> \index{arek}arek
>
> \placeindex
> \stoptext
>
> ConTeXt sorted only letters without accent and rest of the words just put before index without any kind of sorting, also accented characters inside words are sorted wrong way. Is it just me or the sorting is really broken? Is there some text how to do sorting in MkIV so I can fix it, or I just need to follow rest of the sort-lan.lua to write sorting rules?

sort-lan.lua ... what is the sort order you expect?

> Also in MkII what commands \exportsortreduction and \exportsortshortcut do? I couldn't figured it out from source codes.

they are methods for influencing the sort order (they write some info to 
the tui/tuio file that is used later on)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Tomáš Pustelník]

--------------------------------------------------
From: "Hans Hagen" <pragma@wxs.nl>
Sent: Saturday, May 15, 2010 7:26 PM
To: "Tomáš Pustelník" <xPustelnikT@seznam.cz>
Subject: Re: [NTG-context] czech sorting in MiIV

> On 15-5-2010 7:10, Tomáš Pustelník wrote:
>> I'm using version 2010.05.13 12:15 so if there isn't some special way
>> how to download beta I should be using lates version. Also in MkIV I set
>> sorting with this command?
>
> normally the garden is a few hours behind the website
>
> you can fetch the beta from the website and unzip it in texmf-context and 
> then run luatools --generate
>
>> \enabletrackers[sorters.tests]
>
> just tracking
>
Thanks, it's working now, but still it sort some words bit different 
compared to MkII. But it's ok for now, I will check czech norm and try to 
fix it if needed.

Also I have question. In my bachelor thesis I'm describe language support 
for ConTeXt and when I tried add some experimental language into ConTeXt (I 
define system constants and add \loadcorefile{lang-exp} to context.mkii and 
context.mkiv) and when generating format for MkII everything was ok but for 
MkIV ConTeXt complained he can't find file lang-exp.tex (I paste it in base 
folder to other source codes). Any idea whats wrong? In MkIV must I change 
something else as well?

Tomas P. 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Hans Hagen]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

On 15-5-2010 8:14, Tomáš Pustelník wrote:

> Thanks, it's working now, but still it sort some words bit different
> compared to MkII. But it's ok for now, I will check czech norm and try
> to fix it if needed.

we can support several norms if needed

> Also I have question. In my bachelor thesis I'm describe language
> support for ConTeXt and when I tried add some experimental language into
> ConTeXt (I define system constants and add \loadcorefile{lang-exp} to
> context.mkii and context.mkiv) and when generating format for MkII
> everything was ok but for MkIV ConTeXt complained he can't find file
> lang-exp.tex (I paste it in base folder to other source codes). Any idea
> whats wrong? In MkIV must I change something else as well?

Ah .. you look in the wrong spot. There's a file:

   mult-def.lua

and interfaces are defined there. I generate the mult-* files using

   mtxrun interface --context

Now, if you just want to experiment, you need to patch mult-def.lua.

You can make a mult-def-mine.lua:

   local t = require "mult-def.lua"

   t.commands.framed.xx = "frmd"

   -- print(table.serialize(t.commands.framed))

   return t

I patched mtx-interface.lua so that you can say;

   mtxrun interface --context mult-def-mine.lua

There is also --messages for the message files.

Of course you have to make a cont-xx.tex file as well.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------

[-- Attachment #2: mtx-interface.lua --]
[-- Type: text/x-lua, Size: 11812 bytes --]

if not modules then modules = { } end modules ['mtx-cache'] = {
    version   = 1.001,
    comment   = "companion to mtxrun.lua",
    author    = "Hans Hagen, PRAGMA-ADE, Hasselt NL",
    copyright = "PRAGMA ADE / ConTeXt Development Team",
    license   = "see context related readme files"
}

local format = string.format

scripts           = scripts           or { }
scripts.interface = scripts.interface or { }

local flushers          = { }
local userinterfaces    = { 'en','cs','de','it','nl','ro','fr','pe' }
local messageinterfaces = { 'en','cs','de','it','nl','ro','fr','pe','no' }

function flushers.scite(interface,collection)
    local result, i = {}, 0
    result[#result+1] = format("keywordclass.macros.context.%s=",interface)
    for _, command in ipairs(collection) do
        if i==0 then
            result[#result+1] = "\\\n"
            i = 5
        else
            i = i - 1
        end
        result[#result+1] = format("%s ",command)
    end
    io.savedata(format("cont-%s-scite.properties",interface), table.concat(result),"\n")
    io.savedata(format("cont-%s-scite.lua",interface), table.serialize(collection,true))
end

function flushers.jedit(interface,collection)
    local result = {}
    result[#result+1] = "<?xml version='1.0'?>"
    result[#result+1] = "<!DOCTYPE MODE SYSTEM 'xmode.dtd'>\n"
    result[#result+1] = "<MODE>"
    result[#result+1] = "\t<RULES>"
    result[#result+1] = "\t\t<KEYWORDS>"
    for _, command in ipairs(collection) do
        result[#result+1] = format("\t\t\t<KEYWORD2>%s</KEYWORD2>",command)
    end
    result[#result+1] = "\t\t</KEYWORDS>"
    result[#result+1] = "\t</RULES>"
    result[#result+1] = "</MODE>"
    io.savedata(format("context-jedit-%s.xml",interface), table.concat(result),"\n")
end

function flushers.bbedit(interface,collection)
    local result = {}
    result[#result+1] = "<?xml version='1.0'?>"
    result[#result+1] = "<key>BBLMKeywordList</key>"
    result[#result+1] = "<array>"
    for _, command in ipairs(collection) do
        result[#result+1]  = format("\t<string>\\%s</string>",command)
    end
    result[#result+1] = "</array>"
    io.savedata(format("context-bbedit-%s.xml",interface), table.concat(result),"\n")
end

function flushers.raw(interface,collection)
    for _, command in ipairs(collection) do
        logs.simple(command)
    end
end

function scripts.interface.editor(editor)
    local interfaces= environment.files
    if #interfaces == 0 then
        interfaces= userinterfaces
    end
    local xmlfile = resolvers.find_file("cont-en.xml") or ""
    if xmlfile == "" then
        logs.simple("unable to locate cont-en.xml")
    end
    for _, interface in ipairs(interfaces) do
        local keyfile = resolvers.find_file(format("keys-%s.xml",interface)) or ""
        if keyfile == "" then
            logs.simple("unable to locate keys-*.xml")
        else
            local collection = { }
            local mappings   = { }
            local x = xml.load(keyfile)
            for e, d, k in xml.elements(x,"cd:command") do
                local at = d[k].at
                local name, value = at.name, at.value
                if name and value then
                    mappings[name] = value
                end
            end
            local x = xml.load(xmlfile)
            for e, d, k in xml.elements(x,"cd:command") do
                local at = d[k].at
                local name, type = at.name, at["type"]
                if name and name ~= "" then
                    local remapped = mappings[name] or name
                    if type == "environment" then
                        collection[#collection+1] = "start" .. remapped
                        collection[#collection+1] = "stop" .. remapped
                    else
                        collection[#collection+1] = remapped
                    end
                end
            end
            if #collection > 0 then
                table.sort(collection)
                flushers[editor](interface,collection)
            end
        end
    end
end

function scripts.interface.check()
    local xmlfile = resolvers.find_file("cont-en.xml") or ""
    if xmlfile ~= "" then
        local f = io.open("cont-en-check.tex","w")
        if f then
            f:write("\\starttext\n")
            local x = xml.load(xmlfile)
            for e, d, k in xml.elements(x,"cd:command") do
                local dk = d[k]
                local at = dk.at
                if at then
                    local name = xml.filter(dk,"cd:sequence/cd:string/attribute(value)")
                    if name and name ~= "" then
                        if at.type == "environment" then
                            name = "start" .. name
                        end
                        f:write(format("\\doifundefined{%s}{\\writestatus{check}{command '%s' is undefined}}\n",name,name))
                    end
                end
            end
            f:write("\\stoptext\n")
            f:close()
        end
    end
end

function scripts.interface.context()
    local filename = resolvers.find_file(environment.files[1] or "mult-def.lua") or ""
    if filename ~= "" then
        local interface = dofile(filename)
        if interface and next(interface) then
            local variables, constants, commands, elements = interface.variables, interface.constants, interface.commands, interface.elements
            local filename = resolvers.find_file("cont-en.xml") or ""
            local xmldata = filename ~= "" and (io.loaddata(filename) or "")
            local function flush(texresult,xmlresult,language,what,tag)
                local t = interface[what]
                texresult[#texresult+1] = format("%% definitions for interface %s for language %s\n%%",what,language)
                xmlresult[#xmlresult+1] = format("\t<!-- definitions for interface %s for language %s -->\n",what,language)
                xmlresult[#xmlresult+1] = format("\t<cd:%s>",what)
                for _, key in ipairs(table.sortedkeys(t)) do
                    local v = t[key]
                    local value = v[language] or v["en"]
                    if not value then
                        logs.simple(format("warning, no value for key '%s' for language '%s'",key,language))
                    else
                        local value = t[key][language] or t[key].en
                        texresult[#texresult+1] = format("\\setinterface%s{%s}{%s}",tag,key,value)
                        xmlresult[#xmlresult+1] = format("\t\t<cd:%s name='%s' value='%s'/>",tag,key,value)
                    end
                end
                xmlresult[#xmlresult+1] = format("\t</cd:%s>\n",tag)
            end
            local function replace(str, element, attribute, category, othercategory, language)
                return str:gsub(format("(<%s[^>]-%s=)([\"\'])([^\"\']-)([\"\'])",element,attribute), function(a,b,c)
                    local cc = category[c]
                    if not cc and othercategory then
                        cc = othercategory[c]
                    end
                    if cc then
                        ccl = cc[language]
                        if ccl then
                            return a .. b .. ccl .. b
                        end
                    end
                    return a .. b .. c .. b
                end)
            end
            for language, _ in pairs(commands.setuplayout) do
                local texresult, xmlresult = { }, { }
                texresult[#texresult+1] = format("%% this file is auto-generated, don't edit this file\n%%")
                xmlresult[#xmlresult+1] = format("<?xml version='1.0'?>\n",tag)
                xmlresult[#xmlresult+1] = format("<cd:interface xmlns:cd='http://www.pragma-ade.com/commands' name='context' language='%s' version='2008.10.21 19:42'>\n",language)
                flush(texresult,xmlresult,language,"variables","variable")
                flush(texresult,xmlresult,language,"constants","constant")
                flush(texresult,xmlresult,language,"elements", "element")
                flush(texresult,xmlresult,language,"commands", "command")
                texresult[#texresult+1] = format("%%\n\\endinput")
                xmlresult[#xmlresult+1] = format("</cd:interface>")
                local texfilename = format("mult-%s.tex",language)
                local xmlfilename = format("keys-%s.xml",language)
                io.savedata(texfilename,table.concat(texresult,"\n"))
                logs.simple(format("saving interface definitions '%s'",texfilename))
                io.savedata(xmlfilename,table.concat(xmlresult,"\n"))
                logs.simple(format("saving interface translations '%s'",xmlfilename))
                if language ~= "en" and xmldata ~= "" then
                    local newdata = xmldata:gsub("(<cd:interface.*language=.)en(.)","%1"..language.."%2",1)
                    newdata = replace(newdata, 'cd:string', 'value', interface.commands, interface.elements, language)
                    newdata = replace(newdata, 'cd:variable' , 'value', interface.variables, nil, language)
                    newdata = replace(newdata, 'cd:parameter', 'name', interface.constants, nil, language)
                    newdata = replace(newdata, 'cd:constant', 'type', interface.variables, nil, language)
                    newdata = replace(newdata, 'cd:variable', 'type', interface.variables, nil, language)
                    newdata = replace(newdata, 'cd:inherit', 'name', interface.commands, interface.elements, language)
                    local xmlfilename = format("cont-%s.xml",language)
                    io.savedata(xmlfilename,newdata)
                    logs.simple(format("saving interface specification '%s'",xmlfilename))
                end
            end
        end
    end
end

function scripts.interface.messages()
    local filename = resolvers.find_file(environment.files[1] or "mult-mes.lua") or ""
    if filename ~= "" then
        local messages = dofile(filename)
        for _, interface in ipairs(messageinterfaces) do
            local texresult = { }
            for category, data in next, messages do
                for tag, message in next, data do
                    if tag ~= "files" then
                        local msg = message[interface] or message["all"] or message["en"]
                        if msg then
                            texresult[#texresult+1] = format("\\setinterfacemessage{%s}{%s}{%s}",category,tag,msg)
                        end
                    end
                end
            end
            texresult[#texresult+1] = format("%%\n\\endinput")
            io.savedata(format("mult-m%s.tex",interface),table.concat(texresult,"\n"))
        end
    end
end

logs.extendbanner("ConTeXt Interface Related Goodies 0.11",true)

messages.help = [[
--scite               generate scite interface
--bbedit              generate scite interface
--jedit               generate scite interface
--check               generate check file
--context             generate context definition files
--messages            generate context message files
]]

if environment.argument("context") then
    scripts.interface.context()
elseif environment.argument("messages") then
    scripts.interface.messages()
elseif environment.argument("scite") or environment.argument("bbedit") or environment.argument("jedit") then
    if environment.argument("scite") then
        scripts.interface.editor("scite")
    end
    if environment.argument("bbedit") then
        scripts.interface.editor("bbedit")
    end
    if environment.argument("jedit") then
        scripts.interface.editor("jedit")
    end
elseif environment.argument("check") then
    scripts.interface.check()
else
    logs.help(messages.help)
end

[-- Attachment #3: Type: text/plain, Size: 486 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Tomáš Pustelník]

--------------------------------------------------
From: "Hans Hagen" <pragma@wxs.nl>
Sent: Saturday, May 15, 2010 9:42 PM
To: "mailing list for ConTeXt users" <ntg-context@ntg.nl>
Cc: "Tomáš Pustelník" <xPustelnikT@seznam.cz>
Subject: Re: [NTG-context] czech sorting in MiIV

> On 15-5-2010 8:14, Tomáš Pustelník wrote:
>> Also I have question. In my bachelor thesis I'm describe language
>> support for ConTeXt and when I tried add some experimental language into
>> ConTeXt (I define system constants and add \loadcorefile{lang-exp} to
>> context.mkii and context.mkiv) and when generating format for MkII
>> everything was ok but for MkIV ConTeXt complained he can't find file
>> lang-exp.tex (I paste it in base folder to other source codes). Any idea
>> whats wrong? In MkIV must I change something else as well?
>
> Ah .. you look in the wrong spot. There's a file:
>
>   mult-def.lua
>
> and interfaces are defined there. I generate the mult-* files using
>
>   mtxrun interface --context
>
> Now, if you just want to experiment, you need to patch mult-def.lua.
>
> You can make a mult-def-mine.lua:
>
>   local t = require "mult-def.lua"
>
>   t.commands.framed.xx = "frmd"
>
>   -- print(table.serialize(t.commands.framed))
>
>   return t
>
> I patched mtx-interface.lua so that you can say;
>
>   mtxrun interface --context mult-def-mine.lua
>
> There is also --messages for the message files.
>
> Of course you have to make a cont-xx.tex file as well.
>
> Hans

Thank you, this will be handy (actually I was planing to ask about user 
interface as well). But what I had in mind was new language definition (like 
lang-sla, lang-ger and so on), perhaps I could be more precise.

But anyway, thanks. As I said, I will use this as well.

Tomas P. 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Aditya Mahajan]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1257 bytes --]

On Sat, 15 May 2010, Tomáš Pustelník wrote:
>>  On 15-5-2010 7:10, Tomáš Pustelník wrote:
>> >  I'm using version 2010.05.13 12:15 so if there isn't some special way
>> >  how to download beta I should be using lates version. Also in MkIV I set
>> >  sorting with this command?
>>
>>  normally the garden is a few hours behind the website
>>
>>  you can fetch the beta from the website and unzip it in texmf-context and
>>  then run luatools --generate
>> 
>> >  \enabletrackers[sorters.tests]
>>
>>  just tracking
>> 
> Thanks, it's working now, but still it sort some words bit different compared 
> to MkII. But it's ok for now, I will check czech norm and try to fix it if 
> needed.
>
> Also I have question. In my bachelor thesis I'm describe language support for 
> ConTeXt and when I tried add some experimental language into ConTeXt (I 
> define system constants and add \loadcorefile{lang-exp} to context.mkii and 
> context.mkiv) and when generating format for MkII everything was ok but for 
> MkIV ConTeXt complained he can't find file lang-exp.tex (I paste it in base 
> folder to other source codes). Any idea whats wrong? In MkIV must I change 
> something else as well?

Did you run luatools --generate.

Aditya

[-- Attachment #2: Type: text/plain, Size: 486 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: czech sorting in MiIV

[Tomáš Pustelník]

--------------------------------------------------
From: "Aditya Mahajan" <adityam@umich.edu>
Sent: Sunday, May 16, 2010 5:08 AM
To: "mailing list for ConTeXt users" <ntg-context@ntg.nl>
Subject: Re: [NTG-context] czech sorting in MiIV
>>
>> Also I have question. In my bachelor thesis I'm describe language support 
>> for
>> ConTeXt and when I tried add some experimental language into ConTeXt (I
>> define system constants and add \loadcorefile{lang-exp} to context.mkii 
>> and
>> context.mkiv) and when generating format for MkII everything was ok but 
>> for
>> MkIV ConTeXt complained he can't find file lang-exp.tex (I paste it in 
>> base
>> folder to other source codes). Any idea whats wrong? In MkIV must I 
>> change
>> something else as well?
>
> Did you run luatools --generate.
>
> Aditya

Thanks, I forgot about this. It's working now 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________