characters.tex.toutf bug

characters.tex.toutf bug

[Max Chernoff via ntg-context]

Hi Hans,

"characters.tex.toutf" only works if there is a backslash somewhere in
its input string. This leads to some really surprising behaviour in the
following example:

   \definefontfeature[default][tlig=no, trep=no]
   
   \startbuffer[test-bib]
   @article{one,
       author = {``Author---\v{O}ne''},
   }
   @article{two,
       author = {``Author---Two''},
   }
   \stopbuffer
   
   \usebtxdataset[test-bib.buffer]
   
   \starttext
   \placelistofpublications[method=dataset]
   \stoptext

Removing the following lines

       elseif not find(str,"\\",1,true) then
           return str
   
from the definition of "characters.tex.toutf" in "char-tex.{lmt,lua}"
fixes this issue for me.

Thanks,
-- Max
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: characters.tex.toutf bug

[Hans Hagen via ntg-context]

On 3/30/2023 7:57 AM, Max Chernoff via ntg-context wrote:
> Hi Hans,
> 
> "characters.tex.toutf" only works if there is a backslash somewhere in
> its input string. This leads to some really surprising behaviour in the
> following example:
> 
>     \definefontfeature[default][tlig=no, trep=no]
>     
>     \startbuffer[test-bib]
>     @article{one,
>         author = {``Author---\v{O}ne''},
>     }
>     @article{two,
>         author = {``Author---Two''},
>     }
>     \stopbuffer
>     
>     \usebtxdataset[test-bib.buffer]
>     
>     \starttext
>     \placelistofpublications[method=dataset]
>     \stoptext
> 
> Removing the following lines
> 
>         elseif not find(str,"\\",1,true) then
>             return str
>     
> from the definition of "characters.tex.toutf" in "char-tex.{lmt,lua}"
> fixes this issue for me.
it is more side effect of removing the (declared obsolete some years 
ago) `` '' input which catched it later

i tend to remove

         for k, v in next, ligaturemapping do
             hash[k] = v
         end

in favor of an explicit 'tquo' feature that one can enable if wanted

\definefontfeature[default][default][tlig=yes,tquo=yes,trep=yes]

\startbuffer[test-bib]
@article{one,
    author = {``Author---\v{O}ne''},
}
@article{two,
    author = {``Author---Two''},
}
@article{three,
    author = {Author---Two},
}
@article{three,
    author = {,,Author---Two''},
}
\stopbuffer

so I tested that with the above which seems to work.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: characters.tex.toutf bug

[Max Chernoff via ntg-context]

Hi Hans,

> it is more side effect of removing the (declared obsolete some years 
> ago) `` '' input which catched it later
> 
> i tend to remove
> 
>          for k, v in next, ligaturemapping do
>              hash[k] = v
>          end
>
> in favor of an explicit 'tquo' feature that one can enable if wanted
> 
> \definefontfeature[default][default][tlig=yes,tquo=yes,trep=yes]

That does make the output more consistent, but not in the way that I'm
wanting. My real code looks more like this:

   \definefontfeature[default][default][tlig=yes, tquo=yes, trep=yes]
   
   \startbuffer[test-bib]
   @article{one,
       author = {``Auth\v{o}r---One''},
   }
   @article{two,
       author = {``Authǒr---Two''},
   }
   \stopbuffer
   
   \usebtxdataset[test-bib.buffer]
   
   \startluacode
       for key, value in table.sortedpairs(publications.datasets.default.luadata) do
           print(key, publications.prerollcmdstring(value.author))
       end
   \stopluacode

publications.prerollcmdstring seems to run before any of the
replacements/ligatures, so with your fix I get this as the output:

   one     ``Authǒr---One''
   two     ``Authǒr---Two''
   
while with my suggested fix I get:

   one     “Authǒr—One”
   two     “Authǒr—Two”
   
I'm using a CLD document to convert a bunch of old .bib and .tex files into
Lua tables (which I'll then convert into XML). I can't modify any of the
input files, so I'll need to somehow deal with all the LaTeX-style input.
Using publications.prerollcmdstring + characters.tex.toutf seems like the
best way to do this, but I'm open to a different solution if you have any
suggestions.

Thanks,
-- Max
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________