em-dash ligature is converted to en-dash in the generic fontloader

em-dash ligature is converted to en-dash in the generic fontloader

[Ulrike Fischer]

As reported on the dev-luatex list --- is converted to an en-dash
(instead of em-dash) if there are no spaces around the ---.

We could now reproduce the problem also with the generic fontloader:
it appears only with mode=node. I'm using the files from 2019-02-14,
but the problem appeared first in the files from 2018-11-18):

\font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test

dashes-these % gives dash

dashes--these % gives en-dash

dashes---these % gives en-dash WRONG

dashes --- these % gives em-dash

em-dashes---these % gives em-dash 

\bye

context is not affected, there the code works fine.  


-- 
Ulrike Fischer 
https://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Hans Hagen]

On 2/23/2019 1:50 PM, Ulrike Fischer wrote:
> As reported on the dev-luatex list --- is converted to an en-dash
> (instead of em-dash) if there are no spaces around the ---.
> 
> We could now reproduce the problem also with the generic fontloader:
> it appears only with mode=node. I'm using the files from 2019-02-14,
> but the problem appeared first in the files from 2018-11-18):
> 
> \font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test
> 
> dashes-these % gives dash
> 
> dashes--these % gives en-dash
> 
> dashes---these % gives en-dash WRONG
> 
> dashes --- these % gives em-dash
> 
> em-dashes---these % gives em-dash
> 
> \bye
> 
> context is not affected, there the code works fine.

in context we set:

\automatichyphenmode=1

i'll do the same in luatex-plain

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Ulrike Fischer]

Am Sat, 23 Feb 2019 15:55:53 +0100 schrieb Hans Hagen:

>> As reported on the dev-luatex list --- is converted to an en-dash
>> (instead of em-dash) if there are no spaces around the ---.

> in context we set:
> \automatichyphenmode=1

This solves the problem and after some pondering over the
documentation it also looks like a sensible default. But why does
\automatichyphenmode=0 suddenly eat a hyphen?


-- 
Ulrike Fischer 
https://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Hans Hagen]

On 2/23/2019 4:30 PM, Ulrike Fischer wrote:
> Am Sat, 23 Feb 2019 15:55:53 +0100 schrieb Hans Hagen:
> 
>>> As reported on the dev-luatex list --- is converted to an en-dash
>>> (instead of em-dash) if there are no spaces around the ---.
> 
>> in context we set:
>> \automatichyphenmode=1
> 
> This solves the problem and after some pondering over the
> documentation it also looks like a sensible default. But why does
> \automatichyphenmode=0 suddenly eat a hyphen?

because <disc -> followed by -- is pretty obscure ... a replace could be 
seen as --- and when hyphenated as - -- and such ... it's about time 
that texies start using the proper unicode symbols instead of these 
funny ligatures

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Ulrike Fischer]

Am Sat, 23 Feb 2019 17:11:12 +0100 schrieb Hans Hagen:

> because <disc -> followed by -- is pretty obscure ... a replace could be 
> seen as --- and when hyphenated as - -- and such ... it's about time 
> that texies start using the proper unicode symbols instead of these 
> funny ligatures

well the problem with the proper unicode symbols is that at first
they are not so easy to input (that could be handled by the editor)
and that they are not so visible. - - — look quite the same in my
editor, even more if the screen is not so large and when my eyes
tire. If -- wouldn't work I would resort to a command like \ndash
instead only to get a better visual clue what's in the document. 

But regardless from the input method. How can one enable a line
break after an em-dash when \automichyphenmode is 1?  The only way I
found is to explicitly insert a penalty:

\starttext
\hsize=2pt

dash---\penalty1 dash

dash—\penalty1 dash

\stoptext




-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Henri Menke]

On 26/02/19 3:55 AM, Ulrike Fischer wrote:
> Am Sat, 23 Feb 2019 17:11:12 +0100 schrieb Hans Hagen:
> 
>> because <disc -> followed by -- is pretty obscure ... a replace could be 
>> seen as --- and when hyphenated as - -- and such ... it's about time 
>> that texies start using the proper unicode symbols instead of these 
>> funny ligatures
> 
> well the problem with the proper unicode symbols is that at first
> they are not so easy to input (that could be handled by the editor)
> and that they are not so visible. - - — look quite the same in my
> editor, even more if the screen is not so large and when my eyes
> tire. If -- wouldn't work I would resort to a command like \ndash
> instead only to get a better visual clue what's in the document. 
> 
> But regardless from the input method. How can one enable a line
> break after an em-dash when \automichyphenmode is 1?  The only way I
> found is to explicitly insert a penalty:
> 
> \starttext
> \hsize=2pt
> 
> dash---\penalty1 dash
> 
> dash—\penalty1 dash

dash\discretionary{---}{}{---}dash

dash\discretionary{—}{}{—}dash

> 
> \stoptext
> 
> 
> 
> 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Rik Kabel]

On 2/25/2019 15:45, Henri Menke wrote:
> On 26/02/19 3:55 AM, Ulrike Fischer wrote:
>> Am Sat, 23 Feb 2019 17:11:12 +0100 schrieb Hans Hagen:
>>
>>> because <disc -> followed by -- is pretty obscure ... a replace could be
>>> seen as --- and when hyphenated as - -- and such ... it's about time
>>> that texies start using the proper unicode symbols instead of these
>>> funny ligatures
>> well the problem with the proper unicode symbols is that at first
>> they are not so easy to input (that could be handled by the editor)
>> and that they are not so visible. - - — look quite the same in my
>> editor, even more if the screen is not so large and when my eyes
>> tire. If -- wouldn't work I would resort to a command like \ndash
>> instead only to get a better visual clue what's in the document.
>>
>> But regardless from the input method. How can one enable a line
>> break after an em-dash when \automichyphenmode is 1?  The only way I
>> found is to explicitly insert a penalty:
>>
>> \starttext
>> \hsize=2pt
>>
>> dash---\penalty1 dash
>>
>> dash—\penalty1 dash
> dash\discretionary{---}{}{---}dash
>
> dash\discretionary{—}{}{—}dash
>
>> \stoptext

dash|---|dash

dash|—|dash

-- 
Rik

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Henri Menke]

On 26/02/19 3:55 AM, Ulrike Fischer wrote:
> Am Sat, 23 Feb 2019 17:11:12 +0100 schrieb Hans Hagen:
> 
>> because <disc -> followed by -- is pretty obscure ... a replace could be 
>> seen as --- and when hyphenated as - -- and such ... it's about time 
>> that texies start using the proper unicode symbols instead of these 
>> funny ligatures
> 
> well the problem with the proper unicode symbols is that at first
> they are not so easy to input (that could be handled by the editor)
> and that they are not so visible. - - — look quite the same in my
> editor, even more if the screen is not so large and when my eyes
> tire. If -- wouldn't work I would resort to a command like \ndash
> instead only to get a better visual clue what's in the document. 
> 
> But regardless from the input method. How can one enable a line
> break after an em-dash when \automichyphenmode is 1?  The only way I
> found is to explicitly insert a penalty:

\setbreakpoints[compound] makes dash---dash breakable but not dash—dash

> 
> \starttext
> \hsize=2pt
> 
> dash---\penalty1 dash
> 
> dash—\penalty1 dash
> 
> \stoptext
> 
> 
> 
> 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

em-dash ligature is converted to en-dash in the generic fontloader

[Marcel Krüger]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On 2/23/2019 1:50 PM, Ulrike Fischer wrote:
> As reported on the dev-luatex list --- is converted to an en-dash
> (instead of em-dash) if there are no spaces around the ---.
> 
> We could now reproduce the problem also with the generic fontloader:
> it appears only with mode=node. I'm using the files from 2019-02-14,
> but the problem appeared first in the files from 2018-11-18):
> 
> \font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test
> 
> dashes-these % gives dash
> 
> dashes--these % gives en-dash
> 
> dashes---these % gives en-dash WRONG
> 
> dashes --- these % gives em-dash
> 
> em-dashes---these % gives em-dash
> 
> \bye
> 
> context is not affected, there the code works fine.

This issue can be fixed in the fontloader with the attached patch.

Best regards,
Marcel Krüger

[-- Attachment #2: 0001-Fix-ligature-followed-by-disc.diff --]
[-- Type: application/octet-stream, Size: 653 bytes --]

diff --git a/tex/context/base/mkiv/font-ots.lua b/tex/context/base/mkiv/font-ots-fixed.lua
index 563094f..b1068f7 100644
--- a/tex/context/base/mkiv/font-ots.lua
+++ b/tex/context/base/mkiv/font-ots-fixed.lua
@@ -841,7 +841,7 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature,rlmode,skip
                 local prev   = getprev(start)
                 if stop then
                     setnext(stop)
-                    local tail = getprev(stop)
+                    local tail = stop
                     local copy = copy_node_list(start)
                     local liat = find_node_tail(copy)
                     if pre then

[-- Attachment #3: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Hans Hagen]

On 2/28/2019 12:38 PM, Marcel Krüger wrote:
> On 2/23/2019 1:50 PM, Ulrike Fischer wrote:
>> As reported on the dev-luatex list --- is converted to an en-dash
>> (instead of em-dash) if there are no spaces around the ---.
>>
>> We could now reproduce the problem also with the generic fontloader:
>> it appears only with mode=node. I'm using the files from 2019-02-14,
>> but the problem appeared first in the files from 2018-11-18):
>>
>> \font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test
>>
>> dashes-these % gives dash
>>
>> dashes--these % gives en-dash
>>
>> dashes---these % gives en-dash WRONG
>>
>> dashes --- these % gives em-dash
>>
>> em-dashes---these % gives em-dash
>>
>> \bye
>>
>> context is not affected, there the code works fine.
> 
> This issue can be fixed in the fontloader with the attached patch.
Hm, I don't see those lines here and it looks ok in context so ... no 
patches applied (and definitely not in the tex live code freeze window)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Marcel Krüger]

 ---- On Fri, 01 Mar 2019 11:17:56 +0100 Hans Hagen <j.hagen@xs4all.nl> wrote ----
 > On 2/28/2019 12:38 PM, Marcel Kr&uuml;ger wrote: 
 > > On 2/23/2019 1:50 PM, Ulrike Fischer wrote: 
 > >> As reported on the dev-luatex list --- is converted to an en-dash 
 > >> (instead of em-dash) if there are no spaces around the ---. 
 > >> 
 > >> We could now reproduce the problem also with the generic fontloader: 
 > >> it appears only with mode=node. I'm using the files from 2019-02-14, 
 > >> but the problem appeared first in the files from 2018-11-18): 
 > >> 
 > >> \font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test 
 > >> 
 > >> dashes-these % gives dash 
 > >> 
 > >> dashes--these % gives en-dash 
 > >> 
 > >> dashes---these % gives en-dash WRONG 
 > >> 
 > >> dashes --- these % gives em-dash 
 > >> 
 > >> em-dashes---these % gives em-dash 
 > >> 
 > >> \bye 
 > >> 
 > >> context is not affected, there the code works fine. 
 > >  
 > > This issue can be fixed in the fontloader with the attached patch. 
 > Hm, I don't see those lines here and it looks ok in context so ... no  

For reproduction in context, the code

    \starttext
    ff\discretionary{l}{}{l}
    \stoptext

currently produces a single `fl` ligature instead of `ffl`.

Interesting that you didn't see those lines, maybe we have different versions?
The patched line is line 844 in the file `font-ots.lua` of ConTeXt beta, updated today.

Best regards
Marcel

 > patches applied (and definitely not in the tex live code freeze window) 
 >  
 > Hans 
 >  
 > ----------------------------------------------------------------- 
 >                                            Hans Hagen | PRAGMA ADE 
 >                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands 
 >         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl 
 > ----------------------------------------------------------------- 
 > 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Hans Hagen]

On 3/1/2019 12:29 PM, Marcel Krüger wrote:
>   ---- On Fri, 01 Mar 2019 11:17:56 +0100 Hans Hagen <j.hagen@xs4all.nl> wrote ----
>   > On 2/28/2019 12:38 PM, Marcel Kr&uuml;ger wrote:
>   > > On 2/23/2019 1:50 PM, Ulrike Fischer wrote:
>   > >> As reported on the dev-luatex list --- is converted to an en-dash
>   > >> (instead of em-dash) if there are no spaces around the ---.
>   > >>
>   > >> We could now reproduce the problem also with the generic fontloader:
>   > >> it appears only with mode=node. I'm using the files from 2019-02-14,
>   > >> but the problem appeared first in the files from 2018-11-18):
>   > >>
>   > >> \font\test={file:texgyreheros-regular.otf:+tlig;mode=node;}\test
>   > >>
>   > >> dashes-these % gives dash
>   > >>
>   > >> dashes--these % gives en-dash
>   > >>
>   > >> dashes---these % gives en-dash WRONG
>   > >>
>   > >> dashes --- these % gives em-dash
>   > >>
>   > >> em-dashes---these % gives em-dash
>   > >>
>   > >> \bye
>   > >>
>   > >> context is not affected, there the code works fine.
>   > >
>   > > This issue can be fixed in the fontloader with the attached patch.
>   > Hm, I don't see those lines here and it looks ok in context so ... no
> 
> For reproduction in context, the code
> 
>      \starttext
>      ff\discretionary{l}{}{l}
>      \stoptext
> 
> currently produces a single `fl` ligature instead of `ffl`.
> 
> Interesting that you didn't see those lines, maybe we have different versions?
> The patched line is line 844 in the file `font-ots.lua` of ConTeXt beta, updated today.
your patch referred to other numbers

i'll look into it (also why it is as it is)

thanks for the example

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Ulrike Fischer]

Am Fri, 1 Mar 2019 11:17:56 +0100 schrieb Hans Hagen:

>  it looks ok in context 

You need to set automatichyphenmode=0 to see the problem in context:
the handling of the --- ligature is clearly broken in some cases:

\starttext
\automatichyphenmode=0
A---B A --- B
\stoptext

> Hm, I don't see those lines here 

I see 

 local tail = getprev(stop) 

in line 844 of font-ots.lua in a just updated context minimals. 

If I change this to 

 local tail = stop

as suggested by Marcel and rebuild the formats, the ligature behaves
again. 

> (and definitely not in the tex live code freeze window)

TeXLive is frozen and pretest has begun, so I don't see a danger to
apply patches now. 


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: em-dash ligature is converted to en-dash in the generic fontloader

[Hans Hagen]

On 3/1/2019 1:15 PM, Ulrike Fischer wrote:

> TeXLive is frozen and pretest has begun, so I don't see a danger to
> apply patches now.
binaries get frozen at 22/3 and context isn't checked in yet (comes 
after first binary generation)

anyway, i'll look at it (maybe this weekend)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________