[NTG-context] Toggling the symbol for the zero-width joiner and related Unicode control characters

[NTG-context] Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 1294 bytes --]

Dear gang,

Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.

In some text editors, such as Windows Notepad (Uniscribe), there is the option

"Show Unicode control characters"

which toggles the symbols and keeps the effects.

I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.

Does anyone know or recall where the symbol-toggling mechanism is?
If not, what would be the simplest way to implement it?

Here is a MWE (note that the three control chars might be invisible in your email client):

==============
\startTEXpage[offset=1em]
\definedfont[almfixed at 14pt]
ZWJ: ‌

ZWNJ: ‍

NBSP:
\stopTEXpage
==============

In Notepad, all three control characters can be turned on or off at once.
In Scintilla, only the NBSP appears and there is no option to toggle.
In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.

Thank you in advance for your guidance.

Best wishes
Idris
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523


[-- Attachment #1.2: Type: text/html, Size: 2915 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen]

On 9/21/2023 3:31 PM, Hamid,Idris wrote:
> Dear gang,
> 
> Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.
> 
> In some text editors, such as Windows Notepad (Uniscribe), there is the option
> 
> "Show Unicode control characters"
> 
> which toggles the symbols and keeps the effects.
> 
> I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.
> 
> Does anyone know or recall where the symbol-toggling mechanism is?
> If not, what would be the simplest way to implement it?
> 
> Here is a MWE (note that the three control chars might be invisible in your email client):
> 
> ==============
> \startTEXpage[offset=1em]
> \definedfont[almfixed at 14pt]
> ZWJ: ‌
> 
> ZWNJ: ‍
> 
> NBSP:
> \stopTEXpage
> ==============
> 
> In Notepad, all three control characters can be turned on or off at once.
> In Scintilla, only the NBSP appears and there is no option to toggle.
> In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.
> 
> Thank you in advance for your guidance.
\enabletrackers[typesetters.nbsp]


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 2147 bytes --]


------ Original Message ------
From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Date 9/21/2023 8:52:45 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

** Caution: EXTERNAL Sender **

On 9/21/2023 3:31 PM, Hamid,Idris wrote:
Dear gang,

Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.

In some text editors, such as Windows Notepad (Uniscribe), there is the option

"Show Unicode control characters"

which toggles the symbols and keeps the effects.

I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.

Does anyone know or recall where the symbol-toggling mechanism is?
If not, what would be the simplest way to implement it?

Here is a MWE (note that the three control chars might be invisible in your email client):

==============
\startTEXpage[offset=1em]
\definedfont[almfixed at 14pt]
ZWJ: ‌

ZWNJ: ‍

NBSP:
\stopTEXpage
==============

In Notepad, all three control characters can be turned on or off at once.
In Scintilla, only the NBSP appears and there is no option to toggle.
In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.

Thank you in advance for your guidance.
\enabletrackers[typesetters.nbsp]

Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:

=======
\startTEXpage[offset=1em]
\disabletrackers[typesetters.directions]
\disabletrackers[typesetters.zwj]
\disabletrackers[typesetters.zwnj]
\disabletrackers[typesetters.nbsp]
\definedfont[almfixed at 14pt]
ZWJ: ‌
ZWNJ: ‍
NBSP:
\stopTEXpage
=======

See attached, please advise.

Best wishes
Idris
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 5955 bytes --]

[-- Attachment #2: test-zwj.pdf --]
[-- Type: application/pdf, Size: 6712 bytes --]

[-- Attachment #3: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen]

On 9/21/2023 10:53 PM, Hamid,Idris wrote:
> 
> ------ Original Message ------
>  From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
> To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
> Date 9/21/2023 8:52:45 AM
> Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
> 
> ** Caution: EXTERNAL Sender **
> 
> On 9/21/2023 3:31 PM, Hamid,Idris wrote:
> Dear gang,
> 
> Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.
> 
> In some text editors, such as Windows Notepad (Uniscribe), there is the option
> 
> "Show Unicode control characters"
> 
> which toggles the symbols and keeps the effects.
> 
> I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.
> 
> Does anyone know or recall where the symbol-toggling mechanism is?
> If not, what would be the simplest way to implement it?
> 
> Here is a MWE (note that the three control chars might be invisible in your email client):
> 
> ==============
> \startTEXpage[offset=1em]
> \definedfont[almfixed at 14pt]
> ZWJ: ‌
> 
> ZWNJ: ‍
> 
> NBSP:
> \stopTEXpage
> ==============
> 
> In Notepad, all three control characters can be turned on or off at once.
> In Scintilla, only the NBSP appears and there is no option to toggle.
> In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.
> 
> Thank you in advance for your guidance.
> \enabletrackers[typesetters.nbsp]
> 
> Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:
> 
> =======
> \startTEXpage[offset=1em]
> \disabletrackers[typesetters.directions]
> \disabletrackers[typesetters.zwj]
> \disabletrackers[typesetters.zwnj]
> \disabletrackers[typesetters.nbsp]
> \definedfont[almfixed at 14pt]
> ZWJ: ‌
> ZWNJ: ‍
> NBSP:
> \stopTEXpage
> =======
> 
> See attached, please advise.
joiners are part of replacement etc and can come and go ... they are 
characters (we could visualize them but one never knows for sure if one 
sees them)

nbsp are spaces and become glue that we can trace reliable in the node list

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 3791 bytes --]



------ Original Message ------
From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
To "Hamid,Idris" <Idris.Hamid@ColoState.EDU<mailto:Idris.Hamid@ColoState.EDU>>; "mailing list for ConTeXt users" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Date 9/21/2023 3:29:22 PM
Subject Re: [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
  Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:

=======
\startTEXpage[offset=1em]
\disabletrackers[typesetters.directions]
\disabletrackers[typesetters.zwj]
\disabletrackers[typesetters.zwnj]
\disabletrackers[typesetters.nbsp]
\definedfont[almfixed at 14pt]
ZWJ: ‌
ZWNJ: ‍
NBSP:
\stopTEXpage
=======

See attached, please advise.
joiners are part of replacement etc and can come and go ... they are
characters (we could visualize them but one never knows for sure if one
sees them)

nbsp are spaces and become glue that we can trace reliable in the node list

Many thanks. Ok, here is another MWE featuring a workaround using fallbacks:

==============
\definefontfallback[nosymbols] [file:lmmono10-regular] [200C,200D] [force=yes]
\starttypescript [serif] [alm] [name]
    \definefontsynonym [Serif] [ArabicLatinSerif]
\stoptypescript
\starttypescript [mono] [alm] [name]
    \definefontsynonym [Mono]  [ArabicLatinMono]
\stoptypescript
\starttypescript [serif] [alm]
    \definefontsynonym [ArabicLatinSerif] [file:almfixed] % [fallbacks=nosymbols]
\stoptypescript
\starttypescript [mono] [alm]
    \definefontsynonym [ArabicLatinMono] [file:almfixed] [fallbacks=nosymbols]
\stoptypescript
\starttypescript [almfixed-nosymbols]
\definetypeface [\typescriptone] [rm] [serif] [alm] [default]
\definetypeface [\typescriptone] [tt] [mono] [alm] [default]
\stoptypescript
\usetypescript[almfixed-nosymbols]
\setupbodyfont[almfixed-nosymbols,12pt]
\startTEXpage[offset=1em]
\rm
ZWJ: ‌
ZWNJ: ‍
NBSP:
\tt
ZWJ: ‌
ZWNJ: ‍
NBSP:
\stopTEXpage
==============

Under \rm we get the symbols, and under \tt they are suppressed. Of course it doesn't matter what fallback font one uses, as long as it has no control-character symbols.

1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols?

2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font?

3. Ideally:
a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor);
b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge.

And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge.  AlmFixed, of course, has them.

Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts.

Industry workhorses like Linotype Lotus (Arabic) also have them.

Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font.

The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing.

Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out!

Best wishes
Idris
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 10012 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen via ntg-context]

On 9/22/2023 6:16 AM, Hamid,Idris wrote:
> 
> 
> ------ Original Message ------
>  From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
> To "Hamid,Idris" <Idris.Hamid@ColoState.EDU<mailto:Idris.Hamid@ColoState.EDU>>; "mailing list for ConTeXt users" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
> Date 9/21/2023 3:29:22 PM
> Subject Re: [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
>    Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:
> 
> =======
> \startTEXpage[offset=1em]
> \disabletrackers[typesetters.directions]
> \disabletrackers[typesetters.zwj]
> \disabletrackers[typesetters.zwnj]
> \disabletrackers[typesetters.nbsp]
> \definedfont[almfixed at 14pt]
> ZWJ: ‌
> ZWNJ: ‍
> NBSP:
> \stopTEXpage
> =======
> 
> See attached, please advise.
> joiners are part of replacement etc and can come and go ... they are
> characters (we could visualize them but one never knows for sure if one
> sees them)
> 
> nbsp are spaces and become glue that we can trace reliable in the node list
> 
> Many thanks. Ok, here is another MWE featuring a workaround using fallbacks:
> 
> ==============
> \definefontfallback[nosymbols] [file:lmmono10-regular] [200C,200D] [force=yes]
> \starttypescript [serif] [alm] [name]
>      \definefontsynonym [Serif] [ArabicLatinSerif]
> \stoptypescript
> \starttypescript [mono] [alm] [name]
>      \definefontsynonym [Mono]  [ArabicLatinMono]
> \stoptypescript
> \starttypescript [serif] [alm]
>      \definefontsynonym [ArabicLatinSerif] [file:almfixed] % [fallbacks=nosymbols]
> \stoptypescript
> \starttypescript [mono] [alm]
>      \definefontsynonym [ArabicLatinMono] [file:almfixed] [fallbacks=nosymbols]
> \stoptypescript
> \starttypescript [almfixed-nosymbols]
> \definetypeface [\typescriptone] [rm] [serif] [alm] [default]
> \definetypeface [\typescriptone] [tt] [mono] [alm] [default]
> \stoptypescript
> \usetypescript[almfixed-nosymbols]
> \setupbodyfont[almfixed-nosymbols,12pt]
> \startTEXpage[offset=1em]
> \rm
> ZWJ: ‌
> ZWNJ: ‍
> NBSP:
> \tt
> ZWJ: ‌
> ZWNJ: ‍
> NBSP:
> \stopTEXpage
> ==============
> 
> Under \rm we get the symbols, and under \tt they are suppressed. Of course it doesn't matter what fallback font one uses, as long as it has no control-character symbols.
> 
> 1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols?

given the inconsistency in what is or is not in a font the only way out 
is to have our own visualization (consistent across fonts) and even then 
it would add some mess because we're talking of a mix of characters that 
can have gone (as part of rendering) or are not characters at all but 
spacing

so, in that case only 'verbatim' is a candidate for visualization, not 
so much typeset text

> 2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font?

it's gone by that time ... the line break mmechanism uses glue, not 
characters

> 3. Ideally:
> a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor);

only there (with some non interfering rendering i guess) and even then 
it's probably an additonal pass over the node list

> b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is 
out of our control

> But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge.

so an inconsistent mess not worth wasting time on (as this is hobbyism 
only fun can be a motivational factir)

> And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge.  AlmFixed, of course, has them.
> 
> Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts.
> 
> Industry workhorses like Linotype Lotus (Arabic) also have them.

i'm not interested in those .. can't afford them for playing around
purposes .. we only look into commercial fonts if we get a dozen 
unresticted copies for context developers

> Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font.

sure, as long as there is no rendering ... they show the input

> The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing.

a bit subjectiev arguing -)

> Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out!
sore, but not with 'instant priority' (unless it is some project)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 5260 bytes --]



------ Original Message ------
From "Hans Hagen via ntg-context" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Cc "Hans Hagen" <j.hagen@freedom.nl<mailto:j.hagen@freedom.nl>>
Date 9/22/2023 3:53:03 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols?

given the inconsistency in what is or is not in a font the only way out
is to have our own visualization (consistent across fonts) and even then
it would add some mess because we're talking of a mix of characters that
can have gone (as part of rendering) or are not characters at all but
spacing

so, in that case only 'verbatim' is a candidate for visualization, not
so much typeset text

Hm, ok. Since almfixed is based on Knuth's mono, perhaps its visuals of the control characters can be extracted and used as fallback symbols.

Yes: For typeset text/printing visualization is generally unnecessary (the point of this thread).

2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font?

it's gone by that time ... the line break mmechanism uses glue, not
characters

Ok

3. Ideally:
a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor);

only there (with some non interfering rendering i guess) and even then
it's probably an additonal pass over the node list

Ok, that would be good.

b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is
out of our control

If the symbols are in the font, then they are not suppressed. See below.

But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge.

so an inconsistent mess not worth wasting time on (as this is hobbyism
only fun can be a motivational factir)

But there is a certain consistency -- see below.

And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge. AlmFixed, of course, has them.

Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts.

Industry workhorses like Linotype Lotus (Arabic) also have them.

i'm not interested in those .. can't afford them for playing around
purposes .. we only look into commercial fonts if we get a dozen
unresticted copies for context developers

Except for Linotype Lotus, each of the Arabic-script fonts mentioned above is free, not commercial -)

(There is also a free version of Lotus -- it also has the symbolic rendering of the contol chars.)

Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font.

sure, as long as there is no rendering ... they show the input

But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

Yet another curse of the WYSIWYG paradigm, which mixes form and content -)

The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing.

a bit subjectiev arguing -)

Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

(Yes, for the upcoming Husayni I can add a font feature that does the trick, but that will be an exception to the rule.)

Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out!
sore, but not with 'instant priority' (unless it is some project)

My immediate project (no Husayni) is a book that features English translation of an Arabic text (hence the interest in the recent streams thread). Using some Unicode control characters will be unavoidable to get the rendering effects correct, but the symbols will need to be suppressed.

Am thinking/hoping that a ConTeXt-specific font feature can do the trick. Since there appears to be consistency across Arabic fonts in this matter it should not be messy at all, simply a fallback that sends the symbols to some no-man's land.

(A thought: Some of the code you kindly provided for transliteration might be reusable as well.. But a general solution for all ConTeXt users would be ideal.)

In any case, many thanks for your help in thinking this through.

Best wishes
Idris
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 11675 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen]

Hi,

I found it ...

\startbuffer
\definedfont[almfixed*default]hello w\zwnj o\zwj r\zwnj l\zwj d
\stopbuffer

\getbuffer

\start
\setcharacterstripping[1]
\getbuffer
\stop

so now, being its only user, you have to wikify it ...

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen]

On 9/22/2023 2:39 PM, Hamid,Idris wrote:

> b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).
> 
> they basically are unless some font features keeps them around which is
> out of our control

irr it was you who wanted them to be wiped decades ago as some fonts 
visualized them by default

> But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

?

> Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

so you want to see soem zwj sumbol in a rendered text?
  Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 770 bytes --]


------ Original Message ------
From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Date 9/22/2023 7:03:34 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters


Hi,

I found it ...

\startbuffer
\definedfont[almfixed*default]hello w\zwnj o\zwj r\zwnj l\zwj d
\stopbuffer

\getbuffer

\start
\setcharacterstripping[1]
\getbuffer
\stop

so now, being its only user, you have to wikify it ...

Ah, there it is! Many thanks, will wikify it.
Best wishes
Idris

--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 3018 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 3207 bytes --]


------ Original Message ------
From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Date 9/22/2023 7:15:25 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

** Caution: EXTERNAL Sender **

On 9/22/2023 2:39 PM, Hamid,Idris wrote:

b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is
out of our control

irr it was you who wanted them to be wiped decades ago as some fonts
visualized them by default

Yes, that's exactly the point: Somewhere along the course of history, it became standard for Arabic-script fonts (and other cursive-script fonts as well) to include symbols for the control characters.

In typo-rep there is also

%D \starttyping
%D \definefontfeature[default][default][mode=node,formatters=strip]
%D \stoptyping

You included some notes about Khaled, so I guess he faced the same issue. His Amiri font displays the symbols by default, as do other Arabic fonts.

(It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.)



But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

?

Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted.

(I suppose that InDesign behaves the same way.)

That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue.



Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

so you want to see soem zwj sumbol in a rendered text?

Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -)

Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does.

(Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!)

We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.)

I hope the above made sense.

Best wishes
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 7591 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hans Hagen via ntg-context]

On 9/22/2023 3:51 PM, Hamid,Idris wrote:
> 
> ------ Original Message ------
>  From "Hans Hagen" <j.hagen@xs4all.nl<mailto:j.hagen@xs4all.nl>>
> To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
> Date 9/22/2023 7:15:25 AM
> Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
> 
> ** Caution: EXTERNAL Sender **
> 
> On 9/22/2023 2:39 PM, Hamid,Idris wrote:
> 
> b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).
> 
> they basically are unless some font features keeps them around which is
> out of our control
> 
> irr it was you who wanted them to be wiped decades ago as some fonts
> visualized them by default
> 
> Yes, that's exactly the point: Somewhere along the course of history, it became standard for Arabic-script fonts (and other cursive-script fonts as well) to include symbols for the control characters.
> 
> In typo-rep there is also
> 
> %D \starttyping
> %D \definefontfeature[default][default][mode=node,formatters=strip]
> %D \stoptyping
> 
> You included some notes about Khaled, so I guess he faced the same issue. His Amiri font displays the symbols by default, as do other Arabic fonts.
> 
> (It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.)

I admit that I don't follow what happens with xetex (they changed the 
rendere at some point indeed) not HB (I only notice that it gets updates 
frequently in the tex live repository which makes me wonder how one 
retains compatility unless one freezes). I actually kept the lib binding 
code that can use it around for your font testing (we wanted to see what 
uniscribe does), not sure if it still works.

Anyway, we're entering the bug cq. side effect becomes feature area 
here; just like yesterdays perfect bidi algorithm is todays less pefect 
one replaced by ...

> But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

if context shows it then it is not a feature but hard coded shapes which 
is weird; how does one know what to 'remove' or not? And in what stage? 
If they are zero width it is simple to ignore them in the backend, if 
they have dimensions (w/h/d) then they contributed and wiping is tricky

> Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted.
> 
> (I suppose that InDesign behaves the same way.)

I don't know ... irr these dtp programs are more like "if you want this 
feature applied select a range of characters and apply it"

> That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue.

In some way it's also flaws in the open type approach. Basically that 
happens when application stuff becomes a standard and one forgets that 
it was (is) application driven. (And you haven't seen variable fonts and 
color fonts yet ... no pretty standards either.)

> Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

Funny rules ... but I'm not going top enable 'wipe' by default: after 
all, one gets what one deserves, nto what one likes (which can differ 
per day). But you can enable the wiping. We can of course ignore in the 
backend when zero width but then how to explain that they contributed to 
the ht/dp (unless we wipe these dimensions) ... all slow-downers

> so you want to see soem zwj sumbol in a rendered text?
> 
> Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -)
> 
> Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does.
> 
> (Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!)
> 
> We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.)
so what is it now:

- for verbatim you can use almfixed and they show up (when they have a 
glyph)
- for other fonts if they have them they show up (unless gone in the 
process ot rendering)
- but you can wipe them optionally

not sure what more we need

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Install fonts on MS Windows

[Jean-Pierre Delange]

Dear list,
I have a very simple question, perhaps too simple...
I've installed CTX on MSWindows x64, but I can't use the fonts I've 
installed, despite the command: set 
OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

I cleared the Windows font cache, restarted the machine, then reloaded 
with the command :
mtxrun --script fonts --reload

A search for the fonts I had installed (theano didot and gfs didot) gave 
no results with :

mtxrun --script fonts --list --pattern=*gfs* --all.

I admit that I don't know in which directory it is best to install the 
fonts for optimum performance.

Any ideas?

-- 
Jean-Pierre Delange
Ancients&Moderns
"Few discoveries are more irritating than those which expose the pedigree of ideas". Lord Acton

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Pablo Rodriguez]

On 9/23/23 14:40, Jean-Pierre Delange wrote:
> Dear list,
> I have a very simple question, perhaps too simple...
> I've installed CTX on MSWindows x64, but I can't use the fonts I've
> installed, despite the command: set
> OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

Dear Jean-Pierre,

only if you don’t have no way to install fonts on the OS directory for
fonts (%WINDIR%\Fonts in Windows), I would use (in that order):

  1. %USERPROFILE%\texmf\texmf-fonts\, or as last option
  2. [context-dir]\tex\texmf-local\texmf-fonts\

Excuse me if this is obvious to you, %USERPROFILE% is a Windows variable
(Windows variables, once defined, are invoked enclosed in percent signs
[such as in %variable%] and they are case-insensitive).

%USERPROFILE% in Windows is $HOME in Unix.

[context-dir] is no variable. It means the path were you have your
ConTeXt distribution installed (in case you have a ConTeXt only
distribution, and not TeX Live installed [I mean, a full text distribution).

BTW, do you really have a directory C:\Windows\Users\adeimantos? Sorry,
but I think it reads C:\Users\adeimantos (actually, this would be the
value of the Windows variable %USERPROFILE% when you are logged in).

[More on Windows variables, https://ss64.com/nt/syntax-variables.html.]

> I cleared the Windows font cache, restarted the machine, then reloaded
> with the command :
> mtxrun --script fonts --reload

I think it is better the following:

  mtxrun --script cache --erase && mtxrun --generate

It erases the whole cache, which will be generated in your next ConTeXt run.

> A search for the fonts I had installed (theano didot and gfs didot) gave
> no results with :
>
> mtxrun --script fonts --list --pattern=*gfs* --all.

I think this would do the trick. Compile the following source:

  \mainlanguage[agr]
  \definefontfamily[mainface][rm][GFS Didot]
  \definefontfamily[mainface][ss][Theano Didot]
  \setupbodyfont[mainface]
  \starttext
  \doloopoverlist{\tf, \it, \bf, \bi}
    {\recursestring\input aristotle-grc\par}

  \ss\input aristotle-grc
  \stoptext

So you will see whether ConTeXt is able to read the fonts or not (if
they are embedded in the output PDF document or not).

> I admit that I don't know in which directory it is best to install the
> fonts for optimum performance.

If you install the fonts in %WINDIR%\Fonts (Windows font directory), the
will be available for any program. In some cases, this is important,
even when using only with ConTeXt.

If you need only in TeX, %USERPROFILE%\texmf\texmf-fonts\ would be a
good option.

But even if you choose the third option above (numbered as two), you
don’t need to set the OSFONTDIR variable.

On Windows and macOS, the OS font directory is set by ConTeXt itself.
The other two directories are also defined.

BTW, where have you copied both fonts?

Just in case it might help,

Pablo
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Henning Hraban Ramm]

Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:
> Dear list,
> I have a very simple question, perhaps too simple...
> I've installed CTX on MSWindows x64, but I can't use the fonts I've 
> installed, despite the command: set 
> OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

That should be ok.

> I cleared the Windows font cache, restarted the machine, then reloaded 
> with the command :
> mtxrun --script fonts --reload

That should be enough. Maybe try
mtxrun --script font --reload --force
I need that regularly for cambria.ttc to work (on OSX).

> A search for the fonts I had installed (theano didot and gfs didot) gave 
> no results with :
> 
> mtxrun --script fonts --list --pattern=*gfs* --all.

The pattern doesn’t need jokers. Did you try without?

> I admit that I don't know in which directory it is best to install the 
> fonts for optimum performance.

If you need the fonts also for other programs, the OS directory is good, 
otherwise <ConTeXt installation directory>/texmf-fonts/fonts/data/ is 
recommended.

Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Wolfgang Schuster]

Henning Hraban Ramm schrieb am 23.09.2023 um 20:05:
> Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:
>> Dear list,
>> I have a very simple question, perhaps too simple...
>> I've installed CTX on MSWindows x64, but I can't use the fonts I've 
>> installed, despite the command: set 
>> OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts. 
>>
>
> That should be ok.

There is no need to set OSFONTDIR on Windows to get system fonts or 
fonts which are placed in the local TeX directory in the home folder.

Wolfgang

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Jean-Pierre Delange]

[-- Attachment #1.1: Type: text/plain, Size: 2124 bytes --]

Dear Hraban,

Even the general command to show all installed fonts :

mtxrun --script fonts --list --all

doesn't show the new fonts. I didn't use mtxrun --script font --reload 
--force

I'll install fonts in /texmf-fonts/fonts/data/ directory as you say.

Is it a good trick to create separate OTF and TTF etc. directories ?

Merci beaucoup !

JP


Le 23/09/2023 à 20:05, Henning Hraban Ramm a écrit :
> Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:
>> Dear list,
>> I have a very simple question, perhaps too simple...
>> I've installed CTX on MSWindows x64, but I can't use the fonts I've 
>> installed, despite the command: set 
>> OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.
>
> That should be ok.
>
>> I cleared the Windows font cache, restarted the machine, then 
>> reloaded with the command :
>> mtxrun --script fonts --reload
>
> That should be enough. Maybe try
> mtxrun --script font --reload --force
> I need that regularly for cambria.ttc to work (on OSX).
>
>> A search for the fonts I had installed (theano didot and gfs didot) 
>> gave no results with :
>>
>> mtxrun --script fonts --list --pattern=*gfs* --all.
>
> The pattern doesn’t need jokers. Did you try without?
>
>> I admit that I don't know in which directory it is best to install 
>> the fonts for optimum performance.
>
> If you need the fonts also for other programs, the OS directory is 
> good, otherwise <ConTeXt installation 
> directory>/texmf-fonts/fonts/data/ is recommended.
>
> Hraban
> ___________________________________________________________________________________ 
>
> If your question is of interest to others as well, please add an entry 
> to the Wiki!
>
> maillist : ntg-context@ntg.nl / 
> https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : https://contextgarden.net
> ___________________________________________________________________________________

-- 
Jean-Pierre Delange
Ancients&Moderns
Professeur Agrégé de Philosophie (HC)

[-- Attachment #1.2: Type: text/html, Size: 3724 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Jean-Pierre Delange]

Thanks to Pablo, Hraban and Wolfgang for their advice.

Precisely, I've never had any problems before with installing additional 
fonts for CTX with MS Windows : it usually "works".

That's why I was wondering... So I'm going to put the extra fonts in 
/texmf-fonts/fonts/data as Hraban advises.

But to make a "set OSFONTDIR=c:\windows\fonts" is allways necessary I 
presume (in order to make CTX able to use windows fonts) ?

Many thanks !

JP


Le 23/09/2023 à 20:15, Wolfgang Schuster a écrit :
> Henning Hraban Ramm schrieb am 23.09.2023 um 20:05:
>> Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:
>>> Dear list,
>>> I have a very simple question, perhaps too simple...
>>> I've installed CTX on MSWindows x64, but I can't use the fonts I've 
>>> installed, despite the command: set 
>>> OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts. 
>>>
>>
>> That should be ok.
>
> There is no need to set OSFONTDIR on Windows to get system fonts or 
> fonts which are placed in the local TeX directory in the home folder.
>
> Wolfgang
>
> ___________________________________________________________________________________ 
>
> If your question is of interest to others as well, please add an entry 
> to the Wiki!
>
> maillist : ntg-context@ntg.nl / 
> https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : https://contextgarden.net
> ___________________________________________________________________________________ 
>

-- 
Jean-Pierre Delange
Ancients&Moderns
Professeur Agrégé de Philosophie (HC)

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Henning Hraban Ramm]

Am 23.09.23 um 20:30 schrieb Jean-Pierre Delange:
> Dear Hraban,
> 
> Even the general command to show all installed fonts :
> 
> mtxrun --script fonts --list --all
> 
> doesn't show the new fonts. I didn't use mtxrun --script font --reload 
> --force
> 
> I'll install fonts in /texmf-fonts/fonts/data/ directory as you say.

Well, I don’t know if it will work if the OS font dir didn’t.

> Is it a good trick to create separate OTF and TTF etc. directories ?

It’s not necessary. In 8bit times it made sense to separate metrics, 
outlines, mappings and stuff, but since OpenType fonts (.ttf or .otf 
doesn’t matter) contain everything, just order them as you like.

According to TDS, a folder below fonts is expected, so Hans suggested 
"data", but you can use ttf, otf or wtf. Below that, you can use more 
folders to keep an overview.

Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Wolfgang Schuster]

Jean-Pierre Delange schrieb am 23.09.2023 um 20:37:
> Thanks to Pablo, Hraban and Wolfgang for their advice.
>
> Precisely, I've never had any problems before with installing 
> additional fonts for CTX with MS Windows : it usually "works".
>
> That's why I was wondering... So I'm going to put the extra fonts in 
> /texmf-fonts/fonts/data as Hraban advises.
>
> But to make a "set OSFONTDIR=c:\windows\fonts" is allways necessary I 
> presume (in order to make CTX able to use windows fonts) ?

The file mtxrun.lua contains the following check which sets default 
paths on Windows and Mac when no value for OSFONTDIR is set.

do
  local osfontdir=osgetenv("OSFONTDIR")
  if osfontdir and osfontdir~="" then
  elseif osname=="windows" then
   ossetenv("OSFONTDIR","c:/windows/fonts//")
  elseif osname=="macosx" then
ossetenv("OSFONTDIR","$HOME/Library/Fonts//;/Library/Fonts//;/System/Library/Fonts//")
  end
end

Wolfgang

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

[Hamid,Idris]

[-- Attachment #1.1: Type: text/plain, Size: 6153 bytes --]


------ Original Message ------
From "Hans Hagen via ntg-context" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
To "ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>" <ntg-context@ntg.nl<mailto:ntg-context@ntg.nl>>
Cc "Hans Hagen" <j.hagen@freedom.nl<mailto:j.hagen@freedom.nl>>
Date 9/22/2023 10:44:38 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

(It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.)

I admit that I don't follow what happens with xetex (they changed the
rendere at some point indeed) not HB (I only notice that it gets updates
frequently in the tex live repository which makes me wonder how one
retains compatility unless one freezes). I actually kept the lib binding
code that can use it around for your font testing (we wanted to see what
uniscribe does), not sure if it still works.
I remember the experimental uniscribe binding, don't recall that it was ever officially released. (Did Kai Bormann ever work with it?).

I hope that it still works or can be resurrected when the time comes for testing -)


Anyway, we're entering the bug cq. side effect becomes feature area
here; just like yesterdays perfect bidi algorithm is todays less pefect
one replaced by ...

But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

if context shows it then it is not a feature but hard coded shapes which
is weird; how does one know what to 'remove' or not? And in what stage?
If they are zero width it is simple to ignore them in the backend, if
they have dimensions (w/h/d) then they contributed and wiping is tricky

The metrics should be dimensionless and most Arabic fonts do that, but Lotus Linotype gives them width (one wonders why).

Perhaps the Lotus designers thought that giving them width makes them easier to edit, which is indeed the case. And perhaps they were satisfied that Word/InDesign would be able, as needed, to toggle the metrics on and off as well as the shape. That WYSIWYG-DTP curse again..

\setcharacterstripping[1] gives the right results with Lotus (at least so far) -- see attached. Was pleasantly surprised that the spurious width was also suppressed.

\setcharacterstripping[0] toggles back -- another pleasant result -)


Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted.

(I suppose that InDesign behaves the same way.)

I don't know ... irr these dtp programs are more like "if you want this
feature applied select a range of characters and apply it"

That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue.

In some way it's also flaws in the open type approach. Basically that
happens when application stuff becomes a standard and one forgets that
it was (is) application driven. (And you haven't seen variable fonts and
color fonts yet ... no pretty standards either.)

I'm in the process of switching from Multiple Masters sources to variable, so will surely encounter it soon.

Unfortunately there is indeed a frustrating gray area in this arena between knowing where the standards end and where the diktats of corporations (or even free-software consortiums like HB) begin, as we've discussed in the past. With variable fonts, one has to add some degree of arbitrariness..


Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

Funny rules ... but I'm not going top enable 'wipe' by default: after
all, one gets what one deserves, nto what one likes (which can differ
per day). But you can enable the wiping. We can of course ignore in the
backend when zero width but then how to explain that they contributed to
the ht/dp (unless we wipe these dimensions) ... all slow-downers
So far \setcharacterstripping[1] is giving the right results. If the relevant trackers are available, could add them to the attached test. But presumably you know how and why it works even for a non-dimensionless glyph.


so you want to see soem zwj sumbol in a rendered text?

Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -)

Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does.

(Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!)

We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.)
so what is it now:

- for verbatim you can use almfixed and they show up (when they have a
glyph)

Except for \nobreakspace I think. At some point it may be worth seeing if there is a way to get the NBSP glyph to show in a typed buffer, or if the color-representation can be configured as some symbol instead of a color. But not urgent.


- for other fonts if they have them they show up (unless gone in the
process ot rendering)
- but you can wipe them optionally

not sure what more we need

For now I think we're good. If anything comes up during the immediate project, will post it.

Many thanks as always Hans.

Best wishes
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

[-- Attachment #1.2: Type: text/html, Size: 11927 bytes --]

[-- Attachment #2: test-characterstripping-lotus.pdf --]
[-- Type: application/pdf, Size: 6761 bytes --]

[-- Attachment #3: test-characterstripping-lotus.tex --]
[-- Type: application/octet-stream, Size: 546 bytes --]


\definefont [LotusRegular] [Lotus-Light.ttf*arabic at 12pt]
\definefont [LotusBold]    [Lotus-Bold.ttf*arabic at 12pt]

\setupdirections[bidi=global,method=one] 
\righttoleft

\usemodule[translate]
\enableinputtranslation
\translateinput[لْحَ][لْ‍حَ]  % ٱلْحَمْدِ
\translateinput[ٱ][ا]

\startTEXpage[offset=1em]
\LotusRegular
الْحَمْدُ

\setcharacterstripping[1]

الْحَمْدُ

\setcharacterstripping[0]

الْحَمْدُ

\setcharacterstripping[1]

الْحَمْدُ
\stopTEXpage

[-- Attachment #4: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Pablo Rodriguez]

On 9/24/23 10:04, Jean-Pierre Delange wrote:
> […]
>  mtxrun --script cache --erase && mtxrun --generate
> […]
> Now, I must confess that I don't know why installing new fonts in
> c:\windows\fonts, even after laboriously purging the Windows font cache
> and reloading the CTX font cache, didn't have the expected effects ...

I had the same problem on Linux, Jean-Pierre.

Before having the script to remove the cache as an option (at least, to
me), I manually deleted the contents of context/tex/temxf-cache/.

Rebuilding font cache hasn’t worked for me (most of the times), so I
came to the conclusion that generating the cache from the start worked
better for me.

The fonts installed in %WINDIR%\Fonts\ are fine. ConTeXt had a problem
adding the paths of these new fonts to its font database.

Just in case it might be clearer now,

Pablo
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Jean-Pierre Delange]

[-- Attachment #1.1: Type: text/plain, Size: 4223 bytes --]

Thanks a lot everybody !

Thank you all for the clarifying information! The problem has been 
solved using the command recommended by Pablo. All my attempts to clear 
the Windows font cache have failed, except for this one:

  mtxrun --script cache --erase && mtxrun --generate

I started by copying the fonts into the \texmf-fontes\data\ directory, 
then I emptied the cache. Then, the following command:

  mtxrun --script fonts --list --pattern=didot --all

works and correctly returns otf and ttf theano and gfs fonts.

And thanks to Andres : your MWE compiles perfectly !

Now, I must confess that I don't know why installing new fonts in 
c:\windows\fonts, even after laboriously purging the Windows font cache 
and reloading the CTX font cache, didn't have the expected effects ...

Le 24/09/2023 à 07:50, Andres Conrado Montoya a écrit :
> I did a test with Windows 10.
>
> Installed a fresh copy from the garden, Got the GFS Didot font from 
> Google Fonts, and installed it system-wide (C:\Windows\fonts). After 
> that, I ran
>
> mtxrun --script fonts --reload
>
> Then tried the pattern suggested but did not work. However, then I 
> tried a pattern without asterisks:
>
> mtxrun --script fonts --list --pattern=didot --all
>
> This returned the font:
>
> C:\griego>mtxrun --script fonts --list --pattern=didot --all
> identifier           familyname    fontname filename                   
>                 subfont instances
>
> gfsdidot             gfsdidot      gfsdidotregular 
>  c:/windows/fonts/GFSDidot-Regular.ttf
> gfsdidotnormal       gfsdidot      gfsdidotregular 
>  c:/windows/fonts/GFSDidot-Regular.ttf
> gfsdidotregular      gfsdidot      gfsdidotregular 
>  c:/windows/fonts/GFSDidot-Regular.ttf
> theanodidot          theanodidot   theanodidotregular 
> c:/windows/fonts/TheanoDidot-Regular.ttf
> theanodidotnormal    theanodidot   theanodidotregular 
> c:/windows/fonts/TheanoDidot-Regular.ttf
> theanodidotregular   theanodidot   theanodidotregular 
> c:/windows/fonts/TheanoDidot-Regular.ttf
>
> trying to render Pablo's example didn't work at first, but after i 
> called the fonts with the name returned by mtxrun:
>
> \mainlanguage[agr]
> \definefontfamily[mainface][rm][gfsdidot]
> \definefontfamily[mainface][ss][theanodidot]
> \setupbodyfont[mainface]
> \starttext
> GFS Didot:
>
> \doloopoverlist{\tf, \it, \bf, \bi}
>   {\recursestring\input aristotle-grc\par}
>
> \ss
> Theano Didot:
>
> \input aristotle-grc
> \stoptext
>
> Everything compiles correctly, and the font is used. Since it's only a 
> single font file with no variants, you don´t get italics, bold etc. 
> But the font is being used.
>
> -- 
> Andrés Conrado Montoya
> Andi Kú
> andresconrado@gmail.com
> http://sesentaycuatro.com
> http://messier87.com
> http://chiquitico.org
> ----------------------------------------
> Los fines no justifican los medios, porque la medida verdadera de 
> nuestro carácter está dada por los medios que estamos dispuestos a 
> utilizar, no por los fines que proclamamos.
> ----------------------------------------
>
> “You develop an instant global consciousness, a people orientation, an 
> intense dissatisfaction with the state of the world, and a compulsion 
> to do something about it. From out there on the moon, international 
> politics look so petty. You want to grab a politician by the scruff of 
> the neck and drag him a quarter of a million miles out and say, ‘Look 
> at that, you son of a bitch.’” — Apollo 14 astronaut Edgar Mitchell
>
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
>
> maillist :ntg-context@ntg.nl  /https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  :https://www.pragma-ade.nl  /http://context.aanhet.net
> archive  :https://bitbucket.org/phg/context-mirror/commits/
> wiki     :https://contextgarden.net
> ___________________________________________________________________________________

-- 
Jean-Pierre Delange
Ancients&Moderns
"Few discoveries are more irritating than those which expose the pedigree of ideas". Lord Acton

[-- Attachment #1.2: Type: text/html, Size: 7980 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Install fonts on MS Windows

[Andres Conrado Montoya]

[-- Attachment #1.1: Type: text/plain, Size: 2586 bytes --]

I did a test with Windows 10.

Installed a fresh copy from the garden, Got the GFS Didot font from Google
Fonts, and installed it system-wide (C:\Windows\fonts). After that, I ran

mtxrun --script fonts --reload

Then tried the pattern suggested but did not work. However, then I tried a
pattern without asterisks:

mtxrun --script fonts --list --pattern=didot --all

This returned the font:

C:\griego>mtxrun --script fonts --list --pattern=didot --all
identifier           familyname    fontname             filename
                        subfont   instances

gfsdidot             gfsdidot      gfsdidotregular
 c:/windows/fonts/GFSDidot-Regular.ttf
gfsdidotnormal       gfsdidot      gfsdidotregular
 c:/windows/fonts/GFSDidot-Regular.ttf
gfsdidotregular      gfsdidot      gfsdidotregular
 c:/windows/fonts/GFSDidot-Regular.ttf
theanodidot          theanodidot   theanodidotregular
c:/windows/fonts/TheanoDidot-Regular.ttf
theanodidotnormal    theanodidot   theanodidotregular
c:/windows/fonts/TheanoDidot-Regular.ttf
theanodidotregular   theanodidot   theanodidotregular
c:/windows/fonts/TheanoDidot-Regular.ttf

trying to render Pablo's example didn't work at first, but after i called
the fonts with the name returned by mtxrun:

\mainlanguage[agr]
\definefontfamily[mainface][rm][gfsdidot]
\definefontfamily[mainface][ss][theanodidot]
\setupbodyfont[mainface]
\starttext
GFS Didot:

\doloopoverlist{\tf, \it, \bf, \bi}
  {\recursestring\input aristotle-grc\par}

\ss
Theano Didot:

\input aristotle-grc
\stoptext

Everything compiles correctly, and the font is used. Since it's only a
single font file with no variants, you don´t get italics, bold etc. But the
font is being used.

-- 
Andrés Conrado Montoya
Andi Kú
andresconrado@gmail.com
http://sesentaycuatro.com
http://messier87.com
http://chiquitico.org
----------------------------------------
Los fines no justifican los medios, porque la medida verdadera de nuestro
carácter está dada por los medios que estamos dispuestos a utilizar, no por
los fines que proclamamos.
----------------------------------------

“You develop an instant global consciousness, a people orientation, an
intense dissatisfaction with the state of the world, and a compulsion to do
something about it. From out there on the moon, international politics look
so petty. You want to grab a politician by the scruff of the neck and drag
him a quarter of a million miles out and say, ‘Look at that, you son of a
bitch.’” — Apollo 14 astronaut Edgar Mitchell

[-- Attachment #1.2: Type: text/html, Size: 3451 bytes --]

[-- Attachment #2: Type: text/plain, Size: 495 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________