Rendering of ZWNJ and ZWJ

Rendering of ZWNJ and ZWJ

[Khaled Hosny]

[-- Attachment #1.1.1: Type: text/plain, Size: 389 bytes --]


Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
font has a glyph for it, it does the expected behaviour (force
non-joining or joining respectively) but the glyph get rendered while it
shouldn't. But if the font has no glyphs for it, it has no effect at
all. See the attached example.

-- 
 Khaled Hosny
 Arabic localizer and member of Arabeyes.org team

[-- Attachment #1.1.2: zwnj_ScheherazadeRegOT.pdf --]
[-- Type: application/pdf, Size: 9367 bytes --]

[-- Attachment #1.1.3: zwnj_ae_AlMohanad.pdf --]
[-- Type: application/pdf, Size: 5918 bytes --]

[-- Attachment #1.1.4: zwnj.tex --]
[-- Type: text/x-tex, Size: 470 bytes --]

% engine=luatex 
\noheaderandfooterlines

\starttext

 \definefontfeature
   [arabic]
   [analyze=yes, clig=yes, curs=yes, dlig=yes,
    fina=yes, init=yes, isol=yes, kern=yes,
    language=dflt, liga=yes, mark=yes, medi=yes,
    mkmk=yes, mode=node, rlig=yes, script=arab]

\font \Arabic = ScheherazadeRegOT*arabic at 24pt
%\font \Arabic = ae_AlMohanad*arabic at 24pt

\pardir TRT \textdir TRT

\Arabic
خ‌ج‌ح
خ‍ ج‍ ح‍
\stoptext 

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Hans Hagen]

Khaled Hosny wrote:
> Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
> font has a glyph for it, it does the expected behaviour (force
> non-joining or joining respectively) but the glyph get rendered while it
> shouldn't. But if the font has no glyphs for it, it has no effect at
> all. See the attached example.

ok, the case when it's not in the font is easy to fix by a slightly 
different test (aroun dline 4500 in font-otf.lua)

local chardata = characters[char]
if chardata and chardata.description.class == "mark" then
     set_attribute(current,state,5) -- mark
     if trace then fcs(current,"font:mark") end
elseif isol[char] then -- can be zwj or zwnj too
     .....

you need to remake the format

the problem of the glyph showing up ... i have to think of an option 
(because i guess that sometimes it's handy for tracing)



-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Hans Hagen]

Khaled Hosny wrote:
> Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
> font has a glyph for it, it does the expected behaviour (force
> non-joining or joining respectively) but the glyph get rendered while it
> shouldn't. But if the font has no glyphs for it, it has no effect at
> all. See the attached example.

i uploaded a beta

after analyzing i now remove the joiners


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Khaled Hosny]

[-- Attachment #1.1: Type: text/plain, Size: 1585 bytes --]

On Mon, Jun 09, 2008 at 05:45:10PM +0200, Hans Hagen wrote:
> Khaled Hosny wrote:
> > Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
> > font has a glyph for it, it does the expected behaviour (force
> > non-joining or joining respectively) but the glyph get rendered while it
> > shouldn't. But if the font has no glyphs for it, it has no effect at
> > all. See the attached example.
> 
> i uploaded a beta
> 
> after analyzing i now remove the joiners

Thanks for the quick response, I'll try it as soon as I can.

Regards,
 Khaled
> 
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>       tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
>                                               | www.pragma-pod.nl
> -----------------------------------------------------------------
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
> archive  : https://foundry.supelec.fr/projects/contextrev/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________

-- 
 Khaled Hosny
 Arabic localizer and member of Arabeyes.org team

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Idris Samawi Hamid]

Hi Hans,

On Mon, 09 Jun 2008 09:45:10 -0600, Hans Hagen <pragma@wxs.nl> wrote:

> Khaled Hosny wrote:
>> Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
>> font has a glyph for it, it does the expected behaviour (force
>> non-joining or joining respectively) but the glyph get rendered while it
>> shouldn't. But if the font has no glyphs for it, it has no effect at
>> all. See the attached example.
>
> i uploaded a beta
>
> after analyzing i now remove the joiners

Not sure if this is the best approach:

Despite what MS Typography says, I think it's important to maintain the  
joiner info after analysis. If I select text from a pdf, the joiner info  
should be there. This is important for, eg, switching from Arabic-language  
to Farsi-language typography. So If I select some pdf text formatted  
according to Farsi rules, I should be able to save that text with the  
rules included.

Can this be made an option, so fonts that contain zwj zwnj are not  
affected?

Best wishes
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Hans Hagen]

Idris Samawi Hamid wrote:
> Can this be made an option, so fonts that contain zwj zwnj are not  
> affected?

ok, the next version will have

otf.remove_joiners = true -- default

later we can make it an option in the font definition




-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Idris Samawi Hamid]

On Tue, 10 Jun 2008 10:45:31 -0600, Hans Hagen <pragma@wxs.nl> wrote:

> Idris Samawi Hamid wrote:
>> Can this be made an option, so fonts that contain zwj zwnj are not
>> affected?
>
> ok, the next version will have
>
> otf.remove_joiners = true -- default

Our main test fonts + professional fonts generally have zwj zwnj so  
actually the default should be

otf.remove_joiners = false

at least until we can

> make it an option in the font definition

Otherwise I have to keep modifying the file everytime I download a beta...

Best wishes
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Hans Hagen]

Idris Samawi Hamid wrote:
> On Tue, 10 Jun 2008 10:45:31 -0600, Hans Hagen <pragma@wxs.nl> wrote:
> 
>> Idris Samawi Hamid wrote:
>>> Can this be made an option, so fonts that contain zwj zwnj are not
>>> affected?
>> ok, the next version will have
>>
>> otf.remove_joiners = true -- default
> 
> Our main test fonts + professional fonts generally have zwj zwnj so  
> actually the default should be

sure, but i need to check in what way they interfere in the lookups later on

> otf.remove_joiners = false
> 
> at least until we can
> 
>> make it an option in the font definition
> 
> Otherwise I have to keep modifying the file everytime I download a beta...

you can put

\ctxlua{otf.remove_joiners = false}

on cont-loc.mkiv or so



-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Khaled Hosny]

[-- Attachment #1.1: Type: text/plain, Size: 1744 bytes --]

On Mon, Jun 09, 2008 at 06:32:13PM -0600, Idris Samawi Hamid wrote:
> Hi Hans,
> 
> On Mon, 09 Jun 2008 09:45:10 -0600, Hans Hagen <pragma@wxs.nl> wrote:
> 
> > Khaled Hosny wrote:
> >> Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
> >> font has a glyph for it, it does the expected behaviour (force
> >> non-joining or joining respectively) but the glyph get rendered while it
> >> shouldn't. But if the font has no glyphs for it, it has no effect at
> >> all. See the attached example.
> >
> > i uploaded a beta
> >
> > after analyzing i now remove the joiners
> 
> Not sure if this is the best approach:
> 
> Despite what MS Typography says, I think it's important to maintain the  
> joiner info after analysis. If I select text from a pdf, the joiner info  
> should be there. This is important for, eg, switching from Arabic-language  
> to Farsi-language typography. So If I select some pdf text formatted  
> according to Farsi rules, I should be able to save that text with the  
> rules included.
> 
> Can this be made an option, so fonts that contain zwj zwnj are not  
> affected?

Well, I've very little ConTeXt knowledge to actually understand this. My
main point was that non-printable formatting characters like ZWNJ, ZWJ
should be font independent i.e. when I insert a ZWNJ it inhibits the
joining of Arabic letters without being printed, the font having or not
having a glyph for it shouldn't affect this. In the other hand, it
should be their so that when I copy the text I get it (since it is part
of the input and has semantical value).

Is this what we have here?

Regards,
 Khaled

-- 
 Khaled Hosny
 Arabic localizer and member of Arabeyes.org team

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Rendering of ZWNJ and ZWJ

[Hans Hagen]

Khaled Hosny wrote:
> On Mon, Jun 09, 2008 at 06:32:13PM -0600, Idris Samawi Hamid wrote:
>> Hi Hans,
>>
>> On Mon, 09 Jun 2008 09:45:10 -0600, Hans Hagen <pragma@wxs.nl> wrote:
>>
>>> Khaled Hosny wrote:
>>>> Using Luatex engine, ZWNJ and ZWJ chars are rendered incorrectly. If the
>>>> font has a glyph for it, it does the expected behaviour (force
>>>> non-joining or joining respectively) but the glyph get rendered while it
>>>> shouldn't. But if the font has no glyphs for it, it has no effect at
>>>> all. See the attached example.
>>> i uploaded a beta
>>>
>>> after analyzing i now remove the joiners
>> Not sure if this is the best approach:
>>
>> Despite what MS Typography says, I think it's important to maintain the  
>> joiner info after analysis. If I select text from a pdf, the joiner info  
>> should be there. This is important for, eg, switching from Arabic-language  
>> to Farsi-language typography. So If I select some pdf text formatted  
>> according to Farsi rules, I should be able to save that text with the  
>> rules included.
>>
>> Can this be made an option, so fonts that contain zwj zwnj are not  
>> affected?
> 
> Well, I've very little ConTeXt knowledge to actually understand this. My
> main point was that non-printable formatting characters like ZWNJ, ZWJ
> should be font independent i.e. when I insert a ZWNJ it inhibits the
> joining of Arabic letters without being printed, the font having or not
> having a glyph for it shouldn't affect this. In the other hand, it
> should be their so that when I copy the text I get it (since it is part
> of the input and has semantical value).
> 
> Is this what we have here?

i made them font independent in the sense that they need not be present 
in the font; however, the rendering is optional


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________