devanagari rendering (2 errors)

devanagari rendering (2 errors)

[Peter Graif]

Hello,

I recently learned that ConTeXt-mkiv and LuaTeX now support devanagari fonts. An enormous thank you to Kai and Hans and anyone else involved in making that happen. I know it's a niche application, but for some of us it makes all the difference.

I ran some tests on the output and the rendering is lovely, even on the extremely complex ligatures. I noticed only two errors, both of which are simple to describe and consistent:

1) Conjuncts of the form (consonant + virama + र "ra") render the opposite sequence intended. For example:
[U+0915] [U+094D][U+0930] (क्र "kra") incorrectly produces the glyph intended for [U+0930][U+094D][U+0915] (र्क "rka").

2) Word-final virama causes words to render with a half-consonant at the end. They should instead display the whole-consonant with the virama diacritic. Half-consonants should be used only when conjoining with another subsequent consonant in the same word.

A picture, in case it's more helpful: http://i.imgur.com/MpIwd9w.png

I tried to look at the code in hopes of offering a fix, but I'm afraid it was way over my head. If there's anything else I can do to help, I'd be happy for the opportunity.

Best regards,
Peter
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: devanagari rendering (2 errors)

[Hans Hagen]

On 9/9/2013 10:05 PM, Peter Graif wrote:
> Hello,
>
> I recently learned that ConTeXt-mkiv and LuaTeX now support devanagari fonts. An enormous thank you to Kai and Hans and anyone else involved in making that happen. I know it's a niche application, but for some of us it makes all the difference.
>
> I ran some tests on the output and the rendering is lovely, even on the extremely complex ligatures. I noticed only two errors, both of which are simple to describe and consistent:
>
> 1) Conjuncts of the form (consonant + virama + र "ra") render the opposite sequence intended. For example:
> [U+0915] [U+094D][U+0930] (क्र "kra") incorrectly produces the glyph intended for [U+0930][U+094D][U+0915] (र्क "rka").
>
> 2) Word-final virama causes words to render with a half-consonant at the end. They should instead display the whole-consonant with the virama diacritic. Half-consonants should be used only when conjoining with another subsequent consonant in the same word.
>
> A picture, in case it's more helpful: http://i.imgur.com/MpIwd9w.png
>
> I tried to look at the code in hopes of offering a fix, but I'm afraid it was way over my head. If there's anything else I can do to help, I'd be happy for the opportunity.

Can you make a as-small-as-possible test file?

Preferable with proper utf-8 sequences (+ comment mentioning 
numbers/classes), good and wrong.

I can probably figure it out but I cannot read or input easily this kind 
of scripts.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: devanagari rendering (2 errors)

[Peter Graif]

> 
> Can you make a as-small-as-possible test file?
> 
> Preferable with proper utf-8 sequences (+ comment mentioning numbers/classes), good and wrong.
> 
> I can probably figure it out but I cannot read or input easily this kind of scripts.
> 
> Hans
> 

Sure thing. I don't understand how opentype features are represented nor how the ligature logic is dividd up between the font and the engine, so I'm not quite sure what information is helpful or relevant. But, here's a minimal document that contains all the problem sequences I've identified.

As a rough guide, here's another image with the current ConTeXt rendering in black and the desired rendering in red:
http://i.imgur.com/MyvMneg.png

Is that the information you need? Or is there something else I can do?
-Peter

-----------------

\definefont [DevaOne] [file:chandas.ttf*devanagari-one at 16pt]
\starttext
\DevaOne

% The following combinations render incorrectly.
% They are all of the following form:
% {consonant(s)} + U+094D + U+0930
\char"0915 \char"094D \char"0930 ~
\char"0916 \char"094D \char"0930 ~
\char"0917 \char"094D \char"0930 ~
\char"0918 \char"094D \char"0930 ~
\char"0919 \char"094D \char"0930 ~ 
\char"091A \char"094D \char"0930 ~ 
\char"091B \char"094D \char"0930 ~ 
\char"091C \char"094D \char"0930 ~ 
\char"091D \char"094D \char"0930 ~ 
\char"091E \char"094D \char"0930 ~\crlf  
\char"091F \char"094D \char"0930 ~ 
\char"0920 \char"094D \char"0930 ~
\char"0921 \char"094D \char"0930 ~
\char"0922 \char"094D \char"0930 ~
\char"0923 \char"094D \char"0930 ~
\char"0924 \char"094D \char"0930 ~ 
\char"0925 \char"094D \char"0930 ~
\char"0926 \char"094D \char"0930 ~
\char"0927 \char"094D \char"0930 ~
\char"0928 \char"094D \char"0930 ~\crlf 
\char"0929 \char"094D \char"0930 ~ 
\char"092A \char"094D \char"0930 ~ 
\char"092B \char"094D \char"0930 ~ 
\char"092C \char"094D \char"0930 ~ 
\char"092D \char"094D \char"0930 ~ 
\char"092E \char"094D \char"0930 ~ 
\char"092F \char"094D \char"0930 ~ 
\char"0930 \char"094D \char"0930 ~
\char"0931 \char"094D \char"0930 ~
\char"0932 \char"094D \char"0930 ~\crlf 
\char"0933 \char"094D \char"0930 ~
\char"0934 \char"094D \char"0930 ~
\char"0935 \char"094D \char"0930 ~ 
\char"0936 \char"094D \char"0930 ~
\char"0937 \char"094D \char"0930 ~
\char"0938 \char"094D \char"0930 ~
\char"0939 \char"094D \char"0930 ~ 

\char"0958 \char"094D \char"0930 ~
\char"0959 \char"094D \char"0930 ~
\char"095A \char"094D \char"0930 ~
\char"095B \char"094D \char"0930 ~
\char"095C \char"094D \char"0930 ~
\char"095D \char"094D \char"0930 ~
\char"095E \char"094D \char"0930 ~
\char"095F \char"094D \char"0930 ~ 

\crlf \crlf \crlf

% The following cominations also render incorrectly.
% The problem occurs at the end of words with sequences:
% {consonant(s)} + U+094D + [end of word]
\char"0915 \char"094D ~
\char"0916 \char"094D ~
\char"0917 \char"094D ~
\char"0918 \char"094D ~
\char"0919 \char"094D ~
\char"091A \char"094D ~
\char"091B \char"094D ~
\char"091C \char"094D ~
\char"091D \char"094D ~
\char"091E \char"094D ~\crlf
\char"091F \char"094D ~
\char"0920 \char"094D ~
\char"0921 \char"094D ~
\char"0922 \char"094D ~
\char"0923 \char"094D ~
\char"0924 \char"094D ~ 
\char"0925 \char"094D ~
\char"0926 \char"094D ~
\char"0927 \char"094D ~
\char"0928 \char"094D ~\crlf
\char"0929 \char"094D ~ 
\char"0929 \char"094D ~ 
\char"092A \char"094D ~ 
\char"092C \char"094D ~ 
\char"092D \char"094D ~ 
\char"092E \char"094D ~ 
\char"092F \char"094D ~ 
\char"0930 \char"094D ~
\char"0931 \char"094D ~
\char"0932 \char"094D ~\crlf
\char"0933 \char"094D ~
\char"0934 \char"094D ~
\char"0935 \char"094D ~ 
\char"0936 \char"094D ~
\char"0937 \char"094D ~
\char"0938 \char"094D ~
\char"0939 \char"094D ~ 

\char"0958 \char"094D ~
\char"0959 \char"094D ~
\char"095A \char"094D ~
\char"095B \char"094D ~
\char"095C \char"094D ~
\char"095D \char"094D ~
\char"095E \char"094D ~
\char"095F \char"094D ~ 


\stoptext
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: devanagari rendering (2 errors)

[Hans Hagen]

On 9/10/2013 6:43 AM, Peter Graif wrote:
>
>>
>> Can you make a as-small-as-possible test file?
>>
>> Preferable with proper utf-8 sequences (+ comment mentioning numbers/classes), good and wrong.
>>
>> I can probably figure it out but I cannot read or input easily this kind of scripts.
>>
>> Hans
>>
>
> Sure thing. I don't understand how opentype features are represented nor how the ligature logic is dividd up between the font and the engine, so I'm not quite sure what information is helpful or relevant. But, here's a minimal document that contains all the problem sequences I've identified.
>
> As a rough guide, here's another image with the current ConTeXt rendering in black and the desired rendering in red:
> http://i.imgur.com/MyvMneg.png
>
> Is that the information you need? Or is there something else I can do?

it's ok

it took me quite a while to notice that it's a side effect of two 
features that are turned on

you can play with

\definefontfeature
   [devanagari-one]
   [devanagari-one]
   [rphf=no]

and

\definefontfeature
   [devanagari-one]
   [devanagari-one]
   [half=no]

maybe Kai can have a look at it: as these are dealt with in a different 
we probably don't need to enable them in the feature processor (even 
block them)

btw, you can play with

\showotfcomposition{file:chandas.ttf*devanagari-one}{l2r}{\char"0915 
\char"094D \char"0930}

to see what happens

(i also noticed a few possible further optimizations but that's for later)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: devanagari rendering (2 errors)

[Peter Graif]

> it took me quite a while to notice that it's a side effect of two features that are turned on
> 
> you can play with
> 
> \definefontfeature
>  [devanagari-one]
>  [devanagari-one]
>  [rphf=no]
> 
> and
> 
> \definefontfeature
>  [devanagari-one]
>  [devanagari-one]
>  [half=no]
> 
> maybe Kai can have a look at it: as these are dealt with in a different we probably don't need to enable them in the feature processor (even block them)
> 
> btw, you can play with
> 
> \showotfcomposition{file:chandas.ttf*devanagari-one}{l2r}{\char"0915 \char"094D \char"0930}
> 
> to see what happens
> 
> (i also noticed a few possible further optimizations but that's for later)
> 
> Hans

Thanks, Hans.

Those are indeed the relevant features. Unfortunately, turning them off completely breaks other things. I have attached another demonstration below. The good news is that we can now produce proper devanagari by switching the features on and off as needed in the middle of words. I am understanding OpenType better now and I will try again to look at the relevant source code tonight.

-Peter
--------------------------
\definefontfeature[devyes][devanagari-one][rphf=yes,half=yes]
\definefontfeature[devno][devanagari-one][rphf=no,half=no]

\definefont [DevaWith] [file:chandas.ttf*devyes at 14pt]
\definefont [DevaWithout] [file:chandas.ttf*devno at 14pt]

\starttext

{\bf When rphf=yes (current default)}

r-first conjucts ([r]+[half]+\{consonant\}) work:

{\DevaWith [U+0930 \char"0930][U+094D \char"094D][U+0915 \char"0915]: \char"0930 \char"094D \char"0915 ~(correct)}

but r-second conjucts (\{consonant\} +[half]+[r]) don't:

{\DevaWith [U+0915 \char"0915][U+094D \char"094D][U+0930 \char"0930]: \char"0915 \char"094D \char"0930 ~(incorrect)}

\blank[big]

{\bf When rphf=no}

r-first conjucts ([r]+[half]+\{consonant\}) stop working:

{\DevaWithout [U+0930 \char"0930][U+094D \char"094D][U+0915 \char"0915]: \char"0930 \char"094D \char"0915 ~(incorrect)}

but r-second conjucts (\{consonant\} +[half]+[r]) now work properly:

{\DevaWithout [U+0915 \char"0915][U+094D \char"094D][U+0930 \char"0930]: \char"0915 \char"094D \char"0930 ~(correct)}


\blank[3*big]


{\bf When half=yes (current default)}

consonant clusters (\{consonant\}+[half]+\{consonant\}) work:

{\DevaWith [U+0915 \char"0915][U+094D \char"094D][U+0915 \char"0915]: \char"0915 \char"094D \char"0915 ~(correct)}

but word-final viramas (\{consonant\} +[half]) don't:

{\DevaWith [U+0915 \char"0915][U+094D \char"094D] : \char"0915 \char"094D ~(incorrect)}

\blank[big]

{\bf When half=no}

consonant clusters (\{consonant\}+[half]+\{consonant\}) stop working:

{\DevaWithout [U+0915 \char"0915][U+094D \char"094D][U+0915 \char"0915]: \char"0915 \char"094D \char"0915 ~(incorrect)}

but word-final viramas (\{consonant\} +[half]) work properly:

{\DevaWithout [U+0915 \char"0915][U+094D \char"094D] : \char"0915 \char"094D ~(correct)}


\blank[2*big]

With careful bracketing, it's possible to produce correct devanagari.

On: {\DevaWith \char"092E\char"0915\char"094D\char"0915\char"092E\char"0915\char"094D} (no good)

Off: {\DevaWithout \char"092E\char"0915\char"094D\char"0915\char"092E\char"0915\char"094D} (no good)

Mixed: {\DevaWith \char"092E\char"0915\char"094D\char"0915\char"092E{\DevaWithout \char"0915\char"094D}} (perfect)

That's not sustainable for long texts, perhaps, but it's a great move in that direction.

\blank[big]

Note: I've used the consonant [U+0915: {\DevaWith \char"0915}] here throughout, but you can substitute any consonant \{U+0915--U+0939, U+0958--U+095F\} and see the same patterns.

\stoptext
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: devanagari rendering (2 errors)

[Rajeesh K Nambiar]

Hi Hans,

On Tue, Sep 10, 2013 at 2:04 PM, Hans Hagen <pragma@wxs.nl> wrote:

>
> it took me quite a while to notice that it's a side effect of two features
> that are turned on
>
> you can play with
>
> \definefontfeature
>   [devanagari-one]
>   [devanagari-one]
>   [rphf=no]
>
> and
>
> \definefontfeature
>   [devanagari-one]
>   [devanagari-one]
>   [half=no]
>
> maybe Kai can have a look at it: as these are dealt with in a different we
> probably don't need to enable them in the feature processor (even block
> them)
>

As far as I understand, 'rphf' and 'half' forms are essential for
Devanagari, so skipping them completely might not be a good idea...

> btw, you can play with
>
> \showotfcomposition{file:chandas.ttf*devanagari-one}{l2r}{\char"0915
> \char"094D \char"0930}
>


-- 
Cheers,
Rajeesh
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________