improving kannada script

improving kannada script

[Ulrike Fischer]

According to the xelatex output in the following example both
variants are not correct. What would be needed would be the first
two glyphs from variant 1 (knd2 script) and the third from variant 2
(knda script).

Pictures comparing the output can be found at
https://github.com/u-fischer/luaotfload/issues/20

(Disclaimer: I have no idea if xelatex/harfbuzz is correct here, but 
https://tex.stackexchange.com/a/457427/2388 says so).

\definefontfeature
  [kannada-testA]
  [indic-common]
  [script=knd2,
   % basic shaping
   pref=yes,
   blwf=yes,
   half=yes,
   pstf=yes,
   cjct=yes
   ]
\definefontfeature
  [kannada-testB] %identical to default kannada-one
  [indic-common]
  [script=knda,
   % basic shaping
   pref=yes,
   blwf=yes,
   half=yes,
   pstf=yes,
   cjct=yes
   ]   

\definefont
[kannadatestA][file:notosanskannada-regular.ttf*kannada-testA]
\definefont
[kannadatestB][file:notosanskannada-regular.ttf*kannada-testB]
\starttext 
\section{Skript knd2}
\kannadatestA
ಕಿ

ಕ್ರ

ರ್ಕ

\section{Skript knda (default)}

\kannadatestB
ಕಿ

ಕ್ರ

ರ್ಕ
\stoptext




-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 584 bytes --]

On Tue, Oct 30, 2018 at 10:18 AM Ulrike Fischer <news3@nililand.de> wrote:

> According to the xelatex output in the following example both
> variants are not correct. What would be needed would be the first
> two glyphs from variant 1 (knd2 script) and the third from variant 2
> (knda script).
>
>
dunno, but to start with at  least I visually see what's going on:

\showotfcomposition{file:notosanskannada-regular.ttf*kannada-testA at
48pt}{-1}{ಕ್ರ}


\showotfcomposition{file:notosanskannada-regular.ttf*kannada-testB at
48pt}{-1}{ಕ್ರ}


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 1032 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 974 bytes --]

On Tue, Oct 30, 2018 at 11:22 AM luigi scarso <luigi.scarso@gmail.com>
wrote:

>
>
> On Tue, Oct 30, 2018 at 10:18 AM Ulrike Fischer <news3@nililand.de> wrote:
>
>> According to the xelatex output in the following example both
>> variants are not correct. What would be needed would be the first
>> two glyphs from variant 1 (knd2 script) and the third from variant 2
>> (knda script).
>>
>>
> dunno, but to start with at  least I visually see what's going on:
>
> \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testA at
> 48pt}{-1}{ಕ್ರ}
>
>
> \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testB at
> 48pt}{-1}{ಕ್ರ}
>
> support for knd2 is still in progress,
and  in knda
glyph ಕ (U+00C95) +  glyph ್ (U+00CCD) + glyph ರ (U+00CB0)
also is not ok ( at least for what I understand until now).
I see where it fails ( and where hb does it right), let's see if I am able
to find a patch.


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 1797 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[Hans Hagen]

On 10/31/2018 3:38 PM, luigi scarso wrote:

> On Tue, Oct 30, 2018 at 11:22 AM luigi scarso <luigi.scarso@gmail.com 
> <mailto:luigi.scarso@gmail.com>> wrote:
> 
> 
> 
>     On Tue, Oct 30, 2018 at 10:18 AM Ulrike Fischer <news3@nililand.de
>     <mailto:news3@nililand.de>> wrote:
> 
>         According to the xelatex output in the following example both
>         variants are not correct. What would be needed would be the first
>         two glyphs from variant 1 (knd2 script) and the third from variant 2
>         (knda script).
> 
> 
>     dunno, but to start with at  least I visually see what's going on:
> 
>     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testA
>     at 48pt}{-1}{ಕ್ರ}
> 
> 
>     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testB
>     at 48pt}{-1}{ಕ್ರ}
> 
> support for knd2 is still in progress,
> and  in knda
> glyph ಕ (U+00C95) +  glyph ್ (U+00CCD) + glyph ರ (U+00CB0)
> also is not ok ( at least for what I understand until now).
> I see where it fails ( and where hb does it right), let's see if I am 
> able to find a patch.

fyi, there will be an update for some set set of indic script later this 
year (most is done but it all relates to available time of those involved)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --]

On Wed, Oct 31, 2018 at 4:01 PM Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 10/31/2018 3:38 PM, luigi scarso wrote:
>
> > On Tue, Oct 30, 2018 at 11:22 AM luigi scarso <luigi.scarso@gmail.com
> > <mailto:luigi.scarso@gmail.com>> wrote:
> >
> >
> >
> >     On Tue, Oct 30, 2018 at 10:18 AM Ulrike Fischer <news3@nililand.de
> >     <mailto:news3@nililand.de>> wrote:
> >
> >         According to the xelatex output in the following example both
> >         variants are not correct. What would be needed would be the first
> >         two glyphs from variant 1 (knd2 script) and the third from
> variant 2
> >         (knda script).
> >
> >
> >     dunno, but to start with at  least I visually see what's going on:
> >
> >     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testA
> >     at 48pt}{-1}{ಕ್ರ}
> >
> >
> >     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testB
> >     at 48pt}{-1}{ಕ್ರ}
> >
> > support for knd2 is still in progress,
> > and  in knda
> > glyph ಕ (U+00C95) +  glyph ್ (U+00CCD) + glyph ರ (U+00CB0)
> > also is not ok ( at least for what I understand until now).
> > I see where it fails ( and where hb does it right), let's see if I am
> > able to find a patch.
>
> fyi, there will be an update for some set set of indic script later this
> year (most is done but it all relates to available time of those involved)
>
yes,
it's more a matter to see the details under the hood

-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 2340 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[Ulrike Fischer]

Am Wed, 31 Oct 2018 16:01:37 +0100 schrieb Hans Hagen:


> fyi, there will be an update for some set set of indic script later this 
> year (most is done but it all relates to available time of those involved)

That's good news. I got quite a list of probably broken scripts:

https://tex.stackexchange.com/a/457459/2388

(But I couldn't check yet for most of them, if it is a fontloader or
a luaotfload problem and if the newest version solves some of the
problems, deva e.g. should be already better).


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 905 bytes --]

On Fri, Nov 2, 2018 at 11:28 AM Ulrike Fischer <news3@nililand.de> wrote:

> Am Wed, 31 Oct 2018 16:01:37 +0100 schrieb Hans Hagen:
>
>
> > fyi, there will be an update for some set set of indic script later this
> > year (most is done but it all relates to available time of those
> involved)
>
> That's good news. I got quite a list of probably broken scripts:
>
> https://tex.stackexchange.com/a/457459/2388
>
> (But I couldn't check yet for most of them, if it is a fontloader or
> a luaotfload problem and if the newest version solves some of the
> problems, deva e.g. should be already better).
>
> good , thank you.
I am planning to play with a harfbuzz plugin to  check them (sort of tool)
,
with the ongoing stability of the lua api in luatex  it makes sense a
luaffi binding
but luaffi is not stable (yet)
(ok, due  lack of time could be that I use the filter module and the hb
exe).

-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 1455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[Ulrike Fischer]

Am Fri, 2 Nov 2018 11:56:52 +0100 schrieb luigi scarso:

> I am planning to play with a harfbuzz plugin to  check them (sort of tool)

You could try https://github.com/khaledhosny/luahbtex. Khaled is
trying to marry luatex + harfbuzz there (I couldn't try it yet due
to missing windows binaries). 

-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 734 bytes --]

On Fri, Nov 2, 2018 at 12:26 PM Ulrike Fischer <news3@nililand.de> wrote:

> Am Fri, 2 Nov 2018 11:56:52 +0100 schrieb luigi scarso:
>
> > I am planning to play with a harfbuzz plugin to  check them (sort of
> tool)
>
> You could try https://github.com/khaledhosny/luahbtex. Khaled is
> trying to marry luatex + harfbuzz there


sure, it's hardcoded , the final binary is quite heavy ...
Anyway, I am talking of dll/so  plugin and tool,
it's  more  on the track of
https://www.guitex.org/home/images/ArsTeXnica/AT023/luaffi-article.pdf
It's always nice and useful to have an "orthogonal way " to do things, more
chances to fix bugs
and customizations. And at some point,  learn by doing (for me) is the only
way to learn.


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 1516 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[Ulrike Fischer]

Am Fri, 2 Nov 2018 12:53:02 +0100 schrieb luigi scarso:


>> You could try https://github.com/khaledhosny/luahbtex. Khaled is
>> trying to marry luatex + harfbuzz there

> sure, it's hardcoded , the final binary is quite heavy ...
> Anyway, I am talking of dll/so  plugin and tool,
> it's  more  on the track of
> https://www.guitex.org/home/images/ArsTeXnica/AT023/luaffi-article.pdf

Well we (some of the latex team) tried to test this as we have quite
an interest to have an option to use harfbuzz for some fonts but not
every code needed seems to be available publicitly. 

Also if I remember correctly this requires --shell-escape and this
is not really sensible for a user interface. 


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[Aditya Mahajan]

On Fri, 2 Nov 2018, Ulrike Fischer wrote:

> Also if I remember correctly this requires --shell-escape and this
> is not really sensible for a user interface.

IIRC, it is possible to 'whitelist' programs in texmf.cnf so that 
`--shell-escape` is not needed. I believe this is how the automatic eps to 
pdf conversion in latex works.

Aditya
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 2520 bytes --]

On Fri, Nov 2, 2018 at 1:05 PM Ulrike Fischer <news3@nililand.de> wrote:

> Am Fri, 2 Nov 2018 12:53:02 +0100 schrieb luigi scarso:
>
>
> >> You could try https://github.com/khaledhosny/luahbtex. Khaled is
> >> trying to marry luatex + harfbuzz there
>
> > sure, it's hardcoded , the final binary is quite heavy ...
> > Anyway, I am talking of dll/so  plugin and tool,
> > it's  more  on the track of
> > https://www.guitex.org/home/images/ArsTeXnica/AT023/luaffi-article.pdf
>
> Well we (some of the latex team) tried to test this as we have quite
> an interest to have an option to use harfbuzz for some fonts but not
> every code needed seems to be available publicitly.
>
> it's complex thing...  a binding depends on lua API and
the target lib API .. At that time the lua API was still in flux
(so a swiglib binding is not so useful) and the luaffi interface
non tested (just see the apart on the FFT) .
A (stable) luaffi could resolve the luatex side of  the binding
and being lua code is more easy to adapt if the  target API changes.

Also if I remember correctly this requires --shell-escape and this
> is not really sensible for a user interface.
>

 it's the context ml,   enable system commands is default (and we are still
here :-)  . Anyway ConteXt has a sandbox too. )

But yes, the
--safer                       disable easily exploitable lua commands
   --[no-]shell-escape           disable/enable system commands
   --shell-restricted            restrict system commands to a list of
commands given in texmf.cnf
are part of the picture. Hans and I  have to discuss this point.

Just to say: on my linux box, xetex from the official deb package has not
hb hardcoded:
# ldd `which  xetex `
linux-vdso.so.1 (0x00007ffcb26d2000)
libharfbuzz-icu.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz-icu.so.0
(0x00007f5fe89c5000)
libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0
(0x00007f5fe8727000)
Even if I set all the paranoia flags, xetex will  load these shared objects.
With a luaffi things doesn't change, the point is that luatex will load the
libs only when/if  the user (script) will  to do.
Of course, the xetex from texlive is statically compiled ... well almost. I
still see freetype as shared object.
Anyway, as I have said I am now focused on this issue now because I would
like to complete/fix it for the next texlive.

(to be honest: I think that all these safer shell-* switches are a bit
outdated nowadays, but they are there and I don't think they will disappear
. )


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 3838 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: improving kannada script

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 3519 bytes --]

On Wed, Oct 31, 2018 at 4:03 PM luigi scarso <luigi.scarso@gmail.com> wrote:

>
>
> On Wed, Oct 31, 2018 at 4:01 PM Hans Hagen <j.hagen@xs4all.nl> wrote:
>
>> On 10/31/2018 3:38 PM, luigi scarso wrote:
>>
>> > On Tue, Oct 30, 2018 at 11:22 AM luigi scarso <luigi.scarso@gmail.com
>> > <mailto:luigi.scarso@gmail.com>> wrote:
>> >
>> >
>> >
>> >     On Tue, Oct 30, 2018 at 10:18 AM Ulrike Fischer <news3@nililand.de
>> >     <mailto:news3@nililand.de>> wrote:
>> >
>> >         According to the xelatex output in the following example both
>> >         variants are not correct. What would be needed would be the
>> first
>> >         two glyphs from variant 1 (knd2 script) and the third from
>> variant 2
>> >         (knda script).
>> >
>> >
>> >     dunno, but to start with at  least I visually see what's going on:
>> >
>> >     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testA
>> >     at 48pt}{-1}{ಕ್ರ}
>> >
>> >
>> >     \showotfcomposition{file:notosanskannada-regular.ttf*kannada-testB
>> >     at 48pt}{-1}{ಕ್ರ}
>> >
>> > support for knd2 is still in progress,
>> > and  in knda
>> > glyph ಕ (U+00C95) +  glyph ್ (U+00CCD) + glyph ರ (U+00CB0)
>> > also is not ok ( at least for what I understand until now).
>> > I see where it fails ( and where hb does it right), let's see if I am
>> > able to find a patch.
>>
>
Well..not so easy (at least for me , never seen devanagari before )

glyph ಕ (U+00C95) +  glyph ್ (U+00CCD) + glyph ರ (U+00CB0)
are
KA +  VIRAMA + RA
and it is quite complex case (well,  really it could be a moderately
complex case ...)
For Kannada script we  can start by considering
 http://brahmi.sourceforge.net/docs/KannadaComputing.html
it's an old doc but
1) it shows the rules for kannada
2) its says that
"""
2.5 Rendering Rules
(Based on Microsoft Uniscribe-OpenType implementation of the UNICODE
Rendering Rules)
"""
So Uniscribe is the reference for  the rendering rules of this doc, which
in turn is the base for unicode.

Today the reference  unicode is
http://www.unicode.org/versions/Unicode11.0.0/ch12.pdf
The rules are a bit different from
http://brahmi.sourceforge.net/docs/KannadaComputing.html
but  at page 500 we see
U+0C95 ಕ  ka + U+0CCD  ್  halant + U+0CB0 ರ ra →  ಕ್ರ kra
which is our case (unicode says that the preferred name for  VIRAMA is
halant, so halant is also frequently used )
What happens is explained at page 464 for devanagari:
we should consider KA +  VIRAMA + RA as
KAn +  VIRAMAn + RAi, so

R1
We have KAn + VIRAMAn , hence
KAn + VIRAMAn → KAd

R6:
We have KAd + RAi, hence
KAd + RAi → KAn + RAsub

and then
R13
KAn + RAsub → K.RAn
and indeed ಕ್ರ is K.RAn

All these rules are re-explained at
https://docs.microsoft.com/en-us/typography/script-development/kannada#introduction
and font-osd.lua  is based  exactly  on these rules.
It makes sense, given that, as seen,  these are the reference.
The functions  for knda are those ending  in *_one  and for what I have
seen
after that the base syllable is correctly isolated,  reorder_one doesn't
implement R6 and R13 .
I am quite sure that spending a bit more time I can isolate the point,
but, as Hans has said, there is the plan to review these scripts.

-- 
luigi

PS:
Take these notes  cum grano salis: devanagari is not exactly equal to
 kannada,
and the ms site uses its own terms, not always one finds a match with
unicode.

[-- Attachment #1.2: Type: text/html, Size: 5854 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________