Malayalam shaping with mkiv/LuaTeX

* Malayalam shaping with mkiv/LuaTeX
@ 2014-01-04 21:36 Rajeesh K Nambiar
  2014-01-06 17:03 ` Hans Hagen
  0 siblings, 1 reply; 12+ messages in thread
From: Rajeesh K Nambiar @ 2014-01-04 21:36 UTC (permalink / raw)
  To: mailing list for ConTeXt users

[-- Attachment #1: Type: text/plain, Size: 4037 bytes --]

Hello, especially Hans and Kai,

I have succeeded to an extent in adding Malayalam shaping to the
existing Devanagari shaping. Being a complete novice in Lua and mkiv,
I'm at a point where your expertise is needed to proceed further.
Please find the attached patch, test case listed below and expected
rendering for comparison. Additionally needed to support:

1.  Decomposing 2 part Matras (dependent vowels) - 0D4A, 0D4B, 0D4C
for Malayalam [Test cases 2 & 4]. This is needed (at least) for Tamil
also. I guess it is as simple as replacing one code point with 2
others - example: replace 0D4A with 0D46, 0D3E. See page 3 of this
Unicode standard: http://www.unicode.org/charts/PDF/U0D00.pdf

2. Handling "pref"  opentype feature for v2 opentype spec (script tag
"mlm2"). This is required for Malayalam Ra ligature, the glyph is
formed correctly but it needs to be reordered as pre-base to the base
syllable [Test case 1]. (Note that the old spec "mlym" handles "Ra
form" as "pstf" and no reordering required). You can observe the
difference in Test case 5 - Ra form should be attached to first
consonant instead of second one ("mlm2" needs glyph formed by
Virama+Ra to be reordered as pre-base to first consonant, while "mlym"
doesn't need to).

3. Handling "pstf" opentype feature. It works mostly [Test case 8],
except when a post-base Matra follows [Test case 9].

4. Test cases 6 & 7 shows an example where glyph formation fails when
a "pstf" Ya follows. This works correctly with some other conjuncts,
though.

4. font-odv.lua hard coded Devanagari Virama (094D) and Ra (0930) in
certain places, which needs to use table values to handle other
scripts.

5. I had to change "methods.deva" to "methods.mlym" and "methods.dev2"
to "methods.mlm2" to get this working, not sure how best to handle
multiple scripts.

6. There are other cases like Dot-reph (0D4E) which uses GPOS table
and requires reordering as post-base to the syllable follows it, but I
haven't looked into them yet.


i) The font used is Rachana, which supports both "mlm2" and "mlym"
specs, can be downloaded from here:
http://download.savannah.gnu.org/releases/smc/fonts/malayalam-fonts-6.0/Rachana/Rachana.ttf
You can switch to "mlm2" and "mlym" in test case.
ii) Expected rendering of this test case:
http://rajeeshknambiar.fedorapeople.org/ConTeXt/context-test.png
iii) Test case:

----------- 8< ---------
\usemodule[font-odv]

\definefontfeature[malayalam][mode=node,language=dflt,script=mlm2,akhn=yes,blwf=yes,half=yes,pres=yes,blws=yes,psts=yes,haln=no]
%\definefontfeature[malayalam][mode=node,language=dflt,script=mlym,akhn=yes,blwf=yes,half=yes,pres=yes,blws=yes,psts=yes,haln=no]

\starttypescript [serif] [rachana] [name]
   \definefontsynonym[Serif] [file:rachana][features={malayalam}]
\stoptypescript

\starttypescript [rachana]
   \definetypeface[rachana][rm][serif][rachana][default]
   \definetypeface[rachana][mm][math] [modern][default]
\stoptypescript

\setupbodyfont[rachana]

\starttext

1.  \char"0D15 \char"0D4D \char"0D24 \char"0D4D \char"0D30 \\ % ക്ത്ര
2.  \char"0D15 \char"0D4D \char"0D24 \char"0D4D \char"0D30 \char"0D4B
\\ % ക്ത്രോ
3.  \char"0D38 \char"0D4D \char"0D2A \char"0D4D \char"0D30 \\ % സ്പ്ര
4.  \char"0D38 \char"0D4D \char"0D2A \char"0D4D \char"0D30 \char"0D4C
\\ % സ്പ്രൌ
5.  \char"0D24 \char"0D4D \char"0D30 \char"0D2A \\ % ത്രപ
6.  \char"0D24 \char"0D4D \char"0D38 \\ % ത്സ
7.  \char"0D24 \char"0D4D \char"0D38 \char"0D4D \char"0D2F \\ % ത്സ്യ
8.  \char"0D15 \char"0D4D \char"0D2F \\ % ക്യ
9.  \char"0D15 \char"0D4D \char"0D2F \char"0D42 \\ % ക്യൂ

\stoptext
----------- >8 ---------

Let me know if you need additional information or clarifications on test cases.
[Patch, test case, expected output image etc are also available here:
http://rajeeshknambiar.fedorapeople.org/ConTeXt/]

-- 
Cheers,
Rajeesh
http://rajeeshknambiar.wordpress.com

[-- Attachment #2: malayalam-shaping-luatex.patch --]
[-- Type: text/x-patch, Size: 11413 bytes --]

--- tex/texmf-context/tex/context/base/font-odv.lua	2013-10-01 12:53:06.000000000 +0200
+++ tex/texmf-context/tex/context/base/font-oin.lua	2014-01-04 19:38:52.874015300 +0100
@@ -1,7 +1,7 @@
-if not modules then modules = { } end modules ['font-odv'] = {
+if not modules then modules = { } end modules ['font-odv'] = {
     version   = 1.001,
-    comment   = "companion to font-ini.mkiv",
-    author    = "Kai Eigner, TAT Zetwerk / Hans Hagen, PRAGMA ADE",
+    comment   = "Indic shaping companion to font-ini.mkiv",
+    author    = "Kai Eigner, TAT Zetwerk / Hans Hagen, PRAGMA ADE / Rajeesh K Nambiar",
     copyright = "TAT Zetwerk / PRAGMA ADE / ConTeXt Development Team",
     license   = "see context related readme files"
 }
@@ -182,6 +182,17 @@ local consonant = {
     [0x0CB5] = true, [0x0CB6] = true, [0x0CB7] = true, [0x0CB8] = true,
     [0x0CB9] = true,
     [0x0CDE] = true, -- obsolete
+    -- malayalam
+    [0x0D15] = true, [0x0D16] = true, [0x0D17] = true, [0x0D18] = true,
+    [0x0D19] = true, [0x0D1A] = true, [0x0D1B] = true, [0x0D1C] = true,
+    [0x0D1D] = true, [0x0D1E] = true, [0x0D1F] = true, [0x0D20] = true,
+    [0x0D21] = true, [0x0D22] = true, [0x0D23] = true, [0x0D24] = true,
+    [0x0D25] = true, [0x0D26] = true, [0x0D27] = true, [0x0D28] = true,
+    [0x0D29] = true, [0x0D2A] = true, [0x0D2B] = true, [0x0D2C] = true,
+    [0x0D2D] = true, [0x0D2E] = true, [0x0D2F] = true, [0x0D30] = true,
+    [0x0D31] = true, [0x0D32] = true, [0x0D33] = true, [0x0D34] = true,
+    [0x0D35] = true, [0x0D36] = true, [0x0D37] = true, [0x0D38] = true,
+    [0x0D39] = true, [0x0D3A] = true,
 }
 
 local independent_vowel = {
@@ -198,6 +209,11 @@ local independent_vowel = {
     [0x0C89] = true, [0x0C8A] = true, [0x0C8B] = true, [0x0C8C] = true,
     [0x0C8D] = true, [0x0C8E] = true, [0x0C8F] = true, [0x0C90] = true,
     [0x0C91] = true, [0x0C92] = true, [0x0C93] = true, [0x0C94] = true,
+    -- malayalam
+    [0x0D05] = true, [0x0D06] = true, [0x0D07] = true, [0x0D08] = true,
+    [0x0D09] = true, [0x0D0A] = true, [0x0D0B] = true, [0x0D0C] = true,
+    [0x0D0E] = true, [0x0D0F] = true, [0x0D10] = true, [0x0D12] = true,
+    [0x0D13] = true, [0x0D14] = true,
 }
 
 local dependent_vowel = { -- matra
@@ -213,6 +229,11 @@ local dependent_vowel = { -- matra
     [0x0CC2] = true, [0x0CC3] = true, [0x0CC4] = true, [0x0CC5] = true,
     [0x0CC6] = true, [0x0CC7] = true, [0x0CC8] = true, [0x0CC9] = true,
     [0x0CCA] = true, [0x0CCB] = true, [0x0CCC] = true,
+    -- malayalam
+    [0x0D3E] = true, [0x0D3F] = true, [0x0D40] = true, [0x0D41] = true,
+    [0x0D42] = true, [0x0D43] = true, [0x0D44] = true, [0x0D46] = true,
+    [0x0D47] = true, [0x0D48] = true, [0x0D4A] = true, [0x0D4B] = true,
+    [0x0D4C] = true, [0x0D57] = true,
 }
 
 local vowel_modifier = {
@@ -224,6 +245,8 @@ local vowel_modifier = {
     [0xA8E8] = true, [0xA8E9] = true, [0xA8EA] = true, [0xA8EB] = true,
     [0xA8EC] = true, [0xA8ED] = true, [0xA8EE] = true, [0xA8EF] = true,
     [0xA8F0] = true, [0xA8F1] = true,
+    -- malayalam
+    [0x0D02] = true, [0x0D03] = true,
 }
 
 local stress_tone_mark = {
@@ -242,9 +265,19 @@ local halant = {
     [0x094D] = true,
     -- kannada
     [0x0CCD] = true,
+    -- malayalam
+    [0x0D4D] = true,
+}
+
+local char_ra = {
+    -- devanagari
+    [0x0930] = true,
+    -- kannada
+    [0x0CB0] = true,
+    -- malayalam
+    [0x0D30] = true,
 }
 
-local c_ra       = 0x0930 -- used to be tables (also used as constant)
 local c_anudatta = 0x0952 -- used to be tables
 local c_nbsp     = 0x00A0 -- used to be tables
 local c_zwnj     = 0x200C -- used to be tables
@@ -269,10 +302,14 @@ local zw_char = { -- could also be inlin
 -- OCE6 - OCEF digits
 
 local pre_mark = {
+    -- devanagari
     [0x093F] = true, [0x094E] = true,
+    -- malayalam
+    [0x0D46] = true, [0x0D47] = true, [0x0D48] = true,
 }
 
 local above_mark = {
+    -- devanagari
     [0x0900] = true, [0x0901] = true, [0x0902] = true, [0x093A] = true,
     [0x0945] = true, [0x0946] = true, [0x0947] = true, [0x0948] = true,
     [0x0951] = true, [0x0953] = true, [0x0954] = true, [0x0955] = true,
@@ -281,6 +318,8 @@ local above_mark = {
     [0xA8E8] = true, [0xA8E9] = true, [0xA8EA] = true, [0xA8EB] = true,
     [0xA8EC] = true, [0xA8ED] = true, [0xA8EE] = true, [0xA8EF] = true,
     [0xA8F0] = true, [0xA8F1] = true,
+    -- malayalam
+    [0x0D4E] = true,
 }
 
 local below_mark = {
@@ -295,6 +334,13 @@ local post_mark = {
     [0x094F] = true,
 }
 
+local twopart_mark = {
+    -- malayalam
+    [0x0D4A] = { 0x0D46, 0x0D3E, },	-- ൊ
+    [0x0D4B] = { 0x0D47, 0x0D3E, },	-- ോ
+    [0x0D4C] = { 0x0D46, 0x0D57, },	-- ൌ
+}
+
 local mark_four = { } -- As we access these frequently an extra hash is used.
 
 for k, v in next, pre_mark   do mark_four[k] = pre_mark   end
@@ -426,7 +472,7 @@ local basic_shaping_forms =  {
 
 local function initializedevanagi(tfmdata)
     local script, language = otf.scriptandlanguage(tfmdata,attr) -- todo: take fast variant
-    if script == "deva" or script == "dev2" then
+    if script == "deva" or script == "dev2" or script =="mlym" or script == "mlm2" then
         local resources  = tfmdata.resources
         local lookuphash = resources.lookuphash
         if not lookuphash["dv01"] then
@@ -473,6 +519,16 @@ local function initializedevanagi(tfmdat
                 sharedfeatures["dv04"] = true -- dv04_remove_joiners
             end
             --
+	    if script == "mlym" or script == "mlm2" then
+		sharedfeatures["pstf"] = true
+	    end
+	    if script == "mlm2" then
+		sharedfeatures["pref"] = true
+		sharedfeatures["dv03"] = true -- dv03_reorder_pre_base_reordering_consonants
+		gsubfeatures["dv03"] = dev2_defaults -- reorder pre base reordering consonants
+		insert(sequences,insertindex,sequence_reorder_pre_base_reordering_consonants)
+	    end
+	    --
         end
     end
 end
@@ -560,7 +616,7 @@ local function deva_reorder(head,start,s
     local lastcons  = nil
     local basefound = false
 
-    if start.char == c_ra and halant[n.char] and reph then
+    if char_ra[start.char] and halant[n.char] and reph then
         -- if syllable starts with Ra + H and script has 'Reph' then exclude Reph
         -- from candidates for base consonants
         if n == stop then
@@ -686,7 +742,7 @@ local function deva_reorder(head,start,s
 
     n = start.next
  -- if start.char == c_ra and halant[n.char] and not (n ~= stop and zw_char[n.next.char]) then
-    if n ~= stop and start.char == c_ra and halant[n.char] and not zw_char[n.next.char] then
+    if n ~= stop and char_ra[start.char] and halant[n.char] and not zw_char[n.next.char] then
         -- if syllable starts with Ra + H then move this combination so that it follows either:
         -- the post-base 'matra' (if any) or the base consonant
         local matra = base
@@ -817,7 +873,7 @@ local function deva_reorder(head,start,s
         while current ~= stop do
             local c = current
             local n = current.next
-            if current.char == c_ra and halant[n.char] then
+            if char_ra[current.char] and halant[n.char] then
                 c = n
                 n = n.next
                 local b, bn = base, base
@@ -1160,11 +1216,15 @@ function handlers.devanagari_remove_join
 end
 
 local valid = {
+    akhn = true,
     rphf = true,
     pref = true,
     half = true,
     blwf = true,
     pstf = true,
+    pres = true,
+    blws = true,
+    psts = true,
 }
 
 local function dev2_initialize(font,attr)
@@ -1208,9 +1268,9 @@ local function dev2_initialize(font,attr
                             if chain ~= 0 then --rphf is result of of chain
                                 --ToDo: rphf might be result of other handler/chainproc
                             else
-                                reph = lookupcache[0x0930]
+                                reph = lookupcache[0x0930]	--TODO: Ra for other scripts
                                 if reph then
-                                    reph = reph[0x094D]
+                                    reph = reph[0x094D]		--TODO: Halant for other scripts
                                     if reph then
                                         reph = reph["ligature"]
                                     end
@@ -1259,9 +1319,9 @@ local function dev2_reorder(head,start,s
             -- todo: rphf might be result of other handler/chainproc
             -- todo: rphf actualy acts on consonant + halant.
             -- todo: the consonant might not necesseraly be 0x0930 ... (but for devanagari it is)
-            local lookup = lookupcache[0x0930]
+            local lookup = lookupcache[0x0930]	--TODO: Ra for other scripts
             if lookup then
-                local hit = lookup[0x094D]
+                local hit = lookup[0x094D]	--TODO: Halant for other scripts
                 if hit then
                     reph = hit["ligature"]
                 end
@@ -1295,7 +1355,10 @@ local function dev2_reorder(head,start,s
             -- why not global? pretty ineffient this way
             -- this will move to the initializer and we will store the hash in dataset
             -- todo: reph might also be result of chain
-            for k, v in lookupcache[0x094D], next do
+            -- for k, v in lookupcache[0x094D], next do		   --TODO: Ra for other scripts
+            --     pre_base_reordering_consonants[k] = v and v["ligature"]    --ToDo: reph might also be result of chain
+            -- end
+	    for k, v in pairs(lookupcache[0x0D4D]) do			   --TODO: Ra for other scripts
                 pre_base_reordering_consonants[k] = v and v["ligature"]    --ToDo: reph might also be result of chain
             end
             --
@@ -1943,7 +2006,7 @@ end
 -- It looks like these two analyzers were written independently but they share
 -- a lot. Common code has been synced.
 
-function methods.deva(head,font,attr)
+function methods.mlym(head,font,attr)
     local current  = head
     local start    = true
     local done     = false
@@ -1955,7 +2018,7 @@ function methods.deva(head,font,attr)
             local syllableend = nil
             local c = current
             local n = c.next
-            if n and c.char == c_ra and n.id == glyph_code and halant[n.char] and n.subtype<256 and n.font == font then
+            if n and char_ra[c.char] and n.id == glyph_code and halant[n.char] and n.subtype<256 and n.font == font then
                 local n = n.next
                 if n and n.id == glyph_code and n.subtype<256 and n.font == font then
                     c = n
@@ -2141,7 +2204,7 @@ end
 
 -- handler(head,start,kind,lookupname,lookupmatch,sequence,lookuphash,1)
 
-function methods.dev2(head,font,attr)
+function methods.mlm2(head,font,attr)
     local current  = head
     local start    = true
     local done     = false
@@ -2154,7 +2217,7 @@ function methods.dev2(head,font,attr)
             syllablestart = current
             local c = current
             local n = current.next
-            if n and c.char == c_ra and n.id == glyph_code and halant[n.char] and n.subtype<256 and n.font == font then
+            if n and char_ra[c.char] and n.id == glyph_code and halant[n.char] and n.subtype<256 and n.font == font then
                 local n = n.next
                 if n and n.id == glyph_code and n.subtype<256 and n.font == font then
                     c = n

[-- Attachment #3: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread