Re: Chinese

Re: Chinese

[Duncan Hothersall]

Hans wrote:

> chinese is not yet defined in utf so if you want that, we need to do it
...
> assuming this, how about making a set of tfm,enc,map files that match 
> the unicode positions (volunteers ...)

I'm very willing to help, especially if there is some drudge work
involved in constructing the files. I don't know enough (yet) about the
logic of it all to help with setting up the system, but if someone can
supply skeleton files and/or a method for constructing the necessary
files, I'm happy to do any leg-work.

Duncan

Re: Re: Chinese

[Hans Hagen]

Duncan Hothersall wrote:

>Hans wrote:
>
>  
>
>>chinese is not yet defined in utf so if you want that, we need to do it
>>    
>>
>...
>  
>
>>assuming this, how about making a set of tfm,enc,map files that match 
>>the unicode positions (volunteers ...)
>>    
>>
>
>I'm very willing to help, especially if there is some drudge work
>involved in constructing the files. I don't know enough (yet) about the
>logic of it all to help with setting up the system, but if someone can
>supply skeleton files and/or a method for constructing the necessary
>files, I'm happy to do any leg-work.
>  
>
what we need is a set of encoding files like

/UniEncoding52 [
....
/uni52DF
/uni52E0
/uni52E1
/uni52E2
/uni52E3
/uni52E4
...
/.notdef
....
] def

that represent the ranges and can be used to construct tfm files.

(or whatever index entry is needed in order to filter the metrics from 
the ttf file)

maybe patricks font code already can do that:

- read in a ttf file (or a glyph list produced by ttf2tfm or ttf2afm)
- make a range of enc and tfm files

actually, this is rather generic, since pdftex can handle symbolic names 
like /index... and /uni..., so if we have such a set, we can stick to 
one bunch of enc files

the utf handler can then simply access char E1 from htsong-52.tfm

testing is rather simple:

\pdfmapline{htsong-52 <uni-52.enc <htsong.ttf}

\font\test=htsong-52 \char"e1


Hans

Hans

Re: Re: Chinese

[sjoerd siebinga]

On 13 Dec 2005, at 10:52, Hans Hagen wrote:

> Duncan Hothersall wrote:
>
>> Hans wrote:
>>
>>
>>> chinese is not yet defined in utf so if you want that, we need to  
>>> do it
>>>
>> ...
>>
>>> assuming this, how about making a set of tfm,enc,map files that  
>>> match the unicode positions (volunteers ...)
>>>
>>
>> I'm very willing to help, especially if there is some drudge work
>> involved in constructing the files. I don't know enough (yet)  
>> about the
>> logic of it all to help with setting up the system, but if someone  
>> can
>> supply skeleton files and/or a method for constructing the necessary
>> files, I'm happy to do any leg-work.
>>
> what we need is a set of encoding files like
>
> /UniEncoding52 [
> ....
> /uni52DF
> /uni52E0
> /uni52E1
> /uni52E2
> /uni52E3
> /uni52E4
> ...
> /.notdef
> ....
> ] def

I have made a Ruby-script (for personal use loosely based on Adam's  
xsl-files) which generates all the encoding- and symbolfiles from a  
given cmapfile. If someone could send me the ttf-font, I can generate  
all the necessary encodingfiles for you.

Sjoerd

Re: Re: Chinese

[Hans Hagen]

sjoerd siebinga wrote:

> I have made a Ruby-script (for personal use loosely based on Adam's  
> xsl-files) which generates all the encoding- and symbolfiles from a  
> given cmapfile. If someone could send me the ttf-font, I can generate  
> all the necessary encodingfiles for you.

the chinese fonts mentioned in the context garden qualify for such a 
treatment (htsong cum suis)

Hans

Re: Re: Chinese

[Tobias Burnus]

Hi,

sjoerd siebinga wrote:
> I have made a Ruby-script (for personal use loosely based on Adam's 
> xsl-files) which generates all the encoding- and symbolfiles from a 
> given cmapfile. If someone could send me the ttf-font, I can generate 
> all the necessary encodingfiles for you.
Nice! The recommended (by Xiao Jianfeng) TrueType fonts are given at 
http://wiki.contextgarden.net/Chinese
They are
ftp://ftp.ctex.org/pub/tex/fonts/truetype/ttf/htfs.ttf
ftp://ftp.ctex.org/pub/tex/fonts/truetype/ttf/hthei.ttf
ftp://ftp.ctex.org/pub/tex/fonts/truetype/ttf/htkai.ttf
ftp://ftp.ctex.org/pub/tex/fonts/truetype/ttf/htsong.ttf


Richard Gabriel wrote:
> But yet another question: What about Japanese? I've made only small 
> research so far, but unlike Chinese, there's almost no information 
> about Japanese in TeX. How much of work would be to adjust the current 
> "chinese" ConTeXt module for Japanese? What would you need for it?
> [Of course, meanwhile I'll investigate some other ways of typesetting 
> Japanese...]
(I don't know much about Japanese.)

In Japanese contrary to Chinese they mix different character sets:
- The Chinese characters ("Kanji"), which seem to make up most of the 
(scientific) text (I'v seen);
in addition some pronouncation based characters are used:
- ("Kana":) Hiragana and Katagana; the former are rather round 
characters in Japanese texts, most prominent should be "の" [means 
something like "of" in English]. They are mostly used for 
suffixes/prefixes where no Chinese equivalent exists. Whereas Katagana 
is used to write words which have been taken from (mostly) European 
languages.

For Kanji there should be no problem with the Chinese module, for Kana 
you need additional support for these characters. Since they are 
pronouncation based, they only consisted of < 50 Characters each.

Tobias

(Hmm, I never though I would end up such deep in linguistics duing my 
PhD theses in physics. But having three Chinese in the group and doing 
regularily some measurements at a research centre in Taiwan - I couldn't 
help picking up something.)

Re: Re: Chinese

[Hans Hagen]

Tobias Burnus wrote:

> (Hmm, I never though I would end up such deep in linguistics duing my 
> PhD theses in physics. But having three Chinese in the group and doing 
> regularily some measurements at a research centre in Taiwan - I 
> couldn't help picking up something.)


well, there is a certain charm in those characters, even if you cannot 
read them (during a 2*10 hour trip in a chinese bus during the last tug 
conference one quickly learns to recognize the symbols for gas stations 
and such -)

browsing a chinese-english dictionary is also fun (i have a small one on 
my desk; some day i should start collecting dictionaries of all 
languages that context supports -); with a bit of puzzling one can find 
out the system behind the way words are made up

Hans

Re: Re: Chinese

[sjoerd siebinga]

On 13 Dec 2005, at 11:34, Hans Hagen wrote:

> sjoerd siebinga wrote:
>
>> I have made a Ruby-script (for personal use loosely based on  
>> Adam's  xsl-files) which generates all the encoding- and  
>> symbolfiles from a  given cmapfile. If someone could send me the  
>> ttf-font, I can generate  all the necessary encodingfiles for you.
>
> the chinese fonts mentioned in the context garden qualify for such  
> a treatment (htsong cum suis)
>

Ok. Where can I send the chinese encodingfiles?

Re: Re: Chinese

[Adam Lindsay]

Hans Hagen wrote:

> what we need is a set of encoding files like
> 
> /UniEncoding52 [
> ....
> /uni52DF
> /uni52E0

I hate to be negative, but I have doubts about how generic this approach 
may be. In some tentative experiments, I discovered that many (most?) 
CJK fonts don't use traditional postscript names, but rather map from 
unicode to an indexed glyph number.

Fortunately, ttf2tfm's -w enco@Unicode@ notation seems to address this 
in most of the old test cases I tried.

adam
-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  Adam T. Lindsay, Computing Dept.     atl@comp.lancs.ac.uk
  Lancaster University, InfoLab21        +44(0)1524/510.514
  Lancaster, LA1 4WA, UK             Fax:+44(0)1524/510.492
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

your mails at go

[Hans Hagen]

sjoerd siebinga wrote:

>
> On 13 Dec 2005, at 11:34, Hans Hagen wrote:
>
>> sjoerd siebinga wrote:
>>
>>> I have made a Ruby-script (for personal use loosely based on  
>>> Adam's  xsl-files) which generates all the encoding- and  
>>> symbolfiles from a  given cmapfile. If someone could send me the  
>>> ttf-font, I can generate  all the necessary encodingfiles for you.
>>
>>
>> the chinese fonts mentioned in the context garden qualify for such  a 
>> treatment (htsong cum suis)
>>
>
> Ok. Where can I send the chinese encodingfiles?

you can send me a zip

maybe we should start thinking on how to set up a repository at

  https://foundry.supelec.fr/

taco and patrick have more experience in this area than i have so maybe 
they have some ideas on how to organize this

Hans

Re: Re: Chinese

[Hans Hagen]

Adam Lindsay wrote:

> Hans Hagen wrote:
>
>> what we need is a set of encoding files like
>>
>> /UniEncoding52 [
>> ....
>> /uni52DF
>> /uni52E0
>
>
> I hate to be negative, but I have doubts about how generic this 
> approach may be. In some tentative experiments, I discovered that many 
> (most?) CJK fonts don't use traditional postscript names, but rather 
> map from unicode to an indexed glyph number.
>
> Fortunately, ttf2tfm's -w enco@Unicode@ notation seems to address this 
> in most of the old test cases I tried.

afaik pdftex can handle the indexXXXX and unicXXXX entries as 
alternatives for glyphnames

Hans

Re: Re: Chinese

[Adam Lindsay]

Hans Hagen wrote:
> Adam Lindsay wrote:
>> Fortunately, ttf2tfm's -w enco@Unicode@ notation seems to address this 
>> in most of the old test cases I tried.
> 
> 
> afaik pdftex can handle the indexXXXX and unicXXXX entries as 
> alternatives for glyphnames

Yes. Sorry I wasn't clear on that.
It's just that ttf2tfm is the tool that does a good job at extracting 
those entries when other tools fail.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  Adam T. Lindsay, Computing Dept.     atl@comp.lancs.ac.uk
  Lancaster University, InfoLab21        +44(0)1524/510.514
  Lancaster, LA1 4WA, UK             Fax:+44(0)1524/510.492
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: Chinese

[Hans Hagen]

Tobias Burnus wrote:

> Hi,
>
> Hans Hagen wrote:
>
>> chinese is not yet defined in utf so if you want that, we need to do it
>
> We probably should.
>
>> question: do the unicode tables cover gbk and big 5 well?
>
> There exists a one-to-one correspondence between GBK and Unicode [1], 
> for Big5 there are 7 characters which cannot be mapped one-to-one (see 
> comment at top of [2]); thee of which appear twice in Big5 but only 
> once in Unicode, two are not in unicode and there are two mapping 
> problems.
> In practive one can thus say: Both GBK and Big5 can be mapped to Unicode.

so, if we can make things utf deep down and remap gbk and big 5 to utf, 
we can do with one set of fonts, metrics etc

i got chinese working in utf now, but need an enco table for teh special 
cases (see enco-chi); when someone made me that table, i can upload an 
alpha

Hans

Re: Chinese

[Xiao Jianfeng]

Richard Gabriel wrote:

> Hello Hans,
>
> to be honest: I don't speak Chinese and don't know much about it.
> A few days ago, I was told that we'll let translate some of our 
> documents (XML) into Chinese and Japanese and I 'll have to typeset them.
> So I started playing with Chinese in ConTeXt. I've reported the 
> results which other users (e.g. Tobias) have also noticed.
> In fact, all the sample Simplified Chinese documents I've tested it on 
> were easily convertible to CP936 (GBK) and could be typeset. This 
> doesn't mean that you shall not extend the Unicode support, I only 
> think I will not hardly require it... :-)
>
> But yet another question: What about Japanese? I've made only small 
> research so far, but unlike Chinese, there's almost no information 
> about Japanese in TeX. How much of work would be to adjust the current 
> "chinese" ConTeXt module for Japanese? What would you need for it?
> [Of course, meanwhile I'll investigate some other ways of typesetting 
> Japanese...]
>
> Thanks,
> Richard
>
 As far as I know, LaTeX coupled with CJK package can process Chinese, 
Japanese, Korea.

 Regards,

xiaojf

>
>
>     ------------------------------------------------------------------------
>     *From:* Hans Hagen [mailto:pragma@wxs.nl]
>     *To:* mailing list for ConTeXt users [mailto:ntg-context@ntg.nl]
>     *Sent:* Mon, 12 Dec 2005 16:53:51 +0100
>     *Subject:* Re: [NTG-context] Chinese
>
>     Richard Gabriel wrote:
>
>     > Hi guys,
>     >
>     > I can confirm that the UTF-8 input doesn't work for me too.
>     > If I convert the file info GBK (CP936), it works fine [I suggest to
>     > use the 'iconv' utility for the conversion :-)].
>     >
>     > I tested the UTF-8 output the followin ways:
>     >
>     > 1)
>     > \enableregime[utf]
>     > \usemodule[chinese]
>     >
>
>     chinese is not yet defined in utf so if you want that, we need to
>     do it
>
>     now, since the chinese remapping stuff is rather complex, the best
>     method is to consider a dedicated mechanism
>
>     question: do the unicode tables cover gbk and big 5 well?
>
>     assuming this, how about making a set of tfm,enc,map files that match
>     the unicode positions (volunteers ...)
>
>     we can extend the utf handler with a kind of plugin mechanism:
>
>     \unprotect
>
>     \def\utfunihashglyph#1%
>     {\@EA\doutfunihashglyph\@EA{\number\utfdiv{#1}}{#1}} % only div once
>
>     \def\doutfunihashglyph#1#2% div raw
>     {\csname
>     \ifnum#2<\utf@i <mailto:utf@i>
>     \strippedcsname\unicodeasciicharacter
>     \else\ifcsname\@@unicommand#1\endcsname
>     \@@unicommand#1%
>     \else\ifcsname\@@univector#1\endcsname
>     \@@univector#1%
>     \else
>     \strippedcsname\unicodeunknowncharacter
>     \fi\fi\fi
>     \@EA\endcsname\@EA{\number\utfmod{#2}}} % only mod once
>
>     \def\unicodeunknowncharacter#1%
>     {\unknownchar}
>
>     \let\utfunihash\utfunihashglyph
>
>     \def\@@unicommand{@@unicommand}
>
>     \def\defineutfcommand #1 #2%
>     {\setvalue{\@@unicommand#1}##1{#2{#1}{##1}}}
>
>     so we can define pluig in handlers for e.g. chinese
>
>     \defineutfcommand 81 {\uchar}
>
>     (bombs due to missing fonts, so for testing)
>
>     \def\NotYet#1#2{[#1 #2]}
>
>     \defineutfcommand 81 {\NotYet}
>
>     (next comes adapting the chinese files; i can imagine that we redo
>     the
>     big5 and gbk definitions so that they remap to ut8 as common encoding)
>
>     so .. the question is ... who is going to make the tfm/enc/map files
>
>     Hans
>
>     _______________________________________________
>     ntg-context mailing list
>     ntg-context@ntg.nl <mailto:ntg-context@ntg.nl>
>     http://www.ntg.nl/mailman/listinfo/ntg-context
>
>------------------------------------------------------------------------
>
>_______________________________________________
>ntg-context mailing list
>ntg-context@ntg.nl
>http://www.ntg.nl/mailman/listinfo/ntg-context
>  
>

Re: Chinese

[Hans Hagen]

Richard Gabriel wrote:

> Hello Hans,
>
> to be honest: I don't speak Chinese and don't know much about it.
> A few days ago, I was told that we'll let translate some of our 
> documents (XML) into Chinese and Japanese and I 'll have to typeset them.
> So I started playing with Chinese in ConTeXt. I've reported the 
> results which other users (e.g. Tobias) have also noticed.
> In fact, all the sample Simplified Chinese documents I've tested it on 
> were easily convertible to CP936 (GBK) and could be typeset. This 
> doesn't mean that you shall not extend the Unicode support, I only 
> think I will not hardly require it... :-)
>
> But yet another question: What about Japanese? I've made only small 
> research so far, but unlike Chinese, there's almost no information 
> about Japanese in TeX. How much of work would be to adjust the current 
> "chinese" ConTeXt module for Japanese? What would you need for it?
> [Of course, meanwhile I'll investigate some other ways of typesetting 
> Japanese...]

it's all a matter of fonts and specs; handling those languages is not 
that complex

the main complication is in the encodings (related to the input) and 
utf8 makes sense as common encoding; context (newtexexec) has provisions 
for runtime recoding

what do the chinese users think of it ...

Hans

Re: Chinese

[Richard Gabriel]

[-- Attachment #1.1: Type: text/plain, Size: 3228 bytes --]

Hello Hans,

to be honest: I don't speak Chinese and don't know much about it.
A few days ago, I was told that we'll let translate some of our documents (XML) into Chinese and Japanese and I 'll have to typeset them. 
So I started playing with Chinese in ConTeXt. I've reported the results which other users (e.g. Tobias) have also noticed. 
In fact, all the sample Simplified Chinese documents I've tested it on were easily convertible to CP936 (GBK) and could be typeset. This doesn't mean that you shall not extend the Unicode support, I only think I will not hardly require it... :-)

But yet another question: What about Japanese? I've made only small research so far, but unlike Chinese, there's almost no information about Japanese in TeX. How much of work would be to adjust the current "chinese" ConTeXt module for Japanese? What would you need for it?
[Of course, meanwhile I'll investigate some other ways of typesetting Japanese...]

Thanks,
Richard



  _____  

From: Hans Hagen [mailto:pragma@wxs.nl]
To: mailing list for ConTeXt users [mailto:ntg-context@ntg.nl]
Sent: Mon, 12 Dec 2005 16:53:51 +0100
Subject: Re: [NTG-context] Chinese

Richard Gabriel wrote:

> Hi guys,
>
> I can confirm that the UTF-8 input doesn't work for me too.
> If I convert the file info GBK (CP936), it works fine [I suggest to 
> use the 'iconv' utility for the conversion :-)].
>
> I tested the UTF-8 output the followin ways:
>
> 1)
> \enableregime[utf]
> \usemodule[chinese]
>

chinese is not yet defined in utf so if you want that, we need to do it

now, since the chinese remapping stuff is rather complex, the best 
method is to consider a dedicated mechanism

question: do the unicode tables cover gbk and big 5 well?

assuming this, how about making a set of tfm,enc,map files that match 
the unicode positions (volunteers ...)

we can extend the utf handler with a kind of plugin mechanism:

\unprotect

\def\utfunihashglyph#1%
  {\@EA\doutfunihashglyph\@EA{\number\utfdiv{#1}}{#1}} % only div once

\def\doutfunihashglyph#1#2% div raw
  {\csname
     \ifnum#2<\utf@i
       \strippedcsname\unicodeasciicharacter
     \else\ifcsname\@@unicommand#1\endcsname
       \@@unicommand#1%
     \else\ifcsname\@@univector#1\endcsname
       \@@univector#1%
     \else
       \strippedcsname\unicodeunknowncharacter
     \fi\fi\fi
     \@EA\endcsname\@EA{\number\utfmod{#2}}} % only mod once

\def\unicodeunknowncharacter#1%
  {\unknownchar}

\let\utfunihash\utfunihashglyph

\def\@@unicommand{@@unicommand}

\def\defineutfcommand #1 #2%
  {\setvalue{\@@unicommand#1}##1{#2{#1}{##1}}}

so we can define pluig in handlers for e.g. chinese

\defineutfcommand 81 {\uchar}

(bombs due to missing fonts, so for testing)

\def\NotYet#1#2{[#1 #2]}

\defineutfcommand 81 {\NotYet}

(next comes adapting the chinese files; i can imagine that we redo the 
big5 and gbk definitions so that they remap to ut8 as common encoding)

so .. the question is ... who is going to make the tfm/enc/map files

Hans

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context
  

[-- Attachment #1.2: Type: text/html, Size: 4029 bytes --]

[-- Attachment #2: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: Chinese

[Tobias Burnus]

Hi,

Hans Hagen wrote:
> chinese is not yet defined in utf so if you want that, we need to do it
We probably should.

> question: do the unicode tables cover gbk and big 5 well?
There exists a one-to-one correspondence between GBK and Unicode [1], 
for Big5 there are 7 characters which cannot be mapped one-to-one (see 
comment at top of [2]); thee of which appear twice in Big5 but only once 
in Unicode, two are not in unicode and there are two mapping problems.
In practive one can thus say: Both GBK and Big5 can be mapped to Unicode.

> assuming this, how about making a set of tfm,enc,map files that match 
> the unicode positions (volunteers ...)
If I get the time, I will play around with the four fonts mentioned on 
http://wiki.contextgarden.net/Chinese
Adam mentioned 
http://www.physik.fu-berlin.de/~tburnus/linlibertine/LibertineUnicode.zip, 
which contains LibertineInConTeXt.txt, which describes a possible way to 
do so.
But I wouldn't mind if someone who has more experience in playing around 
with .enc files would do the job ;)

Tobias

Re: Chinese

[Hans Hagen]

Richard Gabriel wrote:

> Hi guys,
>
> I can confirm that the UTF-8 input doesn't work for me too.
> If I convert the file info GBK (CP936), it works fine [I suggest to 
> use the 'iconv' utility for the conversion :-)].
>
> I tested the UTF-8 output the followin ways:
>
> 1)
> \enableregime[utf]
> \usemodule[chinese]
>

chinese is not yet defined in utf so if you want that, we need to do it

now, since the chinese remapping stuff is rather complex, the best 
method is to consider a dedicated mechanism

question: do the unicode tables cover gbk and big 5 well?

assuming this, how about making a set of tfm,enc,map files that match 
the unicode positions (volunteers ...)

we can extend the utf handler with a kind of plugin mechanism:

\unprotect

\def\utfunihashglyph#1%
  {\@EA\doutfunihashglyph\@EA{\number\utfdiv{#1}}{#1}} % only div once

\def\doutfunihashglyph#1#2% div raw
  {\csname
     \ifnum#2<\utf@i
       \strippedcsname\unicodeasciicharacter
     \else\ifcsname\@@unicommand#1\endcsname
       \@@unicommand#1%
     \else\ifcsname\@@univector#1\endcsname
       \@@univector#1%
     \else
       \strippedcsname\unicodeunknowncharacter
     \fi\fi\fi
     \@EA\endcsname\@EA{\number\utfmod{#2}}} % only mod once

\def\unicodeunknowncharacter#1%
  {\unknownchar}

\let\utfunihash\utfunihashglyph

\def\@@unicommand{@@unicommand}

\def\defineutfcommand #1 #2%
  {\setvalue{\@@unicommand#1}##1{#2{#1}{##1}}}

so we can define pluig in handlers for e.g. chinese

\defineutfcommand 81 {\uchar}

(bombs due to missing fonts, so for testing)

\def\NotYet#1#2{[#1 #2]}

\defineutfcommand 81 {\NotYet}

(next comes adapting the chinese files; i can imagine that we redo the 
big5 and gbk definitions so that they remap to ut8 as common encoding)

so .. the question is ... who is going to make the tfm/enc/map files

Hans

Re: Chinese

[Hans Hagen]

Tobias Burnus wrote:

> Hi,
>
> Adam Lindsay wrote:
>
>> Is there less kerning among CJK fonts? I would expect so.
>
> Classically any Chinese character has exactly the same width, which is 
> the same as the height (square). Nowadays some are taller than wide. 
> I'm quite certain that there is hardly any Chinese font with kerning 
> as this would break the grid.
>
>> Thinking aloud, you'd probably want to include some 
>> language-switching commands, to mediate between the calling of 
>> unicode fonts for un-named CJK glyphs (just raw conversion from 
>> Unicode to font switch + glyph number) to named roman (and other 
>> alphabetic) glyphs (conversion from UTF-8 to named glyphs to 
>> font+glyph, which retains kerning where it can).
>
> Well, I think one uses most of the time different fonts for Chinese 
> and non-CJK texts as many Chinese fonts don't include that many roman 
> letters (at least the ones quoted at 
> http://wiki.contextgarden.net/Chinese miss the ß and ä).

if there is a utf-8 mapping for chinese, then th eother chars can come 
from the main text font

Hans

Re: Chinese

[Tobias Burnus]

Hi,

Adam Lindsay wrote:
> Is there less kerning among CJK fonts? I would expect so.
Classically any Chinese character has exactly the same width, which is 
the same as the height (square). Nowadays some are taller than wide. I'm 
quite certain that there is hardly any Chinese font with kerning as this 
would break the grid.

> Thinking aloud, you'd probably want to include some language-switching 
> commands, to mediate between the calling of unicode fonts for un-named 
> CJK glyphs (just raw conversion from Unicode to font switch + glyph 
> number) to named roman (and other alphabetic) glyphs (conversion from 
> UTF-8 to named glyphs to font+glyph, which retains kerning where it can).
Well, I think one uses most of the time different fonts for Chinese and 
non-CJK texts as many Chinese fonts don't include that many roman 
letters (at least the ones quoted at 
http://wiki.contextgarden.net/Chinese miss the ß and ä).

Tobias

Re: Chinese

[Richard Gabriel]

[-- Attachment #1.1: Type: text/plain, Size: 2726 bytes --]

Hi guys,

I can confirm that the UTF-8 input doesn't work for me too. 
If I convert the file info GBK (CP936), it works fine [I suggest to use the 'iconv' utility for the conversion :-)].

I tested the UTF-8 output the followin ways:

1) 
\enableregime[utf]
\usemodule[chinese]

Processing a file with this setup ends with an error:

--- cut ---

kpathsea: Running mktextfm gbsong80
mktextfm: Could not map typeface abbreviation bs for gbsong80.
mktextfm: Need to update c:/WinApp/TeXLive/texmf-dist/fonts/map/fontname/special
.map?
mktextfm: Running mf "\mode:=ljfour; mag:=1; nonstopmode; input gbsong80"
This is METAFONT, Version 2.71828 (Web2c 7.5.5)

kpathsea: Running mktexmf gbsong80
mktexmf: empty or non-existent rootfile!
! I can't find file `gbsong80'.
<*> ...ljfour; mag:=1; nonstopmode; input gbsong80

Please type another input file name
! Emergency stop.
<*> ...ljfour; mag:=1; nonstopmode; input gbsong80

Transcript written on mfput.log.
mktextfm: warning: can't open log file gbsong80.log.
mktextfm: `mf "\mode:=ljfour; mag:=1; nonstopmode; input gbsong80"' failed.
kpathsea: Appending font creation commands to missfont.log.
! Font \unicodefont=gbsong80 at 24.88806pt not loadable: Metric (TFM) file not
found.

--- cut ---
  
I cannot figure out why it wants "gbsong80" when the encoding vector set starts from the 0x81 offset. 
Maybe some error in the UTF mapping?


2)
\usemodule[chinese]
  \enableregime[utf]
  
No error, but not output!
The PDF contains only black squares instead of glyphs.
The log shows that no fonts were used at all!

-Richard


  _____  

From: Xiao Jianfeng [mailto:fdu.xiaojf@gmail.com]
To: mailing list for ConTeXt users [mailto:ntg-context@ntg.nl]
Sent: Fri, 09 Dec 2005 15:24:22 +0100
Subject: Re: [NTG-context] Chinese

Tobias Burnus wrote:

> Hi,
>
> Xiao Jianfeng wrote:
>
>>> What would be needed to get UTF-8 input running with Chinese?
>>
>> If you use vim to edit your tex file, maybe you can try "set 
>> encoding=utf8", then save and compile.
>> As far as I know, GBK is compatible with unicode.
>
> No, that does not work - that is the reason I started this mail thread.
> You get the wrong characters and you may get some TeX errors.
> (And that is the reason Lutz wrote a UTF-8 to GBK converted.)
>
> Tobias
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>
What is the value of your environment variables about LC_CTYPE and LANG ?

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context
  

[-- Attachment #1.2: Type: text/html, Size: 3652 bytes --]

[-- Attachment #2: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: Chinese

[Tobias Burnus]

Hi,

Xiao Jianfeng wrote:
> What is the value of your environment variables about LC_CTYPE and LANG ?
Well, I use SCIM to input the characters and my locale is de_DE.UTF-8.
As the input works everywhere (OpenOffice, vim in Xterm, gvim etc.) I'm 
positiv that the problem is the lacking support of UTF-8 encoding of 
Chinese in ConTeXt and that this is not a problem of the encoding being 
wrong.

Looking at the cp936 to Unicode table 
(http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP936.TXT) 
one sees that they are not the same.

A rough big5 to Unicode table can be found at 
http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT


Tobias

Re: Chinese

[Adam Lindsay]

Tobias Burnus wrote:
> Hi,
> 
> Xiao Jianfeng wrote:
> 
>>> What would be needed to get UTF-8 input running with Chinese?
>>
>> If you use vim to edit your tex file, maybe you can try "set 
>> encoding=utf8", then save and compile.
>> As far as I know, GBK is compatible with unicode.
> 
> No, that does not work - that is the reason I started this mail thread.
> You get the wrong characters and you may get some TeX errors.
> (And that is the reason Lutz wrote a UTF-8 to GBK converted.)

Hmm, I suspect that some remix between my old (deprecated) Libertine in 
ConTeXt recipe and the ttf2tfm automatic unicode splitting would have 
some positive effects.

(I would discourage using that recipe for alphabetic (incl Roman) 
Unicode fonts because it blows away any kerning that would happen 
between unicode blocks. Is there less kerning among CJK fonts? I would 
expect so.)

Thinking aloud, you'd probably want to include some language-switching 
commands, to mediate between the calling of unicode fonts for un-named 
CJK glyphs (just raw conversion from Unicode to font switch + glyph 
number) to named roman (and other alphabetic) glyphs (conversion from 
UTF-8 to named glyphs to font+glyph, which retains kerning where it can).

I know it's sketchy and vague, but have a look inside font-uni. It's not 
the most complicated file in the distro.

adam
-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  Adam T. Lindsay, Computing Dept.     atl@comp.lancs.ac.uk
  Lancaster University, InfoLab21        +44(0)1524/510.514
  Lancaster, LA1 4WA, UK             Fax:+44(0)1524/510.492
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: Chinese

[Xiao Jianfeng]

Tobias Burnus wrote:

> Hi,
>
> Xiao Jianfeng wrote:
>
>>> What would be needed to get UTF-8 input running with Chinese?
>>
>> If you use vim to edit your tex file, maybe you can try "set 
>> encoding=utf8", then save and compile.
>> As far as I know, GBK is compatible with unicode.
>
> No, that does not work - that is the reason I started this mail thread.
> You get the wrong characters and you may get some TeX errors.
> (And that is the reason Lutz wrote a UTF-8 to GBK converted.)
>
> Tobias
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>
What is the value of your environment variables about LC_CTYPE and LANG ?

Re: Chinese

[Xiao Jianfeng]

Tobias Burnus wrote:

> Hi,
>
> Xiao Jianfeng wrote:
>
>>> What would be needed to get UTF-8 input running with Chinese?
>>
>> If you use vim to edit your tex file, maybe you can try "set 
>> encoding=utf8", then save and compile.
>> As far as I know, GBK is compatible with unicode.
>
> No, that does not work - that is the reason I started this mail thread.
> You get the wrong characters and you may get some TeX errors.
> (And that is the reason Lutz wrote a UTF-8 to GBK converted.)
>
> Tobias
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>
So, maybe the problem has something to do with you input method.
How do you input Chinese in your system ?

Re: Chinese

[Tobias Burnus]

Hi,

Xiao Jianfeng wrote:
>> What would be needed to get UTF-8 input running with Chinese?
> If you use vim to edit your tex file, maybe you can try "set 
> encoding=utf8", then save and compile.
> As far as I know, GBK is compatible with unicode.
No, that does not work - that is the reason I started this mail thread.
You get the wrong characters and you may get some TeX errors.
(And that is the reason Lutz wrote a UTF-8 to GBK converted.)

Tobias

Re: Chinese

[Xiao Jianfeng]

Hi,

Tobias Burnus wrote:

> Hi,
>
> Xiao Jianfeng wrote:
>
>> Maybe it is beacuse of the encoding of your .tex  file.
>> The encoding of my tex source file is cp936 and I edit with gvim.
>> ConTeXt compiles OK when processing Chinese. I din't use 
>> \enableregime[utf] or \language[cn] to typeset Chinese.
>
> Ok this works. Another possibility is to use the script by Lutz (see 
> link in the wiki) which converts UTF8 to gbk.
> However, both solutions have the drawback that e.g. "ä" does not get 
> typset (there is no ä in gbk).
>
> What would be needed to get UTF-8 input running with Chinese?
>
> Tobias
>
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>
If you use vim to edit your tex file, maybe you can try "set 
encoding=utf8", then save and compile.
As far as I know, GBK is compatible with unicode.

I'm not sure if it works or not. I don't konw too much about encoding.

Regards,
xiaojf

Re: Chinese

[Duncan Hothersall]

Tobias Burnus asks:

> What would be needed to get UTF-8 input running with Chinese?

If there is a recipe, I would like to help cook it. Getting UTF-8 input
running with Chinese would be a godsend to me too.

Duncan

Re: Chinese

[Tobias Burnus]

Hi,

Xiao Jianfeng wrote:
> Maybe it is beacuse of the encoding of your .tex  file.
> The encoding of my tex source file is cp936 and I edit with gvim.
> ConTeXt compiles OK when processing Chinese. I din't use 
> \enableregime[utf] or \language[cn] to typeset Chinese.
Ok this works. Another possibility is to use the script by Lutz (see 
link in the wiki) which converts UTF8 to gbk.
However, both solutions have the drawback that e.g. "ä" does not get 
typset (there is no ä in gbk).

What would be needed to get UTF-8 input running with Chinese?

Tobias

Re: Chinese

[Xiao Jianfeng]

Tobias Burnus wrote:

> Hi,
>
> Tobias Burnus wrote:
>
>> Seemingly the UTF-8 encoding makes some trouble - I get either errors 
>> or the wrong characters.
>
> Ok, I played around a bit more:
> \usemodule[chinese]
> \enableregime[utf]
> ...
> Hä? 中文？
>
> Prints as "Hä? ***" (* denotes black boxes).
>
> Whereas
> \enableregime[utf] % or without this line
> \usemodule[chinese]
> prints "H盲? ***" (* denotes three other characters which are not 中文？)
>
> In addition I get:
> ! Paragraph ended before \handleunicodeflowglyph was complete.
>
> Using \startitemize[c] or \placefigure{} with \mainlanguage[cn] shows 
> the proper characters.
>
> Any ideas? Looking at 
> http://www.pragma-ade.com/general/manuals/mchinese.pdf one seems to be 
> able to simply mix European and Chinese letters; ok using 
> {\language[de] Hä}{\language[cn]中文} I can combine them (if I 
> wouldn't get an error at '}'.
>
> Tobias
>
> PS: ConTeXt live at ConTeXtgarden does not like chinese at all; the 
> transcript shows:
> ! Misplaced alignment tab character &.
> l.6 &
>     
> #36825;里什么饮料也没有... 
>
>
>
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>
Maybe it is beacuse of the encoding of your .tex  file.
The encoding of my tex source file is cp936 and I edit with gvim.
ConTeXt compiles OK when processing Chinese. I din't use 
\enableregime[utf] or \language[cn] to typeset Chinese.

Regards,
xiaojf

Re: Chinese

[Patrick Gundlach]

[...]

> PS: ConTeXt live at ConTeXtgarden does not like chinese at all; the
> transcript shows:
> ! Misplaced alignment tab character &.
> l.6 &
>      #36825;里什么饮料也没有...

try again next week. I have this on my to do list.

Patrick
-- 
ConTeXt wiki and more: http://contextgarden.net

Re: Chinese

[Tobias Burnus]

Hi,

Tobias Burnus wrote:
> Seemingly the UTF-8 encoding makes some trouble - I get either errors 
> or the wrong characters.
Ok, I played around a bit more:
\usemodule[chinese]
\enableregime[utf]
...
Hä? 中文？

Prints as "Hä? ***" (* denotes black boxes).

Whereas
\enableregime[utf] % or without this line
\usemodule[chinese]
prints "H盲? ***" (* denotes three other characters which are not 中文？)

In addition I get:
! Paragraph ended before \handleunicodeflowglyph was complete.

Using \startitemize[c] or \placefigure{} with \mainlanguage[cn] shows 
the proper characters.

Any ideas? Looking at 
http://www.pragma-ade.com/general/manuals/mchinese.pdf one seems to be 
able to simply mix European and Chinese letters; ok using {\language[de] 
Hä}{\language[cn]中文} I can combine them (if I wouldn't get an error at 
'}'.

Tobias

PS: ConTeXt live at ConTeXtgarden does not like chinese at all; the 
transcript shows:
! Misplaced alignment tab character &.
l.6 &
     #36825;里什么饮料也没有...

Re: chinese

[Hans Hagen]

At 01:42 PM 5/26/2002 -0500, Mr. Lei Wang wrote:
> > how did you produce the pdf file (using mptopdf or texexec --fig=c --pdf
> > file.1 ?)
>
>The pdf file produced by above two methods is inserted correctly.
>
>My testing pdf file is converted by epstopdf. Yesterday I try the newest
>version, but the output pdf file can not insert correctly, either.

ok, then forget about epstopdf for metapost, the tex methods 
(texexec/mptopdf) are better for mp anyway and also provide you features 
like transparency (see latest metafun manual)

Hans
-------------------------------------------------------------------------
                                   Hans Hagen | PRAGMA ADE | pragma@wxs.nl
                       Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
  tel: +31 (0)38 477 53 69 | fax: +31 (0)38 477 53 74 | www.pragma-ade.com
-------------------------------------------------------------------------
                        information: http://www.pragma-ade.com/roadmap.pdf
                     documentation: http://www.pragma-ade.com/showcase.pdf
-------------------------------------------------------------------------

Re: chinese

[Mr. Lei Wang]

> how did you produce the pdf file (using mptopdf or texexec --fig=c --pdf 
> file.1 ?)

The pdf file produced by above two methods is inserted correctly.

My testing pdf file is converted by epstopdf. Yesterday I try the newest
version, but the output pdf file can not insert correctly, either.

Other test show that the MP files used as overlay also work although
it seems not correctly when use the presentation modules. I try to 
test more functions.

Wang

Re: chinese

[Hans Hagen]

At 12:55 PM 5/25/2002 -0500, Mr. Lei Wang wrote:
> > \setupoutput[dvipdfm]
>
>It works for most plain documents.
>
> > should work; as far as i know most features work but i'm not sure about mp
> > inclusion (one could always make independent pdf's first); if features are
> > missing but possible we can add them of course
>
>When I test place figure in documents, the MP inclusion seems work.
>I use:
>   \startuseMPgraphic{test}
>   ... some mp code here
>   \stopuseMPgraphic
>
>\starttext
>
>    \placefigure[here][fig1]{Test MP}{\useMPgraphic{test}}
>
>    \placefigure[here][fig2]{Test PDF}{\externalfigure[test2.pdf]}
>
>\stoptext
>
>here the test2.pdf is a pdf figure convert from the same mp code.
>the first figure is OK. but the second can not placed because tex
>could not determine its dimension correctly. I try to use
>" ebb test2.pdf " get the boundingbox information of test2.pdf
>and stored it in test2.bb. this usually works in latex. But texexec
>seems not read the test2.bb file to get the dimension.

how did you produce the pdf file (using mptopdf or texexec --fig=c --pdf 
file.1 ?)

(actually, there is a module that also treats the bb file, but this is 
related to a new fig labeling mechanism, but this is not yet in the 
distribution)

Hans

-------------------------------------------------------------------------
                                   Hans Hagen | PRAGMA ADE | pragma@wxs.nl
                       Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
  tel: +31 (0)38 477 53 69 | fax: +31 (0)38 477 53 74 | www.pragma-ade.com
-------------------------------------------------------------------------
                        information: http://www.pragma-ade.com/roadmap.pdf
                     documentation: http://www.pragma-ade.com/showcase.pdf
-------------------------------------------------------------------------

Re: chinese

[Hans Hagen]

At 02:42 AM 5/18/2002 +0200, Jens-Uwe Morawski wrote:

>yesterday i found the mchinese manual on the server. Previously
>i though that the up-to-date-7 is the only documentation about
>chinese in ConTeXT.
>
>The current version has some unfinished sections. Is there any plan
>that this will change soon (in some weeks)?
>Especially i would like to know what program is a good choice on Linux
>and Windows to edit chinese TeX files. Emacs-MuLE?
>What encoding should i use to save TeX files, UTF-8?
>Any other important hints?

Last week

   "Lei Wang" <wanglei@sbcglobal.net>

announced to pick up the chinese thread; he can best advice on editing 
(there are some tools for windows), encoding etc.

[the missing will certainly be filled in, but i depend on Lei Wangs 
expertise]

Hans
-------------------------------------------------------------------------
                                   Hans Hagen | PRAGMA ADE | pragma@wxs.nl
                       Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
  tel: +31 (0)38 477 53 69 | fax: +31 (0)38 477 53 74 | www.pragma-ade.com
-------------------------------------------------------------------------
                        information: http://www.pragma-ade.com/roadmap.pdf
                     documentation: http://www.pragma-ade.com/showcase.pdf
-------------------------------------------------------------------------

chinese

[Jens-Uwe Morawski]

Hullo,

yesterday i found the mchinese manual on the server. Previously
i though that the up-to-date-7 is the only documentation about
chinese in ConTeXT.

The current version has some unfinished sections. Is there any plan
that this will change soon (in some weeks)?
Especially i would like to know what program is a good choice on Linux
and Windows to edit chinese TeX files. Emacs-MuLE?
What encoding should i use to save TeX files, UTF-8?
Any other important hints?

Many thanks. Best,
  Jens