Problem with AER-Font : ß becomes SS

Problem with AER-Font : ß becomes SS

[Robert F. Beeger]

Hi!

I'm using now the AER-Font to make my hyphenation for german words work.
All seemed to work fine until a more careful reading of my text

When I run a file :
\mainlanguage[de]
\language[de]

\setupbodyfont[aer, 10pt]
\useencoding[pro]

\input xhyphen

\starttext
äÄ üÜ öÖ ß
\stoptext

I get in my pdf-file the output :
äÄ üÜ öÖ SS

I want the ß to stay an ß and not to become an SS. This looks wrong in 
words like "auSSerdem"

Changing \useencoding[pro] to \useencoding[ec] doesn't change this.
Changing \useencoding[pro] to \useencoding[win] brings up again that 
hyphenation-error..

Changing my file this way :
\mainlanguage[de]
\language[de]

\setupbodyfont[aer, 10pt]

\input xhyphen

\useencoding[win]

\starttext
äÄ üÜ öÖ ß
\stoptext

seems to do it, but it doesn't look right to me to change the encoding 
after the hyphenation is done.

And what do I need \useencoding[win] for. I thought that the AER-Font would 
contain the Umlauts and the ß so that there be no need to
map them again. Is there a mistake in the mapping of the ß in the AER-Font?

I'm using the latest stable version of ConTeXt - 2001.2.27 - and fpTeX with 
all the updates that are available on ftp.dante.de.

Greetings
      Robert

Re: Problem with AER-Font : ß becomes SS

[Uwe Koloska]

Am Sonntag,  4. März 2001 13:34 schrieb Robert F. Beeger:
> Hi!
>
> I'm using now the AER-Font to make my hyphenation for german words work.
> All seemed to work fine until a more careful reading of my text
>
> Changing my file this way :
> \mainlanguage[de]
> \language[de]
>
> \setupbodyfont[aer, 10pt]
>
> \input xhyphen

what is this supposed to do?  I don't have such a file ...

> \useencoding[win]

yes, that' the right thing to do (R) ;-)

> \starttext
> äÄ üÜ öÖ ß
> \stoptext
>
> seems to do it, but it doesn't look right to me to change the encoding
> after the hyphenation is done.

where do you do the hyphenation???

> And what do I need \useencoding[win] for.

It is for changing the _input_encoding!  It's a little bit confusing that 
there is no differentiation between input and output encoding.  "ec" is an 
output encoding used to map internal codes to external ones.  "pro" is a 
special case, cause it is needed for protruding (that is something I don't 
understand in depth ...).  "win" is an input encoding.  So maybe "\input 
xhyphen" uses some characters that are remapped by "win".

Though "win" is a name I don't like ;-)  it now is the right encoding when 
"mostly ansi 8859-1" is meant.

 I thought that the AER-Font
> would contain the Umlauts and the ß so that there be no need to
> map them again. Is there a mistake in the mapping of the ß in the
> AER-Font?

Yes, if you think from the ansi 8859-1 direction.  The characters in AER 
are mapped with the T1 encoding.  An if you think about the presentation of 
characters for computers -- they don't see the glyphs, they only see a 
number, and a convention (an encoding) is used to associate glyphs with 
this number -- you may finally be able to understand the problem:

You enter your text in a specific encoding (for nearly all western 
encodings the codes for the ASCII section are the same, so an "A" is an "A" 
is an "A" ;-)) and the fonts you use have another one.

Hope this helps
Uwe

-- 
mailto:koloska@rcs.urz.tu-dresden.de
http://rcswww.urz.tu-dresden.de/~koloska/
--                                    --
right now the web page is in german only
but this will change as time goes by ;-)

Re: Problem with AER-Font : ß becomes SS

[Robert F. Beeger]

Hi, Uwe!
Hi, all!

At 23:08 04.03.2001 +0100, Uwe Koloska wrote:
>Am Sonntag,  4. März 2001 13:34 schrieb Robert F. Beeger:
> > Hi!
> >
> > I'm using now the AER-Font to make my hyphenation for german words work.
> > All seemed to work fine until a more careful reading of my text
> >
> > Changing my file this way :
> > \mainlanguage[de]
> > \language[de]
> >
> > \setupbodyfont[aer, 10pt]
> >
> > \input xhyphen
>
>what is this supposed to do?  I don't have such a file ...

It's my eXtra HYPHENation file . :)

> > \useencoding[win]
>
>yes, that' the right thing to do (R) ;-)
>
> > \starttext
> > äÄ üÜ öÖ ß
> > \stoptext
> >
> > seems to do it, but it doesn't look right to me to change the encoding
> > after the hyphenation is done.
>
>where do you do the hyphenation???

Up in that xhyphen.tex

> > And what do I need \useencoding[win] for.
>
>It is for changing the _input_encoding!  It's a little bit confusing that
>there is no differentiation between input and output encoding.  "ec" is an
>output encoding used to map internal codes to external ones.  "pro" is a
>special case, cause it is needed for protruding (that is something I don't
>understand in depth ...).  "win" is an input encoding.  So maybe "\input
>xhyphen" uses some characters that are remapped by "win".
>
>Though "win" is a name I don't like ;-)  it now is the right encoding when
>"mostly ansi 8859-1" is meant.
>
>  I thought that the AER-Font
> > would contain the Umlauts and the ß so that there be no need to
> > map them again. Is there a mistake in the mapping of the ß in the
> > AER-Font?
>
>Yes, if you think from the ansi 8859-1 direction.  The characters in AER
>are mapped with the T1 encoding.  An if you think about the presentation of
>characters for computers -- they don't see the glyphs, they only see a
>number, and a convention (an encoding) is used to associate glyphs with
>this number -- you may finally be able to understand the problem:
>
>You enter your text in a specific encoding (for nearly all western
>encodings the codes for the ASCII section are the same, so an "A" is an "A"
>is an "A" ;-)) and the fonts you use have another one.

Thanks for this info, but my problem still remains.
\useencoding[win] is changing the input encoding. So I should use it before 
there is any text-input.
In my understanding this should be before my hyphenation file (xhyphen.tex) 
is input.
When I do this, I get again the hypenation error that complains that the 
word to be hyphenated
  must contain only letters and hyphens.
So, using the AER-Font doesn't change anything in this situation.
Only when I use this \useencoding[win] after the hyphenation is done, all 
seems to work.

My concern is that when I hyphenate \hyphenate{au\-ßer\-dem} what it will 
be hyphenated like.
Will TeX now have "au\-ßer\-dem" in its hyphenation table or 
"au\-SSer\-dem"? Will an "außerdem"
after the change to the win-input-encoding be recognized as the one 
hyphenated before the change?

>Hope this helps

Not really, but thanks for the try.

Greetings
     Robert

Re: Problem with AER-Font : ß becomes SS

[Uwe Koloska]

Am Montag,  5. März 2001 08:31 hat Robert F. Beeger geschrieben:
> Thanks for this info, but my problem still remains.
> \useencoding[win] is changing the input encoding. So I should use it
> before there is any text-input.
> In my understanding this should be before my hyphenation file
> (xhyphen.tex) is input.
> When I do this, I get again the hypenation error that complains that the
> word to be hyphenated must contain only letters and hyphens.

Here it would have been a good idea to cite the error message correctly.  
It appears on the "\hyphenation" command and not when the hyphenation is 
really done (that was what I have understand ...)

> So, using the AER-Font doesn't change anything in this situation.
> Only when I use this \useencoding[win] after the hyphenation is done, all
> seems to work.

Have you tried a minimal file without any hyphenation-exception?

> My concern is that when I hyphenate \hyphenate{au\-ßer\-dem} what it will

I think you have mistyped the command here:
  \hyphenation{auß-er-dem}
is the internal TeX command that is meant for hyphenation exceptions

> be hyphenated like.
> Will TeX now have "au\-ßer\-dem" in its hyphenation table or
> "au\-SSer\-dem"? Will an "außerdem"
> after the change to the win-input-encoding be recognized as the one
> hyphenated before the change?

I don't know.  Is there any chance to get the internal hypenation table?

By reproducing your problem with a minimal input for context and latex,  I 
understand it better:

Because "außerdem" will be hyphenated properly, it is a bad example.  So I 
changed the right hyphenation to a wrong one ...

Here is the latex example:
---------------------------------------------------------------
\documentclass{article}
\usepackage[latin1]{inputenc}
\usepackage{german}
\usepackage{ae}

\hyphenation{auß-er-dem}

\begin{document}

außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem
außerdem außerdem außerdem außerdem außerdem außerdem außerdem 

\end{document}
---------------------------------------------------------------

and now the one for context:
---------------------------------------------------------------
\mainlanguage[de]
\language[de]

\setupbodyfont[aer, 10pt]

\hyphenation{auß-er-dem}

\useencoding[win]

\starttext
\dorecurse{100}{außerdem }
\stoptext
---------------------------------------------------------------

The latex example runs correctly and hyphenates like it is given.  Context 
runs properly but doesn't hyphenate the way it is given (it uses the right 
hypenation though)

So I think, there is a problem with contexts input encoding.

Uwe

-- 
mailto:koloska@rcs.urz.tu-dresden.de
http://rcswww.urz.tu-dresden.de/~koloska/
--                                    --
right now the web page is in german only
but this will change as time goes by ;-)

Re: Problem with AER-Font : ß becomes SS

[Robert F. Beeger]

At 11:45 05.03.2001 +0100, Uwe Koloska wrote:
>Am Montag,  5. März 2001 08:31 hat Robert F. Beeger geschrieben:
> > Thanks for this info, but my problem still remains.
> > \useencoding[win] is changing the input encoding. So I should use it
> > before there is any text-input.
> > In my understanding this should be before my hyphenation file
> > (xhyphen.tex) is input.
> > When I do this, I get again the hypenation error that complains that the
> > word to be hyphenated must contain only letters and hyphens.
>
>Here it would have been a good idea to cite the error message correctly.

Sorry, I thought that everyone would understand my e-mail as a follow-up to 
the one I posted some two weeks ago
to this mailing-list ("Hyphenation of german words and em-dashes in german 
texts."), in which I have described this error and
also have put it into the message. That's the advantage of news-groups. 
They are structured.
Here's again the error:
encoding       : coding win is loaded
(g:/TeX/texmf/tex/context/base/enco-win.tex) (./xhyphen.tex
! Improper \hyphenation will be flushed.
\dohandleaccent #1#2->\def
                            \glyph {#2}\ifx \glyph \empty \dohandleaccent 
#1\...
l.1 \hyphenation{Prõ
                     -sen-ta-tion}
? h
Hyphenation exceptions must contain only letters
and hyphens. But continue; I'll forgive and forget.

>It appears on the "\hyphenation" command and not when the hyphenation is
>really done (that was what I have understand ...)

That's what I meant.

> > So, using the AER-Font doesn't change anything in this situation.
> > Only when I use this \useencoding[win] after the hyphenation is done, all
> > seems to work.
>
>Have you tried a minimal file without any hyphenation-exception?

Yes. When using \useencoding[win] it all works. But without any 
hyphenation-exception I don't need the AER-Font.
\useencoding[win] does the right mapping with CMR, too. I changed to AER to 
make a hyphenation like
\hyphenation{Prä-sen-ta-tion} (see it: now it's the right command :) ) 
possible. I don't see any other advantages in
using AER. I read that there are misadvantages in using AER when the text 
has formulas with greek letters in it.
In my current text this is lukily not the problem. So sometimes it's more 
an misadvantage to use AER.

> > My concern is that when I hyphenate \hyphenate{au\-ßer\-dem} what it will
>
>I think you have mistyped the command here:
>   \hyphenation{auß-er-dem}

That's right. I have written my hyphenation-file some weeks ago and am not 
using thís hyphenation
command very often. I should have had another look into this file before 
writing this e-mail.
In my hyphenation-file I use the right command.

>is the internal TeX command that is meant for hyphenation exceptions
>
> > be hyphenated like.
> > Will TeX now have "au\-ßer\-dem" in its hyphenation table or
> > "au\-SSer\-dem"? Will an "außerdem"
> > after the change to the win-input-encoding be recognized as the one
> > hyphenated before the change?
>
>I don't know.  Is there any chance to get the internal hypenation table?

Don't know, either.

>By reproducing your problem with a minimal input for context and latex,  I
>understand it better:
>
>Because "außerdem" will be hyphenated properly, it is a bad example.  So I
>changed the right hyphenation to a wrong one ...
>
>Here is the latex example:
>---------------------------------------------------------------
>\documentclass{article}
>\usepackage[latin1]{inputenc}
>\usepackage{german}
>\usepackage{ae}
>
>\hyphenation{auß-er-dem}
>
>\begin{document}
>
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>außerdem außerdem außerdem außerdem außerdem außerdem außerdem
>
>\end{document}
>---------------------------------------------------------------
>
>
>and now the one for context:
>---------------------------------------------------------------
>\mainlanguage[de]
>\language[de]
>
>\setupbodyfont[aer, 10pt]
>
>\hyphenation{auß-er-dem}
>
>\useencoding[win]
>
>\starttext
>\dorecurse{100}{außerdem }
>\stoptext
>---------------------------------------------------------------
>
>The latex example runs correctly and hyphenates like it is given.  Context
>runs properly but doesn't hyphenate the way it is given (it uses the right
>hypenation though)
>
>So I think, there is a problem with contexts input encoding.

You did a deeper exploring than I had thought of and probably you have also 
found a bug I haven't thought of.

My problem is - to make it short - that I cannot have a
\hyphenation{Prä-sen-ta-tion}
after a
\useencoding[win]
and that using AER doesn't help here.

I tested my text for the bug you found.
In my hyphenation file I have changed \hyphenate{Prä-sen-ta-tion} to
\hyphenate{Prä-ßen-ta-tion}. This is not the right writing of the word, but 
it gives an interesting test case.

In my text I have an overful \hbox with the word "Präsentation" in it. This 
I also changed to "Präßentation".

Here's this overful box in a run without my own hyphenations:
Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
[]/aer10/Ein An-satz wä-re be-stimmt, die Prä-ÿen-ta-ti-on so zu neh-men, wie s
ie ist und sie über die ViewProvider-

And here it is with the my own hyphenations
Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
[]/aer10/Ein An-satz wä-re be-stimmt, die Prä-ÿen-ta-ti-on so zu neh-men, wie s
ie ist und sie über die ViewProvider-

Tex doesn't recognize this "Präßentation" to be the one in the 
hyphenation-command before the \useencoding[win].
How could it. The input encoding has changed in the meanwhile.

To make it clearer:

\setupbodyfont[aer, 10pt]
\hyphenation{Prä-ßen-ta-tion}
\useencoding[win]
\starttext
Präßentation  % This is not the word specified in the hyphenation-command
\stoptext

Another test. Now with Prääsentation instead of Präsentation.
Without own hyphenation:
Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
[]/aer10/Ein An-satz wä-re be-stimmt, die Prää-en-ta-ti-on so zu neh-men, wie s
ie ist und sie über die ViewProvider-

and with own hyphenation
Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
[]/aer10/Ein An-satz wä-re be-stimmt, die Prä-äen-ta-tion so zu neh-men, wie si
e ist und sie über die ViewProvider-

So it seems to be a problem with the ß or with the encoding in general.
You can't use \useencoding[win] before the \hyphenation-commands.
You can't use it afer these commands, because then the words aren't recognize.

I'm not an expert in this encoding-topic, but it seems to me that AER 
should map ß to ß as it does with ä to ä, ü to ü etc.
When using AER there should be no need for using \useencoding[win] since 
this doesn't work with \hyphenation.

Maybe there's a needing for  AERWin, AERUnix etc. to make the right mapping.
Or maybe \useencoding[win] should work with \hyphenation, but then there is 
no need for AER.

Robert

Re: Problem with AER-Font : ß becomes SS

[Uwe Koloska]

Am Dienstag,  6. März 2001 11:12 schrieb Robert F. Beeger:
> At 11:45 05.03.2001 +0100, Uwe Koloska wrote:
> >
> >Here it would have been a good idea to cite the error message correctly.
>
> Sorry, I thought that everyone would understand my e-mail as a follow-up
> to the one I posted some two weeks ago
> to this mailing-list ("Hyphenation of german words and em-dashes in
> german texts."), in which I have described this error and
> also have put it into the message. That's the advantage of news-groups.
> They are structured.

Ahem, if the message is not written in reply to another one, it isn't 
ordered in the same thread.  So your message would have started a new 
thread either in a newsgroup ...

> > > So, using the AER-Font doesn't change anything in this situation.
> > > Only when I use this \useencoding[win] after the hyphenation is done,
> > > all seems to work.

Just a remark to this older statement.  It does make a difference wether 
you use AER-fonts or not!  Try it!  If aer is left out "ß" and the umlauts 
is not recognized as a character.

> >Have you tried a minimal file without any hyphenation-exception?
>
> Yes. When using \useencoding[win] it all works. But without any
> hyphenation-exception I don't need the AER-Font.

I don't know if the following is true for context too, but it is for Tex 
and LaTeX.  Without T1 font encoding, "ß" and umlauts have to be build with 
two or more other glyphs.  And TeX is not able to hyphenate words that 
contain token (in other words, it only can hyphenate words that contain 
plain characters).  You can examine this problem by commenting the 
following lines from my previous example:

%\setupbodyfont[aer, 10pt]
%\hyphenation{auß-er-dem}

(the second one because without aer, "ß" isn't recognized as a character)

> \useencoding[win] does the right mapping with CMR, too. I changed to AER
> to make a hyphenation like
> \hyphenation{Prä-sen-ta-tion} (see it: now it's the right command :) )
> possible. I don't see any other advantages in
> using AER. I read that there are misadvantages in using AER when the text
> has formulas with greek letters in it.
> In my current text this is lukily not the problem. So sometimes it's more
> an misadvantage to use AER.

There is another package with faked T1 fonts named ze
  CTAN/fonts/zefonts
(it is a good idea to read the de-tex-faq!)

> >The latex example runs correctly and hyphenates like it is given. 
> > Context runs properly but doesn't hyphenate the way it is given (it
> > uses the right hypenation though)
> >
> >So I think, there is a problem with contexts input encoding.
>
> You did a deeper exploring than I had thought of and probably you have
> also found a bug I haven't thought of.

No, it's the same problem!  aer makes it possible to insert character with 
codes above 128 into hyphenation patterns, but [win] changes the character 
to a command, and this command can't be recognized as the same letter that 
is included in the hyphenation pattern.

> My problem is - to make it short - that I cannot have a
> \hyphenation{Prä-sen-ta-tion}
> after a
> \useencoding[win]
> and that using AER doesn't help here.

Yes, that's an adequate statement about the problem ;-)

> I tested my text for the bug you found.
> In my hyphenation file I have changed \hyphenate{Prä-sen-ta-tion} to
> \hyphenate{Prä-ßen-ta-tion}. This is not the right writing of the word,
> but it gives an interesting test case.

You didn't understand :-(  I changed the hyphenation to make it possible to 
observe wether the hyphenation is controlled by the hyphenation exception 
or the normal hyphenation algorithm!

> In my text I have an overful \hbox with the word "Präsentation" in it.
> This I also changed to "Präßentation".
>
> Here's this overful box in a run without my own hyphenations:
> Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
> []/aer10/Ein An-satz wä-re be-stimmt, die Prä-ÿen-ta-ti-on so zu neh-men,
> wie s ie ist und sie über die ViewProvider-

If you look carefully at the message, you can see the code that is used for 
the "ß".  So I tried to insert "ÿ" instead of "ß" in the hyphenation pattern
  \hyphenation{auÿ-er-dem}
and -- guess what -- the hyphenation is considered by context!!!

> Another test. Now with Prääsentation instead of Präsentation.
> Without own hyphenation:
> Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
> []/aer10/Ein An-satz wä-re be-stimmt, die Prää-en-ta-ti-on so zu neh-men,
> wie s ie ist und sie über die ViewProvider-
>
> and with own hyphenation
> Overfull \hbox (2.53653pt too wide) in paragraph at lines 898--903
> []/aer10/Ein An-satz wä-re be-stimmt, die Prä-äen-ta-tion so zu neh-men,
> wie si e ist und sie über die ViewProvider-
>
> So it seems to be a problem with the ß or with the encoding in general.
> You can't use \useencoding[win] before the \hyphenation-commands.
> You can't use it afer these commands, because then the words aren't
> recognize.

It is a problem with T1 encoding.  In the de-tex-faq you can find:

  Diese Kodierung entspricht im wesentlichen dem Code ISO 8859-1 (Latin-1) 
  mit dem Unterschied, daß das scharfe s zu den anderen akzentuierten 
  Kleinbuchstaben auf Position 255 gesetzt wurde

  this encoding is essentially in accordance to iso 8859-1 with the
  difference, that the sharp s "ß" is changed to position 255, where the
  other accented characters are located

This means: "ß" has a different position in iso 8859-1 (input encoding) and 
ec (output encoding) "ä" has the same position.

So if someone makes a version of ae (or ze) with texnansi (and not T1), 
this would be a workaround till we fix this encoding behaviour to match the 
one that latex uses -- and that works.

Uwe

-- 
mailto:koloska@rcs.urz.tu-dresden.de
http://rcswww.urz.tu-dresden.de/~koloska/
--                                    --
right now the web page is in german only
but this will change as time goes by ;-)