Re: FAQ, German Umlauts (patch)

Re: FAQ, German Umlauts (patch)

[Peter Stephenson]

Uli Zappe wrote:
> > Zsh doesn't even decide; the system does and it seems to be
> > getting it wrong.
> 
> But zsh is the *only* program with this problem on NEXTSTEP; so it  
> can't be as simple as "zsh is right, but NEXTSTEP is wrong"... :-(  
> perl, for instance, works perfectly well.

That's because they're simply not deciding.  Zsh, having a fairly
sophisticated editor, is trying to spare the user unreadable
characters on the terminal.  Perl doesn't know about characters, it
simply deals with byte streams.  You can probably see what I mean if
you have `less' and try to view something with non-ASCII characters.

> > The old zsh, so far as I remember, didn't even have the locale
> > mechanism, so is unlikely to be able to do non-ASCII characters.
> > Even a reasonable attempt at 8-bit characters is fairly new.
> 
> zsh 2.51 *can* deal perfectly with lower case Umlauts, as I said before.

That must be because it simply passed all 8-bit characters through.
This is unfortunately not a sensible default:  it'll screw up your old
VT100 something rotten, for example.

> > To clarify, zsh doesn't need the locale system
> > either, it just supports it if it is available.
> 
> Does this locale system consist of only the file /usr/lib/locale  
> (so that I simply could install such a file), or is there a  
> corresponding program that's necessary?

No, there's a whole mess of stuff in the operating system; either it
comes with it or it doesn't.

> > The alternative is Wolfgang Hukriede's suggestion of passing
> > through all characters.  That's dangerous in general, so it would
> > have to be an option.
> 
> Maybe it's not necessary to really pass thru *all* characters; I  
> can't judge, though.
> 
> Anyway, what do I have to do to make this option being implemented  
> in future versions of zsh?

It's very simple and I've just done it.  I've assumed that only eight
bit characters are a problem.  The option is called PRINT_EIGHT_BIT
(or printeightbit etc.); see the manual entry.  (Would PASS_EIGHT_BIT
be better? It sounded a little opaque to me.)

*** Doc/Zsh/options.yo.p8	Wed Apr 30 09:18:38 1997
--- Doc/Zsh/options.yo	Thu Jun 26 18:33:51 1997
***************
*** 622,627 ****
--- 622,634 ----
  tt(trap) and
  tt(unset).
  )
+ pindex(PRINT_EIGHT_BIT)
+ cindex(exit status, printing)
+ item(tt(PRINT_EIGHT_BIT) (tt(-1)))(
+ Print eight bit characters literally in completion lists, etc.
+ This option is not necessary if your system correctly returns the
+ printability of eight bit characters (see manref(ctype)(3)).
+ )
  pindex(PRINT_EXIT_VALUE)
  cindex(exit status, printing)
  item(tt(PRINT_EXIT_VALUE) (tt(-1)))(
*** Src/options.c.p8	Thu Jun 26 18:17:28 1997
--- Src/options.c	Thu Jun 26 18:14:26 1997
***************
*** 144,149 ****
--- 144,150 ----
  {NULL, "overstrike",	      0,			 OVERSTRIKE},
  {NULL, "pathdirs",	      0,			 PATHDIRS},
  {NULL, "posixbuiltins",	      OPT_EMULATE|OPT_BOURNE,	 POSIXBUILTINS},
+ {NULL, "printeightbit",       0,                         PRINTEIGHTBIT},
  {NULL, "printexitvalue",      0,			 PRINTEXITVALUE},
  {NULL, "privileged",	      OPT_SPECIAL,		 PRIVILEGED},
  {NULL, "promptcr",	      OPT_ALL,			 PROMPTCR},
*** Src/utils.c.p8	Thu Jun 26 18:17:18 1997
--- Src/utils.c	Thu Jun 26 18:39:11 1997
***************
*** 175,180 ****
--- 175,182 ----
      if (isprint(c))
  	goto done;
      if (c & 0x80) {
+ 	if (isset(PRINTEIGHTBIT))
+ 	    goto done;
  	*s++ = '\\';
  	*s++ = 'M';
  	*s++ = '-';
*** Src/zsh.h.p8	Tue Jun 24 09:09:11 1997
--- Src/zsh.h	Thu Jun 26 18:13:27 1997
***************
*** 1062,1068 ****
      VERBOSE,
      XTRACE,
      USEZLE,
!     OPT_SIZE
  };
  
  #undef isset
--- 1062,1069 ----
      VERBOSE,
      XTRACE,
      USEZLE,
!     OPT_SIZE,
!     PRINTEIGHTBIT
  };
  
  #undef isset

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutsches Elektronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: FAQ, German Umlauts (patch)

[Zefram]

Peter Stephenson wrote:
>+ item(tt(PRINT_EIGHT_BIT) (tt(-1)))(
>+ Print eight bit characters literally in completion lists, etc.
>+ This option is not necessary if your system correctly returns the
>+ printability of eight bit characters (see manref(ctype)(3)).

If it's only needed if the system has a bug, then a run-time option is the
wrong way to do it.  Make it a configure-time option (--enable-pass8),
and conditionally compile in such a special version of isprint(), using
it instead of the system's version.

>      VERBOSE,
>      XTRACE,
>      USEZLE,
>!     OPT_SIZE,
>!     PRINTEIGHTBIT
>  };

Also this is wrong.  OPT_SIZE must be the last entry in the list, and
the options in the list are in alphabetical order.

-zefram

Re: FAQ, German Umlauts (patch)

[Uli Zappe]

Zefram wrote:

> If it's only needed if the system has a bug, then a run-time
> option is the wrong way to do it.  Make it a configure-time option
> (--enable-pass8), and conditionally compile in such a special
> version of isprint(), using it instead of the system's version.

This sounds very reasonable.

Is there a source for isprint() lying around somewhere? I could  
modify and/or at least test its correct behaviour for NEXTSTEP.


                Bye
                        Uli

_____________________________________________________________________

Uli Zappe               E-Mail: uli@tallowcross.uni-frankfurt.de
                                (NeXTMail,Mime,ASCII) PGP on request
Lorscher Strasse 5      WWW:    -
D-60489 Frankfurt       Fon:    +49 (69) 9784 0007
Germany                 Fax:    +49 (69) 9784 0042

staff member of NEXTTOYOU - the German NEXTSTEP/OPENSTEP magazine
_____________________________________________________________________

Re: FAQ, German Umlauts (patch)

[Uli Zappe]

I wrote:

> Is there a source for isprint() lying around somewhere? I could
> modify and/or at least test its correct behaviour for NEXTSTEP.

Aehem, to add to my own remark:

I've had the time now to look in the docs and found there is a  
NEXTSTEP equivalent to isprint, called NXIsPrint.

So obviously all that needs to be done is to make zsh use an  
alternative isprint and then use NXIsPrint for it (or "write" an  
alternative isprint that wraps NXIsPrint).

How could this be integrated in the source?


                Bye
                        Uli

_____________________________________________________________________

Uli Zappe               E-Mail: uli@tallowcross.uni-frankfurt.de
                                (NeXTMail,Mime,ASCII) PGP on request
Lorscher Strasse 5      WWW:    -
D-60489 Frankfurt       Fon:    +49 (69) 9784 0007
Germany                 Fax:    +49 (69) 9784 0042

staff member of NEXTTOYOU - the German NEXTSTEP/OPENSTEP magazine
_____________________________________________________________________

Re: FAQ, German Umlauts (patch)

[Peter Stephenson]

Zefram wrote:
> Peter Stephenson wrote:
> >+ item(tt(PRINT_EIGHT_BIT) (tt(-1)))(
> >+ Print eight bit characters literally in completion lists, etc.
> >+ This option is not necessary if your system correctly returns the
> >+ printability of eight bit characters (see manref(ctype)(3)).
> 
> If it's only needed if the system has a bug, then a run-time option is the
> wrong way to do it.  Make it a configure-time option (--enable-pass8),
> and conditionally compile in such a special version of isprint(), using
> it instead of the system's version.

I agree with Wolfgang on this:  you simply can't tell from the system
set-up whether it supports the hardware, i.e. whether the terminal
Does the Right Thing with eight bit characters or not. If you're using
a dumb character terminal as well as a smart terminal on the same system
you definitely need the run-time option.  In any case, there's no
way of deciding automatically and (as Uli has been saying) these NEXTSTEP
people like the thing just to work.  Configuring at compile-time whether
to dump raw characters at run-time to a terminal which could be anything
is just too brutal.  (What you possibly could configure is whether or not
to compile in the *option*, but I think it's obvious that that has less
than welcome side effects for code.  More plausibly you could decide at
compile time system by system whether the option should be in effect by
default.)

> >      VERBOSE,
> >      XTRACE,
> >      USEZLE,
> >!     OPT_SIZE,
> >!     PRINTEIGHTBIT
> >  };
> 
> Also this is wrong.  OPT_SIZE must be the last entry in the list, and
> the options in the list are in alphabetical order.

that was stupid... I simply noticed the last two were out of order and
thought therefore I could dump the new one on the end.  Here's a revised
version.

*** Doc/Zsh/options.yo.p8	Wed Apr 30 09:18:38 1997
--- Doc/Zsh/options.yo	Thu Jun 26 18:33:51 1997
***************
*** 622,627 ****
--- 622,634 ----
  tt(trap) and
  tt(unset).
  )
+ pindex(PRINT_EIGHT_BIT)
+ cindex(exit status, printing)
+ item(tt(PRINT_EIGHT_BIT) (tt(-1)))(
+ Print eight bit characters literally in completion lists, etc.
+ This option is not necessary if your system correctly returns the
+ printability of eight bit characters (see manref(ctype)(3)).
+ )
  pindex(PRINT_EXIT_VALUE)
  cindex(exit status, printing)
  item(tt(PRINT_EXIT_VALUE) (tt(-1)))(
*** Src/options.c.p8	Thu Jun 26 18:17:28 1997
--- Src/options.c	Thu Jun 26 18:14:26 1997
***************
*** 144,149 ****
--- 144,150 ----
  {NULL, "overstrike",	      0,			 OVERSTRIKE},
  {NULL, "pathdirs",	      0,			 PATHDIRS},
  {NULL, "posixbuiltins",	      OPT_EMULATE|OPT_BOURNE,	 POSIXBUILTINS},
+ {NULL, "printeightbit",       0,                         PRINTEIGHTBIT},
  {NULL, "printexitvalue",      0,			 PRINTEXITVALUE},
  {NULL, "privileged",	      OPT_SPECIAL,		 PRIVILEGED},
  {NULL, "promptcr",	      OPT_ALL,			 PROMPTCR},
*** Src/utils.c.p8	Thu Jun 26 18:17:18 1997
--- Src/utils.c	Thu Jun 26 18:39:11 1997
***************
*** 175,180 ****
--- 175,182 ----
      if (isprint(c))
  	goto done;
      if (c & 0x80) {
+ 	if (isset(PRINTEIGHTBIT))
+ 	    goto done;
  	*s++ = '\\';
  	*s++ = 'M';
  	*s++ = '-';
*** Src/zsh.h.p8	Tue Jun 24 09:09:11 1997
--- Src/zsh.h	Fri Jun 27 09:35:25 1997
***************
*** 1035,1040 ****
--- 1035,1041 ----
      OVERSTRIKE,
      PATHDIRS,
      POSIXBUILTINS,
+     PRINTEIGHTBIT,
      PRINTEXITVALUE,
      PRIVILEGED,
      PROMPTCR,


-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutsches Elektronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

German Umlauts still don't work

[Uli Zappe]

Hi all,

the German Umlaut problem I brought up a few days ago is more  
difficult to solve than expected.

Peter Stephenson suggested to patch utils.c which is the only  
source file in zsh making use of isprint(c) which delivers incorrect  
results for NEXTSTEP.

I tried that, but it didn't change a thing. Obviously there is  
another part of the zsh source that's limiting zsh's behaviour; it's  
definitely *not* the isprint(c) part (which would be easy to  
replace by the correct NXIsPrint(c) for the NEXTSTEP version, BTW).

Does anyone have an idea where to look? Because of the pathnames on  
my system (which I can't change for various reasons) I really NEED  
those Umlauts... :-(

Thank you!


                Bye
                        Uli

_____________________________________________________________________

Uli Zappe               E-Mail: uli@tallowcross.uni-frankfurt.de
                                (NeXTMail,Mime,ASCII) PGP on request
Lorscher Strasse 5      WWW:    -
D-60489 Frankfurt       Fon:    +49 (69) 9784 0007
Germany                 Fax:    +49 (69) 9784 0042

staff member of NEXTTOYOU - the German NEXTSTEP/OPENSTEP magazine
_____________________________________________________________________

Re: German Umlauts still don't work

[Bart Schaefer]

On Jul 3,  3:52am, Uli Zappe wrote:
} Subject: German Umlauts still don't work
}
} the German Umlaut problem I brought up a few days ago is more  
} difficult to solve than expected.
} 
} Peter Stephenson suggested to patch utils.c which is the only  
} source file in zsh making use of isprint(c) which delivers incorrect  
} results for NEXTSTEP.

I presume you also did `setopt PRINTEIGHTBIT` after applying that patch?
Just checking ...

} I tried that, but it didn't change a thing. Obviously there is  
} another part of the zsh source that's limiting zsh's behaviour; it's  
} definitely *not* the isprint(c) part (which would be easy to  
} replace by the correct NXIsPrint(c) for the NEXTSTEP version, BTW).

Is the symptom that the characters display as \M-a, or simply that they
lose the umlauts?  What's your `stty -a` output?

} Does anyone have an idea where to look?

With some grepping, I can't find any output of file names that doesn't
pass through the right functions, except fprintdir() which seems to always
output the raw string (so it can't be accidentally \M-ifying it).

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: German Umlauts still don't work

[Zoltan Hidvegi]

> } Peter Stephenson suggested to patch utils.c which is the only  
> } source file in zsh making use of isprint(c) which delivers incorrect  
> } results for NEXTSTEP.
[...]
> } Does anyone have an idea where to look?

Yes.

> With some grepping, I can't find any output of file names that doesn't
> pass through the right functions, except fprintdir() which seems to always
> output the raw string (so it can't be accidentally \M-ifying it).

Actually, many functions use icntrl(), especially zle_refresh.c.  I
wanted to mention this right after Peter posted his patch, but I just
wanted to post the right fix with this answer.  Sorry.  I think that this
whole chechking stuff should be done with typtab, and typtab should be
modified according to the localle and the printeightbit option.  And
isprint should only be used to set up typtab.  Probably we can set up
typtab so that we can assume that isprint() is equivalent to !ictrl().

Zoltan

Re: German Umlauts still don't work

[Uli Zappe]

Bart Schaefer wrote:

> I presume you also did `setopt PRINTEIGHTBIT` after applying that
> patch? Just checking ...

Yes I did :-)

> Is the symptom that the characters display as \M-a, or simply
> that they lose the umlauts?

The characters are lost (but only <AE>, <OE> and <UE> (the capital  
ones), <ae>, <oe>, <ue> and <ss> work).

Zoltan Hidvegi wrote:

> Actually, many functions use icntrl(), especially zle_refresh.c.
> I wanted to mention this right after Peter posted his patch, but
> I just wanted to post the right fix with this answer.  Sorry.

Can we still hope for that patch? ;-)

>  I think that this whole chechking stuff should be done with
> typtab, and typtab should be modified according to the localle
> and the printeightbit option.  And isprint should only be used
> to set up typtab.  Probably we can set up typtab so that we can
> assume that isprint() is equivalent to !ictrl().

>From a NEXTSTEP point of view, it is certainly preferable to make  
use of the correct NXIsPrint (or NXIsCntrl) function instead of  
permitting *all* 8 bit characters via a printeightbit option.

I can't judge how this is with other operating systems.

Maybe it's possible to make this configurable so that for every  
operating system one can choose whatever way is best for it to  
decide what is printable?


                Bye
                        Uli

_____________________________________________________________________

Uli Zappe               E-Mail: uli@tallowcross.uni-frankfurt.de
                                (NeXTMail,Mime,ASCII) PGP on request
Lorscher Strasse 5      WWW:    -
D-60489 Frankfurt       Fon:    +49 (69) 9784 0007
Germany                 Fax:    +49 (69) 9784 0042

staff member of NEXTTOYOU - the German NEXTSTEP/OPENSTEP magazine
_____________________________________________________________________

Re: FAQ, German Umlauts (patch)

[Peter Stephenson]

Amol Deshpande wrote:
> Peter,
> Should this also be applicable to 3.0.x ? Also, would printing eight-bit
> characters  be affected by inittyptab ? 
> thanks,

Here's a patch for zsh 3.0.x.  The internal zsh table (set up in
ininittyptab()) isn't used in this case; isprint() is used in order to
give the system a chance of deciding what's printable.

*** Doc/zshoptions.man.p8	Mon Oct 21 21:01:15 1996
--- Doc/zshoptions.man	Mon Jun 30 09:36:36 1997
***************
*** 416,421 ****
--- 416,426 ----
  .BR trap " and"
  .BR unset .
  .TP
+ \fBPRINT_EIGHT_BIT\fP
+ Print eight bit characters literally in completion lists, etc.
+ This option is not necessary if your system correctly returns the
+ printability of eight bit characters (see ctype(3)).
+ .TP
  \fBPRINT_EXIT_VALUE\fP (\-\fB1\fP)
  Print the exit value of programs with non-zero exit status.
  .TP
*** Src/globals.h.p8	Tue Jun  3 07:11:26 1997
--- Src/globals.h	Mon Jun 30 10:04:01 1997
***************
*** 762,767 ****
--- 762,768 ----
      {"overstrike", 		0,    0,    0},
      {"pathdirs", 		'Q',  0,    0},
      {"posixbuiltins",		0,    0,    OPT_EMULATE|OPT_BOURNE},
+     {"printeightbit", 		0,    0,    0},
      {"printexitvalue", 		'1',  0,    0},
      {"privileged", 		'p',  'p',  OPT_SPECIAL},
      {"promptcr", 		x'V', 0,    OPT_ALL},
*** Src/utils.c.p8	Tue Jun  3 07:11:28 1997
--- Src/utils.c	Mon Jun 30 09:39:48 1997
***************
*** 174,179 ****
--- 174,181 ----
      if (isprint(c))
  	goto done;
      if (c & 0x80) {
+ 	if (isset(PRINTEIGHTBIT))
+ 	    goto done;
  	*s++ = '\\';
  	*s++ = 'M';
  	*s++ = '-';
*** Src/zsh.h.p8	Tue Jun  3 07:11:28 1997
--- Src/zsh.h	Mon Jun 30 09:37:01 1997
***************
*** 1124,1129 ****
--- 1124,1130 ----
      OVERSTRIKE,
      PATHDIRS,
      POSIXBUILTINS,
+     PRINTEIGHTBIT,
      PRINTEXITVALUE,
      PRIVILEGED,
      PROMPTCR,

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutsches Elektronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

RE: FAQ, German Umlauts (patch)

[Amol Deshpande]

Peter,
Should this also be applicable to 3.0.x ? Also, would printing eight-bit
characters  be affected by inittyptab ? 
thanks,
-amol

> *** Doc/Zsh/options.yo.p8	Wed Apr 30 09:18:38 1997
> --- Doc/Zsh/options.yo	Thu Jun 26 18:33:51 1997
> ***************
> *** 622,627 ****
> --- 622,634 ----
>   tt(trap) and
>   tt(unset).
>   )
> + pindex(PRINT_EIGHT_BIT)
> + cindex(exit status, printing)
> + item(tt(PRINT_EIGHT_BIT) (tt(-1)))(
> + Print eight bit characters literally in completion lists, etc.
> + This option is not necessary if your system correctly returns the
> + printability of eight bit characters (see manref(ctype)(3)).
> + )
>   pindex(PRINT_EXIT_VALUE)
>   cindex(exit status, printing)
>   item(tt(PRINT_EXIT_VALUE) (tt(-1)))(
> *** Src/options.c.p8	Thu Jun 26 18:17:28 1997
> --- Src/options.c	Thu Jun 26 18:14:26 1997
> ***************
> *** 144,149 ****
> --- 144,150 ----
>   {NULL, "overstrike",	      0,
> OVERSTRIKE},
>   {NULL, "pathdirs",	      0,			 PATHDIRS},
>   {NULL, "posixbuiltins",	      OPT_EMULATE|OPT_BOURNE,
> POSIXBUILTINS},
> + {NULL, "printeightbit",       0,
> PRINTEIGHTBIT},
>   {NULL, "printexitvalue",      0,
> PRINTEXITVALUE},
>   {NULL, "privileged",	      OPT_SPECIAL,
> PRIVILEGED},
>   {NULL, "promptcr",	      OPT_ALL,			 PROMPTCR},
> *** Src/utils.c.p8	Thu Jun 26 18:17:18 1997
> --- Src/utils.c	Thu Jun 26 18:39:11 1997
> ***************
> *** 175,180 ****
> --- 175,182 ----
>       if (isprint(c))
>   	goto done;
>       if (c & 0x80) {
> + 	if (isset(PRINTEIGHTBIT))
> + 	    goto done;
>   	*s++ = '\\';
>   	*s++ = 'M';
>   	*s++ = '-';
> *** Src/zsh.h.p8	Tue Jun 24 09:09:11 1997
> --- Src/zsh.h	Fri Jun 27 09:35:25 1997
> ***************
> *** 1035,1040 ****
> --- 1035,1041 ----
>       OVERSTRIKE,
>       PATHDIRS,
>       POSIXBUILTINS,
> +     PRINTEIGHTBIT,
>       PRINTEXITVALUE,
>       PRIVILEGED,
>       PROMPTCR,
> 
> 
> -- 
> Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
> WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
> Deutsches Elektronen-Synchrotron --- Institut fuer Hochenergiephysik
> Zeuthen
> DESY-IfH, 15735 Zeuthen, Germany.
>