Re: zsh changing cp1250 letters beyond recognition

Re: zsh changing cp1250 letters beyond recognition

[Peter Stephenson]

> I have written simple script (attached) that gets filenames (in $*).
> The filenames possibly contain letters in cp1250 (windows encoding for
> middle and eastern Europe) or iso-8859-2.
> 
> Now, if I use bash to run the script, everything works just fine. But
> when I replace #!/bin/bash with #!/bin/zsh, all those strange cp1250
> characters (\232\235\236\212\215\216) get replaced with other (and
> equally strange) ones. Renaming and tr don't work correctly then.

There's a bug that characters in this range present on the command line
don't get turned properly into zsh's internal representation.  Thanks for
spotting this --- it's very hard to see these things if you're using
iso-8859-1, as most of us presumably are.

You didn't say what version you're using, but the problem is there in both
3.0.8 and 3.1.9.  The patch below is for 3.1.9, I'll send a separate one
for 3.0.8 (the fix is the same, but the context for the patch is
different).

Details for those interested:  the type table doesn't get set up until well
on into initialisation, while the command line arguments called metafy()
straight away, which uses the type table to decide what needs to be
metafied.  I don't dare reorder all the stuff at the start, since there
always turns out to be unpleasant dependencies, so I've done the least
invasive surgery I can think of, which is simply set up the IMETA bit of
the type table right at the start and then initialise the whole thing
properly as before.  I think the ideal solution would be something like
this: set up the bits of the type table which never change at this point,
then for the rest of the life of the shell only meddle with the bits which
can change according to options.  Contributions invited.

This needs to be in one of the tests, too.  Probably we need a special
metafication test.

Index: Src/main.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/main.c,v
retrieving revision 1.1.1.6
diff -u -r1.1.1.6 main.c
--- Src/main.c	2000/02/23 15:18:44	1.1.1.6
+++ Src/main.c	2000/06/08 09:37:31
@@ -35,11 +35,23 @@
 main(int argc, char **argv)
 {
     char **t;
+    int t0;
 #ifdef USE_LOCALE
     setlocale(LC_ALL, "");
 #endif
 
     init_hackzero(argv, environ);
+
+    /*
+     * Provisionally set up the type table to allow metafication.
+     * This will be done properly when we have decided if we are
+     * interactive
+     */
+    typtab['\0'] |= IMETA;
+    typtab[STOUC(Meta)  ] |= IMETA;
+    typtab[STOUC(Marker)] |= IMETA;
+    for (t0 = (int)STOUC(Pound); t0 <= (int)STOUC(Nularg); t0++)
+	typtab[t0] |= ITOK | IMETA;
 
     for (t = argv; *t; *t = metafy(*t, -1, META_ALLOC), t++);
 
-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: zsh changing cp1250 letters beyond recognition

[Peter Stephenson]

> There's a bug that characters in this range present on the command line
> don't get turned properly into zsh's internal representation.
>...
> I'll send a separate one
> for 3.0.8 (the fix is the same, but the context for the patch is
> different).

Here's the patch for 3.0.8.

--- Src/init.c.old	Thu Jun  8 10:33:34 2000
+++ Src/init.c	Thu Jun  8 10:35:22 2000
@@ -37,11 +37,23 @@
 main(int argc, char **argv)
 {
     char **t;
+    int t0;
 #ifdef USE_LOCALE
     setlocale(LC_ALL, "");
 #endif
 
     global_permalloc();
+
+    /*
+     * Provisionally set up the type table to allow metafication.
+     * This will be done properly when we have decided if we are
+     * interactive
+     */
+    typtab['\0'] |= IMETA;
+    typtab[STOUC(Meta)  ] |= IMETA;
+    typtab[STOUC(Marker)] |= IMETA;
+    for (t0 = (int)STOUC(Pound); t0 <= (int)STOUC(Nularg); t0++)
+	typtab[t0] |= ITOK | IMETA;
 
     for (t = argv; *t; *t = metafy(*t, -1, META_ALLOC), t++);
 

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: zsh changing cp1250 letters beyond recognition

[Bart Schaefer]

On Jun 8, 10:55am, Peter Stephenson wrote:
} Subject: Re: zsh changing cp1250 letters beyond recognition
}
} Here's the patch for 3.0.8.

I don't consider this patch by itself critical enough to do a 3.0.9.

Therefore, what I've done is this:

I created a "Patches for old releases" category in the patch manager on
the zsh sourceforge.net project, and posted Peter's patch there.

I'll also be adding the zsh.texi patch for the @' foolishness.

The status is "Open" because the sourceforge UI only displays open,
closed, or postponed patches, not accepted, rejected, or out-of-date
ones, and displays open ones by default. I want these patches to be
visible.

My intention is to keep this patch manager category up to date with
anything that comes across the mailing lists that is applicable to
3.0.8. If a sufficient number of them accumulates before 4.0 goes out,
or if any particularly critical ones appear, I'll consider doing a
3.0.9; otherwise, the patch manager should be considered the canonical
repository of "approved" patches for the 3.0 series.

Since anyone can submit a patch in any category, the way to tell an
"approved" patch from a random submission is that the "assigned to"
column of an approved patch will be "barts".

Peter, if you'd throw something about this into the next FAQ, that'd
be great.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net