Re: completion bug in UTF-8 locale

Re: completion bug in UTF-8 locale

[Peter Stephenson]

On Sun, 20 Aug 2006 10:18:33 +0300
Roman Cheplyaka <roman.cheplyaka@gmail.com> wrote:
> I have the following issue in UTF-8 locale:
> zsh completion works ok in current folder, no matter which files are
> here.
> But suppose I have some directory named 'тест' (cyrillic letters).
> Then
> % ls те<Tab>
> completes it to 'тест', but
> % ls тест/<Tab>
> doesn't show anything, even if there are no cyrillic-named items in that
> directory.

I can't get this, even with zsh -f; it seems to work fine both with my
own completion set up and the plain old-fashioned defaul.  I'm using the
locale en_GB.UTF-8, but I tried it with ru_RU.UTF-8 as well.  What
actually are the contents of the directory (ls -a)?  What locale are you
using?  What version of the shell?  (UTF-8 support is new in 3.2, so if
it's older it won't work well.)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: completion bug in UTF-8 locale

[Roman Cheplyaka]

On Sun, Aug 20, 2006 at 06:01:51PM +0100, Peter Stephenson wrote:
> On Sun, 20 Aug 2006 10:18:33 +0300
> Roman Cheplyaka <roman.cheplyaka@gmail.com> wrote:
> > I have the following issue in UTF-8 locale:
> > zsh completion works ok in current folder, no matter which files are
> > here.
> > But suppose I have some directory named 'тест' (cyrillic letters).
> > Then
> > % ls те<Tab>
> > completes it to 'тест', but
> > % ls тест/<Tab>
> > doesn't show anything, even if there are no cyrillic-named items in that
> > directory.
> I can't get this, even with zsh -f; it seems to work fine both with my
> own completion set up and the plain old-fashioned defaul.
I've made some more tests. It seems to break at certain letters.. Try for
example directory named "ц" (w/o quotes). 


> I'm using the
> locale en_GB.UTF-8, but I tried it with ru_RU.UTF-8 as well.  What
> actually are the contents of the directory (ls -a)?
It's irrelevant, I think. Ok, here it is:
% ls -a ц
./  ../  aaaa

> What locale are you
> using?
% locale   
LANG=POSIX
LC_CTYPE=ru_UA.UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE=ru_UA.UTF-8
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=


> What version of the shell?  (UTF-8 support is new in 3.2, so if
> it's older it won't work well.)
% zsh --version
zsh 4.3.2 (i686-pc-linux-gnu)
% dpkg-query -W zsh 
zsh	4.3.2-14

-- 
Roman I. Cheplyaka

Re: completion bug in UTF-8 locale

[Peter Stephenson]

On Sun, 20 Aug 2006 23:08:08 +0300
Roman Cheplyaka <roman.cheplyaka@gmail.com> wrote:
> I've made some more tests. It seems to break at certain letters.. Try for
> example directory named "ц" (w/o quotes). 

Aha.  That needs one of the bytes to be encoded as a "Meta" byte (it
overlaps with zsh's range of internal tokens).

That's still broken in the patches I made for listing multibyte
characters quite recently; the following should fix it.  This means it's
not surprising it's buggy in 4.3.2.

Would you be able to try it with the code currently in CVS?  Otherwise
I'll probably make a pre-4.3.3 test version shortly.

In the following patch I've assumed that any null-terminated string must
be metafied.  That's the usual rule, but nothing would surprise me about
the completion code.

Index: Src/Zle/complist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/complist.c,v
retrieving revision 1.94
diff -u -r1.94 complist.c
--- Src/Zle/complist.c	17 Aug 2006 09:34:12 -0000	1.94
+++ Src/Zle/complist.c	20 Aug 2006 22:17:13 -0000
@@ -621,7 +621,11 @@
 	if (ml == mlend - 1 && (cc % columns) == columns - 1)
 	    return 0;
 
-	putc(*p, shout);
+	if (*p == Meta) {
+	    p++;
+	    putc(*p ^ 32, shout);
+	} else
+	    putc(*p, shout);
 	if ((beg = !(cc % columns)))
 	    ml++;
 	if (mscroll && !(cc % columns) &&
@@ -1137,8 +1141,14 @@
 		    dopr = 0;
 		    continue;
 		}
-		while (len--)
-		    putc(*p++, shout);
+		while (len--) {
+		    if (*p == Meta) {
+			len--;
+			p++;
+			putc(*p++ ^ 32, shout);
+		    } else
+			putc(*p++, shout);
+		}
 		if ((beg = !(cc % columns)) && !stat) {
 		    ml++;
                     fputs(" \010", shout);
Index: Src/Zle/zle_tricky.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_tricky.c,v
retrieving revision 1.74
diff -u -r1.74 zle_tricky.c
--- Src/Zle/zle_tricky.c	20 Aug 2006 18:07:49 -0000	1.74
+++ Src/Zle/zle_tricky.c	20 Aug 2006 22:17:15 -0000
@@ -2123,9 +2123,15 @@
 			tcout(TCUNDERLINEEND);
 		    break;
 		case '{':
-		    for (p++; *p && (*p != '%' || p[1] != '}'); p++)
-			if (dopr)
+		    for (p++; *p && (*p != '%' || p[1] != '}'); p++) {
+			if (*p == Meta) {
+			    p++;
+			    if (dopr) 
+				putc(*p ^ 32, shout);
+			}
+			else if (dopr)
 			    putc(*p, shout);
+		    }
 		    if (*p)
 			p++;
 		    else
@@ -2164,8 +2170,14 @@
 		convchar_t cchar;
 		int clen = MB_METACHARLENCONV(p, &cchar);
 		if (dopr) {
-		    while (clen--)
-			putc(*p++, shout);
+		    while (clen--) {
+			if (*p == Meta) {
+			    p++;
+			    clen--;
+			    putc(*p++ ^ 32, shout);
+			} else
+			    putc(*p++, shout);
+		    }
 		} else
 		    p += clen;
 		cc += WCWIDTH(cchar);

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: completion bug in UTF-8 locale

[David Gómez]

Hi Peter,

On Aug 20 at 11:18:38, Peter Stephenson wrote:
> Roman Cheplyaka <roman.cheplyaka@gmail.com> wrote:
> > I've made some more tests. It seems to break at certain letters.. Try for
> > example directory named "ц" (w/o quotes). 
> 
> Aha.  That needs one of the bytes to be encoded as a "Meta" byte (it
> overlaps with zsh's range of internal tokens).
> 
> That's still broken in the patches I made for listing multibyte
> characters quite recently; the following should fix it.  This means it's
> not surprising it's buggy in 4.3.2.

I'm testing latest CVS and the completion bug is still present. My locale
is different from Roman's one (es_ES.UTF-8) and my tests are with japanese
characters:

[~] % ls 花 
aaaa  bbbb

And 

[~] % cd 花/<Tab>

doesn't complete.


cheers,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: completion bug in UTF-8 locale

[Peter Stephenson]

On Sat, 26 Aug 2006 20:24:33 +0200
David Gómez <david@pleyades.net> wrote:
> Hi Peter,
> I'm testing latest CVS and the completion bug is still present. My locale
> is different from Roman's one (es_ES.UTF-8) and my tests are with japanese
> characters:
> 
> [~] % ls 花 
> aaaa  bbbb
> 
> And 
> 
> [~] % cd 花/<Tab>
> 
> doesn't complete.

Is this after plain "zsh -f"?  I can see the problem there, although I
may not have a chance to look at it for a while since I'm in the middle
of some holiday (it's probably a fairly trivial missing piece of Meta
handling again).  This isn't too surprising since I haven't been paying
much attention to old-style completion.

Or is this after compinit, possibly with some styles set?  I can't see
any problem there at the moment (and unless aaaa and bbbb are
directories they wouldn't complete at that point).

Generally speaking, the exact context is crucial in looking at completion
bugs.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: completion bug in UTF-8 locale

[Peter Stephenson]

Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Sat, 26 Aug 2006 20:24:33 +0200
> David Gómez <david@pleyades.net> wrote:
> > Hi Peter,
> > I'm testing latest CVS and the completion bug is still present. My locale
> > is different from Roman's one (es_ES.UTF-8) and my tests are with japanese
> > characters:
> > 
> > [~] % ls 花 
> > aaaa  bbbb
> > 
> > And 
> > 
> > [~] % cd 花/<Tab>
> > 
> > doesn't complete.
> 
> Is this after plain "zsh -f"?  I can see the problem there, although I
> may not have a chance to look at it for a while since I'm in the middle
> of some holiday (it's probably a fairly trivial missing piece of Meta
> handling again).

This fixes the problem in this case, although I still don't know if this is
the only problem.  This isn't new with MULTIBYTE_SUPPORT, although in the
case of this character you'd be seeing other strange effects if you didn't
have that turned on.

Index: Src/Zle/compctl.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/compctl.c,v
retrieving revision 1.27
diff -u -r1.27 compctl.c
--- Src/Zle/compctl.c	30 May 2006 22:35:04 -0000	1.27
+++ Src/Zle/compctl.c	30 Aug 2006 15:50:33 -0000
@@ -2127,9 +2127,9 @@
 {
     DIR *d;
     struct stat buf;
-    char *n, p[PATH_MAX], *q = NULL, *e;
+    char *n, p[PATH_MAX], *q = NULL, *e, *pathpref;
     LinkList l = NULL;
-    int ns = 0, ng = opts[NULLGLOB], test, aw = addwhat;
+    int ns = 0, ng = opts[NULLGLOB], test, aw = addwhat, pathpreflen;
 
     opts[NULLGLOB] = 1;
 
@@ -2145,12 +2145,18 @@
 	all = execs = 0;
     }
     /* Open directory. */
-    if ((d = opendir((prpre && *prpre) ? prpre : "."))) {
+    if (prpre && *prpre) {
+	pathpref = dupstring(prpre);
+	unmetafy(pathpref, &pathpreflen);
+    } else {
+	pathpref = NULL;
+	pathpreflen = 0;
+    }
+    if ((d = opendir(pathpref ? pathpref : "."))) {
 	/* If we search only special files, prepare a path buffer for stat. */
-	if (!all && prpre) {
-	    strcpy(p, prpre);
-	    q = p + strlen(prpre);
-	}
+	if (!all && pathpreflen)
+	    memcpy(p, pathpref, pathpreflen);
+	q = p + pathpreflen;
 	/* Fine, now read the directory. */
 	while ((n = zreaddir(d, 1)) && !errflag) {
 	    /* Ignore files beginning with `.' unless the thing we found on *

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: completion bug in UTF-8 locale

[David Gómez]

Hi Peter,

On Aug 30 at 04:54:15, Peter Stephenson wrote:
> > Is this after plain "zsh -f"?  I can see the problem there, although I
> > may not have a chance to look at it for a while since I'm in the middle
> > of some holiday (it's probably a fairly trivial missing piece of Meta
> > handling again).
> 
> This fixes the problem in this case, although I still don't know if this is
> the only problem.  This isn't new with MULTIBYTE_SUPPORT, although in the
> case of this character you'd be seeing other strange effects if you didn't
> have that turned on.

Great! That patch did the trick. I've tested it with another directories with
different characters and with subdirectories and completion works in all cases.
And yes, i was using old style completion, with no compinit at all loaded.

Thanks for fixing this in your holiday ;-)

cheers,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org