crash in tabcompleting

crash in tabcompleting

[Mikael Magnusson]

I'm using latest cvs (as of 20th march) and I'm getting segfaults when
i tabcomplete some files with utf-8 names. I'm not running in a UTF-8
locale though, and I think this worked earlier. I have in one
directory two files called
"大å¡\232æ\204\233 - 大好ã\201\215ã\201\237ã\202\231ã\202\210ã\200\202.avi"
and
"大å¡\232æ\204\233 - é\207\221é­\232è\212±ç\201«.mpg"
(listed with ls --quoting-style=c)

if i try to tabcomplete in that directory, i get the following backtrace:
If i'm leaving some vital information out, please say so and i will do
my best to help.

Program received signal SIGSEGV, Segmentation fault.
ztrsub (t=0x82ac9fe "", s=0x82d9000 <Address 0x82d9000 out of bounds>)
at utils.c:2992
2992		if (*s++ == Meta) {
(gdb) bt
#0  ztrsub (t=0x82ac9fe "", s=0x82d9000 <Address 0x82d9000 out of
bounds>) at utils.c:2992
#1  0x080c5d2a in pattryrefs (prog=0x811e9b8, string=0x442fc700 "",
stringlen=6, patoffset=0,
    nump=0x0, begp=0x0, endp=0x0) at pattern.c:1567
#2  0x080c6420 in pattry (prog=0x0, string=0x0) at pattern.c:1499
#3  0xb7d7bf94 in do_comp_vars (test=186, na=1, sa=0xb7d29928 "\213",
nb=137021950,
    sb=0xb7d29928 "\213", mod=1) at complete.c:829
#4  0xb7d7c255 in bin_compset (name=0xb7d29928 "\213", argv=0x0,
ops=0xbffee4a0, func=0)
    at complete.c:939
#5  0x08053b54 in execbuiltin (args=0xbffee4a0, bn=0xb7d93ba4) at builtin.c:439
#6  0x08070c8e in execcmd (state=0xbfffdec0, input=0, output=0,
how=18, last1=2) at exec.c:2435
#7  0x08071865 in execpline2 (state=0xbfffdec0, pcode=0, how=18,
input=0, output=0, last1=0)
    at exec.c:1283
#8  0x08071e71 in execpline (state=0xbfffdec0, slcode=1143981824,
how=18, last1=0) at exec.c:1069
#9  0x08073833 in execlist (state=0xbfffdec0, dont_change_job=1,
exiting=0) at exec.c:875
#10 0x0809a6f0 in execif (state=0xbfffdec0, do_exec=0) at loop.c:505
#11 0x08070027 in execcmd (state=0xbfffdec0, input=0, output=0,
how=18, last1=2) at exec.c:2382
#12 0x08071865 in execpline2 (state=0xbfffdec0, pcode=0, how=18,
input=0, output=0, last1=0)
    at exec.c:1283
#13 0x08071e71 in execpline (state=0xbfffdec0, slcode=1143981824,
how=18, last1=0) at exec.c:1069
#14 0x08073833 in execlist (state=0xbfffdec0, dont_change_job=1,
exiting=0) at exec.c:875
#15 0x0809a799 in execif (state=0xbfffdec0, do_exec=0) at loop.c:520
#16 0x08070027 in execcmd (state=0xbfffdec0, input=0, output=0, how=2,
last1=2) at exec.c:2382
#17 0x08071865 in execpline2 (state=0xbfffdec0, pcode=0, how=2,
input=0, output=0, last1=0)
    at exec.c:1283
#18 0x08071e71 in execpline (state=0xbfffdec0, slcode=1143981824,
how=2, last1=0) at exec.c:1069
#19 0x08073833 in execlist (state=0xbfffdec0, dont_change_job=1,
exiting=0) at exec.c:875
#20 0x08073b6a in runshfunc (prog=0x82a7c28, wrap=0x82a7c28,
name=0xb7d28030 "_main_complete")
    at exec.c:775
#21 0xb7d7ccba in comp_wrapper (prog=0x0, w=0x0, name=0x0) at complete.c:1298
---Type <return> to continue, or q <return> to quit---
#22 0x08073f6a in doshfunc (name=0x814b918 "_main_complete",
prog=0x82a7c28, doshargs=0xb7d28030,
    flags=0, noreturnval=0) at exec.c:3692
#23 0xb7d842b6 in makecomplist (s=0x8151d70 "大å¡\203ºæ\203¤\203» -
", incmd=0, lst=0)
    at compcore.c:787
#24 0xb7d85368 in do_completion (dummy=0xb7dc7e74, dat=0x442fc700) at
compcore.c:342
#25 0xb7db7ef4 in docomplete (lst=0) at zle_tricky.c:1845
#26 0xb7db94cf in completeword (args=0x0) at zle_tricky.c:208
#27 0xb7db3a09 in completecall (args=0x442fc700) at zle_tricky.c:184
#28 0xb7da7042 in execzlefunc (func=0xb7dc5330, args=0xb7dc817c) at
zle_main.c:1053
#29 0xb7da7868 in zlecore () at zle_main.c:828
#30 0xb7da829a in zleread (lp=0x810ab90, rp=0x0, flags=0, context=0)
at zle_main.c:982
#31 0x0808ac74 in inputline () at input.c:278
#32 0x0808b31a in ingetc () at input.c:214
#33 0x08081777 in ihgetc () at hist.c:240
#34 0x08097126 in gettok () at lex.c:627
#35 0x08098e86 in yylex () at lex.c:343
#36 0x080bba5a in parse_event () at parse.c:449
#37 0x08087c38 in loop (toplevel=1, justonce=0) at init.c:128
#38 0x0808a854 in zsh_main (argc=1, argv=0xbfffeca4) at init.c:1282
#39 0x080534ce in main (argc=0, argv=0x0) at main.c:93

-- 
Mikael Magnusson

Re: crash in tabcompleting

[Peter Stephenson]

Mikael Magnusson wrote:
> I'm using latest cvs (as of 20th march) and I'm getting segfaults when
> i tabcomplete some files with utf-8 names. I'm not running in a UTF-8
> locale though, and I think this worked earlier. I have in one
> directory two files called
> "=E5=A4=A7=E5=A1\232=E6\204\233 - =E5=A4=A7=E5=A5=BD=E3\201\215=E3\201\237=
> =E3\202\231=E3\202\210=E3\200\202.avi"
> and
> "=E5=A4=A7=E5=A1\232=E6\204\233 - =E9\207\221=E9=AD\232=E8\212=B1=E7\201=AB=
> .mpg"
> (listed with ls --quoting-style=3Dc)

(Deliberately left as undigested quoted printable for viewers in ASCII.)

This looks fishy, but I can't get it to happen.  I can get some weird
and wonderful display effects, but that's not surprising.  Are you able
to create a script including only 7-bit characters that starts from zsh
-f, creates appropriate files (using escapes such as e.g. $'\xe5') that
sets up the shell to show the problem?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: crash in tabcompleting

[Peter Stephenson]

Mikael Magnusson wrote:
> It seems one file is enough. I also failed to mention that zsh doesn't
> crash if there are any other files in the same directory. It has to be
> only ones which cause the problem I think. (not sure exactly what that
> is though).
> This script succeeds in creating the file for me:
> 
> 
> a='+WSdYWmEb - +WSdZfTBNMF8wmTCIMAI.avi'
> b="`echo $a|iconv -f utf7 -t utf8`"
> touch "$b"

This is good enough for me to see that somewhere the completion system
is messing up the use of metafied characters: with debugging turned on,
there's an error message from "ztrsub" because there's a Meta at the end
of the variable.

Somewhere in the undocmented morass it generates a "compprefix" without
checking properly if the string contains Meta characters and truncates
it incorrectly.  There are lots of ways of checking compprefix.  Someone
is going to have to trace which it is.  We had something similar before
and it took a lot of tracking down.

This is even before we've thought about handling multibyte characters.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: crash in tabcompleting

[Peter Stephenson]

Peter Stephenson wrote:
> Mikael Magnusson wrote:
> > It seems one file is enough. I also failed to mention that zsh doesn't
> > crash if there are any other files in the same directory. It has to be
> > only ones which cause the problem I think. (not sure exactly what that
> > is though).
> > This script succeeds in creating the file for me:
> > 
> > 
> > a='+WSdYWmEb - +WSdZfTBNMF8wmTCIMAI.avi'
> > b="`echo $a|iconv -f utf7 -t utf8`"
> > touch "$b"
> 
> This is good enough for me to see that somewhere the completion system
> is messing up the use of metafied characters: with debugging turned on,
> there's an error message from "ztrsub" because there's a Meta at the end
> of the variable.

It looks like the code to move along compprefix (and compsuffix, though
that didn't seem to show up here) didn't take account of Meta
characters.  I'd guess there's much, much more like this.  This has been
there for ages, but it's liable to show up in different ways.

Index: Src/Zle/compcore.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/compcore.c,v
retrieving revision 1.69
diff -u -r1.69 compcore.c
--- Src/Zle/compcore.c	14 Jan 2005 13:05:22 -0000	1.69
+++ Src/Zle/compcore.c	21 Mar 2005 18:38:44 -0000
@@ -1532,8 +1532,8 @@
 	    untokenize(ss);
 	    compsuffix = ztrdup(ss);
 	}
-        if ((i = strlen(compprefix)) &&
-            compprefix[i - 1] == '\\' && compprefix[i - 2] != '\\')
+        if ((i = strlen(compprefix)) > 1 && compprefix[i - 1] == '\\' &&
+	    compprefix[i - 2] != '\\' && compprefix[i - 2] != Meta)
             compprefix[i - 1] = '\0';
         
 	tmp = tricat(compqiprefix, compiprefix, multiquote(qp, 1));
Index: Src/Zle/complete.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/complete.c,v
retrieving revision 1.27
diff -u -r1.27 complete.c
--- Src/Zle/complete.c	7 Dec 2004 16:55:11 -0000	1.27
+++ Src/Zle/complete.c	21 Mar 2005 18:38:44 -0000
@@ -821,18 +821,32 @@
 		    add = -1;
 		} else {
 		    p = compprefix + 1;
+		    if (*p == Meta)
+			p++;
 		    add = 1;
 		}
-		for (; l; l--, p += add) {
+		for (;;) {
 		    sav = *p;
 		    *p = '\0';
 		    test = pattry(pp, compprefix);
 		    *p = sav;
 		    if (test && !--na)
 			break;
+		    if (add > 0) {
+			if (p == compprefix + l)
+			    return 0;
+			if (*p == Meta)
+			    p += 2;
+			else
+			    p++;
+		    } else {
+			if (p == compprefix)
+			    return 0;
+			p--;
+			if (p > compprefix && p[-1] == Meta)
+			    p--;
+		    }
 		}
-		if (!l)
-		    return 0;
 		if (mod)
 		    ignore_prefix(p - compprefix);
 	    } else {
@@ -847,14 +861,30 @@
 		    add = 1;
 		} else {
 		    p = compsuffix + l - 1;
+		    if (p > compsuffix && p[-1] == Meta)
+			p--;
 		    add = -1;
 		}
-		for (; l; l--, p += add)
+		for (;;) {
 		    if (pattry(pp, p) && !--na)
 			break;
 
-		if (!l)
-		    return 0;
+		    if (add > 0) {
+			if (p == compsuffix + l)
+			    return 0;
+			if (*p == Meta)
+			    p += 2;
+			else
+			    p++;
+		    } else {
+			if (p == compsuffix)
+			    return 0;
+			p--;
+			if (p > compsuffix && p[-1] == Meta)
+			    p--;
+		    }
+		}
+
 		if (mod)
 		    ignore_suffix(ol - (p - compsuffix));
 	    }

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************