PATCH: fix 4, was Re: unpatch: metafying zle line

PATCH: fix 4, was Re: unpatch: metafying zle line

[Peter Stephenson]

Conversion of the cursor position when metafying and unmetafying the
line was broken in both directions when the cursor was at the end of the
line: this is the bug David Gomez noticed.

After this, completion is basically working with ZLE_UNICODE_SUPPORT
defined when you stick to using single-byte characters.  It's almost
time to allow that to be enabled by default, which will help picking up
the remaining problems with multi-byte characters:  those are likely to
be widespread, but probably in most cases fixable by local changes.

In fact the first test that fails is the third from last, and it's not
catastrophic; a stray space appears at the end of the line in
Y03arguments.ztst test "words array in reset arguments".  There ought to
be a prize for anyone who tracks that down.  But there isn't.

Index: Src/Zle/zle_utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_utils.c,v
retrieving revision 1.26
diff -u -r1.26 zle_utils.c
--- Src/Zle/zle_utils.c	10 Aug 2005 10:56:41 -0000	1.26
+++ Src/Zle/zle_utils.c	12 Aug 2005 10:17:48 -0000
@@ -140,16 +140,19 @@
     s = zalloc(inll * MB_CUR_MAX + 1);
 
     outcs = 0;
-    for(i=0; i < inll; i++, incs--) {
+    for (i=0; i < inll; i++, incs--) {
 	if (incs == 0)
 	    outcs = mb_len;
 	j = wctomb(s + mb_len, instr[i]);
 	if (j == -1) {
 	    /* invalid char; what to do? */
+	    s[mb_len++] = ZWC('?');
 	} else {
 	    mb_len += j;
 	}
     }
+    if (incs == 0)
+	outcs = mb_len;
     s[mb_len] = '\0';
 
     outll = mb_len;
@@ -279,7 +282,7 @@
 	/* Reset shift state to input complete string */
 	memset(&ps, '\0', sizeof(ps));
 
-	while (ll) {
+	while (ll > 0) {
 	    size_t ret = mbrtowc(outptr, inptr, ll, &ps);
 
 	    /*
@@ -311,6 +314,8 @@
 	    outptr++;
 	    ll -= ret;
 	}
+	if (outcs && inptr <= (char *)instr + incs)
+	    *outcs = outptr - outstr;
 	*outll = outptr - outstr;
     } else {
 	*outll = 0;

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: PATCH: fix 4, was Re: unpatch: metafying zle line

[David Gómez]

Hi Peter ;),

On Aug 12 at 11:21:00, Peter Stephenson wrote:
> Conversion of the cursor position when metafying and unmetafying the
> line was broken in both directions when the cursor was at the end of the
> line: this is the bug David Gomez noticed.

Great, i'll update my copy to test it. By the way, since yesterday i've been
using cvs code with ZLE_UNICODE_SUPPORT disabled and i haven't seen any negative
side effects of the new patchs on completion ;).

bye

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: PATCH: fix 4, was Re: unpatch: metafying zle line

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 4840 bytes --]

On Friday 12 August 2005 14:21, Peter Stephenson wrote:
> After this, completion is basically working with ZLE_UNICODE_SUPPORT
> defined when you stick to using single-byte characters.  It's almost
> time to allow that to be enabled by default, which will help picking up
> the remaining problems with multi-byte characters:  those are likely to
> be widespread, but probably in most cases fixable by local changes.
>


Hat off to you, Sir! It is worth more than pizza ...

Multibyte basically works (ru_RU.UTF-8, Mandrake^H^H^Hiva cooker, gcc 4.0.1, 
glibc 2.3.5). I.e. basic line editing (movement char, word, emacs or vi 
mode). Completion - i.e. listing etc.

Some random problems.

1. attempt to use a-a-m-c results in

{pts/2}% ll кккккккк ккккккккккк/home/bor/src/zsh/Src/Zle/compresult.c:690: 
line not metafied
zsh: segmentation fault (core dumped)  pkg/bin/zsh
#0  0xb7bef892 in hasbrpsfx (m=0x8178a50, pre=0x0, suf=0x0)
    at /home/bor/src/zsh/Src/Zle/compresult.c:699
699             memcpy(oline, zlemetaline, zlemetall);
(gdb) bt
#0  0xb7bef892 in hasbrpsfx (m=0x8178a50, pre=0x0, suf=0x0)
    at /home/bor/src/zsh/Src/Zle/compresult.c:699
#1  0xb7bf1e67 in calclist (showall=1)
    at /home/bor/src/zsh/Src/Zle/compresult.c:1489
#2  0xb7f4e3a4 in complistmatches (dummy=0xb7bf6ab0, dat=0xbff669b0)
    at /home/bor/src/zsh/Src/Zle/complist.c:1608
#3  0x080961f4 in runhookdef (h=0xb7bf6ab0, d=0xbff669b0)
    at /home/bor/src/zsh/Src/module.c:1889
#4  0xb7bf4761 in list_matches (dummy=0xb7c2cec0, dummy2=0x0)
    at /home/bor/src/zsh/Src/Zle/compresult.c:2200
#5  0x080961f4 in runhookdef (h=0xb7c2cec0, d=0x0)
    at /home/bor/src/zsh/Src/module.c:1889
#6  0xb7c14a5e in zrefresh ()
    at /home/bor/src/zsh/Src/Zle/zle_refresh.c:752
#7  0xb7f5027c in domenuselect (dummy=0xb7bf6a74, dat=0xbff66fe0)
    at /home/bor/src/zsh/Src/Zle/complist.c:2153
#8  0x0809618f in runhookdef (h=0xb7bf6a74, d=0xbff66fe0)
    at /home/bor/src/zsh/Src/module.c:1883
#9  0xb7bdea62 in after_complete (dummy=0xb7c2cefc, dat=0xbff6706c)
    at /home/bor/src/zsh/Src/Zle/compcore.c:506
#10 0x080961f4 in runhookdef (h=0xb7c2cefc, d=0xbff6706c)
    at /home/bor/src/zsh/Src/module.c:1889
#11 0xb7c19e55 in docomplete (lst=0)
    at /home/bor/src/zsh/Src/Zle/zle_tricky.c:851
#12 0xb7c18544 in completeword (args=0xb7c2d13c)
    at /home/bor/src/zsh/Src/Zle/zle_tricky.c:232
#13 0xb7c18421 in completecall (args=0xb7c2d13c)
    at /home/bor/src/zsh/Src/Zle/zle_tricky.c:208
#14 0xb7c0b09f in execzlefunc (func=0xb7c2b050, args=0xb7c2d13c)
    at /home/bor/src/zsh/Src/Zle/zle_main.c:1063
#15 0xb7c0a62b in zlecore ()
    at /home/bor/src/zsh/Src/Zle/zle_main.c:838
#16 0xb7c0ad1b in zleread (lp=0x80e63e0, rp=0x0, flags=7, context=0)
    at /home/bor/src/zsh/Src/Zle/zle_main.c:992
#17 0x08080ac3 in inputline () at /home/bor/src/zsh/Src/input.c:278
#18 0x08080978 in ingetc () at /home/bor/src/zsh/Src/input.c:214
#19 0x0807775b in ihgetc () at /home/bor/src/zsh/Src/hist.c:240
#20 0x080880f4 in gettok () at /home/bor/src/zsh/Src/lex.c:628
#21 0x0808792c in yylex () at /home/bor/src/zsh/Src/lex.c:344
#22 0x080a1269 in parse_event () at /home/bor/src/zsh/Src/parse.c:451
#23 0x0807d8d6 in loop (toplevel=1, justonce=0)
    at /home/bor/src/zsh/Src/init.c:128
#24 0x080805ba in zsh_main (argc=1, argv=0xbff67614)
    at /home/bor/src/zsh/Src/init.c:1315
#25 0x08052652 in main (argc=1, argv=0xbff67614)
    at /home/bor/src/zsh/Src/main.c:93

apparently it correctly inserts first match but crashes at recursive 
completion attempt.


2. Attempt to complete anything beyond U'd183' results in

{pts/2}% ll уBUG: substring ends in the middle of a metachar in 
ztrsub()mpleting `files'
         ll у
{pts/2}% echo у | xxd
0000000: d183 0a                                  ...

I do not know what is magic about it; all characters before are completed 
normally. Probably it somehow clashes with META characters. The above is in 
UTF-8, it is 0400 unicode plane, character in question is 0443.

3. emacs ^W sometimes removes too much not just a previous word in Russian 
(where word delimiter in this case is space):

{pts/2}% ll    АААААА ББББББББ ВВВВВВВВВ^W
{pts/2}% 

4. Undo (in emacs or vi) often puts cursor not at previous position but at 
some random position. In the above example, if I have anyhing after cursor:

{pts/2}% ll А Б В^W
                 ^cursor here
{pts/2}% ^_
         ^ cursor here
{pts/2}% ll А Б В
            ^ cursor here

the position is really random, sometimes it is end of buffer sometimes more.

But yes, I think it makes sense enable UNICODE by default now. It seems to be 
usable, at least for simple filename completion.

Thank you

-andrey

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH] fix zrefresh recursive completion call

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

On Sunday 14 August 2005 19:55, Andrey Borzenkov wrote:
>
> 1. attempt to use a-a-m-c results in
>
> {pts/2}% ll кккккккк ккккккккккк/home/bor/src/zsh/Src/Zle/compresult.c:690:
> line not metafied
> zsh: segmentation fault (core dumped)  pkg/bin/zsh

This happens with ASCII as well; the attached patch fixes it (zrefresh calls 
completion recursively).

Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.26
diff -u -p -r1.26 zle_refresh.c
--- Src/Zle/zle_refresh.c       11 Aug 2005 18:40:53 -0000      1.26
+++ Src/Zle/zle_refresh.c       15 Aug 2005 17:18:25 -0000
@@ -746,8 +746,13 @@ singlelineout:
        zfree(tmpline, tmpll);

     /* if we have a new list showing, note it; if part of the list has been
-    overwritten, redisplay it. */
+    overwritten, redisplay it. We have to metafy line back before calling
+    completion code */
     if (showinglist == -2 || (showinglist > 0 && showinglist < nlnct)) {
+       if (remetafy) {
+           metafy_line();
+           remetafy = 0;
+       }
        inlist = 1;
        listmatches();
        inlist = 0;

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: PATCH: fix 4, was Re: unpatch: metafying zle line

[Peter Stephenson]

Peter Stephenson <pws@csr.com> wrote:
> In fact the first test that fails is the third from last, and it's not
> catastrophic; a stray space appears at the end of the line in
> Y03arguments.ztst test "words array in reset arguments".  There ought to
> be a prize for anyone who tracks that down.  But there isn't.

I win! (Surprised?)  This was a bug in handling of automatic removing of
suffixes.

This turns on ZLE_UNICODE_SUPPORT by default when the environment supports
it.  For now we require full ISO 10646 support (__STDC_ISO_10646__).  I'm
happy to relax that, but it will need a certain amount---probably a
lot---of probing for features in configure to make sure they are there.
Recent Linux distributions definitely work as things stand, and I would
guess therefore BSD-based ones.  It would be good to make a list if anyone
has more information.  (I don't know whether Solaris 9 is currently
supported, for example; I know 8 isn't yet.)

The problems Andrej's been seeing are likely to be mostly specific to real
wide characters: since the completion code uses multibyte characters, with
normal ASCII it sees exactly what it would have with ZLE_UNICODE_SUPPORT
turned off.  So we will be relying a lot on people who need characters not
in ASCII.

Other fixes in this patch: insert-last-word couldn't be repeated because
the line wasn't handled consistently with the stored previous line.  That's
because the last word comes from parsing the command line and hence is a
metafied character string.  Using metafy_line() here is therefore the
simplest fix.

Also, I've added a warning when "bindkey -m" is used, since this will
completely trash use of non-ASCII characters.  Actually, it always did (you
needed ^V which should still work) but I think it's now worth pointing out
to users that bindkey -m doesn't really fit into the brave new world.

Also, any widget that removes completion suffixes is allowed to manipulate
the line directly so should always run with the line unmetafied.

Index: Src/system.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/system.h,v
retrieving revision 1.32
diff -u -r1.32 system.h
--- Src/system.h	1 Aug 2005 09:55:00 -0000	1.32
+++ Src/system.h	15 Aug 2005 09:34:48 -0000
@@ -705,7 +705,7 @@
  * between wide characters and multibyte strings.
  */
 #if defined(HAVE_MBRTOWC) && defined(HAVE_WCRTOMB)
-/*#define ZLE_UNICODE_SUPPORT	1*/
+#define ZLE_UNICODE_SUPPORT	1
 #endif
 #else
 # ifdef HAVE_LANGINFO_H
Index: Src/Zle/zle_hist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_hist.c,v
retrieving revision 1.26
diff -u -r1.26 zle_hist.c
--- Src/Zle/zle_hist.c	11 Aug 2005 18:40:53 -0000	1.26
+++ Src/Zle/zle_hist.c	15 Aug 2005 09:34:49 -0000
@@ -531,10 +531,11 @@
 	}
     }
 
+    metafy_line();
     if (lastinsert && lastlen &&
-	lastpos <= zlecs &&
-	lastlen == zlecs - lastpos &&
-	memcmp(lastinsert, (char *)&zleline[lastpos], lastlen) == 0)
+	lastpos <= zlemetacs &&
+	lastlen == zlemetacs - lastpos &&
+	memcmp(lastinsert, (char *)&zlemetaline[lastpos], lastlen) == 0)
 	deleteword = 1;
     else
 	lasthist = curhist;
@@ -548,9 +549,9 @@
 	 * confusion.
 	 */
 	if (deleteword) {
-	    int pos = zlecs;
-	    zlecs = lastpos;
-	    foredel(pos - zlecs);
+	    int pos = zlemetacs;
+	    zlemetacs = lastpos;
+	    foredel(pos - zlemetacs);
 	    /*
 	     * Mark that this has been deleted.
 	     * For consistency with history lines, we really ought to
@@ -604,9 +605,9 @@
      * successfully found a new one to insert.
      */
     if (deleteword > 0) {
-	int pos = zlecs;
-	zlecs = lastpos;
-	foredel(pos - zlecs);
+	int pos = zlemetacs;
+	zlemetacs = lastpos;
+	foredel(pos - zlemetacs);
     }
     if (lastinsert) {
 	zfree(lastinsert, lastlen);
@@ -625,13 +626,15 @@
     save = *t;
     *t = '\0';			/* ignore trailing whitespace */
     lasthist = evhist;
-    lastpos = zlecs;
+    lastpos = zlemetacs;
     lastlen = t - s;
     lastinsert = zalloc(t - s);
     memcpy(lastinsert, s, lastlen);
     n = zmult;
     zmult = 1;
 
+    unmetafy_line();
+
     zs = stringaszleline((unsigned char *)s, 0, &len, NULL, NULL);
     doinsert(zs, len);
     free(zs);
Index: Src/Zle/zle_keymap.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_keymap.c,v
retrieving revision 1.16
diff -u -r1.16 zle_keymap.c
--- Src/Zle/zle_keymap.c	18 Feb 2005 13:57:28 -0000	1.16
+++ Src/Zle/zle_keymap.c	15 Aug 2005 09:34:49 -0000
@@ -818,6 +818,10 @@
 	zwarnnam(name, "keymap `%s' is protected", kmname, 0);
 	return 1;
     }
+#ifdef ZLE_UNICODE_SUPPORT
+    zwarnnam(name, "warning: `bindkey -m' disables multibyte support",
+	     NULL, 0);
+#endif
     for(i = 128; i < 256; i++)
 	if(metabind[i - 128] != z_undefinedkey) {
 	    m[0] = i;
Index: Src/Zle/zle_misc.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_misc.c,v
retrieving revision 1.24
diff -u -r1.24 zle_misc.c
--- Src/Zle/zle_misc.c	10 Aug 2005 10:56:41 -0000	1.24
+++ Src/Zle/zle_misc.c	15 Aug 2005 09:34:49 -0000
@@ -30,7 +30,7 @@
 #include "zle.mdh"
 #include "zle_misc.pro"
 
-/* insert a metafied string, with repetition and suffix removal */
+/* insert a zle string, with repetition and suffix removal */
 
 /**/
 void
@@ -1107,6 +1107,18 @@
 	    LinkList args = newlinklist();
 	    char buf[20];
 	    int osc = sfcontext;
+	    int wasmeta = (zlemetaline != 0);
+
+	    if (wasmeta) {
+		/*
+		 * The suffix removal function runs as a normal
+		 * ZLE function, not a completion function, so
+		 * the line should be unmetafied if this was
+		 * called from completion.  (It may not be since
+		 * we may decide to remove the suffix later.)
+		 */
+		umetafy_line();
+	    }
 
 	    sprintf(buf, "%d", suffixlen[0]);
 	    addlinknode(args, suffixfunc);
@@ -1118,13 +1130,16 @@
 	    doshfunc(suffixfunc, prog, args, 0, 1);
 	    sfcontext = osc;
 	    endparamscope();
+
+	    if (wasmeta)
+		metafy_line();
 	}
 	zsfree(suffixfunc);
 	suffixfunc = NULL;
     } else {
 #ifdef ZLE_UNICODE_SUPPORT
 	/* TODO: best I can think of for now... */
-	int sl = (unsigned int)c < 256 ? suffixlen[c] : 0;
+	int sl = (unsigned int)c <= 256 ? suffixlen[c] : 0;
 #else
 	int sl = suffixlen[c];
 #endif


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Clint Adams]

> This turns on ZLE_UNICODE_SUPPORT by default when the environment supports
> it.  For now we require full ISO 10646 support (__STDC_ISO_10646__).  I'm

This is great!

> The problems Andrej's been seeing are likely to be mostly specific to real
> wide characters: since the completion code uses multibyte characters, with
> normal ASCII it sees exactly what it would have with ZLE_UNICODE_SUPPORT
> turned off.  So we will be relying a lot on people who need characters not
> in ASCII.

I notice two areas that need attention.

% ls
한국어/  English/    español/   ish中文/    Türkçe/    Ελληνικάeek/  中文/
català/  Englishий/  français/  Português/  Ελληνικά/  Русский/      中文ish/

% cd
Completing local directory
한국어            Englishий/         ish中文            Ελληνικά     中文                                  
català/             español/            Português/          Ελληνικάeek  中文ish                               
English/             français/           Türkçe/            Русский                       

The directories with the / suffix are colored red, and the unmarked ones are uncolored.
Display width is also miscalculated when tabbing through the completions.

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Peter Stephenson]

Clint Adams wrote:
> Display width is also miscalculated when tabbing through the completions.

Display width has not yet been looked at all.  It would be good if
someone did.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Peter Stephenson]

Peter Stephenson wrote:
> Clint Adams wrote:
> > Display width is also miscalculated when tabbing through the completions.
> 
> Display width has not yet been looked at all.  It would be good if
> someone did.

(Though I should point out that the problem in this case is probably that
multibyte strings are still treated as one character per metafied byte.)

pws


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Clint Adams]

> Display width has not yet been looked at all.  It would be good if
> someone did.

I made a huge mess, but this part of it may be useful.

--- orig/Src/Zle/zle.h
+++ mod/Src/Zle/zle.h
@@ -58,6 +58,11 @@
 #define ZS_strcpy wcscpy
 #define ZS_strncpy wcsncpy
 #define ZS_strncmp wcsncmp
+#define ZS_zarrdup wcs_zarrdup
+#define ZS_width wcslen
+#define ZS_strchr wcschr
+#define ZS_zputs wcs_zputs
+#define ZS_nicewidth wcs_niceztrlen
 
 #define ZC_iblank iswspace
 #define ZC_icntrl iswcntrl
@@ -89,6 +94,11 @@
 #define ZS_memmove memmove
 #define ZS_memset memset
 #define ZS_memcmp memcmp
+#define ZS_zarrdup zarrdup
+#define ZS_width ztrlen
+#define ZS_strchr strchr
+#define ZS_zputs zputs
+#define ZS_nicewidth niceztrlen
 
 #ifdef __GNUC__
 static inline size_t ZS_strlen(ZLE_STRING_T s)
--- orig/Src/string.c
+++ mod/Src/string.c
@@ -54,6 +54,22 @@
     return t;
 }
 
+#ifdef ZLE_UNICODE_SUPPORT
+/**/
+mod_export wchar_t *
+wcs_ztrdup(const wchar_t *s)
+{
+    wchar_t *t;
+
+    if (!s)
+	return NULL;
+    t = (wchar_t *)zalloc(wcslen((wchar_t *)s) + 1);
+    wcscpy(t, s);
+    return t;
+}
+#endif /* ZLE_UNICODE_SUPPORT */
+
+
 /* concatenate s1, s2, and s3 in dynamically allocated buffer */
 
 /**/
--- orig/Src/utils.c
+++ mod/Src/utils.c
@@ -243,6 +243,46 @@
     return buf;
 }
 
+#ifdef ZLE_UNICODE_SUPPORT
+/**/
+mod_export wchar_t *
+wcs_nicechar(wint_t c)
+{
+    static wchar_t buf[6];
+    wchar_t *s = buf;
+    if (iswprint(c))
+	goto done;
+    if (c > 0x80) {
+	if (isset(PRINTEIGHTBIT))
+	    goto done;
+	*s++ = '\\';
+	*s++ = 'M';
+	*s++ = '-';
+	c &= 0x7f;
+	if(iswprint(c))
+	    goto done;
+    }
+    if (c == 0x7f) {
+	*s++ = '^';
+	c = '?';
+    } else if (c == '\n') {
+	*s++ = '\\';
+	c = 'n';
+    } else if (c == '\t') {
+	*s++ = '\\';
+	c = 't';
+    } else if (c < 0x20) {
+	*s++ = '^';
+	c += 0x40;
+    }
+    done:
+    *s++ = c;
+    *s = 0;
+    return buf;
+}
+#endif /* ZLE_UNICODE_SUPPORT */
+
+
 /* Output a string's visible representation. */
 
 #if 0 /**/
@@ -2453,6 +2493,21 @@
     return y;
 }
 
+#ifdef ZLE_UNICODE_SUPPORT
+/**/
+mod_export wchar_t **
+wcs_zarrdup(wchar_t **s)
+{
+    wchar_t **x, **y;
+
+    y = x = (wchar_t **) zalloc(sizeof(wchar_t *) * (arrlen((char **)s) + 1));
+
+    while ((*x++ = wcs_ztrdup(*s++)));
+
+    return y;
+}
+#endif /* ZLE_UNICODE_SUPPORT */
+
 /**/
 static char *
 spname(char *oldname)
@@ -3051,6 +3106,29 @@
     return 0;
 }
 
+#ifdef ZLE_UNICODE_SUPPORT
+/**/
+mod_export int
+wcs_zputs(wchar_t const *s, FILE *stream)
+{
+    wint_t c;
+
+    while (*s) {
+	if (*s == Meta)
+	    c = *++s ^ 32;
+	else if(itok(*s)) {
+	    s++;
+	    continue;
+	} else
+	    c = *s;
+	s++;
+	if (fputwc(c, stream) == WEOF)
+	    return EOF;
+    }
+    return 0;
+}
+#endif /* ZLE_UNICODE_SUPPORT */
+
 /* Create a visibly-represented duplicate of a string. */
 
 /**/
@@ -3137,6 +3215,29 @@
     return l;
 }
 
+#ifdef ZLE_UNICODE_SUPPORT
+/**/
+mod_export size_t
+wcs_nicewidth(wchar_t const *s)
+{
+    size_t l = 0;
+    wint_t c;
+
+    while ((c = *s++)) {
+	if (itok(c)) {
+	    if (c <= (wint_t)Comma)
+		c = ztokens[c - Pound];
+	    else 
+		continue;
+	}
+	if (c == Meta)
+	    c = *s++ ^ 32;
+	l += wcswidth(wcs_nicechar(c), 6);
+    }
+    return l;
+}
+#endif /* ZLE_UNICODE_SUPPORT */
+
 /* check for special characters in the string */
 
 /**/

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Peter Stephenson]

Clint Adams wrote:
> > Display width has not yet been looked at all.  It would be good if
> > someone did.
> 
> I made a huge mess, but this part of it may be useful.

It's a good start, although wcs_niceztrlen() appears to be missing.
(Getting niceztrlen() right ought to be a big help in getting the
completion lists to use the right widths.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Clint Adams]

> It's a good start, although wcs_niceztrlen() appears to be missing.
> (Getting niceztrlen() right ought to be a big help in getting the
> completion lists to use the right widths.)

That is because I confusingly named it wcs_nicewidth().  Beyond that,
the question is when and how to get the matches into a ZLE_STRING_T upon
which to do width calculations.

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Clint Adams]

> 한국어            Englishий/         ish中文            Ελληνικά     中文                                  
> català/             español/            Português/          Ελληνικάeek  中文ish                               
> English/             français/           Türkçe/            Русский                       
> 
> The directories with the / suffix are colored red, and the unmarked ones are uncolored.
> Display width is also miscalculated when tabbing through the completions.

This addresses that problem, though I suspect that the string should be
unmetafied elsewhere.  Yes? No?

--- orig/Src/Zle/compcore.c
+++ mod/Src/Zle/compcore.c
@@ -2465,16 +2465,23 @@
     if ((flags & CMF_FILE) && orig[0] && orig[strlen(orig) - 1] != '/') {
         struct stat buf;
         char *pb;
+	int blahl;
 
         pb = (char *) zhalloc((cm->prpre ? strlen(cm->prpre) : 0) +
                               3 + strlen(orig));
         sprintf(pb, "%s%s", (cm->prpre ? cm->prpre : "./"), orig);
 
-        if (!ztat(pb, &buf, 1)) {
+	char *blah = ztrdup(pb);
+
+	unmetafy(blah, &blahl);
+
+        if (!ztat(blah, &buf, 1)) {
             cm->mode = buf.st_mode;
             if ((cm->modec = file_type(buf.st_mode)) == ' ')
                 cm->modec = '\0';
         }
+
+	free(blah);
     }
     if ((*compqstack == '\\' && compqstack[1]) ||
 	(autoq && *compqstack && compqstack[1] == '\\'))

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Wayne Davison]

On Mon, Aug 15, 2005 at 08:45:57PM -0400, Clint Adams wrote:
>          sprintf(pb, "%s%s", (cm->prpre ? cm->prpre : "./"), orig);
>  
> -        if (!ztat(pb, &buf, 1)) {
> +	char *blah = ztrdup(pb);

I checked in a change that moved that char* definition up to the start
of the local code block.

..wayne..

Re: some unicode issues [was Re: PATCH: fix 4, was Re: unpatch: metafying zle line]

[Peter Stephenson]

Clint Adams <clint@zsh.org> wrote:
> This addresses that problem, though I suspect that the string should be
> unmetafied elsewhere.  Yes? No?

Possibly, but most completion strings spend most of their life metafied.
In zrefresh() characters come in as (wide) characters for display;
completion lists, however, are initially metafied multibyte strings which
need to be unmetafied.  clnicezputs(), compzputs() and compnicezputs()
handle this character by character.  So it's normal to unmetafy at the last
possible moment.  Their may well be other special cases.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************