unpatch: metafying zle line

unpatch: metafying zle line

[Peter Stephenson]

I have a patch for the first step in making completion work with wide
characters.  It's quite long, so instead of posting a patch which will
fill people's mailboxes and no one will read I'll describe it here and
commit it some time during the day.

People who aren't interested in the implementation should simply note
that it might make completion unstable for a while.  This is in a good
cause and any bug reports will help the Unicode development, even
without the definition ZLE_UNICODE_SUPPORT that turns it on (at the
moment this is off unless you alter zsh.h).

Now the details.

Completion works on a multibyte string: that's tied to the way the main
shell works, because of the degree of interaction between them.(*) So
when we enter the completion section we take the variables zleline,
zlecs and zlell and turn them into metafied multibyte counterparts,
zlemetaline, zlemetacs and zlemetall.  The metafication exists already;
the new feature is the conversion from wide characters to multibyte
strings at the same time.  This necessitated different variables for the
line itself; it didn't do so for the cursor and length, but it seemed
neater to use different variables for those as well.  It's supposed to
be the case that only the metafied line is used from the main shell; the
interface with lex.c and hist.c is on the horrific side, but I don't
propose to play with that for the time being.

I had to catch various places (inevitably rather more places than I had
hoped) where metafy_line()/unmetafy_line() pairs were needed, for
example:  some occurrences of zrefresh(); the read-only uses of variables
BUFFER, LBUFFER, RBUFFER, CURSOR; zle -M and relatives.  If it's any
consolation, the lack of such pairs was already a bug in the existing
code and could have resulted in Meta characters popping up on the
command line.  With the new system, the likely result of a missed case
is a crash, since I deliberately set zleline or zlemetaline to NULL when
in the other state.

Note the existence of METACHECK() and UNMETACHECK() macros which are
turned on with the DEBUG definition and report the file name and line number
if the metafication state is wrong.  (By the way, there's a good
argument for adopting the use of file name and line number in the
standard DPUTS definition.)  These can be added to any other places
where it seems appropriate.

One thing I haven't done is handle the mark variable properly.  This
should really be updated with metafy_line() and unmetafy_line(), too,
but it was too minor for me to worry about at this stage.

There is one additional fix, for the lprompt output in singlerefresh().
This is completely untested!  singlerefresh() isn't used much nowadays.
Still no attempt to get character widths correct.  (This would need
either multibyte or wide character support in the prompt code.)

Indeed, I haven't yet tried any of this with ZLE_UNICODE_SUPPORT turned
on.  That's the next step: the first step is to ensure that the patch
works with the old system.  When that's done, we can be reasonably sure
that the conversion between zleline and zlemetaline is sound.  Then, I
hope, the basics of completion with ZLE_UNICODE_SUPPORT will work
without too much extra work, in that any text with only single-byte
characters will (in theory) be handled straight away.

The step after that will be to teach the metafied areas of completion,
as well as (ouch) the main shell about multibyte characters.  However,
at this point it should be possible to turn on ZLE_UNICODE_SUPPORT for
systems that have all the required support(**) as the functionality
should be no worse than what we have at present without
ZLE_UNICODE_SUPPORT.  This will make life easier since we will at least
be debugging a basically working system.

(*) It might theoretically be possible to move the interface so that
more of the completion code runs with wide characters; I have simply
intercepted the points where metafy_line currently runs (or should run).
However, that's substantially more work.  As I said before, changing the
main shell to use wide chars throughout isn't an option: we don't even
know that the entire byte stream input maps one-to-one to characters at
all (indeed, with tokens in it we know for sure it doesn't); the only
safe assumption is that a 7-bit subset of it contains characters we can
interpret.  Then we provide additional facilities (all TBD) when the
user knows a certain chunk is multibyte characters.

(**) Once we turn on ZLE_UNICODE_SUPPORT by default for systems where it
works, we can start thinking about relaxing some of the assumptions
underlying it since, as Oliver pointed out, they shouldn't all be
necessary.  However, we'll probably have to do this system by system to
find out which assumptions we can safely relax.  One interesting case is
Solaris 8 which currently doesn't meet all the tests but should
nonetheless have all the features we actually require.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: unpatch: metafying zle line

[DervishD]

    Hi Peter :)

 * Peter Stephenson <pws@csr.com> dixit:
> I have a patch for the first step in making completion work with wide
> characters.  It's quite long, so instead of posting a patch which will
> fill people's mailboxes and no one will read I'll describe it here and
> commit it some time during the day.

    It's good to see that step by step, zsh is becoming Unicode
compliant :) It is a very big step beyond. I just hope someday we
have another big step beyond: zle as a library, because all these
poor console apps deserve a good way of editing lines ;))

    Thanks for your effort, zsh is getting better day by day.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

PATCH: fix 1, was Re: unpatch: metafying zle line

[Peter Stephenson]

Peter Stephenson wrote:
> I have a patch for the first step in making completion work with wide
> characters.  It's quite long, so instead of posting a patch which will
> fill people's mailboxes and no one will read I'll describe it here and
> commit it some time during the day.

It's committed.  Here's the first bug: accept-and-infer-next-history in
menu selection gets the metafication state confused.  (That function is
probably the most confused with regard to the state, see the comment
further up.)

Index: Src/Zle/complist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/complist.c,v
retrieving revision 1.69
diff -u -r1.69 complist.c
--- Src/Zle/complist.c	10 Aug 2005 10:56:41 -0000	1.69
+++ Src/Zle/complist.c	10 Aug 2005 11:55:11 -0000
@@ -2303,7 +2303,10 @@
                 iforcemenu = -1;
             } else
                 mode = 0;
+	    /* Nested completion assumes line is unmetafied */
+	    unmetafy_line();
 	    menucomplete(zlenoargs);
+	    metafy_line();
 	    iforcemenu = 0;
 
             if (cmd != Th(z_acceptandinfernexthistory))

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

PATCH: fix 2, was Re: unpatch: metafying zle line

[Peter Stephenson]

Remove two problems with character arrays with ZLE_UNICODE_SUPPORT by
transferring a "*" from one file to another.  Very economical.

Index: Src/Zle/complist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/complist.c,v
retrieving revision 1.70
diff -u -r1.70 complist.c
--- Src/Zle/complist.c	10 Aug 2005 11:59:29 -0000	1.70
+++ Src/Zle/complist.c	10 Aug 2005 13:15:52 -0000
@@ -1865,7 +1865,7 @@
 {
 #ifdef ZLE_UNICODE_SUPPORT
     /* MB_CUR_MAX may not be constant */
-    VARARR(char *, s, MB_CUR_MAX+1);
+    VARARR(char, s, MB_CUR_MAX+1);
 #else
     char s[2];
 #endif
Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.24
diff -u -r1.24 zle_refresh.c
--- Src/Zle/zle_refresh.c	10 Aug 2005 10:56:41 -0000	1.24
+++ Src/Zle/zle_refresh.c	10 Aug 2005 13:15:52 -0000
@@ -1207,7 +1207,7 @@
 #ifdef ZLE_UNICODE_SUPPORT
     ZLE_STRING_T lpwbuf, lpwp;	/* converted lprompt and pointer */
     char *lpptr,		/* pointer into multibyte lprompt */
-	lpend;			/* end of multibyte lprompt */
+	*lpend;			/* end of multibyte lprompt */
     mbstate_t ps;		/* shift state */
 #endif
 
-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: unpatch: metafying zle line

[David Gómez]

Hi Peter ;),

On Aug 10 at 11:34:17, Peter Stephenson wrote:
> I have a patch for the first step in making completion work with wide
> characters.  It's quite long, so instead of posting a patch which will
> fill people's mailboxes and no one will read I'll describe it here and
> commit it some time during the day.

I'm really glad to hear that :).

> People who aren't interested in the implementation should simply note
> that it might make completion unstable for a while.  This is in a good
> cause and any bug reports will help the Unicode development, even
> without the definition ZLE_UNICODE_SUPPORT that turns it on (at the
> moment this is off unless you alter zsh.h).

Cause i'm of no much help with the source, i'll do my best with the
tests, so i'll start testing it the moment you commit the patch.

Thanks for you great work!

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: unpatch: metafying zle line

[David Gómez]

Hi Peter ;),

On Aug 10 at 11:34:17, Peter Stephenson wrote:
> Indeed, I haven't yet tried any of this with ZLE_UNICODE_SUPPORT turned
> on.  That's the next step:

I've already tried it with ZLE_UNICODE_SUPPORT enabled ;). It's not so easy
to get a segmentation fault as before. Fist thing i noticed is that when you
try to complete a line, it always tries to complete the first word.

> works with the old system.  When that's done, we can be reasonably sure
> that the conversion between zleline and zlemetaline is sound.

Then i'll test it without ZLE_UNICODE

bye

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

PATCH: fix 3, was Re: unpatch: metafying zle line

[Peter Stephenson]

Two more:

which-command and run-help were broken because I forgot to copy the code
that moves the cursor to the end, so the strings were inserted in the
wrong order.

Some more advanced menu features including accept-and-menu-complete were
broken by yet another complication with the metafication state when
using menus.

Index: Src/Zle/compresult.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/compresult.c,v
retrieving revision 1.55
diff -u -r1.55 compresult.c
--- Src/Zle/compresult.c	10 Aug 2005 10:56:41 -0000	1.55
+++ Src/Zle/compresult.c	10 Aug 2005 19:44:28 -0000
@@ -687,6 +687,8 @@
 mod_export int
 hasbrpsfx(Cmatch m, char *pre, char *suf)
 {
+    METACHECK();
+
     if (m->flags & CMF_ALL)
 	return 1;
     else {
@@ -1153,6 +1155,15 @@
 mod_export void
 do_menucmp(int lst)
 {
+    int was_meta;
+
+    /* Already metafied when called from domenuselect already */
+    if (zlemetaline == NULL) {
+	was_meta = 0;
+	metafy_line();
+    } else
+	was_meta = 1;
+
     /* Just list the matches if the list was requested. */
     if (lst == COMP_LIST_COMPLETE) {
 	showinglist = -2;
@@ -1173,13 +1184,10 @@
 	     (((*minfo.cur)->flags & (CMF_NOLIST | CMF_MULT)) &&
 	      (!(*minfo.cur)->str || !*(*minfo.cur)->str)));
     /* ... and insert it into the command line. */
-    /* Already metafied when called from domenuselect already */
-    if (zlemetaline == NULL) {
-	metafy_line();
-	do_single(*minfo.cur);
+    do_single(*minfo.cur);
+
+    if (!was_meta)
 	unmetafy_line();
-    } else
-	do_single(*minfo.cur);
 }
 
 /**/
Index: Src/Zle/zle_tricky.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_tricky.c,v
retrieving revision 1.54
diff -u -r1.54 zle_tricky.c
--- Src/Zle/zle_tricky.c	10 Aug 2005 10:56:41 -0000	1.54
+++ Src/Zle/zle_tricky.c	10 Aug 2005 19:44:51 -0000
@@ -1800,6 +1800,8 @@
 	ZS_strncpy(zleline + zlecs, zlestr, zlelen);
 	free(zlestr);
 	zsfree((char *)instr);
+	if (move)
+	    zlecs += len;
     }
     return len;
 }

-- 
Peter Stephenson <pws@pwstephenson.fsnet.co.uk>
Work: pws@csr.com
Web: http://www.pwstephenson.fsnet.co.uk