BUG: tied parameters with NUL-separator

BUG: tied parameters with NUL-separator

[Matthias B.]

The following works as expected in zsh 4.2.1
(NOTE: hexd is a hexdump script that displays 
offset: hex   correspondingchars)

pmt> unset foo
pmt> unset bar
pmt> sep=:
pmt> foo='a:b:c'
pmt> typeset -T foo bar "$sep"
pmt> echo -n "${bar[1]}" | hexd
00000: 61                 a
pmt> echo -n "${bar[2]}" | hexd
00000: 62                 b
pmt> echo -n "${bar[3]}" | hexd
00000: 63                 c
pmt> bar[3]='1 2'
pmt> echo -n "${bar[3]}" | hexd
00000: 31 20 32               1 2
pmt> echo -n "$foo" | hexd
00000: 61 3a 62 3a 31 20 32           a:b:1 2


Now look what happens when I use NUL as separator

pmt> unset foo
pmt> unset bar
pmt> sep=$'\0'
pmt> foo=$'a\0b\0c'
pmt> typeset -T foo bar "$sep"
pmt> echo -n "${bar[1]}" | hexd
00000: 61                 a
pmt> echo -n "${bar[2]}" | hexd
00000: 20 62                 b
pmt> echo -n "${bar[3]}" | hexd
00000: 20 63                 c
pmt> bar[3]='1 2'
pmt> echo -n "${bar[3]}" | hexd
00000: 31 20 32               1 2
pmt> echo -n "$foo" | hexd
00000: 61 20 62 31 20             a b1 


So we have 2 bugs here:

1) When typeset -T splits foo it inserts a space in front of all elements
but the 1st one. This only happens when splitting apparently. Assigning to
an element directly does not insert a space.

2) Re-assembling the scalar doesn't work. The separators are not inserted
at all and the part after the space in the string '1 2' assigned to
element 3 gets lost somehow.


I know that I probably shouldn't complain. I'm actually quite impressed
that embedding NULs in strings works at all. Few C-programs have
binary-clean string-handling.
But it's such a useful feature and since it works almost perfectly already
I'm hopeful that this can be made to work fully. I'm really looking
forward to this, because I want to parse output from find into an array
and since pathnames can in theory contain every character but NUL, using
NUL as separator seems to be the only way to achieve this. 

Bug 1) is the important one for me as I'm only interested in splitting, so
a fix for that would be appreciated even if Bug 2) would take too much
effort to fix.

MSB

-- 
pragma est dogma.

PATCH: incorrect tied param splitting/joining when imeta(separator) is true

[Matthias B.]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

Since no one seemed to be interested in the NUL-separator bugs I've taken
a look at the code myself and I've managed to locate the bugs. 

The function tiedarrsetfn() does not convert characters for which imeta()
is true into separator strings properly. This means that the bug is not
limited to NUL-characters but affects all characters for which imeta() is
true (e.g. 0x83). 

There are also 2 bugs in zjoin():
1. zjoin() doesn't put the NUL-terminator in the right place when delim is
a meta-character.

2. zjoin() refuses to add 0-delimiters (although it does count them when
computing the length). This behaviour does not seem to be used anywhere in
zsh code AFAICS and even if it were, that would be a bug in the respective
call.


The attached patch fixes both problems and adds 2 regression tests to
B02typeset.ztst (both tests are necessary, since 0 and 0x83 are treated
differently in some places, so that one case could regress independent of
the other).

MSB

[-- Attachment #2: zsh-4.2.1-split-join-with-meta-sep-fix.patch --]
[-- Type: text/plain, Size: 4287 bytes --]

diff -r --unified=23 zsh-4.2.1/Src/params.c zsh-4.2.1-split-join-with-meta-sep-fix/Src/params.c
--- zsh-4.2.1/Src/params.c	Fri Aug 13 12:22:45 2004
+++ zsh-4.2.1-split-join-with-meta-sep-fix/Src/params.c	Thu Sep 16 19:34:59 2004
@@ -2623,47 +2623,47 @@
 }
 
 /**/
 char *
 tiedarrgetfn(Param pm)
 {
     struct tieddata *dptr = (struct tieddata *)pm->u.data;
     return *dptr->arrptr ? zjoin(*dptr->arrptr, dptr->joinchar, 1) : "";
 }
 
 /**/
 void
 tiedarrsetfn(Param pm, char *x)
 {
     struct tieddata *dptr = (struct tieddata *)pm->u.data;
 
     if (*dptr->arrptr)
 	freearray(*dptr->arrptr);
     if (x) {
 	char sepbuf[3];
 	if (imeta(dptr->joinchar))
 	{
 	    sepbuf[0] = Meta;
-	    sepbuf[1] = dptr->joinchar;
+	    sepbuf[1] = dptr->joinchar ^ 32;
 	    sepbuf[2] = '\0';
 	}
 	else
 	{
 	    sepbuf[0] = dptr->joinchar;
 	    sepbuf[1] = '\0';
 	}
 	*dptr->arrptr = sepsplit(x, sepbuf, 0, 0);
 	if (pm->flags & PM_UNIQUE)
 	    uniqarray(*dptr->arrptr);
     } else
 	*dptr->arrptr = NULL;
     if (pm->ename)
 	arrfixenv(pm->nam, *dptr->arrptr);
     zsfree(x);
 }
 
 /**/
 void
 tiedarrunsetfn(Param pm, UNUSED(int exp))
 {
     /*
      * Special unset function because we allocated a struct tieddata
diff -r --unified=23 zsh-4.2.1/Src/utils.c zsh-4.2.1-split-join-with-meta-sep-fix/Src/utils.c
--- zsh-4.2.1/Src/utils.c	Fri Aug 13 12:22:46 2004
+++ zsh-4.2.1-split-join-with-meta-sep-fix/Src/utils.c	Thu Sep 16 21:25:56 2004
@@ -1853,56 +1853,54 @@
 	} else {
 	    if (ztrftimebuf(&bufsize, 1))
 		return 0;
 	    *buf++ = *fmt++;
 	}
     *buf = '\0';
     return buf - origbuf;
 }
 
 /**/
 mod_export char *
 zjoin(char **arr, int delim, int heap)
 {
     int len = 0;
     char **s, *ret, *ptr;
 
     for (s = arr; *s; s++)
 	len += strlen(*s) + 1 + (imeta(delim) ? 1 : 0);
     if (!len)
 	return heap? "" : ztrdup("");
     ptr = ret = (heap ? (char *) hcalloc(len) : (char *) zshcalloc(len));
     for (s = arr; *s; s++) {
 	strucpy(&ptr, *s);
-	if (delim) {
 	    if (imeta(delim)) {
 		*ptr++ = Meta;
 		*ptr++ = delim ^ 32;
 	    }
 	    else
 		*ptr++ = delim;
-	}
     }
-    ptr[-1] = '\0';
+    ptr[-1 - (imeta(delim) ? 1 : 0)] = '\0';
     return ret;
 }
 
 /* Split a string containing a colon separated list *
  * of items into an array of strings.               */
 
 /**/
 char **
 colonsplit(char *s, int uniq)
 {
     int ct;
     char *t, **ret, **ptr, **p;
 
     for (t = s, ct = 0; *t; t++) /* count number of colons */
 	if (*t == ':')
 	    ct++;
     ptr = ret = (char **) zalloc(sizeof(char **) * (ct + 2));
 
     t = s;
     do {
 	s = t;
         /* move t to point at next colon */
 	for (; *t && *t != ':'; t++);
diff -r --unified=23 zsh-4.2.1/Test/B02typeset.ztst zsh-4.2.1-split-join-with-meta-sep-fix/Test/B02typeset.ztst
--- zsh-4.2.1/Test/B02typeset.ztst	Fri Aug 13 12:22:54 2004
+++ zsh-4.2.1-split-join-with-meta-sep-fix/Test/B02typeset.ztst	Thu Sep 16 21:58:41 2004
@@ -163,46 +163,61 @@
 ?(eval):3: read-only variable: r
 
  typeset r=success
  readonly r
  print $r
  r=failure
 1:Convert to readonly
 >success
 ?(eval):4: read-only variable: r
 
  typeset -gU array
  print $array
 0:Uniquified arrays and non-local scope
 >a r y
 
  typeset -T SCALAR=l:o:c:a:l array
  print $array
  typeset -U SCALAR
  print $SCALAR $array
 0:Tied parameters and uniquified colon-arrays
 >l o c a l
 >l:o:c:a l o c a
 
+ typeset -T SCALAR=$'l\x83o\x83c\x83a\x83l' array $'\x83'
+ print $array
+ typeset -U SCALAR
+ print $SCALAR $array
+0:Tied parameters and uniquified arrays with meta-character as separator
+>l o c a l
+>lƒoƒcƒa l o c a
+
+ typeset -T SCALAR=$'l\000o\000c\000a\000l' array $'\000'
+ typeset -U SCALAR
+ print $array
+ [[ $SCALAR == $'l\000o\000c\000a' ]]
+0:Tied parameters and uniquified arrays with NUL-character as separator
+>l o c a
+
  typeset -T SCALAR array
  typeset +T SCALAR
 1:Untying is prohibited
 ?(eval):typeset:2: use unset to remove tied variables
 
  OUTER=outer
  scope13
  print $OUTER
 0:Export of tied parameters
 >i:n:n:e:r
 >outer
 
  typeset -TU MORESTUFF=here-we-go-go-again morestuff '-'
  print -l $morestuff
 0:Tied arrays with separator specified
 >here
 >we
 >go
 >again
 
  typeset -T THIS will not work
 1:Tied array syntax
 ?(eval):typeset:1: -T requires names of scalar and array

Re: PATCH: incorrect tied param splitting/joining when imeta(separator) is true

[Peter Stephenson]

"Matthias B." wrote:
> Since no one seemed to be interested in the NUL-separator bugs I've taken
> a look at the code myself and I've managed to locate the bugs. 

Thanks, this one passed me by.  I've committed it (trying not to commit
anything to do with ignore_eof).

It will probably need updating for Unicode, but no one's holding their
breath.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************