Certain unicode in hostname breaks with zsh 4.3.17

Certain unicode in hostname breaks with zsh 4.3.17

[Kevin N.]

[-- Attachment #1: Type: text/plain, Size: 131 bytes --]

When using the hostname さわこ zsh produces core dumps with zsh: bad pattern:
\M-c\M-^A?\M-c\M-^B[\M-c\M-^A>
Works in 4.3.15

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Phil Pennock]

On 2012-03-10 at 21:10 -0800, Kevin N. wrote:
> When using the hostname さわこ zsh produces core dumps with zsh: bad pattern:
> \M-c\M-^A?\M-c\M-^B[\M-c\M-^A>
> Works in 4.3.15

gdb zsh /path/to/zsh.core
(gdb) bt

A backtrace to show where the core-dump is being triggered would be
helpful.

Thanks,
-Phil

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Bart Schaefer]

On Sat, Mar 10, 2012 at 9:10 PM, Kevin N. <vekinn@gmail.com> wrote:
> When using the hostname さわこ zsh produces core dumps with zsh: bad patter

When I try to copy-and-paste that hostname into gnome-terminal, I get

** (gnome-terminal:11819): WARNING **: Error (Invalid or incomplete
multibyte or wide character) converting data for child, dropping.

zsh is probably encountering a similar error.

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Bart Schaefer]

On Sun, Mar 11, 2012 at 12:08 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> ** (gnome-terminal:11819): WARNING **: Error (Invalid or incomplete
> multibyte or wide character) converting data for child, dropping.
>
> zsh is probably encountering a similar error.

However, I don't reproduce a crash:

torch% print $ZSH_VERSION
4.3.17-dev-0
torch% HOST="$(perl -MMIME::QuotedPrint -e "print
decode_qp('=E3=81=95=E3=82=8F=E3=81=93')")"
ã\M-^A\M-^Uã\M-^B\M-^Oã\M-^A\M-^S%

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Mikael Magnusson]

On 11 March 2012 20:17, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sun, Mar 11, 2012 at 12:08 PM, Bart Schaefer
> <schaefer@brasslantern.com> wrote:
>> ** (gnome-terminal:11819): WARNING **: Error (Invalid or incomplete
>> multibyte or wide character) converting data for child, dropping.
>>
>> zsh is probably encountering a similar error.
>
> However, I don't reproduce a crash:
>
> torch% print $ZSH_VERSION
> 4.3.17-dev-0
> torch% HOST="$(perl -MMIME::QuotedPrint -e "print
> decode_qp('=E3=81=95=E3=82=8F=E3=81=93')")"
> ã\M-^A\M-^Uã\M-^B\M-^Oã\M-^A\M-^S%

Regardless of how the data is encoded, shouldn't zsh not be trying to
interpret the value of $HOST as a pattern?

-- 
Mikael Magnusson

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Bart Schaefer]

On Mar 11,  8:20pm, Mikael Magnusson wrote:
}
} Regardless of how the data is encoded, shouldn't zsh not be trying to
} interpret the value of $HOST as a pattern?

Depends on context, which the OP didn't provide.  He must be doing
something with $HOST or $(hostname) -- using it in a conditional
expression maybe?

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Mikael Magnusson]

On 11 March 2012 22:19, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Mar 11,  8:20pm, Mikael Magnusson wrote:
> }
> } Regardless of how the data is encoded, shouldn't zsh not be trying to
> } interpret the value of $HOST as a pattern?
>
> Depends on context, which the OP didn't provide.  He must be doing
> something with $HOST or $(hostname) -- using it in a conditional
> expression maybe?

% su
# hostname れにな
# exit
% zsh -f
れにな% echo $HOST
zsh: no matches found: \M-c\M-^B|にな
れにな% typeset HOST
HOST=ãにな
れにな% vared HOST
れにな
れにな% echo $HOST
れにな

-- 
Mikael Magnusson

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Bart Schaefer]

On Mar 11, 10:28pm, Mikael Magnusson wrote:
}
} 0x308C0x306B0x306A% echo $HOST
} zsh: no matches found: \M-c\M-^B|0x306B0x306A

Hm.

    hostnam = (char *)zalloc(256);
    gethostname(hostnam, 256);
    setsparam("HOST", ztrdup(hostnam));
    zfree(hostnam, 256);

Apparently that ztrdup(hostnam) should be a metafy(hostnam, -1, META_DUP).

Should that get done with LOGNAME, MACHTYPE, OSTYPE, and VENDOR as well?

None of that explains the OP's crash, though.  The above code excerpt has
not changed in at least a dozen years, and he says 4.3.15 did not show a
problem.

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Peter Stephenson]

On Sun, 11 Mar 2012 20:56:38 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Mar 11, 10:28pm, Mikael Magnusson wrote:
> }
> } 0x308C0x306B0x306A% echo $HOST
> } zsh: no matches found: \M-c\M-^B|0x306B0x306A
> 
> Hm.
> 
>     hostnam = (char *)zalloc(256);
>     gethostname(hostnam, 256);
>     setsparam("HOST", ztrdup(hostnam));
>     zfree(hostnam, 256);
> 
> Apparently that ztrdup(hostnam) should be a metafy(hostnam, -1, META_DUP).
> 
> Should that get done with LOGNAME, MACHTYPE, OSTYPE, and VENDOR as well?

LOGNAME is imported, so in principle needs this treatment; the others
are from static defines so you might imagine they wouldn't.  Still, I
suppose it makes sense to be consistent when defining parameters
initially, the hit is tiny.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Mikael Magnusson]

On 12 March 2012 04:56, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Mar 11, 10:28pm, Mikael Magnusson wrote:
> }
> } 0x308C0x306B0x306A% echo $HOST
> } zsh: no matches found: \M-c\M-^B|0x306B0x306A
>
> Hm.
>
>    hostnam = (char *)zalloc(256);
>    gethostname(hostnam, 256);
>    setsparam("HOST", ztrdup(hostnam));
>    zfree(hostnam, 256);
>
> Apparently that ztrdup(hostnam) should be a metafy(hostnam, -1, META_DUP).
>
> Should that get done with LOGNAME, MACHTYPE, OSTYPE, and VENDOR as well?
>
> None of that explains the OP's crash, though.  The above code excerpt has
> not changed in at least a dozen years, and he says 4.3.15 did not show a
> problem.

I asked him on irc, and his crash comes from compdump, i think this line,
mv -f $_d_file ${_d_file%.$HOST.$$}
It doesn't crash for me though,
% zsh -f
さわこ% autoload compinit
さわこ% compinit
compdump:135: bad pattern: .\M-c\M-^A?\M-c\M-^B[\M-c\M-^A>.11450
さわこ%

-- 
Mikael Magnusson

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Mikael Magnusson]

On 12 March 2012 11:56, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 12 March 2012 04:56, Bart Schaefer <schaefer@brasslantern.com> wrote:
>> On Mar 11, 10:28pm, Mikael Magnusson wrote:
>> }
>> } 0x308C0x306B0x306A% echo $HOST
>> } zsh: no matches found: \M-c\M-^B|0x306B0x306A
>>
>> Hm.
>>
>>    hostnam = (char *)zalloc(256);
>>    gethostname(hostnam, 256);
>>    setsparam("HOST", ztrdup(hostnam));
>>    zfree(hostnam, 256);
>>
>> Apparently that ztrdup(hostnam) should be a metafy(hostnam, -1, META_DUP).
>>
>> Should that get done with LOGNAME, MACHTYPE, OSTYPE, and VENDOR as well?
>>
>> None of that explains the OP's crash, though.  The above code excerpt has
>> not changed in at least a dozen years, and he says 4.3.15 did not show a
>> problem.
>
> I asked him on irc, and his crash comes from compdump, i think this line,
> mv -f $_d_file ${_d_file%.$HOST.$$}
> It doesn't crash for me though,
> % zsh -f
> さわこ% autoload compinit
> さわこ% compinit
> compdump:135: bad pattern: .\M-c\M-^A?\M-c\M-^B[\M-c\M-^A>.11450
> さわこ%

Apparently he can't reproduce the crash anymore, only the bad pattern message.

-- 
Mikael Magnusson

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Mikael Magnusson]

On 12 March 2012 11:59, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 12 March 2012 11:56, Mikael Magnusson <mikachu@gmail.com> wrote:
>> On 12 March 2012 04:56, Bart Schaefer <schaefer@brasslantern.com> wrote:
>>> On Mar 11, 10:28pm, Mikael Magnusson wrote:
>>> }
>>> } 0x308C0x306B0x306A% echo $HOST
>>> } zsh: no matches found: \M-c\M-^B|0x306B0x306A
>>>
>>> Hm.
>>>
>>>    hostnam = (char *)zalloc(256);
>>>    gethostname(hostnam, 256);
>>>    setsparam("HOST", ztrdup(hostnam));
>>>    zfree(hostnam, 256);
>>>
>>> Apparently that ztrdup(hostnam) should be a metafy(hostnam, -1, META_DUP).
>>>
>>> Should that get done with LOGNAME, MACHTYPE, OSTYPE, and VENDOR as well?
>>>
>>> None of that explains the OP's crash, though.  The above code excerpt has
>>> not changed in at least a dozen years, and he says 4.3.15 did not show a
>>> problem.
>>
>> I asked him on irc, and his crash comes from compdump, i think this line,
>> mv -f $_d_file ${_d_file%.$HOST.$$}
>> It doesn't crash for me though,
>> % zsh -f
>> さわこ% autoload compinit
>> さわこ% compinit
>> compdump:135: bad pattern: .\M-c\M-^A?\M-c\M-^B[\M-c\M-^A>.11450
>> さわこ%
>
> Apparently he can't reproduce the crash anymore, only the bad pattern message.

Sorry for triple posting, apparently there was some confusion with
compdump vs coredump, there was never a crash :).

-- 
Mikael Magnusson

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Peter Stephenson]

On Mon, 12 Mar 2012 10:08:31 +0000
Peter Stephenson <Peter.Stephenson@csr.com> wrote:
> LOGNAME is imported, so in principle needs this treatment; the others
> are from static defines so you might imagine they wouldn't.  Still, I
> suppose it makes sense to be consistent when defining parameters
> initially, the hit is tiny.

Like this, for example.

Index: Src/params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/params.c,v
retrieving revision 1.177
diff -p -u -r1.177 params.c
--- Src/params.c	7 Jan 2012 23:21:00 -0000	1.177
+++ Src/params.c	12 Mar 2012 11:09:13 -0000
@@ -698,16 +698,18 @@ createparamtable(void)
      * So allow the user to set it in the special cases where it's
      * useful.
      */
-    setsparam("TMPPREFIX", ztrdup(DEFAULT_TMPPREFIX));
-    setsparam("TIMEFMT", ztrdup(DEFAULT_TIMEFMT));
-    setsparam("WATCHFMT", ztrdup(default_watchfmt));
+    setsparam("TMPPREFIX", ztrdup_metafy(DEFAULT_TMPPREFIX));
+    setsparam("TIMEFMT", ztrdup_metafy(DEFAULT_TIMEFMT));
+    setsparam("WATCHFMT", ztrdup_metafy(default_watchfmt));
 
     hostnam = (char *)zalloc(256);
     gethostname(hostnam, 256);
-    setsparam("HOST", ztrdup(hostnam));
+    setsparam("HOST", ztrdup_metafy(hostnam));
     zfree(hostnam, 256);
 
-    setsparam("LOGNAME", ztrdup((str = getlogin()) && *str ? str : cached_username));
+    setsparam("LOGNAME",
+	      ztrdup_metafy((str = getlogin()) && *str ?
+			    str : cached_username));
 
 #if !defined(HAVE_PUTENV) && !defined(USE_SET_UNSET_ENV)
     /* Copy the environment variables we are inheriting to dynamic *
@@ -739,7 +741,7 @@ createparamtable(void)
 			pm->env = mkenvstr (pm->node.nam,
 					    getsparam(pm->node.nam), pm->node.flags);
 		    else
-			pm->env = ztrdup(*envp2);
+			pm->env = ztrdup_metafy(*envp2);
 #ifndef USE_SET_UNSET_ENV
 		    *envp++ = pm->env;
 #endif
@@ -778,22 +780,22 @@ createparamtable(void)
     if(uname(&unamebuf)) setsparam("CPUTYPE", ztrdup("unknown"));
     else
     {
-       machinebuf = ztrdup(unamebuf.machine);
+       machinebuf = ztrdup_metafy(unamebuf.machine);
        setsparam("CPUTYPE", machinebuf);
     }
-	
+
 #else
-    setsparam("CPUTYPE", ztrdup("unknown"));
+    setsparam("CPUTYPE", ztrdup_metafy("unknown"));
 #endif
-    setsparam("MACHTYPE", ztrdup(MACHTYPE));
-    setsparam("OSTYPE", ztrdup(OSTYPE));
-    setsparam("TTY", ztrdup(ttystrname));
-    setsparam("VENDOR", ztrdup(VENDOR));
-    setsparam("ZSH_NAME", ztrdup(zsh_name));
-    setsparam("ZSH_VERSION", ztrdup(ZSH_VERSION));
-    setsparam("ZSH_PATCHLEVEL", ztrdup(ZSH_PATCHLEVEL));
+    setsparam("MACHTYPE", ztrdup_metafy(MACHTYPE));
+    setsparam("OSTYPE", ztrdup_metafy(OSTYPE));
+    setsparam("TTY", ztrdup_metafy(ttystrname));
+    setsparam("VENDOR", ztrdup_metafy(VENDOR));
+    setsparam("ZSH_NAME", ztrdup_metafy(zsh_name));
+    setsparam("ZSH_VERSION", ztrdup_metafy(ZSH_VERSION));
+    setsparam("ZSH_PATCHLEVEL", ztrdup_metafy(ZSH_PATCHLEVEL));
     setaparam("signals", sigptr = zalloc((SIGCOUNT+4) * sizeof(char *)));
-    for (t = sigs; (*sigptr++ = ztrdup(*t++)); );
+    for (t = sigs; (*sigptr++ = ztrdup_metafy(*t++)); );
 
     noerrs = 0;
 }
Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.266
diff -p -u -r1.266 utils.c
--- Src/utils.c	5 Mar 2012 10:06:29 -0000	1.266
+++ Src/utils.c	12 Mar 2012 11:09:13 -0000
@@ -4013,6 +4013,28 @@ metafy(char *buf, int len, int heap)
 
 
 /*
+ * Duplicate a string, metafying it as we go.
+ *
+ * Typically, this is used only for strings imported from outside
+ * zsh, as strings internally are either already metafied or passed
+ * around with an associated length.
+ */
+/**/
+mod_export char *
+ztrdup_metafy(const char *s)
+{
+    /* To mimic ztrdup() behaviour */
+    if (!s)
+	return NULL;
+    /*
+     * metafy() does lots of different things, so the pointer
+     * isn't const.  Using it with META_DUP should be safe.
+     */
+    return metafy((char *)s, -1, META_DUP);
+}
+
+
+/*
  * Take a null-terminated, metafied string in s into a literal
  * representation by converting in place.  The length is in *len
  * len is non-NULL; if len is NULL, you don't know the length of

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Bart Schaefer]

On Mar 12, 11:10am, Peter Stephenson wrote:
>
> @@ -739,7 +741,7 @@ createparamtable(void)
>  			pm->env = mkenvstr (pm->node.nam,
>  					    getsparam(pm->node.nam), pm->node.flags);
>  		    else
> -			pm->env = ztrdup(*envp2);
> +			pm->env = ztrdup_metafy(*envp2);
>  #ifndef USE_SET_UNSET_ENV
>  		    *envp++ = pm->env;
>  #endif

I'm not sure that's correct.  pm->env is being copied directly into the
global environ array in the USE_SET_UNSET_ENV case.  Surely it'll play
havok to put metafied strings into the environment of forked children?
Or is that cleaned up somewhere else?

Re: Certain unicode in hostname breaks with zsh 4.3.17

[Peter Stephenson]

On Mon, 12 Mar 2012 10:07:37 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Mar 12, 11:10am, Peter Stephenson wrote:
> >
> > @@ -739,7 +741,7 @@ createparamtable(void)
> >  			pm->env = mkenvstr (pm->node.nam,
> >  					    getsparam(pm->node.nam), pm->node.flags);
> >  		    else
> > -			pm->env = ztrdup(*envp2);
> > +			pm->env = ztrdup_metafy(*envp2);
> >  #ifndef USE_SET_UNSET_ENV
> >  		    *envp++ = pm->env;
> >  #endif
> 
> I'm not sure that's correct.  pm->env is being copied directly into the
> global environ array in the USE_SET_UNSET_ENV case.

Yes, you're right, so that shouldn't be metafied and I'll correct that
bit.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog