zsh appears to be eating characters when called with -c

zsh appears to be eating characters when called with -c

[Matt Watson]

Can anyone explain what's going on here? I have a file with a UTF-8 
name, yet zsh seems to be munging the input when I pass this file name 
on the command line. We link /bin/sh to /bin/zsh. Here's a diagnosis 
from our bug database:

bash-2.03$ /bin/tcsh -c 'echo NTT-ME\ 
MN128\343\202\267\343\203\252\343\203\274\343\202\271\343\202\231\ 128K\ 
ARA' | od -x
0000000     4e54    542d    4d45    204d    4e31    3238    e382    b7e3
0000020     83aa    e383    bce3    82b9    e382    9920    3132    384b
0000040     2041    5241    0a00
0000045

bash-2.03$ /usr/local/bin/bash -c 'echo NTT-ME\ 
MN128\343\202\267\343\203\252\343\203\274\343\202\271\343\202\231\ 128K\ 
ARA' | od -x
0000000     4e54    542d    4d45    204d    4e31    3238    e382    b7e3
0000020     83aa    e383    bce3    82b9    e382    9920    3132    384b
0000040     2041    5241    0a00
0000045

/bin/sh is zsh
bash-2.03$ /bin/sh -c 'echo NTT-ME\ 
MN128\343\202\267\343\203\252\343\203\274\343\202\271\343\202\231\ 128K\ 
ARA' | od -x
0000000     4e54    542d    4d45    204d    4e31    3238    e382    b7e3
0000020     8ae3    9ce3    82b9    e382    2031    3238    4b20    4152
0000040     410a
0000042


the actual file name is :
NTT-ME\ 
MN128\343\202\267\343\203\252\343\203\274\343\202\271\343\202\231\ 128K\ 
ARA

It seems that zsh is interpreting some of the bytes (removing some, 
replacing others), while tcsh and bash just work as expected.
ppp executes a command with parameters using /bin/sh, and will fail when 
the parameters contain this kind of file names (japanese modem scripts).
As a work around, ppp can use /bin/tcsh.
But is zsh supposed to interpret bytes in single quotes ?

Try this (no use of echo this time):

#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>

void testshell(const char *shell, const char *command)
{
         int status = 0;
         int r = fork();
         switch (r) {
         case -1:
                 fprintf(stderr, "fork() failed: %s\n", strerror(errno));
                 break;
         case 0:
                 fprintf(stdout, "%s: ", shell);
                 fflush(stdout);
                 execlp(shell, shell, "-c", command, NULL);
                 fprintf(stderr, "execlp() failed: %s\n", 
strerror(errno));
                 break;
         default:
                 waitpid(r, &status, 0);
                 break;
         }
}

int main(void)
{
         char file[] = { 'N',  'T',  'T',  '-',  'M',  'E',  ' ',  'M',
                         'N',  '1',  '2',  '8', 0343, 0202, 0267, 0343,
                        0203, 0252, 0343, 0203, 0274, 0343, 0202, 0271,
                        0343, 0202, 0231,  ' ',  '1',  '2',  '8',  'K',
                         ' ',  'A',  'R',  'A',   0                    };

         char tmpfile[256] = "/tmp/";
         char cmd[256] = "cat ";
         FILE *tmpf;

         strcat(tmpfile, file);
         tmpf = fopen(tmpfile, "w");

         if (tmpf == NULL) {
                 fprintf(stderr, "failed to open: %s\n", tmpfile);
                 exit(1);
         }

         fprintf(tmpf, "Hello World!\n");
         fclose(tmpf);

         strcat(cmd, "'");
         strcat(cmd, tmpfile);
         strcat(cmd, "'");

         testshell("/usr/local/bin/bash", cmd);
         testshell("/bin/tcsh", cmd);
         testshell("/bin/zsh", cmd);
         testshell("/bin/sh", cmd);

         exit(0);
}

Re: zsh appears to be eating characters when called with -c

[Sven Wischnowsky]

Matt Watson wrote:

> ...
> 
> /bin/sh is zsh
> bash-2.03$ /bin/sh -c 'echo NTT-ME\ 
> MN128\343\202\267\343\203\252\343\203\274\343\202\271\343\202\231\ 128K\ 
> ARA' | od -x
> 0000000     4e54    542d    4d45    204d    4e31    3238    e382    b7e3
> 0000020     8ae3    9ce3    82b9    e382    2031    3238    4b20    4152
> 0000040     410a
> 0000042

There is a patch for this at SourceForge:

  https://sourceforge.net/patch/?func=detailpatch&patch_id=100530&group_id=4068

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: zsh appears to be eating characters when called with -c

[Matt Watson]

Okay, this appears to be caused by the Meta character being interpreted. 
This portion of the command string:

0x83aa 0xe383 0xbce3 0x82b9 0xe382 0x9920

gets munged into:

0x8ae3 0x9ce3 0x82b9 0xe382 20

So, the 0x83aa gets turned into 0xaa ^ 32 (0x8a), the 0x83bc gets turned 
into 0x9c, and the 0x99 (Dnull) is summarily dispatched.

matt.

Re: zsh appears to be eating characters when called with -c

[Matt Watson]

Forgot to mention, this is 3.0.8 running on Darwin/Mac OS X.

matt.