Bug when $USERNAME is cyryllic

Bug when $USERNAME is cyryllic

[Максим Щербаков]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

Hello, this is the first time I am using mailing lists, don't be mad at me if I do something wrong.
Sorry for errors, English is not my native language.


I found a bug in zsh:

If I add this line to the top of /etc/passwd (before the "root" line, also works with other UIDs):
> quote_type
> Студент:x:0:0:root:/home/Студент:/usr/bin/zsh

After logging in as "root" (so environment will be good, as in this case "root" line is used for login), and running this commands:
> quote_type
> mkdir -p '/home/Студент'
> env -i LANG=C.UTF-8 zsh -fc 'cd; echo "$USERNAME | ${(%):-%n} | $LOGNAME"'

I will get:
> quote_type
> �т��ент | Ст\M-Q\M-p\M-4ент | Студент

Instead of:
> quote_type
> Студент | Студент | Студент

As we can see, $LOGNAME is fine, but $USERNAME and %n are bad.

-----------------------------

Also, probably related, zcompile don't work on cyryllic paths:
> quote_type
> touch '/home/Студент/.zc/.zcompdump-5.9'
> zcompile '/home/Студент/.zc/.zcompdump-5.9'

Results in:
> quote_type
> zcompile: can't write zwc file: /home/Студент/.zc/.zcompdump-5.9.zwc


[-- Attachment #2: Type: text/html, Size: 2433 bytes --]

Re: Bug when $USERNAME is cyryllic

[Ellenor Bjornsdottir]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

Maksim,

What UNIX are you using where that's a valid username? If Linux, what 
distribution?

Not disparaging it, just wondering. I agree that this is not how zsh 
should be behaving. Are other shells (e.g. bash, Korn, tcsh) behaving 
correctly?

In English, we call the script used in Russia, Ukraine, Belarus, and a 
few other countries (including most internal vassals of Russia, as well 
as Bulgaria, Serbia (sometimes), Montenegro, Mongolia, Tajikistan, and 
until recently Kazakhstan) to write their native languages "Cyrillic", 
not "Cyryllic". Your English spelling and grammar is otherwise entirely 
acceptable.

Le 1 octobre 2023 05:04:40 UTC, "Максим Щербаков" 
<herobrine135111@gmail.com> a écrit :

    Hello, this is the first time I am using mailing lists, don't be mad
    at me if I do something wrong.
    Sorry for errors, English is not my native language.


    I found a bug in zsh:

    If I add this line to the top of _/etc/passwd_ (before the "root"
    line, also works with other UIDs):

        Студент:x:0:0:root:/home/Студент:/usr/bin/zsh


    After logging in as "root" (so environment will be good, as in this
    case "root" line is used for login), and running this commands:

        mkdir -p '/home/Студент'
        env -i LANG=C.UTF-8 zsh -fc 'cd; echo "$USERNAME | ${(%):-%n} |
        $LOGNAME"'


    I will get:

        �т��ент | Ст\M-Q\M-p\M-4ент | Студент


    Instead of:

        Студент | Студент | Студент


    As we can see, _$LOGNAME_ is fine, but _$USERNAME_ and _%n_ are bad.

    -----------------------------

    Also, probably related, _zcompile_ don't work on cyryllic paths:

        touch '/home/Студент/.zc/.zcompdump-5.9'
        zcompile '/home/Студент/.zc/.zcompdump-5.9'


    Results in:

        zcompile: can't write zwc file: /home/Студент/.zc/.zcompdump-5.9.zwc


-- 
Ellenor Agnes Bjornsdottir (she)
sysadmin umbrellix.net
jabber: ellenor ~on~ umbrellix.net

[-- Attachment #2: Type: text/html, Size: 4482 bytes --]

Re: Bug when $USERNAME is cyryllic

[Максим Щербаков]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

> What UNIX are you using where that's a valid username? If Linux, what distribution?
Cygwin, as it uses usernames from underlying Windows
> Not disparaging it, just wondering. I agree that this is not how zsh should be behaving. Are other shells (e.g. bash, Korn, tcsh) behaving correctly?
Works fine in bash:
> quote_type
> PS1='\u >'
> Студент >
>  to write their native languages "Cyrillic", not "Cyryllic". Your English spelling and grammar is otherwise entirely acceptable.
Oh, "Cyrillic", thanks for correcting.


[-- Attachment #2: Type: text/html, Size: 1715 bytes --]

Re: Bug when $USERNAME is cyryllic

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Sat, Sep 30, 2023 at 10:05 PM Максим Щербаков
<herobrine135111@gmail.com> wrote:
>
> As we can see, $LOGNAME is fine, but $USERNAME and %n are bad.

Patch appended.

As mentioned in another thread, this presumes that it's harmless to
metafy something that's already metafied.  If that's not true then the
patch will fix USERNAME but break LOGNAME.

Aside:  If USE_GETPWUID is not defined and DISABLE_DYNAMIC_NSS is also
defined, then both USERNAME and LOGNAME will be empty.

> zcompile: can't write zwc file: /home/Студент/.zc/.zcompdump-5.9.zwc

The opposite problem.

[-- Attachment #2: meta-username.txt --]
[-- Type: text/plain, Size: 1309 bytes --]

diff --git a/Src/init.c b/Src/init.c
index ffb017e22..799ad19f6 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -1212,8 +1212,8 @@ setupvals(char *cmd, char *runscript, char *zsh_name)
 #ifdef USE_GETPWUID
     if ((pswd = getpwuid(cached_uid))) {
 	if (EMULATION(EMULATE_ZSH))
-	    home = metafy(pswd->pw_dir, -1, META_DUP);
-	cached_username = ztrdup(pswd->pw_name);
+	    home = ztrdup_metafy(pswd->pw_dir);
+	cached_username = ztrdup_metafy(pswd->pw_name);
     }
     else
 #endif /* USE_GETPWUID */
diff --git a/Src/params.c b/Src/params.c
index 5841308d7..50e8627d1 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -4561,7 +4561,7 @@ usernamesetfn(UNUSED(Param pm), char *x)
 	    zwarn("failed to change user ID: %e", errno);
 	else {
 	    zsfree(cached_username);
-	    cached_username = ztrdup(pswd->pw_name);
+	    cached_username = ztrdup_metafy(pswd->pw_name);
 	    cached_uid = pswd->pw_uid;
 	}
     }
diff --git a/Src/utils.c b/Src/utils.c
index 7028c155f..790625379 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -1069,7 +1069,7 @@ get_username(void)
 	cached_uid = current_uid;
 	zsfree(cached_username);
 	if ((pswd = getpwuid(current_uid)))
-	    cached_username = ztrdup(pswd->pw_name);
+	    cached_username = ztrdup_metafy(pswd->pw_name);
 	else
 	    cached_username = ztrdup("");
     }

[-- Attachment #3: unmeta-zcompile.txt --]
[-- Type: text/plain, Size: 1815 bytes --]

diff --git a/Src/parse.c b/Src/parse.c
index a07a6cc71..e76602524 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -3217,12 +3217,14 @@ bin_zcompile(char *nam, char **args, Options ops, UNUSED(int func))
 
     if (!args[1] && !(OPT_ISSET(ops,'c') || OPT_ISSET(ops,'a'))) {
 	queue_signals();
-	ret = build_dump(nam, dyncat(*args, FD_EXT), args, OPT_ISSET(ops,'U'),
+	dump = unmetafy(dyncat(*args, FD_EXT), NULL);
+	ret = build_dump(nam, dump, args, OPT_ISSET(ops,'U'),
 			 map, flags);
 	unqueue_signals();
 	return ret;
     }
-    dump = (strsfx(FD_EXT, *args) ? *args : dyncat(*args, FD_EXT));
+    dump = (strsfx(FD_EXT, *args) ? ztrdup(*args) : dyncat(*args, FD_EXT));
+    unmetafy(dump, NULL);
 
     queue_signals();
     ret = ((OPT_ISSET(ops,'c') || OPT_ISSET(ops,'a')) ?
@@ -3400,6 +3402,7 @@ build_dump(char *nam, char *dump, char **files, int ali, int map, int flags)
 
     for (hlen = FD_PRELEN, tlen = 0; *files; files++) {
 	struct stat st;
+	char *fnam;
 
 	if (check_cond(*files, "k")) {
 	    flags = (flags & ~(FDHF_KSHLOAD | FDHF_ZSHLOAD)) | FDHF_KSHLOAD;
@@ -3408,7 +3411,8 @@ build_dump(char *nam, char *dump, char **files, int ali, int map, int flags)
 	    flags = (flags & ~(FDHF_KSHLOAD | FDHF_ZSHLOAD)) | FDHF_ZSHLOAD;
 	    continue;
 	}
-	if ((fd = open(*files, O_RDONLY)) < 0 ||
+	fnam = unmeta(*files);
+	if ((fd = open(fnam, O_RDONLY)) < 0 ||
 	    fstat(fd, &st) != 0 || !S_ISREG(st.st_mode) ||
 	    (flen = lseek(fd, 0, 2)) == -1) {
 	    if (fd >= 0)
@@ -3417,8 +3421,10 @@ build_dump(char *nam, char *dump, char **files, int ali, int map, int flags)
 	    zwarnnam(nam, "can't open file: %s", *files);
 	    noaliases = ona;
 	    unlink(dump);
+	    zsfree(fnam);
 	    return 1;
 	}
+	zsfree(fnam);
 	file = (char *) zalloc(flen + 1);
 	file[flen] = '\0';
 	lseek(fd, 0, 0);

Re: Bug when $USERNAME is cyryllic

[Максим]

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

Tested both patches:

    env -i LANG=C =zsh-5.9.0.1-dev -fc 'cd; echo "$USERNAME | ${(%):-%n} |
$LOGNAME"'

Now results in:

    Студент | Студент | Ѓ�ту�дент

*USERNAME* and *%n* works, but *LOGNAME* is broken now (both in C and
C.UTF-8 locales)


Btw, *zcompile* now works just fine

вс, 1 окт. 2023 г., 21:05 Bart Schaefer <schaefer@brasslantern.com>:

> On Sat, Sep 30, 2023 at 10:05 PM Максим Щербаков
> <herobrine135111@gmail.com> wrote:
> >
> > As we can see, $LOGNAME is fine, but $USERNAME and %n are bad.
>
> Patch appended.
>
> As mentioned in another thread, this presumes that it's harmless to
> metafy something that's already metafied.  If that's not true then the
> patch will fix USERNAME but break LOGNAME.
>
> Aside:  If USE_GETPWUID is not defined and DISABLE_DYNAMIC_NSS is also
> defined, then both USERNAME and LOGNAME will be empty.
>
> > zcompile: can't write zwc file: /home/Студент/.zc/.zcompdump-5.9.zwc
>
> The opposite problem.
>
>

[-- Attachment #2: Type: text/html, Size: 1789 bytes --]

Re: Bug when $USERNAME is cyryllic

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

On Sun, Oct 1, 2023 at 11:55 AM Максим <herobrine135111@gmail.com> wrote:
>
>     env -i LANG=C =zsh-5.9.0.1-dev -fc 'cd; echo "$USERNAME | ${(%):-%n} | $LOGNAME"'
>
> Now results in:
>
>     Студент | Студент | Ѓ�ту�дент

This should fix that.

Aside, my recollection was wrong:  It's tokenize() that appears
harmless to apply more than once, not metafy().

[-- Attachment #2: meta-logname.txt --]
[-- Type: text/plain, Size: 629 bytes --]

diff --git a/Src/params.c b/Src/params.c
index 50e8627d1..957656e3f 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -850,12 +850,11 @@ createparamtable(void)
     setsparam("HOST", ztrdup_metafy(hostnam));
     zfree(hostnam, 256);
 
-    setsparam("LOGNAME", ztrdup_metafy(
+    setsparam("LOGNAME",
 #ifndef DISABLE_DYNAMIC_NSS
-			(str = getlogin()) && *str ?  str :
+	      (str = getlogin()) && *str ? ztrdup_metafy(str) :
 #endif
-				cached_username
-			));
+	      ztrdup(cached_username));
 
 #if !defined(HAVE_PUTENV) && !defined(USE_SET_UNSET_ENV)
     /* Copy the environment variables we are inheriting to dynamic *

Re: Bug when $USERNAME is cyryllic

[Максим]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

Everything works now, thanks!

вс, 1 окт. 2023 г., 23:35 Bart Schaefer <schaefer@brasslantern.com>:

> On Sun, Oct 1, 2023 at 11:55 AM Максим <herobrine135111@gmail.com> wrote:
> >
> >     env -i LANG=C =zsh-5.9.0.1-dev -fc 'cd; echo "$USERNAME | ${(%):-%n}
> | $LOGNAME"'
> >
> > Now results in:
> >
> >     Студент | Студент | Ѓ�ту�дент
>
> This should fix that.
>
> Aside, my recollection was wrong:  It's tokenize() that appears
> harmless to apply more than once, not metafy().
>

[-- Attachment #2: Type: text/html, Size: 958 bytes --]

Re: Bug when $USERNAME is cyryllic

[Peter Stephenson]

> On 01/10/2023 21:35 Bart Schaefer <schaefer@brasslantern.com> wrote:
> Aside, my recollection was wrong:  It's tokenize() that appears
> harmless to apply more than once, not metafy().

In case this is useful to jog memories in future...

I think the difference is that metafy() is operating on something
completely raw, so we can't rely on any octet having any special
internal meaning.  tokenize(), on the other hand, always operates
on an already metafied string, hence we at least have the assurance that
anything stuck behind a Meta marker is special and can be left alone.

pws