* print -v with multibyte characters
@ 2020-02-20 19:03 zsugabubus
2020-02-20 22:58 ` Mikael Magnusson
0 siblings, 1 reply; 8+ messages in thread
From: zsugabubus @ 2020-02-20 19:03 UTC (permalink / raw)
To: zsh-workers
Hi,
$ echo $ZSH_VERSION
5.7.1
$ export {LC_ALL,LANG}=en_US.UTF-8
$ set -o multibyte && echo ok
ok
Good:
$ print ÖÓŐöóő
ÖÓŐöóő
$ printf -v var ÖÓŐöóő; echo $var
ÖÓŐöóő
Bad:
$ print -v var ÖÓŐ; echo $var
öó
$ print -v var öóő; echo $var
öóŃ
--
zsugabubus
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: print -v with multibyte characters
2020-02-20 19:03 print -v with multibyte characters zsugabubus
@ 2020-02-20 22:58 ` Mikael Magnusson
2020-08-05 16:13 ` Mikael Magnusson
0 siblings, 1 reply; 8+ messages in thread
From: Mikael Magnusson @ 2020-02-20 22:58 UTC (permalink / raw)
To: zsugabubus; +Cc: zsh-workers
On 2/20/20, zsugabubus <zsugabubus@national.shitposting.agency> wrote:
> Hi,
>
> $ echo $ZSH_VERSION
> 5.7.1
> $ export {LC_ALL,LANG}=en_US.UTF-8
> $ set -o multibyte && echo ok
> ok
>
> Good:
> $ print ÖÓŐöóő
> ÖÓŐöóő
> $ printf -v var ÖÓŐöóő; echo $var
> ÖÓŐöóő
>
> Bad:
> $ print -v var ÖÓŐ; echo $var
> öó
> $ print -v var öóő; echo $var
> öóŃ
This gets closer to correct, but seems to leave out the final byte or
two, or change it somehow,
diff --git i/Src/builtin.c w/Src/builtin.c
index 168bf8863b..ed26717b5b 100644
--- i/Src/builtin.c
+++ w/Src/builtin.c
@@ -4848,8 +4848,7 @@ bin_print(char *name, char **args, Options ops, int func)
if (ret)
free(buf);
else
- setsparam(OPT_ARG(ops, 'v'),
- metafy(buf, rcount, META_REALLOC));
+ setsparam(OPT_ARG(ops, 'v'), buf);
unqueue_signals();
}
return ret;
@@ -4972,8 +4971,7 @@ bin_print(char *name, char **args, Options ops, int func)
if (ret)
free(buf);
else
- setsparam(OPT_ARG(ops, 'v'),
- metafy(buf, rcount, META_REALLOC));
+ setsparam(OPT_ARG(ops, 'v'), buf);
unqueue_signals();
}
return ret;
Incidentally you can use print -v var -f %s ÖÓŐ; echo $var to work
around the problem (the handling for -f uses different code which
doesn't have the bug).
--
Mikael Magnusson
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: print -v with multibyte characters
2020-02-20 22:58 ` Mikael Magnusson
@ 2020-08-05 16:13 ` Mikael Magnusson
2020-08-06 6:51 ` Daniel Shahaf
2020-08-07 10:10 ` Jun T
0 siblings, 2 replies; 8+ messages in thread
From: Mikael Magnusson @ 2020-08-05 16:13 UTC (permalink / raw)
To: zsh-workers
On 2/20/20, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 2/20/20, zsugabubus <zsugabubus@national.shitposting.agency> wrote:
>> Hi,
>>
>> $ echo $ZSH_VERSION
>> 5.7.1
>> $ export {LC_ALL,LANG}=en_US.UTF-8
>> $ set -o multibyte && echo ok
>> ok
>>
>> Good:
>> $ print ÖÓŐöóő
>> ÖÓŐöóő
>> $ printf -v var ÖÓŐöóő; echo $var
>> ÖÓŐöóő
>>
>> Bad:
>> $ print -v var ÖÓŐ; echo $var
>> öó
>> $ print -v var öóő; echo $var
>> öóŃ
>
> This gets closer to correct, but seems to leave out the final byte or
> two, or change it somehow,
> diff --git i/Src/builtin.c w/Src/builtin.c
> index 168bf8863b..ed26717b5b 100644
> --- i/Src/builtin.c
> +++ w/Src/builtin.c
> @@ -4848,8 +4848,7 @@ bin_print(char *name, char **args, Options ops, int
> func)
> if (ret)
> free(buf);
> else
> - setsparam(OPT_ARG(ops, 'v'),
> - metafy(buf, rcount, META_REALLOC));
> + setsparam(OPT_ARG(ops, 'v'), buf);
> unqueue_signals();
> }
> return ret;
> @@ -4972,8 +4971,7 @@ bin_print(char *name, char **args, Options ops, int
> func)
> if (ret)
> free(buf);
> else
> - setsparam(OPT_ARG(ops, 'v'),
> - metafy(buf, rcount, META_REALLOC));
> + setsparam(OPT_ARG(ops, 'v'), buf);
> unqueue_signals();
> }
> return ret;
>
> Incidentally you can use print -v var -f %s ÖÓŐ; echo $var to work
> around the problem (the handling for -f uses different code which
> doesn't have the bug).
Just looking through some pending patches and saw this is unresolved.
Does anyone have any ideas on how this can be fixed?
--
Mikael Magnusson
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: print -v with multibyte characters
2020-08-05 16:13 ` Mikael Magnusson
@ 2020-08-06 6:51 ` Daniel Shahaf
2020-08-07 10:10 ` Jun T
1 sibling, 0 replies; 8+ messages in thread
From: Daniel Shahaf @ 2020-08-06 6:51 UTC (permalink / raw)
To: zsh-workers
Mikael Magnusson wrote on Wed, 05 Aug 2020 18:13 +0200:
> Just looking through some pending patches and saw this is unresolved.
> Does anyone have any ideas on how this can be fixed?
No idea, but if you notice other outstanding issues, feel free to add them to Etc/BUGS.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: print -v with multibyte characters
2020-08-05 16:13 ` Mikael Magnusson
2020-08-06 6:51 ` Daniel Shahaf
@ 2020-08-07 10:10 ` Jun T
2020-08-07 22:29 ` PATCH: Test for print -v fix Mikael Magnusson
1 sibling, 1 reply; 8+ messages in thread
From: Jun T @ 2020-08-07 10:10 UTC (permalink / raw)
To: zsh-workers
In the case of 'print -v var', args[] are RE-metafied at line 4871
(builtin.c), but the length len[] is not updated.
At line 4959,
fwrite(*args, *len, 1, fout);
this metafied args[] are written to buf with the wrong length len[].
I think args[] should not have been RE-metafied at line 4871, because
without '-v var' args[] are no metafied at this fwrite().
We need to metafy buf for setsparam(), as the current code wrongly
does at line 4981.
diff --git a/Src/builtin.c b/Src/builtin.c
index ff84ce936..09eb3728c 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -4862,7 +4862,7 @@ bin_print(char *name, char **args, Options ops, int func)
/* normal output */
if (!fmt) {
- if (OPT_ISSET(ops, 'z') || OPT_ISSET(ops, 'v') ||
+ if (OPT_ISSET(ops, 'z') ||
OPT_ISSET(ops, 's') || OPT_ISSET(ops, 'S')) {
/*
* We don't want the arguments unmetafied after all.
^ permalink raw reply [flat|nested] 8+ messages in thread
* PATCH: Test for print -v fix
2020-08-07 10:10 ` Jun T
@ 2020-08-07 22:29 ` Mikael Magnusson
2020-08-11 4:34 ` Jun T
0 siblings, 1 reply; 8+ messages in thread
From: Mikael Magnusson @ 2020-08-07 22:29 UTC (permalink / raw)
To: zsh-workers
---
This seems to work for me, thanks! Here's a test for it.
Test/B03print.ztst | 1 +
Test/D07multibyte.ztst | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/Test/B03print.ztst b/Test/B03print.ztst
index 0ef3743ce3..5634239346 100644
--- a/Test/B03print.ztst
+++ b/Test/B03print.ztst
@@ -4,6 +4,7 @@
# Use of print -p to output to coprocess A01grammar
# Prompt expansion with print -P D01prompt
# -l, -r, -R and -n indirectly tested in various places
+# multibyte tests in D07multibyte
# Not yet tested:
# echo and pushln
diff --git a/Test/D07multibyte.ztst b/Test/D07multibyte.ztst
index 989f451837..345e061d94 100644
--- a/Test/D07multibyte.ztst
+++ b/Test/D07multibyte.ztst
@@ -570,6 +570,15 @@
0:printf %q and quotestring and general metafy / token madness
>你你
+ typeset -a foo
+ print -v foo 'ÖÓŐ'
+ echo $foo
+ printf -v foo 'ÖÓŐ'
+ echo $foo
+0:print and printf into a variable with multibyte text
+>ÖÓŐ
+>ÖÓŐ
+
# This test is kept last as it introduces an additional
# dependency on the system regex library.
if zmodload zsh/regex 2>/dev/null; then
--
2.15.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: PATCH: Test for print -v fix
2020-08-07 22:29 ` PATCH: Test for print -v fix Mikael Magnusson
@ 2020-08-11 4:34 ` Jun T
2020-08-12 9:45 ` Mikael Magnusson
0 siblings, 1 reply; 8+ messages in thread
From: Jun T @ 2020-08-11 4:34 UTC (permalink / raw)
To: zsh-workers
> 2020/08/08 7:29, Mikael Magnusson <mikachu@gmail.com> wrote:
>
> --- a/Test/D07multibyte.ztst
> +++ b/Test/D07multibyte.ztst
> @@ -570,6 +570,15 @@
> 0:printf %q and quotestring and general metafy / token madness
>> 你你
>
> + typeset -a foo
Why -a?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: PATCH: Test for print -v fix
2020-08-11 4:34 ` Jun T
@ 2020-08-12 9:45 ` Mikael Magnusson
0 siblings, 0 replies; 8+ messages in thread
From: Mikael Magnusson @ 2020-08-12 9:45 UTC (permalink / raw)
To: Jun T; +Cc: zsh-workers
On 8/11/20, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
>
>> 2020/08/08 7:29, Mikael Magnusson <mikachu@gmail.com> wrote:
>>
>> --- a/Test/D07multibyte.ztst
>> +++ b/Test/D07multibyte.ztst
>> @@ -570,6 +570,15 @@
>> 0:printf %q and quotestring and general metafy / token madness
>>> 你你
>>
>> + typeset -a foo
>
> Why -a?
Ah, must have been because I copypasted from another testcase :). Will
fix before pushing it.
--
Mikael Magnusson
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-08-12 9:45 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-20 19:03 print -v with multibyte characters zsugabubus
2020-02-20 22:58 ` Mikael Magnusson
2020-08-05 16:13 ` Mikael Magnusson
2020-08-06 6:51 ` Daniel Shahaf
2020-08-07 10:10 ` Jun T
2020-08-07 22:29 ` PATCH: Test for print -v fix Mikael Magnusson
2020-08-11 4:34 ` Jun T
2020-08-12 9:45 ` Mikael Magnusson
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/zsh/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).