Encoding bug?

Encoding bug?

[☈king]

In v5.0.0 (revision 361e171), if I do:

% echo ♔
% r

I get Mojibake.

If I instead do:

% echo ♔
% fc

I also get a case of the 'baks.

My locale settings are no different than from when I ran an older zsh:
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=en_US.utf8


I did talk to one other user on #zsh that is using 5.0.0 and didn't
experience the problem, and one who did. I have no clue what the
difference between the odd man out might be.

Thanks!
—☈

Re: Encoding bug?

[Benjamin R. Haskell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1482 bytes --]

On Fri, 7 Sep 2012, ☈king wrote:

> In v5.0.0 (revision 361e171), if I do:
>
> % echo ♔
> % r
>
> I get Mojibake.
>
> If I instead do:
>
> % echo ♔
> % fc
>
> I also get a case of the 'baks.
>
> My locale settings are no different than from when I ran an older zsh:
> LANG=en_US.utf8
> LC_CTYPE="en_US.utf8"
> LC_NUMERIC="en_US.utf8"
> LC_TIME="en_US.utf8"
> LC_COLLATE="en_US.utf8"
> LC_MONETARY="en_US.utf8"
> LC_MESSAGES="en_US.utf8"
> LC_PAPER="en_US.utf8"
> LC_NAME="en_US.utf8"
> LC_ADDRESS="en_US.utf8"
> LC_TELEPHONE="en_US.utf8"
> LC_MEASUREMENT="en_US.utf8"
> LC_IDENTIFICATION="en_US.utf8"
> LC_ALL=en_US.utf8
>
>
> I did talk to one other user on #zsh that is using 5.0.0 and didn't 
> experience the problem, and one who did. I have no clue what the 
> difference between the odd man out might be.

Not sure where something's not being encoded properly, but I get the 
same results here under 4.3.12 patch 1.5346 w/ LANG/LC_* set to 
en_US.UTF-8.  Also fails on git tag zsh-4.3.10.  So, it's not a new 
issue, AFAICT.

♔ = \U2654 = UTF-8: 0xe2 0x99 0x94

In my HISTORYFILE it ends up being encoded as:

0xe2 0x83 0xb9 0x83 0xb4

If I run:

$ echo ♔
$ r

and then arrow-up, it ends up displaying as: echo <20f9><83><b4> (which 
jives with the HISTORYFILE encoding, but seems notable inasmuch as zsh 
is 'aware' that the characters are different).

In the process of git-bisecting, but I won't be able to finish til 
tomorrow sometime (probably).

-- 
Best,
Ben

Re: Encoding bug?

[Damien Thébault]

On Fri, Sep 7, 2012 at 6:45 PM, ☈king <rkingdesk@sharpsaw.org> wrote:
> In v5.0.0 (revision 361e171), if I do:
>
> % echo ♔
> % r
>
> I get Mojibake.
>
> If I instead do:
>
> % echo ♔
> % fc

Hello,

I have the same issue with v5.0.0 . And it seems not to be the same
behaviour with every non-ascii charater I tried.

echo é # works
echo eée # works
echo eéàe # r prints garbage
echo eéèe # works
echo oôöo # works
echo æ«€¶ŧ←↓→øþ # r prints garbage starting with arrows
echo ←↓→ # r prints garbage

Hope it helps locating the issue.
-- 
Damien Thebault

Re: Encoding bug?

[Peter Stephenson]

> echo eéàe # r prints garbage

Sorry if this gets screwed up, I'm using webmail on the other laptop
at the moment.

Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.261
diff -u -r1.261 builtin.c
--- Src/builtin.c	21 Aug 2012 18:03:03 -0000	1.261
+++ Src/builtin.c	8 Sep 2012 19:12:38 -0000
@@ -1727,8 +1727,12 @@
 	    if (f == stdout) {
 		nicezputs(s, f);
 		putc('\n', f);
-	    } else
-		fprintf(f, "%s\n", s);
+	    } else {
+		int len;
+		unmetafy(s, &len);
+		fwrite(s, 1, len, f);
+		putc('\n', f);
+	    }
 	}
 	/* move on to the next history line, or quit the loop */
 	if (first < last) {

pws

Re: Encoding bug?

[Benjamin R. Haskell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1019 bytes --]

On Sat, 8 Sep 2012, Peter Stephenson wrote:

>> echo eéàe # r prints garbage
>
> Sorry if this gets screwed up, I'm using webmail on the other laptop
> at the moment.

Verified that this solves the problem.

Here's your patch against the git repo.  (Also hoping it solves the 
tabs-vs.-spaces discrepancies probably caused by webmail.)

Is the git repo the official source now?  I thought that was another 
5.0.0 goal/plan.

Best,
Ben

---
  Src/builtin.c |    8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/Src/builtin.c b/Src/builtin.c
index 3925edd..d8493bb 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -1727,8 +1727,12 @@ fclist(FILE *f, Options ops, zlong first, zlong last,
  	    if (f == stdout) {
  		nicezputs(s, f);
  		putc('\n', f);
-	    } else
-		fprintf(f, "%s\n", s);
+	    } else {
+		int len;
+		unmetafy(s, &len);
+		fwrite(s, 1, len, f);
+		putc('\n', f);
+	    }
  	}
  	/* move on to the next history line, or quit the loop */
  	if (first < last) {
-- 
1.7.7

Re: Encoding bug?

[Bart Schaefer]

On Sat, Sep 8, 2012 at 12:34 PM, Benjamin R. Haskell <zsh@benizi.com> wrote:
>
> Is the git repo the official source now?  I thought that was another 5.0.0
> goal/plan.

I believe the git repo is still mirroring CVS.  At least, I have not
heard otherwise.

Re: Encoding bug?

[☈king]

Commit dd3a8ea perfectly fixes my issue.

Thanks for the prompt fix. Woohoo!
—☈