Multibyte test fails on MacOSX Leopard

Multibyte test fails on MacOSX Leopard

[İsmail Dönmez]

Hi all,

Using MacOSX Leopard 10.5.2 with latest CVS I get,

Testing multibyte with locale en_US.UTF-8
*** /tmp/zsh.ztst.out.91960	2008-02-26 22:58:32.000000000 +0200
--- /tmp/zsh.ztst.tout.91960	2008-02-26 22:58:32.000000000 +0200
***************
*** 1,2 ****
! HAH HEH HÉH HÈH HUH
  HAH HEH HUH HÈH HÉH
--- 1,2 ----
! HAH HEH HUH HÈH HÉH
  HAH HEH HUH HÈH HÉH
Test ./D07multibyte.ztst failed: output differs from expected as shown
above for:
  print -oi HAH HUH HEH HÉH HÈH
  (LC_ALL=C; print -oi HAH HUH HEH HÉH HÈH)
Was testing: Multibyte characters in print sorting


Did anyone see this before? I don't see any functional problem with
respect to multibyte handling but I think its a good idea to report
this anyway.

Regards,
ismail

-- 
Never learn by your mistakes, if you do you may never dare to try again

Re: Multibyte test fails on MacOSX Leopard

[Peter Stephenson]

On Tue, 26 Feb 2008 23:00:16 +0200
"İsmail Dönmez" <ismail@namtrac.org> wrote:
> Hi all,
> 
> Using MacOSX Leopard 10.5.2 with latest CVS I get,
> 
> Testing multibyte with locale en_US.UTF-8
> *** /tmp/zsh.ztst.out.91960	2008-02-26 22:58:32.000000000 +0200
> --- /tmp/zsh.ztst.tout.91960	2008-02-26 22:58:32.000000000 +0200
> ***************
> *** 1,2 ****
> ! HAH HEH HÉH HÈH HUH
>   HAH HEH HUH HÈH HÉH
> --- 1,2 ----
> ! HAH HEH HUH HÈH HÉH
>   HAH HEH HUH HÈH HÉH
> Test ./D07multibyte.ztst failed: output differs from expected as shown
> above for:
>   print -oi HAH HUH HEH HÉH HÈH
>   (LC_ALL=C; print -oi HAH HUH HEH HÉH HÈH)
> Was testing: Multibyte characters in print sorting
> 
> 
> Did anyone see this before? I don't see any functional problem with
> respect to multibyte handling but I think its a good idea to report
> this anyway.

Yes, it looks like the collation sequence for accented characters is
different in your library.  It's a nuisance since that leaves no way of
testing this---it's already forced into en_US.UTF-8 which gives the
expected answer on other systems.

We might be able to compare the output of "sort", I suppose, but that does
rely on sort being internationalized.  It's probably a reasonable bet if
we've got this far.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Multibyte test fails on MacOSX Leopard

[Peter Stephenson]

On Wed, 27 Feb 2008 10:34:56 +0000
Peter Stephenson <pws@csr.com> wrote:
> We might be able to compare the output of "sort", I suppose, but that does
> rely on sort being internationalized.  It's probably a reasonable bet if
> we've got this far.

Let's see what happens.

Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.22
diff -u -r1.22 D07multibyte.ztst
--- Test/D07multibyte.ztst	17 Dec 2007 17:11:29 -0000	1.22
+++ Test/D07multibyte.ztst	27 Feb 2008 19:37:37 -0000
@@ -322,10 +322,16 @@
 # We ask for case-insensitive sorting here (and supply upper case
 # characters) so that we exercise the logic in the shell that lowers the
 # case of the string for case-insensitive sorting.
-  print -oi HAH HUH HEH HÉH HÈH
+# As all letters are upper case, however, sort should produce the same order.
+  print -loi HAH HUH HEH HÉH HÈH >zshsort.txt
+  print -l HAH HUH HEH HÉH HÈH | sort >sortsort.txt
+  print Diff output should be empty
+  diff zshsort.txt sortsort.txt
+  print Sort in C locale
   (LC_ALL=C; print -oi HAH HUH HEH HÉH HÈH)
 0:Multibyte characters in print sorting
->HAH HEH HÉH HÈH HUH
+>Diff output should be empty
+>Sort in C locale
 >HAH HEH HUH HÈH HÉH
 
 # These are control characters in Unicode, so don't show up.

Re: Multibyte test fails on MacOSX Leopard

[İsmail Dönmez]

Hi,

On Wed, Feb 27, 2008 at 9:37 PM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> On Wed, 27 Feb 2008 10:34:56 +0000
>  Peter Stephenson <pws@csr.com> wrote:
>  > We might be able to compare the output of "sort", I suppose, but that does
>  > rely on sort being internationalized.  It's probably a reasonable bet if
>  > we've got this far.
>
>  Let's see what happens.
>
>  Index: Test/D07multibyte.ztst
>  ===================================================================
>  RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
>  retrieving revision 1.22
>  diff -u -r1.22 D07multibyte.ztst
>  --- Test/D07multibyte.ztst      17 Dec 2007 17:11:29 -0000      1.22
>  +++ Test/D07multibyte.ztst      27 Feb 2008 19:37:37 -0000
>  @@ -322,10 +322,16 @@
>   # We ask for case-insensitive sorting here (and supply upper case
>   # characters) so that we exercise the logic in the shell that lowers the
>   # case of the string for case-insensitive sorting.
>  -  print -oi HAH HUH HEH HÉH HÈH
>  +# As all letters are upper case, however, sort should produce the same order.
>  +  print -loi HAH HUH HEH HÉH HÈH >zshsort.txt
>  +  print -l HAH HUH HEH HÉH HÈH | sort >sortsort.txt
>  +  print Diff output should be empty
>  +  diff zshsort.txt sortsort.txt
>  +  print Sort in C locale
>
>    (LC_ALL=C; print -oi HAH HUH HEH HÉH HÈH)
>   0:Multibyte characters in print sorting
>  ->HAH HEH HÉH HÈH HUH
>  +>Diff output should be empty
>  +>Sort in C locale
>
>  >HAH HEH HUH HÈH HÉH
>
>   # These are control characters in Unicode, so don't show up.
>

Gives another error now :

./D07multibyte.ztst: starting.
Testing multibyte with locale en_US.UTF-8
*** /tmp/zsh.ztst.out.2929	2008-02-27 23:10:23.000000000 +0200
--- /tmp/zsh.ztst.tout.2929	2008-02-27 23:10:23.000000000 +0200
***************
*** 4,12 ****
  ngs20.txt
  ngs100.txt
  ngs200.txt
- ngs100.txt
- ngs10.txt
  ngs1.txt
! ngs200.txt
! ngs20.txt
  ngs2.txt
--- 4,12 ----
  ngs20.txt
  ngs100.txt
  ngs200.txt
  ngs1.txt
! ngs10.txt
! ngs100.txt
  ngs2.txt
+ ngs20.txt
+ ngs200.txt
Test ./D07multibyte.ztst failed: output differs from expected as shown
above for:
  touch ngs1.txt ngs2.txt ngs10.txt ngs20.txt ngs100.txt ngs200.txt
  setopt numericglobsort
  print -l ngs*
  unsetopt numericglobsort
  print -l ngs*
Was testing: NUMERIC_GLOB_SORT option in UTF-8 locale

Thanks,
ismail

-- 
Never learn by your mistakes, if you do you may never dare to try again