D07multibyte.ztst failure on HP-UX 11.11

D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 728 bytes --]

Now that I have this executing, it's giving me the following result.
I don't see a difference, so suspect the test itself rather than the shell.

./D07multibyte.ztst: starting.
Testing multibyte with locale en_US.utf8
*** /tmp/zsh.ztst.out.5327      2009-04-24 16:28:27.000000000 -0400
--- /tmp/zsh.ztst.tout.5327     2009-04-24 16:28:27.000000000 -0400
***************
*** 1,2 ****
! HÎH HÎH HÎH HÎH HÎH
  HAH HEH HUH HÃH HÃH
--- 1,2 ----
! HÎH HÎH HÎH HÎH HÎH
  HAH HEH HUH HÃH HÃH
Test ./D07multibyte.ztst failed: output differs from expected as shown above for:
  print -oi HÎH HÎH HÎH HÎH HÎH
  (LC_ALL=C; print -oi HAH HUH HEH HÃH HÃH)
Was testing: Multibyte characters in print sorting
./D07multibyte.ztst: test failed.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Monday 27 of April 2009 07:17:03 Paul Ackersviller wrote:
> Now that I have this executing, it's giving me the following result.
> I don't see a difference, so suspect the test itself rather than the
> shell.
>

Could you verify exact byte sequence with od, xxd or like? It is quite 
possible to have combined vs. non-combined characters here (which look 
alike in printable form but have different internal representation).

> ./D07multibyte.ztst: starting.
> Testing multibyte with locale en_US.utf8
> *** /tmp/zsh.ztst.out.5327      2009-04-24 16:28:27.000000000 -0400
> --- /tmp/zsh.ztst.tout.5327     2009-04-24 16:28:27.000000000 -0400
> ***************
> *** 1,2 ****
> ! HÎH HÎH HÎH HÎH HÎH
>   HAH HEH HUH HÃH HÃH
> --- 1,2 ----
> ! HÎH HÎH HÎH HÎH HÎH
>   HAH HEH HUH HÃH HÃH
> Test ./D07multibyte.ztst failed: output differs from expected as
> shown above for: print -oi HÎH HÎH HÎH HÎH HÎH
>   (LC_ALL=C; print -oi HAH HUH HEH HÃH HÃH)
> Was testing: Multibyte characters in print sorting
> ./D07multibyte.ztst: test failed.


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Mon, Apr 27, 2009 at 08:42:46AM +0400, Andrey Borzenkov wrote:
> Could you verify exact byte sequence with od, xxd or like? It is quite 
> possible to have combined vs. non-combined characters here (which look 
> alike in printable form but have different internal representation).

Of course, should've thought of that the first time.  I've attempted
to annotate the mismatches, but could've missed something.


0000000: 2a2a 2a20 312c 3220 2a2a 2a2a 0a21 2048  *** 1,2 ****.! H
0000010: ce91 4820 48ce 9248 2048 ce93 4820 48ce  ..H H..H H..H H.
            ^            ^
0000020: 9448 2048 ce95 480a 2020 4841 4820 4845  .H H..H.  HAH HE
                      ^
0000030: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.


0000040: 2d2d 2d20 312c 3220 2d2d 2d2d 0a21 2048  --- 1,2 ----.! H
0000050: ce95 4820 48ce 9448 2048 ce93 4820 48ce  ..H H..H H..H H.
            ^            ^
0000060: 9248 2048 ce91 480a 2020 4841 4820 4845  .H H..H.  HAH HE
                      ^
0000070: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Mon, 27 Apr 2009 19:26:43 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> On Mon, Apr 27, 2009 at 08:42:46AM +0400, Andrey Borzenkov wrote:
> > Could you verify exact byte sequence with od, xxd or like? It is quite 
> > possible to have combined vs. non-combined characters here (which look 
> > alike in printable form but have different internal representation).
> 
> Of course, should've thought of that the first time.  I've attempted
> to annotate the mismatches, but could've missed something.
> 
> 
> 0000000: 2a2a 2a20 312c 3220 2a2a 2a2a 0a21 2048  *** 1,2 ****.! H
> 0000010: ce91 4820 48ce 9248 2048 ce93 4820 48ce  ..H H..H H..H H.
>             ^            ^
> 0000020: 9448 2048 ce95 480a 2020 4841 4820 4845  .H H..H.  HAH HE
>           ^           ^
> 0000030: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.
> 
> 
> 0000040: 2d2d 2d20 312c 3220 2d2d 2d2d 0a21 2048  --- 1,2 ----.! H
> 0000050: ce95 4820 48ce 9448 2048 ce93 4820 48ce  ..H H..H H..H H.
>             ^            ^
> 0000060: 9248 2048 ce91 480a 2020 4841 4820 4845  .H H..H.  HAH HE
>           ^           ^
> 0000070: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.

You missed a 94 and a 92 which I've marked: the problem is again that the
sort order isn't quite as deterministic as one might hope.  It looks like
something funny happened to the characters in your original post; this may
or may not be related.  It's possible the problem is in case modification.
The desired answer is that (in the selected UTF-8 locale)

  print -oi HΕH HΔH HΓH HΒH HΑH

outputs

  HΑH HΒH HΓH HΔH HΕH

(the middle letters are all upper case Greek).  Does it work without the
-i?

The sort tests have always been arguably more trouble than their worth,
though I suppose it probably is worth spotlighting where the problems are.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Tue, Apr 28, 2009 at 09:48:32AM +0100, Peter Stephenson wrote:
> On Mon, 27 Apr 2009 19:26:43 +0000
> Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > 
> > 0000000: 2a2a 2a20 312c 3220 2a2a 2a2a 0a21 2048  *** 1,2 ****.! H
> > 0000010: ce91 4820 48ce 9248 2048 ce93 4820 48ce  ..H H..H H..H H.
> >             ^            ^
> > 0000020: 9448 2048 ce95 480a 2020 4841 4820 4845  .H H..H.  HAH HE
> >           ^           ^
> > 0000030: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.
> > 
> > 
> > 0000040: 2d2d 2d20 312c 3220 2d2d 2d2d 0a21 2048  --- 1,2 ----.! H
> > 0000050: ce95 4820 48ce 9448 2048 ce93 4820 48ce  ..H H..H H..H H.
> >             ^            ^
> > 0000060: 9248 2048 ce91 480a 2020 4841 4820 4845  .H H..H.  HAH HE
> >           ^           ^
> > 0000070: 4820 4855 4820 48c3 8848 2048 c389 480a  H HUH H..H H..H.
> 
> You missed a 94 and a 92 which I've marked: the problem is again that the
> sort order isn't quite as deterministic as one might hope.  It looks like
> something funny happened to the characters in your original post; this may
> or may not be related.  It's possible the problem is in case modification.

I don't recall how I did the original post, but it may have been a copy
and paste (sorry).  The xxd output is from a run on the original file.

> The desired answer is that (in the selected UTF-8 locale)
> 
>   print -oi HΕH HΔH HΓH HΒH HΑH
> 
> outputs
> 
>   HΑH HΒH HΓH HΔH HΕH
> 
> (the middle letters are all upper case Greek).  Does it work without the
> -i?

No, no difference on that machine.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Tue, 28 Apr 2009 19:19:20 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> On Tue, Apr 28, 2009 at 09:48:32AM +0100, Peter Stephenson wrote:
>> The desired answer is that (in the selected UTF-8 locale)
>> 
>>   print -oi HΕH HΔH HΓH HΒH HΑH
>> 
>> outputs 
>> 
>>   HΑH HΒH HΓH HΔH HΕH
>> 
>> (the middle letters are all upper case Greek).  Does it work without the
>> -i?
> 
> No, no difference on that machine.

How about accented European characters like

% print -oi HÛH HÔH HÎH HÊH HÂH
HÂH HÊH HÎH HÔH HÛH

?

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Tue, Apr 28, 2009 at 08:48:17PM +0100, Peter Stephenson wrote:
> On Tue, 28 Apr 2009 19:19:20 +0000
> Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > On Tue, Apr 28, 2009 at 09:48:32AM +0100, Peter Stephenson wrote:
> >> The desired answer is that (in the selected UTF-8 locale)
> >> 
> >>   print -oi H?H H?H H?H H?H H?H
> >> 
> >> outputs 
> >> 
> >>   H?H H?H H?H H?H H?H
> >> 
> >> (the middle letters are all upper case Greek).  Does it work without the
> >> -i?
> > 
> > No, no difference on that machine.
> 
> How about accented European characters like
> 
> % print -oi H?H H?H H?H H?H H?H
> H?H H?H H?H H?H H?H

Yes, that appears to give the desired result.  Sorry for not correctly
quoting the utf8 in your last post, either my editor or utf8 locale
isn't behaving on this system.  At least I think my mail client
managed to extract it successfully for this test.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Thu, 30 Apr 2009 03:01:51 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > How about accented European characters like
> > 
> > % print -oi H?H H?H H?H H?H H?H
> > H?H H?H H?H H?H H?H
> 
> Yes, that appears to give the desired result.

OK, I'll limit it to those.  The fact that western European characters work
better may be a hold over from supporting ISO-8859-1, I suppose.

> Sorry for not correctly
> quoting the utf8 in your last post, either my editor or utf8 locale
> isn't behaving on this system.  At least I think my mail client
> managed to extract it successfully for this test.

It seemed to happen again in the reply, not that it's significant this time
round.  I've quoted what I got, although it's possible something nearer
this end decided on the "?"s as a fallback for something it didn't understand.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Thu, Apr 30, 2009 at 09:41:16AM +0100, Peter Stephenson wrote:
> On Thu, 30 Apr 2009 03:01:51 +0000
> Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > > How about accented European characters like
> > > 
> > > % print -oi H?H H?H H?H H?H H?H
> > > H?H H?H H?H H?H H?H
> > 
> > Yes, that appears to give the desired result.
> 
> OK, I'll limit it to those.  The fact that western European characters work
> better may be a hold over from supporting ISO-8859-1, I suppose.

That gets me past that test, then a subsequent one fails.  Hope you
don't mind me keeping the same subject...

./D07multibyte.ztst: starting.
locale en_US.UTF-8 failed
*** /tmp/zsh.ztst.out.4937	2009-04-30 11:23:24.000000000 -0400
--- /tmp/zsh.ztst.tout.4937	2009-04-30 11:23:24.000000000 -0400
***************
*** 1,4 ****
- OK
- OK
- OK
- OK
--- 0 ----
Test ./D07multibyte.ztst failed: output differs from expected as shown above for:
  testfn() { (LC_ALL=C; print $'\u00e9') }
  repeat 4 testfn 2>&1 | while read line; do
    if [[ $line = *"character not in range"* ]]; then
      print OK
    elif [[ $line = "?" ]]; then
      print OK
    else
      print Failed: no error message and no question mark
    fi
  done
  true
Was testing: error handling in Unicode quoting
./D07multibyte.ztst: test failed.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

Paul Ackersviller wrote:
> That gets me past that test, then a subsequent one fails.

Urgh.  I can't help feeling it might be better if they just fixed the
system.

> ./D07multibyte.ztst: starting.
> locale en_US.UTF-8 failed
> *** /tmp/zsh.ztst.out.4937	2009-04-30 11:23:24.000000000 -0400
> --- /tmp/zsh.ztst.tout.4937	2009-04-30 11:23:24.000000000 -0400
> ***************
> *** 1,4 ****
> - OK
> - OK
> - OK
> - OK
> --- 0 ----
> Test ./D07multibyte.ztst failed: output differs from expected as shown above 
> for:
>   testfn() { (LC_ALL=C; print $'\u00e9') }
>   repeat 4 testfn 2>&1 | while read line; do
>     if [[ $line = *"character not in range"* ]]; then
>       print OK
>     elif [[ $line = "?" ]]; then
>       print OK
>     else
>       print Failed: no error message and no question mark
>     fi
>   done
>   true
> Was testing: error handling in Unicode quoting
> ./D07multibyte.ztst: test failed.

Eh?  How did you manage to get *no* output from that test?  What does
testfn do when run on it's own?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

Sorry if anyone gets this twice, I sent it yesterday but haven't seen
it show up on the list yet.

On Thu, Apr 30, 2009 at 05:03:59PM +0100, Peter Stephenson wrote:
> Paul Ackersviller wrote:
> > That gets me past that test, then a subsequent one fails.
> 
> Urgh.  I can't help feeling it might be better if they just fixed the
> system.

Granted, it is and old, if not already-obsolete, system.

> > ./D07multibyte.ztst: starting.
> > locale en_US.UTF-8 failed
> > *** /tmp/zsh.ztst.out.4937	2009-04-30 11:23:24.000000000 -0400
> > --- /tmp/zsh.ztst.tout.4937	2009-04-30 11:23:24.000000000 -0400
> > ***************
> > *** 1,4 ****
> > - OK
> > - OK
> > - OK
> > - OK
> > --- 0 ----
> > Test ./D07multibyte.ztst failed: output differs from expected as shown above 
> > for:
> >   testfn() { (LC_ALL=C; print $'\u00e9') }
> >   repeat 4 testfn 2>&1 | while read line; do
> >     if [[ $line = *"character not in range"* ]]; then
> >       print OK
> >     elif [[ $line = "?" ]]; then
> >       print OK
> >     else
> >       print Failed: no error message and no question mark
> >     fi
> >   done
> >   true
> > Was testing: error handling in Unicode quoting
> > ./D07multibyte.ztst: test failed.
> 
> Eh?  How did you manage to get *no* output from that test?  What does
> testfn do when run on it's own?

Same thing -- it seems the read is either detecting EOF right off,
or else failing silently somehow.  I don't do the if at all,
but I see I can get there by using a non-utf8 value of LANG.

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

Paul Ackersviller wrote:
> Same thing -- it seems the read is either detecting EOF right off,
> or else failing silently somehow.  I don't do the if at all,
> but I see I can get there by using a non-utf8 value of LANG.
> 
> 
>  To report this email as spam click https://www.mailcontrol.com/sr/dVFeMF0h0+
> nTndxI!oX7Us7Qlo!t9IHtDeKYakRZVp+8CRI1Q160mc558qJIqPSzuTDRjhhY0p1rzgsKYpIG1w=
> = .

Presumably then we need to find out what

LC_ALL=C; print $'\u00e9'

is doing in the code.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Fri, May 01, 2009 at 04:18:47PM +0100, Peter Stephenson wrote:
> Presumably then we need to find out what
> 
> LC_ALL=C; print $'\u00e9'
> 
> is doing in the code.

I meant to get to this on the weekend, but my internet service at home
wasn't working worth crap.  Anyway, I'm about to trace into the code
for this.  First I thought a couple of other details may tweak something
for you, they don't make any sense to me.

I can get read to silently fail on the HP box with

env -i LANG=en_US.utf8 ../Src/zsh -fc \
	"(LC_ALL=C; print \$'\\u00e9') | read || print failure"

Taking out any of the read, the LC_ALL=C, or the sub-shelling gives a
zero return to the left of the "print failure".

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Tue, 5 May 2009 19:39:31 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> I can get read to silently fail on the HP box with
> 
> env -i LANG=en_US.utf8 ../Src/zsh -fc \
> 	"(LC_ALL=C; print \$'\\u00e9') | read || print failure"
> 
> Taking out any of the read, the LC_ALL=C, or the sub-shelling gives a
> zero return to the left of the "print failure".

So presumably the underlying problem is the same as the failure in the
test that needs investigating.

Taking out the LC_ALL should produce some sensible output if you omit
the read.  (Replacing it with xxd or failing that od -x might make it
clearer what's going on.)

If you're simply taking out the subshell and not replacing it with
anything then the LC_ALL=C covers the "read" as well as the "print".
So possibly something strange is happening in the read.  Replacing it
with xxd might be even more instructive here.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Wed, May 06, 2009 at 08:22:06PM +0100, Peter Stephenson wrote:
> On Tue, 5 May 2009 19:39:31 +0000
> Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > I can get read to silently fail on the HP box with
> > 
> > env -i LANG=en_US.utf8 ../Src/zsh -fc \
> > 	"(LC_ALL=C; print \$'\\u00e9') | read || print failure"
> > 
> > Taking out any of the read, the LC_ALL=C, or the sub-shelling gives a
> > zero return to the left of the "print failure".
> 
> So presumably the underlying problem is the same as the failure in the
> test that needs investigating.
> 
> Taking out the LC_ALL should produce some sensible output if you omit
> the read.  (Replacing it with xxd or failing that od -x might make it
> clearer what's going on.)

Not quite: "zsh:1: cannot do charset conversion (iconv failed)"

> If you're simply taking out the subshell and not replacing it with
> anything then the LC_ALL=C covers the "read" as well as the "print".
> So possibly something strange is happening in the read.  Replacing it
> with xxd might be even more instructive here.

This gives
	0000000 c50a
Does this mean the 0a should be the second byte, but is perhaps being
interpreted as newline?

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Wed, 6 May 2009 21:50:26 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> On Wed, May 06, 2009 at 08:22:06PM +0100, Peter Stephenson wrote:
> > On Tue, 5 May 2009 19:39:31 +0000
> > Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > > I can get read to silently fail on the HP box with
> > > 
> > > env -i LANG=en_US.utf8 ../Src/zsh -fc \
> > > 	"(LC_ALL=C; print \$'\\u00e9') | read || print failure"
>
>> > Taking out the LC_ALL should produce some sensible output if you omit
> > the read.  (Replacing it with xxd or failing that od -x might make it
> > clearer what's going on.)
> 
> Not quite: "zsh:1: cannot do charset conversion (iconv failed)"

It's not clear why it should fail, but the error message is OK and allowed
for by the test.

> > If you're simply taking out the subshell and not replacing it with
> > anything then the LC_ALL=C covers the "read" as well as the "print".
> > So possibly something strange is happening in the read.  Replacing it
> > with xxd might be even more instructive here.
> 
> This gives
> 	0000000 c50a
> Does this mean the 0a should be the second byte, but is perhaps being
> interpreted as newline?

So this comes from

 env -i LANG=en_US.utf8 ../Src/zsh -fc \
   "LC_ALL=C; print \$'\\u00e9' | read || print failure"

I get "character not in range" here.  It looks like your system is
outputting 0xc5, which I wouldn't expect to be a valid character in the C
locale, and I can't work out why it comes from Unicode character 0xe9.  The
UTF-8 would be 0xc3a9, the ISO-8859-1 or -15 would be 0xe9.

The 0x0a really is a newline.

In the test you show, read is running with UTF-8.  I can confirm that
on my system (where I happen already to be in the en_GB.UTF-8 locale)

  (unsetopt multibyte; print $'\xc5') | xxd

gives what you're sending to read, and

  (unsetopt multibyte; print $'\xc5') | read

returns status 1 with no output.

So this all tallies, and I think we've found out all we need, but I'm not
sure about the fix; possibly read should output an error on an invalid
character in MULTIBYTE mode (which we could add to the test)?  Does anyone
see a problem with that?

I'm fairly happy this isn't a shell bug, but I'd still like the shell to
have enough facilities to be able to detect the problem.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Thu, 7 May 2009 16:38:19 +0100
Peter Stephenson <pws@csr.com> wrote:
>   (unsetopt multibyte; print $'\xc5') | read
> 
> returns status 1 with no output.
> 
> I'm fairly happy this isn't a shell bug...

Having said that, actually there's no good reason why "read" shouldn't just
accept any old junk and let whoever is looking at the variable handle it.
This fixes that, and adds a test for it.

This means your test still fails, however.  I'm not sure what to do about
this since clearly the test *does* fail; you don't get a valid character or
an error.  Since character set conversion isn't working on your system (as
evinced by the result of not setting the C locale), I suppose we could test
that and if so skip the test?

Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.224
diff -u -r1.224 builtin.c
--- Src/builtin.c	25 Mar 2009 09:52:41 -0000	1.224
+++ Src/builtin.c	7 May 2009 15:55:58 -0000
@@ -5546,8 +5546,12 @@
 		wc = (wchar_t)c;
 	    }
 	    if (ret != MB_INCOMPLETE) {
-		if (ret == MB_INVALID)
+		if (ret == MB_INVALID) {
 		    memset(&mbs, 0, sizeof(mbs));
+		    /* Treat this as a single character */
+		    wc = (wchar_t)c;
+		    laststart = bptr;
+		}
 		/*
 		 * \ at the end of a line introduces a continuation line,
 		 * except in raw mode (-r option)
Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.31
diff -u -r1.31 D07multibyte.ztst
--- Test/D07multibyte.ztst	30 Apr 2009 08:37:13 -0000	1.31
+++ Test/D07multibyte.ztst	7 May 2009 15:55:58 -0000
@@ -427,3 +427,9 @@
 >Ą Пётр Ильич Чайковский 梶浦由記
 >Ą Пётр Ильич Чайковский 梶浦由記
 >Ą Пётр Ильич Чайковский 梶浦由記
+
+  setopt cbases
+  print $'\xc5' | read
+  print $(( [#16] #REPLY ))
+0:read passes through invalid multibyte characters
+>0xC5


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Paul Ackersviller]

On Thu, May 07, 2009 at 05:02:19PM +0100, Peter Stephenson wrote:
> On Thu, 7 May 2009 16:38:19 +0100
> Peter Stephenson <pws@csr.com> wrote:
> >   (unsetopt multibyte; print $'\xc5') | read
> > 
> > returns status 1 with no output.
> > 
> > I'm fairly happy this isn't a shell bug...
> 
> Having said that, actually there's no good reason why "read" shouldn't just
> accept any old junk and let whoever is looking at the variable handle it.
> This fixes that, and adds a test for it.

Thanks, it works as advertised.

> This means your test still fails, however.  I'm not sure what to do about
> this since clearly the test *does* fail; you don't get a valid character or
> an error.  Since character set conversion isn't working on your system (as
> evinced by the result of not setting the C locale), I suppose we could test
> that and if so skip the test?

Sounds like about the best that can be done under the circumstances.

BTW, there's one other thing with HP-UX tests I've been ignoring until
now, but would like to look at shortly.  They end with:
	rm: cannot remove `Modules/zsh/parameter.so': Text file busy
	rm: cannot remove `Modules/zsh/zutil.so': Text file busy
	rm: cannot remove `Modules/zsh/complete.so': Text file busy
	rm: cannot remove `Modules/zsh/complist.so': Text file busy
	rm: cannot remove `Modules/zsh/computil.so': Text file busy
	rm: cannot remove `Modules/zsh/zle.so': Text file busy

Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.11)

[Bart Schaefer]

On May 7, 10:08pm, Paul Ackersviller wrote:
} Subject: Re: D07multibyte.ztst failure on HP-UX 11.11
}
} BTW, there's one other thing with HP-UX tests I've been ignoring until
} now, but would like to look at shortly.  They end with:
} 	rm: cannot remove `Modules/zsh/parameter.so': Text file busy
} 	rm: cannot remove `Modules/zsh/zutil.so': Text file busy
} 	rm: cannot remove `Modules/zsh/complete.so': Text file busy
} 	rm: cannot remove `Modules/zsh/complist.so': Text file busy
} 	rm: cannot remove `Modules/zsh/computil.so': Text file busy
} 	rm: cannot remove `Modules/zsh/zle.so': Text file busy

The test script creates a shadow directory with copies of all the
shared objects so that it has a stable module_path for zmodload.
HP-UX does paging for shared objects directly out of their files,
so those files can't be removed while loaded by an application.

It might work to "zmodload -u" the modules before "rm"ing the files.
However:

schaefer<501> zmodload -u $(zmodload)
zmodload: module zsh/complete is in use by another module and cannot be unloaded
zmodload: module zsh/zle is in use by another module and cannot be unloaded
schaefer<502> zmodload -u $(zmodload)
Program received signal SIGSEGV, Segmentation fault.
0x081284c7 in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
739             if (nbuf[ln]) {

(gdb) where
#0  0x081284c7 in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
#1  0x08129087 in zrefresh () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:1123
#2  0x08120c3f in zleread (lp=0x816c9b0, rp=0x0, flags=3, context=0)
    at ../../../zsh-4.0/Src/Zle/zle_main.c:1202
#3  0x08122a34 in zle_main_entry (cmd=1, ap=0xbffb30b4 "À0û¿")
    at ../../../zsh-4.0/Src/Zle/zle_main.c:1866
#4  0x0807dc91 in zleentry (cmd=1) at ../../zsh-4.0/Src/init.c:1247
#5  0x0807e4ef in inputline () at ../../zsh-4.0/Src/input.c:278
#6  0x0807e3a4 in ingetc () at ../../zsh-4.0/Src/input.c:214
#7  0x0807445a in ihgetc () at ../../zsh-4.0/Src/hist.c:263
#8  0x08085c19 in gettok () at ../../zsh-4.0/Src/lex.c:677
#9  0x080854f9 in zshlex () at ../../zsh-4.0/Src/lex.c:364
#10 0x080a012c in parse_event () at ../../zsh-4.0/Src/parse.c:451
#11 0x0807b035 in loop (toplevel=1, justonce=0) at ../../zsh-4.0/Src/init.c:131
#12 0x0807dfed in zsh_main (argc=1, argv=0xbffb3294)
    at ../../zsh-4.0/Src/init.c:1409
#13 0x0804cbea in main (argc=1, argv=0xbffb3294) at ../../zsh-4.0/Src/main.c:93

Apparently zle's cleanup_() routine is not quite comprehensive enough;
maybe some of what's in finish_() needs to be in cleanup_() instead?
Or perhaps it needs to unsetopt ZLE ...

schaefer<501> zmodload -u $(zmodload)
zmodload: module zsh/complete is in use by another module and cannot be unloaded
zmodload: module zsh/zle is in use by another module and cannot be unloaded
schaefer<502> zmodload
zsh/complete
zsh/parameter
zsh/zle
schaefer<503> zmodload -u zsh/complete
schaefer<504> zmodload -u zsh/parameter
schaefer<505> unsetopt zle
schaefer<506> zmodload -u zsh/zle
schaefer<507> setopt zle

Program received signal SIGSEGV, Segmentation fault.
0x081284db in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
739             if (nbuf[ln]) {

Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.11)

[Peter Stephenson]

On Thu, 07 May 2009 16:30:27 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> schaefer<501> zmodload -u $(zmodload)
> zmodload: module zsh/complete is in use by another module and cannot be unloaded
> zmodload: module zsh/zle is in use by another module and cannot be unloaded
> schaefer<502> zmodload -u $(zmodload)
> Program received signal SIGSEGV, Segmentation fault.
> 0x081284c7 in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
> 739             if (nbuf[ln]) {

Does this help?

Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.74
diff -u -r1.74 zle_refresh.c
--- Src/Zle/zle_refresh.c	13 Jan 2009 16:05:46 -0000	1.74
+++ Src/Zle/zle_refresh.c	8 May 2009 08:32:10 -0000
@@ -699,6 +699,8 @@
 #endif
 	nbuf = NULL;
 	obuf = NULL;
+	winw_alloc = -1;
+	winh_alloc = -1;
     }
 }
 
-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.11)

[Bart Schaefer]

On May 8,  9:34am, Peter Stephenson wrote:
} Subject: Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.1
}
} On Thu, 07 May 2009 16:30:27 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > schaefer<501> zmodload -u $(zmodload)
} > zmodload: module zsh/complete is in use by another module and cannot be unloaded
} > zmodload: module zsh/zle is in use by another module and cannot be unloaded
} > schaefer<502> zmodload -u $(zmodload)
} > Program received signal SIGSEGV, Segmentation fault.
} > 0x081284c7 in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
} > 739             if (nbuf[ln]) {
} 
} Does this help?

It moves the segfault somewhere else, at least; now it craps out when
referencing the value of $COLUMNS in my precmd.  (Read more below the
stack trace.)

(gdb) where
#0  0x00462ed0 in _int_malloc () from /lib/tls/libc.so.6
#1  0x00464aa1 in malloc () from /lib/tls/libc.so.6
#2  0x0808ee3d in zshcalloc (size=48) at ../../zsh-4.0/Src/mem.c:601
#3  0x080973fb in createparam (name=0xb7cd63d8 "colno", flags=2097152)
    at ../../zsh-4.0/Src/params.c:841
#4  0x08052923 in typeset_single (cname=0xb7cd63a8 "local", 
    pname=0xb7cd63d8 "colno", pm=0x0, func=0, on=2097152, off=0, roff=0, 
    value=0x0, altpm=0x0, ops=0xbff46650, joinchar=0)
    at ../../zsh-4.0/Src/builtin.c:2168
#5  0x0805429d in bin_typeset (name=0xb7cd63a8 "local", argv=0xbff465fc, 
    ops=0xbff46650, func=0) at ../../zsh-4.0/Src/builtin.c:2521
#6  0x0804d5b7 in execbuiltin (args=0xb7cd6360, bn=0x8153b7c)
    at ../../zsh-4.0/Src/builtin.c:439
#7  0x080679fa in execcmd (state=0xbff46e70, input=0, output=0, how=2, last1=2)
    at ../../zsh-4.0/Src/exec.c:3067
#8  0x08063975 in execpline2 (state=0xbff46e70, pcode=195, how=2, input=0, 
    output=0, last1=0) at ../../zsh-4.0/Src/exec.c:1561
#9  0x08062d76 in execpline (state=0xbff46e70, slcode=7170, how=2, last1=0)
    at ../../zsh-4.0/Src/exec.c:1347
#10 0x080626d4 in execlist (state=0xbff46e70, dont_change_job=1, exiting=0)
    at ../../zsh-4.0/Src/exec.c:1144
#11 0x0806219f in execode (p=0x837e470, dont_change_job=1, exiting=0)
    at ../../zsh-4.0/Src/exec.c:975
#12 0x0806a6bf in runshfunc (prog=0x837e470, wrap=0x0, 
    name=0xb7cd6268 "prompt_bart_precmd") at ../../zsh-4.0/Src/exec.c:4459
#13 0x0806a410 in doshfunc (shfunc=0x837b9a8, doshargs=0x0, noreturnval=1)
    at ../../zsh-4.0/Src/exec.c:4353
#14 0x080be2c3 in callhookfunc (name=0x814796f "precmd", lnklst=0x0, arrayp=1, 
    retval=0x0) at ../../zsh-4.0/Src/utils.c:1184
#15 0x080be425 in preprompt () at ../../zsh-4.0/Src/utils.c:1237
#16 0x0807b006 in loop (toplevel=1, justonce=0) at ../../zsh-4.0/Src/init.c:120
#17 0x0807e001 in zsh_main (argc=1, argv=0xbff47304)
    at ../../zsh-4.0/Src/init.c:1409
#18 0x0804cbea in main (argc=1, argv=0xbff47304) at ../../zsh-4.0/Src/main.c:93

If I turn off the precmd, it dies as soon as I type anything other than
a newline or a history motion at the prompt.  (Still more after this next
stack trace.)

(gdb) where
#0  0x00462ed0 in _int_malloc () from /lib/tls/libc.so.6
#1  0x00464aa1 in malloc () from /lib/tls/libc.so.6
#2  0x0808eccb in zalloc (size=44) at ../../zsh-4.0/Src/mem.c:583
#3  0x0813654f in mkundoent () at ../../../zsh-4.0/Src/Zle/zle_utils.c:1072
#4  0x0813633a in handleundo () at ../../../zsh-4.0/Src/Zle/zle_utils.c:1029
#5  0x0812078d in zlecore () at ../../../zsh-4.0/Src/Zle/zle_main.c:1061
#6  0x08120cb7 in zleread (lp=0x816c9b0, rp=0x0, flags=3, context=0)
    at ../../../zsh-4.0/Src/Zle/zle_main.c:1213
#7  0x08122a48 in zle_main_entry (cmd=1, ap=0xbfe63154 "`1æ¿")
    at ../../../zsh-4.0/Src/Zle/zle_main.c:1866
#8  0x0807dcd0 in zleentry (cmd=1) at ../../zsh-4.0/Src/init.c:1258
#9  0x0807e503 in inputline () at ../../zsh-4.0/Src/input.c:278
#10 0x0807e3b8 in ingetc () at ../../zsh-4.0/Src/input.c:214
#11 0x0807446e in ihgetc () at ../../zsh-4.0/Src/hist.c:263
#12 0x08085c2d in gettok () at ../../zsh-4.0/Src/lex.c:677
#13 0x0808550d in zshlex () at ../../zsh-4.0/Src/lex.c:364
#14 0x080a0140 in parse_event () at ../../zsh-4.0/Src/parse.c:451
#15 0x0807b049 in loop (toplevel=1, justonce=0) at ../../zsh-4.0/Src/init.c:131
#16 0x0807e001 in zsh_main (argc=1, argv=0xbfe63334)
    at ../../zsh-4.0/Src/init.c:1409
#17 0x0804cbea in main (argc=1, argv=0xbfe63334) at ../../zsh-4.0/Src/main.c:93

So I tried something slightly different:

% zmodload -u $(zmodload)
zmodload: module zsh/complete is in use by another module and cannot be
unloaded
zmodload: module zsh/zle is in use by another module and cannot be unloaded
% zmodload -u $(zmodload); zmodload
% zmodload                         
zsh/compctl
zsh/complete
zsh/zle
%  

Note that after the second zmodload -u, all modules really are unloaded,
but as soon as the prompt is printed the zle-based modules all have now
been reloaded again.  It's at this point that it segfaults.  So really
it appears to be a problem with booting up again after unloading, not
with unloading in the first place.

-- 

Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.11)

[Peter Stephenson]

On Fri, 08 May 2009 07:20:04 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On May 8,  9:34am, Peter Stephenson wrote:
> } Subject: Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.1
> }
> } On Thu, 07 May 2009 16:30:27 -0700
> } Bart Schaefer <schaefer@brasslantern.com> wrote:
> } > schaefer<501> zmodload -u $(zmodload)
> } > zmodload: module zsh/complete is in use by another module and cannot be unloaded
> } > zmodload: module zsh/zle is in use by another module and cannot be unloaded
> } > schaefer<502> zmodload -u $(zmodload)
> } > Program received signal SIGSEGV, Segmentation fault.
> } > 0x081284c7 in resetvideo () at ../../../zsh-4.0/Src/Zle/zle_refresh.c:739
> } > 739             if (nbuf[ln]) {
> } 
> } Does this help?
> 
> It moves the segfault somewhere else, at least; now it craps out when
> referencing the value of $COLUMNS in my precmd.  (Read more below the
> stack trace.)
> 
> (gdb) where
> #0  0x00462ed0 in _int_malloc () from /lib/tls/libc.so.6
> #1  0x00464aa1 in malloc () from /lib/tls/libc.so.6
> #2  0x0808ee3d in zshcalloc (size=48) at ../../zsh-4.0/Src/mem.c:601
> #3  0x080973fb in createparam (name=0xb7cd63d8 "colno", flags=2097152)
>     at ../../zsh-4.0/Src/params.c:841
>...
> (gdb) where
> #0  0x00462ed0 in _int_malloc () from /lib/tls/libc.so.6
> #1  0x00464aa1 in malloc () from /lib/tls/libc.so.6
> #2  0x0808eccb in zalloc (size=44) at ../../zsh-4.0/Src/mem.c:583
> #3  0x0813654f in mkundoent () at ../../../zsh-4.0/Src/Zle/zle_utils.c:1072

Looks to me like some basic memory corruption:  is it worth running
valgrind?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.11)

[Peter Stephenson]

On Fri, 08 May 2009 08:41:23 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> [off-list]
> 
> On May 8,  3:29pm, Peter Stephenson wrote:
> } Subject: Re: Modules on HP-UX (Re: D07multibyte.ztst failure on HP-UX 11.1
> }
> } Looks to me like some basic memory corruption:  is it worth running
> } valgrind?
> 
> (skipping stuff up to the point of actual error)
>...
> ==17685==    at 0x8127AA0: zle_set_highlight (zle_refresh.c:354)

There are lots of these, but all seem to boil down to the fact that
region_highlights and n_region_highlights are not set to zero when the
module is unloaded and the same values are used when the module is loaded
again.

It seemed like this was worth a note in the development guide.  The main
problem is no one reads that.

Index: Etc/zsh-development-guide
===================================================================
RCS file: /cvsroot/zsh/zsh/Etc/zsh-development-guide,v
retrieving revision 1.21
diff -u -r1.21 zsh-development-guide
--- Etc/zsh-development-guide	22 Jan 2009 12:12:14 -0000	1.21
+++ Etc/zsh-development-guide	11 May 2009 08:51:11 -0000
@@ -279,6 +279,10 @@
 not rely on the module successfully being set up.
 The state from `finish_' module is currently ignored; it is called
 too late to prevent the module from being unloaded.
+*Note* in addition to freeing memory, variables associated with allocated
+memory should be set to NULL or to indicate arrays are empty, etc.  It
+should not be assumed that the variables will automatically be zeroed if
+the module is reloaded (though some configurations may do this).
 
 In short, the `cleanup_'-function should undo what the `boot_'-function did
 (together with handling any residual effects of `enables_'), but should
Index: Src/Zle/zle_refresh.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_refresh.c,v
retrieving revision 1.75
diff -u -r1.75 zle_refresh.c
--- Src/Zle/zle_refresh.c	8 May 2009 08:37:31 -0000	1.75
+++ Src/Zle/zle_refresh.c	11 May 2009 08:51:11 -0000
@@ -2644,6 +2644,10 @@
     freevideo();
 
     if (region_highlights)
+    {
 	zfree(region_highlights,
 	      sizeof(struct region_highlight) * n_region_highlights);
+	region_highlights = NULL;
+	n_region_highlights = 0;
+    }
 }


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Modules on HP-UX

[Paul Ackersviller]

On Thu, May 07, 2009 at 04:30:27PM -0700, Bart Schaefer wrote:
> On May 7, 10:08pm, Paul Ackersviller wrote:
> } Subject: Re: D07multibyte.ztst failure on HP-UX 11.11
> }
> } BTW, there's one other thing with HP-UX tests I've been ignoring until
> } now, but would like to look at shortly.  They end with:
> } 	rm: cannot remove `Modules/zsh/parameter.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/zutil.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/complete.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/complist.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/computil.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/zle.so': Text file busy
> 
> The test script creates a shadow directory with copies of all the
> shared objects so that it has a stable module_path for zmodload.
> HP-UX does paging for shared objects directly out of their files,
> so those files can't be removed while loaded by an application.
> 
> It might work to "zmodload -u" the modules before "rm"ing the files.

I tried this on two of the lesser-used modules, namely complist and
zutil, but it made no difference.  It's possible I put the "zmodload
-u" too early, but I can't see a better place than the end of the tests
that do the loading.  I see no problems running zmodload interactively.

P.S.  A tiny documentation nit -- zmodload says modules are usually call
.sl on HPUX, which is true, however the HP build names them .so's anyway.

Re: Modules on HP-UX, with small PATCH

[Paul Ackersviller]

On Thu, May 07, 2009 at 04:30:27PM -0700, Bart Schaefer wrote:
> On May 7, 10:08pm, Paul Ackersviller wrote:
> } Subject: Re: D07multibyte.ztst failure on HP-UX 11.11
> }
> } BTW, there's one other thing with HP-UX tests I've been ignoring until
> } now, but would like to look at shortly.  They end with:
> } 	rm: cannot remove `Modules/zsh/parameter.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/zutil.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/complete.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/complist.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/computil.so': Text file busy
> } 	rm: cannot remove `Modules/zsh/zle.so': Text file busy
> 
> The test script creates a shadow directory with copies of all the
> shared objects so that it has a stable module_path for zmodload.
> HP-UX does paging for shared objects directly out of their files,
> so those files can't be removed while loaded by an application.

Peter's subsequent change didn't get rid of this, but I noticed I'm
able to remove the files as soon as the makefile finishes.  I suspect
timing, and the following less-than-optimal tweak seems to fix it.

Index: Test/Makefile.in
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/Makefile.in,v
retrieving revision 1.12
diff -u -r1.12 Makefile.in
--- Test/Makefile.in	17 Oct 2008 08:31:22 -0000	1.12
+++ Test/Makefile.in	12 May 2009 20:20:31 -0000
@@ -54,6 +54,7 @@
 	else \
 	 stat=1; \
 	fi; \
+	sleep 1; \
 	rm -rf Modules .zcompdump; \
 	exit $$stat

Re: D07multibyte.ztst failure on HP-UX 11.11

[Peter Stephenson]

On Thu, 7 May 2009 22:08:19 +0000
Paul Ackersviller <pda@sdf.lonestar.org> wrote:
> > This means your test still fails, however.  I'm not sure what to do about
> > this since clearly the test *does* fail; you don't get a valid character or
> > an error.  Since character set conversion isn't working on your system (as
> > evinced by the result of not setting the C locale), I suppose we could test
> > that and if so skip the test?
> 
> Sounds like about the best that can be done under the circumstances.

OK, let's do the following.  It prints a message but avoids looking like a
shell bug if character set conversion doesn't work in a simple case.

Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.32
diff -u -r1.32 D07multibyte.ztst
--- Test/D07multibyte.ztst	7 May 2009 16:04:42 -0000	1.32
+++ Test/D07multibyte.ztst	8 May 2009 14:19:38 -0000
@@ -378,17 +378,25 @@
 >X$'\300'Y$'\a'Z$'\177'T
 
 # This also isn't strictly multibyte and is here to reduce the
-# likelihood of a "can't do character set conversion" error.
-  testfn() { (LC_ALL=C; print $'\u00e9') }
-  repeat 4 testfn 2>&1 | while read line; do
-    if [[ $line = *"character not in range"* ]]; then
-      print OK
-    elif [[ $line = "?" ]]; then
-      print OK
-    else
-      print Failed: no error message and no question mark
-    fi
-  done
+# likelihood of a "cannot do character set conversion" error.
+  (print $'\u00e9') 2>&1 | read
+  if [[ $REPLY != é ]]; then
+    print "warning: your system can't do simple Unicode conversion." >&$ZTST_fd
+    print "Check you have a correctly installed iconv library." >&$ZTST_fd
+    # cheat
+    repeat 4 print OK
+  else
+    testfn() { (LC_ALL=C; print $'\u00e9') }
+    repeat 4 testfn 2>&1 | while read line; do
+      if [[ $line = *"character not in range"* ]]; then
+        print OK
+      elif [[ $line = "?" ]]; then
+        print OK
+      else
+        print Failed: no error message and no question mark
+      fi
+    done
+  fi
   true
 0:error handling in Unicode quoting
 >OK



-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D07multibyte.ztst failure on HP-UX 11.11

[Phil Pennock]

On 2009-04-30 at 09:41 +0100, Peter Stephenson wrote:
> It seemed to happen again in the reply, not that it's significant this time
> round.  I've quoted what I got, although it's possible something nearer
> this end decided on the "?"s as a fallback for something it didn't understand.

Paul's original mail was charset=unknown-8bit:
  Content-Type: text/plain; charset=unknown-8bit
  Content-Transfer-Encoding: 8bit
  User-Agent: Mutt/1.4.2.3i

In one reply to you, where you used utf-8, his mail client also used
utf-8 and the characters came through fine; but in the reply which you
comment upon, his mail client replies with:
  Content-Type: text/plain; charset=us-ascii

and it's the same version of mutt each time.  So since mutt claimed
us-ascii, I believe that means that the characters it saw fit within
us-ascii; certainly with current mutt 1.5, there's a send_charset
(described below) setting which would mean that the text received back
from the text editor no longer had the UTF-8 characters intact.

Paul, can you look at your text editor and mail client config please?  I
use vim from mutt and in the special vimrc I use for email (disables
modelines, etc) I set:
  set filetype=mail
  set fileencoding=utf-8

and in my .muttrc I set:
  set send_charset="us-ascii:utf-8"
which is the ordered list of charsets to try, until one works; this is
mutt 1.5.x though; I also set "charset" explicitly to utf-8 rather than
relying upon locale, but I forget why -- I also forget whether or not
the effective charset affects conversion for the text editor.

Ah, the joys of trying to do things Right.
-Phil