Re: zsh 4.3.6 FreeBSD bug

[Phil Pennock <zsh-workers+phil.pennock@spodhuis.org>]

On 2008-05-04 at 19:37 +0100, Peter Stephenson wrote:
> On Sun, 4 May 2008 13:19:13 +0100
> Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> > I've now defined _XOPEN_SOURCE_EXTENDED to get wcwidth() even if curses
> > doesn't need it, because that's an XSI function; presumably that doesn't
> > want to happen on BSD either (it'll appear in the scope of the curses
> > files)?  If so we'll need to revise the scope of the definition you
> > changed to mean never define _XOPEN_SOURCE_EXTENDED.
> 
> I think we probably want something like this.

cvs HEAD (with your fixes) builds and runs on FreeBSD 6.2/amd64.

There are display glitches though; these occur both with 4.3.6 with
directly hacked configure and with cvs HEAD.  I don't know if these do
not occur without the changes needed for amd64 when
_XOPEN_SOURCE_EXTENDED is defined.  It seems somewhat unusual that
simply defining that behaviour flag results in mutually incompatible
standard system headers.  *sigh*

This is with LC_CTYPE=en_US.UTF-8 and connecting remotely (iTerm on
MacOS 10.4.x).

Can anyone confirm if these problems, outlined below, are widely seen or
a platform issue?

Note the differences between EURO SIGN first and POUND SIGN first,
perhaps something to do with character width:
€: EURO SIGN    [0x20ac]
£: POUND SIGN   [0xa3]

Minimum steps to reproduce:

  ./dbg/bin/zsh -f
  autoload ${^fpath}/*(N-.:t)
  zle -N insert-composed-char
  bindkey '^Xk' insert-composed-char

  Type: <Ctrl-X>k Pd <Ctrl-X>k Eu
  See: £###
  Press Enter and see:
  zsh: command not found: £€
                             %

  Add: <Ctrl-L>
  See: £€ [K
  where the blank character is ^[.

  Type: <Ctrl-X>k Eu <Ctrl-X>k Pd
  See: €£
  erase, etc, work fine

  With deliberately invalid combination l- (X11 combination for £):
  Type: <Ctrl-X>k Eu <Ctrl-X>l Pd <Ctrl-X>k l-
  See: €£ [K
    Cut&Paste into od(1) confirms that the blank character is ^[, so
    that when I paste the output into vim, it self-corrects to €£

  With cursor left alone from previous output, press backspace (sends
  ^?) and see: €£ €
  The blank space is again/still an ESC.
  Type <Ctrl-L> to redraw, see just a €.

The display glitch seems to require both a wide character and a
single-byte character, in UTF-8, in that order.  Any wide-character
followed by a single-byte character (per UTF-8 encoding) is sufficient.

Is this the problem which wcwidth() should fix?

I tried removing *freebsd* from the configure check, rebuild to confirm
it failed, make clean, hacked Src/zshcurses.h to read:
  #define __wchar_t wchar_t
  #define __wint_t wint_t
  #include <curses.h>
and this did NOT resolve the problem, despite leaving ZSH_NO_XOPEN
undefined and _XOPEN_SOURCE_EXTENDED defined.

Before I look any further, can someone with ready access to another
UTF-8 capable platform confirm what they see with the above combining
chars?

Thanks,
-Phil