zsh 4.3.6 FreeBSD bug

zsh 4.3.6 FreeBSD bug

[Phil Pennock]

Hi,

FreeBSD's conflict in definitions for wchar_t and wint_t if ncurses.h is
included with _XOPEN_SOURCE_EXTENDED defined (conflict with stddef.h and
wchar.h) means that zsh 4.3.6 fails to build on FreeBSD 6.2/amd64.

There's already logic in configure/configure.ac to handle OpenBSD
specially.  If I just add *freebsd* to the OS check, zsh builds fine for
me.  I don't know enough to know why this wasn't needed for you, unless
it only affects 64-bit.

I just hacked configure directly to avoid an autoconf rebuild.

----------------------------8< cut here >8------------------------------
--- configure.old	Sat May  3 07:27:20 2008
+++ configure	Sat May  3 07:27:53 2008
@@ -6880,7 +6880,7 @@
   echo $ECHO_N "(cached) $ECHO_C" >&6
 else
   case "$host_os" in
-  *openbsd*)
+  *openbsd*|*freebsd*)
   zsh_cv_curses_needs_xopen=no
   ;;
   *)
----------------------------8< cut here >8------------------------------

This _should_ do it for a rebuild case (untested).
----------------------------8< cut here >8------------------------------
--- configure.ac.old	Sat May  3 07:34:09 2008
+++ configure.ac	Sat May  3 07:34:24 2008
@@ -674,7 +674,7 @@
 AC_CACHE_CHECK(if the curses library needs _XOPEN_SOURCE_EXTENDED,
 zsh_cv_curses_needs_xopen,
 [case "$host_os" in
-  *openbsd*)
+  *openbsd*|*freebsd*)
   zsh_cv_curses_needs_xopen=no
   ;;
   *)
----------------------------8< cut here >8------------------------------

Regards,
-Phil

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> writes:
> FreeBSD's conflict in definitions for wchar_t and wint_t if ncurses.h is
> included with _XOPEN_SOURCE_EXTENDED defined (conflict with stddef.h and
> wchar.h) means that zsh 4.3.6 fails to build on FreeBSD 6.2/amd64.
>
> There's already logic in configure/configure.ac to handle OpenBSD
> specially.  If I just add *freebsd* to the OS check, zsh builds fine for
> me.  I don't know enough to know why this wasn't needed for you, unless
> it only affects 64-bit.

I run CURRENT on amd64; it has a different version of ncurses.

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Dag-Erling Smørgrav <des@des.no> writes:
> Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> writes:
> > FreeBSD's conflict in definitions for wchar_t and wint_t if ncurses.h is
> > included with _XOPEN_SOURCE_EXTENDED defined (conflict with stddef.h and
> > wchar.h) means that zsh 4.3.6 fails to build on FreeBSD 6.2/amd64.
> >
> > There's already logic in configure/configure.ac to handle OpenBSD
> > specially.  If I just add *freebsd* to the OS check, zsh builds fine for
> > me.  I don't know enough to know why this wasn't needed for you, unless
> > it only affects 64-bit.
> I run CURRENT on amd64; it has a different version of ncurses.

Your patch doesn't work, btw; it breaks CURRENT (and presumably
RELENG_[78])

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Phil Pennock]

On 2008-05-04 at 14:25 +0200, Dag-Erling Smørgrav wrote:
> Your patch doesn't work, btw; it breaks CURRENT (and presumably
> RELENG_[78])

*sigh*  CURRENT now requires the _XOPEN_foo flag to work, where previous
versions didn't and instead broke with it?  Joy.

config.log shows: host_os='freebsd62'

Peter, it needs to be *freebsd6*; I don't believe FreeBSD 5 is still
supported and I don't have such a test system anyway.

-Phil

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

On Mon, 5 May 2008 15:02:36 -0700
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> On 2008-05-04 at 14:25 +0200, Dag-Erling Smørgrav wrote:
> > Your patch doesn't work, btw; it breaks CURRENT (and presumably
> > RELENG_[78])
> 
> *sigh*  CURRENT now requires the _XOPEN_foo flag to work, where previous
> versions didn't and instead broke with it?  Joy.
> 
> config.log shows: host_os='freebsd62'
> 
> Peter, it needs to be *freebsd6*; I don't believe FreeBSD 5 is still
> supported and I don't have such a test system anyway.

OK, I've changed it.

pws

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Peter Stephenson <pws@csr.com> writes:
> Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> writes:
> > Peter, it needs to be *freebsd6*; I don't believe FreeBSD 5 is still
> > supported and I don't have such a test system anyway.
> OK, I've changed it.

IOW, you've broken 6.3...

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> writes:
> *sigh*  CURRENT now requires the _XOPEN_foo flag to work, where previous
> versions didn't and instead broke with it?  Joy.

CURRENT works out of the box, and so do 7.0 and 6.3.  6.2 and older
don't.  The difference between versions that work and versions that
don't is that those that work have ncurses 5.6 (with wide character
support) while those that don't have ncurses 5.2.

For now, I've fixed the port by conditionally applying the attached
patch when building on a FreeBSD version that still has the old ncurses
library.

> config.log shows: host_os='freebsd62'
>
> Peter, it needs to be *freebsd6*; I don't believe FreeBSD 5 is still
> supported and I don't have such a test system anyway.

Yes, 5.5 is still supported.

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

Dag-Erling Smørgrav <des@des.no> writes:
> For now, I've fixed the port by conditionally applying the attached
> patch when building on a FreeBSD version that still has the old ncurses
> library.

Once more, with patch.

DES
-- 
Dag-Erling Smørgrav - des@des.no


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: extra-patch-ncurses --]
[-- Type: text/x-patch, Size: 1786 bytes --]

--- configure.ac.orig	2008-03-28 12:49:32.000000000 +0100
+++ configure.ac	2008-05-05 09:47:30.860369561 +0200
@@ -669,20 +669,20 @@
    termcap_curses_order="$ncursesw_test tinfo termcap $ncurses_test curses" ;;
 esac])dnl
 
-AH_TEMPLATE([ZSH_CURSES_NEEDS_XOPEN],
-[Define if the curses libraries need _XOPEN_SOURCE_EXTENDED defined])
-AC_CACHE_CHECK(if the curses library needs _XOPEN_SOURCE_EXTENDED,
-zsh_cv_curses_needs_xopen,
+AH_TEMPLATE([ZSH_NO_XOPEN],
+[Define if _XOPEN_SOURCE_EXTENDED should not be defined to avoid clashes])
+AC_CACHE_CHECK(if _XOPEN_SOURCE_EXTENDED should not be defined,
+zsh_cv_no_xopen,
 [case "$host_os" in
-  *openbsd*)
-  zsh_cv_curses_needs_xopen=no
+  *openbsd*|*freebsd*)
+  zsh_cv_no_xopen=yes
   ;;
   *)
-  zsh_cv_curses_needs_xopen=yes
+  zsh_cv_no_xopen=no
   ;;
 esac])
-if test x$zsh_cv_curses_needs_xopen = xyes; then
-  AC_DEFINE(ZSH_CURSES_NEEDS_XOPEN)
+if test x$zsh_cv_no_xopen = xyes; then
+  AC_DEFINE(ZSH_NO_XOPEN)
 fi
 
 dnl Check for tigetflag (terminfo) before tgetent (termcap).
--- Src/system.h.orig	2007-12-14 13:43:33.000000000 +0100
+++ Src/system.h	2008-05-05 09:50:17.333164563 +0200
@@ -52,9 +52,20 @@
 # undef HAVE_SYS_UTSNAME_H
 #endif
 
-#if defined(ZSH_CURSES_SOURCE) && defined(ZSH_CURSES_NEEDS_XOPEN)
-#define _XOPEN_SOURCE_EXTENDED 1
-#endif
+#ifndef ZSH_NO_XOPEN
+# ifdef ZSH_CURSES_SOURCE
+#  define _XOPEN_SOURCE_EXTENDED 1
+# else
+#  ifdef MULTIBYTE_SUPPORT
+/*
+ * Needed for wcwidth() which is part of XSI.
+ * Various other uses of the interface mean we can't get away with just
+ * _XOPEN_SOURCE.
+ */
+#   define _XOPEN_SOURCE_EXTENDED 1
+#  endif /* MULTIBYTE_SUPPORT */
+# endif /* ZSH_CURSES_SOURCE */
+#endif /* ZSH_NO_XOPEN */
 
 /*
  * Solaris by default zeroes all elements of the tm structure in

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

On Tue, 06 May 2008 11:13:52 +0200
Dag-Erling Smørgrav <des@des.no> wrote:
> Dag-Erling Smørgrav <des@des.no> writes:
> > For now, I've fixed the port by conditionally applying the attached
> > patch when building on a FreeBSD version that still has the old ncurses
> > library.
> 
> Once more, with patch.

This looks like my original patch; presumably there's more stuff for
handling ZSH_NO_XOPEN somewhere?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Peter Stephenson <pws@csr.com> writes:
> This looks like my original patch

not quite, your patch did not apply to 4.3.6.

> presumably there's more stuff for handling ZSH_NO_XOPEN somewhere?

I'm not sure what you mean.  It is obvious from the patch that
ZSH_NO_XOPEN controls whether _XOPEN_SOURCE_EXTENDED is defined.

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote:
> Peter Stephenson <pws@csr.com> writes:
> > This looks like my original patch
> 
> not quite, your patch did not apply to 4.3.6.
> 
> > presumably there's more stuff for handling ZSH_NO_XOPEN somewhere?
> 
> I'm not sure what you mean.  It is obvious from the patch that
> ZSH_NO_XOPEN controls whether _XOPEN_SOURCE_EXTENDED is defined.

I mean you're saying it's different for versions up to 6.2 than
for versions after and this needs some handling.  Presumably this
can be added to the host_os test, but I don't know what the appropriate
test patterns are.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Dag-Erling Smørgrav <des@des.no> writes:
> Peter Stephenson <pws@csr.com> writes:
> > This looks like my original patch
> not quite, your patch did not apply to 4.3.6.
> > presumably there's more stuff for handling ZSH_NO_XOPEN somewhere?
> I'm not sure what you mean.  It is obvious from the patch that
> ZSH_NO_XOPEN controls whether _XOPEN_SOURCE_EXTENDED is defined.

I should add that the patched glob in configure.ac is incorrect, but it
works because of the way the port selectively applies the patch.  The
correct glob would be something like:

  *openbsd*|*freebsd5*|*freebsd6[012]*)

FreeBSD 4 and older are no longer supported, so I won't bother trying to
make them work properly.

It didn't occur to me earlier that with this change, the patch could be
applied unconditionally.

DES
-- 
Dag-Erling Smørgrav - des@des.no

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote:
> Dag-Erling Sm=C3=B8rgrav <des@des.no> writes:
> > Peter Stephenson <pws@csr.com> writes:
> > > This looks like my original patch
> > not quite, your patch did not apply to 4.3.6.
> > > presumably there's more stuff for handling ZSH_NO_XOPEN somewhere?
> > I'm not sure what you mean.  It is obvious from the patch that
> > ZSH_NO_XOPEN controls whether _XOPEN_SOURCE_EXTENDED is defined.
> 
> I should add that the patched glob in configure.ac is incorrect, but it
> works because of the way the port selectively applies the patch.  The
> correct glob would be something like:
> 
>   *openbsd*|*freebsd5*|*freebsd6[012]*)

Thanks, that's what I was after... I've committed that.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

Peter Stephenson <pws@csr.com> writes:
> Dag-Erling Smørgrav <des@des.no> writes:
> > I should add that the patched glob in configure.ac is incorrect, but it
> > works because of the way the port selectively applies the patch.  The
> > correct glob would be something like:
> > 
> >   *openbsd*|*freebsd5*|*freebsd6[012]*)
>
> Thanks, that's what I was after... I've committed that.

two problems I just discovered:

1) there should be a . after the 6.

2) m4 will gobble the square brackets, so the entire case statement
   needs to be quoted.

New patch (relative to 4.3.6) attached.  I've tested it on 6.2, 6.3, 7.0
and CURRENT.

DES
-- 
Dag-Erling Smørgrav - des@des.no


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch-ncurses --]
[-- Type: text/x-patch, Size: 1835 bytes --]

--- configure.ac.orig	2008-03-28 12:49:32.000000000 +0100
+++ configure.ac	2008-05-05 09:47:30.860369561 +0200
@@ -669,20 +669,20 @@
    termcap_curses_order="$ncursesw_test tinfo termcap $ncurses_test curses" ;;
 esac])dnl
 
-AH_TEMPLATE([ZSH_CURSES_NEEDS_XOPEN],
-[Define if the curses libraries need _XOPEN_SOURCE_EXTENDED defined])
-AC_CACHE_CHECK(if the curses library needs _XOPEN_SOURCE_EXTENDED,
-zsh_cv_curses_needs_xopen,
+AH_TEMPLATE([ZSH_NO_XOPEN],
+[Define if _XOPEN_SOURCE_EXTENDED should not be defined to avoid clashes])
+AC_CACHE_CHECK(if _XOPEN_SOURCE_EXTENDED should not be defined,
+zsh_cv_no_xopen,
-[case "$host_os" in
-  *openbsd*)
-  zsh_cv_curses_needs_xopen=no
+[[case "$host_os" in
+  *openbsd*|*freebsd5*|*freebsd6.[012]*)
+  zsh_cv_no_xopen=yes
   ;;
   *)
-  zsh_cv_curses_needs_xopen=yes
+  zsh_cv_no_xopen=no
   ;;
-esac])
+esac]])
-if test x$zsh_cv_curses_needs_xopen = xyes; then
-  AC_DEFINE(ZSH_CURSES_NEEDS_XOPEN)
+if test x$zsh_cv_no_xopen = xyes; then
+  AC_DEFINE(ZSH_NO_XOPEN)
 fi
 
 dnl Check for tigetflag (terminfo) before tgetent (termcap).
--- Src/system.h.orig	2007-12-14 13:43:33.000000000 +0100
+++ Src/system.h	2008-05-05 09:50:17.333164563 +0200
@@ -52,9 +52,20 @@
 # undef HAVE_SYS_UTSNAME_H
 #endif
 
-#if defined(ZSH_CURSES_SOURCE) && defined(ZSH_CURSES_NEEDS_XOPEN)
-#define _XOPEN_SOURCE_EXTENDED 1
-#endif
+#ifndef ZSH_NO_XOPEN
+# ifdef ZSH_CURSES_SOURCE
+#  define _XOPEN_SOURCE_EXTENDED 1
+# else
+#  ifdef MULTIBYTE_SUPPORT
+/*
+ * Needed for wcwidth() which is part of XSI.
+ * Various other uses of the interface mean we can't get away with just
+ * _XOPEN_SOURCE.
+ */
+#   define _XOPEN_SOURCE_EXTENDED 1
+#  endif /* MULTIBYTE_SUPPORT */
+# endif /* ZSH_CURSES_SOURCE */
+#endif /* ZSH_NO_XOPEN */
 
 /*
  * Solaris by default zeroes all elements of the tm structure in

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

On Sat, 3 May 2008 00:39:47 -0700
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> FreeBSD's conflict in definitions for wchar_t and wint_t if ncurses.h is
> included with _XOPEN_SOURCE_EXTENDED defined (conflict with stddef.h and
> wchar.h) means that zsh 4.3.6 fails to build on FreeBSD 6.2/amd64.

I've now defined _XOPEN_SOURCE_EXTENDED to get wcwidth() even if curses
doesn't need it, because that's an XSI function; presumably that doesn't
want to happen on BSD either (it'll appear in the scope of the curses
files)?  If so we'll need to revise the scope of the definition you
changed to mean never define _XOPEN_SOURCE_EXTENDED.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

On Sun, 4 May 2008 13:19:13 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> I've now defined _XOPEN_SOURCE_EXTENDED to get wcwidth() even if curses
> doesn't need it, because that's an XSI function; presumably that doesn't
> want to happen on BSD either (it'll appear in the scope of the curses
> files)?  If so we'll need to revise the scope of the definition you
> changed to mean never define _XOPEN_SOURCE_EXTENDED.

I think we probably want something like this.

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.99
diff -u -r1.99 configure.ac
--- configure.ac	26 Apr 2008 17:46:46 -0000	1.99
+++ configure.ac	4 May 2008 18:37:06 -0000
@@ -682,20 +682,20 @@
    termcap_curses_order="$ncursesw_test tinfo termcap $ncurses_test curses" ;;
 esac])dnl
 
-AH_TEMPLATE([ZSH_CURSES_NEEDS_XOPEN],
-[Define if the curses libraries need _XOPEN_SOURCE_EXTENDED defined])
-AC_CACHE_CHECK(if the curses library needs _XOPEN_SOURCE_EXTENDED,
-zsh_cv_curses_needs_xopen,
+AH_TEMPLATE([ZSH_NO_XOPEN],
+[Define if _XOPEN_SOURCE_EXTENDED should not be defined to avoid clashes])
+AC_CACHE_CHECK(if _XOPEN_SOURCE_EXTENDED should not be defined,
+zsh_cv_no_xopen,
 [case "$host_os" in
-  *openbsd*)
-  zsh_cv_curses_needs_xopen=no
+  *openbsd*|*freebsd*)
+  zsh_cv_no_xopen=yes
   ;;
   *)
-  zsh_cv_curses_needs_xopen=yes
+  zsh_cv_no_xopen=no
   ;;
 esac])
-if test x$zsh_cv_curses_needs_xopen = xyes; then
-  AC_DEFINE(ZSH_CURSES_NEEDS_XOPEN)
+if test x$zsh_cv_no_xopen = xyes; then
+  AC_DEFINE(ZSH_NO_XOPEN)
 fi
 
 dnl Check for tigetflag (terminfo) before tgetent (termcap).
Index: Src/system.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/system.h,v
retrieving revision 1.51
diff -u -r1.51 system.h
--- Src/system.h	23 Apr 2008 10:35:29 -0000	1.51
+++ Src/system.h	4 May 2008 18:37:10 -0000
@@ -52,19 +52,20 @@
 # undef HAVE_SYS_UTSNAME_H
 #endif
 
-#if defined(ZSH_CURSES_SOURCE) && defined(ZSH_CURSES_NEEDS_XOPEN)
-# define _XOPEN_SOURCE_EXTENDED 1
-#else
-# ifdef MULTIBYTE_SUPPORT
+#ifndef ZSH_NO_XOPEN
+# ifdef ZSH_CURSES_SOURCE
+#  define _XOPEN_SOURCE_EXTENDED 1
+# else
+#  ifdef MULTIBYTE_SUPPORT
 /*
  * Needed for wcwidth() which is part of XSI.
  * Various other uses of the interface mean we can't get away with just
  * _XOPEN_SOURCE.
  */
-/*#  define _XOPEN_SOURCE 1*/
-#  define _XOPEN_SOURCE_EXTENDED 1
-# endif
-#endif
+#   define _XOPEN_SOURCE_EXTENDED 1
+#  endif /* MULTIBYTE_SUPPORT */
+# endif /* ZSH_CURSES_SOURCE */
+#endif /* ZSH_NO_XOPEN */
 
 /*
  * Solaris by default zeroes all elements of the tm structure in

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh 4.3.6 FreeBSD bug

[Phil Pennock]

On 2008-05-04 at 19:37 +0100, Peter Stephenson wrote:
> On Sun, 4 May 2008 13:19:13 +0100
> Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> > I've now defined _XOPEN_SOURCE_EXTENDED to get wcwidth() even if curses
> > doesn't need it, because that's an XSI function; presumably that doesn't
> > want to happen on BSD either (it'll appear in the scope of the curses
> > files)?  If so we'll need to revise the scope of the definition you
> > changed to mean never define _XOPEN_SOURCE_EXTENDED.
> 
> I think we probably want something like this.

cvs HEAD (with your fixes) builds and runs on FreeBSD 6.2/amd64.

There are display glitches though; these occur both with 4.3.6 with
directly hacked configure and with cvs HEAD.  I don't know if these do
not occur without the changes needed for amd64 when
_XOPEN_SOURCE_EXTENDED is defined.  It seems somewhat unusual that
simply defining that behaviour flag results in mutually incompatible
standard system headers.  *sigh*

This is with LC_CTYPE=en_US.UTF-8 and connecting remotely (iTerm on
MacOS 10.4.x).

Can anyone confirm if these problems, outlined below, are widely seen or
a platform issue?

Note the differences between EURO SIGN first and POUND SIGN first,
perhaps something to do with character width:
€: EURO SIGN    [0x20ac]
£: POUND SIGN   [0xa3]

Minimum steps to reproduce:

  ./dbg/bin/zsh -f
  autoload ${^fpath}/*(N-.:t)
  zle -N insert-composed-char
  bindkey '^Xk' insert-composed-char

  Type: <Ctrl-X>k Pd <Ctrl-X>k Eu
  See: £###
  Press Enter and see:
  zsh: command not found: £€
                             %

  Add: <Ctrl-L>
  See: £€ [K
  where the blank character is ^[.

  Type: <Ctrl-X>k Eu <Ctrl-X>k Pd
  See: €£
  erase, etc, work fine

  With deliberately invalid combination l- (X11 combination for £):
  Type: <Ctrl-X>k Eu <Ctrl-X>l Pd <Ctrl-X>k l-
  See: €£ [K
    Cut&Paste into od(1) confirms that the blank character is ^[, so
    that when I paste the output into vim, it self-corrects to €£

  With cursor left alone from previous output, press backspace (sends
  ^?) and see: €£ €
  The blank space is again/still an ESC.
  Type <Ctrl-L> to redraw, see just a €.

The display glitch seems to require both a wide character and a
single-byte character, in UTF-8, in that order.  Any wide-character
followed by a single-byte character (per UTF-8 encoding) is sufficient.

Is this the problem which wcwidth() should fix?

I tried removing *freebsd* from the configure check, rebuild to confirm
it failed, make clean, hacked Src/zshcurses.h to read:
  #define __wchar_t wchar_t
  #define __wint_t wint_t
  #include <curses.h>
and this did NOT resolve the problem, despite leaving ZSH_NO_XOPEN
undefined and _XOPEN_SOURCE_EXTENDED defined.

Before I look any further, can someone with ready access to another
UTF-8 capable platform confirm what they see with the above combining
chars?

Thanks,
-Phil

Re: zsh 4.3.6 FreeBSD bug

[Phil Pennock]

[ -cc des ]

On 2008-05-04 at 17:38 -0700, Phil Pennock wrote:
> cvs HEAD (with your fixes) builds and runs on FreeBSD 6.2/amd64.

NB: the HEAD included the 24915 fixes for combining characters and the
problems were still seen.

-Phil

Re: zsh 4.3.6 FreeBSD bug

[Peter Stephenson]

On Sun, 4 May 2008 17:38:58 -0700
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> Can anyone confirm if these problems, outlined below, are widely seen or
> a platform issue?
>
> Note the differences between EURO SIGN first and POUND SIGN first,
> perhaps something to do with character width:
> €: EURO SIGN    [0x20ac]
> £: POUND SIGN   [0xa3]

I haven't seen any problems with either of these two characters on
any of the Fedora versions I've been using.

> Is this the problem which wcwidth() should fix?

No, it looks like something different.  wcwidth() tells the shell the
printing width; it's nothing to do with the number of bytes that make up
the character.  The symptom of wcwidth() failing is that the character
prints OK but the shell doesn't count the characters it's moving over or
deleting properly.

What does the following give for you (with pound sterling followed by Euro)?

% echo £€ | xxd
0000000: c2a3 e282 ac0a                           ......

0xc2 0xa3 is the pound sign in UTF-8 and 0xe2 0x82 0xac is the Euro
symbol.  If it's the same as what I get, then the conversion to UTF-8 is
correct.  One possibility for what's going wrong is that 0xe2 0x82 0xac
is being reported as an invalid character by mbtowc() and its
relatives.  The shell's line editor would pick this up and show the
individual bytes; a dumb display would simply dump out the bytes and if
the terminal is working they would show up OK.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: zsh 4.3.6 FreeBSD bug

[Jun T.]

On 2008/05/05, at 09:38, Phil Pennock wrote:
> This is with LC_CTYPE=en_US.UTF-8 and connecting remotely (iTerm on
> MacOS 10.4.x).

It seems iTerm has some problems in displaying multibyte characters.

I can't access any FreeBSD machine so I tested on MacOSX 10.5 locally
with iTerm (i.e., zsh running on MacOSX), and found the same problem.
Terminal.app on 10.5 is much better than iTerm in multibyte support
and can display £€ without any problem.

I can't test on MacOSX 10.4 now, and don't know Terminal.app on 10.4
is better than iTerm or not.
Could you please try Terminal.app on 10.4?

Re: zsh 4.3.6 FreeBSD bug

[Phil Pennock]

On 2008-05-06 at 05:02 +0900, Jun T. wrote:
> On 2008/05/05, at 09:38, Phil Pennock wrote:
>> This is with LC_CTYPE=en_US.UTF-8 and connecting remotely (iTerm on
>> MacOS 10.4.x).
> 
> It seems iTerm has some problems in displaying multibyte characters.
> 
> I can't access any FreeBSD machine so I tested on MacOSX 10.5 locally
> with iTerm (i.e., zsh running on MacOSX), and found the same problem.
> Terminal.app on 10.5 is much better than iTerm in multibyte support
> and can display £€ without any problem.
> 
> I can't test on MacOSX 10.4 now, and don't know Terminal.app on 10.4
> is better than iTerm or not.
> Could you please try Terminal.app on 10.4?

Confirmed, no problems with Terminal.app; thanks.

Okay, not a bug in zsh, so it's _just_ the problems in getting the code
to compile in the first place and which versions do or don't need
special-casing.  *sigh*

Regards,
-Phil

Re: zsh 4.3.6 FreeBSD bug

[Dag-Erling Smørgrav]

Peter Stephenson <p.w.stephenson@ntlworld.com> writes:
> I think we probably want something like this.

This patch breaks on FreeBSD-CURRENT, just like Phil's.

DES
-- 
Dag-Erling Smørgrav - des@des.no