Cannot paste unicode <0221>, <0234> - <024f>

Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1734 bytes --]

Hello,
I use additional Unicode blocks from Latin script, like "Latin-1 Supplement" or "Latin Extended-A". The data is processed and displayed correctly, however when I copy, and paste, then a few chars are presented as <CODE>:

ȮȯȰȱȲȳ<0234><0235><0236><0237><0238><0239>...

This should be:

ȮȯȰȱȲȳȴȵȶȷȸȹ

Screenshot attached. Bash works. Also tested on -f zsh-5.3.1-dev-0. Despite wrong display, characters work correctly with the argument-receiving script, however not when copying the pasted text again – "<CODE>" is copied literally, not as single unicode character.

The complete subset of Latin script that I use is below:

0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZµºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáãäæçèéêëìíîïðñòóôõöøúûüýþÿĀāĂ㥹ĆćĈĉĊċČčĎďĐđĒēĔĕĖėĘęĚěĜĝĞğĠġĢģĤĥĦħĨĩĪīĬĭĮįİıIJijĴĵĶķĸĹĺĻļĽľĿŀŁłŃńŅņŇňʼnŊŋŌōŎŏŐőŒœŔŕŖŗŘřŚśŜŝŞşŠšŢţŤťŦŧŨũŪūŬŭŮůŰűŲųŴŵŶŷŸŹźŻżŽžſƀƁƂƃƄƅƆƇƈƉƊƋƌƍƎƏƐƑƒƓƔƕƖƗƘƙƚƛƜƝƞƟƠơƢƣƤƥƦƧƨƩƪƫƬƭƮƯưƱƲƳƴƵƶƷƸƹƺƻƼƽƾƿǀǁǂǃDŽDždžLJLjljNJNjnjǍǎǏǐǑǒǓǔǕǖǗǘǙǚǛǜǝǞǟǠǡǢǣǤǥǦǧǨǩǪǫǬǭǮǯǰDZDzdzǴǵǶǷǸǹǺǻǼǽǾǿȀȁȂȃȄȅȆȇȈȉȊȋȌȍȎȏȐȑȒȓȔȕȖȗȘșȚțȜȝȞȟȠȡȢȣȤȥȦȧȨȩȪȫȬȭȮȯȰȱȲȳȴȵȶȷȸȹȺȻȼȽȾȿɀɁɂɃɄɅɆɇɈɉɊɋɌɍɎɏḀḁḂḃḄḅḆḇḈḉḊḋḌḍḎḏḐḑḒḓḔḕḖḗḘḙḚḛḜḝḞḟḠḡḢḣḤḥḦḧḨḩḪḫḬḭḮḯḰḱḲḳḴ

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

[-- Attachment #2: cannot_paste2.png --]
[-- Type: image/png, Size: 139468 bytes --]

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

On Fri, 28 Apr 2017 12:55:10 +0200
Sebastian Gniazdowski <psprint@zdharma.org> wrote:
> I use additional Unicode blocks from Latin script, like "Latin-1
> Supplement" or "Latin Extended-A". The data is processed and displayed
> correctly, however when I copy, and paste, then a few chars are
> presented as <CODE>:
> 
> ȮȯȰȱȲȳ<0234><0235><0236><0237><0238><0239>...

Looks like they're not being recognised as printable characters on that
system for some reason (it's clearly not a terminal issue).  Presumably
you get the same effect with the insert-unicode-char widget and the four
hex digits (you need to use the widget at the end to terminate the
sequence)?

Another possibility is it's a broken character width test --- we
sanitise that the character is printable by checking WCWIDTH() is
greater than zero.  That usually maps to wcwidth(), but if we detect it
looks broken we'll use an internal replacement mk_wcwidth().

pws

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 28 Apr 2017 at 13:55:22, Peter Stephenson (p.stephenson@samsung.com) wrote:
> On Fri, 28 Apr 2017 12:55:10 +0200
> Sebastian Gniazdowski wrote:
> > ȮȯȰȱȲȳ<0234><0235><0236><0237><0238><0239>...
>  
> Looks like they're not being recognised as printable characters on that
> system for some reason (it's clearly not a terminal issue). Presumably
> you get the same effect with the insert-unicode-char widget and the four
> hex digits (you need to use the widget at the end to terminate the
> sequence)?

Yes, checked a few, 0234,0235,0236,0237. They result in <CODE>

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

Probably worth checking if BROKEN_WCWIDTH is already defined in config.h
and if not what happens if you define it.  It has no effect unless
one of __STDC_ISO_10646__ or __APPLE__ is also defined.

An other option is --enable-unicode9, but you'll need the appropriate
header and library support for that to work.

You might want to confirm also if it actually is wcwidth() rather than
iswprint() by hacking zle_refresh.c e.g. the following (untested).  This
might be a plausible patch, actually --- if iswprint() succeeds,
presumably the printing width actually is positive (combining characters
are handled as a special case afterwards).

pws

diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8391739..835eed7 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -1278,8 +1278,11 @@ zrefresh(void)
 #ifdef __STDC_ISO_10646__
 		 !ZSH_INVALID_WCHAR_TEST(*t) &&
 #endif
-		 iswprint(*t) && (width = WCWIDTH(*t)) > 0) {
+		 iswprint(*t)) {
 	    int ichars;
+	    width = WCWIDTH(*t);
+	    if (width <= 0)
+		width = 1;
 	    if (width > rpms.sen - rpms.s) {
 		int started = 0;
 		/*

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 28 April 2017 at 15:17:40, Peter Stephenson (p.stephenson@samsung.com) wrote:
> Probably worth checking if BROKEN_WCWIDTH is already defined in config.h
> and if not what happens if you define it. It has no effect unless
> one of __STDC_ISO_10646__ or __APPLE__ is also defined.

Tried defining it with the wcwidth hack and without – no change. System is OS X so there is __APPLE__.

> An other option is --enable-unicode9, but you'll need the appropriate
> header and library support for that to work.

I've installed iconv, it got somewhat detected:
checking iconv.h usability... yes
checking iconv.h presence... yes
checking for iconv.h... yes
...
checking for iconv... no
checking for iconv in -liconv... no
checking for libiconv in -liconv... yes
checking for iconv declaration...

-liconv was in configure status, and zsh binary is actually linked with it. However, no luck with pasting "ȮȯȰȱȲȳȴȵȶȷȸȹ", result is the same, also with insert-unicode-char

> You might want to confirm also if it actually is wcwidth() rather than
> iswprint() by hacking zle_refresh.c e.g. the following (untested). This
> might be a plausible patch, actually --- if iswprint() succeeds,
> presumably the printing width actually is positive (combining characters
> are handled as a special case afterwards).

Tried the patch, no luck, result is the same

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

On Fri, 28 Apr 2017 15:54:23 +0200
Sebastian Gniazdowski <psprint@zdharma.org> wrote:
> > You might want to confirm also if it actually is wcwidth() rather than
> > iswprint() by hacking zle_refresh.c e.g. the following (untested). This
> > might be a plausible patch, actually --- if iswprint() succeeds,
> > presumably the printing width actually is positive (combining characters
> > are handled as a special case afterwards).
> 
> Tried the patch, no luck, result is the same

So most likely iswprint() is returning false for those characters for
some reason, although it could be wrong somewhere else entirely.
Doesn't look like it matches the range of metafied characters for
once --- the first character in the affected run is U+0234 which
is UTF-8 c8 b4 20 none of which is metafied.

pws

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

It might be possible to rely *only* on wcwidth() in cases where we're
using a replacement?  Does anyone from the Apple world have any
opinions?

pws

diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8391739..d0384af 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -1278,7 +1278,15 @@ zrefresh(void)
 #ifdef __STDC_ISO_10646__
 		 !ZSH_INVALID_WCHAR_TEST(*t) &&
 #endif
-		 iswprint(*t) && (width = WCWIDTH(*t)) > 0) {
+		 /*
+		  * If we are using a replacement wcwidth(), rely
+		  * on it also for checking printability: width <= 0
+		  * if not printable.
+		  */
+#ifndef BROKEN_WCWIDTH
+		 iswprint(*t) &&
+#endif
+		 (width = WCWIDTH(*t)) > 0) {
 	    int ichars;
 	    if (width > rpms.sen - rpms.s) {
 		int started = 0;

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 28 kwietnia 2017 at 16:42:17, Peter Stephenson (p.stephenson@samsung.com) wrote:
> It might be possible to rely *only* on wcwidth() in cases where we're
> using a replacement? Does anyone from the Apple world have any
> opinions?
>  
> pws
>  
> diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c

I've checked the patch, no luck, but I knew what to track, so I've added:

    fprintf( stderr, "Path 1, char/string: %S\n", t );
and
    fprintf( stderr, "Path 3, char/string: %S, iswprint:%d, wcwidth: %d\n", t, iswprint(*t), WCWIDTH(*t));

for printable / unprintable if-paths, and the result is:

Path 1, char/string: ȯȰȱȲȳȴȵȶȷȸȹ
Path 1, char/string: ȰȱȲȳȴȵȶȷȸȹ
Path 1, char/string: ȱȲȳȴȵȶȷȸȹ
Path 1, char/string: Ȳȳȴȵȶȷȸȹ
Path 1, char/string: ȳȴȵȶȷȸȹ
Path 3, char/string: ȴȵȶȷȸȹ, iswprint:0, wcwidth: -1
Path 3, char/string: ȵȶȷȸȹ, iswprint:0, wcwidth: -1
Path 3, char/string: ȶȷȸȹ, iswprint:0, wcwidth: -1
Path 3, char/string: ȷȸȹ, iswprint:0, wcwidth: -1
Path 3, char/string: ȸȹ, iswprint:0, wcwidth: -1
Path 3, char/string: ȹ, iswprint:0, wcwidth: -1

What's interesting, once or twice the print looked like following (recompile fixed this):

Path 1, char/string: ȯȰȱȲȳȴȵȶȷȸȹiⱁ
                                  ÍA䰂Ӄ࢑࠱䙁怲དྷĝ²ũǝ
Path 1, char/string: ȰȱȲȳȴȵȶȷȸȹiⱁ
                                 ÍA䰂Ӄ࢑࠱䙁怲དྷĝ²ũǝ
Path 1, char/string: ȱȲȳȴȵȶȷȸȹiⱁ
                                ÍA䰂Ӄ࢑࠱䙁怲དྷĝ²ũǝ
...

like if printf() wasn't sure itself should %S print those characters correctly or not. That said, vim or mentioned bash somehow manage this, just as a reference point if stdlib is broken or not(-really).

-- 
Sebastian Gniazdowski

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

On Fri, 28 Apr 2017 17:27:16 +0200
Sebastian Gniazdowski <psprint@zdharma.org> wrote:
> On 28 kwietnia 2017 at 16:42:17, Peter Stephenson (p.stephenson@samsung.com) wrote:
> > It might be possible to rely *only* on wcwidth() in cases where we're
> > using a replacement? Does anyone from the Apple world have any
> > opinions?
> >  
> > pws
> >  
> > diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
> 
> I've checked the patch, no luck, but I knew what to track, so I've added:

Hmmm... with the last patch *and* BROKEN_WCWIDTH defined *and* __APPLE__, I
think we have all-known software (no iswprint() and compiled-in
replacement wcwidth()) which probably ought to work.  So maybe it's not
detecting this particular form of problem?

pws

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 28 kwietnia 2017 at 17:44:33, Peter Stephenson (p.stephenson@samsung.com) wrote:
> On Fri, 28 Apr 2017 17:27:16 +0200
> Sebastian Gniazdowski wrote:
> > On 28 kwietnia 2017 at 16:42:17, Peter Stephenson (p.stephenson@samsung.com) wrote:  
> > > It might be possible to rely *only* on wcwidth() in cases where we're
> > > using a replacement? Does anyone from the Apple world have any
> > > opinions?
> > >
> > > pws
> > >
> > > diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
> >
> > I've checked the patch, no luck, but I knew what to track, so I've added:
>  
> Hmmm... with the last patch *and* BROKEN_WCWIDTH defined *and* __APPLE__, I
> think we have all-known software (no iswprint() and compiled-in
> replacement wcwidth()) which probably ought to work. So maybe it's not
> detecting this particular form of problem?

Didn't define BROKEN_WCWIDTH when testing – after doing this now, the last patch works. Not sure what you meant about the detection of problem. It would be good if someone would test the paste (ȮȯȰȱȲȳȴȵȶȷȸȹ) on OS X Sierra, I have one-step-behind OS X El Captain.

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 28 kwietnia 2017 at 16:42:17, Peter Stephenson (p.stephenson@samsung.com) wrote:
> It might be possible to rely *only* on wcwidth() in cases where we're
> using a replacement? Does anyone from the Apple world have any
> opinions?

This is a situation where having compile-time tests and equipped redundant survival-thing should yield a edge case where problem can be managed and solved despite hard conditions. There is always a difficult for eyes question: Why not always rely on internal wcwidth? Sorry for this question, I'm pro clever redundancy.

A fact: it is easy to detect <0234>-alike problem – test 0234 character in wcwidth configure check, define BROKEN_WCWIDTH. Ideal following solution: add iswprint test of 0234 to configure, use BROKEN_ISWPRINT. It should be easy.

Broken wcwidth might imply broken iswprint, so you might be right about simple approach – to rely only on internal wcwidth when BROKEN_WCWIDTH, i.e. skip iswprint. Are there some opinions on this?

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Jun T.]

Both iswprint() and wcwidth() are broken on macOS for characters
like U+0234. I will prepare a patch, but I'm rather busy today.
Maybe tomorrow...

Jun

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Jun T.]

Here is a quick (maybe too simple) patch.

wcwidth() on MacOSX was broken for combining characters,
but Apple has fixed *this* problem a few years ago,
probably in OSX 10.8 (Mavericks). So BROKEN_WCWIDTH is
NOT defined on recent macOS.

In the patch below, I added a test in configure.ac using U+0234
for both wcwidth() and iswprint() (both are broken on macOS;
wcwidth() returns -1 and iswprint() returns 0=false).

As a replacement for the broken iswprint(), I added a very (or
too) simple function wc_isprint(), which returns false only for
those characters for which mk_wcwidth() returns -1, i.e.,
0 <= wc <= 0x1f and 0x7f <= wc <= 0x9f (8bit control chars).

Another possibility is to use --enable-unicode9 if wcwidth()
and/or iswprint() are broken (--enable-unicode9 works fine
without any additional libraries). There is no iswprint-replacement
in wcwidth.h, but implementing it would be easy if we can use the
array wcwidth9_nonprint in wcwidth9.h.
# But I must say I couldn't understand the array; for example,
# why U+00ad is not printable while U+2028 is printable?


diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8391739..d0dd1ef 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -1278,7 +1278,7 @@ zrefresh(void)
 #ifdef __STDC_ISO_10646__
 		 !ZSH_INVALID_WCHAR_TEST(*t) &&
 #endif
-		 iswprint(*t) && (width = WCWIDTH(*t)) > 0) {
+		 WC_ISPRINT(*t) && (width = WCWIDTH(*t)) > 0) {
 	    int ichars;
 	    if (width > rpms.sen - rpms.s) {
 		int started = 0;
@@ -1460,7 +1460,7 @@ zrefresh(void)
 	u = outputline;
 	for (; u < outputline + outll; u++) {
 #ifdef MULTIBYTE_SUPPORT
-	    if (iswprint(*u)) {
+	    if (WC_ISPRINT(*u)) {
 		int width = WCWIDTH(*u);
 		/* Handle wide characters as above */
 		if (width > rpms.sen - rpms.s) {
@@ -2468,7 +2468,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	if (tmpline[t0] == ZWC('\t'))
 	    vsiz = (vsiz | 7) + 2;
 #ifdef MULTIBYTE_SUPPORT
-	else if (iswprint(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
+	else if (WC_ISPRINT(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
 	    vsiz += width;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
 		while (t0 < tmpll-1 && IS_COMBINING(tmpline[t0+1]))
@@ -2556,7 +2556,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	    vp->atr = all_atr_on | all_atr_off;
 	    vp++;
 #ifdef MULTIBYTE_SUPPORT
-	} else if (iswprint(tmpline[t0]) &&
+	} else if (WC_ISPRINT(tmpline[t0]) &&
 		   (width = WCWIDTH(tmpline[t0])) > 0) {
 	    int ichars;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
diff --git a/Src/compat.c b/Src/compat.c
index a295694..ca9713b 100644
--- a/Src/compat.c
+++ b/Src/compat.c
@@ -1017,3 +1017,20 @@ isprint_ascii(int c)
 
 /**/
 #endif /* __APPLE__ && BROKEN_ISPRINT */
+
+/**/
+#if defined(__APPLE__) && defined(BROKEN_ISWPRINT)
+
+/**/
+int
+wc_isprint(wint_t ucs)
+{
+    if (ucs <= 0)
+	return 0;
+    if (ucs < 32 || (ucs >= 0x7f && ucs < 0xa0))
+	return 0;
+    return 1;
+}
+
+/**/
+#endif /* __APPLE__ && BROKEN_ISWPRINT */
diff --git a/Src/pattern.c b/Src/pattern.c
index 75db016..fc7c737 100644
--- a/Src/pattern.c
+++ b/Src/pattern.c
@@ -3625,7 +3625,7 @@ mb_patmatchrange(char *range, wchar_t ch, int zmb_ind, wint_t *indptr, int *mtp)
 		    return 1;
 		break;
 	    case PP_PRINT:
-		if (iswprint(ch))
+		if (WC_ISPRINT(ch))
 		    return 1;
 		break;
 	    case PP_PUNCT:
diff --git a/Src/utils.c b/Src/utils.c
index ea4b34b..8aceb79 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -629,7 +629,7 @@ wcs_nicechar_sel(wchar_t c, size_t *widthp, char **swidep, int quotable)
     }
 
     s = buf;
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f) {
 	    if (quotable) {
 		*s++ = '\\';
@@ -734,7 +734,7 @@ wcs_nicechar(wchar_t c, size_t *widthp, char **swidep)
 /**/
 mod_export int is_wcs_nicechar(wchar_t c)
 {
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f || c == L'\n' || c == L'\t' || c < 0x20)
 	    return 1;
 	if (c >= 0x80) {
diff --git a/Src/ztype.h b/Src/ztype.h
index 76589b1..a8f5fe5 100644
--- a/Src/ztype.h
+++ b/Src/ztype.h
@@ -72,7 +72,11 @@
 
 #ifdef MULTIBYTE_SUPPORT
 #define WC_ZISTYPE(X,Y) wcsitype((X),(Y))
-#define WC_ISPRINT(X)	iswprint(X)
+# if defined(__APPLE__) && defined(BROKEN_ISWPRINT)
+#  define WC_ISPRINT(X)	wc_isprint(X)
+# else
+#  define WC_ISPRINT(X)	iswprint(X)
+# endif
 #else
 #define WC_ZISTYPE(X,Y)	zistype((X),(Y))
 #define WC_ISPRINT(X)	isprint(X)
diff --git a/configure.ac b/configure.ac
index 911cc45..d2f418d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2591,14 +2591,18 @@ fi])
 
 AH_TEMPLATE([BROKEN_WCWIDTH],
 [Define to 1 if the wcwidth() function is present but broken.])
+AH_TEMPLATE([BROKEN_ISWPRINT],
+[Define to 1 if the iswprint() function is present but broken.])
 AH_TEMPLATE([BROKEN_ISPRINT],
 [Define to 1 if the isprint() function is broken under UTF-8 locale.])
 if test x$zsh_cv_c_unicode_support = xyes; then
   AC_DEFINE(MULTIBYTE_SUPPORT)
 
-  dnl Test for a wcwidth() implementation that gives the wrong width for
-  dnl zero-width combining characters.
-  dnl For the test we use a combining acute accent (\u0301).
+  dnl Test for a wcwidth() implementation that gives the wrong width for either
+  dnl   zero-width combining characters, or
+  dnl   some characters in the Latin Extended-B.
+  dnl For the test we use a combining acute accent (\u0301) or
+  dnl a LATIN SMALL LETTER L WITH CURL (\u0234).
   dnl We input it as UTF-8 since that is the standard we can rely
   dnl upon most:  we can't rely on a wchar_t being stored as a
   dnl Unicode code point on all systems.
@@ -2607,9 +2611,8 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   dnl - the programme compiled, linked and ran
   dnl - we successfully set a UTF-8 locale
   dnl - the locale we set plausibly converted the UTF-8 string
-  dnl   for a zero-width combining character (the only way to be
-  dnl   100% sure would be to output it and ask if it looked right)
-  dnl - the converted wide character gave a non-zero width.
+  dnl   into the correct wide character
+  dnl - but the converted wide character gave a wrong width.
   dnl locale -a is a fallback; on most systems we should find en_US.UTF-8.
   [locale_prog='char *my_locales[] = {
   "en_US.UTF-8", "en_GB.UTF-8", "en.UTF-8", '
@@ -2625,17 +2628,19 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   int main() {
     char **localep;
     char comb_acute_mb[] = { (char)0xcc, (char)0x81 };
+    char u_0234[] = { (char)0xc8, (char)0xb4 };
     wchar_t wc;
 
     for (localep = my_locales; *localep; localep++)
-      if (setlocale(LC_ALL, *localep) &&
-          mbtowc(&wc, comb_acute_mb, 2) == 2)
+      if (setlocale(LC_ALL, *localep))
 	  break;
     if (!*localep)
       return 1;
-    if (wcwidth(wc) == 0)
-      return 1;
-    return 0;
+    if (mbtowc(&wc, comb_acute_mb, 2) == 2 && wcwidth(wc) != 0)
+      return 0;
+    if (mbtowc(&wc, u_0234, 2) == 2 && wcwidth(wc) != 1)
+      return 0;
+    return 1;
   }
   "]
 
@@ -2649,6 +2654,43 @@ if test x$zsh_cv_c_unicode_support = xyes; then
     AC_DEFINE(BROKEN_WCWIDTH)
   fi
 
+  dnl Check if iswprint() is broken.
+  [locale_prog='char *my_locales[] = {
+  "en_US.UTF-8", "en_GB.UTF-8", "en.UTF-8", '
+  locale_prog="$locale_prog"`locale -a 2>/dev/null | \
+    sed -e 's/utf8/UTF-8/' | grep UTF-8 | \
+    while read line; do echo " \"$line\","; done;`
+  locale_prog="$locale_prog 0 };
+  #include <stdlib.h>
+  #include <locale.h>
+  #include <wchar.h>
+  #include <wctype.h>
+
+  int main() {
+    char **localep;
+    char u_0234[] = { (char)0xc8, (char)0xb4 };
+    wchar_t wc;
+    for (localep = my_locales; *localep; localep++)
+      if (setlocale(LC_ALL, *localep))
+	break;
+    if (!*localep)
+      return 1;
+    if (mbtowc(&wc, u_0234, 2) == 2 && !iswprint(wc))
+      return 0;
+    return 1;
+  }
+  "]
+
+  AC_CACHE_CHECK(if the iswprint() function is broken,
+  zsh_cv_c_broken_iswprint,
+  [AC_TRY_RUN([$locale_prog],
+  zsh_cv_c_broken_iswprint=yes,
+  zsh_cv_c_broken_iswprint=no,
+  zsh_cv_c_broken_iswprint=no)])
+  if test x$zsh_cv_c_broken_iswprint = xyes; then
+    AC_DEFINE(BROKEN_ISWPRINT)
+  fi
+
   dnl Check if isprint() behaves correctly under UTF-8 locale.
   dnl On some platform (maybe only on Mac OS X), isprint() returns
   dnl true for all characters in the range from 0xa0 to 0xff if

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Peter Stephenson]

On Tue, 2 May 2017 00:52:03 +0900
Jun T. <takimoto-j@kba.biglobe.ne.jp> wrote:
> Another possibility is to use --enable-unicode9 if wcwidth()
> and/or iswprint() are broken (--enable-unicode9 works fine
> without any additional libraries). There is no iswprint-replacement
> in wcwidth.h, but implementing it would be easy if we can use the
> array wcwidth9_nonprint in wcwidth9.h.

That sounds like a future-proof approach, though I'm not sure how we'd
determine whether the contents of wcwdith9.h are appropriate for the
system.  Probably just ad hoc testing for now.

You can probably email Joshua Rubin <joshua@rubixconsulting.com> if you
have calculations about that file (see message 40037).

pws

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 1 maja 2017 at 18:38:07, Jun T. (takimoto-j@kba.biglobe.ne.jp) wrote:
> ...
> In the patch below, I added a test in configure.ac using U+0234
> for both wcwidth() and iswprint() (both are broken on macOS;
> wcwidth() returns -1 and iswprint() returns 0=false).

This is what I was thinking about this, to just test U+0234 or other character from the broken ranges. However I cannot test your patch. Google removed ability to download message, so I even configured getmail and Mutt but patch still fails, could you attach it? I would test it.

I look forward for this patch to be commited. I coded a tool that allows to pack Git repository data into base-1024 URL:

% giturl https://github.com/zdharma/giturl -r devel -p lib/coding_functions.cpp

Protocol: https
Site:     github.com
Repo:     zdharma/giturl
Revision: devel
File:     lib/coding_functions.cpp

gitu://ŬṽǚǫoŒẗ6ẏȅcЭÑẩőn4ầŘїệαЃȣϟṈӛŀї

It is coded in Zsh (and also a second one in C++11), but the point is: pasting to Zsh should work for all characters from the base-1024 range. I.e. for Latin, Greek, Cyrillic, Hebrew *letters* (not symbols).

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Bart Schaefer]

On May 6,  5:58pm, Sebastian Gniazdowski wrote:
}
} Google removed ability to download message

?? In gmail, view the message, click the triangle to the right of the
swoopy reply-arrow (top right), choose "Show Original" -- at bottom
left of the message headers table is a link for "Download Original".

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Sat, May 06, 2017 at 17:58:48 +0200:
> I even configured getmail and Mutt but patch still fails, could you
> attach it?

I've done so offlist.  I've verified the patch applies to
aa8157b463c1.

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 6 maja 2017 at 20:12:01, Bart Schaefer (schaefer@brasslantern.com) wrote:
> On May 6, 5:58pm, Sebastian Gniazdowski wrote:
> }
> } Google removed ability to download message
> 
> ?? In gmail, view the message, click the triangle to the right of the
> swoopy reply-arrow (top right), choose "Show Original" -- at bottom
> left of the message headers table is a link for "Download Original".

Thanks, I think that changed recently (together with removal of "Message text garbled", this one is certain), and when I use the link, I get e.g.:

int width =3D WCWIDTH(*u);

instead of:

int width = WCWIDTH(*u);

It's the same (the =3D and other changes) what getmail obtains.
--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 1 maja 2017 at 18:38:07, Jun T. (takimoto-j@kba.biglobe.ne.jp) wrote:
> Here is a quick (maybe too simple) patch.
>  
> wcwidth() on MacOSX was broken for combining characters,
> but Apple has fixed *this* problem a few years ago,
> probably in OSX 10.8 (Mavericks). So BROKEN_WCWIDTH is
> NOT defined on recent macOS.

Tested the patch. I get:

checking if the wcwidth() function is broken... yes
checking if the iswprint() function is broken... yes
checking if the isprint() function is broken... yes

and pasting of "ȮȯȰȱȲȳȴȵȶȷȸȹ" works. Also tried to paste the 1024 characters that I use and it worked.

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Sun, May 07, 2017 at 11:50:39 +0200:
> int width =3D WCWIDTH(*u);
> 
> instead of:
> 
> int width = WCWIDTH(*u);
> 
> It's the same (the =3D and other changes) what getmail obtains.

That's normal for quoted-printable encoded MIME parts.  To decode the
=-escaping (it uses = as an escape character), either view the email in
a mail client, or extract the first body MIME part and pass it through
a quoted-printable decoder.

Re: Gmail patch extraction (was: Cannot paste unicode <0221>, <0234> - <024f>)

[Sebastian Gniazdowski]

On 8 maja 2017 at 04:45:21, Daniel Shahaf (d.s@daniel.shahaf.name) wrote:
> Sebastian Gniazdowski wrote on Sun, May 07, 2017 at 11:50:39 +0200:
> > It's the same (the =3D and other changes) what getmail obtains.
> 
> That's normal for quoted-printable encoded MIME parts. To decode the
> =-escaping (it uses = as an escape character), either view the email in
> a mail client, or extract the first body MIME part and pass it through
> a quoted-printable decoder.

Ok but it was working before on gmail, I've applied many patches by downloading .eml file and removing discussion text, they've had to have "=" in them.

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Gmail patch extraction (was: Cannot paste unicode <0221>, <0234> - <024f>)

[Sebastian Gniazdowski]

On 8 maja 2017 at 06:05:38, Sebastian Gniazdowski (psprint@zdharma.org) wrote:
> Ok but it was working before on gmail, I've applied many patches by downloading .eml file 
> and removing discussion text, they've had to have "=" in them.

Hm but it's probably client that encoded = into =3D and breaked long lines with =. At some point I'll probably create a tool that will remove discussion text and decode, I've tried copy-paste from Mutt and it didn't work

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Gmail patch extraction (was: Cannot paste unicode <0221>, <0234> - <024f>)

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Mon, May 08, 2017 at 06:33:38 +0200:
> On 8 maja 2017 at 06:05:38, Sebastian Gniazdowski (psprint@zdharma.org) wrote:
> > Ok but it was working before on gmail, I've applied many patches by downloading .eml file 
> > and removing discussion text, they've had to have "=" in them.
> 
> Hm but it's probably client that encoded = into =3D and breaked long
> lines with =. At some point I'll probably create a tool that will
> remove discussion text and decode, I've tried copy-paste from Mutt and
> it didn't work

When you use mutt you need to use the 'v' command (from the pager (read
one message) screen) and then <Enter> on the relevant body part,
otherwise there'll be "+" signs at the start of visual lines.  (Instead
of pressing <Enter> you can actually press '|' and pipe the body part to
patch(1) as-is.)

I suspect we might be outstaying our welcome in the list subscribers'
mailboxes at this point, so let's continue on #zsh IRC.

Cheers,

Daniel

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Jun T.]

[-- Attachment #1: Type: text/plain, Size: 25322 bytes --]


On 2017/05/02, at 0:52, Jun T. <takimoto-j@kba.biglobe.ne.jp> wrote:
> 
> Another possibility is to use --enable-unicode9 if wcwidth()
> and/or iswprint() are broken

I had some discussion with Joshua Rubin and concluded that this
is the way to go.

In the patch below (against the current master), if wcwidth() OR
iswprint() is broken then BOTH will be replaced by the ones derived
from wcwidth9.h.
(the patch is also attached as a file u9-diff.txt)

BROKEN_WCWIDTH (and BROKEN_ISWPRINT introduced in the previous patch)
are removed; we will set ENABLE_UNICODE9 if either (or both) is broken
(and __STDC_ISO_10646__ is defined, or building on macOS).

BROKEN_WCWIDTH was used in zsh.h (second hunk in the patch) when
defining IS_COMBINING(), but I think this is not necessary because the
table for combining chars in wcwidth9.h is sufficiently reliable and need
not be checked against system's iswcntrl() (which might also be broken).

I also modified wcwidth9.h so that wcwidth9() return 0 for combining
characters (it was returning -1), and added U+2028/2029 to the list
of non-printable characters (I believe there will be no font glyphs
for these two characters).


diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8391739..d0dd1ef 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -1278,7 +1278,7 @@ zrefresh(void)
 #ifdef __STDC_ISO_10646__
 		 !ZSH_INVALID_WCHAR_TEST(*t) &&
 #endif
-		 iswprint(*t) && (width = WCWIDTH(*t)) > 0) {
+		 WC_ISPRINT(*t) && (width = WCWIDTH(*t)) > 0) {
 	    int ichars;
 	    if (width > rpms.sen - rpms.s) {
 		int started = 0;
@@ -1460,7 +1460,7 @@ zrefresh(void)
 	u = outputline;
 	for (; u < outputline + outll; u++) {
 #ifdef MULTIBYTE_SUPPORT
-	    if (iswprint(*u)) {
+	    if (WC_ISPRINT(*u)) {
 		int width = WCWIDTH(*u);
 		/* Handle wide characters as above */
 		if (width > rpms.sen - rpms.s) {
@@ -2468,7 +2468,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	if (tmpline[t0] == ZWC('\t'))
 	    vsiz = (vsiz | 7) + 2;
 #ifdef MULTIBYTE_SUPPORT
-	else if (iswprint(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
+	else if (WC_ISPRINT(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
 	    vsiz += width;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
 		while (t0 < tmpll-1 && IS_COMBINING(tmpline[t0+1]))
@@ -2556,7 +2556,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	    vp->atr = all_atr_on | all_atr_off;
 	    vp++;
 #ifdef MULTIBYTE_SUPPORT
-	} else if (iswprint(tmpline[t0]) &&
+	} else if (WC_ISPRINT(tmpline[t0]) &&
 		   (width = WCWIDTH(tmpline[t0])) > 0) {
 	    int ichars;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
diff --git a/Src/compat.c b/Src/compat.c
index a295694..a130d92 100644
--- a/Src/compat.c
+++ b/Src/compat.c
@@ -672,7 +672,7 @@ strtoul(nptr, endptr, base)
 
 /**/
 int
-mk_wcwidth(wchar_t ucs)
+u9_wcwidth(wchar_t ucs)
 {
   int w = wcwidth9(ucs);
   if (w < -1)
@@ -681,326 +681,16 @@ mk_wcwidth(wchar_t ucs)
 }
 
 /**/
-#elif defined(BROKEN_WCWIDTH) && (defined(__STDC_ISO_10646__) || defined(__APPLE__))
-
-/*
- * This is an implementation of wcwidth() and wcswidth() (defined in
- * IEEE Std 1002.1-2001) for Unicode.
- *
- * http://www.opengroup.org/onlinepubs/007904975/functions/wcwidth.html
- * http://www.opengroup.org/onlinepubs/007904975/functions/wcswidth.html
- *
- * In fixed-width output devices, Latin characters all occupy a single
- * "cell" position of equal width, whereas ideographic CJK characters
- * occupy two such cells. Interoperability between terminal-line
- * applications and (teletype-style) character terminals using the
- * UTF-8 encoding requires agreement on which character should advance
- * the cursor by how many cell positions. No established formal
- * standards exist at present on which Unicode character shall occupy
- * how many cell positions on character terminals. These routines are
- * a first attempt of defining such behavior based on simple rules
- * applied to data provided by the Unicode Consortium.
- *
- * For some graphical characters, the Unicode standard explicitly
- * defines a character-cell width via the definition of the East Asian
- * FullWidth (F), Wide (W), Half-width (H), and Narrow (Na) classes.
- * In all these cases, there is no ambiguity about which width a
- * terminal shall use. For characters in the East Asian Ambiguous (A)
- * class, the width choice depends purely on a preference of backward
- * compatibility with either historic CJK or Western practice.
- * Choosing single-width for these characters is easy to justify as
- * the appropriate long-term solution, as the CJK practice of
- * displaying these characters as double-width comes from historic
- * implementation simplicity (8-bit encoded characters were displayed
- * single-width and 16-bit ones double-width, even for Greek,
- * Cyrillic, etc.) and not any typographic considerations.
- *
- * Much less clear is the choice of width for the Not East Asian
- * (Neutral) class. Existing practice does not dictate a width for any
- * of these characters. It would nevertheless make sense
- * typographically to allocate two character cells to characters such
- * as for instance EM SPACE or VOLUME INTEGRAL, which cannot be
- * represented adequately with a single-width glyph. The following
- * routines at present merely assign a single-cell width to all
- * neutral characters, in the interest of simplicity. This is not
- * entirely satisfactory and should be reconsidered before
- * establishing a formal standard in this area. At the moment, the
- * decision which Not East Asian (Neutral) characters should be
- * represented by double-width glyphs cannot yet be answered by
- * applying a simple rule from the Unicode database content. Setting
- * up a proper standard for the behavior of UTF-8 character terminals
- * will require a careful analysis not only of each Unicode character,
- * but also of each presentation form, something the author of these
- * routines has avoided to do so far.
- *
- * http://www.unicode.org/unicode/reports/tr11/
- *
- * Markus Kuhn -- 2007-05-26 (Unicode 5.0)
- *
- * Permission to use, copy, modify, and distribute this software
- * for any purpose and without fee is hereby granted. The author
- * disclaims all warranties with regard to this software.
- *
- * Latest version: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
- */
-
-struct interval {
-  int first;
-  int last;
-};
-
-/* auxiliary function for binary search in interval table */
-static int bisearch(wchar_t ucs, const struct interval *table, int max) {
-  int min = 0;
-  int mid;
-
-  if (ucs < table[0].first || ucs > table[max].last)
-    return 0;
-  while (max >= min) {
-    mid = (min + max) / 2;
-    if (ucs > table[mid].last)
-      min = mid + 1;
-    else if (ucs < table[mid].first)
-      max = mid - 1;
-    else
-      return 1;
-  }
-
-  return 0;
-}
-
-
-/* The following two functions define the column width of an ISO 10646
- * character as follows:
- *
- *    - The null character (U+0000) has a column width of 0.
- *
- *    - Other C0/C1 control characters and DEL will lead to a return
- *      value of -1.
- *
- *    - Non-spacing and enclosing combining characters (general
- *      category code Mn or Me in the Unicode database) have a
- *      column width of 0.
- *
- *    - SOFT HYPHEN (U+00AD) has a column width of 1.
- *
- *    - Other format characters (general category code Cf in the Unicode
- *      database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.
- *
- *    - Hangul Jamo medial vowels and final consonants (U+1160-U+11FF)
- *      have a column width of 0.
- *
- *    - Spacing characters in the East Asian Wide (W) or East Asian
- *      Full-width (F) category as defined in Unicode Technical
- *      Report #11 have a column width of 2.
- *
- *    - All remaining characters (including all printable
- *      ISO 8859-1 and WGL4 characters, Unicode control characters,
- *      etc.) have a column width of 1.
- *
- * This implementation assumes that wchar_t characters are encoded
- * in ISO 10646.
- */
-
-/**/
 int
-mk_wcwidth(wchar_t ucs)
-{
-  /* sorted list of non-overlapping intervals of non-spacing characters */
-  /* generated by "uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c" */
-  static const struct interval combining[] = {
-    { 0x0300, 0x036F }, { 0x0483, 0x0486 }, { 0x0488, 0x0489 },
-    { 0x0591, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },
-    { 0x05C4, 0x05C5 }, { 0x05C7, 0x05C7 }, { 0x0600, 0x0603 },
-    { 0x0610, 0x0615 }, { 0x064B, 0x065E }, { 0x0670, 0x0670 },
-    { 0x06D6, 0x06E4 }, { 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED },
-    { 0x070F, 0x070F }, { 0x0711, 0x0711 }, { 0x0730, 0x074A },
-    { 0x07A6, 0x07B0 }, { 0x07EB, 0x07F3 }, { 0x0901, 0x0902 },
-    { 0x093C, 0x093C }, { 0x0941, 0x0948 }, { 0x094D, 0x094D },
-    { 0x0951, 0x0954 }, { 0x0962, 0x0963 }, { 0x0981, 0x0981 },
-    { 0x09BC, 0x09BC }, { 0x09C1, 0x09C4 }, { 0x09CD, 0x09CD },
-    { 0x09E2, 0x09E3 }, { 0x0A01, 0x0A02 }, { 0x0A3C, 0x0A3C },
-    { 0x0A41, 0x0A42 }, { 0x0A47, 0x0A48 }, { 0x0A4B, 0x0A4D },
-    { 0x0A70, 0x0A71 }, { 0x0A81, 0x0A82 }, { 0x0ABC, 0x0ABC },
-    { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 }, { 0x0ACD, 0x0ACD },
-    { 0x0AE2, 0x0AE3 }, { 0x0B01, 0x0B01 }, { 0x0B3C, 0x0B3C },
-    { 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B43 }, { 0x0B4D, 0x0B4D },
-    { 0x0B56, 0x0B56 }, { 0x0B82, 0x0B82 }, { 0x0BC0, 0x0BC0 },
-    { 0x0BCD, 0x0BCD }, { 0x0C3E, 0x0C40 }, { 0x0C46, 0x0C48 },
-    { 0x0C4A, 0x0C4D }, { 0x0C55, 0x0C56 }, { 0x0CBC, 0x0CBC },
-    { 0x0CBF, 0x0CBF }, { 0x0CC6, 0x0CC6 }, { 0x0CCC, 0x0CCD },
-    { 0x0CE2, 0x0CE3 }, { 0x0D41, 0x0D43 }, { 0x0D4D, 0x0D4D },
-    { 0x0DCA, 0x0DCA }, { 0x0DD2, 0x0DD4 }, { 0x0DD6, 0x0DD6 },
-    { 0x0E31, 0x0E31 }, { 0x0E34, 0x0E3A }, { 0x0E47, 0x0E4E },
-    { 0x0EB1, 0x0EB1 }, { 0x0EB4, 0x0EB9 }, { 0x0EBB, 0x0EBC },
-    { 0x0EC8, 0x0ECD }, { 0x0F18, 0x0F19 }, { 0x0F35, 0x0F35 },
-    { 0x0F37, 0x0F37 }, { 0x0F39, 0x0F39 }, { 0x0F71, 0x0F7E },
-    { 0x0F80, 0x0F84 }, { 0x0F86, 0x0F87 }, { 0x0F90, 0x0F97 },
-    { 0x0F99, 0x0FBC }, { 0x0FC6, 0x0FC6 }, { 0x102D, 0x1030 },
-    { 0x1032, 0x1032 }, { 0x1036, 0x1037 }, { 0x1039, 0x1039 },
-    { 0x1058, 0x1059 }, { 0x1160, 0x11FF }, { 0x135F, 0x135F },
-    { 0x1712, 0x1714 }, { 0x1732, 0x1734 }, { 0x1752, 0x1753 },
-    { 0x1772, 0x1773 }, { 0x17B4, 0x17B5 }, { 0x17B7, 0x17BD },
-    { 0x17C6, 0x17C6 }, { 0x17C9, 0x17D3 }, { 0x17DD, 0x17DD },
-    { 0x180B, 0x180D }, { 0x18A9, 0x18A9 }, { 0x1920, 0x1922 },
-    { 0x1927, 0x1928 }, { 0x1932, 0x1932 }, { 0x1939, 0x193B },
-    { 0x1A17, 0x1A18 }, { 0x1B00, 0x1B03 }, { 0x1B34, 0x1B34 },
-    { 0x1B36, 0x1B3A }, { 0x1B3C, 0x1B3C }, { 0x1B42, 0x1B42 },
-    { 0x1B6B, 0x1B73 }, { 0x1DC0, 0x1DCA }, { 0x1DFE, 0x1DFF },
-    { 0x200B, 0x200F }, { 0x202A, 0x202E }, { 0x2060, 0x2063 },
-    { 0x206A, 0x206F }, { 0x20D0, 0x20EF }, { 0x302A, 0x302F },
-    { 0x3099, 0x309A }, { 0xA806, 0xA806 }, { 0xA80B, 0xA80B },
-    { 0xA825, 0xA826 }, { 0xFB1E, 0xFB1E }, { 0xFE00, 0xFE0F },
-    { 0xFE20, 0xFE23 }, { 0xFEFF, 0xFEFF }, { 0xFFF9, 0xFFFB },
-    { 0x10A01, 0x10A03 }, { 0x10A05, 0x10A06 }, { 0x10A0C, 0x10A0F },
-    { 0x10A38, 0x10A3A }, { 0x10A3F, 0x10A3F }, { 0x1D167, 0x1D169 },
-    { 0x1D173, 0x1D182 }, { 0x1D185, 0x1D18B }, { 0x1D1AA, 0x1D1AD },
-    { 0x1D242, 0x1D244 }, { 0xE0001, 0xE0001 }, { 0xE0020, 0xE007F },
-    { 0xE0100, 0xE01EF }
-  };
-
-  /* test for 8-bit control characters */
-  if (ucs == 0)
-    return 0;
-  if (ucs < 32 || (ucs >= 0x7f && ucs < 0xa0))
-    return -1;
-
-  /* binary search in table of non-spacing characters */
-  if (bisearch(ucs, combining,
-	       sizeof(combining) / sizeof(struct interval) - 1))
-    return 0;
-
-  /* if we arrive here, ucs is not a combining or C0/C1 control character */
-
-  return 1 +
-    (ucs >= 0x1100 &&
-     (ucs <= 0x115f ||                    /* Hangul Jamo init. consonants */
-      ucs == 0x2329 || ucs == 0x232a ||
-      (ucs >= 0x2e80 && ucs <= 0xa4cf &&
-       ucs != 0x303f) ||                  /* CJK ... Yi */
-      (ucs >= 0xac00 && ucs <= 0xd7a3) || /* Hangul Syllables */
-      (ucs >= 0xf900 && ucs <= 0xfaff) || /* CJK Compatibility Ideographs */
-      (ucs >= 0xfe10 && ucs <= 0xfe19) || /* Vertical forms */
-      (ucs >= 0xfe30 && ucs <= 0xfe6f) || /* CJK Compatibility Forms */
-      (ucs >= 0xff00 && ucs <= 0xff60) || /* Fullwidth Forms */
-      (ucs >= 0xffe0 && ucs <= 0xffe6) ||
-      (ucs >= 0x20000 && ucs <= 0x2fffd) ||
-      (ucs >= 0x30000 && ucs <= 0x3fffd)));
-}
-
-
-/*
- * The following functions are part of the original wcwidth.c:
- * we don't use them but I've kept them in case - pws.
- */
-#if 0
-int mk_wcswidth(const wchar_t *pwcs, size_t n)
-{
-  int w, width = 0;
-
-  for (;*pwcs && n-- > 0; pwcs++)
-    if ((w = mk_wcwidth(*pwcs)) < 0)
-      return -1;
-    else
-      width += w;
-
-  return width;
-}
-
-
-/*
- * The following functions are the same as mk_wcwidth() and
- * mk_wcswidth(), except that spacing characters in the East Asian
- * Ambiguous (A) category as defined in Unicode Technical Report #11
- * have a column width of 2. This variant might be useful for users of
- * CJK legacy encodings who want to migrate to UCS without changing
- * the traditional terminal character-width behaviour. It is not
- * otherwise recommended for general use.
- */
-int mk_wcwidth_cjk(wchar_t ucs)
+u9_iswprint(wint_t ucs)
 {
-  /* sorted list of non-overlapping intervals of East Asian Ambiguous
-   * characters, generated by "uniset +WIDTH-A -cat=Me -cat=Mn -cat=Cf c" */
-  static const struct interval ambiguous[] = {
-    { 0x00A1, 0x00A1 }, { 0x00A4, 0x00A4 }, { 0x00A7, 0x00A8 },
-    { 0x00AA, 0x00AA }, { 0x00AE, 0x00AE }, { 0x00B0, 0x00B4 },
-    { 0x00B6, 0x00BA }, { 0x00BC, 0x00BF }, { 0x00C6, 0x00C6 },
-    { 0x00D0, 0x00D0 }, { 0x00D7, 0x00D8 }, { 0x00DE, 0x00E1 },
-    { 0x00E6, 0x00E6 }, { 0x00E8, 0x00EA }, { 0x00EC, 0x00ED },
-    { 0x00F0, 0x00F0 }, { 0x00F2, 0x00F3 }, { 0x00F7, 0x00FA },
-    { 0x00FC, 0x00FC }, { 0x00FE, 0x00FE }, { 0x0101, 0x0101 },
-    { 0x0111, 0x0111 }, { 0x0113, 0x0113 }, { 0x011B, 0x011B },
-    { 0x0126, 0x0127 }, { 0x012B, 0x012B }, { 0x0131, 0x0133 },
-    { 0x0138, 0x0138 }, { 0x013F, 0x0142 }, { 0x0144, 0x0144 },
-    { 0x0148, 0x014B }, { 0x014D, 0x014D }, { 0x0152, 0x0153 },
-    { 0x0166, 0x0167 }, { 0x016B, 0x016B }, { 0x01CE, 0x01CE },
-    { 0x01D0, 0x01D0 }, { 0x01D2, 0x01D2 }, { 0x01D4, 0x01D4 },
-    { 0x01D6, 0x01D6 }, { 0x01D8, 0x01D8 }, { 0x01DA, 0x01DA },
-    { 0x01DC, 0x01DC }, { 0x0251, 0x0251 }, { 0x0261, 0x0261 },
-    { 0x02C4, 0x02C4 }, { 0x02C7, 0x02C7 }, { 0x02C9, 0x02CB },
-    { 0x02CD, 0x02CD }, { 0x02D0, 0x02D0 }, { 0x02D8, 0x02DB },
-    { 0x02DD, 0x02DD }, { 0x02DF, 0x02DF }, { 0x0391, 0x03A1 },
-    { 0x03A3, 0x03A9 }, { 0x03B1, 0x03C1 }, { 0x03C3, 0x03C9 },
-    { 0x0401, 0x0401 }, { 0x0410, 0x044F }, { 0x0451, 0x0451 },
-    { 0x2010, 0x2010 }, { 0x2013, 0x2016 }, { 0x2018, 0x2019 },
-    { 0x201C, 0x201D }, { 0x2020, 0x2022 }, { 0x2024, 0x2027 },
-    { 0x2030, 0x2030 }, { 0x2032, 0x2033 }, { 0x2035, 0x2035 },
-    { 0x203B, 0x203B }, { 0x203E, 0x203E }, { 0x2074, 0x2074 },
-    { 0x207F, 0x207F }, { 0x2081, 0x2084 }, { 0x20AC, 0x20AC },
-    { 0x2103, 0x2103 }, { 0x2105, 0x2105 }, { 0x2109, 0x2109 },
-    { 0x2113, 0x2113 }, { 0x2116, 0x2116 }, { 0x2121, 0x2122 },
-    { 0x2126, 0x2126 }, { 0x212B, 0x212B }, { 0x2153, 0x2154 },
-    { 0x215B, 0x215E }, { 0x2160, 0x216B }, { 0x2170, 0x2179 },
-    { 0x2190, 0x2199 }, { 0x21B8, 0x21B9 }, { 0x21D2, 0x21D2 },
-    { 0x21D4, 0x21D4 }, { 0x21E7, 0x21E7 }, { 0x2200, 0x2200 },
-    { 0x2202, 0x2203 }, { 0x2207, 0x2208 }, { 0x220B, 0x220B },
-    { 0x220F, 0x220F }, { 0x2211, 0x2211 }, { 0x2215, 0x2215 },
-    { 0x221A, 0x221A }, { 0x221D, 0x2220 }, { 0x2223, 0x2223 },
-    { 0x2225, 0x2225 }, { 0x2227, 0x222C }, { 0x222E, 0x222E },
-    { 0x2234, 0x2237 }, { 0x223C, 0x223D }, { 0x2248, 0x2248 },
-    { 0x224C, 0x224C }, { 0x2252, 0x2252 }, { 0x2260, 0x2261 },
-    { 0x2264, 0x2267 }, { 0x226A, 0x226B }, { 0x226E, 0x226F },
-    { 0x2282, 0x2283 }, { 0x2286, 0x2287 }, { 0x2295, 0x2295 },
-    { 0x2299, 0x2299 }, { 0x22A5, 0x22A5 }, { 0x22BF, 0x22BF },
-    { 0x2312, 0x2312 }, { 0x2460, 0x24E9 }, { 0x24EB, 0x254B },
-    { 0x2550, 0x2573 }, { 0x2580, 0x258F }, { 0x2592, 0x2595 },
-    { 0x25A0, 0x25A1 }, { 0x25A3, 0x25A9 }, { 0x25B2, 0x25B3 },
-    { 0x25B6, 0x25B7 }, { 0x25BC, 0x25BD }, { 0x25C0, 0x25C1 },
-    { 0x25C6, 0x25C8 }, { 0x25CB, 0x25CB }, { 0x25CE, 0x25D1 },
-    { 0x25E2, 0x25E5 }, { 0x25EF, 0x25EF }, { 0x2605, 0x2606 },
-    { 0x2609, 0x2609 }, { 0x260E, 0x260F }, { 0x2614, 0x2615 },
-    { 0x261C, 0x261C }, { 0x261E, 0x261E }, { 0x2640, 0x2640 },
-    { 0x2642, 0x2642 }, { 0x2660, 0x2661 }, { 0x2663, 0x2665 },
-    { 0x2667, 0x266A }, { 0x266C, 0x266D }, { 0x266F, 0x266F },
-    { 0x273D, 0x273D }, { 0x2776, 0x277F }, { 0xE000, 0xF8FF },
-    { 0xFFFD, 0xFFFD }, { 0xF0000, 0xFFFFD }, { 0x100000, 0x10FFFD }
-  };
-
-  /* binary search in table of non-spacing characters */
-  if (bisearch(ucs, ambiguous,
-	       sizeof(ambiguous) / sizeof(struct interval) - 1))
-    return 2;
-
-  return mk_wcwidth(ucs);
+    if (ucs == 0)
+	return 0;
+    return wcwidth9(ucs) != -1;
 }
 
-
-int mk_wcswidth_cjk(const wchar_t *pwcs, size_t n)
-{
-  int w, width = 0;
-
-  for (;*pwcs && n-- > 0; pwcs++)
-    if ((w = mk_wcwidth_cjk(*pwcs)) < 0)
-      return -1;
-    else
-      width += w;
-
-  return width;
-}
-#endif /* 0 */
-
 /**/
-#endif /* BROKEN_WCWIDTH && (__STDC_ISO_10646__ || __APPLE__) */
+#endif	/* ENABLE_UNICODE9 */
 
 /**/
 #if defined(__APPLE__) && defined(BROKEN_ISPRINT)
diff --git a/Src/pattern.c b/Src/pattern.c
index 75db016..fc7c737 100644
--- a/Src/pattern.c
+++ b/Src/pattern.c
@@ -3625,7 +3625,7 @@ mb_patmatchrange(char *range, wchar_t ch, int zmb_ind, wint_t *indptr, int *mtp)
 		    return 1;
 		break;
 	    case PP_PRINT:
-		if (iswprint(ch))
+		if (WC_ISPRINT(ch))
 		    return 1;
 		break;
 	    case PP_PUNCT:
diff --git a/Src/utils.c b/Src/utils.c
index 5eb936b..acb891d 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -629,7 +629,7 @@ wcs_nicechar_sel(wchar_t c, size_t *widthp, char **swidep, int quotable)
     }
 
     s = buf;
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f) {
 	    if (quotable) {
 		*s++ = '\\';
@@ -734,7 +734,7 @@ wcs_nicechar(wchar_t c, size_t *widthp, char **swidep)
 /**/
 mod_export int is_wcs_nicechar(wchar_t c)
 {
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f || c == L'\n' || c == L'\t' || c < 0x20)
 	    return 1;
 	if (c >= 0x80) {
diff --git a/Src/wcwidth9.h b/Src/wcwidth9.h
index 07e6bae..448f548 100644
--- a/Src/wcwidth9.h
+++ b/Src/wcwidth9.h
@@ -22,6 +22,7 @@ static const struct wcwidth9_interval wcwidth9_nonprint[] = {
   {0x070f, 0x070f},
   {0x180b, 0x180e},
   {0x200b, 0x200f},
+  {0x2028, 0x2029},
   {0x202a, 0x202e},
   {0x206a, 0x206f},
   {0xd800, 0xdfff},
@@ -1283,6 +1284,9 @@ static inline bool wcwidth9_intable(const struct wcwidth9_interval *table, size_
 }
 
 static inline int wcwidth9(int c) {
+  if (c == 0) {
+    return 0;
+  }
   if (c < 0|| c > 0x10ffff) {
     return -1;
   }
@@ -1292,7 +1296,7 @@ static inline int wcwidth9(int c) {
   }
 
   if (wcwidth9_intable(wcwidth9_combining, WCWIDTH9_ARRAY_SIZE(wcwidth9_combining), c)) {
-    return -1;
+    return 0;
   }
 
   if (wcwidth9_intable(wcwidth9_not_assigned, WCWIDTH9_ARRAY_SIZE(wcwidth9_not_assigned), c)) {
diff --git a/Src/zsh.h b/Src/zsh.h
index 1093151..22f73f8 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -3158,9 +3158,7 @@ typedef wint_t convchar_t;
  * works on MacOS which doesn't define that.
  */
 #ifdef ENABLE_UNICODE9
-#define WCWIDTH(wc)	mk_wcwidth(wc)
-#elif defined(BROKEN_WCWIDTH) && (defined(__STDC_ISO_10646__) || defined(__APPLE__))
-#define WCWIDTH(wc)	mk_wcwidth(wc)
+#define WCWIDTH(wc)	u9_wcwidth(wc)
 #else
 #define WCWIDTH(wc)	wcwidth(wc)
 #endif
@@ -3205,15 +3203,7 @@ typedef wint_t convchar_t;
  * sense throughout the shell.  I am not aware of a way of
  * detecting the Unicode trait in standard libraries.
  */
-#ifdef BROKEN_WCWIDTH
-/*
- * We can't be quite sure the wcwidth we've provided is entirely
- * in agreement with the system's, so be extra safe.
- */
-#define IS_COMBINING(wc)	(wc != 0 && WCWIDTH(wc) == 0 && !iswcntrl(wc))
-#else
 #define IS_COMBINING(wc)	(wc != 0 && WCWIDTH(wc) == 0)
-#endif
 /*
  * Test for the base of a combining character.
  *
diff --git a/Src/ztype.h b/Src/ztype.h
index 76589b1..ae72367 100644
--- a/Src/ztype.h
+++ b/Src/ztype.h
@@ -72,7 +72,11 @@
 
 #ifdef MULTIBYTE_SUPPORT
 #define WC_ZISTYPE(X,Y) wcsitype((X),(Y))
-#define WC_ISPRINT(X)	iswprint(X)
+# ifdef ENABLE_UNICODE9
+#  define WC_ISPRINT(X)	u9_iswprint(X)
+# else
+#  define WC_ISPRINT(X)	iswprint(X)
+# endif
 #else
 #define WC_ZISTYPE(X,Y)	zistype((X),(Y))
 #define WC_ISPRINT(X)	isprint(X)
diff --git a/configure.ac b/configure.ac
index 911cc45..88da89e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2589,16 +2589,21 @@ AC_HELP_STRING([--enable-unicode9], [compile with unicode9 character widths]),
   AC_DEFINE(ENABLE_UNICODE9)
 fi])
 
-AH_TEMPLATE([BROKEN_WCWIDTH],
-[Define to 1 if the wcwidth() function is present but broken.])
 AH_TEMPLATE([BROKEN_ISPRINT],
 [Define to 1 if the isprint() function is broken under UTF-8 locale.])
+
 if test x$zsh_cv_c_unicode_support = xyes; then
   AC_DEFINE(MULTIBYTE_SUPPORT)
 
-  dnl Test for a wcwidth() implementation that gives the wrong width for
-  dnl zero-width combining characters.
-  dnl For the test we use a combining acute accent (\u0301).
+  dnl Test if wcwidth() and/or iswprint() is broken for
+  dnl   zero-width combining characters, or
+  dnl   some characters in the Latin Extended-B.
+  dnl If either of the functions is broken, both functions will be replaced
+  dnl by the ones from wcwidth9.h by defining ENABLE_UNICODE9. We will do
+  dnl this only if __STDC_ISO_10646__ is defined (or if building on macOS,
+  dnl where __STDC_ISO_10646__ is not defined but wchar_t is UCS).
+  dnl For the test we use a combining acute accent (\u0301) or
+  dnl a LATIN SMALL LETTER L WITH CURL (\u0234).
   dnl We input it as UTF-8 since that is the standard we can rely
   dnl upon most:  we can't rely on a wchar_t being stored as a
   dnl Unicode code point on all systems.
@@ -2607,9 +2612,8 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   dnl - the programme compiled, linked and ran
   dnl - we successfully set a UTF-8 locale
   dnl - the locale we set plausibly converted the UTF-8 string
-  dnl   for a zero-width combining character (the only way to be
-  dnl   100% sure would be to output it and ask if it looked right)
-  dnl - the converted wide character gave a non-zero width.
+  dnl   into the correct wide character
+  dnl - but wcwidth() or iswprint() is broken for the converted wide character.
   dnl locale -a is a fallback; on most systems we should find en_US.UTF-8.
   [locale_prog='char *my_locales[] = {
   "en_US.UTF-8", "en_GB.UTF-8", "en.UTF-8", '
@@ -2621,32 +2625,38 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   #include <stdlib.h>
   #include <locale.h>
   #include <wchar.h>
+  #include <wctype.h>
 
   int main() {
     char **localep;
     char comb_acute_mb[] = { (char)0xcc, (char)0x81 };
+    char u_0234[] = { (char)0xc8, (char)0xb4 };
     wchar_t wc;
+  #if !defined(__STDC_ISO_10646__) && !defined(__APPLE__)
+    return 1;
+  #endif
 
     for (localep = my_locales; *localep; localep++)
-      if (setlocale(LC_ALL, *localep) &&
-          mbtowc(&wc, comb_acute_mb, 2) == 2)
+      if (setlocale(LC_ALL, *localep))
 	  break;
     if (!*localep)
       return 1;
-    if (wcwidth(wc) == 0)
-      return 1;
-    return 0;
+    if (mbtowc(&wc, comb_acute_mb, 2) == 2 && (wcwidth(wc) != 0 || !iswprint(wc)))
+      return 0;
+    if (mbtowc(&wc, u_0234, 2) == 2 && (wcwidth(wc) != 1 || !iswprint(wc)))
+      return 0;
+    return 1;
   }
   "]
 
-  AC_CACHE_CHECK(if the wcwidth() function is broken,
+  AC_CACHE_CHECK(if the wcwidth() and/or iswprint() functions are broken,
   zsh_cv_c_broken_wcwidth,
   [AC_TRY_RUN([$locale_prog],
   zsh_cv_c_broken_wcwidth=yes,
   zsh_cv_c_broken_wcwidth=no,
   zsh_cv_c_broken_wcwidth=no)])
   if test x$zsh_cv_c_broken_wcwidth = xyes; then
-    AC_DEFINE(BROKEN_WCWIDTH)
+    AC_DEFINE(ENABLE_UNICODE9)
   fi
 
   dnl Check if isprint() behaves correctly under UTF-8 locale.





[-- Attachment #2: u9-diff.txt --]
[-- Type: text/plain, Size: 24113 bytes --]

diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8391739..d0dd1ef 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -1278,7 +1278,7 @@ zrefresh(void)
 #ifdef __STDC_ISO_10646__
 		 !ZSH_INVALID_WCHAR_TEST(*t) &&
 #endif
-		 iswprint(*t) && (width = WCWIDTH(*t)) > 0) {
+		 WC_ISPRINT(*t) && (width = WCWIDTH(*t)) > 0) {
 	    int ichars;
 	    if (width > rpms.sen - rpms.s) {
 		int started = 0;
@@ -1460,7 +1460,7 @@ zrefresh(void)
 	u = outputline;
 	for (; u < outputline + outll; u++) {
 #ifdef MULTIBYTE_SUPPORT
-	    if (iswprint(*u)) {
+	    if (WC_ISPRINT(*u)) {
 		int width = WCWIDTH(*u);
 		/* Handle wide characters as above */
 		if (width > rpms.sen - rpms.s) {
@@ -2468,7 +2468,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	if (tmpline[t0] == ZWC('\t'))
 	    vsiz = (vsiz | 7) + 2;
 #ifdef MULTIBYTE_SUPPORT
-	else if (iswprint(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
+	else if (WC_ISPRINT(tmpline[t0]) && ((width = WCWIDTH(tmpline[t0])) > 0)) {
 	    vsiz += width;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
 		while (t0 < tmpll-1 && IS_COMBINING(tmpline[t0+1]))
@@ -2556,7 +2556,7 @@ singlerefresh(ZLE_STRING_T tmpline, int tmpll, int tmpcs)
 	    vp->atr = all_atr_on | all_atr_off;
 	    vp++;
 #ifdef MULTIBYTE_SUPPORT
-	} else if (iswprint(tmpline[t0]) &&
+	} else if (WC_ISPRINT(tmpline[t0]) &&
 		   (width = WCWIDTH(tmpline[t0])) > 0) {
 	    int ichars;
 	    if (isset(COMBININGCHARS) && IS_BASECHAR(tmpline[t0])) {
diff --git a/Src/compat.c b/Src/compat.c
index a295694..a130d92 100644
--- a/Src/compat.c
+++ b/Src/compat.c
@@ -672,7 +672,7 @@ strtoul(nptr, endptr, base)
 
 /**/
 int
-mk_wcwidth(wchar_t ucs)
+u9_wcwidth(wchar_t ucs)
 {
   int w = wcwidth9(ucs);
   if (w < -1)
@@ -681,326 +681,16 @@ mk_wcwidth(wchar_t ucs)
 }
 
 /**/
-#elif defined(BROKEN_WCWIDTH) && (defined(__STDC_ISO_10646__) || defined(__APPLE__))
-
-/*
- * This is an implementation of wcwidth() and wcswidth() (defined in
- * IEEE Std 1002.1-2001) for Unicode.
- *
- * http://www.opengroup.org/onlinepubs/007904975/functions/wcwidth.html
- * http://www.opengroup.org/onlinepubs/007904975/functions/wcswidth.html
- *
- * In fixed-width output devices, Latin characters all occupy a single
- * "cell" position of equal width, whereas ideographic CJK characters
- * occupy two such cells. Interoperability between terminal-line
- * applications and (teletype-style) character terminals using the
- * UTF-8 encoding requires agreement on which character should advance
- * the cursor by how many cell positions. No established formal
- * standards exist at present on which Unicode character shall occupy
- * how many cell positions on character terminals. These routines are
- * a first attempt of defining such behavior based on simple rules
- * applied to data provided by the Unicode Consortium.
- *
- * For some graphical characters, the Unicode standard explicitly
- * defines a character-cell width via the definition of the East Asian
- * FullWidth (F), Wide (W), Half-width (H), and Narrow (Na) classes.
- * In all these cases, there is no ambiguity about which width a
- * terminal shall use. For characters in the East Asian Ambiguous (A)
- * class, the width choice depends purely on a preference of backward
- * compatibility with either historic CJK or Western practice.
- * Choosing single-width for these characters is easy to justify as
- * the appropriate long-term solution, as the CJK practice of
- * displaying these characters as double-width comes from historic
- * implementation simplicity (8-bit encoded characters were displayed
- * single-width and 16-bit ones double-width, even for Greek,
- * Cyrillic, etc.) and not any typographic considerations.
- *
- * Much less clear is the choice of width for the Not East Asian
- * (Neutral) class. Existing practice does not dictate a width for any
- * of these characters. It would nevertheless make sense
- * typographically to allocate two character cells to characters such
- * as for instance EM SPACE or VOLUME INTEGRAL, which cannot be
- * represented adequately with a single-width glyph. The following
- * routines at present merely assign a single-cell width to all
- * neutral characters, in the interest of simplicity. This is not
- * entirely satisfactory and should be reconsidered before
- * establishing a formal standard in this area. At the moment, the
- * decision which Not East Asian (Neutral) characters should be
- * represented by double-width glyphs cannot yet be answered by
- * applying a simple rule from the Unicode database content. Setting
- * up a proper standard for the behavior of UTF-8 character terminals
- * will require a careful analysis not only of each Unicode character,
- * but also of each presentation form, something the author of these
- * routines has avoided to do so far.
- *
- * http://www.unicode.org/unicode/reports/tr11/
- *
- * Markus Kuhn -- 2007-05-26 (Unicode 5.0)
- *
- * Permission to use, copy, modify, and distribute this software
- * for any purpose and without fee is hereby granted. The author
- * disclaims all warranties with regard to this software.
- *
- * Latest version: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
- */
-
-struct interval {
-  int first;
-  int last;
-};
-
-/* auxiliary function for binary search in interval table */
-static int bisearch(wchar_t ucs, const struct interval *table, int max) {
-  int min = 0;
-  int mid;
-
-  if (ucs < table[0].first || ucs > table[max].last)
-    return 0;
-  while (max >= min) {
-    mid = (min + max) / 2;
-    if (ucs > table[mid].last)
-      min = mid + 1;
-    else if (ucs < table[mid].first)
-      max = mid - 1;
-    else
-      return 1;
-  }
-
-  return 0;
-}
-
-
-/* The following two functions define the column width of an ISO 10646
- * character as follows:
- *
- *    - The null character (U+0000) has a column width of 0.
- *
- *    - Other C0/C1 control characters and DEL will lead to a return
- *      value of -1.
- *
- *    - Non-spacing and enclosing combining characters (general
- *      category code Mn or Me in the Unicode database) have a
- *      column width of 0.
- *
- *    - SOFT HYPHEN (U+00AD) has a column width of 1.
- *
- *    - Other format characters (general category code Cf in the Unicode
- *      database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.
- *
- *    - Hangul Jamo medial vowels and final consonants (U+1160-U+11FF)
- *      have a column width of 0.
- *
- *    - Spacing characters in the East Asian Wide (W) or East Asian
- *      Full-width (F) category as defined in Unicode Technical
- *      Report #11 have a column width of 2.
- *
- *    - All remaining characters (including all printable
- *      ISO 8859-1 and WGL4 characters, Unicode control characters,
- *      etc.) have a column width of 1.
- *
- * This implementation assumes that wchar_t characters are encoded
- * in ISO 10646.
- */
-
-/**/
 int
-mk_wcwidth(wchar_t ucs)
-{
-  /* sorted list of non-overlapping intervals of non-spacing characters */
-  /* generated by "uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c" */
-  static const struct interval combining[] = {
-    { 0x0300, 0x036F }, { 0x0483, 0x0486 }, { 0x0488, 0x0489 },
-    { 0x0591, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },
-    { 0x05C4, 0x05C5 }, { 0x05C7, 0x05C7 }, { 0x0600, 0x0603 },
-    { 0x0610, 0x0615 }, { 0x064B, 0x065E }, { 0x0670, 0x0670 },
-    { 0x06D6, 0x06E4 }, { 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED },
-    { 0x070F, 0x070F }, { 0x0711, 0x0711 }, { 0x0730, 0x074A },
-    { 0x07A6, 0x07B0 }, { 0x07EB, 0x07F3 }, { 0x0901, 0x0902 },
-    { 0x093C, 0x093C }, { 0x0941, 0x0948 }, { 0x094D, 0x094D },
-    { 0x0951, 0x0954 }, { 0x0962, 0x0963 }, { 0x0981, 0x0981 },
-    { 0x09BC, 0x09BC }, { 0x09C1, 0x09C4 }, { 0x09CD, 0x09CD },
-    { 0x09E2, 0x09E3 }, { 0x0A01, 0x0A02 }, { 0x0A3C, 0x0A3C },
-    { 0x0A41, 0x0A42 }, { 0x0A47, 0x0A48 }, { 0x0A4B, 0x0A4D },
-    { 0x0A70, 0x0A71 }, { 0x0A81, 0x0A82 }, { 0x0ABC, 0x0ABC },
-    { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 }, { 0x0ACD, 0x0ACD },
-    { 0x0AE2, 0x0AE3 }, { 0x0B01, 0x0B01 }, { 0x0B3C, 0x0B3C },
-    { 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B43 }, { 0x0B4D, 0x0B4D },
-    { 0x0B56, 0x0B56 }, { 0x0B82, 0x0B82 }, { 0x0BC0, 0x0BC0 },
-    { 0x0BCD, 0x0BCD }, { 0x0C3E, 0x0C40 }, { 0x0C46, 0x0C48 },
-    { 0x0C4A, 0x0C4D }, { 0x0C55, 0x0C56 }, { 0x0CBC, 0x0CBC },
-    { 0x0CBF, 0x0CBF }, { 0x0CC6, 0x0CC6 }, { 0x0CCC, 0x0CCD },
-    { 0x0CE2, 0x0CE3 }, { 0x0D41, 0x0D43 }, { 0x0D4D, 0x0D4D },
-    { 0x0DCA, 0x0DCA }, { 0x0DD2, 0x0DD4 }, { 0x0DD6, 0x0DD6 },
-    { 0x0E31, 0x0E31 }, { 0x0E34, 0x0E3A }, { 0x0E47, 0x0E4E },
-    { 0x0EB1, 0x0EB1 }, { 0x0EB4, 0x0EB9 }, { 0x0EBB, 0x0EBC },
-    { 0x0EC8, 0x0ECD }, { 0x0F18, 0x0F19 }, { 0x0F35, 0x0F35 },
-    { 0x0F37, 0x0F37 }, { 0x0F39, 0x0F39 }, { 0x0F71, 0x0F7E },
-    { 0x0F80, 0x0F84 }, { 0x0F86, 0x0F87 }, { 0x0F90, 0x0F97 },
-    { 0x0F99, 0x0FBC }, { 0x0FC6, 0x0FC6 }, { 0x102D, 0x1030 },
-    { 0x1032, 0x1032 }, { 0x1036, 0x1037 }, { 0x1039, 0x1039 },
-    { 0x1058, 0x1059 }, { 0x1160, 0x11FF }, { 0x135F, 0x135F },
-    { 0x1712, 0x1714 }, { 0x1732, 0x1734 }, { 0x1752, 0x1753 },
-    { 0x1772, 0x1773 }, { 0x17B4, 0x17B5 }, { 0x17B7, 0x17BD },
-    { 0x17C6, 0x17C6 }, { 0x17C9, 0x17D3 }, { 0x17DD, 0x17DD },
-    { 0x180B, 0x180D }, { 0x18A9, 0x18A9 }, { 0x1920, 0x1922 },
-    { 0x1927, 0x1928 }, { 0x1932, 0x1932 }, { 0x1939, 0x193B },
-    { 0x1A17, 0x1A18 }, { 0x1B00, 0x1B03 }, { 0x1B34, 0x1B34 },
-    { 0x1B36, 0x1B3A }, { 0x1B3C, 0x1B3C }, { 0x1B42, 0x1B42 },
-    { 0x1B6B, 0x1B73 }, { 0x1DC0, 0x1DCA }, { 0x1DFE, 0x1DFF },
-    { 0x200B, 0x200F }, { 0x202A, 0x202E }, { 0x2060, 0x2063 },
-    { 0x206A, 0x206F }, { 0x20D0, 0x20EF }, { 0x302A, 0x302F },
-    { 0x3099, 0x309A }, { 0xA806, 0xA806 }, { 0xA80B, 0xA80B },
-    { 0xA825, 0xA826 }, { 0xFB1E, 0xFB1E }, { 0xFE00, 0xFE0F },
-    { 0xFE20, 0xFE23 }, { 0xFEFF, 0xFEFF }, { 0xFFF9, 0xFFFB },
-    { 0x10A01, 0x10A03 }, { 0x10A05, 0x10A06 }, { 0x10A0C, 0x10A0F },
-    { 0x10A38, 0x10A3A }, { 0x10A3F, 0x10A3F }, { 0x1D167, 0x1D169 },
-    { 0x1D173, 0x1D182 }, { 0x1D185, 0x1D18B }, { 0x1D1AA, 0x1D1AD },
-    { 0x1D242, 0x1D244 }, { 0xE0001, 0xE0001 }, { 0xE0020, 0xE007F },
-    { 0xE0100, 0xE01EF }
-  };
-
-  /* test for 8-bit control characters */
-  if (ucs == 0)
-    return 0;
-  if (ucs < 32 || (ucs >= 0x7f && ucs < 0xa0))
-    return -1;
-
-  /* binary search in table of non-spacing characters */
-  if (bisearch(ucs, combining,
-	       sizeof(combining) / sizeof(struct interval) - 1))
-    return 0;
-
-  /* if we arrive here, ucs is not a combining or C0/C1 control character */
-
-  return 1 +
-    (ucs >= 0x1100 &&
-     (ucs <= 0x115f ||                    /* Hangul Jamo init. consonants */
-      ucs == 0x2329 || ucs == 0x232a ||
-      (ucs >= 0x2e80 && ucs <= 0xa4cf &&
-       ucs != 0x303f) ||                  /* CJK ... Yi */
-      (ucs >= 0xac00 && ucs <= 0xd7a3) || /* Hangul Syllables */
-      (ucs >= 0xf900 && ucs <= 0xfaff) || /* CJK Compatibility Ideographs */
-      (ucs >= 0xfe10 && ucs <= 0xfe19) || /* Vertical forms */
-      (ucs >= 0xfe30 && ucs <= 0xfe6f) || /* CJK Compatibility Forms */
-      (ucs >= 0xff00 && ucs <= 0xff60) || /* Fullwidth Forms */
-      (ucs >= 0xffe0 && ucs <= 0xffe6) ||
-      (ucs >= 0x20000 && ucs <= 0x2fffd) ||
-      (ucs >= 0x30000 && ucs <= 0x3fffd)));
-}
-
-
-/*
- * The following functions are part of the original wcwidth.c:
- * we don't use them but I've kept them in case - pws.
- */
-#if 0
-int mk_wcswidth(const wchar_t *pwcs, size_t n)
-{
-  int w, width = 0;
-
-  for (;*pwcs && n-- > 0; pwcs++)
-    if ((w = mk_wcwidth(*pwcs)) < 0)
-      return -1;
-    else
-      width += w;
-
-  return width;
-}
-
-
-/*
- * The following functions are the same as mk_wcwidth() and
- * mk_wcswidth(), except that spacing characters in the East Asian
- * Ambiguous (A) category as defined in Unicode Technical Report #11
- * have a column width of 2. This variant might be useful for users of
- * CJK legacy encodings who want to migrate to UCS without changing
- * the traditional terminal character-width behaviour. It is not
- * otherwise recommended for general use.
- */
-int mk_wcwidth_cjk(wchar_t ucs)
+u9_iswprint(wint_t ucs)
 {
-  /* sorted list of non-overlapping intervals of East Asian Ambiguous
-   * characters, generated by "uniset +WIDTH-A -cat=Me -cat=Mn -cat=Cf c" */
-  static const struct interval ambiguous[] = {
-    { 0x00A1, 0x00A1 }, { 0x00A4, 0x00A4 }, { 0x00A7, 0x00A8 },
-    { 0x00AA, 0x00AA }, { 0x00AE, 0x00AE }, { 0x00B0, 0x00B4 },
-    { 0x00B6, 0x00BA }, { 0x00BC, 0x00BF }, { 0x00C6, 0x00C6 },
-    { 0x00D0, 0x00D0 }, { 0x00D7, 0x00D8 }, { 0x00DE, 0x00E1 },
-    { 0x00E6, 0x00E6 }, { 0x00E8, 0x00EA }, { 0x00EC, 0x00ED },
-    { 0x00F0, 0x00F0 }, { 0x00F2, 0x00F3 }, { 0x00F7, 0x00FA },
-    { 0x00FC, 0x00FC }, { 0x00FE, 0x00FE }, { 0x0101, 0x0101 },
-    { 0x0111, 0x0111 }, { 0x0113, 0x0113 }, { 0x011B, 0x011B },
-    { 0x0126, 0x0127 }, { 0x012B, 0x012B }, { 0x0131, 0x0133 },
-    { 0x0138, 0x0138 }, { 0x013F, 0x0142 }, { 0x0144, 0x0144 },
-    { 0x0148, 0x014B }, { 0x014D, 0x014D }, { 0x0152, 0x0153 },
-    { 0x0166, 0x0167 }, { 0x016B, 0x016B }, { 0x01CE, 0x01CE },
-    { 0x01D0, 0x01D0 }, { 0x01D2, 0x01D2 }, { 0x01D4, 0x01D4 },
-    { 0x01D6, 0x01D6 }, { 0x01D8, 0x01D8 }, { 0x01DA, 0x01DA },
-    { 0x01DC, 0x01DC }, { 0x0251, 0x0251 }, { 0x0261, 0x0261 },
-    { 0x02C4, 0x02C4 }, { 0x02C7, 0x02C7 }, { 0x02C9, 0x02CB },
-    { 0x02CD, 0x02CD }, { 0x02D0, 0x02D0 }, { 0x02D8, 0x02DB },
-    { 0x02DD, 0x02DD }, { 0x02DF, 0x02DF }, { 0x0391, 0x03A1 },
-    { 0x03A3, 0x03A9 }, { 0x03B1, 0x03C1 }, { 0x03C3, 0x03C9 },
-    { 0x0401, 0x0401 }, { 0x0410, 0x044F }, { 0x0451, 0x0451 },
-    { 0x2010, 0x2010 }, { 0x2013, 0x2016 }, { 0x2018, 0x2019 },
-    { 0x201C, 0x201D }, { 0x2020, 0x2022 }, { 0x2024, 0x2027 },
-    { 0x2030, 0x2030 }, { 0x2032, 0x2033 }, { 0x2035, 0x2035 },
-    { 0x203B, 0x203B }, { 0x203E, 0x203E }, { 0x2074, 0x2074 },
-    { 0x207F, 0x207F }, { 0x2081, 0x2084 }, { 0x20AC, 0x20AC },
-    { 0x2103, 0x2103 }, { 0x2105, 0x2105 }, { 0x2109, 0x2109 },
-    { 0x2113, 0x2113 }, { 0x2116, 0x2116 }, { 0x2121, 0x2122 },
-    { 0x2126, 0x2126 }, { 0x212B, 0x212B }, { 0x2153, 0x2154 },
-    { 0x215B, 0x215E }, { 0x2160, 0x216B }, { 0x2170, 0x2179 },
-    { 0x2190, 0x2199 }, { 0x21B8, 0x21B9 }, { 0x21D2, 0x21D2 },
-    { 0x21D4, 0x21D4 }, { 0x21E7, 0x21E7 }, { 0x2200, 0x2200 },
-    { 0x2202, 0x2203 }, { 0x2207, 0x2208 }, { 0x220B, 0x220B },
-    { 0x220F, 0x220F }, { 0x2211, 0x2211 }, { 0x2215, 0x2215 },
-    { 0x221A, 0x221A }, { 0x221D, 0x2220 }, { 0x2223, 0x2223 },
-    { 0x2225, 0x2225 }, { 0x2227, 0x222C }, { 0x222E, 0x222E },
-    { 0x2234, 0x2237 }, { 0x223C, 0x223D }, { 0x2248, 0x2248 },
-    { 0x224C, 0x224C }, { 0x2252, 0x2252 }, { 0x2260, 0x2261 },
-    { 0x2264, 0x2267 }, { 0x226A, 0x226B }, { 0x226E, 0x226F },
-    { 0x2282, 0x2283 }, { 0x2286, 0x2287 }, { 0x2295, 0x2295 },
-    { 0x2299, 0x2299 }, { 0x22A5, 0x22A5 }, { 0x22BF, 0x22BF },
-    { 0x2312, 0x2312 }, { 0x2460, 0x24E9 }, { 0x24EB, 0x254B },
-    { 0x2550, 0x2573 }, { 0x2580, 0x258F }, { 0x2592, 0x2595 },
-    { 0x25A0, 0x25A1 }, { 0x25A3, 0x25A9 }, { 0x25B2, 0x25B3 },
-    { 0x25B6, 0x25B7 }, { 0x25BC, 0x25BD }, { 0x25C0, 0x25C1 },
-    { 0x25C6, 0x25C8 }, { 0x25CB, 0x25CB }, { 0x25CE, 0x25D1 },
-    { 0x25E2, 0x25E5 }, { 0x25EF, 0x25EF }, { 0x2605, 0x2606 },
-    { 0x2609, 0x2609 }, { 0x260E, 0x260F }, { 0x2614, 0x2615 },
-    { 0x261C, 0x261C }, { 0x261E, 0x261E }, { 0x2640, 0x2640 },
-    { 0x2642, 0x2642 }, { 0x2660, 0x2661 }, { 0x2663, 0x2665 },
-    { 0x2667, 0x266A }, { 0x266C, 0x266D }, { 0x266F, 0x266F },
-    { 0x273D, 0x273D }, { 0x2776, 0x277F }, { 0xE000, 0xF8FF },
-    { 0xFFFD, 0xFFFD }, { 0xF0000, 0xFFFFD }, { 0x100000, 0x10FFFD }
-  };
-
-  /* binary search in table of non-spacing characters */
-  if (bisearch(ucs, ambiguous,
-	       sizeof(ambiguous) / sizeof(struct interval) - 1))
-    return 2;
-
-  return mk_wcwidth(ucs);
+    if (ucs == 0)
+	return 0;
+    return wcwidth9(ucs) != -1;
 }
 
-
-int mk_wcswidth_cjk(const wchar_t *pwcs, size_t n)
-{
-  int w, width = 0;
-
-  for (;*pwcs && n-- > 0; pwcs++)
-    if ((w = mk_wcwidth_cjk(*pwcs)) < 0)
-      return -1;
-    else
-      width += w;
-
-  return width;
-}
-#endif /* 0 */
-
 /**/
-#endif /* BROKEN_WCWIDTH && (__STDC_ISO_10646__ || __APPLE__) */
+#endif	/* ENABLE_UNICODE9 */
 
 /**/
 #if defined(__APPLE__) && defined(BROKEN_ISPRINT)
diff --git a/Src/pattern.c b/Src/pattern.c
index 75db016..fc7c737 100644
--- a/Src/pattern.c
+++ b/Src/pattern.c
@@ -3625,7 +3625,7 @@ mb_patmatchrange(char *range, wchar_t ch, int zmb_ind, wint_t *indptr, int *mtp)
 		    return 1;
 		break;
 	    case PP_PRINT:
-		if (iswprint(ch))
+		if (WC_ISPRINT(ch))
 		    return 1;
 		break;
 	    case PP_PUNCT:
diff --git a/Src/utils.c b/Src/utils.c
index 5eb936b..acb891d 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -629,7 +629,7 @@ wcs_nicechar_sel(wchar_t c, size_t *widthp, char **swidep, int quotable)
     }
 
     s = buf;
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f) {
 	    if (quotable) {
 		*s++ = '\\';
@@ -734,7 +734,7 @@ wcs_nicechar(wchar_t c, size_t *widthp, char **swidep)
 /**/
 mod_export int is_wcs_nicechar(wchar_t c)
 {
-    if (!iswprint(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
+    if (!WC_ISPRINT(c) && (c < 0x80 || !isset(PRINTEIGHTBIT))) {
 	if (c == 0x7f || c == L'\n' || c == L'\t' || c < 0x20)
 	    return 1;
 	if (c >= 0x80) {
diff --git a/Src/wcwidth9.h b/Src/wcwidth9.h
index 07e6bae..448f548 100644
--- a/Src/wcwidth9.h
+++ b/Src/wcwidth9.h
@@ -22,6 +22,7 @@ static const struct wcwidth9_interval wcwidth9_nonprint[] = {
   {0x070f, 0x070f},
   {0x180b, 0x180e},
   {0x200b, 0x200f},
+  {0x2028, 0x2029},
   {0x202a, 0x202e},
   {0x206a, 0x206f},
   {0xd800, 0xdfff},
@@ -1283,6 +1284,9 @@ static inline bool wcwidth9_intable(const struct wcwidth9_interval *table, size_
 }
 
 static inline int wcwidth9(int c) {
+  if (c == 0) {
+    return 0;
+  }
   if (c < 0|| c > 0x10ffff) {
     return -1;
   }
@@ -1292,7 +1296,7 @@ static inline int wcwidth9(int c) {
   }
 
   if (wcwidth9_intable(wcwidth9_combining, WCWIDTH9_ARRAY_SIZE(wcwidth9_combining), c)) {
-    return -1;
+    return 0;
   }
 
   if (wcwidth9_intable(wcwidth9_not_assigned, WCWIDTH9_ARRAY_SIZE(wcwidth9_not_assigned), c)) {
diff --git a/Src/zsh.h b/Src/zsh.h
index 1093151..22f73f8 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -3158,9 +3158,7 @@ typedef wint_t convchar_t;
  * works on MacOS which doesn't define that.
  */
 #ifdef ENABLE_UNICODE9
-#define WCWIDTH(wc)	mk_wcwidth(wc)
-#elif defined(BROKEN_WCWIDTH) && (defined(__STDC_ISO_10646__) || defined(__APPLE__))
-#define WCWIDTH(wc)	mk_wcwidth(wc)
+#define WCWIDTH(wc)	u9_wcwidth(wc)
 #else
 #define WCWIDTH(wc)	wcwidth(wc)
 #endif
@@ -3205,15 +3203,7 @@ typedef wint_t convchar_t;
  * sense throughout the shell.  I am not aware of a way of
  * detecting the Unicode trait in standard libraries.
  */
-#ifdef BROKEN_WCWIDTH
-/*
- * We can't be quite sure the wcwidth we've provided is entirely
- * in agreement with the system's, so be extra safe.
- */
-#define IS_COMBINING(wc)	(wc != 0 && WCWIDTH(wc) == 0 && !iswcntrl(wc))
-#else
 #define IS_COMBINING(wc)	(wc != 0 && WCWIDTH(wc) == 0)
-#endif
 /*
  * Test for the base of a combining character.
  *
diff --git a/Src/ztype.h b/Src/ztype.h
index 76589b1..ae72367 100644
--- a/Src/ztype.h
+++ b/Src/ztype.h
@@ -72,7 +72,11 @@
 
 #ifdef MULTIBYTE_SUPPORT
 #define WC_ZISTYPE(X,Y) wcsitype((X),(Y))
-#define WC_ISPRINT(X)	iswprint(X)
+# ifdef ENABLE_UNICODE9
+#  define WC_ISPRINT(X)	u9_iswprint(X)
+# else
+#  define WC_ISPRINT(X)	iswprint(X)
+# endif
 #else
 #define WC_ZISTYPE(X,Y)	zistype((X),(Y))
 #define WC_ISPRINT(X)	isprint(X)
diff --git a/configure.ac b/configure.ac
index 911cc45..88da89e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2589,16 +2589,21 @@ AC_HELP_STRING([--enable-unicode9], [compile with unicode9 character widths]),
   AC_DEFINE(ENABLE_UNICODE9)
 fi])
 
-AH_TEMPLATE([BROKEN_WCWIDTH],
-[Define to 1 if the wcwidth() function is present but broken.])
 AH_TEMPLATE([BROKEN_ISPRINT],
 [Define to 1 if the isprint() function is broken under UTF-8 locale.])
+
 if test x$zsh_cv_c_unicode_support = xyes; then
   AC_DEFINE(MULTIBYTE_SUPPORT)
 
-  dnl Test for a wcwidth() implementation that gives the wrong width for
-  dnl zero-width combining characters.
-  dnl For the test we use a combining acute accent (\u0301).
+  dnl Test if wcwidth() and/or iswprint() is broken for
+  dnl   zero-width combining characters, or
+  dnl   some characters in the Latin Extended-B.
+  dnl If either of the functions is broken, both functions will be replaced
+  dnl by the ones from wcwidth9.h by defining ENABLE_UNICODE9. We will do
+  dnl this only if __STDC_ISO_10646__ is defined (or if building on macOS,
+  dnl where __STDC_ISO_10646__ is not defined but wchar_t is UCS).
+  dnl For the test we use a combining acute accent (\u0301) or
+  dnl a LATIN SMALL LETTER L WITH CURL (\u0234).
   dnl We input it as UTF-8 since that is the standard we can rely
   dnl upon most:  we can't rely on a wchar_t being stored as a
   dnl Unicode code point on all systems.
@@ -2607,9 +2612,8 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   dnl - the programme compiled, linked and ran
   dnl - we successfully set a UTF-8 locale
   dnl - the locale we set plausibly converted the UTF-8 string
-  dnl   for a zero-width combining character (the only way to be
-  dnl   100% sure would be to output it and ask if it looked right)
-  dnl - the converted wide character gave a non-zero width.
+  dnl   into the correct wide character
+  dnl - but wcwidth() or iswprint() is broken for the converted wide character.
   dnl locale -a is a fallback; on most systems we should find en_US.UTF-8.
   [locale_prog='char *my_locales[] = {
   "en_US.UTF-8", "en_GB.UTF-8", "en.UTF-8", '
@@ -2621,32 +2625,38 @@ if test x$zsh_cv_c_unicode_support = xyes; then
   #include <stdlib.h>
   #include <locale.h>
   #include <wchar.h>
+  #include <wctype.h>
 
   int main() {
     char **localep;
     char comb_acute_mb[] = { (char)0xcc, (char)0x81 };
+    char u_0234[] = { (char)0xc8, (char)0xb4 };
     wchar_t wc;
+  #if !defined(__STDC_ISO_10646__) && !defined(__APPLE__)
+    return 1;
+  #endif
 
     for (localep = my_locales; *localep; localep++)
-      if (setlocale(LC_ALL, *localep) &&
-          mbtowc(&wc, comb_acute_mb, 2) == 2)
+      if (setlocale(LC_ALL, *localep))
 	  break;
     if (!*localep)
       return 1;
-    if (wcwidth(wc) == 0)
-      return 1;
-    return 0;
+    if (mbtowc(&wc, comb_acute_mb, 2) == 2 && (wcwidth(wc) != 0 || !iswprint(wc)))
+      return 0;
+    if (mbtowc(&wc, u_0234, 2) == 2 && (wcwidth(wc) != 1 || !iswprint(wc)))
+      return 0;
+    return 1;
   }
   "]
 
-  AC_CACHE_CHECK(if the wcwidth() function is broken,
+  AC_CACHE_CHECK(if the wcwidth() and/or iswprint() functions are broken,
   zsh_cv_c_broken_wcwidth,
   [AC_TRY_RUN([$locale_prog],
   zsh_cv_c_broken_wcwidth=yes,
   zsh_cv_c_broken_wcwidth=no,
   zsh_cv_c_broken_wcwidth=no)])
   if test x$zsh_cv_c_broken_wcwidth = xyes; then
-    AC_DEFINE(BROKEN_WCWIDTH)
+    AC_DEFINE(ENABLE_UNICODE9)
   fi
 
   dnl Check if isprint() behaves correctly under UTF-8 locale.

Re: Cannot paste unicode <0221>, <0234> - <024f>

[Sebastian Gniazdowski]

On 10 maja 2017 at 14:13:00, Jun T. (takimoto-j@kba.biglobe.ne.jp) wrote:
>  
> On 2017/05/02, at 0:52, Jun T. wrote:
> >
> > Another possibility is to use --enable-unicode9 if wcwidth()
> > and/or iswprint() are broken
>  
> I had some discussion with Joshua Rubin and concluded that this
> is the way to go.
>  
> In the patch below (against the current master), if wcwidth() OR
> iswprint() is broken then BOTH will be replaced by the ones derived
> from wcwidth9.h.
> (the patch is also attached as a file u9-diff.txt)

Tested current head (patch is merged), I get:

configure: all functions found, multibyte support enabled
checking if the wcwidth() and/or iswprint() functions are broken... yes
checking if the isprint() function is broken... yes

and pasting of the 1024 Unicode letters works.

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org