[BUG] E02 test failure / ztst *> issue

[BUG] E02 test failure / ztst *> issue

[dana]

When i run E02 on master, on macOS, i get an failure for the new 'preserves
tracing' test (see output below). On my machine, `which` outputs the name of
the ヌ function without quoting.

I was going to just change it like this:

  ->$'\M-c\M-\C-C\M-\C-L' () {
  +*>(ヌ|$'\M-c\M-\C-C\M-\C-L') \(\) {

But that doesn't work the way i expected — it breaks the surrounding lines.
The documentation (B01) says this about `*>`:

  # Each line of a '>' and '?' chunk may be preceded by a '*', so the line
  # starts '*>' or '*?'.  This signifies that for any line with '*' in front
  # the actual output should be pattern matched against the corresponding
  # lines in the test output.

In actuality, if *any* `>` line is preceded by `*`, *all* of them are treated
as patterns to be matched against the corresponding lines in the output. Thus,
if the surrounding lines contain pattern meta-characters (which they do), they
need to be escaped to work correctly.

I'm not sure if the documentation is wrong or if the behaviour is. It seems
like it would be helpful if it worked the way it said, but it looks like we'd
have to pass pattern-containing line numbers or something to ZTST_diff for it
to do that. idk

dana


--- /tmp/zsh.ztst.28982/ztst.out	2020-03-11 15:57:36.000000000 -0500
+++ /tmp/zsh.ztst.28982/ztst.tout	2020-03-11 15:57:36.000000000 -0500
@@ -6,7 +6,7 @@
 	# traced
 	echo inner
 }
-$'\M-c\M-\C-C\M-\C-L' () {
+ヌ () {
 	# traced
 	echo inner
 }
Test ./E02xtrace.ztst failed: output differs from expected as shown above for:
  test_cases=(
      f            # baseline
      foo-bar      # Dash
      ヌ           # Meta (the UTF-8 representation of this character has an 0x83 byte)
      \$\'ba\\0z\' # Nul, escaped as though by ${(qqqq)}
  )
  for 1 in "$test_cases[@]"; do
    eval "
      ${1}() {
        ${1}() { echo inner }
      }
      functions -T ${1}
      ${1}
      which ${1}
    "
  done
Was testing: a function that redefines itself preserves tracing

Re: [BUG] E02 test failure / ztst *> issue

[Jun T]

> 2020/03/12 6:15, dana <dana@dana.is> wrote:
> 
> When i run E02 on master, on macOS, i get an failure for the new 'preserves
> tracing' test (see output below). On my machine, `which` outputs the name of
> the ヌ function without quoting.

I noticed this on macOS few days ago or so, and was testing on other BSDs,
and yes, the test fails (at least) on Free/Net/Open BSDs. It turned out
that the problem is in zsh itself; a patch is at the end of this post.


The test is run under C-locale, but on BSDs "which" outputs the function
name ヌ (utdf-8: e3 83 8c) as raw bytes instead of escaping by \M and \C.
On Linux the test succeeds.

The function name is printed by quotedzputs(), which calls
is_mb_niceformat(), and mbrtowc() (line 5422, utils.c). Under C locale,
mbrtowc() behaves differently on Linux and BSDs.

On Linux, mbrtowc(&c, 0xe3;0x83;0x8c, ..) returns MB_INVALID if called
under C locale. So the 1st byte 0xe3 is an invalid byte.

On BSDs it returns 1, indicating that 0xe3 is a valid single byte
character (but not printable because isprint(0xe3) is 0).

But this should not make any difference, because zsh is *expected* to
print either invalid or unprintable byte by escaping with \M- and \C-
(by using one of nicechar-family functions).

The real problem is in is_wcs_nicechar() that is called from
is_mb_niceformat() (line 5452, utils.c); is_mb_niceformat(0xe3)
should return 1 but actually returns 0. The 2nd hunk in the path below
fixes this. But only with this hunk, the function name is output as
$'\M-c\M-^C\M-^L'
i.e., ^X instead of \C-X. The first hunk in the patch fixes this.



diff --git a/Src/utils.c b/Src/utils.c
index f9c2d4a2b..5ffc459f2 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -711,7 +711,7 @@ wcs_nicechar_sel(wchar_t c, size_t *widthp, char **swidep, int quotable)
 	    if (widthp)
 		*widthp = 6;
 	} else {
-	    strcpy(buf, nicechar((int)c));
+	    strcpy(buf, nicechar_sel((int)c, quotable));
 	    /*
 	     * There may be metafied characters from nicechar(),
 	     * so compute width and end position independently.
@@ -771,7 +771,7 @@ mod_export int is_wcs_nicechar(wchar_t c)
 	if (c == 0x7f || c == L'\n' || c == L'\t' || c < 0x20)
 	    return 1;
 	if (c >= 0x80) {
-	    return (c >= 0x100);
+	    return (c >= 0x100 || is_nicechar(c));
 	}
     }
     return 0;

Re: [BUG] E02 test failure / ztst *> issue

[dana]

On 11 Mar 2020, at 21:37, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
> I noticed this on macOS few days ago or so, and was testing on other BSDs,
> and yes, the test fails (at least) on Free/Net/Open BSDs. It turned out
> that the problem is in zsh itself; a patch is at the end of this post.

Oh, i'd just assumed it was a normal platform/locale difference.

On Mojave, your patch fixes the test for me only if i manually set LC_CTYPE or
LC_ALL to C when running `make check` — which didn't have any effect before,
so it is definitely an improvement. If i run it normally (where i have
LC_CTYPE=en_GB.UTF-8), it still fails in the same way.

We could set LC_CTYPE=C in ztst, but that'd cause problems for some of the
multi-byte tests. So should we just set it for this one test? If i combine
this patch with yours, everything passes for me.

dana


diff --git a/Test/E02xtrace.ztst b/Test/E02xtrace.ztst
index a5a7bc55c..77088001d 100644
--- a/Test/E02xtrace.ztst
+++ b/Test/E02xtrace.ztst
@@ -147,22 +147,25 @@
 ?+(anon):0> true
 ?+fn:0> gn
 
-  test_cases=(
-      f            # baseline
-      foo-bar      # Dash
-      ヌ           # Meta (the UTF-8 representation of this character has an 0x83 byte)
-      \$\'ba\\0z\' # Nul, escaped as though by ${(qqqq)}
+  (
+    LC_ALL=C # Make `which` output consistent
+    test_cases=(
+        f            # baseline
+        foo-bar      # Dash
+        ヌ           # Meta (the UTF-8 representation of this character has an 0x83 byte)
+        \$\'ba\\0z\' # Nul, escaped as though by ${(qqqq)}
+    )
+    for 1 in "$test_cases[@]"; do
+      eval "
+        ${1}() {
+          ${1}() { echo inner }
+        }
+        functions -T ${1}
+        ${1}
+        which ${1}
+      "
+    done
   )
-  for 1 in "$test_cases[@]"; do
-    eval "
-      ${1}() {
-        ${1}() { echo inner }
-      }
-      functions -T ${1}
-      ${1}
-      which ${1}
-    "
-  done
 0:a function that redefines itself preserves tracing
 >f () {
 >  # traced

Re: [BUG] E02 test failure / ztst *> issue

[Peter Stephenson]

On Thu, 2020-03-12 at 12:31 -0500, dana wrote:
> We could set LC_CTYPE=C in ztst, but that'd cause problems for some of the
> multi-byte tests. So should we just set it for this one test? If i combine
> this patch with yours, everything passes for me.

Yes, I think that's reasonable.  It's just this one that's making some
assumption that things won't get *too* clever.

pws

Re: [BUG] E02 test failure / ztst *> issue

[Jun T]

> 2020/03/13 2:31, dana <dana@dana.is> wrote:
> 
> On 11 Mar 2020, at 21:37, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
> 
> On Mojave, your patch fixes the test for me only if i manually set LC_CTYPE or
> LC_ALL to C when running `make check` — which didn't have any effect before,
> so it is definitely an improvement. If i run it normally (where i have
> LC_CTYPE=en_GB.UTF-8), it still fails in the same way.

ztst.zsh (lines 27-31) resets LC_* and LANG to C if they are non-empty.
So I was thinking tests were run under C locale by default ...
But now I've noticed LC_CTYPE is NOT reset here.

I guess you can avoid subushell by

LC_ALL=C which ${1}

Re: [BUG] E02 test failure / ztst *> issue

[dana]

On 13 Mar 2020, at 05:08, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
> I guess you can avoid subushell by
> LC_ALL=C which ${1}

Yeah, and it's a cleaner diff. Committed like that, thanks

dana