PATCH: (2): Bug#255788: $'' does not work after <<<

[Peter Stephenson <pws@csr.com>]

OK, it's Monday morning and I spent the entire weekend thinking about
book rewrites and tax returns, so...

Here's a second go, with additions to the test suite.  Since I'm now
very confused, here's an exhausting (sic) list of changes.

Here documents: Nothing changed in the body of the here document.  In
parsing the word after <<, handle all forms of quoting.  ' is unchanged;
" unchanged; ` now does the same thing as double quotes, which means \\
gets turned into \, etc.; backquotes don't do command substitution ---
they never did, but they wouldn't have been stripped properly.  $'...'
now has its standard effect.  (We should probably do that during the
lexical analysis phase, which corresponds to Bart's last point.
However, it seems to be working using the new function quotesubst(), so
I haven't looked at this.)

Here strings: They sort of did expansion before (and this was
documented), but it was a bit botched because the string was massaged to
remove quotes too early.  (The dupstring() has now gone along with the
remnulargs(), for those who care about such things.)  So the test in
A04redirect.ztst with a double-quoted $foo worked but $'...' didn't
work.  This should be more consistent, doing zsh's standard forced
single-word substitution (make sure arrays don't turn it into multiple
words, etc.).  I still don't see any reason why this would be the wrong
thing to do.

Index: Doc/Zsh/redirect.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/redirect.yo,v
retrieving revision 1.6
diff -u -r1.6 redirect.yo
--- Doc/Zsh/redirect.yo	2 May 2003 10:25:32 -0000	1.6
+++ Doc/Zsh/redirect.yo	28 Jun 2004 11:36:26 -0000
@@ -70,12 +70,20 @@
 and `tt(\)' must be used to quote the characters
 `tt(\)', `tt($)', `tt(`)' and the first character of var(word).
 
+Note that var(word) itself does not undergo shell expansion.
+Backquotes in var(word) do not have their usual effect, and
+behave similarly to double quotes.  Quotes in the
+form tt($')var(...)tt(') have their standard effect of expanding
+backslashed references to special characters.
+
 If tt(<<-) is used, then all leading
 tabs are stripped from var(word) and from the document.
 )
 item(tt(<<<) var(word))(
 Perform shell expansion on var(word) and pass the result
 to standard input.  This is known as a em(here-string).
+Compare the use of var(word) in here-documents above, where var(word)
+does not undergo shell expansion.
 )
 xitem(tt(<&) var(number))
 item(tt(>&) var(number))(
Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.65
diff -u -r1.65 exec.c
--- Src/exec.c	22 Jun 2004 13:10:02 -0000	1.65
+++ Src/exec.c	28 Jun 2004 11:36:27 -0000
@@ -2714,9 +2714,10 @@
 
     for (s = str; *s; s++)
 	if (INULL(*s)) {
-	    *s = Nularg;
 	    qt = 1;
+	    break;
 	}
+    quotesubst(str);
     untokenize(str);
     if (typ == REDIR_HEREDOCDASH) {
 	strip = 1;
Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.45
diff -u -r1.45 parse.c
--- Src/parse.c	22 Jun 2004 13:10:02 -0000	1.45
+++ Src/parse.c	28 Jun 2004 11:36:28 -0000
@@ -1754,9 +1754,6 @@
 	if ((tokstr[0] == Inang || tokstr[0] == Outang) && tokstr[1] == Inpar)
 	    type = tokstr[0] == Inang ? REDIR_INPIPE : REDIR_OUTPIPE;
 	break;
-    case REDIR_HERESTR:
-        remnulargs(name = dupstring(name));
-        break;
     }
     yylex();
 
Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.38
diff -u -r1.38 subst.c
--- Src/subst.c	2 Jun 2004 22:14:26 -0000	1.38
+++ Src/subst.c	28 Jun 2004 11:36:29 -0000
@@ -231,6 +231,37 @@
     return errflag ? NULL : node;
 }
 
+/*
+ * Simplified version of the prefork/singsub processing where
+ * we only do substitutions appropriate to quoting.  Currently
+ * this means only the expansions in $'....'.  This is used
+ * for the end tag for here documents.  As we are not doing
+ * `...` expansions, we just use those for quoting and removing
+ * them from the text (they do not get removed by remnulargs()).
+ *
+ * The remnulargs() makes this consistent with the other forms
+ * of substitution, indicating that quotes have been fully
+ * processed.
+ */
+
+/**/
+void
+quotesubst(char *str)
+{
+    char *s = str;
+
+    while (*s) {
+	if (*s == String && s[1] == Snull) {
+	    s = getkeystring(s, NULL, 4, NULL);
+	} else if (*s == Tick || *s == Qtick) {
+	    chuck(s);
+	} else {
+	    s++;
+	}
+    }
+    remnulargs(str);
+}
+
 /**/
 mod_export void
 globlist(LinkList list, int nountok)
Index: Test/A04redirect.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A04redirect.ztst,v
retrieving revision 1.4
diff -u -r1.4 A04redirect.ztst
--- Test/A04redirect.ztst	2 May 2003 10:25:33 -0000	1.4
+++ Test/A04redirect.ztst	28 Jun 2004 11:36:29 -0000
@@ -50,10 +50,34 @@
 0:Here-documents stripping tabs
 >barbar
 
+  cat <<-$'$HERE '`$(THERE) `'$((AND)) '"\EVERYWHERE"
+# tabs again.  sorry about the max miller.
+	Here's a funny thing.  Here is a funny thing.
+	I went home last night.  There's a funny thing.
+	Man walks into a $foo.  Ouch, it's an iron $foo.
+	$HERE $(THERE) $((AND)) \EVERYWHERE
+0:Here-documents don't perform shell expansion on the initial word
+>Here's a funny thing.  Here is a funny thing.
+>I went home last night.  There's a funny thing.
+>Man walks into a $foo.  Ouch, it's an iron $foo.
+
+  cat <<-$'\x45\x4e\x44\t\x44\x4f\x43'
+# tabs again
+	This message is unfathomable.
+	END	DOC
+0:Here-documents do perform $'...' expansion on the initial word
+>This message is unfathomable.
+
   cat <<<"This is a line with a $foo in it"
 0:'<<<' redirection
 >This is a line with a bar in it
 
+  cat <<<$'a\nb\nc'
+0:here-strings with $'...' quoting
+>a
+>b
+>c
+
   exec 3>redir  &&  print hello >&3  &&  print goodbye >&3  && cat redir
 0:'>&' redirection
 >hello

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************