Inconsistent history expansion of characters adjacent to histchar

Inconsistent history expansion of characters adjacent to histchar

[Hauke Petersen]

zshexpn(1), heavily elided:

    HISTORY EXPANSION

    Following [the exclamation mark] is an optional event designator and then
    an optional word designator; _if neither of these designators is present,
    no history expansion occurs._

AFAICT, that's only partially true.

    % print !
    !
    % print !; print foo
    print print !; print foo
    print !
    foo

It does not immediately make sense to me why a single `!' would be
interpreted as a double `!!' before a `;' and tried, unsuccessfully,
to find an explanation.

    Event Designators
    !   Start a history expansion, except when followed by a blank, new-line,
    `=' or `('.  If followed immediately by a word designator, this forms a
    history reference with no event designator.

    [!!, !n, !-n, !str, !?str[?], !#, !{...}]

There is also this:

    a history reference with no event designator refers to the same event as
    any preceding history reference on that command line; if it is the only
    history reference in a command, it refers to the previous command

but that only applies to references with a word designator, which
there isn't in `!;'.  Nor does it satisfy the definition of any of the
event designators except `!str', but then it would a) eat the `;' as
being part of the reference, and b) fail with `event not found: ;' if
there is no previous command starting with ';'.  Like otherwise
similar `&':

    % print !& print foo
    zsh: event not found: &

Instead, as above, it expands like `!!;' which every once a while
makes my heart skip two beats when I realize what could have happened
in lines like the following with a previous command other than `:'.

    % :
    % ./foo & touch foo.$!; wait $!; rm foo.$!; printf \\a
    ./foo & touch foo.$:; wait $:; rm foo.$:; printf \\a
    [1] 3828
    wait: job not found: $:

I know expansion is always potentially destructive and, yes,
HIST_VERIFY could prevent the worst of it, but I do not think the
current behavior makes sense in the first place.  I'd expect and
prefer `!;'  not to be interpreted as a history reference, but seeing
the behavior of `!&', I think it should at least behave consistently.

(Sorry about the verbosity.)

Re: Inconsistent history expansion of characters adjacent to histchar

[Bart Schaefer]

On Mon, Oct 7, 2013 at 8:55 AM, Hauke Petersen <hkptrsn@gmail.com> wrote:

> It does not immediately make sense to me why a single `!' would be
> interpreted as a double `!!' before a `;' and tried, unsuccessfully,
> to find an explanation.

Bluntly, it's because that history documentation is lying to you. The
correct bit is the first sentence here:

>     Event Designators
>     !   Start a history expansion, except when followed by a blank, new-line,
>     `=' or `('.  If followed immediately by a word designator, this forms a
>     history reference with no event designator.

Semicolon is not one of the things that prevents a "!" from starting
a history expansion, so everything from there forward behaves like a
history reference with no event.

This bug appears to have been around for longer than we've been keeping
source control copies of the shell. For some reason ';' is called out
in histsubchar() along with things that begin (or end) event or word
designators. The same is true of any kind of quotes, e.g.:

    print !`echo foo`
    print !!`echo foo`

are equivalent; "!}" also has the same problem though it usually causes
a parse error.

I think the intent is that those characters only end a history reference
after at least one other character [not in that set] has been read, but
that's not the way the loop is written, and it doesn't account for other
characters that can't be word designators, as seen here:

> there is no previous command starting with ';'.  Like otherwise
> similar `&':
>
>     % print !& print foo
>     zsh: event not found: &

Same happens with "!|" and probably a few other obscure variations.

In short, the history parser has always been very ad-hoc and deviates
from its own spec in several ways.

Here's a patch that fixes the unintended expansion, but it doesn't fix
the inconsistent behavior with !& !| et al.  There should probably be
more method to this madness instead of multiple series of individual
character comparisons.

The (c == '"') condition is probably unnecessary because '!"' has been
handled much earlier, but the loop above this also tests all three
quotes, so ...  

diff --git a/Src/hist.c b/Src/hist.c
index d1af30a..bd650e8 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -521,6 +521,12 @@ histsubchar(int c)
 		}
 		c = ingetc();
 	    }
+	    if (ptr == buf &&
+		(c == '}' ||  c == ';' || c == '\'' || c == '"' || c == '`')) {
+	      /* Neither event nor word designator, no expansion */
+	      safeinungetc(c);
+	      return bangchar;
+	    }
 	    *ptr = 0;
 	    if (!*buf) {
 		if (c != '%') {

Re: Inconsistent history expansion of characters adjacent to histchar

[Bart Schaefer]

On Oct 7,  7:56pm, Bart Schaefer wrote:
}
} In short, the history parser has always been very ad-hoc and deviates
} from its own spec in several ways.
} 
} Here's a patch that fixes the unintended expansion, but it doesn't fix
} the inconsistent behavior with !& !| et al.  There should probably be
} more method to this madness instead of multiple series of individual
} character comparisons.

Just to illustrate this further:

torch% print !\; print foo                
zsh: event not found: \
torch% \print bar
bar
torch% print !\; print foo                
print \print bar; print foo
print bar
foo
torch% 

I'm wondering if backslash ought to be treated specially in more of the
history code.  In particular:

torch% print !\!; print foo       
zsh: event not found: \!
torch% 

One might think one could use that form to search for a command line
that begins with a literal exclamation point, but no.

Re: Inconsistent history expansion of characters adjacent to histchar

[Peter Stephenson]

On Tue, 08 Oct 2013 07:31:25 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm wondering if backslash ought to be treated specially in more of the
> history code.  In particular:
>
> torch% print !\!; print foo       
> zsh: event not found: \!
> torch% 
> 
> One might think one could use that form to search for a command line
> that begins with a literal exclamation point, but no.

Hmm... the documentation does indeed imply backslash is some sort of a
quote for history.  Actually, I think it's just talking about escaping
the initial ! and then goes into its usual "it sort of vaguely works
like this but nobody ever really thought it through from beginning to
end" mode.  However, you might think you were morally entitled to have a
backslash at that point either isolate the backslashed expression from
what went before (so it behaves just like a single !), or to be treated
as a string argument, and I would think the second, the one you
mentioned, is rather more useful.

pws

Re: Inconsistent history expansion of characters adjacent to histchar

[Peter Stephenson]

On Tue, 08 Oct 2013 15:44:04 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Tue, 08 Oct 2013 07:31:25 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > I'm wondering if backslash ought to be treated specially in more of the
> > history code.  In particular:
> >
> > torch% print !\!; print foo       
> > zsh: event not found: \!
> > torch% 
> > 
> > One might think one could use that form to search for a command line
> > that begins with a literal exclamation point, but no.
> 
> Hmm... the documentation does indeed imply backslash is some sort of a
> quote for history.  Actually, I think it's just talking about escaping
> the initial ! and then goes into its usual "it sort of vaguely works
> like this but nobody ever really thought it through from beginning to
> end" mode.  However, you might think you were morally entitled to have a
> backslash at that point either isolate the backslashed expression from
> what went before (so it behaves just like a single !), or to be treated
> as a string argument, and I would think the second, the one you
> mentioned, is rather more useful.

I put this badly --- I meant, of course, the "!" should be treated as a
string argument, not the "\".

pws