'patch -Rp0<!$' loses the < in expansion

'patch -Rp0<!$' loses the < in expansion

[Daniel Shahaf]

[[[
% echo >foo
% : -Rp0<!$
: -Rp0foo
% echo >foo
% : -Rp0 <!$
: -Rp0 <foo
]]]

This reproduces using 4.3.10 (me) and HEAD (Frank Terbeck).

Shouldn't the < be retained?

Re: 'patch -Rp0<!$' loses the < in expansion

[Bart Schaefer]

On Aug 16, 12:46pm, Daniel Shahaf wrote:
} Subject: 'patch -Rp0<!$' loses the < in expansion
}
} [[[
} % echo >foo
} % : -Rp0<!$
} : -Rp0foo
} % echo >foo
} % : -Rp0 <!$
} : -Rp0 <foo
} ]]]
} 
} This reproduces using 4.3.10 (me) and HEAD (Frank Terbeck).

It reproduces back to 4.2.0 at least.

} Shouldn't the < be retained?

Presumably.

What's going on here is that the lexer is looking ahead to discover
whether "<" is introducing a numeric glob instead of a redirection.  
It actually tracks back and forth across "<f" a couple of times before
deciding it's safe to move ahead to "o".  Right up until the point
where Src/hist.c:ihwaddc('o') is called, the history line (chline in
the C code) looks fine; somewhere in processing "o" it backtracks too
far and overwrites "<f" with "fo".

But only in the history, not in the lexer state; so the redirection
is correctly processed and except for a garbled history entry it all
works fine.

Here's the first spot where hptr backs up to the "<":

(gdb) where
#0  ihungetc (c=60) at ../../zsh-4.0/Src/hist.c:785
#1  0x080871cb in gettokstr (c=60, sub=0) at ../../zsh-4.0/Src/lex.c:1435
#2  0x0808652e in gettok () at ../../zsh-4.0/Src/lex.c:1001
#3  0x08085344 in zshlex () at ../../zsh-4.0/Src/lex.c:395
#4  0x080a3620 in par_simple (complex=0xbff8a45c, nr=0)
    at ../../zsh-4.0/Src/parse.c:1678
#5  0x080a15ed in par_cmd (complex=0xbff8a45c) at ../../zsh-4.0/Src/parse.c:879
#6  0x080a104b in par_pline (complex=0xbff8a45c)
    at ../../zsh-4.0/Src/parse.c:728
#7  0x080a0ff8 in par_sublist2 (complex=0xbff8a45c)
    at ../../zsh-4.0/Src/parse.c:709
#8  0x080a0e8b in par_sublist (complex=0xbff8a488)
    at ../../zsh-4.0/Src/parse.c:664
#9  0x080a0910 in par_event () at ../../zsh-4.0/Src/parse.c:477
#10 0x080a0886 in parse_event () at ../../zsh-4.0/Src/parse.c:454
#11 0x0807aa68 in loop (toplevel=1, justonce=0) at ../../zsh-4.0/Src/init.c:132
#12 0x0807da85 in zsh_main (argc=2, argv=0xbff8a5d4)
    at ../../zsh-4.0/Src/init.c:1528

It appears to go forward from there OK, but then backs up again here:

(gdb) where
#0  0x08076d57 in ihwend () at ../../zsh-4.0/Src/hist.c:1362
#1  0x08087def in exalias () at ../../zsh-4.0/Src/lex.c:1801
#2  0x08085357 in zshlex () at ../../zsh-4.0/Src/lex.c:395
#3  0x080a3620 in par_simple (complex=0xbff8a45c, nr=0)
    at ../../zsh-4.0/Src/parse.c:1678
#4  0x080a15ed in par_cmd (complex=0xbff8a45c) at ../../zsh-4.0/Src/parse.c:879
#5  0x080a104b in par_pline (complex=0xbff8a45c)
    at ../../zsh-4.0/Src/parse.c:728
#6  0x080a0ff8 in par_sublist2 (complex=0xbff8a45c)
    at ../../zsh-4.0/Src/parse.c:709
#7  0x080a0e8b in par_sublist (complex=0xbff8a488)
    at ../../zsh-4.0/Src/parse.c:664
#8  0x080a0910 in par_event () at ../../zsh-4.0/Src/parse.c:477
#9  0x080a0886 in parse_event () at ../../zsh-4.0/Src/parse.c:454
#10 0x0807aa68 in loop (toplevel=1, justonce=0) at ../../zsh-4.0/Src/init.c:132
#11 0x0807da85 in zsh_main (argc=2, argv=0xbff8a5d4)
    at ../../zsh-4.0/Src/init.c:1528

That's this:

1358                if (hwgetword > -1) {
1359                    /* We want to reuse the current word position */
1360                    chwordpos = hwgetword;
1361                    /* Start from where previous word ended, if possible */
1362                    hptr = chline + chwords[chwordpos ? chwordpos - 1 : 0];
1363                }

This makes me suspect that hwgetword was not incremented at some place
where it should have been, but I don't know where -- or whether that is
really the problem.

Re: 'patch -Rp0<!$' loses the < in expansion

[Bart Schaefer]

On Aug 16,  8:36pm, Bart Schaefer wrote:
}
} 1358                if (hwgetword > -1) {
} 1359                    /* We want to reuse the current word position */
} 1360                    chwordpos = hwgetword;
} 1361                    /* Start from where previous word ended, if possible */
} 1362                    hptr = chline + chwords[chwordpos ? chwordpos - 1 : 0];
} 1363                }
} 
} This makes me suspect that hwgetword was not incremented at some place
} where it should have been, but I don't know where -- or whether that is
} really the problem.

I just put a debug statement in there and ran "make check" and that
particular code branch is never excercised.

Immediately before that, there's this:

1350            /* end of word reached and we've already begun a word */
1351            if (hptr > chline + chwords[chwordpos-1]) {
1352                chwords[chwordpos++] = hptr - chline;

Everything is fine at this point.  We've got chwords[4] pointing at "<"
and chwords[5] (which represents the end of that word) pointing at "f".
We should be ready to start the next word.

So why do we want to do the stuff at line 1360 & 1361?  What is the
point of "reuse the current word position"?  I suspect the answer is
up in ihwbegin():

1331	    /* If we're expanding an alias, we should overwrite the expansion
1332	     * in the history.
1333	     */
1334	    if ((inbufflags & INP_ALIAS) && !(inbufflags & INP_HIST))
1335		hwgetword = chwordpos;
1336	    else
1337		hwgetword = -1;

And sure enough:

torch% alias -g foo=bar
torch% echo foo
 ../../zsh-4.0/Src/hist.c:1359: Re-using current word position
bar

And:

torch% : -Rp0<!$
 ../../zsh-4.0/Src/hist.c:1359: Re-using current word position
 ../../zsh-4.0/Src/hist.c:1359: Re-using current word position
: -Rp0foo
torch% 

It's possible that what hwgetword should be getting reset to -1 somewhere
that it is not, but I on a few tracethroughs I can't find anyplace else
where we detect that we've transitioned lexer words in the middle of a
contiguous string without separators.  So without much finesse:

--- ../zsh-forge/current/Src/hist.c	2011-08-14 11:12:30.000000000 -0700
+++ Src/hist.c	2011-08-16 23:20:35.000000000 -0700
@@ -1355,7 +1355,8 @@
 					    (chwordlen += 32) * 
 					    sizeof(short));
 	    }
-	    if (hwgetword > -1) {
+	    if (hwgetword > -1 &&
+		(inbufflags & INP_ALIAS) && !(inbufflags & INP_HIST)) {
 		/* We want to reuse the current word position */
 		chwordpos = hwgetword;
 		/* Start from where previous word ended, if possible */

It's possible that we also want to assign hwgetword = -1 at this point,
but I think that would be wrong in a recursively-expanding alias.

Re: 'patch -Rp0<!$' loses the < in expansion

[Peter Stephenson]

On Tue, 16 Aug 2011 23:26:16 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> It's possible that what hwgetword should be getting reset to -1 somewhere
> that it is not, but I on a few tracethroughs I can't find anyplace else
> where we detect that we've transitioned lexer words in the middle of a
> contiguous string without separators.

I couldn't see any obvious logical place to reset it, either.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog