!!$ unitialized at first prompt

!!$ unitialized at first prompt

[Benjamin R. Haskell]

Experiencing bizarre behavior when the first thing I do in the shell is 
to call expand-or-complete on !$.

If the value of !$ contains something from $IFS, it expands improperly 
during the first command to just the last word.

# get something into history that has a space-y last arg
first-shell$ echo 'a b'
a b
first-shell$ <ctrl-d> to logout

# first noticed the weird tab-completion
second-shell$ echo !$<tab>
zsh: do you want to see all 294 possibilities (49 lines)?

# but it also doesn't expand to the correct value:
# (without an intervening command)
second-shell$ echo x!{$}y
quote><ctrl-c>
echo xb'y

I first thought it was related to having just added csh_junkie_history 
to my preferred options, but it's not.  Reproducible with 
HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f.  Seems like 
something's not being initialized in the correct order.

-- 
Best,
Ben

Re: !!$ unitialized at first prompt

[Benjamin R. Haskell]

On Fri, 24 Sep 2010, Benjamin R. Haskell wrote:

> # but it also doesn't expand to the correct value:
> # (without an intervening command)
> second-shell$ echo x!{$}y
> quote><ctrl-c>
> echo xb'y
>

The words are definitely split wrong.  Not just '$':

first-shell$ echo 'a b' 'c d'
a b c d
first-shell$ <ctrl-d>

second-shell$ !!0-2
echo 'a b'
a b

> [...]
> Reproducible with HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f.
                                                                     ^^
('-f' should be '-d' -- took me a while to figure out why -f was 
preventing history from being saved -- finally found the mention in the 
manual.)

-- 
Best,
Ben

Re: !!$ unitialized at first prompt

[Mikael Magnusson]

On 24 September 2010 06:43, Benjamin R. Haskell <zsh@benizi.com> wrote:
> On Fri, 24 Sep 2010, Benjamin R. Haskell wrote:
>
>> # but it also doesn't expand to the correct value:
>> # (without an intervening command)
>> second-shell$ echo x!{$}y
>> quote><ctrl-c>
>> echo xb'y
>>
>
> The words are definitely split wrong.  Not just '$':
>
> first-shell$ echo 'a b' 'c d'
> a b c d
> first-shell$ <ctrl-d>
>
> second-shell$ !!0-2
> echo 'a b'
> a b
>
>> [...]
>> Reproducible with HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f.

Could this be related to
http://thread.gmane.org/gmane.comp.shells.zsh.devel/20904 ? That also
only happens when reloading history.

-- 
Mikael Magnusson

Re: !!$ unitialized at first prompt

[Peter Stephenson]

The shell doesn't split words in the same way when importing history.
In normal operation it relies on the lines having been passed through
the lexical analyser to generate the words, which doesn't happen when
history is read from a file.  Instead it just blindly splits on
whitespace.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: !!$ unitialized at first prompt

[Benjamin R. Haskell]

On Fri, 24 Sep 2010, Peter Stephenson wrote:

> The shell doesn't split words in the same way when importing history. 
> In normal operation it relies on the lines having been passed through 
> the lexical analyser to generate the words, which doesn't happen when 
> history is read from a file.  Instead it just blindly splits on 
> whitespace.

Interesting.

I'm surprised it splits the history when it's read in.  (I would've 
expected it to read in whole command lines and only split into words 
when necessary.)

But, glad there's a simple explanation.

-- 
Thanks,
Ben

Re: !!$ unitialized at first prompt

[Peter Stephenson]

On Fri, 24 Sep 2010 10:18:23 -0400 (EDT)
"Benjamin R. Haskell" <zsh@benizi.com> wrote:
> On Fri, 24 Sep 2010, Peter Stephenson wrote:
> 
> > The shell doesn't split words in the same way when importing
> > history. In normal operation it relies on the lines having been
> > passed through the lexical analyser to generate the words, which
> > doesn't happen when history is read from a file.  Instead it just
> > blindly splits on whitespace.
> 
> Interesting.
> 
> I'm surprised it splits the history when it's read in.  (I would've 
> expected it to read in whole command lines and only split into words 
> when necessary.)

There isn't a "when necessary", exactly: what's recorded in the history
list are the word pointers, which is what you get for free (well, OK, for
me spending a few hours a few years ago improving the mechanism, which used
to record words in a slightly messier way) when they've come from "live"
history.  So, without adding additional complexity, doing it on reading in
is the only sensible place in the other case (what it happens to do at that
point is a different matter but if you've seen the completion system you'll
realise interfacing to the lexical analyser just to split up words isn't
much fun with our current implementation).

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: !!$ unitialized at first prompt

[Bart Schaefer]

On Sep 24,  3:26pm, Peter Stephenson wrote:
}
} So, without adding additional complexity, doing it on reading in is
} the only sensible place in the other case (what it happens to do at
} that point is a different matter but if you've seen the completion
} system you'll realise interfacing to the lexical analyser just to
} split up words isn't much fun with our current implementation).

The complexity here seems to be that the history is stored as an
array of lines with pointers into each line for the word positions,
so it's not as simple as calling bufferwords() to get the parsed
result and then store all the individual words.

History is doing a complex task, attempting to maintain both the
lexical interpretation and literal text of the history.  In fact it
originally stored both separately -- excerpt from an ancient manual:

`toggle-literal-history (ESC-R ESC-r)'
     Toggle between literal and lexical history.  The default is lexical
     history unless the `HIST_LIT' option is set.

Re: !!$ unitialized at first prompt

[Bart Schaefer]

On Sep 24,  1:39pm, Peter Stephenson wrote:
}
} The shell doesn't split words in the same way when importing history.
} In normal operation it relies on the lines having been passed through
} the lexical analyser to generate the words, which doesn't happen when
} history is read from a file.  Instead it just blindly splits on
} whitespace.

... and at the time this was designed, it would make history import an
agonizingly slow process if the entire history file had to be passed
through the lexical analyzer.

Nowadays that's probably not so much a concern for most people.  There's
probably an argument to be made that SHARE_HISTORY should force use of
the lexical analyzer, because (although no one has ever complained) the
current implementation means that history words referenced from shared
history behave differently than those from the local history.

You can load your own history something like this provided that you
*don't* use shared or extended history:

    readhistfile() {
	emulate -LR zsh
	local histline
	local -a histwords
	while read -r histline
	do
	    if [[ $histline = *\\ ]]
	    then
		histline[-1]=''
		histwords+=( ${(z)histline}$'\n' )
	    else
		histwords+=( ${(z)histline} )
		print -s $histwords
		histwords=()
	    fi
	done
    }

That's not perfect, as it does things like insert a space at the front
of every continuation line in a multi-line command, but it's passable.
Note the function as written expects the history to be standard input.

-- 

Re: !!$ unitialized at first prompt

[Mikael Magnusson]

On 24 September 2010 17:10, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sep 24,  1:39pm, Peter Stephenson wrote:
> }
> } The shell doesn't split words in the same way when importing history.
> } In normal operation it relies on the lines having been passed through
> } the lexical analyser to generate the words, which doesn't happen when
> } history is read from a file.  Instead it just blindly splits on
> } whitespace.
>
> ... and at the time this was designed, it would make history import an
> agonizingly slow process if the entire history file had to be passed
> through the lexical analyzer.
>
> Nowadays that's probably not so much a concern for most people.  There's
> probably an argument to be made that SHARE_HISTORY should force use of
> the lexical analyzer, because (although no one has ever complained) the
> current implementation means that history words referenced from shared
> history behave differently than those from the local history.
>
> You can load your own history something like this provided that you
> *don't* use shared or extended history:
>
>    readhistfile() {
>        emulate -LR zsh
>        local histline
>        local -a histwords
>        while read -r histline
>        do
>            if [[ $histline = *\\ ]]
>            then
>                histline[-1]=''
>                histwords+=( ${(z)histline}$'\n' )
>            else
>                histwords+=( ${(z)histline} )
>                print -s $histwords
>                histwords=()
>            fi
>        done
>    }
>
> That's not perfect, as it does things like insert a space at the front
> of every continuation line in a multi-line command, but it's passable.
> Note the function as written expects the history to be standard input.

Unless I'm missing something, it's also not unmetafying anything.
Which is not a problem if you're american/english :).

-- 
Mikael Magnusson

Re: !!$ unitialized at first prompt

[Peter Stephenson]

On Fri, 24 Sep 2010 08:10:40 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> You can load your own history something like this provided that you
> *don't* use shared or extended history:
> 
>     readhistfile() {
> 	emulate -LR zsh
> 	local histline
> 	local -a histwords
> 	while read -r histline
> 	do
> 	    if [[ $histline = *\\ ]]
> 	    then
> 		histline[-1]=''
> 		histwords+=( ${(z)histline}$'\n' )
> 	    else
> 		histwords+=( ${(z)histline} )
> 		print -s $histwords
> 		histwords=()
> 	    fi
> 	done
>     }

This, we can do internally without the side effects.  I'd forgotten this
already did use the lexer, so I was too pessimistic.

The mechanism for word splitting in the manner of the (z) flag is a bit
iffy.  I've picked up that newlines get turned into semicolons --- there
was a discussion about this recently --- and also that &! turns into &|,
which I didn't know about.  If you have debugging turned on you'll see
warning messages that should be enough to fix any remaining problems.
(I could just assume the words are at least the right length but I'd
rather find out the problems.)

Index: Src/hist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/hist.c,v
retrieving revision 1.99
diff -p -u -r1.99 hist.c
--- Src/hist.c	22 Jun 2010 11:00:32 -0000	1.99
+++ Src/hist.c	24 Sep 2010 19:59:41 -0000
@@ -2228,11 +2228,12 @@ readhistfile(char *fn, int err, int read
     Histent he;
     time_t stim, ftim, tim = time(NULL);
     off_t fpos;
-    short *wordlist;
+    short *words;
     struct stat sb;
-    int nwordpos, nwordlist, bufsiz;
+    int nwordpos, nwords, bufsiz;
     int searching, newflags, l, ret;
-
+    LinkList wordlist;
+   
     if (!fn && !(fn = getsparam("HISTFILE")))
 	return;
     if (readflags & HFILE_FAST) {
@@ -2251,8 +2252,8 @@ readhistfile(char *fn, int err, int read
 	}
     }
     if ((in = fopen(unmeta(fn), "r"))) {
-	nwordlist = 64;
-	wordlist = (short *)zalloc(nwordlist*sizeof(short));
+	nwords = 64;
+	words = (short *)zalloc(nwords*sizeof(short));
 	bufsiz = 1024;
 	buf = zalloc(bufsiz);
 
@@ -2334,29 +2335,63 @@ readhistfile(char *fn, int err, int read
 	    else
 		he->ftim = ftim;
 
-	    /* Divide up the words.  We don't know how it lexes,
-	       so just look for white-space.
-	       */
+	    /*
+	     * Divide up the words.  Attempt to do this using the lexer.
+	     */
 	    nwordpos = 0;
 	    start = pt;
-	    do {
+	    wordlist = bufferwords(NULL, pt, NULL);
+	    he->nwords = countlinknodes(wordlist);
+	    if (2*he->nwords > nwords) {
+		nwords = 2*he->nwords;
+		words = (short *)realloc(words, nwords*sizeof(short));
+	    }
+	    while (firstnode(wordlist)) {
+		char *word = uremnode(wordlist, firstnode(wordlist));
+		
 		while (inblank(*pt))
 		    pt++;
-		if (*pt) {
-		    if (nwordpos >= nwordlist)
-			wordlist = (short *) realloc(wordlist,
-					(nwordlist += 64)*sizeof(short));
-		    wordlist[nwordpos++] = pt - start;
-		    while (*pt && !inblank(*pt))
-			pt++;
-		    wordlist[nwordpos++] = pt - start;
+		if (!strpfx(word, pt)) {
+		    int bad = 0;
+		    /*
+		     * Oddity 1: newlines turn into semicolons.
+		     */
+		    if (!strcmp(word, ";"))
+			continue;
+		    /*
+		     * Oddity 2: !'s turn into |'s.
+		     */
+		    while (*pt) {
+			if (!*word) {
+			    bad = 1;
+			    break;
+			}
+			if (*pt == *word ||
+			    (*pt == '!' && *word == '|')) {
+			    pt++;
+			    word++;
+			} else {
+			    bad = 1;
+			    break;
+			}
+		    }
+		    if (bad) {
+#ifdef DEBUG
+			dputs(ERRMSG("bad wordsplit reading history: %s\nat: %s"
+				     "\nword: %s"),
+			      start, pt, word);
+#endif
+			break;
+		    }
 		}
-	    } while (*pt);
+		words[nwordpos++] = pt - start;
+		pt += strlen(word);
+		words[nwordpos++] = pt - start;
+	    }
 
-	    he->nwords = nwordpos/2;
 	    if (he->nwords) {
 		he->words = (short *)zalloc(nwordpos*sizeof(short));
-		memcpy(he->words, wordlist, nwordpos*sizeof(short));
+		memcpy(he->words, words, nwordpos*sizeof(short));
 	    } else
 		he->words = (short *)NULL;
 	    addhistnode(histtab, he->node.nam, he);
@@ -2369,7 +2404,7 @@ readhistfile(char *fn, int err, int read
 	    zsfree(lasthist.text);
 	    lasthist.text = ztrdup(start);
 	}
-	zfree(wordlist, nwordlist*sizeof(short));
+	zfree(words, nwords*sizeof(short));
 	zfree(buf, bufsiz);
 
 	fclose(in);

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: !!$ unitialized at first prompt

[Bart Schaefer]

On Sep 24,  9:12pm, Peter Stephenson wrote:
}
} The mechanism for word splitting in the manner of the (z) flag is a
} bit iffy. I've picked up that newlines get turned into semicolons ---
} there was a discussion about this recently --- and also that &! turns
} into &|, which I didn't know about.

Now that you mention it, I'll bet >! also turns into >| ... but it
appears your test is generic enough to handle that.

Another history oddity that this brings to mind is that "print -s"
puts each of its arguments into the history as a separate word, but
on reload those will be re-parsed and are likely to end up with new
word boundaries.

It's unlikely there's anything useful to be done about this, but now
that we have the zshaddhistory hook and people are manipulating what
gets stored by calling "print -s", it may be worth pointing out.

Re: !!$ unitialized at first prompt

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

It might be nice to introduce a new history format that would indicate the
word splitting in a simple manner.  One possibility is to output all spaces
inside a word as some other sequence of characters.  If the new encoding
were indicated in the prefixed extended-history header, the parsing code
would know if it should split the line based on the new, simple decoder or
the old-style parser.  One complicating factor is a syntax that splits into
separate words without intervening spaces.  To deal with that we'd either
need a character sequence to indicate this, or we'd need to still run the
line through the parser before decoding the words.

Here's one possible encoding would use URI encoding to fully encode all the
history semantics of the line.  I'm indicating its presence with a ":%" in
the prefix, like this:

: 1226351062:0:%;cat file%20name%20with%20spaces%20(100%25)
>%00/tmp/file%20name

In the above, I indicated a word-split that didn't have a space with a null
(%00), so the above is 4 words.  The idea is to come up with an encoding
that directly translates between the internal representation and a hist-text
representation.  Would we need more encoding rules?  Maybe something for
metafied chars?

..wayne..

Re: !!$ unitialized at first prompt

[Peter Stephenson]

On Sat, 25 Sep 2010 10:26:28 -0700
Wayne Davison <wayned@users.sourceforge.net> wrote:
> It might be nice to introduce a new history format that would indicate the
> word splitting in a simple manner.

(1) I'm not sure how much more this actually gains (2) I'd rather not
have another incompatible history format (3) I prefer that the history
file always contains a readable line that can be copied directly for use
from outside the shell.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: !!$ unitialized at first prompt

[Bart Schaefer]

On Sep 25,  7:49pm, Peter Stephenson wrote:
} Subject: Re: !!$ unitialized at first prompt
}
} On Sat, 25 Sep 2010 10:26:28 -0700
} Wayne Davison <wayned@users.sourceforge.net> wrote:
} > It might be nice to introduce a new history format that would indicate the
} > word splitting in a simple manner.
} 
} (1) I'm not sure how much more this actually gains (2) I'd rather not
} have another incompatible history format (3) I prefer that the history
} file always contains a readable line that can be copied directly for use
} from outside the shell.

I agree.

If we were going to do something like this, I'd instead suggest that we
take the known word positions as offsets into the buffer, encode that
somehow, and stick it in the extended-history prefix, rather than
embed an encoding in the literal history text.