* !!$ unitialized at first prompt @ 2010-09-24 4:36 Benjamin R. Haskell 2010-09-24 4:43 ` Benjamin R. Haskell 0 siblings, 1 reply; 14+ messages in thread From: Benjamin R. Haskell @ 2010-09-24 4:36 UTC (permalink / raw) To: Zsh Workers Experiencing bizarre behavior when the first thing I do in the shell is to call expand-or-complete on !$. If the value of !$ contains something from $IFS, it expands improperly during the first command to just the last word. # get something into history that has a space-y last arg first-shell$ echo 'a b' a b first-shell$ <ctrl-d> to logout # first noticed the weird tab-completion second-shell$ echo !$<tab> zsh: do you want to see all 294 possibilities (49 lines)? # but it also doesn't expand to the correct value: # (without an intervening command) second-shell$ echo x!{$}y quote><ctrl-c> echo xb'y I first thought it was related to having just added csh_junkie_history to my preferred options, but it's not. Reproducible with HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f. Seems like something's not being initialized in the correct order. -- Best, Ben ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 4:36 !!$ unitialized at first prompt Benjamin R. Haskell @ 2010-09-24 4:43 ` Benjamin R. Haskell 2010-09-24 11:48 ` Mikael Magnusson 0 siblings, 1 reply; 14+ messages in thread From: Benjamin R. Haskell @ 2010-09-24 4:43 UTC (permalink / raw) To: Zsh Workers On Fri, 24 Sep 2010, Benjamin R. Haskell wrote: > # but it also doesn't expand to the correct value: > # (without an intervening command) > second-shell$ echo x!{$}y > quote><ctrl-c> > echo xb'y > The words are definitely split wrong. Not just '$': first-shell$ echo 'a b' 'c d' a b c d first-shell$ <ctrl-d> second-shell$ !!0-2 echo 'a b' a b > [...] > Reproducible with HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f. ^^ ('-f' should be '-d' -- took me a while to figure out why -f was preventing history from being saved -- finally found the mention in the manual.) -- Best, Ben ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 4:43 ` Benjamin R. Haskell @ 2010-09-24 11:48 ` Mikael Magnusson 2010-09-24 12:39 ` Peter Stephenson 0 siblings, 1 reply; 14+ messages in thread From: Mikael Magnusson @ 2010-09-24 11:48 UTC (permalink / raw) To: Zsh Workers On 24 September 2010 06:43, Benjamin R. Haskell <zsh@benizi.com> wrote: > On Fri, 24 Sep 2010, Benjamin R. Haskell wrote: > >> # but it also doesn't expand to the correct value: >> # (without an intervening command) >> second-shell$ echo x!{$}y >> quote><ctrl-c> >> echo xb'y >> > > The words are definitely split wrong. Not just '$': > > first-shell$ echo 'a b' 'c d' > a b c d > first-shell$ <ctrl-d> > > second-shell$ !!0-2 > echo 'a b' > a b > >> [...] >> Reproducible with HISTFILE=/tmp/testhist ZDOTDIR=/nonexistent zsh -f. Could this be related to http://thread.gmane.org/gmane.comp.shells.zsh.devel/20904 ? That also only happens when reloading history. -- Mikael Magnusson ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 11:48 ` Mikael Magnusson @ 2010-09-24 12:39 ` Peter Stephenson 2010-09-24 14:18 ` Benjamin R. Haskell 2010-09-24 15:10 ` Bart Schaefer 0 siblings, 2 replies; 14+ messages in thread From: Peter Stephenson @ 2010-09-24 12:39 UTC (permalink / raw) To: Zsh Workers The shell doesn't split words in the same way when importing history. In normal operation it relies on the lines having been passed through the lexical analyser to generate the words, which doesn't happen when history is read from a file. Instead it just blindly splits on whitespace. -- Peter Stephenson <pws@csr.com> Software Engineer Tel: +44 (0)1223 692070 Cambridge Silicon Radio Limited Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 12:39 ` Peter Stephenson @ 2010-09-24 14:18 ` Benjamin R. Haskell 2010-09-24 14:26 ` Peter Stephenson 2010-09-24 15:10 ` Bart Schaefer 1 sibling, 1 reply; 14+ messages in thread From: Benjamin R. Haskell @ 2010-09-24 14:18 UTC (permalink / raw) To: Peter Stephenson; +Cc: Zsh Workers On Fri, 24 Sep 2010, Peter Stephenson wrote: > The shell doesn't split words in the same way when importing history. > In normal operation it relies on the lines having been passed through > the lexical analyser to generate the words, which doesn't happen when > history is read from a file. Instead it just blindly splits on > whitespace. Interesting. I'm surprised it splits the history when it's read in. (I would've expected it to read in whole command lines and only split into words when necessary.) But, glad there's a simple explanation. -- Thanks, Ben ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 14:18 ` Benjamin R. Haskell @ 2010-09-24 14:26 ` Peter Stephenson 2010-09-24 15:30 ` Bart Schaefer 0 siblings, 1 reply; 14+ messages in thread From: Peter Stephenson @ 2010-09-24 14:26 UTC (permalink / raw) To: Zsh Workers On Fri, 24 Sep 2010 10:18:23 -0400 (EDT) "Benjamin R. Haskell" <zsh@benizi.com> wrote: > On Fri, 24 Sep 2010, Peter Stephenson wrote: > > > The shell doesn't split words in the same way when importing > > history. In normal operation it relies on the lines having been > > passed through the lexical analyser to generate the words, which > > doesn't happen when history is read from a file. Instead it just > > blindly splits on whitespace. > > Interesting. > > I'm surprised it splits the history when it's read in. (I would've > expected it to read in whole command lines and only split into words > when necessary.) There isn't a "when necessary", exactly: what's recorded in the history list are the word pointers, which is what you get for free (well, OK, for me spending a few hours a few years ago improving the mechanism, which used to record words in a slightly messier way) when they've come from "live" history. So, without adding additional complexity, doing it on reading in is the only sensible place in the other case (what it happens to do at that point is a different matter but if you've seen the completion system you'll realise interfacing to the lexical analyser just to split up words isn't much fun with our current implementation). -- Peter Stephenson <pws@csr.com> Software Engineer Tel: +44 (0)1223 692070 Cambridge Silicon Radio Limited Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 14:26 ` Peter Stephenson @ 2010-09-24 15:30 ` Bart Schaefer 0 siblings, 0 replies; 14+ messages in thread From: Bart Schaefer @ 2010-09-24 15:30 UTC (permalink / raw) To: Zsh Workers On Sep 24, 3:26pm, Peter Stephenson wrote: } } So, without adding additional complexity, doing it on reading in is } the only sensible place in the other case (what it happens to do at } that point is a different matter but if you've seen the completion } system you'll realise interfacing to the lexical analyser just to } split up words isn't much fun with our current implementation). The complexity here seems to be that the history is stored as an array of lines with pointers into each line for the word positions, so it's not as simple as calling bufferwords() to get the parsed result and then store all the individual words. History is doing a complex task, attempting to maintain both the lexical interpretation and literal text of the history. In fact it originally stored both separately -- excerpt from an ancient manual: `toggle-literal-history (ESC-R ESC-r)' Toggle between literal and lexical history. The default is lexical history unless the `HIST_LIT' option is set. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 12:39 ` Peter Stephenson 2010-09-24 14:18 ` Benjamin R. Haskell @ 2010-09-24 15:10 ` Bart Schaefer 2010-09-24 15:20 ` Mikael Magnusson 2010-09-24 20:12 ` Peter Stephenson 1 sibling, 2 replies; 14+ messages in thread From: Bart Schaefer @ 2010-09-24 15:10 UTC (permalink / raw) To: Zsh Workers On Sep 24, 1:39pm, Peter Stephenson wrote: } } The shell doesn't split words in the same way when importing history. } In normal operation it relies on the lines having been passed through } the lexical analyser to generate the words, which doesn't happen when } history is read from a file. Instead it just blindly splits on } whitespace. ... and at the time this was designed, it would make history import an agonizingly slow process if the entire history file had to be passed through the lexical analyzer. Nowadays that's probably not so much a concern for most people. There's probably an argument to be made that SHARE_HISTORY should force use of the lexical analyzer, because (although no one has ever complained) the current implementation means that history words referenced from shared history behave differently than those from the local history. You can load your own history something like this provided that you *don't* use shared or extended history: readhistfile() { emulate -LR zsh local histline local -a histwords while read -r histline do if [[ $histline = *\\ ]] then histline[-1]='' histwords+=( ${(z)histline}$'\n' ) else histwords+=( ${(z)histline} ) print -s $histwords histwords=() fi done } That's not perfect, as it does things like insert a space at the front of every continuation line in a multi-line command, but it's passable. Note the function as written expects the history to be standard input. -- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 15:10 ` Bart Schaefer @ 2010-09-24 15:20 ` Mikael Magnusson 2010-09-24 20:12 ` Peter Stephenson 1 sibling, 0 replies; 14+ messages in thread From: Mikael Magnusson @ 2010-09-24 15:20 UTC (permalink / raw) To: Zsh Workers On 24 September 2010 17:10, Bart Schaefer <schaefer@brasslantern.com> wrote: > On Sep 24, 1:39pm, Peter Stephenson wrote: > } > } The shell doesn't split words in the same way when importing history. > } In normal operation it relies on the lines having been passed through > } the lexical analyser to generate the words, which doesn't happen when > } history is read from a file. Instead it just blindly splits on > } whitespace. > > ... and at the time this was designed, it would make history import an > agonizingly slow process if the entire history file had to be passed > through the lexical analyzer. > > Nowadays that's probably not so much a concern for most people. There's > probably an argument to be made that SHARE_HISTORY should force use of > the lexical analyzer, because (although no one has ever complained) the > current implementation means that history words referenced from shared > history behave differently than those from the local history. > > You can load your own history something like this provided that you > *don't* use shared or extended history: > > readhistfile() { > emulate -LR zsh > local histline > local -a histwords > while read -r histline > do > if [[ $histline = *\\ ]] > then > histline[-1]='' > histwords+=( ${(z)histline}$'\n' ) > else > histwords+=( ${(z)histline} ) > print -s $histwords > histwords=() > fi > done > } > > That's not perfect, as it does things like insert a space at the front > of every continuation line in a multi-line command, but it's passable. > Note the function as written expects the history to be standard input. Unless I'm missing something, it's also not unmetafying anything. Which is not a problem if you're american/english :). -- Mikael Magnusson ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 15:10 ` Bart Schaefer 2010-09-24 15:20 ` Mikael Magnusson @ 2010-09-24 20:12 ` Peter Stephenson 2010-09-25 6:53 ` Bart Schaefer 1 sibling, 1 reply; 14+ messages in thread From: Peter Stephenson @ 2010-09-24 20:12 UTC (permalink / raw) To: Zsh Workers On Fri, 24 Sep 2010 08:10:40 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > You can load your own history something like this provided that you > *don't* use shared or extended history: > > readhistfile() { > emulate -LR zsh > local histline > local -a histwords > while read -r histline > do > if [[ $histline = *\\ ]] > then > histline[-1]='' > histwords+=( ${(z)histline}$'\n' ) > else > histwords+=( ${(z)histline} ) > print -s $histwords > histwords=() > fi > done > } This, we can do internally without the side effects. I'd forgotten this already did use the lexer, so I was too pessimistic. The mechanism for word splitting in the manner of the (z) flag is a bit iffy. I've picked up that newlines get turned into semicolons --- there was a discussion about this recently --- and also that &! turns into &|, which I didn't know about. If you have debugging turned on you'll see warning messages that should be enough to fix any remaining problems. (I could just assume the words are at least the right length but I'd rather find out the problems.) Index: Src/hist.c =================================================================== RCS file: /cvsroot/zsh/zsh/Src/hist.c,v retrieving revision 1.99 diff -p -u -r1.99 hist.c --- Src/hist.c 22 Jun 2010 11:00:32 -0000 1.99 +++ Src/hist.c 24 Sep 2010 19:59:41 -0000 @@ -2228,11 +2228,12 @@ readhistfile(char *fn, int err, int read Histent he; time_t stim, ftim, tim = time(NULL); off_t fpos; - short *wordlist; + short *words; struct stat sb; - int nwordpos, nwordlist, bufsiz; + int nwordpos, nwords, bufsiz; int searching, newflags, l, ret; - + LinkList wordlist; + if (!fn && !(fn = getsparam("HISTFILE"))) return; if (readflags & HFILE_FAST) { @@ -2251,8 +2252,8 @@ readhistfile(char *fn, int err, int read } } if ((in = fopen(unmeta(fn), "r"))) { - nwordlist = 64; - wordlist = (short *)zalloc(nwordlist*sizeof(short)); + nwords = 64; + words = (short *)zalloc(nwords*sizeof(short)); bufsiz = 1024; buf = zalloc(bufsiz); @@ -2334,29 +2335,63 @@ readhistfile(char *fn, int err, int read else he->ftim = ftim; - /* Divide up the words. We don't know how it lexes, - so just look for white-space. - */ + /* + * Divide up the words. Attempt to do this using the lexer. + */ nwordpos = 0; start = pt; - do { + wordlist = bufferwords(NULL, pt, NULL); + he->nwords = countlinknodes(wordlist); + if (2*he->nwords > nwords) { + nwords = 2*he->nwords; + words = (short *)realloc(words, nwords*sizeof(short)); + } + while (firstnode(wordlist)) { + char *word = uremnode(wordlist, firstnode(wordlist)); + while (inblank(*pt)) pt++; - if (*pt) { - if (nwordpos >= nwordlist) - wordlist = (short *) realloc(wordlist, - (nwordlist += 64)*sizeof(short)); - wordlist[nwordpos++] = pt - start; - while (*pt && !inblank(*pt)) - pt++; - wordlist[nwordpos++] = pt - start; + if (!strpfx(word, pt)) { + int bad = 0; + /* + * Oddity 1: newlines turn into semicolons. + */ + if (!strcmp(word, ";")) + continue; + /* + * Oddity 2: !'s turn into |'s. + */ + while (*pt) { + if (!*word) { + bad = 1; + break; + } + if (*pt == *word || + (*pt == '!' && *word == '|')) { + pt++; + word++; + } else { + bad = 1; + break; + } + } + if (bad) { +#ifdef DEBUG + dputs(ERRMSG("bad wordsplit reading history: %s\nat: %s" + "\nword: %s"), + start, pt, word); +#endif + break; + } } - } while (*pt); + words[nwordpos++] = pt - start; + pt += strlen(word); + words[nwordpos++] = pt - start; + } - he->nwords = nwordpos/2; if (he->nwords) { he->words = (short *)zalloc(nwordpos*sizeof(short)); - memcpy(he->words, wordlist, nwordpos*sizeof(short)); + memcpy(he->words, words, nwordpos*sizeof(short)); } else he->words = (short *)NULL; addhistnode(histtab, he->node.nam, he); @@ -2369,7 +2404,7 @@ readhistfile(char *fn, int err, int read zsfree(lasthist.text); lasthist.text = ztrdup(start); } - zfree(wordlist, nwordlist*sizeof(short)); + zfree(words, nwords*sizeof(short)); zfree(buf, bufsiz); fclose(in); -- Peter Stephenson <p.w.stephenson@ntlworld.com> Web page now at http://homepage.ntlworld.com/p.w.stephenson/ ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-24 20:12 ` Peter Stephenson @ 2010-09-25 6:53 ` Bart Schaefer 2010-09-25 17:26 ` Wayne Davison 0 siblings, 1 reply; 14+ messages in thread From: Bart Schaefer @ 2010-09-25 6:53 UTC (permalink / raw) To: Zsh Workers On Sep 24, 9:12pm, Peter Stephenson wrote: } } The mechanism for word splitting in the manner of the (z) flag is a } bit iffy. I've picked up that newlines get turned into semicolons --- } there was a discussion about this recently --- and also that &! turns } into &|, which I didn't know about. Now that you mention it, I'll bet >! also turns into >| ... but it appears your test is generic enough to handle that. Another history oddity that this brings to mind is that "print -s" puts each of its arguments into the history as a separate word, but on reload those will be re-parsed and are likely to end up with new word boundaries. It's unlikely there's anything useful to be done about this, but now that we have the zshaddhistory hook and people are manipulating what gets stored by calling "print -s", it may be worth pointing out. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-25 6:53 ` Bart Schaefer @ 2010-09-25 17:26 ` Wayne Davison 2010-09-25 18:49 ` Peter Stephenson 0 siblings, 1 reply; 14+ messages in thread From: Wayne Davison @ 2010-09-25 17:26 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh Workers [-- Attachment #1: Type: text/plain, Size: 1231 bytes --] It might be nice to introduce a new history format that would indicate the word splitting in a simple manner. One possibility is to output all spaces inside a word as some other sequence of characters. If the new encoding were indicated in the prefixed extended-history header, the parsing code would know if it should split the line based on the new, simple decoder or the old-style parser. One complicating factor is a syntax that splits into separate words without intervening spaces. To deal with that we'd either need a character sequence to indicate this, or we'd need to still run the line through the parser before decoding the words. Here's one possible encoding would use URI encoding to fully encode all the history semantics of the line. I'm indicating its presence with a ":%" in the prefix, like this: : 1226351062:0:%;cat file%20name%20with%20spaces%20(100%25) >%00/tmp/file%20name In the above, I indicated a word-split that didn't have a space with a null (%00), so the above is 4 words. The idea is to come up with an encoding that directly translates between the internal representation and a hist-text representation. Would we need more encoding rules? Maybe something for metafied chars? ..wayne.. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-25 17:26 ` Wayne Davison @ 2010-09-25 18:49 ` Peter Stephenson 2010-09-26 1:14 ` Bart Schaefer 0 siblings, 1 reply; 14+ messages in thread From: Peter Stephenson @ 2010-09-25 18:49 UTC (permalink / raw) To: Zsh Workers On Sat, 25 Sep 2010 10:26:28 -0700 Wayne Davison <wayned@users.sourceforge.net> wrote: > It might be nice to introduce a new history format that would indicate the > word splitting in a simple manner. (1) I'm not sure how much more this actually gains (2) I'd rather not have another incompatible history format (3) I prefer that the history file always contains a readable line that can be copied directly for use from outside the shell. -- Peter Stephenson <p.w.stephenson@ntlworld.com> Web page now at http://homepage.ntlworld.com/p.w.stephenson/ ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: !!$ unitialized at first prompt 2010-09-25 18:49 ` Peter Stephenson @ 2010-09-26 1:14 ` Bart Schaefer 0 siblings, 0 replies; 14+ messages in thread From: Bart Schaefer @ 2010-09-26 1:14 UTC (permalink / raw) To: Zsh Workers On Sep 25, 7:49pm, Peter Stephenson wrote: } Subject: Re: !!$ unitialized at first prompt } } On Sat, 25 Sep 2010 10:26:28 -0700 } Wayne Davison <wayned@users.sourceforge.net> wrote: } > It might be nice to introduce a new history format that would indicate the } > word splitting in a simple manner. } } (1) I'm not sure how much more this actually gains (2) I'd rather not } have another incompatible history format (3) I prefer that the history } file always contains a readable line that can be copied directly for use } from outside the shell. I agree. If we were going to do something like this, I'd instead suggest that we take the known word positions as offsets into the buffer, encode that somehow, and stick it in the extended-history prefix, rather than embed an encoding in the literal history text. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2010-09-26 1:14 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-09-24 4:36 !!$ unitialized at first prompt Benjamin R. Haskell 2010-09-24 4:43 ` Benjamin R. Haskell 2010-09-24 11:48 ` Mikael Magnusson 2010-09-24 12:39 ` Peter Stephenson 2010-09-24 14:18 ` Benjamin R. Haskell 2010-09-24 14:26 ` Peter Stephenson 2010-09-24 15:30 ` Bart Schaefer 2010-09-24 15:10 ` Bart Schaefer 2010-09-24 15:20 ` Mikael Magnusson 2010-09-24 20:12 ` Peter Stephenson 2010-09-25 6:53 ` Bart Schaefer 2010-09-25 17:26 ` Wayne Davison 2010-09-25 18:49 ` Peter Stephenson 2010-09-26 1:14 ` Bart Schaefer
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).