Working with the historywords special parameter

Working with the historywords special parameter

[Felix Rosencrantz]

I'm trying to create a completer that uses the previous word on the line
and the special parameter historywords to determine what values current
word should be.

It looks to me like there is no single expression that can be used to
get just the list of all the elements that match an expression from an
array.  It seems like you need to iterate over the elements of an array
via an expression like:

    #the_word is $words[CURRENT-1]
    cnt=2
    words=()
    while [[ $historywords[(in:cnt:)the_word] -lt  $#historywords  ]] ; do
       words=($words $historywords[$historywords[(in:cnt:)the_word]-1]
       let "cnt = cnt + 1" 
    done

Is there a faster way to do this?

I'd like to do this with a large historywords (currently I have
4728 elements) doing this takes between 1 and 2 seconds. If I copy
historywords to a temporary array to do the search it goes much faster.
But if the search could be done in one operation, it would be faster and
wouldn't require a temporary copy of historywords.

Would anyone object to a new special parameter (maybe historywordsnums)
that has corresponding elements to historywords saying with which
history line the word is associated?  I ask since one of the problems
with this completer is that it picks up false "matches" when the
previous word is the last word of a previous command, it gets the next command.

It would be nice to be able to determine if a word is the last word of
the line, or what the first word of the line is.

-FR.


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Working with the historywords special parameter

[Sven Wischnowsky]

Felix Rosencrantz wrote:

> I'm trying to create a completer that uses the previous word on the line
> and the special parameter historywords to determine what values current
> word should be.
> 
> It looks to me like there is no single expression that can be used to
> get just the list of all the elements that match an expression from an
> array.  It seems like you need to iterate over the elements of an array
> via an expression like:
> 
>     #the_word is $words[CURRENT-1]
>     cnt=2
>     words=()
>     while [[ $historywords[(in:cnt:)the_word] -lt  $#historywords  ]] ; do
>        words=($words $historywords[$historywords[(in:cnt:)the_word]-1]
>        let "cnt = cnt + 1" 
>     done
> 
> Is there a faster way to do this?

I don't see anything faster than:

  local i length cnt=2 words hist
  words=()
  hist=($historywords)
  length=$#hist
  while (( ( i=$hist[(in:cnt++:)$the_word] ) <= length )); do
    words=($words $hist[i-1])
  done

Maybe ${(Mk)historywords:#$the_word} should expand to the list of
the indices of the words equal to $the_word.

> ...
> 
> Would anyone object to a new special parameter (maybe historywordsnums)
> that has corresponding elements to historywords saying with which
> history line the word is associated?  I ask since one of the problems
> with this completer is that it picks up false "matches" when the
> previous word is the last word of a previous command, it gets the next command.
> 
> It would be nice to be able to determine if a word is the last word of
> the line, or what the first word of the line is.

Or maybe give that new array another name and let it report not only
the line but also other information, e.g. the position on the line
(word number and character positions so that that could be used to
index into the values of the history assoc).


Bye
  Sven

-- 
Sven Wischnowsky                    wischnow@informatik.hu-berlin.de

Re: Working with the historywords special parameter

[Bart Schaefer]

On Aug 23, 11:54pm, Felix Rosencrantz wrote:
}
} I'm trying to create a completer that uses the previous word on the line
} and the special parameter historywords to determine what values current
} word should be.

Have you tried using $history instead of $historywords?  You want the
word that comes immediately after $words[CURRENT-1] in every history line
that contains $word[CURRENT-1], right?

    local w p h r
    w=${(q)words[CURRENT-1]}
    p=$'\0'$w$'\0'
    h=$'\0'${(pj:\0:)${(z)history[(R)*$w*]}}
    r=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
    compadd -a r

This assumes there are no literal NUL or ctrl-A characters in the history,
but that seems a pretty safe assumption.

} It looks to me like there is no single expression that can be used to
} get just the list of all the elements that match an expression from an
} array.

That's a different question.  You want a list of the indices of all the
elements that match $words[CURRENT-1]?

    integer n=0
    local ixs p="(#b)((#s)(${(q)words[CURRENT-1]})(#e)|*)"
    ixs=( ${${(M)${historywords//$~p/$((++n))=$match[2]}:#<->\=?##}%\=*} )

This takes advantage of the fact that alternate matches with `|' are tried
left to right; it wouldn't work with (*|...).

To get the nth element after the element that matches the word, just
start n at a larger number, e.g. n=1 for your specific example.

} Would anyone object to a new special parameter (maybe historywordsnums)
} that has corresponding elements to historywords saying with which
} history line the word is associated?

I won't object, but it's not really necessary, is it?  You can get it
from ${(k)history[(R)word]}.

On Aug 24, 10:16am, Sven Wischnowsky wrote:
}
} Maybe ${(Mk)historywords:#$the_word} should expand to the list of
} the indices of the words equal to $the_word.

That's an interesting thought.  It's equivalent to what I invented above
but not quite as flexible.


-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Working with the historywords special parameter

[Felix Rosencrantz]

--- Bart Schaefer <schaefer@brasslantern.com> wrote:
>Have you tried using $history instead of $historywords?  You want the
>word that comes immediately after $words[CURRENT-1] in every history
>line that contains $word[CURRENT-1], right?
>
>    local w p h r
>    w=${(q)words[CURRENT-1]}
>    p=$'\0'$w$'\0'
>    h=$'\0'${(pj:\0:)${(z)history[(R)*$w*]}}
>    r=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
>    compadd -a r
>
>This assumes there are no literal NUL or ctrl-A characters in the
>history, but that seems a pretty safe assumption.

I tried Sven's suggestion and that worked fine. I then tried Bart's
suggestion, and that worked really fast.  However, there were two
problems.  One a seg fault in the shell when completing for a word that
had either one or no matches  (stack in a separate post).  The other problem
is that it didn't know about line boundaries.  So if the searched for word was
at the end of the line, then it would return a match for the previous
command, which is wrong.

I didn't try Bart's other suggestion for getting indexes.  So don't have
anything to say about that versus a new "M" modifier.

Also, I was wondering if the (z) modifier applied to the elements of history
would always return the same results as found by historywords?

-FR


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Seg Fault in paramsubst()

[Felix Rosencrantz]

When using this in my completer I would sometimes get seg faults.  Below
is a stack trace from gdb.  I think this stack trace would only occur if there
were either no matches or just one.  Though it might just happen in one of
those situations.

>    local w p h r
>    w=${(q)words[CURRENT-1]}
>    p=$'\0'$w$'\0'
>    h=$'\0'${(pj:\0:)${(z)history[(R)*$w*]}}
>    r=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
>    compadd -a r
>
-FR

#0  0x80b472e in paramsubst (l=0xbfffbe88, n=0xbfffbe70, str=0xbfffbc58, qt=0, 
    ssub=0) at subst.c:1836
#1  0x80af2ec in stringsubst (list=0xbfffbe88, node=0xbfffbe70, ssub=0, 
    asssub=0) at subst.c:130
#2  0x80aec7f in prefork (list=0xbfffbe88, flags=0) at subst.c:74
#3  0x80af82f in multsub (s=0xbfffc108, a=0xbfffc104, isarr=0xbfffc12c, 
    sep=0x0) at subst.c:282
#4  0x80b216d in paramsubst (l=0xbfffc3f0, n=0xbfffc3e0, str=0xbfffc1b8, qt=0, 
    ssub=4) at subst.c:1065
#5  0x80af2ec in stringsubst (list=0xbfffc3f0, node=0xbfffc3e0, ssub=4, 
    asssub=0) at subst.c:130
#6  0x80aec7f in prefork (list=0xbfffc3f0, flags=6) at subst.c:74
#7  0x8065b6e in addvars (state=0xbfffc4d0, pc=0x8139d5c, export=0)
    at exec.c:1516
#8  0x8063a5e in execsimple (state=0xbfffc4d0) at exec.c:753
#9  0x8063bc8 in execlist (state=0xbfffc4d0, dont_change_job=1, exiting=0)
    at exec.c:806
#10 0x80639c8 in execode (p=0x8139a68, dont_change_job=1, exiting=0)
    at exec.c:729
#11 0x806b388 in runshfunc (prog=0x8139a68, wrap=0x0, 
    name=0x40249b40 "_history_prev_word") at exec.c:3461
#12 0x401cd80f in comp_wrapper (prog=0x8139a68, w=0x0, 
    name=0x40249b40 "_history_prev_word") at complete.c:1271


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

PATCH: Re: Seg Fault in paramsubst()

[Sven Wischnowsky]

Felix Rosencrantz wrote:

> When using this in my completer I would sometimes get seg faults.  Below
> is a stack trace from gdb.  I think this stack trace would only occur if there
> were either no matches or just one.  Though it might just happen in one of
> those situations.

bufferwords() might return a NULL pointer, so we should check if it did.
Everywhere.


Bye
  Sven

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.21
diff -u -r1.21 subst.c
--- Src/subst.c	2001/08/07 10:18:32	1.21
+++ Src/subst.c	2001/08/28 08:32:38
@@ -1833,7 +1833,7 @@
 	} else
 	    list = bufferwords(NULL, val, NULL);
 
-	if (!firstnode(list))
+	if (!list || !firstnode(list))
 	    val = dupstring("");
 	else if (!nextnode(firstnode(list)))
 	    val = getdata(firstnode(list));
Index: Src/Modules/parameter.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/parameter.c,v
retrieving revision 1.21
diff -u -r1.21 parameter.c
--- Src/Modules/parameter.c	2001/07/09 16:05:14	1.21
+++ Src/Modules/parameter.c	2001/08/28 08:32:38
@@ -1105,9 +1105,9 @@
     int i = addhistnum(curhist, -1, HIST_FOREIGN), iw;
     Histent he = gethistent(i, GETHIST_UPWARD);
 
-    ll = bufferwords(NULL, NULL, NULL);
-    for (n = firstnode(ll); n; incnode(n))
-	pushnode(l, getdata(n));
+    if ((ll = bufferwords(NULL, NULL, NULL)))
+        for (n = firstnode(ll); n; incnode(n))
+            pushnode(l, getdata(n));
 
     while (he) {
 	for (iw = he->nwords - 1; iw >= 0; iw--) {
Index: Src/Zle/zle_misc.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/zle_misc.c,v
retrieving revision 1.4
diff -u -r1.4 zle_misc.c
--- Src/Zle/zle_misc.c	2000/05/10 15:01:19	1.4
+++ Src/Zle/zle_misc.c	2001/08/28 08:32:39
@@ -549,13 +549,13 @@
     int i;
     char *p = NULL;
 
-    l = bufferwords(NULL, NULL, &i);
+    if ((l = bufferwords(NULL, NULL, &i)))
+        for (n = firstnode(l); n; incnode(n))
+            if (!i--) {
+                p = getdata(n);
+                break;
+            }
 
-    for (n = firstnode(l); n; incnode(n))
-	if (!i--) {
-	    p = getdata(n);
-	    break;
-	}
     if (p) {
 	int len = strlen(p);
 

-- 
Sven Wischnowsky                    wischnow@informatik.hu-berlin.de

Re: Working with the historywords special parameter

[Bart Schaefer]

On Aug 27, 11:19pm, Felix Rosencrantz wrote:
}
} --- Bart Schaefer <schaefer@brasslantern.com> wrote:
} >Have you tried using $history instead of $historywords?  You want the
} >word that comes immediately after $words[CURRENT-1] in every history
} >line that contains $word[CURRENT-1], right?
} >
} >    local w p h r
} >    w=${(q)words[CURRENT-1]}
} >    p=$'\0'$w$'\0'
} >    h=$'\0'${(pj:\0:)${(z)history[(R)*$w*]}}
} >    r=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
} >    compadd -a r
} >
} >This assumes there are no literal NUL or ctrl-A characters in the
} >history, but that seems a pretty safe assumption.
} 
} The other problem is that it didn't know about line boundaries. So if
} the searched for word was at the end of the line, then it would return
} a match for the previous command, which is wrong.

Stick a $'\1' on the end of each line so that $'\0'$w$'\0' won't match
the last word:

    local w p h r
    w=${(q)words[CURRENT-1]}
    p=$'\0'$w$'\0'
    #
    h=( ${^history[(R)*$w*]}$'\1' )
    h=$'\0'${(pj:\0:)${(z)h}}
    r=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
    #
    compadd -a r

To see which is faster, you can replace the three lines between the `#'
with:

    r=()
    for h in $history[(R)*$w*]
    do
	h=$'\0'${(pj:\0:)${(z)h}}
	r[$#r+1]=( ${${(ps:\1:)h//$~p/$'\1'}%%$'\0'*} )
    done

I suspect the first way is faster because it does fewer, though larger,
memory allocations.

The loop body could be one line if not for the need to prepend $'\0' to
the join.

} Also, I was wondering if the (z) modifier applied to the elements of
} history would always return the same results as found by historywords?

No, it won't.  $historywords is more like $=history, but it isn't quite
that either -- the rules for what is a "word" in $historywords are a bit
odd.  E.g., quoted strings are broken up at whitespace, but =(...) are
kept as single words.  Also, it appears that $historywords has the words
from the current input line, whereas $history doesn't get that line added
until after the command has been parsed -- but I'm not sure about that.

I think ${(z)history} is more accurate for purposes of completion.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net