Re: simulation of dabbrev-expand

Re: simulation of dabbrev-expand

[Adam Spiers]

Adam Spiers, on thelonious (adam@thelonious.new.ox.ac.uk) wrote:
> Bart Schaefer (schaefer@candle.brasslantern.com) wrote:
> > On Sep 9,  1:39pm, Adam Spiers wrote:
> > } It appears to always work fine when grabbing words from history added
> > } by the current shell, but either doesn't work at all, or very
> > } unreliably when you want to grab a word from the saved history.
> > 
> > Can you perhaps give a more specific example?  I'm not having any trouble
> > with this at all.
> 
>   % history 1 | grep ' /' | wc -l
>       650
>   % less /<TAB>
>   % less /usr/share/zsh/functions/_man
>   zsh: do you wish to see all 51 possibilities? 
> 
> Where did all the others go?

[snip]

> The newer entries work fine.  This problem has
> exhibited itself in exactly the same way across zsh-3.1.6-pws-* :-(

I've done a lot of further investigation into this, and *still* cannot
spot an exact pattern to when it does and doesn't work, which is
unbelievably frustrating.  However, I'm beginning to suspect that it
may have something to do with history duplicates.  Sven's latest patch
to _history_complete_word is very nice, but if the `history_sort'
configuration key is unset, no sorting occurs, and hence no removal of
duplicates.  (As we're talking about individual words extracted from
the history, the options dealing with history line duplicates have
little bearing on this.)

How can I ensure that there are no duplicate words in the list of
matches?  Even if this isn't the real problem I'm experiencing, it may
well help narrow it down.  This questions leads me to another: we have
all these variables at our disposal in the widget functions for
examining the completion system's state in great detail ... but the
one thing we still can't do (AFAICS) is actually see what completions
have currently been calculated.  If there was a special array
parameter, `completions' say, then things like removing duplicate
matches, or choosing your own match sorting algorithm (although heaven
forbid I would ever need to implement a quicksort in zsh :-) would
become possible in the shell functions.  Have I missed something here?

Adam

PATCH: 3.1.6-pws-4: Re: simulation of dabbrev-expand

[Bart Schaefer]

On Sep 16,  5:34pm, Adam Spiers wrote:
} Subject: Re: simulation of dabbrev-expand
}
} Bart Schaefer (schaefer@candle.brasslantern.com) wrote:
} > On Sep 9,  1:39pm, Adam Spiers wrote:
} > } Subject: simulation of dabbrev-expand
} > }
} > } It appears to always work fine when grabbing words from history added
} > } by the current shell, but either doesn't work at all, or very
} > } unreliably when you want to grab a word from the saved history.

I'm finally able to reproduce something like this.

}   % history 1 | grep ' /' | wc -l
}       650
}   % less /<>
}   % less /usr/share/zsh/functions/_man
}   zsh: do you wish to see all 51 possibilities? 

I just tried >>ing the histories from five different xterms into a single
file and then loading them all into a fresh zsh with "fc -R".  Here's what
I get:

zagzig<2996> history 1 | grep ' /' | wc -l             
    110
zagzig<2997> less /tmp/histdump2
zsh: do you wish to see all 138 possibilities? 

If I say "y" at this point I get a list with a lot of duplicate entries,
but it certainly seems to span all the history.

If I next setopt the complete collection of options that you quoted:

}   cshjunkiehistory
}   extendedhistory
}   histallowclobber
}   histexpiredupsfirst
}   histignorealldups
}   histignoredups
}   histignorespace
}   histreduceblanks
}   histverify
}   incappendhistory

then suddenly "compgen -H" fails to search back farther than the point at
which I executed the "setopts", and a few commands later it's searching
only a couple of lines back.

The problem is in zle_tricky.c's use of quietgethist().  With Wayne's new
stuff, the history list may have holes in it, but makecomplistflags() is
trying to iterate over the history by calling quietgethist() on consecutive
integers until it returns nothing -- which happens whenever it encounters
such a hole.

I think the patch below should take care of this part, but it's possible
that I've done more than was necessary; in particular, the hist.c hunk
may not be needed: it might suffice to call the original quietgethist()
before the loop in makecomplistflags().  It's also possible that calling
up_histent() N times is the wrong thing to do:  Should the number passed
to compgen -H specify the number of history entries searched or the range
of history numbers searched?  I've interpreted it as the former.

I haven't done an inventory looking for other such misuses of the gethist
family of functions, but perhaps Wayne or someone else very familiar with
them should do so.

On Sep 17, 10:44pm, Adam Spiers wrote:
} Subject: Re: simulation of dabbrev-expand
}
} How can I ensure that there are no duplicate words in the list of
} matches?

The completion system *should* be handling this for you, but I'm going to
leave that bug to Sven, as I don't feel like understanding the completion
system quite that deeply today.  (I actually seem to be having much better
luck with that, too, after the patch below, but I'm mystified as to why.)

Index: Src/hist.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-3.1/Src/hist.c,v
retrieving revision 1.15
diff -u -r1.15 hist.c
--- hist.c	1999/08/30 15:34:03	1.15
+++ hist.c	1999/09/18 19:21:49
@@ -1435,14 +1435,21 @@
 
 /**/
 Histent
-quietgethist(int ev)
+quietgethistent(int ev, int nearmatch)
 {
     if (ev == curhist && (histactive & HA_ACTIVE)) {
 	curline.text = chline;
 	curline.nwords = chwordpos/2;
 	curline.words = chwords;
     }
-    return gethistent(ev, GETHIST_EXACT);
+    return gethistent(ev, nearmatch);
+}
+
+/**/
+Histent
+quietgethist(int ev)
+{
+    return quietgethistent(ev, GETHIST_EXACT);
 }
 
 /**/
Index: Src/Zle/zle_tricky.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-3.1/Src/Zle/zle_tricky.c,v
retrieving revision 1.157
diff -u -r1.157 zle_tricky.c
--- zle_tricky.c	1999/09/18 07:20:15	1.157
+++ zle_tricky.c	1999/09/18 19:32:19
@@ -6723,8 +6723,8 @@
 	/* We have a pattern to take things from the history. */
 	Patprog pprogc = NULL;
 	char *e, *h, hpatsav;
-	Histent he;
 	int i = addhistnum(curhist,-1,HIST_FOREIGN), n = cc->hnum;
+	Histent he = quietgethistent(i, GETHIST_UPWARD);
 
 	/* Parse the pattern, if it isn't the null string. */
 	if (*(cc->hpat)) {
@@ -6738,7 +6738,7 @@
 	    n = -1;
 
 	/* Now search the history. */
-	while (n-- && (he = quietgethist(i--))) {
+	while (n-- && he) {
 	    int iwords;
 	    for (iwords = 0; iwords < he->nwords; iwords++) {
 		h = he->text + he->words[iwords*2];
@@ -6754,6 +6754,7 @@
 		if (hpatsav)
 		    *e = hpatsav;
 	    }
+	    he = up_histent(he);
 	}
     }
     if ((t = cc->mask & (CC_ARRAYS | CC_INTVARS | CC_ENVVARS | CC_SCALARS |

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: 3.1.6-pws-4: Re: simulation of dabbrev-expand

[Adam Spiers]

Bart Schaefer (schaefer@candle.brasslantern.com) wrote:
> } > On Sep 9,  1:39pm, Adam Spiers wrote:
> } > } Subject: simulation of dabbrev-expand
> } > }
> } > } It appears to always work fine when grabbing words from history added
> } > } by the current shell, but either doesn't work at all, or very
> } > } unreliably when you want to grab a word from the saved history.
> 
> I'm finally able to reproduce something like this.

[...]

> The problem is in zle_tricky.c's use of quietgethist().  With Wayne's new
> stuff, the history list may have holes in it, but makecomplistflags() is
> trying to iterate over the history by calling quietgethist() on consecutive
> integers until it returns nothing -- which happens whenever it encounters
> such a hole.

Ahhh!  Of course!  I had looked at quietgethist() but never twigged
about the holes.

> I think the patch below should take care of this part, but it's possible
> that I've done more than was necessary; in particular, the hist.c hunk
> may not be needed: it might suffice to call the original quietgethist()
> before the loop in makecomplistflags().  It's also possible that calling
> up_histent() N times is the wrong thing to do:  Should the number passed
> to compgen -H specify the number of history entries searched or the range
> of history numbers searched?  I've interpreted it as the former.

I think that's a sensible choice.

[snip patch]

Bart, you're a hero.  This bug has been er, bugging me for ages.
Together with Sven's nice duplicate removing options we can finally
have the dabbrev-expand behaviour which up till now has been the only
aspect of zsh I've found inferior to any other shell.  Great work guys :-)