Local/global history with pattern isearch

Local/global history with pattern isearch

[Jan Larres]

Hi,

a few months ago there was a discussion about how to use the shared
history only for isearch, but not for example for the <Up> arrow search.
Since I use the history-incremental-pattern-search-backward widget
instead of the normal isearch widget the solution didn't work for me,
but Bart suggested this instead:

zle-line-init()  { NUMERIC=1 zle set-local-history }
zle -N zle-line-init
zle-isearch-update() { NUMERIC=0 zle set-local-history }
zle -N zle-isearch-update
zle-isearch-exit()  { NUMERIC=1 zle set-local-history }
zle -N zle-isearch-exit

history-incremental-pattern-search-backward() {
    if [[ $LASTWIDGET != history-incremental-pattern-search-backward ]]; then
        NUMERIC=0 zle set-local-history
    fi
    zle .history-incremental-pattern-search-backward "$@"
}
zle -N history-incremental-pattern-search-backward
bindkey '^r' history-incremental-pattern-search-backward

Unfortunately this still isn't quite working for me. It does use the
shared history when starting the search, but when pressing ^r again it
fails to find any other results, neither in the shared nor in the local
history. The custom function also doesn't get called again in that case,
I don't know if that's normal behaviour or not. I would be grateful if
anyone know how to fix this.

Cheers,
Jan

Re: Local/global history with pattern isearch

[Bart Schaefer]

On Mar 3,  5:19pm, Jan Larres wrote:
} Subject: Local/global history with pattern isearch
}
} zle -N history-incremental-pattern-search-backward
} bindkey '^r' history-incremental-pattern-search-backward
} 
} Unfortunately this still isn't quite working for me. It does use the
} shared history when starting the search, but when pressing ^r again it
} fails to find any other results, neither in the shared nor in the local
} history.

This *seems* to be working for me in zsh-5.0.5-71-g965d730 (the latest
pull from git as of my writing this).  I find multiple results with
repeated presses of ^R, and adding   zle -M "$ZLE_STATE"   to the
zle-isearch-update function shows "globalhistory insert" as expected.

I also tried removing   zle set-local-history   from zle-isearch-update
and found that $ZLE_STATE still reports "globalhistory", so the update
hook may not be needed.

} The custom function also doesn't get called again in that case,
} I don't know if that's normal behaviour or not.

As I think was mentioned in the thread when this first came around, the
isearch routines don't use true keymaps; they use the widget names of
regular keymaps but substitute their own widget actions.  This is why
the custom name must be "history-incremental-pattern-search-backward"
in order for this to work at all.  So yes, this is expected.

(Actually you could probably get away with naming it something else as
long as you explicitly bound the search-again key in the isearch keymap
to history-incremental-pattern-search-backward.)

HOWEVER, I think this may be the source of the bug.  Repeating the
search does not exit from the isearch keymap, so zle-isearch-exit is
always called between any two calls of the custom widget.  Thus the
[[ $LASTWIDGET != history-incremental-pattern-search-backward ]]  test
is probably incorrect part of the time.

Try changing to

    history-incremental-pattern-search-backward() {
	NUMERIC=0 zle set-local-history
	zle .history-incremental-pattern-search-backward "$@"
    }

and see if that clears it up?

Re: Local/global history with pattern isearch

[Jan Larres]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

Okay, the good news first: I found the culprit! The issue was caused by
the zsh-syntax-highlighting script
(https://github.com/zsh-users/zsh-syntax-highlighting). If I stop
sourcing that script everything works beautifully. The problems seems to
stem from the fact that that script rebinds all of the widgets to invoke
the highlighting routine after them:
https://github.com/zsh-users/zsh-syntax-highlighting/blob/master/zsh-syntax-highlighting.zsh#L127
Unfortunately I don't know enough about the widget stuff to pinpoint the
exact reason this makes it fail. I've sourced that script after the zle
changes so far, but a quick swapping of order doesn't seem to make any
difference.

During my tests I also found a way to make zsh segfault. While I still
had the above script enabled I occasionally encountered some strange
behaviour, described in this sequence:

  1. Press ^r and enter something common like 'ls', the first result
     gets displayed
  2. Press ^r again, a "failing bck-i-search" message gets displayed
  3. Press ^r again, another result from the history gets displayed (huh?)
  4. Press ^r again, "failing" message again and/or segfault

In the cases where it doesn't segfault in step 4 pressing ^r more often
after that just results in more "failing" messages. I've attached a full
backtrace of the segfault case. This is with release 5.0.5 on Debian, by
the way.

On 03/03/14 18:44, Bart Schaefer wrote:
> HOWEVER, I think this may be the source of the bug.  Repeating the
> search does not exit from the isearch keymap, so zle-isearch-exit is
> always called between any two calls of the custom widget.  Thus the
> [[ $LASTWIDGET != history-incremental-pattern-search-backward ]]  test
> is probably incorrect part of the time.
>
> Try changing to
>
>     history-incremental-pattern-search-backward() {
> 	NUMERIC=0 zle set-local-history
> 	zle .history-incremental-pattern-search-backward "$@"
>     }
>
> and see if that clears it up?

This didn't change anything with the highlighting script still enabled,
and I didn't see any immediate difference when doing some quick tests
after disabling that script, but I'll try to investigate further.

Thanks for the help!

-Jan

[-- Attachment #2: gdb.txt --]
[-- Type: text/plain, Size: 11728 bytes --]

#0  0x000000000048324c in metafy (buf=0x24007c7320 <Address 0x24007c7320 out of bounds>, len=40, heap=heap@entry=6) at ../../Src/utils.c:3985
        meta = 0
        t = <optimized out>
        p = <optimized out>
        e = 0x24007c7321 <Address 0x24007c7321 out of bounds>
        mbuf =           '\000' <repeats 8192 times>
#1  0x000000000042fd51 in igetmatch (sp=sp@entry=0x7fffffffc6d8, p=p@entry=0x7c7300, fl=fl@entry=8710, n=n@entry=0, replstr=replstr@entry=0x0, 
    repllistp=repllistp@entry=0x7fffffffc7e8) at ../../Src/glob.c:2559
        muststr = <optimized out>
        s = 0x80eea0 "less /usr/share/vim/vim73/syntax/dircolors.vim"
        t = 0x7fff00000000 <Address 0x7fff00000000 out of bounds>
        tmatch = <optimized out>
        ioff = <optimized out>
        l = 46
        matched = 1
        umltot = 46
        umlen = <optimized out>
        nmatches = <optimized out>
        repllist = 0x0
#2  0x00000000004349b0 in getmatchlist (str=str@entry=0x80eea0 "less /usr/share/vim/vim73/syntax/dircolors.vim", p=p@entry=0x7c7300, 
    repllistp=repllistp@entry=0x7fffffffc7e8) at ../../Src/glob.c:2445
        sp = 0x7fffffffc6d8
#3  0x00007ffff6abd316 in doisearch (args=<optimized out>, dir=-1, pattern=1) at ../../../Src/Zle/zle_hist.c:1308
        t = 0x0
        ibuf = 0x7ffff5de5220 "failing bck-i-search: ls"
        sbuf = 0x7ffff5de5236 "ls"
        last_line = 0x80ef60 "eval $(dircolors -b ~/.etc/zsh/zsh/dircolors) && ls_colors"
        zt = 0x80eea0 "less /usr/share/vim/vim73/syntax/dircolors.vim"
        sbptr = 2
        top_spot = 5
        sibuf = 80
        nomatch = 0
        skip_line = <optimized out>
        skip_pos = <optimized out>
        odir = -1
        sens = 3
        hl = 976
        pos = 46
        pat_hl = 977
        pat_pos = 49
        revert_patpos = 0
        dup_ok = <optimized out>
        end_pos = 58
        savekeys = -1
        feep = 0
        nosearch = 0
        cmd = <optimized out>
        okeymap = 0x9bafa0 "main"
        he = 0x80ee40
        patprog = 0x7c7300
        matchlist = 0x0
        exitfn = 0x0
        aborted = 0
#4  0x00007ffff6ac311e in execzlefunc (func=func@entry=0x7ffff6cea588 <thingies+9320>, args=args@entry=0x7fffffffca88, set_bindk=set_bindk@entry=0)
    at ../../../Src/Zle/zle_main.c:1356
        wflags = 1
        r = 0
        ret = 0
        remetafy = 0
        w = 0x7ffff6cec110 <widgets+2320>
        save_bindk = 0x7ffff6ce8a30 <thingies+2320>
#5  0x00007ffff6ad158e in bin_zle_call (name=0x7ffff5de51d8 "zle", args=0x7fffffffca88, ops=<optimized out>, func=<optimized out>)
    at ../../../Src/Zle/zle_thingy.c:711
        t = 0x7ffff6cea588 <thingies+9320>
        modsave = {
          flags = 1, 
          mult = 1, 
          tmult = 1, 
          vibuf = 0, 
          base = 10
        }
        ret = <optimized out>
        saveflag = 0
        setbindk = 0
        wname = <optimized out>
        keymap_restore = 0x0
        keymap_tmp = <optimized out>
#6  0x000000000041cd76 in execbuiltin (args=args@entry=0x7ffff5de5178, bn=bn@entry=0x7ffff6ced540 <bintab+128>) at ../../Src/builtin.c:450
        argarr = 0x7fffffffca80
        argv = 0x7fffffffca80
        pp = <optimized out>
        name = 0x7ffff5de51d8 "zle"
        optstr = 0x7ffff6adf1fb "aAcCDFgGIKlLmMNrRTUw"
        flags = 8
        argc = <optimized out>
        execop = <optimized out>
        xtr = 0
        ops = {
          ind =             '\000' <repeats 127 times>, 
          args = 0x0, 
          argscount = 0, 
          argsalloc = 0
        }
#7  0x000000000042ad76 in execcmd (state=0x7fffffffd1c0, input=<optimized out>, output=<optimized out>, how=<optimized out>, last1=2)
    at ../../Src/exec.c:3274
        restorelist = <optimized out>
        removelist = <optimized out>
        hn = <optimized out>
        args = <optimized out>
        filelist = 0x0
        node = <optimized out>
        fn = <optimized out>
        mfds =           {[0] = 0x0,
          [1] = 0x0,
          [2] = 0x0,
          [3] = 0x0,
          [4] = 0x0,
          [5] = 0x0,
          [6] = 0x0,
          [7] = 0x0,
          [8] = 0x0,
          [9] = 0x0}
        text = <optimized out>
        save =           {[0] = -2,
          [1] = -2,
          [2] = -2,
          [3] = -2,
          [4] = -2,
          [5] = -2,
          [6] = -2,
          [7] = -2,
          [8] = -2,
          [9] = -2}
        fil = <optimized out>
        dfil = <optimized out>
        is_cursh = <optimized out>
        type = 6
        do_exec = 0
        redir_err = 0
        i = <optimized out>
        htok = 1
        nullexec = <optimized out>
        assign = <optimized out>
        forked = 0
        is_shfunc = 0
        is_builtin = <optimized out>
        is_exec = 0
        use_defpath = 0
        cflags = <optimized out>
        orig_cflags = 0
        checked = <optimized out>
        oautocont = -1
        redir = <optimized out>
        code = <optimized out>
        beg = <optimized out>
        varspc = 0x0
        oxtrerr = 0x7ffff74a3060 <_IO_2_1_stderr_>
        newxtrerr = 0x0
#8  0x000000000042b2c3 in execpline2 (state=0x7fffffffd1c0, pcode=40, pcode@entry=323, how=18, input=0, output=0, last1=-14360, last1@entry=0)
    at ../../Src/exec.c:1691
        pid = -11840
        pipes =           {[0] = 0,
          [1] = 0}
#9  0x000000000042b7f4 in execpline (state=state@entry=0x7fffffffd1c0, slcode=<optimized out>, how=how@entry=18, last1=0) at ../../Src/exec.c:1478
        ipipe =           {[0] = 0,
          [1] = 0}
        opipe =           {[0] = 0,
          [1] = 0}
        pj = 0
        newjob = 1
        old_simple_pline = 0
        slflags = 0
        code = 323
        lastwj = 2
        lpforked = 0
#10 0x000000000042cb72 in execlist (state=state@entry=0x7fffffffd1c0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0)
    at ../../Src/exec.c:1261
        donedebug = 0
        donetrap = 0
        next = 0x7ffff7dfd4dc
        code = <optimized out>
        ret = <optimized out>
        cj = 0
        csp = 0
        ltype = 18
        old_pline_level = 0
        old_list_pipe = 0
        oldlineno = 1
        oldnoerrexit = 0
#11 0x000000000042ce52 in execode (p=p@entry=0x7bbd60, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0, 
    context=context@entry=0x48d4ce "shfunc") at ../../Src/exec.c:1070
        s = {
          prog = 0x7bbd60, 
          pc = 0x7ffff7dfd4dc, 
          strs = 0x7ffff7dfdd95 "\205LASTWIDGET"
        }
        zsh_eval_context_len = 16
        alen = <optimized out>
#12 0x0000000000426c29 in runshfunc (prog=0x7bbd60, wrap=0x0, name=0x7ffff5de4f68 "history-incremental-pattern-search-backward") at ../../Src/exec.c:4865
        cont = <optimized out>
        ouu = 6
        ou = 0x9bafe0 "_BASE"
#13 0x0000000000427226 in doshfunc (shfunc=shfunc@entry=0x7bbda0, doshargs=doshargs@entry=0x0, noreturnval=noreturnval@entry=1) at ../../Src/exec.c:4756
        pptab = 0x6c1fd0
        x = <optimized out>
        oargv0 = 0x6c1fb0 "/usr/bin/zsh"
        oldzoptind = 1
        oldlastval = 0
        oldoptcind = 0
        oldnumpipestats = 1
        ret = <optimized out>
        oldpipestats = 0x7ffff5de4f30
        saveopts =           "\000\001\000\001\000\001\000\000\001\001\000\001\001\001\001\000\001\001\001\000\000\000\001\001\001\001\001\000\000\000\000\001\000\000\000\001\001\000\000\000\000\000\000\000\000\001\000\001\000\000\001\001\001\001\000\000\001\001\001\001\000\000\000\000\001\001\000\001\000\001\000\000\000\001\000\001\000\000\000\000\001\000\000\000\000\000\000\000\000\001\001\000\000\000\000\000\000\001\001\000\000\001\000\000\000\000\001\000\000\000\000\001\001\001\001\000\001", '\000' <repeats 18 times>, "\001\001\001\001\001\001\001\000\000\000\001\000\000\000\000\000\001\000\000\001\000\000\001\000\000\000\000\000\000\000\000\001\000\000\000\000\001"
        oldscriptname = 0x0
        name = <optimized out>
        flags = <optimized out>
        ooflags = 0
        fname = 0x7ffff5de4f00 "history-incremental-pattern-search-backward"
        obreaks = 0
        saveemulation = 48
        restore_sticky = 0
        prog = <optimized out>
        fstack = {
          prev = 0x0, 
          name = 0x7ffff5de4f68 "history-incremental-pattern-search-backward", 
          filename = 0x7ffff5de4fa8 "/home/user/jan/.etc/zsh/zsh/zshrc.d/80_zle", 
          caller = 0x7ffff5de4f98 "/usr/bin/zsh", 
          flineno = 42, 
          lineno = 1, 
          tp = 1
        }
        oflags = 0
        save_sticky = 0x0
        funcdepth = 1
#14 0x00007ffff6ac32c4 in execzlefunc (func=0x7ffff6ce8a30 <thingies+2320>, args=args@entry=0x7ffff6ced9c8 <zlenoargs>, set_bindk=set_bindk@entry=0)
    at ../../../Src/Zle/zle_main.c:1390
        osc = 0
        osi = 14
        oxt = 0
        largs = 0x0
        shf = 0x7bbda0
        r = 0
        ret = 0
        remetafy = 0
        w = 0x7bc2d0
        save_bindk = 0x7ffff6ce8a30 <thingies+2320>
#15 0x00007ffff6ac3859 in zlecore () at ../../../Src/Zle/zle_main.c:1073
No locals.
#16 0x00007ffff6ac4423 in zleread (lp=<optimized out>, rp=<optimized out>, flags=<optimized out>, context=<optimized out>, 
    init=0x7ffff6adf099 "zle-line-init", finish=0x7ffff6adf089 "zle-line-finish") at ../../../Src/Zle/zle_main.c:1257
        s = <optimized out>
        old_errno = 22
        tmout = <optimized out>
#17 0x000000000044019f in zleentry (cmd=1) at ../../Src/init.c:1477
        ret = 0x0
        ap =           {[0] = {
            gp_offset = 40, 
            fp_offset = 48, 
            overflow_arg_area = 0x7fffffffd900, 
            reg_save_area = 0x7fffffffd840
          }}
#18 0x0000000000441223 in inputline () at ../../Src/input.c:285
        flags = 8155937
        ingetcpmptl = 0x28
        ingetcpmptr = 0x6
        context = 49152
#19 ingetc () at ../../Src/input.c:221
No locals.
#20 0x000000000043a0c6 in ihgetc () at ../../Src/hist.c:282
        c = <optimized out>
#21 0x000000000044b1be in gettok () at ../../Src/lex.c:714
        peekfd = -1
#22 zshlex () at ../../Src/lex.c:395
No locals.
#23 0x0000000000467e77 in parse_event () at ../../Src/parse.c:451
No locals.
#24 0x000000000043d479 in loop (toplevel=toplevel@entry=1, justonce=justonce@entry=0) at ../../Src/init.c:132
        prog = <optimized out>
        err = <optimized out>
        non_empty = 0
#25 0x00000000004407ae in zsh_main (argc=<optimized out>, argv=<optimized out>) at ../../Src/init.c:1633
        errexit = 0
        t = <optimized out>
        runscript = <optimized out>
#26 0x00007ffff7120b45 in __libc_start_main (main=0x40efc0 <main at ../../Src/main.c:93>, argc=1, argv=0x7fffffffdb68, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdb58) at libc-start.c:287
        result = <optimized out>
        unwind_buf = {
          cancel_jmp_buf =             {[0] = {
              jmp_buf =                 {[0] = 0,
                [1] = 5655022532906994327,
                [2] = 4255685,
                [3] = 140737488345952,
                [4] = 0,
                [5] = 0,
                [6] = -5655022534070875497,
                [7] = -5655039900245144937}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad =               {[0] = 0x0,
              [1] = 0x0,
              [2] = 0x48ba90 <__libc_csu_init>,
              [3] = 0x7fffffffdb68}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 4766352
            }
          }
        }
        not_first_call = <optimized out>
#27 0x000000000040efee in _start ()
No symbol table info available.

Re: Local/global history with pattern isearch

[Bart Schaefer]

On Mar 4, 12:44pm, Jan Larres wrote:
} 
} Okay, the good news first: I found the culprit! The issue was caused by
} the zsh-syntax-highlighting script

Hmm.  I don't use that, but looking at it, it should be the case that if
you define your widget first then it will skip trying to redefine it,
and if you define yours second then yours will replace the highlighting
one.  (This is all because your widget has the same name as a builtin.)

That explains why 
} ... a quick swapping of order doesn't seem to make any
} difference.

Moving on:

} During my tests I also found a way to make zsh segfault.  While I still
} had the above script enabled

Users of zsh-syntax-highlighting have reported segfaults before.  There
must be a subtle mistake somewhere in the zsh_highlight internals.  I'm
not going to try to run this one down, so if any other zsh-workers are
listening and inclined ...

} I occasionally encountered some strange
} behaviour, described in this sequence:
} 
}   1. Press ^r and enter something common like 'ls', the first result
}      gets displayed
}   2. Press ^r again, a "failing bck-i-search" message gets displayed
}   3. Press ^r again, another result from the history gets displayed (huh?)

This is almost certainly related to what I said in my previous message
on this thread, about the $LASTWIDGET test being wrong.  So it probably
(but not definitely) is not related to the segfault.

}   4. Press ^r again, "failing" message again and/or segfault

Re: Local/global history with pattern isearch

[Jan Larres]

On 06/03/14 17:53, Bart Schaefer wrote:
> On Mar 4, 12:44pm, Jan Larres wrote:
> }
> } Okay, the good news first: I found the culprit! The issue was caused by
> } the zsh-syntax-highlighting script
>
> Hmm.  I don't use that, but looking at it, it should be the case that if
> you define your widget first then it will skip trying to redefine it,
> and if you define yours second then yours will replace the highlighting
> one.  (This is all because your widget has the same name as a builtin.)
>
> That explains why
> } ... a quick swapping of order doesn't seem to make any
> } difference.

This got me wondering why it wasn't working even though the search
widget got left alone, and so I went trying to figure out which widget
*was* the issue. And, possibly to no-one's surprise, that widget turned
out to be set-local-history. If I redefine that widget so it won't get
redefined again by zsh-syntax-highlighting then the local/global history
functionality works perfectly together with pattern-search. Apparently
the calling of an additional function after that widget confuses it
somehow.

So for those playing along at home this is what I currently have in my
.zshrc:


set-local-history() { zle .set-local-history }
zle -N set-local-history
zle-line-init()  { NUMERIC=1 zle set-local-history }
zle -N zle-line-init
zle-isearch-update() { NUMERIC=0 zle set-local-history }
zle -N zle-isearch-update
zle-isearch-exit()  { NUMERIC=1 zle set-local-history }
zle -N zle-isearch-exit

history-incremental-pattern-search-backward() {
    NUMERIC=0 zle set-local-history
    zle .history-incremental-pattern-search-backward "$@"
}
zle -N history-incremental-pattern-search-backward
bindkey '^r' history-incremental-pattern-search-backward
history-incremental-pattern-search-forward() {
    NUMERIC=0 zle set-local-history
    zle .history-incremental-pattern-search-forward "$@"
}
zle -N history-incremental-pattern-search-forward
bindkey '^s' history-incremental-pattern-search-forward


Thanks for the help, Bart!

-Jan

Re: Local/global history with pattern isearch

[Bart Schaefer]

On Mar 7,  7:49pm, Jan Larres wrote:
} Subject: Re: Local/global history with pattern isearch
}
} This got me wondering why it wasn't working even though the search
} widget got left alone, and so I went trying to figure out which widget
} *was* the issue. And, possibly to no-one's surprise, that widget turned
} out to be set-local-history.

Aha!  Good catch.

zsh-syntax-highlight is a bit too aggressive about remapping nearly all
the widgets that appear in the output of $(builtin zle -la).  It does
filter out run-help|which-command|beep but there are a number of others
that do not alter the buffer or the cursor position, which also ought
to be excluded:  the suffix widgets (auto-*), the argument widgets
(*-argument|argument-base), reset-prompt, split-undo, and probably
describe-key-briefly and what-cursor-position, plus set-local-history.

If anyone reading this is a zsh-syntax-highlighting developer, you might
consider tweaking the filter at line 137 of zsh-syntax-highlighting.zsh.

I'd also argue that the "hook widgets" (matching the pattern zle-*) need
special treatment.  I'm actually suprised that z-s-h doesn't already use
a zle-line-init widget.

Re: Local/global history with pattern isearch

[Jan Larres]

On 07/03/14 20:51, Bart Schaefer wrote:
> zsh-syntax-highlight is a bit too aggressive about remapping nearly all
> the widgets that appear in the output of $(builtin zle -la).  It does
> filter out run-help|which-command|beep but there are a number of others
> that do not alter the buffer or the cursor position, which also ought
> to be excluded:  the suffix widgets (auto-*), the argument widgets
> (*-argument|argument-base), reset-prompt, split-undo, and probably
> describe-key-briefly and what-cursor-position, plus set-local-history.
>
> If anyone reading this is a zsh-syntax-highlighting developer, you might
> consider tweaking the filter at line 137 of zsh-syntax-highlighting.zsh.

I've filed an issue:
https://github.com/zsh-users/zsh-syntax-highlighting/issues/137

So hopefully it will get fixed soon.

-Jan