subscript too small after complete in variable

subscript too small after complete in variable

[Dylan Alex Simon]

Annoying completion issue (though with no ill effects beyond a delay and
message):
1. at prompt, enter "FOOBAR=" (no newline)
2. move cursor back a few characters (say, onto B)
3. activate complete-word (with tab)
4. shell hangs a second and then:
   _subscript:20: subscript too small: -262145

zsh 4.3.9.  Seems to be reproducible with various different options and
bindings, but let me know if you need more info.  Thanks.

:-Dylan

Re: subscript too small after complete in variable

[Bart Schaefer]

On Apr 6, 11:57pm, Dylan Alex Simon wrote:
} Subject: subscript too small after complete in variable
}
} 4. shell hangs a second and then:
}    _subscript:20: subscript too small: -262145

Try the patch below.  Doesn't actually find any useful completions,
but at least fixes the infinite-but-for-memory-limits loop.

--- ../zsh-forge/current/Completion/Zsh/Context/_subscript      2008-10-04
09:28:48.000000000 -0700
+++ Completion/Zsh/Context/_subscript   2009-04-06 21:12:26.000000000 -0700
@@ -17,7 +17,7 @@
 # name expansion.  Also expand in anything that looks like an assignment
 # or colon list.
 integer pos=$((CURSOR+1))
-while [[ pos > 1 && $BUFFER[pos-1] != '[' ]]; do (( pos-- )); done
+while [[ pos -gt 1 && $BUFFER[pos-1] != '[' ]]; do (( pos-- )); done
 if [[ $BUFFER[1,pos-1] = *[[:space:]:=]##\~\[ ]]; then
   _dynamic_directory_name
 elif [[ "$PREFIX" = :* ]]; then

Re: subscript too small after complete in variable

[Peter Stephenson]

On Mon, 06 Apr 2009 21:16:00 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Apr 6, 11:57pm, Dylan Alex Simon wrote:
> } Subject: subscript too small after complete in variable
> }
> } 4. shell hangs a second and then:
> }    _subscript:20: subscript too small: -262145
> 
> Try the patch below.  Doesn't actually find any useful completions,
> but at least fixes the infinite-but-for-memory-limits loop.

Yes, that's correct, it was just a typo I introduced recently before
4.3.9.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

crash when expand-or-complete widget wrapped

[Dylan Alex Simon]

> > Try the patch below.  Doesn't actually find any useful completions,
> > but at least fixes the infinite-but-for-memory-limits loop.
>
> Yes, that's correct, it was just a typo I introduced recently before
> 4.3.9.

Thanks, should have seen that myself.  Now for a harder one:

- Start zsh with at least the following zshrc (all other startup files may be
  empty/missing):
    .zshrc:
      autoload -U compinit
      compinit
- Type anything and use expand-and-complete with tab such that it generates
  matches.
- Create the following zle widget for expand-or-complete:
    widget-wrapper () {
      zle .$WIDGET -- "$@"
    }
    zle -N expand-or-complete widget-wrapper
- Again attempt to use completion with tab, repeatedly if necessary.

Various behaviors are observed.  Sometimes nothing happens (no completion).
Sometimes "BUG: listmatches called with bogus list."  Most often it crashes
due to a corrupted match list of some kind (here m is invalid):

#0  0x00007fb8d3c9ecf3 in do_completion (dummy=0x7fb8d410b188, dat=0x7fffdd162ee0) at /usr/src/zsh-4.3.9/src/Src/Zle/compcore.c:409
#1  0x0000000000456734 in runhookdef (h=0x7fb8d410b188, d=0x7fffdd162ee0) at /usr/src/zsh-4.3.9/src/Src/module.c:996
#2  0x00007fb8d3ef389b in docompletion (s=0xdf3c60 "t", lst=0, incmd=1) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_tricky.c:2135
#3  0x00007fb8d3eef32e in docomplete (lst=0) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_tricky.c:859
#4  0x00007fb8d3eedb0d in expandorcomplete (args=0x7fffdd163388) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_tricky.c:315
#5  0x00007fb8d3edc990 in execzlefunc (func=0x7fb8d4109c60, args=0x7fffdd163388, set_bindk=1) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_main.c:1299
#6  0x00007fb8d3eecf05 in bin_zle_call (name=0x7fb8d515ead0 "zle", args=0x7fffdd163388, ops=0x7fffdd1633e0, func=0 '\0') at /usr/src/zsh-4.3.9/src/Src/Zle/zle_thingy.c:700
#7  0x00007fb8d3eec030 in bin_zle (name=0x7fb8d515ead0 "zle", args=0x7fffdd163370, ops=0x7fffdd1633e0, func=0) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_thingy.c:381
#8  0x000000000040e883 in execbuiltin (args=0x7fb8d515ea40, bn=0x7fb8d410b120) at /usr/src/zsh-4.3.9/src/Src/builtin.c:439
#9  0x000000000042b20f in execcmd (state=0x7fffdd163da0, input=0, output=0, how=18, last1=2) at /usr/src/zsh-4.3.9/src/Src/exec.c:3066
#10 0x0000000000426d4b in execpline2 (state=0x7fffdd163da0, pcode=131, how=18, input=0, output=0, last1=0) at /usr/src/zsh-4.3.9/src/Src/exec.c:1561
#11 0x000000000042619e in execpline (state=0x7fffdd163da0, slcode=7170, how=18, last1=0) at /usr/src/zsh-4.3.9/src/Src/exec.c:1347
#12 0x00000000004259a3 in execlist (state=0x7fffdd163da0, dont_change_job=1, exiting=0) at /usr/src/zsh-4.3.9/src/Src/exec.c:1144
#13 0x0000000000425430 in execode (p=0xebed10, dont_change_job=1, exiting=0) at /usr/src/zsh-4.3.9/src/Src/exec.c:975
#14 0x000000000042e053 in runshfunc (prog=0xebed10, wrap=0x0, name=0x7fb8d515ea00 "widget-wrapper") at /usr/src/zsh-4.3.9/src/Src/exec.c:4416
#15 0x000000000042dd7f in doshfunc (shfunc=0xe9f580, doshargs=0x0, noreturnval=1) at /usr/src/zsh-4.3.9/src/Src/exec.c:4324
#16 0x00007fb8d3edcc7f in execzlefunc (func=0x7fb8d4108130, args=0x7fb8d410b6a8, set_bindk=0) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_main.c:1333
#17 0x00007fb8d3edbdf9 in zlecore () at /usr/src/zsh-4.3.9/src/Src/Zle/zle_main.c:1043
#18 0x00007fb8d3edc553 in zleread (lp=0x6c7e80, rp=0x0, flags=3, context=0) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_main.c:1206
#19 0x00007fb8d3ede912 in zle_main_entry (cmd=1, ap=0x7fffdd164440) at /usr/src/zsh-4.3.9/src/Src/Zle/zle_main.c:1836
#20 0x00000000004421d2 in zleentry (cmd=1) at /usr/src/zsh-4.3.9/src/Src/init.c:1255
#21 0x0000000000442c0c in inputline () at /usr/src/zsh-4.3.9/src/Src/input.c:278
#22 0x0000000000442a8a in ingetc () at /usr/src/zsh-4.3.9/src/Src/input.c:214
#23 0x0000000000438bb2 in ihgetc () at /usr/src/zsh-4.3.9/src/Src/hist.c:263
#24 0x000000000044ae98 in gettok () at /usr/src/zsh-4.3.9/src/Src/lex.c:677
#25 0x000000000044a65e in yylex () at /usr/src/zsh-4.3.9/src/Src/lex.c:364
#26 0x0000000000467e52 in parse_event () at /usr/src/zsh-4.3.9/src/Src/parse.c:451
#27 0x000000000043f787 in loop (toplevel=1, justonce=0) at /usr/src/zsh-4.3.9/src/Src/init.c:129
#28 0x000000000044269f in zsh_main (argc=1, argv=0x7fffdd164838) at /usr/src/zsh-4.3.9/src/Src/init.c:1406
#29 0x000000000040de27 in main (argc=1, argv=0x7fffdd164838) at /usr/src/zsh-4.3.9/src/Src/main.c:93

Is there some reason this is not an okay thing to do or that completion widgets
should be called differently?  (I'm working on improving predict-on.)

Observed on zsh 4.3.9 and 4.2.7 on x86_64.

:-Dylan

Re: crash when expand-or-complete widget wrapped

[Bart Schaefer]

On Apr 8, 12:49am, Dylan Alex Simon wrote:
}
} > > Try the patch below.  Doesn't actually find any useful completions,
} > > but at least fixes the infinite-but-for-memory-limits loop.
} >
} > Yes, that's correct, it was just a typo I introduced recently before
} > 4.3.9.
} 
} Is there some reason this is not an okay thing to do or that
} completion widgets should be called differently? (I'm working on
} improving predict-on.)

It shouldn't outright crash, but ...

You can't call completion widgets from non-completion widgets, in the
general case.  In fact you generally can't call completion widgets
with "zle widget_name", because completion widgets don't share the
normal editor state.

Instead you need to create your wrapper widget as a completion widget
itself, and then call the function that implements the other widget
directly, such as:

widget-wrapper () {
  $WIDGET "$@"
}
zle -C widget-wrapper expand-or-complete widget-wrapper

See the implementations of _complete_help or _complete_debug for
examples, note that they set things up and then call _main_complete
directly.

Yes, this means there is no way to "wrap" a built-in completion widget.

I think you can call completion widgets from normal widgets by pushing
keystrokes with "zle -U" and then invoking "zle recursive-edit" but I'm
not entirely sure it will always work.

Re: crash when expand-or-complete widget wrapped

[Dylan Alex Simon]

> > Is there some reason this is not an okay thing to do or that completion widgets
> > should be called differently?  (I'm working on improving predict-on.)
> 
> You can't call completion widgets from non-completion widgets, in the
> general case.  In fact you generally can't call completion widgets
> with "zle widget_name", because completion widgets don't share the
> normal editor state.

> zle -C widget-wrapper expand-or-complete widget-wrapper

Sorry, I guess my question was a little confused (I shouldn't have said
"completion widgets").  As far as I understand, expand-or-complete and
complete-word are not themselves actual completion widgets (they aren't
generating completion lists) but rather zle widgets that invoke completion.

In fact, complete-word is invoked exactly as "zle complete-word" from
insert-and-predict in predict-on (which works fine).  The problem seems only
to happen when you invoke the dotted version "zle .expand-or-complete" rather
than "zle expand-and-complete".  That is, if I create a new widget to wrap
expand-and-complete that just does "zle expand-and-complete -- "$@"" and bind
tab to it instead, things work fine.

:-Dylan

Re: crash when expand-or-complete widget wrapped

[Bart Schaefer]

On Apr 11, 12:31am, Dylan Alex Simon wrote:
} Subject: Re: crash when expand-or-complete widget wrapped
}
} > > Is there some reason this is not an okay thing to do or that
} > > completion widgets should be called differently? (I'm working on
} > > improving predict-on.)
} >
} > You can't call completion widgets from non-completion widgets, in the
} > general case.  In fact you generally can't call completion widgets
} > with "zle widget_name", because completion widgets don't share the
} > normal editor state.
} 
} > zle -C widget-wrapper expand-or-complete widget-wrapper
} 
} Sorry, I guess my question was a little confused (I shouldn't have
} said "completion widgets"). As far as I understand, expand-or-complete
} and complete-word are not themselves actual completion widgets (they
} aren't generating completion lists) but rather zle widgets that invoke
} completion.

No, you spoke correctly: "completion widgets" are zle widgets (either
built-in or user-defined) to invoke completion; "completion functions"
are shell functions intended to be bound to completion widgets with
"zle -C"; and "completers" are shell functions that generate matches
for use by completion functions.  However, those terms are not always
used with very much precision.  Also, "completion system" refers to
the entire set of completers and completion functions as installed by
"compinit" (and thereby turned into completion widgets).

What's confused is my memory.  It should be OK to invoke a completion
widget from a normal widget; what doesn't work is invoking a normal
widget from a completion function (or from a completer).

} The problem seems only to happen when you invoke the dotted version
} "zle .expand-or-complete" rather than "zle expand-and-complete".

I'm unable to reproduce the crash, but I believe I know at least part
of what's going on here.

"compinit" replaces the built-in complete-word, expand-or-complete,
etc., widgets with the completion function bindings that call through
to _main_complete to invoke the completion system.

However, one of the basic rules of zle widgets is that the dotted
versions are immutable, so compinit can't replace those.  When you
define your widget-wrapper to invoke "zle .$WIDGET", you're calling
directly through to the built-in widgets, bypassing the completion
system entirely and calling into the old compctl system.

Those "old" widgets nevertheless share some guts with the builtins
that form the core of the "new" completion system, and I suspect that
there's an unexpected conflict caused by insert-and-predict playing
with BUFFER and/or CURSOR and/or the values in the compstate hash.