compadd -Q -U completes $(( without inserting upon it

compadd -Q -U completes $(( without inserting upon it

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Hello,
I have two completers, first one:

PREFIX='open'
SUFFIX=''
IPREFIX=""
ISUFFIX=""
words=( 'open' )
builtin compadd -Q -U 'open 0 + 1 close'

second one:
PREFIX='$(('
SUFFIX=''
IPREFIX=""
ISUFFIX=""
words=( '$((' )
builtin compadd -Q -U '$(( 0 + 1 ))'

With first one, if I do "open<TAB>", I get: "open 0 + 1 close". With
second one, when I do "$((<TAB>", I get "$(($(( 0 + 1 ))'.

This looks like zsh source has embedded tweaks to make math
($compstate[context] is "math" in second case) completing work more
reasonably. Is there a way to overcome this?

Video:
https://asciinema.org/a/9o1wwp76t0z2j2fz15aoucwym

To test the attached files:

autoload compadd_test ; zle -N compadd_test
autoload compadd_test2 ; zle -N compadd_test2

then:

zstyle ':completion:*' completer compadd_test _complete

or:

zstyle ':completion:*' completer compadd_test2 _complete

Best regards,
Sebastian Gniazdowski

[-- Attachment #2: compadd_test --]
[-- Type: application/octet-stream, Size: 119 bytes --]

PREFIX='open'
SUFFIX=''
IPREFIX=""
ISUFFIX=""
words=( 'open' )

builtin compadd -Q -U 'open 0 + 1 close'

# vim:ft=zsh

[-- Attachment #3: compadd_test2 --]
[-- Type: application/octet-stream, Size: 113 bytes --]

PREFIX='$(('
SUFFIX=''
IPREFIX=""
ISUFFIX=""
words=( '$((' )

builtin compadd -Q -U '$(( 0 + 1 ))'

# vim:ft=zsh

Re: compadd -Q -U completes $(( without inserting upon it

[Bart Schaefer]

On Jan 16,  6:25pm, Sebastian Gniazdowski wrote:
}
} This looks like zsh source has embedded tweaks to make math
} ($compstate[context] is "math" in second case) completing work more
} reasonably. Is there a way to overcome this?

Internally, when you invoke a completion widget, the ZLE buffer gets
analyzed and divided into three sections:
 1) The words array, which is the context where completion happens.
 2) Anything before the words array, which completion is not allowed
    to modify.
 3) Anything after the words array, similarly immutable.

In "ordinary" command completion, (2) and (3) are empty.

When you complete after a token such as $( or $(( that begins a context,
those tokens are part of the immutable section that precedes the array
of words.  You can assign to the words array to change how completion
interprets the set of possible matches, but anything you subsequently do
has to fit into the part of $BUFFER corresponding to the original $words.

You can make the mutable part smaller and expand the immutable parts by
calling "compset -q", but nobody considered before that the reverse
might also be interesting.  The internals of "compset" are a bit of a
nightmare, programmed by a brilliant Polish student who was fond of
abbreviated variable names and rarely wrote comments.  (The comments
there are others trying mostly unsuccessfully to untangle his logic.)

I've already outlined for you two ways to overcome this:

A) Capture the set of matches using "compadd -O", then remove the bits
that overlap with the immutable parts of the buffer before "compadd"
again of what remains.  (It's possible I've given you bad advice
regarding the need for the -U option; try it without that as well.)

B) Don't invoke completion directly; instead, invoke a normal editing
widget to modify the buffer so the tokens do not begin a context, then
call the completion widget, and finally clean up the buffer again when
the completion widget returns.

The tricky bit of (A) is keeping track of which part of the matches is
really "immutable" and which part is not.  The unfortunate bit of (B)
is that it can't work at all as a completer function (zstyle element).

There's also option (C) write your own completion system from scratch.
When the current compsys was invented, the (z) and (Z) parameter flags
didn't exist, so it was felt that it would be easier to allow the
exiting completion internals to be callable to do most of that work.  
We also didn't have keymaps at that point, nor "zle -M" / "zle -R".
A lot of what the completion system used to be the only way to do, can
probably now be done in the shell language.

Re: compadd -Q -U completes $(( without inserting upon it

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

On 16 January 2016 at 23:55, Bart Schaefer <schaefer@brasslantern.com> wrote:
> B) Don't invoke completion directly; instead, invoke a normal editing
> widget to modify the buffer so the tokens do not begin a context, then
> call the completion widget, and finally clean up the buffer again when
> the completion widget returns.

I went this path and it was quite easy. I have zew-complete-shell-word
widget (attached) that does a simple grep:
        __zew_csw_found=(
"${(@M)historywords:#(#i)$__zew_csw_left*$__zew_csw_right}" )

then zle -M displays the results and Alt-h/H navigates among them. One
minute video:

https://asciinema.org/a/9smut3m7l6njvwfhrkfb6emdp

> The unfortunate bit of (B)
> is that it can't work at all as a completer function (zstyle element).

So no automatic multi-column zle -M, no actual highlighting (tried to
pass ANSI codes to zle -M but it doesn't process them), no zstyles
that configure pagination etc. These are the drawbacks? All this is in
the queue to implement, e.g. I currently display LINES / 3 matches and
there is no access to matches beyond that.

Code is at:

https://github.com/psprint/zsh-editing-workbench/blob/master/zew-complete-shell-word

only 74 lines for a robust _history_complete_older

Thanks,
Sebastian Gniazdowski

[-- Attachment #2: zew-complete-shell-word --]
[-- Type: application/octet-stream, Size: 2088 bytes --]

emulate -LR zsh
setopt typesetsilent extendedglob noshortloops

# Prepare output variables for zew-process-buffer
local ZEW_PB_WORDS ZEW_PB_WORDS_BEGINNINGS ZEW_PB_SPACES 
local ZEW_PB_SELECTED_WORD ZEW_PB_LEFT ZEW_PB_RIGHT

autoload zew-process-buffer
zew-process-buffer "$BUFFER"

typeset -g __zew_csw_index __zew_csw_left __zew_csw_right
typeset -ga __zew_csw_found
__zew_csw_found=( "$reply[@]" )

# Consecutive call?
if [ "${WIDGET%-backwards}" = "${LASTWIDGET%-backwards}" ]; then
    if [[ "$WIDGET" != *-backwards ]]; then
        (( __zew_csw_index ++ ))
    else
        (( __zew_csw_index -- ))
    fi
else
    if [[ "$WIDGET" != *-backwards ]]; then
        __zew_csw_index="1"
    else
        # Will get changed into $to_display limit
        __zew_csw_index="0"
    fi
    __zew_csw_left="$ZEW_PB_LEFT"
    __zew_csw_right="$ZEW_PB_RIGHT"
    __zew_csw_found=( )
fi

# Find history words matching $left ... $right
if [ "$#__zew_csw_found" -eq "0" ]; then
    typeset -U __zew_csw_found
    repeat 1; do
        __zew_csw_found=( "${(@M)historywords:#(#i)$__zew_csw_left*$__zew_csw_right}" )
        # Remember for consecutive calls
        reply=( "$__zew_csw_found[@]" )
    done
fi

# Guard values of the index
integer to_display=$(( LINES / 2 ))
[ "$to_display" -gt "$#__zew_csw_found" ] && to_display="$#__zew_csw_found"
[ "$__zew_csw_index" -le 0 ] && __zew_csw_index="$to_display"
[ "$__zew_csw_index" -gt "$to_display" ] && __zew_csw_index=1

# Display matches
typeset -a disp_list
disp_list=( "${(@)__zew_csw_found[1,to_display]}" )
disp_list[__zew_csw_index]="> ${disp_list[__zew_csw_index]} <"
zle -M -- "${(F)disp_list}"

# Regenerate command line
buf=""
integer nwords="${#ZEW_PB_WORDS}"
integer newcursor=0
for (( i=1; i<=nwords; i++ )); do
    if [ "$i" = "$ZEW_PB_SELECTED_WORD" ]; then
        buf+="${ZEW_PB_SPACES[i]}${__zew_csw_found[__zew_csw_index]}"
        newcursor="$#buf"
    else
        buf+="${ZEW_PB_SPACES[i]}${ZEW_PB_WORDS[i]}"
    fi
done

# Set command line
BUFFER="$buf"
# Move cursor to the end of word
CURSOR="$newcursor"

# vim:ft=zsh

[-- Attachment #3: zew-process-buffer --]
[-- Type: application/octet-stream, Size: 3070 bytes --]

# Input:
# $1 - buffer to process
#
# Output:
# ZEW_PB_WORDS - split of "$1" into shell words; array
# ZEW_PB_WORDS_BEGINNINGS - indexes of first letters of corresponding ZEW_PB_WORDS in ZEW_PB_WORDS
# ZEW_PB_SPACES - white spaces before corresponding ZEW_PB_WORDS in ZEW_PB_WORDS
# ZEW_PB_SELECTED_WORD - index in ZEW_PB_WORDS pointing to word activated by CURSOR position
# ZEW_PB_LEFT - left part of active word
# ZEW_PB_RIGHT - right part of active word
#

emulate -LR zsh
setopt typesetsilent extendedglob noshortloops

local MBEGIN MEND MATCH mbegin mend match

local buf="$1"
ZEW_PB_WORDS=( "${(Z+n+)BUFFER}" )
ZEW_PB_SPACES=( )
ZEW_PB_WORDS_BEGINNINGS=( )
ZEW_PB_SELECTED_WORD="-1"

integer nwords="${#ZEW_PB_WORDS}"

# Remove ZEW_PB_WORDS one by one, counting characters,
# computing beginning of each word, to find
# place to break the word into 2 halves (for
# complete_in_word option)

local i word
integer char_count=0

# (Z) handles spaces nicely, but we need them for the user
# Also compute words beginnings and the selected word
for (( i=1; i<=nwords; i++ )); do
    # Remove spurious space generated by Z-flag when
    # input is an unbound '$(' (happens with zsh < 5.1)
    # and also real spaces gathered by an unbound '$(',
    # to handle them in a way normal to this loop
    ZEW_PB_WORDS[i]="${ZEW_PB_WORDS[i]%% ##}"
    word="${ZEW_PB_WORDS[i]}"

    # In general, $buf can start with white spaces
    # We will not search for them, but instead for
    # leading character of current shell word,
    # negated. This is an ambition to completely
    # avoid character classes

    # Remove white spaces
    buf="${buf##(#m)[^$word[1]]#}"
    # Count them
    char_count=char_count+"$#MATCH"
    # This is the beginning of current word
    ZEW_PB_WORDS_BEGINNINGS[i]=$(( char_count + 1 ))
    # Remember the spaces
    ZEW_PB_SPACES[i]="$MATCH"

    # Remove the word
    MATCH=""
    buf="${buf#(#m)$word}"

    # If shell word not found, return. This shoudln't happen
    [ -z "$MATCH" ] && return 0

    # Spaces point to previous shell word
    # Visual cursor right after spaces (-ge) -> not enough to select previous word (-gt required)
    [[ "$ZEW_PB_SELECTED_WORD" -eq "-1" && "$char_count" -gt "$CURSOR" ]] && ZEW_PB_SELECTED_WORD=$(( i-1 ))

    # Actual characters point to current shell word
    # Visual cursor right after letters (-ge) -> enough to select current word
    char_count=char_count+"$#word"
    [[ "$ZEW_PB_SELECTED_WORD" -eq "-1" && "$char_count" -ge "$CURSOR" ]] && ZEW_PB_SELECTED_WORD="$i"
done 

# What's left in $buf can be only white spaces
char_count=char_count+"$#buf"
ZEW_PB_SPACES[i]="$buf"

# Visual cursor right after spaces (-ge) -> enough to select last word
[[ "$ZEW_PB_SELECTED_WORD" -eq "-1" && "$char_count" -ge "$CURSOR" ]] && ZEW_PB_SELECTED_WORD=$(( i-1 ))

# Divide active word into two halves
integer diff=$(( CURSOR - ZEW_PB_WORDS_BEGINNINGS[ZEW_PB_SELECTED_WORD] + 1 ))
word="${ZEW_PB_WORDS[ZEW_PB_SELECTED_WORD]}"
ZEW_PB_LEFT="${word[1,diff]}"
ZEW_PB_RIGHT="${word[diff+1,-1]}"

# vim:ft=zsh