I got it working using the hook method: I added another script with
`#compdef -first-` that saves the words array.

The thing is, that now I'm faced with a different problem:
When the command line is `./ps.py`, completions are generated correctly.
When the command line is `python ps.py`, the actual completions are joined
by the list of files in the current directory.

This appears to be a byproduct of going through the `_python`
auto-completion script.

I was wondering if there is a way to remove existing completion matches
before adding my own to make sure I provide only correct matches.
Also, might this be a bug in the `_python` completion script?

Thanks for all the help,
Dan

On Thu, May 7, 2020 at 11:40 PM Daniel Shahaf <d.s@daniel.shahaf.name>
wrote:

> > > > > After _python returned, my script was invoked again and printed
> > > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
> > >
> > > I'm not sure offhand where this comes from.  $context, $funcstack, et
> al
> > > may have clues.
> >
> >
> > I may have neglected to say that I aliased `python` to `python3`, so it
> > makes sense since you said that words is the result of alias expansion.
>
> Yes, that'd be it.
>
> > > You might try looking up the place that originally populates $words and
> > > making it stash a copy of $words in another, read-only array; then
> you'll
> > > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > > use of *_orig keys in ${hook_com}.)
> >
> >
> > I tried looking at things related to `vcs_info`, and I saw they were
> > outside of the Completion directory.
> > I was able to find the place where the `*_orig` keys are set, but
> couldn't
> > find where they are used.
> > It would be nice to get some context (or reference) about how this part
> of
> > zsh connects to compsys, and how variables like `hook_com` become
> available
> > to completion functions.
>
> vcs_info has nothing to do with compsys.  Sorry for the unclarity.
>
> *_orig variables would be used by hooks in people's dotfiles.  There
> are examples of hooks in Misc/vcs_info-examples, though they don't
> access the *_orig keys.
>
> I was trying to explain by way of analogy: just like vcs_info sets
> ${hook_com[revision_orig]}, in order that if two hooks are installed
> the later one be able to access the original value even if the
> earlier one had modified ${hook_com[revision]}, so I imagined
> a ${words_orig} array, to let your completion function access the
> value of $words before the _arguments call in _python shifted some
> elements off it.
>
> > > Something along these lines, I think:
> > >
> > >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> > >     words2[-1]=${words2[-1]%x}
> > >     words2[1,${words2[(I)(;|&&|…)]}]=()
> >
> >
> > A lot for me to unpack here and in the paragraphs that follow.
> > I'll try to understand as much as I can, but will probably ask about it
> > later.
>
> Sure, feel free.
>
> «${foo[(I)bar]}» expands to an index into the array $foo; which index is
> determined by the pattern «bar».  (You can get quick help by tab
> completing after the opening round parenthesis; the full description of
> ${[(I)]} is in the manual.)
>
> Do consider the other two alternatives, though.  Just because it's
> _possible_ to reimplement the $BUFFER-to-$words transformation doesn't
> imply that's the best way forward.
>
> Cheers,
>
> Daniel
>
> > Thanks!
> > Dan
> >
> > On Tue, May 5, 2020 at 7:49 PM Daniel Shahaf <d.s@daniel.shahaf.name>
> wrote:
> >
> > > > > > Well, for starters, if you «shift words», you should decrement
> > > CURRENT
> > > > > > as well.
> > > > > >
> > > > > > However, _python uses the «*::…» form of an _arguments optspec,
> > > > > > which should take care of $words/$NUMERIC for you.  I assume
> the
> > > reason
> > > > > > it didn't is that you used «compdef -p».  Try -P instead?  If
> I'm not
> > > > > > mistaken, that would also handle «python … script.py <TAB>» for
> you
> > > > > > (where the ellipsis stands for python's --options).
> > > > >
> > > > >
> > > > > I tested the 4 possible options, and found that switching -p with
> -P
> > > > > doesn't change anything, but shifting words and decrementing
> CURRENT
> > > > > works!  But... I feel like it's a hack I'm not supposed to do
> > > > > (correct me if I'm wrong).
> > >
> > > Those parameters are documented, and manipulated by a number of
> > > functions (including _python, as it happens), so you _are_ using the
> API.
> > > See also the 'compset' builtin.
> > >
> > > However, as mentioned, I suspect that hardcoding a skip/decrement of
> one
> > > element won't DTRT if python is invoked with options (e.g., «python
> > > --foo script.py <TAB>») — not unless you reimplement _python's parsing
> > > of those options and any arguments to them.
> > >
> > > > > I now tried using the BUFFER variable instead of words (since it
> > > remains
> > > > > unchanged), but found that it also has caveats: The BUFFER variable
> > > > > contains the raw value of the line, whereas the words array
> contains
> > > the
> > > > > words after expansions (I think that's the right term).
> > >
> > > Partly.  $words has undergone alias expansion, but not further
> > > expansions.  In particular, the words are still quoted.
> > >
> > > > > To demonstrate this I tried running again completion for `python
> ps,py`
> > > > > with `#compdef -P *`, and printing out BUFFER and words. The
> debug
> > > log
> > > > > (and my terminal) showed the following:
> > > > > First of all _python was called and dispatched my script which
> printed
> > > > > `BUFFER=python ps.py` and `words=ps.py`.
> > >
> > > That's what I'd expect.
> > >
> > > > > After _python returned, my script was invoked again and printed
> > > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
> > >
> > > I'm not sure offhand where this comes from.  $context, $funcstack, et
> al
> > > may have clues.
> > >
> > > > > I do think the right solution is to be dispatched by _python with
> > > > > `#compdef -P *.py`, but to be able to run I need the words array as
> > > > > seen by _python
> > >
> > > Once an element is shifted off $words, it's lost forever (free() is
> > > called on it), so there'll be no way to access a "previous" value of
> $words
> > > unless a copy had been made somewhere.
> > >
> > > Moreover, consider that python may be itself wrapped by some
> precommand:
> > > for example, «env X=Y python», «sudo -u foo python», or «ssh foo
> python».
> > >
> > > You might try looking up the place that originally populates $words and
> > > making it stash a copy of $words in another, read-only array; then
> you'll
> > > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > > use of *_orig keys in ${hook_com}.)
> > >
> > > > > or a to understand how BUFFER can be expanded/converted to its
> > > correct
> > > > > form.  If you have any input on this, I would very much appreciate
> > > > > it.
> > >
> > > Something along these lines, I think:
> > >
> > >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> > >     words2[-1]=${words2[-1]%x}
> > >     words2[1,${words2[(I)(;|&&|…)]}]=()
> > >
> > > To understand the second line, see addx() in Src/Zle/zle_tricky.c.
> With
> > > that out of the way, the first line's a textbook use of the ${(z)}
> > > tokenizer.  The third line just throws out everything until the start
> of
> > > the current _simple command_ (see zshmisc(1)) — or, at least, it will
> > > once the ellipsis is filled out.
> > >
> > > Caveats: that snippet doesn't do alias expansion, doesn't try to deal
> > > with ${RBUFFER}, doesn't try to deal with SHORT_LOOPS, doesn't try to
> > > deal with precommands and precommand assignments, should be changed to
> > > use zZ+c+ if the INTERACTIVE_COMMENTS option is set…
> > >
> > > Although it's a plugin that I worked on myself, I should probably
> > > mention that zsh-syntax-highlighting has faced the same problem and you
> > > and implemented a solution to it, which handles a number of the above
> > > caveats.
> > >
> > > >
> > >
> > > There's also a third approach: instead of trying to run the binary,
> make
> > > it stash the output in a well-known location.  For example,
> /usr/bin/foo
> > > could store the output in /usr/share/argcomplete/foo.  (I've always
> > > wanted to standardize _some_ solution to this problem; I'm tired of
> > > writing completion functions by hand…)
> > >
> > > Cheers,
> > >
> > > Daniel
> > >
> >
> >
>
>

-- 
Dan Arad
dan1994@gmail.com