Fwd: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

Fwd: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Dan Arad]

[-- Attachment #1: Type: text/plain, Size: 6354 bytes --]

Hi there,

I'm forwarding a conversation I started by mistake on the wrong mailing
list (Sorry, I'm new to the concept).
Any help on the topic would be greatly appreciated!

Cheers,
Dan

---------- Forwarded message ---------
From: Daniel Shahaf <d.s@daniel.shahaf.name>
Date: Mon, May 4, 2020 at 4:18 PM
Subject: Re: Help Request/Bug Report: comparguments causes _arguments to
fail in certain cases
To: Dan Arad <dan1994@gmail.com>


Good morning Dan,

Would you re-send that to the mailing list, please?  You sent it only
to me.

Thanks for tracking down the rich text issue — it's much easier to talk
this way. ☺

Cheers,

Daniel


Dan Arad wrote on Mon, 04 May 2020 09:46 +0300:
> First of all, thank you for your swift response!
>
> For future reference, you should have addressed your question to
> > zsh-users@, as the answer to it doesn't involve a change to zsh
itself.
>
>
> Noted! I searched online for the zsh bug tracker, and stumbled into this
> mail address.
>
> Furthermore, it would be helpful to configure your email client to
> > generate a text/plain part that doesn't surround every line by asterisks
> > and add spurious blank lines in code blocks.
> >
>
> I was trying to copy paste the stack overflow entry through Word, so it
> won't keep the web formatting, and that's probably why this happened...
>
> Well, for starters, if you «shift words», you should decrement CURRENT
> > as well.
> >
> > However, _python uses the «*::…» form of an _arguments optspec,
> > which should take care of $words/$NUMERIC for you.  I assume the reason
> > it didn't is that you used «compdef -p».  Try -P instead?  If I'm not
> > mistaken, that would also handle «python … script.py <TAB>» for you
> > (where the ellipsis stands for python's --options).
>
>
> I tested the 4 possible options, and found that switching -p with -P
> doesn't change anything, but shifting words and decrementing CURRENT
works!
> But... I feel like it's a hack I'm not supposed to do (correct me if I'm
> wrong).
>
> Your second paragraph made me look back at a previous solution (that
> doesn't work). I would be glad to discuss it if you have the time.
>
> I was using `#compdef -p *.py` before, and what I saw was that _python was
> being called before my script, and that my script didn't work well. What
> was actually happening was that _python was shifting words before I was
> being called, and that interfered with the logic of my script.
>
> In order to generate completions I rely on running what is currently
> present in the command line, and so this may cause my script to generate
no
> completions (i.e. `python ps.py` becomes `ps.py` which can't be run, and
> `python ./ps.py` becomes `./ps.py` which may or may not be run depending
on
> file permissions).
>
> I now tried using the BUFFER variable instead of words (since it remains
> unchanged), but found that it also has caveats: The BUFFER variable
> contains the raw value of the line, whereas the words array contains the
> words after expansions (I think that's the right term).
> To demonstrate this I tried running again completion for `python ps,py`
> with `#compdef -P *`, and printing out BUFFER and words. The debug log
(and
> my terminal) showed the following:
> First of all _python was called and dispatched my script which printed
> `BUFFER=python ps.py` and `words=ps.py`.
> After _python returned, my script was invoked again and printed
> `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
>
> I do think the right solution is to be dispatched by _python with
`#compdef
> -P *.py`, but to be able to run I need the words array as seen by _python
> or a to understand how BUFFER can be expanded/converted to its correct
form.
> If you have any input on this, I would very much appreciate it.
>
> Best Regards,
> Dan
>
>
> On Sun, May 3, 2020 at 7:58 PM Daniel Shahaf <d.s@daniel.shahaf.name>
wrote:
>
> > Dan Arad wrote on Sun, 03 May 2020 10:40 +0300:
> > > Hi there,
> > >
> > > From what I understand, bug reports should be mailed here. If I'm
wrong
> > I'd
> > > be happy to be redirected to the correct medium.
> >
> > For future reference, you should have addressed your question to
> > zsh-users@, as the answer to it doesn't involve a change to zsh itself.
> > Furthermore, it would be helpful to configure your email client to
> > generate a text/plain part that doesn't surround every line by asterisks
> > and add spurious blank lines in code blocks.
> >
> > > What I want to report is probably not a bug, and stems from my
> > > misunderstanding of compsys. Any help will be greatly appreciated.
> > > The following is a copy paste of the a this question
> > > <
> >
https://stackoverflow.com/questions/61560687/comparguments-causes-arguments-to-fail-in-certain-cases

> > >
> > > I asked on stack overflow:
> > >
> >
> > Thanks for the cross-reference.
> >
> > > *1. The script is invoked directly (e.g. ~/script.py)*
> > >
> > > *2. The script is invoked through python (e.g. python script.py)*
> > >
> > > *3. The script is invoked as a python module (e.g. python -m script)*
> > >
> > >
> > > *I have so far managed to successfully handle the first case, but the
> > > second case fails to retrieve any completions. Using zsh completion
debug
> > > log I was able to see where things went wrong:*
> > ⋮
> > > *EDIT:*
> > >
> > > *In order to temporarily bypass the problem I tried adding a shift
words
> > > before calling _arguments. This caused comparguments to succeed (!),
but
> > > still causes _arguments to fail with a no arguments message later on.*
> > >
> > > *I added the log for this case to the gist linked above.*
> >
> > Well, for starters, if you «shift words», you should decrement CURRENT
> > as well.
> >
> > However, _python uses the «*::…» form of an _arguments optspec,
> > which should take care of $words/$NUMERIC for you.  I assume the reason
> > it didn't is that you used «compdef -p».  Try -P instead?  If I'm not
> > mistaken, that would also handle «python … script.py <TAB>» for you
> > (where the ellipsis stands for python's --options).
> >
> > Cheers,
> >
> > Daniel
> >
>
>



-- 
Dan Arad
dan1994@gmail.com

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Daniel Shahaf]

> > > Well, for starters, if you «shift words», you should decrement CURRENT  
> > > as well.
> > >
> > > However, _python uses the «*::…» form of an _arguments optspec,
> > > which should take care of $words/$NUMERIC for you.  I assume the reason
> > > it didn't is that you used «compdef -p».  Try -P instead?  If I'm not
> > > mistaken, that would also handle «python … script.py <TAB>» for you
> > > (where the ellipsis stands for python's --options).  
> >
> >
> > I tested the 4 possible options, and found that switching -p with -P
> > doesn't change anything, but shifting words and decrementing CURRENT  
> > works!  But... I feel like it's a hack I'm not supposed to do
> > (correct me if I'm wrong).

Those parameters are documented, and manipulated by a number of
functions (including _python, as it happens), so you _are_ using the API.
See also the 'compset' builtin.

However, as mentioned, I suspect that hardcoding a skip/decrement of one
element won't DTRT if python is invoked with options (e.g., «python
--foo script.py <TAB>») — not unless you reimplement _python's parsing
of those options and any arguments to them.

> > I now tried using the BUFFER variable instead of words (since it remains
> > unchanged), but found that it also has caveats: The BUFFER variable
> > contains the raw value of the line, whereas the words array contains the
> > words after expansions (I think that's the right term).

Partly.  $words has undergone alias expansion, but not further
expansions.  In particular, the words are still quoted.

> > To demonstrate this I tried running again completion for `python ps,py`
> > with `#compdef -P *`, and printing out BUFFER and words. The debug log  
> > (and my terminal) showed the following:
> > First of all _python was called and dispatched my script which printed
> > `BUFFER=python ps.py` and `words=ps.py`.

That's what I'd expect.

> > After _python returned, my script was invoked again and printed
> > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).

I'm not sure offhand where this comes from.  $context, $funcstack, et al
may have clues.

> > I do think the right solution is to be dispatched by _python with  
> > `#compdef -P *.py`, but to be able to run I need the words array as
> > seen by _python

Once an element is shifted off $words, it's lost forever (free() is
called on it), so there'll be no way to access a "previous" value of $words
unless a copy had been made somewhere.

Moreover, consider that python may be itself wrapped by some precommand:
for example, «env X=Y python», «sudo -u foo python», or «ssh foo python».

You might try looking up the place that originally populates $words and
making it stash a copy of $words in another, read-only array; then you'll
be able to use «compset -P» and access that array.  (Compare vcs_info's
use of *_orig keys in ${hook_com}.)

> > or a to understand how BUFFER can be expanded/converted to its correct  
> > form.  If you have any input on this, I would very much appreciate
> > it.

Something along these lines, I think:

    local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
    words2[-1]=${words2[-1]%x}
    words2[1,${words2[(I)(;|&&|…)]}]=()

To understand the second line, see addx() in Src/Zle/zle_tricky.c.  With
that out of the way, the first line's a textbook use of the ${(z)}
tokenizer.  The third line just throws out everything until the start of
the current _simple command_ (see zshmisc(1)) — or, at least, it will
once the ellipsis is filled out.  

Caveats: that snippet doesn't do alias expansion, doesn't try to deal
with ${RBUFFER}, doesn't try to deal with SHORT_LOOPS, doesn't try to
deal with precommands and precommand assignments, should be changed to
use zZ+c+ if the INTERACTIVE_COMMENTS option is set…

Although it's a plugin that I worked on myself, I should probably
mention that zsh-syntax-highlighting has faced the same problem and you
and implemented a solution to it, which handles a number of the above
caveats.

>

There's also a third approach: instead of trying to run the binary, make
it stash the output in a well-known location.  For example, /usr/bin/foo
could store the output in /usr/share/argcomplete/foo.  (I've always
wanted to standardize _some_ solution to this problem; I'm tired of
writing completion functions by hand…)

Cheers,

Daniel

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Dan Arad]

[-- Attachment #1: Type: text/plain, Size: 6176 bytes --]

>
> > > After _python returned, my script was invoked again and printed
> > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
>
> I'm not sure offhand where this comes from.  $context, $funcstack, et al
> may have clues.


I may have neglected to say that I aliased `python` to `python3`, so it
makes sense since you said that words is the result of alias expansion.

You might try looking up the place that originally populates $words and
> making it stash a copy of $words in another, read-only array; then you'll
> be able to use «compset -P» and access that array.  (Compare vcs_info's
> use of *_orig keys in ${hook_com}.)


I tried looking at things related to `vcs_info`, and I saw they were
outside of the Completion directory.
I was able to find the place where the `*_orig` keys are set, but couldn't
find where they are used.
It would be nice to get some context (or reference) about how this part of
zsh connects to compsys, and how variables like `hook_com` become available
to completion functions.


> Something along these lines, I think:
>
>     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
>     words2[-1]=${words2[-1]%x}
>     words2[1,${words2[(I)(;|&&|…)]}]=()


A lot for me to unpack here and in the paragraphs that follow.
I'll try to understand as much as I can, but will probably ask about it
later.

Thanks!
Dan

On Tue, May 5, 2020 at 7:49 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:

> > > > Well, for starters, if you «shift words», you should decrement
> CURRENT
> > > > as well.
> > > >
> > > > However, _python uses the «*::…» form of an _arguments optspec,
> > > > which should take care of $words/$NUMERIC for you.  I assume the
> reason
> > > > it didn't is that you used «compdef -p».  Try -P instead?  If I'm not
> > > > mistaken, that would also handle «python … script.py <TAB>» for you
> > > > (where the ellipsis stands for python's --options).
> > >
> > >
> > > I tested the 4 possible options, and found that switching -p with -P
> > > doesn't change anything, but shifting words and decrementing CURRENT
> > > works!  But... I feel like it's a hack I'm not supposed to do
> > > (correct me if I'm wrong).
>
> Those parameters are documented, and manipulated by a number of
> functions (including _python, as it happens), so you _are_ using the API.
> See also the 'compset' builtin.
>
> However, as mentioned, I suspect that hardcoding a skip/decrement of one
> element won't DTRT if python is invoked with options (e.g., «python
> --foo script.py <TAB>») — not unless you reimplement _python's parsing
> of those options and any arguments to them.
>
> > > I now tried using the BUFFER variable instead of words (since it
> remains
> > > unchanged), but found that it also has caveats: The BUFFER variable
> > > contains the raw value of the line, whereas the words array contains
> the
> > > words after expansions (I think that's the right term).
>
> Partly.  $words has undergone alias expansion, but not further
> expansions.  In particular, the words are still quoted.
>
> > > To demonstrate this I tried running again completion for `python ps,py`
> > > with `#compdef -P *`, and printing out BUFFER and words. The debug
> log
> > > (and my terminal) showed the following:
> > > First of all _python was called and dispatched my script which printed
> > > `BUFFER=python ps.py` and `words=ps.py`.
>
> That's what I'd expect.
>
> > > After _python returned, my script was invoked again and printed
> > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
>
> I'm not sure offhand where this comes from.  $context, $funcstack, et al
> may have clues.
>
> > > I do think the right solution is to be dispatched by _python with
> > > `#compdef -P *.py`, but to be able to run I need the words array as
> > > seen by _python
>
> Once an element is shifted off $words, it's lost forever (free() is
> called on it), so there'll be no way to access a "previous" value of $words
> unless a copy had been made somewhere.
>
> Moreover, consider that python may be itself wrapped by some precommand:
> for example, «env X=Y python», «sudo -u foo python», or «ssh foo python».
>
> You might try looking up the place that originally populates $words and
> making it stash a copy of $words in another, read-only array; then you'll
> be able to use «compset -P» and access that array.  (Compare vcs_info's
> use of *_orig keys in ${hook_com}.)
>
> > > or a to understand how BUFFER can be expanded/converted to its
> correct
> > > form.  If you have any input on this, I would very much appreciate
> > > it.
>
> Something along these lines, I think:
>
>     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
>     words2[-1]=${words2[-1]%x}
>     words2[1,${words2[(I)(;|&&|…)]}]=()
>
> To understand the second line, see addx() in Src/Zle/zle_tricky.c.  With
> that out of the way, the first line's a textbook use of the ${(z)}
> tokenizer.  The third line just throws out everything until the start of
> the current _simple command_ (see zshmisc(1)) — or, at least, it will
> once the ellipsis is filled out.
>
> Caveats: that snippet doesn't do alias expansion, doesn't try to deal
> with ${RBUFFER}, doesn't try to deal with SHORT_LOOPS, doesn't try to
> deal with precommands and precommand assignments, should be changed to
> use zZ+c+ if the INTERACTIVE_COMMENTS option is set…
>
> Although it's a plugin that I worked on myself, I should probably
> mention that zsh-syntax-highlighting has faced the same problem and you
> and implemented a solution to it, which handles a number of the above
> caveats.
>
> >
>
> There's also a third approach: instead of trying to run the binary, make
> it stash the output in a well-known location.  For example, /usr/bin/foo
> could store the output in /usr/share/argcomplete/foo.  (I've always
> wanted to standardize _some_ solution to this problem; I'm tired of
> writing completion functions by hand…)
>
> Cheers,
>
> Daniel
>


-- 
Dan Arad
dan1994@gmail.com

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Daniel Shahaf]

> > > > After _python returned, my script was invoked again and printed
> > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).  
> >
> > I'm not sure offhand where this comes from.  $context, $funcstack, et al
> > may have clues.  
> 
> 
> I may have neglected to say that I aliased `python` to `python3`, so it
> makes sense since you said that words is the result of alias expansion.

Yes, that'd be it.

> > You might try looking up the place that originally populates $words and
> > making it stash a copy of $words in another, read-only array; then you'll
> > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > use of *_orig keys in ${hook_com}.)  
> 
> 
> I tried looking at things related to `vcs_info`, and I saw they were
> outside of the Completion directory.
> I was able to find the place where the `*_orig` keys are set, but couldn't
> find where they are used.
> It would be nice to get some context (or reference) about how this part of
> zsh connects to compsys, and how variables like `hook_com` become available
> to completion functions.

vcs_info has nothing to do with compsys.  Sorry for the unclarity.

*_orig variables would be used by hooks in people's dotfiles.  There
are examples of hooks in Misc/vcs_info-examples, though they don't
access the *_orig keys.

I was trying to explain by way of analogy: just like vcs_info sets
${hook_com[revision_orig]}, in order that if two hooks are installed
the later one be able to access the original value even if the
earlier one had modified ${hook_com[revision]}, so I imagined
a ${words_orig} array, to let your completion function access the
value of $words before the _arguments call in _python shifted some
elements off it.

> > Something along these lines, I think:
> >
> >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> >     words2[-1]=${words2[-1]%x}  
> >     words2[1,${words2[(I)(;|&&|…)]}]=()  
> 
> 
> A lot for me to unpack here and in the paragraphs that follow.
> I'll try to understand as much as I can, but will probably ask about it
> later.

Sure, feel free.

«${foo[(I)bar]}» expands to an index into the array $foo; which index is
determined by the pattern «bar».  (You can get quick help by tab
completing after the opening round parenthesis; the full description of
${[(I)]} is in the manual.)

Do consider the other two alternatives, though.  Just because it's
_possible_ to reimplement the $BUFFER-to-$words transformation doesn't
imply that's the best way forward.

Cheers,

Daniel

> Thanks!
> Dan
> 
> On Tue, May 5, 2020 at 7:49 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> 
> > > > > Well, for starters, if you «shift words», you should decrement  
> > CURRENT  
> > > > > as well.
> > > > >
> > > > > However, _python uses the «*::…» form of an _arguments optspec,
> > > > > which should take care of $words/$NUMERIC for you.  I assume the  
> > reason  
> > > > > it didn't is that you used «compdef -p».  Try -P instead?  If I'm not
> > > > > mistaken, that would also handle «python … script.py <TAB>» for you
> > > > > (where the ellipsis stands for python's --options).  
> > > >
> > > >
> > > > I tested the 4 possible options, and found that switching -p with -P
> > > > doesn't change anything, but shifting words and decrementing CURRENT
> > > > works!  But... I feel like it's a hack I'm not supposed to do
> > > > (correct me if I'm wrong).  
> >
> > Those parameters are documented, and manipulated by a number of
> > functions (including _python, as it happens), so you _are_ using the API.
> > See also the 'compset' builtin.
> >
> > However, as mentioned, I suspect that hardcoding a skip/decrement of one
> > element won't DTRT if python is invoked with options (e.g., «python
> > --foo script.py <TAB>») — not unless you reimplement _python's parsing
> > of those options and any arguments to them.
> >  
> > > > I now tried using the BUFFER variable instead of words (since it  
> > remains  
> > > > unchanged), but found that it also has caveats: The BUFFER variable
> > > > contains the raw value of the line, whereas the words array contains  
> > the  
> > > > words after expansions (I think that's the right term).  
> >
> > Partly.  $words has undergone alias expansion, but not further
> > expansions.  In particular, the words are still quoted.
> >  
> > > > To demonstrate this I tried running again completion for `python ps,py`
> > > > with `#compdef -P *`, and printing out BUFFER and words. The debug  
> > log  
> > > > (and my terminal) showed the following:
> > > > First of all _python was called and dispatched my script which printed
> > > > `BUFFER=python ps.py` and `words=ps.py`.  
> >
> > That's what I'd expect.
> >  
> > > > After _python returned, my script was invoked again and printed
> > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).  
> >
> > I'm not sure offhand where this comes from.  $context, $funcstack, et al
> > may have clues.
> >  
> > > > I do think the right solution is to be dispatched by _python with
> > > > `#compdef -P *.py`, but to be able to run I need the words array as
> > > > seen by _python  
> >
> > Once an element is shifted off $words, it's lost forever (free() is
> > called on it), so there'll be no way to access a "previous" value of $words
> > unless a copy had been made somewhere.
> >
> > Moreover, consider that python may be itself wrapped by some precommand:
> > for example, «env X=Y python», «sudo -u foo python», or «ssh foo python».
> >
> > You might try looking up the place that originally populates $words and
> > making it stash a copy of $words in another, read-only array; then you'll
> > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > use of *_orig keys in ${hook_com}.)
> >  
> > > > or a to understand how BUFFER can be expanded/converted to its  
> > correct  
> > > > form.  If you have any input on this, I would very much appreciate
> > > > it.  
> >
> > Something along these lines, I think:
> >
> >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> >     words2[-1]=${words2[-1]%x}  
> >     words2[1,${words2[(I)(;|&&|…)]}]=()  
> >
> > To understand the second line, see addx() in Src/Zle/zle_tricky.c.  With
> > that out of the way, the first line's a textbook use of the ${(z)}
> > tokenizer.  The third line just throws out everything until the start of
> > the current _simple command_ (see zshmisc(1)) — or, at least, it will
> > once the ellipsis is filled out.
> >
> > Caveats: that snippet doesn't do alias expansion, doesn't try to deal
> > with ${RBUFFER}, doesn't try to deal with SHORT_LOOPS, doesn't try to
> > deal with precommands and precommand assignments, should be changed to
> > use zZ+c+ if the INTERACTIVE_COMMENTS option is set…
> >
> > Although it's a plugin that I worked on myself, I should probably
> > mention that zsh-syntax-highlighting has faced the same problem and you
> > and implemented a solution to it, which handles a number of the above
> > caveats.
> >  
> > >  
> >
> > There's also a third approach: instead of trying to run the binary, make
> > it stash the output in a well-known location.  For example, /usr/bin/foo
> > could store the output in /usr/share/argcomplete/foo.  (I've always
> > wanted to standardize _some_ solution to this problem; I'm tired of
> > writing completion functions by hand…)
> >
> > Cheers,
> >
> > Daniel
> >  
> 
> 

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Dan Arad]

[-- Attachment #1: Type: text/plain, Size: 8817 bytes --]

I got it working using the hook method: I added another script with
`#compdef -first-` that saves the words array.

The thing is, that now I'm faced with a different problem:
When the command line is `./ps.py`, completions are generated correctly.
When the command line is `python ps.py`, the actual completions are joined
by the list of files in the current directory.

This appears to be a byproduct of going through the `_python`
auto-completion script.

I was wondering if there is a way to remove existing completion matches
before adding my own to make sure I provide only correct matches.
Also, might this be a bug in the `_python` completion script?

Thanks for all the help,
Dan

On Thu, May 7, 2020 at 11:40 PM Daniel Shahaf <d.s@daniel.shahaf.name>
wrote:

> > > > > After _python returned, my script was invoked again and printed
> > > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
> > >
> > > I'm not sure offhand where this comes from.  $context, $funcstack, et
> al
> > > may have clues.
> >
> >
> > I may have neglected to say that I aliased `python` to `python3`, so it
> > makes sense since you said that words is the result of alias expansion.
>
> Yes, that'd be it.
>
> > > You might try looking up the place that originally populates $words and
> > > making it stash a copy of $words in another, read-only array; then
> you'll
> > > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > > use of *_orig keys in ${hook_com}.)
> >
> >
> > I tried looking at things related to `vcs_info`, and I saw they were
> > outside of the Completion directory.
> > I was able to find the place where the `*_orig` keys are set, but
> couldn't
> > find where they are used.
> > It would be nice to get some context (or reference) about how this part
> of
> > zsh connects to compsys, and how variables like `hook_com` become
> available
> > to completion functions.
>
> vcs_info has nothing to do with compsys.  Sorry for the unclarity.
>
> *_orig variables would be used by hooks in people's dotfiles.  There
> are examples of hooks in Misc/vcs_info-examples, though they don't
> access the *_orig keys.
>
> I was trying to explain by way of analogy: just like vcs_info sets
> ${hook_com[revision_orig]}, in order that if two hooks are installed
> the later one be able to access the original value even if the
> earlier one had modified ${hook_com[revision]}, so I imagined
> a ${words_orig} array, to let your completion function access the
> value of $words before the _arguments call in _python shifted some
> elements off it.
>
> > > Something along these lines, I think:
> > >
> > >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> > >     words2[-1]=${words2[-1]%x}
> > >     words2[1,${words2[(I)(;|&&|…)]}]=()
> >
> >
> > A lot for me to unpack here and in the paragraphs that follow.
> > I'll try to understand as much as I can, but will probably ask about it
> > later.
>
> Sure, feel free.
>
> «${foo[(I)bar]}» expands to an index into the array $foo; which index is
> determined by the pattern «bar».  (You can get quick help by tab
> completing after the opening round parenthesis; the full description of
> ${[(I)]} is in the manual.)
>
> Do consider the other two alternatives, though.  Just because it's
> _possible_ to reimplement the $BUFFER-to-$words transformation doesn't
> imply that's the best way forward.
>
> Cheers,
>
> Daniel
>
> > Thanks!
> > Dan
> >
> > On Tue, May 5, 2020 at 7:49 PM Daniel Shahaf <d.s@daniel.shahaf.name>
> wrote:
> >
> > > > > > Well, for starters, if you «shift words», you should decrement
> > > CURRENT
> > > > > > as well.
> > > > > >
> > > > > > However, _python uses the «*::…» form of an _arguments optspec,
> > > > > > which should take care of $words/$NUMERIC for you.  I assume
> the
> > > reason
> > > > > > it didn't is that you used «compdef -p».  Try -P instead?  If
> I'm not
> > > > > > mistaken, that would also handle «python … script.py <TAB>» for
> you
> > > > > > (where the ellipsis stands for python's --options).
> > > > >
> > > > >
> > > > > I tested the 4 possible options, and found that switching -p with
> -P
> > > > > doesn't change anything, but shifting words and decrementing
> CURRENT
> > > > > works!  But... I feel like it's a hack I'm not supposed to do
> > > > > (correct me if I'm wrong).
> > >
> > > Those parameters are documented, and manipulated by a number of
> > > functions (including _python, as it happens), so you _are_ using the
> API.
> > > See also the 'compset' builtin.
> > >
> > > However, as mentioned, I suspect that hardcoding a skip/decrement of
> one
> > > element won't DTRT if python is invoked with options (e.g., «python
> > > --foo script.py <TAB>») — not unless you reimplement _python's parsing
> > > of those options and any arguments to them.
> > >
> > > > > I now tried using the BUFFER variable instead of words (since it
> > > remains
> > > > > unchanged), but found that it also has caveats: The BUFFER variable
> > > > > contains the raw value of the line, whereas the words array
> contains
> > > the
> > > > > words after expansions (I think that's the right term).
> > >
> > > Partly.  $words has undergone alias expansion, but not further
> > > expansions.  In particular, the words are still quoted.
> > >
> > > > > To demonstrate this I tried running again completion for `python
> ps,py`
> > > > > with `#compdef -P *`, and printing out BUFFER and words. The
> debug
> > > log
> > > > > (and my terminal) showed the following:
> > > > > First of all _python was called and dispatched my script which
> printed
> > > > > `BUFFER=python ps.py` and `words=ps.py`.
> > >
> > > That's what I'd expect.
> > >
> > > > > After _python returned, my script was invoked again and printed
> > > > > `BUFFER=python ps.py` and `words=python*3* ps.py` (note the 3).
> > >
> > > I'm not sure offhand where this comes from.  $context, $funcstack, et
> al
> > > may have clues.
> > >
> > > > > I do think the right solution is to be dispatched by _python with
> > > > > `#compdef -P *.py`, but to be able to run I need the words array as
> > > > > seen by _python
> > >
> > > Once an element is shifted off $words, it's lost forever (free() is
> > > called on it), so there'll be no way to access a "previous" value of
> $words
> > > unless a copy had been made somewhere.
> > >
> > > Moreover, consider that python may be itself wrapped by some
> precommand:
> > > for example, «env X=Y python», «sudo -u foo python», or «ssh foo
> python».
> > >
> > > You might try looking up the place that originally populates $words and
> > > making it stash a copy of $words in another, read-only array; then
> you'll
> > > be able to use «compset -P» and access that array.  (Compare vcs_info's
> > > use of *_orig keys in ${hook_com}.)
> > >
> > > > > or a to understand how BUFFER can be expanded/converted to its
> > > correct
> > > > > form.  If you have any input on this, I would very much appreciate
> > > > > it.
> > >
> > > Something along these lines, I think:
> > >
> > >     local -a words2=( ${(z):-"$PREBUFFER${LBUFFER}x"} )
> > >     words2[-1]=${words2[-1]%x}
> > >     words2[1,${words2[(I)(;|&&|…)]}]=()
> > >
> > > To understand the second line, see addx() in Src/Zle/zle_tricky.c.
> With
> > > that out of the way, the first line's a textbook use of the ${(z)}
> > > tokenizer.  The third line just throws out everything until the start
> of
> > > the current _simple command_ (see zshmisc(1)) — or, at least, it will
> > > once the ellipsis is filled out.
> > >
> > > Caveats: that snippet doesn't do alias expansion, doesn't try to deal
> > > with ${RBUFFER}, doesn't try to deal with SHORT_LOOPS, doesn't try to
> > > deal with precommands and precommand assignments, should be changed to
> > > use zZ+c+ if the INTERACTIVE_COMMENTS option is set…
> > >
> > > Although it's a plugin that I worked on myself, I should probably
> > > mention that zsh-syntax-highlighting has faced the same problem and you
> > > and implemented a solution to it, which handles a number of the above
> > > caveats.
> > >
> > > >
> > >
> > > There's also a third approach: instead of trying to run the binary,
> make
> > > it stash the output in a well-known location.  For example,
> /usr/bin/foo
> > > could store the output in /usr/share/argcomplete/foo.  (I've always
> > > wanted to standardize _some_ solution to this problem; I'm tired of
> > > writing completion functions by hand…)
> > >
> > > Cheers,
> > >
> > > Daniel
> > >
> >
> >
>
>

-- 
Dan Arad
dan1994@gmail.com

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Daniel Shahaf]

Dan Arad wrote on Fri, 15 May 2020 17:39 +0300:
> I got it working using the hook method: I added another script with
> `#compdef -first-` that saves the words array.
> 

Nice :)

> The thing is, that now I'm faced with a different problem:
> When the command line is `./ps.py`, completions are generated correctly.
> When the command line is `python ps.py`, the actual completions are joined
> by the list of files in the current directory.
> 
> This appears to be a byproduct of going through the `_python`
> auto-completion script.
> 
> I was wondering if there is a way to remove existing completion matches
> before adding my own to make sure I provide only correct matches.

I don't think there's a way to remove already-added matches.

How would that help if it existed?  I assume -first- runs before
_python, not after, so you should look for a way to prevent _python from
running when your function has added matches.  For starters, does your
function return 0 when it has added matches?

Or you could arrange for «python ps.py» to call your script without
a -first- completion function.  That should prevent _python from
falling back to completing files.  Additionally, overwriting -first- is
not a composable approach: users who already overwrite -first- won't be
able to install your completion function alongside their existing
configuration.

> Also, might this be a bug in the `_python` completion script?

I don't see how.  It's normal for completion to offer files as a
fallback for unknown commands, as in «nosuchcommand <TAB>».

I don't recall offhand how to disable or modify the fallback behaviour.

> Thanks for all the help,
> Dan

You're welcome.

Cheers,

Daniel

P.S. While I wrote the above I ran into this:

diff --git a/Doc/Zsh/compsys.yo b/Doc/Zsh/compsys.yo
index 98ab46d8a..8d2813c5a 100644
--- a/Doc/Zsh/compsys.yo
+++ b/Doc/Zsh/compsys.yo
@@ -1685,6 +1685,9 @@ example(zstyle ':completion:*' group-name '')
 
 All matches for which no group name is defined will be put in a group
 named tt(-default-).
+
+To display the group name in the output, see the tt(format) style (q.v.)
+under the tt(descriptions) tag.
 )
 kindex(group-order, completion style)
 item(tt(group-order))(

Fwd: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Dan Arad]

[-- Attachment #1: Type: text/plain, Size: 5602 bytes --]

---------- Forwarded message ---------
From: Dan Arad <dan1994@gmail.com>
Date: Sun, May 17, 2020, 11:05
Subject: Re: Help Request/Bug Report: comparguments causes _arguments to
fail in certain cases
To: Daniel Shahaf <d.s@daniel.shahaf.name>


I think you're mixing things up, so I'll try to clarify:

The only thing I do in the `-first-` script is to backup `words` in another
variable. Otherwise it doesn't affect the completion flow (and my other
bug) at all.
The `_python` script is called before my script (assuming your command line
starts with python), and I actually use it to my advantage, since it
updates `words` so that the script name is always the first element, making
my job easier.

To make things easier to understand, my main script is called
`_python_script`, and the `-first-` script is called
`_python_script_words_backup`.

The two scenarios (starting from the first thing called by `_complete`):

Given the command line `./ps.py`:
`_python_script_words_backup`: Backs up `words` and returns to `_complete`
`_complete`: Calls `_normal` which calls `_dispatch` which  calls
`_python_script`
`_python_script`: Uses the first item in the `words` variable to determine
the what is the script

Given the command line `python ./ps.py`:
`_python_script_words_backup`: Backs up `words` and returns to `_complete`
`_complete`: Calls `_normal` which calls `_dispatch` which  calls `_python`
`_python`: Adds its own arguments, shifts `words` and calls `_normal` which
calls `_dispatch` which calls `_python_script`
`_python_script`: Uses the updated `words` the same as in the first
scenario to determine which script is run, and adds the actual completion

How would that help if it existed?  I assume -first- runs before
> _python, not after, so you should look for a way to prevent _python from
> running when your function has added matches.  For starters, does your
> function return 0 when it has added matches?
>

So regarding what you wrote here, I add my matches after `_python` has run
and not before. I also use `_compskip=all` and always return 0, but since
it is after `_python` has already run, it doesn't solve this particular
problem.

In short, if I want to bypass `_python`, I would probably have to change my
`compdef` definition from `-P *.py` to `-P *`, but then I'll have to parse
everything myself again, losing the great advantage `_python` has given me.

I also thought about using the `_ignore` directive, but I'm not exactly
sure how to use it, or if it can even be used for this use case.

And about this:

Additionally, overwriting -first- is
> not a composable approach: users who already overwrite -first- won't be
> able to install your completion function alongside their existing
> configuration.


Would you propose using a different `compdef` specification for
`_python_script_words_backup`? Using something like `-p *` is not right for
this case, as it will be called multiple times (including after `_python`
has run) and will overwrite the backup.

Cheers,
Dan

On Sat, May 16, 2020 at 11:48 PM Daniel Shahaf <d.s@daniel.shahaf.name>
wrote:

> Dan Arad wrote on Fri, 15 May 2020 17:39 +0300:
> > I got it working using the hook method: I added another script with
> > `#compdef -first-` that saves the words array.
> >
>
> Nice :)
>
> > The thing is, that now I'm faced with a different problem:
> > When the command line is `./ps.py`, completions are generated correctly.
> > When the command line is `python ps.py`, the actual completions are
> joined
> > by the list of files in the current directory.
> >
> > This appears to be a byproduct of going through the `_python`
> > auto-completion script.
> >
> > I was wondering if there is a way to remove existing completion matches
> > before adding my own to make sure I provide only correct matches.
>
> I don't think there's a way to remove already-added matches.
>
> How would that help if it existed?  I assume -first- runs before
> _python, not after, so you should look for a way to prevent _python from
> running when your function has added matches.  For starters, does your
> function return 0 when it has added matches?
>
> Or you could arrange for «python ps.py» to call your script without
> a -first- completion function.  That should prevent _python from
> falling back to completing files.  Additionally, overwriting -first- is
> not a composable approach: users who already overwrite -first- won't be
> able to install your completion function alongside their existing
> configuration.
>
> > Also, might this be a bug in the `_python` completion script?
>
> I don't see how.  It's normal for completion to offer files as a
> fallback for unknown commands, as in «nosuchcommand <TAB>».
>
> I don't recall offhand how to disable or modify the fallback behaviour.
>
> > Thanks for all the help,
> > Dan
>
> You're welcome.
>
> Cheers,
>
> Daniel
>
> P.S. While I wrote the above I ran into this:
>
> diff --git a/Doc/Zsh/compsys.yo b/Doc/Zsh/compsys.yo
> index 98ab46d8a..8d2813c5a 100644
> --- a/Doc/Zsh/compsys.yo
> +++ b/Doc/Zsh/compsys.yo
> @@ -1685,6 +1685,9 @@ example(zstyle ':completion:*' group-name '')
>
>  All matches for which no group name is defined will be put in a group
>  named tt(-default-).
> +
> +To display the group name in the output, see the tt(format) style (q.v.)
> +under the tt(descriptions) tag.
>  )
>  kindex(group-order, completion style)
>  item(tt(group-order))(
>


-- 
Dan Arad
dan1994@gmail.com

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Bart Schaefer]

Reordering discussion slightly ...

On Sat, May 16, 2020 at 1:49 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Dan Arad wrote on Fri, 15 May 2020 17:39 +0300:
> > The thing is, that now I'm faced with a different problem:
> > When the command line is `python ps.py`, the actual completions are joined
> > by the list of files in the current directory.
> >
> > This appears to be a byproduct of going through the `_python`
> > auto-completion script.
> >
> > Also, might this be a bug in the `_python` completion script?
>
> I don't see how.  It's normal for completion to offer files as a
> fallback for unknown commands, as in «nosuchcommand <TAB>».

This isn't happening because of an unknown command, it's because the
historic behavior of _normal is to return all files when none match
the specific completer.  I've mentioned this before; the thought at
the time was that it would be even more confusing for completion to
beep failure at you when there was at least one file in the directory.

> > I was wondering if there is a way to remove existing completion matches
> > before adding my own to make sure I provide only correct matches.
>
> I don't think there's a way to remove already-added matches.

Completers that want to do this sort of thing typically do a hack:
Wrap the "compadd" builtin with a function that uses "builtin compadd
-O array" (or "-A array") to capture the completions without actually
adding them, and then later remove the wrapper and make the real
compadd call.  However, I don't think you need or want to do that
here.

On Sun, May 17, 2020 at 12:26 PM Dan Arad <dan1994@gmail.com> wrote:
>
> The `_python` script is called before my script (assuming your command line
> starts with python), and I actually use it to my advantage, since it
> updates `words` so that the script name is always the first element, making
> my job easier.

I think you're mis-stating this.  The _python function (I'm going to
be pedantic that a function is not the same as a script) is called
AROUND your _python_script function.  You have this call sequence
correct:

> Given the command line `python ./ps.py`:
> `_python_script_words_backup`: Backs up `words` and returns to `_complete`
> `_complete`: Calls `_normal` which calls `_dispatch` which  calls `_python`
> `_python`: Adds its own arguments, shifts `words` and calls `_normal` which
> calls `_dispatch` which calls `_python_script`

However, nothing will have been "compadd"ed by _python at the point at
which it calls _normal.  Anything that gets added is being added as
consequence of calling _normal, not as a consequence of calling
_python.

> [Daniel Shahaf again in a later message]:
> > you should look for a way to prevent _python from
> > running when your function has added matches.  For starters, does your
> > function return 0 when it has added matches?
>
> So regarding what you wrote here, I add my matches after `_python` has run

No, you add them while _python is still running, not after.

> and not before. I also use `_compskip=all`

That doesn't actually do very much once you are beyond -first-.  It
aborts the current level of _dispatch but doesn't prevent the outer
_default that called _python from continuing on into -default-, which
I think you'll find to be the place your extra matches are coming
from.

> and always return 0, but since
> it is after `_python` has already run, it doesn't solve this particular
> problem.

What you would need here is for _python to set _compskip, because it's
local to each level of _dispatch.

> > Additionally, overwriting -first- is
> > not a composable approach: users who already overwrite -first- won't be
> > able to install your completion function alongside their existing
> > configuration.
>
> Would you propose using a different `compdef` specification for
> `_python_script_words_backup`?

No, I'd suggest using a different compdef for "python" itself.  E.g.:

_python_or_script() {
  _python_script_words_backup
  _python "$@" && _compskip=all
}
compdef _python_or_script python

(and remove your -first- compdef).

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Bart Schaefer]

On Sun, May 17, 2020 at 3:12 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> aborts the current level of _dispatch but doesn't prevent the outer
> _default that called _python from continuing on into -default-, which

Typo, that should say "... the outer _dispatch that ..."

Re: Help Request/Bug Report: comparguments causes _arguments to fail in certain cases

[Dan Arad]

[-- Attachment #1: Type: text/plain, Size: 2594 bytes --]

Thanks for the great clarifications! I'll try to stick to the correct
technical terms.

This isn't happening because of an unknown command, it's because the
> historic behavior of _normal is to return all files when none match
> the specific completer.

...

However, nothing will have been "compadd"ed by _python at the point at
> which it calls _normal.  Anything that gets added is being added as
> consequence of calling _normal, not as a consequence of calling

_python.


I suspected as much, but I'm new to zsh and completion, so I had some hard
time following the debug logs and figuring this out.

_python_or_script() {
>   _python_script_words_backup
>   _python "$@" && _compskip=all
> }
> compdef _python_or_script python


This works like a charm!
The only thing I did different is to use the `#compdef -p` form at the
beginning of the file so I could give the same pattern as that of the
`_python` function.

> and not before. I also use `_compskip=all`
>
> That doesn't actually do very much once you are beyond -first-.  It
> aborts the current level of _dispatch but doesn't prevent the outer
> _default that called _python from continuing on into -default-, which
> I think you'll find to be the place your extra matches are coming
> from.


This `_compskip` prevents the inner `_dispatch` from calling the
`_python_script` function more than once.
An example I've seen, is with the command line `python .vscode/ps.py` (I
use the .vscode directory as a place for temporary files as it is git
ignored).
In this case, without `_compskip=all`, the inner `_dispatch` calls
`_python_script` once with `service=.vscode/ps.py` and once with
`service=ps.py`.
In my case this doubles the time to produce completions which may be
critical depending on the way the user has written his script.

To conclude, I would really like to thank everybody who took part in this
discussion.
This is my first open source project, and I was really hoping to provide a
good user experience.
It was a great experience for me to have such a productive conversation,
and I really appreciate the time you've put into providing detailed answers
in a welcoming manner.

Have a great day!
Dan

On Mon, May 18, 2020 at 1:14 AM Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Sun, May 17, 2020 at 3:12 PM Bart Schaefer <schaefer@brasslantern.com>
> wrote:
> >
> > aborts the current level of _dispatch but doesn't prevent the outer
> > _default that called _python from continuing on into -default-, which
>
> Typo, that should say "... the outer _dispatch that ..."
>


-- 
Dan Arad
dan1994@gmail.com