Re: Some possible bugs...

Re: Some possible bugs...

[Sven Wischnowsky]

Bart Schaefer wrote:

> } The problem is that in things like `zsh -c "ls <TAB>' one doesn't want 
> } the shell to treat the `"ls ' as one string. So when completing inside 
> } quotes, zsh treats words separated by spaces as outside the quotes.
> 
> How hard would it be to make this context-sensitive?

Well, currently this is done by get_comp_string() which calls the
lexer and this *has* to be done at the very beginning, because we
first have to get the command line words and position to find out
which compctl to use. However, we could remove all this quote-removal
stuff and make it always just return the whole quoted string.

> For example, when
> the effective compctl says to complete file names, treat the entire
> quoted string as one word and attempt to complete it as a file (this is
> what bash appears to do); similary for command names or anything else
> that normally results in a single shell "word"; otherwise, complete the
> way it's presently done.  This might require another compctl option,
> similar to using	compctl -x ... -l '' ...	except that the
> "range of command line words that are considered to be arguments" is a
> single word that has to be split at IFS before completing.

I think this wouldn't be enough. One example is the:

  % zsh -c "ls 'foo b<TAB>

I mentioned. This should complete to 'foo bar' if there is such a
possible match. So we would need a way to `remove one level of
quotes'. But the quotes may also contain things where we would need
the whole lexing/parsing if we suddenly decide to work on the parts of 
the quoted string. And, of course, such a quoted string may contain
other quoted parts on which we may or may not want to work separately.

> So to get the current behavior you'd use something like (writing old style)
> 
> 	compctl -x 'c[-1,-c]' -1 -l '' -- zsh
> 
> where I'm using -1 as this new option I've described for no better reason
> than that I can't remember which (if any) letters are left for compctl.

(The only free letter is `h').

> This would also mean that, in the case where a single word such as a file
> name is expected to result, zsh could automatically close the quotes and
> append a space after the closing quote.

Sure, yes. Enhancing cmatch a bit...

> This so-called -1 option could apply to any other flag that followed it,
> not just -l ... generically, it would mean "unquote and split the current
> word, find a new current word after the split, then complete using the
> following flags, and finally requote the result."

If we really want `all flags *following* it' we would have to change
the argument parsing of compctl, which is already pretty complicated.
If we just let it work on all flags surrounding it, at least the
parsing would be simple (of course).

I confess that I hadn't thought about a flag that `works like -l'
(without trying to implement it I would prefer to make it work like -l 
in that it doesn't have an effect on the surrounding flags -- I think
this would be much easier to implement, but I may be wrong).
This sounds good to me, with that we could even use the lexing code
(calling it on the string we should work on). However, if this gets
mixed with completions for the whole quoted string, keeping track of
the positions (where to re-insert the completed string, etc.) may turn 
out to be complicated. Hm, using the quote-removal code when handling
this flag and not using the lexer may be easier, but wouldn't allow us
to complete at places like `zsh -c "[[ -o <TAB>'.

For the new completion system we would need a way, too. We could use
something I've been thinking about when we were discussing it at the
beginning. We could give an option to compset which gets one string as 
argument, parses it and stores the result in $words, $compstate,
$IPREFIX, and so on.

Bye
 Sven

P.S.: I'm mainly thinking about ways to implement this, so some of my
      remarks may be hard to understand, sorry.

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Better completion in quotes

[Bart Schaefer]

On May 27,  1:10pm, Sven Wischnowsky wrote:
} Subject: Re: Some possible bugs...
}
} > For example, when
} > the effective compctl says to complete file names, treat the entire
} > quoted string as one word and attempt to complete it as a file (this is
} > what bash appears to do); similary for command names or anything else
} > that normally results in a single shell "word"; otherwise, complete the
} > way it's presently done.
} 
} I think [splitting at spaces] wouldn't be enough. One example is the:
} 
}   % zsh -c "ls 'foo b<TAB>
} 
} I mentioned. This should complete to 'foo bar' if there is such a
} possible match.

Ideally, yes, but even knowing whether or not to split again at spaces,
and what context to complete in if not, is more than e.g. bash does.

The problem is that no context at all is causing zsh to make the wrong
assumption about how it should behave in a very common case.

} So we would need a way to `remove one level of
} quotes'. But the quotes may also contain things where we would need
} the whole lexing/parsing if we suddenly decide to work on the parts of 
} the quoted string.

True.  It'd have to re-parse the whole string after one level of quotes
was removed.  That may be overkill.

} > This so-called -1 option could apply to any other flag that followed it,
} 
} If we just let it work on all flags surrounding it, at least the
} parsing would be simple (of course).

That would be fine.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Better completion in quotes

[Sven Wischnowsky]

Just a couple of comments, as far as I was able to think until now,
collecting suggestions made by others...

Personally, I would prefer a solution where we always return the whole 
quoted string (so that the default behavior is the same as in
bash/tcsh) and we have a flag very much like -l which says that the
word we are currently working on should be split into separate words
and completion should then continue on these.
The first question is if we should use the lexer to split it and then
use all the information we can get from it or not. In some cases this
may be good to have (if the string contains shell code we could get
better context information -- if we are in a [[..]]] and things like
that). But in some cases this may not be what we want, so maybe we
should just split the string at spaces (and try to do this right with
nested quotes). This would also make it look more like -l, because
that doesn't recognise shell constructs, either.
The great problem with this is that this is something completely new
to the completion code. Currently we only have one set of words and
the string, they can be handled independently in most cases. With such 
a string-splitting, we would have to temporarily (completely) change
the array of waords and we would have to remember where to insert the
completion for each match because matches may be produced for the
whole string and a part of it. Currently we only save the positions of
the whole string and replace the original with whatever was completed.
This gets worse when thinking about the new completion stuff. I
suggested an option to compadd that sets the special parameters to
make them work on the parts of the current word. But we would have to
preserve information about the original string internally to be able
to insert matches for the whole string and parts of it. And in the new 
completion code, the user can set the parameters to anything he likes
and thus the internally stored information about whether we are
working on the whole string or a part of it can be garbled. We could
make it accessible to the user, but how? Another set of `ignored'
prefix/suffix?

Note also that with this solution the new flag has no effect on the
surrounding flags, but one can always do the same trick as with -l,
using a dummy compctl:

  compctl -1 _files foo
  compctl -f _files

Which may be a bit ugly. I haven't thought much about making it work
on the other flags, being rather fixed on the `like -l' thing.

And, btw., the fact that the lexer knows about the quoting state (as
Peter mentioned) doens't help us here because the completion code gets 
information only about whole strings and I don't want it to be called
on every quote or something like that. It would make get_comp_string() 
even more complicated.

Ok. Some of these things may be simpler to solve than I can see now,
I'll try to hack something soon, but don't hold your breath. (And of
course, others may want to give it a try, too.)

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Better completion in quotes

[Sven Wischnowsky]

No patch yet, because this may get so complicated that I want to hear
your opinions first. So this is just another list of things I'm
thinking about. I've also found some more problems...

There are two main problems: (1) make zsh correctly handle completion
of quoted words and (2) allow completion on separate words in quoted
strings.

1) We make get_comp_string() always return quoted strings as a whole,
   in the form the lexer returns them. We could then look if the word
   begins with a quote, remember that and use this information to
   offer automatically inserted ending-quotes. But there is this
   problem:

     % touch 'foo bar'
     % ls f'o<TAB>
     % ls 'fo'o<TAB>

   I.e. the quotes need not be at the beginning or there may be
   `closed' quotes. Note also that the second example may be more
   complicated if we have a partial-word matching specification.
   This means that we would have to keep information about quotes in
   the word and we have to re-insert them the way we do it for braces
   already. But this is hard to implement and it wouldn't be enough:

     % compctl -M 'r:|-=*'
     % touch 'foo bar-bar foo'
     % ls f-'b<TAB>

   Here we would have to produce foo\ bar-'bar foo'. So we would also
   have quote different parts of the strings to insert differently.
   This would be so hard to implement that I don't want to think any
   further in this direction.
   So, as a much simpler solution I suggest to make the completion
   code turn strings with quotes into a generic quoted form, using
   backslashes. But it is a bit ugly that

     % ls 'foo b<TAB>

   would be turned into foo\ bar. If this is unacceptable, we could
   treat strings beginning with a quote as a special case and have the 
   code re-insert the quote at the beginning and automatically insert
   the closing quote, that wouldn't be too hard, but I'm not sure I
   like this special casing. I could be convinced, though.
   We'll also have to make sure that the quotes aren't removed if they
   are really needed, as in ${"${...}"}, but I think this is mostly a
   question of when to (internally) un-quote the string. If anyone can 
   think of other places where we have to be careful, I'd like to hear 
   about them.

2) I still like the option Bart suggested and would like to make it
   work. Since I've written 6417 mostly about it, only a short
   description:
   We have a flag for compctl, say -1 (until we find the best
   character for it) which says that the string completion is
   currently using hsould be split into separate words and completion
   should work on these. The syntax would be the same as for -l,
   i.e. with an optional command name.
   Internally this would use either the lexer or some space-splitting
   code to get at the words, I'm not yet sure which would be easier
   (splitting at spaces can get complicated with nested quotes, of
   course). With compctl this would just do another search for the
   command name, and complete using the compctl for that, as with -l.
   We could also add a -x test to allow users to test if the current
   string was quoted or not. Since the sub-compctl would complete the
   parts of the original string as if it weren't quoted, there are
   probably some nasty effects waiting for us when matches are
   generated for both the whole string and a part of it. I don't
   expect this to happen very often (or to be of any interest very
   often), though. If you special case words quoted at the beginning
   this will happen even less often and I think making -1 recursive is 
   easier to implement (and probably more powerful) than making -1 an
   option without arguments that changes the way the surrounding
   options use the current word. We'll see.
   For the new completion system, this is a bit more problematic
   because there we can't do such a C-code-controlled recursion.
   Instead, I suggest a new option to compset, say -q, that makes the
   special parameters be set up so that following shell code works on
   the separate parts of the current word. I.e. the parts would be
   stored in $words, ${PRE,SUF}FIX would be change, and so on. The
   problem is that we need a way to tell the completion (C-)code that
   we are currently working only on a part of the original string. We
   need this information anyway, but for the new completion system the 
   question is if we make it user-visible or not. If we want to make
   it visible, we would add another prefix/suffix-pair, say Q{PRE,SUF}FIX.
   So if we do "foo b<TAB>, compset would store `b' in PREFIX, and
   `foo ' in QPREFIX. The C-code would then use those values to build
   the strings that are used to replace the whole original string
   (which is `foo b'). This is a bit ugly, because this string would
   also be stored in $words (as `(foo b)'), but maybe this isn't that
   problematic. We could also make Q{PRE,SUF}FIX read-only, just to
   make sure that the result isn't complete garbage (but that can't be 
   guaranteed in the new completion system anyway...).
   Finally, we wouldhave to change the quote* keys of compstate,
   making them reflect whether a `compset -q' is currently in effect
   (with what kind of quotes removed -- this can get complicated to
   implement), and probably whether calling `compset -q' might be a
   good idea, because there are quotes

Ok, that's as far as I can think now. I really want a (near-to-)final
solution for all this quoting stuff, to avoid such question as we have 
seen several times lately (and to get rid of that comment in
get_comp_string()).

If anyone sees a better solution or has other ideas or comments,
please let me know.


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Better completion in quotes

[Peter Stephenson]

Sven Wischnowsky wrote:
>    So, as a much simpler solution I suggest to make the completion
>    code turn strings with quotes into a generic quoted form, using
>    backslashes. But it is a bit ugly that
> 
>      % ls 'foo b<TAB>
> 
>    would be turned into foo\ bar. If this is unacceptable, we could
>    treat strings beginning with a quote as a special case and have the 
>    code re-insert the quote at the beginning and automatically insert
>    the closing quote, that wouldn't be too hard, but I'm not sure I
>    like this special casing. I could be convinced, though.

This seems to me the key issue in making it work the way people are
expecting, so I'm not too unhappy about the idea of this being a special
case.  It's preferable to the horror of handling quotes in the middle of
words properly, which I can easily believe is too much work.  But I think
it's also preferable to swapping the quotes for backslashes in this case:
it works neatly the way most users will try most of the time, and it offers
you some way of having your quotes left as they are; and bash has this
problem too, only worse as it just beeps if there's a quote in the middle
of the word.  So long as the documentation is explicit about "foo' b" and
"'foo b" completing differently, I would say let them have quotes.

(I would hazard a guess there are not a few users around who don't even
know you can stick quotes in the middle of words... it wasn't obvious to me
when I first came to UNIX some time ago.  People may even think changing
those is a feature.  "Urgh, someone's stuck some quotes in the middle of
the word.  It would look much better with backslashes...")

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: Better completion in quotes

[Bart Schaefer]

On Jun 4, 11:33am, Sven Wischnowsky wrote:
} Subject: Re: Better completion in quotes
}
} 1) We make get_comp_string() always return quoted strings as a whole,
}    in the form the lexer returns them. We could then look if the word
}    begins with a quote, remember that and use this information to
}    offer automatically inserted ending-quotes. But [...]
}    This means that we would have to keep information about quotes in
}    the word and we have to re-insert them the way we do it for braces
}    already.  [...]
}    So, as a much simpler solution I suggest to make the completion
}    code turn strings with quotes into a generic quoted form, using
}    backslashes. But it is a bit ugly that
} 
}      % ls 'foo b<TAB>
} 
}    would be turned into foo\ bar.

How about this:  We turn the string into whatever is the simplest generic
quoted form for the completion code to handle, which I take it is using
backslashes, and remember that the original was quoted.  After generating
the list of matches, we rewrite those to "beautified quote form" which
puts the entire word in some kind of outer quote marks, and use that for
the strings that get inserted on the command line.

So starting from
	'foo b
the completion system would work on
	foo\ b
and then the matches would get rewritten as
	'foo bar' 'foo blat'
before insertion.  This does mean that if you actually started with
	"foo b
you'd still get
	'foo bar'
on the command line, but that's not too bad (unless what you planned
to type was "foo bar $ding" -- but you can't have everything).

For "beautified form" I suggest starting with the assumption that single
quotes will work, and if you encounter a String token or a Qtick, simply
close the single quote and start again with double quote.  So something
like
	"this is a $parameter
becomes
	'this is a '"$parameter
which is IMO better than
	this\ is\ a\ $parameter
because for one thing that last is wrong when SH_WORD_SPLIT is set.

(Z-Mail uses an algorithm much like this for generating quoted strings
in its supposedly-human-readable saved-options files.)

} 2) I still like the option Bart suggested and would like to make it
}    work.

I don't know how what I just said about (1) might mess up what you've
said about (2).  As with most of the new completion stuff, I've have
to actually see some of it written down to judge how baroque it is, but
it sounds OK in theory.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com