Expanding quotes

Expanding quotes

[Yuri D'Elia]

Hi everyone,

mkdir test
cd test
touch file file\' file\"
ls *\"<TAB>

results in:

% ls file file\" file\'

although the result of the pure glob *\" is correct, and only contains
file\". This confused me yesterday, where I almost got the impression
that the quote was breaking the match.

Another tidbit:

% ls *[\']<TAB>

doesn't expand entirely, although it globs correctly file\' when executed.

The problem only exists with the quote/double quote (in both forms).

Re: Expanding quotes

[Bart Schaefer]

On Dec 17,  3:48pm, Yuri D'Elia wrote:
} Subject: Expanding quotes
}
} touch file file\' file\"
} ls *\"<TAB>
} 
} results in:
} 
} % ls file file\" file\'

Hmm, somewhere down in the completion internals the code that sets the
PREFIX string is re-quoting both the backslash and the single quote.
When this is re-interpolated by the expand-word code it turns into
the empty string, so you get the expansion of "*".

} % ls *[\']<TAB>
} 
} doesn't expand

Same problem, except you get the expansion of *[] which is an invalid
pattern and hence nothing.

There's a basic conflict in the completion system in that it can't "know"
if you mean for the literal string backslash-quote to be completed, or if
you mean for the backslash to have its usual semantic effect on the quote.

Re: Expanding quotes

[Yuri D'Elia]

On 12/17/2013 07:26 PM, Bart Schaefer wrote:
> There's a basic conflict in the completion system in that it can't "know"
> if you mean for the literal string backslash-quote to be completed, or if
> you mean for the backslash to have its usual semantic effect on the quote.

Hhhm, how could this happen? Surely, zsh seem to take note if the
argument to be completed starts with a quote.

Re: Expanding quotes

[Bart Schaefer]

On Dec 23,  6:38pm, Yuri D'Elia wrote:
}
} On 12/17/2013 07:26 PM, Bart Schaefer wrote:
} > There's a basic conflict in the completion system in that it can't "know"
} > if you mean for the literal string backslash-quote to be completed, or if
} > you mean for the backslash to have its usual semantic effect on the quote.
} 
} Hhhm, how could this happen? Surely, zsh seem to take note if the
} argument to be completed starts with a quote.

It's a syntactic question as well as a semantic one.  For example, you
typed

    ls *\"

and expected completion to treat backslash-dquote as a unit matching a
literal double quote.  But someone else might type

    ls *"

and expect completion to *supply* the missing backslash, because there
is a file name ending with a quote.  It gets even more complicated if
the backslash is inside another set of quotes, etc.  We spent quite a
lot of time (years ago) responding to complaints about this kind of thing
before finally concluding that we couldn't please everybody and should
stick with making the common cases work.

A third complication is that there are a huge number of different ways
to apply quoting, and trying to figure out which one corresponds to what
the user already typed is difficult to generalize.  The globbing code
that generates the list of possible files returns a particular quoting
scheme that would have to be converted to the user's arbitrary quoting.

Consequently there are a number of cases that either never came up when
this was all being developed, or that have been deemed to seldom come
up in practice, where the results are imperfect.

We'd be happy to find some more volunteers to work on the internals of
completion to resolve more of these cases.

Re: Expanding quotes

[Ray Andrews]

On 23/12/13 11:15 AM, Bart Schaefer wrote:
> and expect completion to *supply* the missing backslash, because there
> is a file name ending with a quote.  It gets even more complicated if
> the backslash is inside another set of quotes, etc.  We spent quite a
> lot of time (years ago) responding to complaints about this kind of thing
> before finally concluding that we couldn't please everybody and should
> stick with making the common cases work.

Pardon for wandering off on a philosophical rant,  but it seems to me 
that there is a very deep error in even trying to solve 'problems' like 
this.  It is not logically possible to permit any syntax and still parse 
a statement.  All meaningful grammars exist--can only exist--because 
they impose limitations on language. It is monstrous that a file name 
is  permitted to end with a quotation mark! Without reserved characters 
we invite chaos.  It's the same with spaces inside a filename--it is 
insanity to permit it.  And then the double quote is legal too?

file name   .................. two identifiers so ...
"file name" .................. is the filename  ... << file name >> or 
are the double quotes include in the filename so that the filename is  
....  << "file name" >> ?

Madness!

Granted I'm still a relative beginner, but it seems to me that the 
syntax of zsh (all sh*) is already vastly over complicated, even 
Byzantine.  As more and more special situations are handled, the code 
must mushroom into an intractable mess, and actually create more 
problems than it solves.  Better IMHO not to even try.

Re: Expanding quotes

[Yuri D'Elia]

On 12/24/2013 04:12 PM, Ray Andrews wrote:
> file name   .................. two identifiers so ...
> "file name" .................. is the filename  ... << file name >> or
> are the double quotes include in the filename so that the filename is 
> ....  << "file name" >> ?
> 
> Madness!
> 
> Granted I'm still a relative beginner, but it seems to me that the
> syntax of zsh (all sh*) is already vastly over complicated, even
> Byzantine.  As more and more special situations are handled, the code
> must mushroom into an intractable mess, and actually create more
> problems than it solves.  Better IMHO not to even try.

Quoting is invariably part of any language construct.

It's really important, especially as a beginner, to understand quoting
right from the start as a "normal event", not as an exception.

Restricting the allowed characters of a file in the file system will not
remove quoting issues of a variable's value (for example).

To wrap your mind against it, you might see quoting as a problem for the
interpreter to *separate arguments* and not to interpret data. That is,
if we could choose # as an argument separator, we could have any
character in the file name except #. It just so happens that it's
/usually/ more readable to type:

$ command argument "a value"

then

$ command#argument#a value

Just my 2c.

Re: Expanding quotes

[Ray Andrews]

On 24/12/13 08:24 AM, Yuri D'Elia wrote:
> On 12/24/2013 04:12 PM, Ray Andrews wrote:
>> file name   .................. two identifiers so ...
>> "file name" .................. is the filename  ... << file name >> or
>> are the double quotes include in the filename so that the filename is
>> ....  << "file name" >> ?
>>
>> Madness!
>>
>> Granted I'm still a relative beginner, but it seems to me that the
>> syntax of zsh (all sh*) is already vastly over complicated, even
>> Byzantine.  As more and more special situations are handled, the code
>> must mushroom into an intractable mess, and actually create more
>> problems than it solves.  Better IMHO not to even try.
> Quoting is invariably part of any language construct.
Of course.
> It's really important, especially as a beginner, to understand quoting
> right from the start as a "normal event", not as an exception.
Of course.
> Restricting the allowed characters of a file in the file system will not
> remove quoting issues of a variable's value (for example).
>
> To wrap your mind against it, you might see quoting as a problem for the
> interpreter to *separate arguments* and not to interpret data. That is,
> if we could choose # as an argument separator, we could have any
> character in the file name except #. It just so happens that it's
> /usually/ more readable to type:
That's just the point: There must be reserved characters, and it seems 
to me that quotation marks are first on that list.   Also, as a policy, 
I prefer simplicity with some resulting limitations on what can be done, 
over complicated efforts to do 'anything'.  It's just as Bart said, some 
issues are better left 'unsolved'. The solution could be worse than the  
problem.  I think back to my DOS days, where almost all of the 'special' 
characters were reserved--it sure made things simpler.
> $ command argument "a value"
>
> then
>
> $ command#argument#a value
>
> Just my 2c.
>
And mine ;-) I come at this, not as a zsh expert, but as a guy who knows 
something about information theory. Noam Chomsky would have something to 
say about this sort of issue.
>

Re: Expanding quotes

[Yuri D'Elia]

On 12/24/2013 06:18 PM, Ray Andrews wrote:
> I prefer simplicity with some resulting limitations on what can be done,
> over complicated efforts to do 'anything'.  It's just as Bart said, some
> issues are better left 'unsolved'. The solution could be worse than the 
> problem.  I think back to my DOS days, where almost all of the 'special'
> characters were reserved--it sure made things simpler.

I think you're conflating the text UI of a shell, the filesystem and the
shell language itself as part of the same issue. Yes, taken together, I
do agree that from my user point of view it's definitely inconvenient
and the interaction is not ideal. However, I would gladly prefer a shell
solution than "fixing" what's clearly not broken: the filesystem.

If you use a visual file manager, users will curse your from not being
allowed to use quotes and spaces as part of the file name. I know
windows users complain all the time about the disallowed characters,
because they don't see the reasons of the restricted character set
anymore (this is true also for users of nautilus/gnome/kde of course). I
also agree with these users.

> And mine ;-) I come at this, not as a zsh expert, but as a guy who knows
> something about information theory. Noam Chomsky would have something to
> say about this sort of issue.

The line-editing interface has definitely limitations, but I would
always like to think about it just as one of the possible interfaces to
the file system.

Imagine a different shell, where you had to different insertion modes,
like "vi". To enter a path, you would enter "pathname" mode with ESCp,
and the shell would keep track internally of the beginning and end of
the file name, while just showing the path maybe in a different color:

% some-command ESCppath name with spacesESCc continue command

Problem solved without escaping entirely?
(damn, I would love such a shell).

Re: Expanding quotes

[Ray Andrews]

On 26/12/13 06:50 AM, Yuri D'Elia wrote:
> On 12/24/2013 06:18 PM, Ray Andrews wrote:
>> I prefer simplicity with some resulting limitations on what can be done,
>> over complicated efforts to do 'anything'.  It's just as Bart said, some
>> issues are better left 'unsolved'. The solution could be worse than the
>> problem.  I think back to my DOS days, where almost all of the 'special'
>> characters were reserved--it sure made things simpler.
> I think you're conflating the text UI of a shell, the filesystem and the
> shell language itself as part of the same issue. Yes, taken together, I
> do agree that from my user point of view it's definitely inconvenient
> and the interaction is not ideal. However, I would gladly prefer a shell
> solution than "fixing" what's clearly not broken: the filesystem.
You're right, I'm looking at this as one collective issue.  But IMHO the 
thing that is broken is to permit things like double quotes in a 
filename at all.  From what I understand, POSIX does not allow it, but 
maybe I misunderstand.  Anyway, if somehow zsh can learn to live with 
this sort of thing, then so be it, but I think the task should not be 
asked.  Perhaps some switch or flag of some sort could be added to 
indicate: "I have quotes in filenames here ... God be with you." Anyway, 
that's how it looks to me, maybe I'm wrong.

Re: Expanding quotes

[Yuri D'Elia]

On 12/23/2013 08:15 PM, Bart Schaefer wrote:
> It's a syntactic question as well as a semantic one.  For example, you
> typed
> 
>     ls *\"
> 
> and expected completion to treat backslash-dquote as a unit matching a
> literal double quote.  But someone else might type
> 
>     ls *"
> 
> and expect completion to *supply* the missing backslash, because there
> is a file name ending with a quote.  It gets even more complicated if

I'm pretty sure I've experienced this behavior before, though it's
definitely rare.

> A third complication is that there are a huge number of different ways
> to apply quoting, and trying to figure out which one corresponds to what
> the user already typed is difficult to generalize.  The globbing code
> that generates the list of possible files returns a particular quoting
> scheme that would have to be converted to the user's arbitrary quoting.

I see.

> We'd be happy to find some more volunteers to work on the internals of
> completion to resolve more of these cases.

Thanks for the clarification, the rationale is sound for me.

Personally, I would actually prefer is there was an option in zsh to
normalize the quoting mechanism to always "double quote" the argument
instead of escaping characters (thus inserting the initial double-quote
if missing). I don't know if this would actually simplify or furthermore
complicate the expansion rules.

With complete_in_word and all the other completion switches I definitely
see the complexity of "doing the right thing" in all the cases. Though
the fact that a glob might not expand in the same way as when executed
makes a bit of uneasy feeling for me. I often use expansion to
proof-check a glob instead of executing the command twice.

Maybe there's a better way to do it that doesn't involve running "ls
glob" before "real-command glob".

Re: Expanding quotes

[Bart Schaefer]

On Dec 24,  5:11pm, Yuri D'Elia wrote:
}
} Personally, I would actually prefer is there was an option in zsh to
} normalize the quoting mechanism to always "double quote" the argument
} instead of escaping characters (thus inserting the initial double-quote
} if missing). I don't know if this would actually simplify or furthermore
} complicate the expansion rules.

Have a look at Functions/Zle/quote-and-complete-word in the distributed
sample functions.

} the fact that a glob might not expand in the same way as when executed
} makes a bit of uneasy feeling for me. I often use expansion to
} proof-check a glob instead of executing the command twice.
} 
} Maybe there's a better way to do it that doesn't involve running "ls
} glob" before "real-command glob".

You can create your own little completer that does nothing but globbing:

    _glob () {
      local -a globbed
      globbed=( ${(Q)~words[CURRENT]} )
      [[ $globbed == $words[CURRENT]] ]] && return 1
      compadd -Uf -a globbed
    }

You can fiddle with the assignment to globbed until you get the behavior
you want.  What I've done with ${(Q)~words[CURRENT]} may not be to your
liking; e.g. you might prefer ${~words[CURRENT]//\\/} to only remove any
backslashes.  Something of the kind is needed, though, because the tilde
operator only makes globbing characters active, it doesn't unquote other
special characters that may be quoted in the value of $words.

The default completers if you don't set a style are (_complete _ignored).
So you just need to add yours:

    zstyle ':completion:*' completer _glob _complete _ignored

(Use "zstlyle -L" to see if you've already changed the completer style,
and if so just add _glob in an appropriate place.)