Re: New options & arguments processing system for ZSH

Re: New options & arguments processing system for ZSH

[Felix Rosencrantz]

I think parse_opts is very interesting.  To me, it's exciting since it
attempts to use the same specifications for command line parsing as
it does for completion.  I think whatever format we use for command
descriptions, the format should be able to help the shell understand the
command line from many different angles (e.g. completion, command line
correction, validation, prediction, generation).  I think this is the
first specifications that attempts to cover the issues of the command
line from multiple sides, which a data format can do.

I think this format is complimentary to the XML stuff with which I've
been playing.  I think it would be possible to generate the parse_opts
call from the XML format.  Also, since parse_opts has additional type
and frequency information that could be added to the XML format, to aid
parse_opts generation.

One thing I noticed when trying to complete options to the supplied
"yes" function, was that it didn't provide much help for the options.
It provided type information, but it didn't actually tell me what the
options did.  The format does provide a mechanism to do that.  Though
it seems like that it requires duplicating information that is already
there in the help text.  So it seems if it was somehow possible to reuse
the help text already there that might be useful.

This format also gives the developer a bit of control over where&how
option information is placed variables.  It would be nice if _arguments
had richer control over that.  Though this extra control seems like it
is more for command line parsing that for completion.

-FR.


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

RE: New options & arguments processing system for ZSH

[Borsenkow Andrej]

> 
> One thing I noticed when trying to complete options to the supplied
> "yes" function, was that it didn't provide much help for the options.
> It provided type information, but it didn't actually tell me what the
> options did.  The format does provide a mechanism to do that.  Though
> it seems like that it requires duplicating information that is already
> there in the help text.  So it seems if it was somehow possible to
reuse
> the help text already there that might be useful.
>

Yes, I was about to comment on this too. I guess if this is adopted,
parse_opts should generate help text output out of options
specifications.

-andrej

Re: New options & arguments processing system for ZSH

[Felix Rosencrantz]

--- martin.ebourne@arcordia.com wrote:
> Aha! Yes, it should do exactly that. ...
> So if  you could put 'yes' into emacs and M-x tabify with the whole thing
> selected, or something else similar, it should start working properly.
> Sorry about that - just more ammo for my personal vendetta against the
> worst email program in the universe.

That what was it.  With tabs it works nicely.

-FR.

__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Re: New options & arguments processing system for ZSH

[martin.ebourne]

> prompt% yes2 -<TAB>
> ---- option
> --escape      -e  -- __Interpret escape characters as for echo (default
unless
> --help        -h  -- __Provide help
> --lines       -l  -- __Output only count times
> --no-escape   -E  -- __Prevent interpretation of escape characters
> --no-newline  -n  -- __Suppress output of automatic newline
> --sleep       -s  -- __Pause for number of seconds between each echo
>
> The way I read your mail, though, is that the yes2 output should be
coming
> from the original "yes" script.  So there might be a problem somewhere,
> or I just didn't understand you.

Aha! Yes, it should do exactly that. I think I know what the problem is as
well. parse_opts detects the difference between the options and the help
text because the latter follows tab characters, and the former is before
any tab characters. The documentation is correct about this, but I wouldn't
expect you to have read all that anyway.

However, I should say that the ever wonderful Lotus *#?! Notes program here
has carefully converted my tabs to spaces and totally stuffed it up. So if
you could put 'yes' into emacs and M-x tabify with the whole thing
selected, or something else similar, it should start working properly.
Sorry about that - just more ammo for my personal vendetta against the
worst email program in the universe.

> If you use a state action, _arguments puts information in the opt_args
> associative array.  I think it could be useful if there was a little
> more control how parsed values are stored like provided by parse_opts.

Ah, ok. I haven't used the state machine facilities yet, but I can see how
that might be handy.

Cheers,

Martin.

Re: New options & arguments processing system for ZSH

[Felix Rosencrantz]

I'll try to be a little more descriptive.  When completing, like so:

prompt% yes -<TAB>
---- option
--lines  --sleep  -l  -s  -- positive integer number                           
--escape       --no-escape    -E             -h                          
--help         --no-newline   -e             -n                          

There is no help on what the options do.  There's nothing that says -l means
"Output only count times".  Actually, there is sort of a problem with the
listing, since it says that the --lines,-l,-s,--sleep options are the same
because of the new "equivalence matching".

I created a new version of yes, called yes2, which I've attached.  It behaves
more like what I would like to see during completion, as such:

prompt% yes2 -<TAB>
---- option
--escape      -e  -- __Interpret escape characters as for echo (default unless 
--help        -h  -- __Provide help                                            
--lines       -l  -- __Output only count times                                 
--no-escape   -E  -- __Prevent interpretation of escape characters             
--no-newline  -n  -- __Suppress output of automatic newline                    
--sleep       -s  -- __Pause for number of seconds between each echo           

But notice the duplication of the option explanation via the  '* help="..."'
syntax.  (I added the double underscores, just to make sure that is where the
text was coming from.)

The way I read your mail, though, is that the yes2 output should be coming
from the original "yes" script.  So there might be a problem somewhere,
or I just didn't understand you.  

>Since _arguments doesn't decode values for use I don't see how that would
>work.

If you use a state action, _arguments puts information in the opt_args
associative array.  I think it could be useful if there was a little 
more control how parsed values are stored like provided by parse_opts. 

-FR.
----------BEGIN yes2
local -A opts
parse_opts - opts -- "$argv[@]" <<'EOF' || return 1
Description:
Repeatedly print a string

Usage:
yes2 [options] [<text> ...]

Options:
  -e, --escape           Interpret escape characters as for echo (default
unless
                    the BSD_ECHO option is set)
                    # [echoopt] += --escape | -e : constant=-e *
help="__Interpret escape characters as for echo (default unless the BSD_ECHO
option is set)"
  -E, --no-escape        Prevent interpretation of escape characters
                    # [echoopt] += --no-escape | -E : constant=-E *
help="__Prevent interpretation of escape characters"
  -h, --help             Provide this help
                    # --help | -h * help="__Provide help"
  -l <count>, --lines=<count> Output only count times
                    # --lines | -l : posinteger * help="__Output only count
times"
  -n, --no-newline       Suppress output of automatic newline
                    # [echoopt] += --no-newline | -n : constant=-n *
help="__Suppress output of automatic newline"
  -s <seconds>, --sleep=<seconds>
                    Pause for number of seconds between each echo
                    # --sleep | -s : posinteger * help="__Pause for number of
seconds between each echo"

Arguments:
  [<text> ...]           Optional text to print. Defaults to 'yes'
                    # [text] += [*] text
EOF

# Output as required
while (( ${opts[lines]:-1} ))
do
  echo $=opts[echoopt] ${opts[text]:-yes}
  (( opts[sleep] )) && sleep $opts[sleep]
  (( opts[lines] && opts[lines]-- ))
done
----------END yes2


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Re: New options & arguments processing system for ZSH

[martin.ebourne]

> - I like it a lot better than the XML version suggested by Felix, but
>   the ideal solution probably lies somewhere between the two.

Quite likely. Certainly I can see advantages of both systems, and even of
converting between the two. As to whether it'll ever be possible to come up
with a format that's ideal both for complicated programs such as find, and
simple ones as cat, I'm not so certain. Coupled with the fact that
different people like to do things in different ways, I think it's helpful
to have different systems.

> - It suffers from the same problems as my _arg_compile scheme, to wit,
>   every function that uses it has to do a lot of extra work that is not
>   directly related to the purpose of the function.

I like _arg_compile, it certainly makes the less than intuitive (in my
opinion anyway!) syntax of _arguments much more readable. However, with
parse_opts I was really trying to tackle a different problem. What I wanted
was something that would decode the command line when a program is run,
splitting options and arguments into data structures as required, and
performing any validation possible (sort of a zparseopts on steroids ;)). I
wanted this to simplify the writing of scripts/functions which were
complicated enough to need some code to parse the command line. While I was
on the way I found I could add automatic completion for almost no extra
hassle on the user's part, so I jumped at the chance.

As to the extra work done by the 'client' function, in terms of processing
there is a fair bit extra - mostly to do with decoding the help. However
since this is intended to be done mainly for functions which are used
directly by a person, not ones used by other functions in a larger system,
I don't think the processing overhead is really an issue. I can think of
some ways of caching this stuff though, which would be of most advantage to
speed when completing I think, and would probably almost eliminate the
overhead.

As to extra work in terms of code in the function, I don't think that's the
case. Most of the lines are real help text, and there's not much you can do
to shorten that apart from not provide it. I've done my best to keep the
calling syntax very simple, and hopefully managed it to a large extent. I
did go through most of my shell functions and convert them to this system.
Almost all functions that did any kind of options parsing, help, or
validation became shorter, and all of the functions became more functional
(long options, completion, etc). Even with a case like the 'yes' example,
the real code became noticably shorter, and the division of responsibility
between decoding and doing became much more clear.

> - Here-documents end up getting encoded as strings in the compiled
>   function definition in memory, so this is probably prohibitive for
>   general use except when using zcompile'd files with memory mapping.

I've already mentioned that most of the text is 'real' help, so that's no
different to you'd get if you provided that anyway. Personally I like
commands to be as helpful as possible, so that's a price I'm prepared to
pay - and as I've said, I don't see a major use for this in decoding
options for internal functions - ie. I wouldn't expect the completion
system or anything similar to use it internally! [nothing stopping it
though... ;)]

On the other hand, tt is fair to say that all of my functions are in
separate autoloaded files, and I do have them all compiled on a
per-directory basis. But then I did that anyway. ;)

> - You should have sent it to zsh-workers, not zsh-users. :-)

Yes, I can certainly see your point. I did think about it for quite a
while. The thing that tipped the balance for me was that I feel it is the
sort of thing that any users of zsh who are writing shell scripts might
find helpful, as opposed to just the developers of Zsh. After all, I'm in
the former category, and obviously find it useful! Also even on a slow
modem the download time should be < 15 seconds, so I didn't think it was
too bad.

Certainly all followup should be on workers, as has happened.

Thanks for looking at it, any further suggestions welcomed if you get
chance.

Cheers,

Martin.

Re: New options & arguments processing system for ZSH

[martin.ebourne]

> I think this format is complimentary to the XML stuff with which I've
> been playing.  I think it would be possible to generate the parse_opts
> call from the XML format

I'm sure that it would be possible to convert XML -> parse_opts, and if
there are other tools that convert to XML from eg. man pages, I can think
of some uses too. However, for most small-medium sized shell scripts I'd
like to think that the format is simple enough to be coded directly.

> One thing I noticed when trying to complete options to the supplied
> "yes" function, was that it didn't provide much help for the options.
> It provided type information, but it didn't actually tell me what the
> options did.

I'm not quite sure what you mean here. It does use the help text for
options, so I'm going to make a couple of guesses what you might mean and
try and explain a little. My appologies in advance if I'm way off track!

The help text for each option/argument is extracted and stored against the
option/argument. It is also made use of in both cases. For options, it is
used in exactly the same way as for all the other completions I've seen,
which is when you are completing the option *name* it lists the
descriptions against that. When you are completing the option *value* it is
not used because there is a separate description for that, which tells you
the type to be completed. I have thought on occasion that it would be nice
to have the option description available at the point of completing the
value as well. parse_opts could be changed to do this but I think it's more
of a general issue. I guess probably _arguments could do this itself, based
on a style, by simply concatenating the option description onto the end of
the value description.

For arguments, there currently is no description of what the argument is
for in any of the completions I've seen - only a description of what you
are completing. I didn't want to waste the help text present in this case
so parse_opts does concatenate the argument help onto the end of the value
descriptions exactly as I've just suggested for options.

I'm sure you're aware that various styles need to be set to be able to view
the various descriptions.

> The format does provide a mechanism to do that.  Though
> it seems like that it requires duplicating information that is already
> there in the help text.  So it seems if it was somehow possible to reuse
> the help text already there that might be useful.

Again, I'm not quite sure what you mean about specifying the help text
separately. I can think of two angles:

1. Defining a type:
    #type newtype basetype completion_description completion_action
Here you can give a full description for the value to be completed, and
could duplicate the option help. As I mentioned before, I'd like to see
this done in a more central place though.

2. Defining an option with helptext override:
    # --option * help="the description"
This simply overrides the help extracted automatically from the help text.
I don't envisage this will be useful except in extremely rare circumstances
when it has trouble extracting help from the main text. The only reason
it's there is because it's a generic method of adding new settings
associated with an option/argument/type, and is also able to override all
current ones. The help specified in this case will be used in exactly the
way I've described above.

I would like to think that there's no need for duplication of information
when using parse_opts, certainly that's one of the major design goals.

> This format also gives the developer a bit of control over where&how
> option information is placed variables.  It would be nice if _arguments
> had richer control over that.  Though this extra control seems like it
> is more for command line parsing that for completion.

Since _arguments doesn't decode values for use I don't see how that would
work. Certainly this information is only used by parse_opts for the
processing of the options/arguments when the command is actually invoked,
not when completed.

Thanks for your interest,

Martin.

Re: New options & arguments processing system for ZSH

[Bart Schaefer]

On Aug 17,  2:59pm, martin.ebourne@arcordia.com wrote:
}
} For now I'd like any comments people have, especially bug reports and
} praise (yeah right!).

I've barely glanced over it, but my initial impressions are:

- I like it a lot better than the XML version suggested by Felix, but
  the ideal solution probably lies somewhere between the two.

- It suffers from the same problems as my _arg_compile scheme, to wit,
  every function that uses it has to do a lot of extra work that is not
  directly related to the purpose of the function.

- Here-documents end up getting encoded as strings in the compiled
  function definition in memory, so this is probably prohibitive for
  general use except when using zcompile'd files with memory mapping.

- You should have sent it to zsh-workers, not zsh-users. :-)

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net