Generating completion functions from XML

Generating completion functions from XML

[Felix Rosencrantz]

Recently there was discussion about how it is harder to create new      
completion functions under the new completion system than the old.      
The new system is easier to use as installed, since there are so many   
great completion functions available, and those do a much better job at 
completion than anything before.                                        

Part of the problem is the new system provides much more functionality. 
To do a good job, it takes more than just listing the flags to          
_arguments, now there is help to provide for the each flag, and then    
help for the flag value.  There are more options on what flags are      
exclusive and equivalent, and many times this information has to be     
repeated.                                                               

It still has been bothering me that it is not easier to create new      
completions.  I've thought about writing shell/perl/python/* scripts    
that could attempt to make good guesses based on usage information or   
man pages, and generate the appropriate _arguments call. (_arguments    
can already do this at runtime with GNU getopts like output.)           

But I suspect any scripts would still hit some sort of 90/10 rule.      
It'll get you 90% of the way but there will still be the final 10%      
which will have to be done manually.                                    

So I feared having to look at partially wrong _arguments calls.  That   
command has a lot of syntactic sugar and there is more on the way (e.g. 
recent changes).  It's not easy for me to remember what each special    
character and grouping does.                                            

It occurred to me that having an intermediate file format would be a    
good idea.  Possibly something that would be a little more descriptive, 
maybe even interspersed with comments and documentation (both from      
zsh & the command).  Also it should be easier to edit.  Also, it        
should make it easier to specify commands that have subcommands (e.g.   
cvs).  There would need to be a tool which transform this intermediate  
file format into the appropriate completion function, generating the    
appropriate call to _arguments, _values, etc.  Also, as the syntax to   
_arguments changed it would be possible to change this generation tool, 
and then regenerate the completion function.  Maybe even _arguments     
could even take this intermediate format file as input.                 

To me it seemed like XML would be a good format to use for this         
intermediate format.  The XML format could be designed with the idea    
that is it meant more for a person to understand, and edit.             

Some things that would then be possible.  It would be possible to write 
code that inserts standard documentation describing how zsh works (e.g. 
how does it handle exclusive flags or equivalent flags) right next to   
the place where the command specific information is being provided.     
Command specific help (such as pulled from the man page) could be       
embedded in the XML.  It would also be possible to embed zsh source     
code for flags that use $state in the XML.                              

It would be possible to provide zstyle information (e.g. default        
values, values to look up) for particular arguments.  Also, editing     
this file would provide another point of customization.                 

I don't think the XML format devised would necessarily have to be able  
to generate completion functions for every possible command.  Just have 
to be good enough to work for a good portion of commands.               

So I've gone on long enough about this.  Does this sound like something 
that could provide an additional easier to use interface to the         
completion system?                                                     

-FR.


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

Re: Generating completion functions from XML

[Sven Wischnowsky]

Felix Rosencrantz wrote:

> ...
> 
> But I suspect any scripts would still hit some sort of 90/10 rule.      
> It'll get you 90% of the way but there will still be the final 10%      
> which will have to be done manually.                                    

It might be useful if it manages to automate more than 50% of the
process, leaving the rest to the user...

> So I feared having to look at partially wrong _arguments calls.  That   
> command has a lot of syntactic sugar and there is more on the way (e.g. 
> recent changes).  It's not easy for me to remember what each special    
> character and grouping does.                                            

Actually, the reason why some of the _argument calls hurt the eyes is
that it has so littly syntactic sugar.

And I hope there won't be any changes to the _arguments syntax (or at
least nothing dramatic).  That's part of the reason why I used `same
description' as the condition on which to group in _describe.

> It occurred to me that having an intermediate file format would be a    
> good idea.  Possibly something that would be a little more descriptive, 
> maybe even interspersed with comments and documentation (both from      
> zsh & the command).  Also it should be easier to edit.  Also, it        
> should make it easier to specify commands that have subcommands (e.g.   
> cvs).  There would need to be a tool which transform this intermediate  
> file format into the appropriate completion function, generating the    
> appropriate call to _arguments, _values, etc.  Also, as the syntax to   
> _arguments changed it would be possible to change this generation tool, 
> and then regenerate the completion function.  Maybe even _arguments     
> could even take this intermediate format file as input.                 

Btw. have you noticed Bart's _arg_compile function?

I'm not opposed to defining some language to make describing commands
easier at all.  Be it XML or something else (everything has to be XML
nowadays, right? at least until the next hype... ;-).

But it's not simple to define, really.  Especially if you think about
commands like _cvs which does *a lot* besides calling _arguments, things
that are best described with the shell code needed to do them, I think.

I'll try to make helpful comments when you start describing the language
you have in mind.


Bye
  Sven


-- 
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Generating completion functions from XML

[Bart Schaefer]

On Jul 19, 10:56am, Sven Wischnowsky wrote:
} Subject: Re: Generating completion functions from XML
}
} Felix Rosencrantz wrote:
} 
} > So I feared having to look at partially wrong _arguments calls.  That   
} > command has a lot of syntactic sugar and there is more on the way (e.g. 
} > recent changes).  It's not easy for me to remember what each special    
} > character and grouping does.                                            
} 
} Actually, the reason why some of the _argument calls hurt the eyes is
} that it has so littly syntactic sugar.
} 
} > It occurred to me that having an intermediate file format would be a    
} > good idea.  Possibly something that would be a little more descriptive, 
} > maybe even interspersed with comments and documentation (both from      
} > zsh & the command).  Also it should be easier to edit.  Also, it        
} > should make it easier to specify commands that have subcommands (e.g.   
} > cvs).  There would need to be a tool which transform this intermediate  
} > file format into the appropriate completion function, generating the    
} > appropriate call to _arguments, _values, etc.  Also, as the syntax to   
} > _arguments changed it would be possible to change this generation tool, 
} > and then regenerate the completion function.  Maybe even _arguments     
} > could even take this intermediate format file as input.                 
} 
} Btw. have you noticed Bart's _arg_compile function?

I haven't updated _arg_compile in quite some time; I'm not sure that it
still produces good _arguments calls.

Another possible output format for a more sugary input language may be
Tanaka's _regex_arguments, which presently is used only for the `xset'
and `xwit' commands and the Debian `apt' family of commands.  It might
be possible to produce better completions for many commands by using
_regex_arguments, but I confess that trying to comprehend that regex
syntax has always exceeded my available time or attention span.

} I'm not opposed to defining some language to make describing commands
} easier at all.  Be it XML or something else (everything has to be XML
} nowadays, right? at least until the next hype... ;-).

That pretty much sums up my feeling about XML.  I have yet to find an
editor that makes XML easier to edit than shell code.  Maybe I just
haven't tried a recent enough version of emacs, or something.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Generating completion functions from XML

[Adam Spiers]

Bart Schaefer (schaefer@candle.brasslantern.com) wrote:
> That pretty much sums up my feeling about XML.  I have yet to find an
> editor that makes XML easier to edit than shell code.  Maybe I just
> haven't tried a recent enough version of emacs, or something.

psgml.el does a pretty decent job, although it is a bit of a pain to
install.

Re: Generating completion functions from XML

[Felix Rosencrantz]

>Btw. have you noticed Bart's _arg_compile function?

I hadn't looked at Bart's _arg_compile function.  It does look
interesting.  It covers things that I would like to see in an
intermediate file format.

XML is sort of interesting to me for the intermediate format for all the
reasons that people are hyping it these days.  It might not be the way
to go, but it's worth a try.  I do agree that XML is not overly easy to
edit (I haven't looked at psgml.el).  Though, I like the possibility of
actually placing documentation in appropriate points in the descriptive
file, and the ability for it to be modified more programmatically.

Also, I don't really expect XML to represent all completion functions.
Though for many simple ones, I think XML could be a good format.

I'll take a first stab at coming up with some sort of grammar.  I've
only created a couple small grammars for some small projects.  I know
though that it can take some effort to design a good schema.

I'd like to see an XML format that can represent any _arguments call.
So that it would be possible to have some sort of equivalent version in
XML.

I'd also like to be able to represent/generate a function that contains
multiple _argument calls based on $service, and then can also handle
$state (e.g. the _zip function).

>But it's not simple to define, really.  Especially if you think about
>commands like _cvs which does *a lot* besides calling _arguments,
>things that are best described with the shell code needed to do them, I
>think.

With a _cvs-like command I'd like to be able to represent in XML what
amounts to the hierarchical call to _arguments.  I realize there is a
lot of specialized code to handle actions.  I don't really see a way
to automate action generation or to represent actions other than in
zsh script.  (Though there is an XML version of java code, javaml...)
I think if the XML format made it simple to share flags between the
various sub-commands, since it is common for the same flag to be shared
among sub-commands, it would help with the creation of such completion
functions.  

>I'll try to make helpful comments when you start describing the
>language you have in mind.

That would be very much appreciated.  I'll try to come up with something
over the next week.

-FR


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

Re: Generating completion functions from XML

[Oliver Kiddle]

Felix Rosencrantz wrote:

> There are more options on what flags are exclusive and equivalent,
> and many times this information has to be repeated.

Note that the exclusion lists are not necessarily repeated. An option
listed in the exclusion list is excluded after the option and there are
some commands where the exclusion is dependant on the ordering. I can't
think of much information repetition in the current _arguments syntax.

Another thing we could do is modify GNU gengetopts (which produces C
calls to getopt_long) to additionally output the basis of a completion
function. Whether they would include our change, I don't know with zsh
not being GNU.

> cvs).  There would need to be a tool which transform this intermediate  
> file format into the appropriate completion function, generating the    
> appropriate call to _arguments, _values, etc.

Have you used XSLT at all? I've used it for some things and it is quite
cunning. If you use XML, I'd recommend you try it with this
transformation in mind. I actually struggled to find a decent program to
do XSL transformations - most of the information on the web relates to
direct browser support yet it makes a lot of sense currently to use XSL
on the server side. libxslt (which needs libxml2) comes with a little
program called xsltproc which is what I have used (and I have a simple
completion for it so I've attached that).

Another thing you might want to think about is if your XML format can
somehow be flexible enough to include support for multiple languages.

It'll be interesting to see but to be honest I'm a bit doubtful that an
XML equivalent will be any easier to modify or understand that the
current script. A gui wizard might do more to easing things but then
that could use the XML. The completion functions may not have gained
much in ease of writing over compctls (and as you said this is largely
because of the provision greater functionality) but I think they are a
lot more readable. I could never remember all the compctl flags and
always needed the manual to write them whereas tcsh completes were easy.

Oliver

#compdef xsltproc

_arguments -C \
  -v --{verbose,timing,repeat,debug,novalid,noout,maxdepth} \
  '1:stylesheet:_files -g \*.xsl' \
  ':file:_files -g \*.xml'

_____________________________________________________________________
This message has been checked for all known viruses by the 
MessageLabs Virus Scanning Service. For further information visit
http://www.messagelabs.com/stats.asp

Re: Generating completion functions from XML

[Felix Rosencrantz]

>Note that the exclusion lists are not necessarily repeated. An option
>listed in the exclusion list is excluded after the option and there
>are some commands where the exclusion is dependent on the ordering. I
>can't think of much information repetition in the current _arguments
>syntax.

I know there are some tricky exclusion list rules where it is not
always repetitive, though I think it is more common to use the same
set.  Also, the other place for repetition is for equivalent flags
(short/long versions).  Also, I'm hoping that there might be ways to
take advantage of shared flags that commands with subcommands (e.g.
cvs, bitkeeper, clearcase) typically have.  Other than that, I agree
there isn't much repetition.

>Have you used XSLT at all? I've used it for some things and it is
>quite cunning. If you use XML, I'd recommend you try it with this
>transformation in mind. I actually struggled to find a decent program
>to do XSL transformations - most of the information on the web relates
>to direct browser support yet it makes a lot of sense currently to use
>XSL on the server side. libxslt (which needs libxml2) comes with a
>little program called xsltproc which is what I have used (and I have a
>simple completion for it so I've attached that).

I got hooked on XSLT, and that has actually fueled my interest in XML.
I will keep it in mind for generating _argument calls.  There are many
constraints for this file, which probably won't all be satisfied in
initial drafts.

I would like to make it simple to create the initial XML; it should
be possible to write simple throw away shell/perl scripts that just
toss out the facts (flag names, help, exclusion, actions) and have XSLT
clean it up.  Another thing that interests me is using XSLT to augment
the XML command description with generic documentation describing the
XML.  Another thing would be to use XSLT to customize an _arguments
call before it is generated.  (I've been using Xalan from Apache&IBM.
I've included my simple completion.)

>Another thing you might want to think about is if your XML format can
>somehow be flexible enough to include support for multiple languages.

That's a great idea.  And this is one of the places where the XML would
actually contain information you can't easily place in _arguments call.

I sent mail recently about the shell standard talking about some
completion-like things (prediction, correction, validation) the shell
might do someday which might also benefit from having a declarative
description of a command.  Also, it might be a useful format to
generate a gui wrapper for a command.

>It'll be interesting to see but to be honest I'm a bit doubtful that
>an XML equivalent will be any easier to modify or understand that the
>current script. A gui wizard might do more to easing things but then
>that could use the XML. The completion functions may not have gained
>much in ease of writing over compctls (and as you said this is largely
>because of the provision greater functionality) but I think they are
>a lot more readable. I could never remember all the compctl flags and
>always needed the manual to write them whereas tcsh completes were
>easy.

I agree I find many are readable, but there are a lot of special cases
for which I would need to read the manual. Also, I don't find them so
editable...  I find it hard to create and edit an _arguments call.
(e.g. What's the difference between help for the flag vs. help for the
flag's argument?  I know now, but it took me a little time to grok.) I
have the manual accessible, and am reading as much as I need to and
then flip back to my completion function.  Or I tend to look at an
example function.  I hope that pertinent documentation can be placed in
the XML right at the place you need it.  Also, folks are at different
levels of understanding of the system, so it might be possible to add
different levels of insertable documentation.

I guess the process I see for creating a new completion function would
be to take the command usage and give it to some tool (maybe in perl),
which would generate some simple XML.  There might be another tool that
could take the man page for the command, and generate more XML that can
augment the XML from the first script.  Together this generated XML
would be a draft that could be processed by an XSLT sytlesheet to come
up with a more canonical form of the XML.  Then the user would edit
this XML (e.g by hand, with a nifty gui tool, or via web based gui).
If editing by hand, the user may also run an XSLT stylesheet that adds
zsh documentation to describe things like how exclusion groups work or
how to handle equivalent flags.  Then when the user is done or wants
to try out the new completion, the user would run some sort of XSLT
stylesheet to generate the function.  (It shouldn't be too hard to
generate some sort of compctl or tcsh completion, if that is wanted.
....)

I guess I'm still hopeful that there will be tools to help with the
process, and I think XML is an easier format to parse than a zsh
script.  XML has the potential to provide a good medium for the user to
understand, even though it will contain a lot more syntactic sugar.

-FR.

#compdef org.apache.xalan.xslt.Process
    local context state line
    typeset -A opt_args

_arguments \
    "-IN:inputXMLURL: _files" \
    "-XSL:XSLTransformationURL:_urls" \
    "-XSL:XSLTransformationURL:_urls" \
    "-LXCIN:compiledStylesheetFileNameIn:_file" \
    "-LXCOUT:compiledStylesheetFileNameOutOut:_file" \
    "-PARSER:fully qualified class name of parser liaison:java_class" \
     "-E[Do not expand entity refs]" \
    "-QC[Quiet Pattern Conflicts Warnings]" \
    "-Q[Quiet Mode]" \
    "-LF[Use linefeeds only on output {default is CR/LF}]" \
    "-CR[Use carriage returns only on output {default is CR/LF}]" \
    "-ESCAPE:(Which characters to escape {default is <>&\"'\\r\\n}:" \
    "-INDENT:(Control how many spaces to indent {default is 0}):" \
    "-TT[Trace the templates as they are being called.]" \
    "-TG[Trace each generation event.]" \
    "-TS[Trace each selection event.]" \
    "-TTC[Trace the template children as they are being processed.]" \
    "-TCLASS[TraceListener class for trace extensions.]" \
    "-VALIDATE[Set whether validation occurs.  Validation is off by default.]"
\
    "-EDUMP[{optional filename} (Do stackdump on error.]:_file" \
    "-XML[Use XML formatter and add XML header.]" \
    "-TEXT[Use simple Text formatter.]" \
    "-HTML[Use HTML formatter.]" \
    "-PARAM:name expression (Set a stylesheet parameter):" \
    "-MEDIA[use media attribute to find stylesheet associated with a
document.]::" \
    "-SX[User Xerces Serializers]"


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Re: Generating completion functions from XML

[Andrej Borsenkow]

On Sun, 22 Jul 2001, Felix Rosencrantz wrote:

>
> I know there are some tricky exclusion list rules where it is not
> always repetitive, though I think it is more common to use the same
> set.  Also, the other place for repetition is for equivalent flags
> (short/long versions).
>

I really wish we had different notation for equivalent flags. Exclusion
lists at this point are already overloaded and loko ugly (most important,
tgey force us to repeat the whole line). Consider

-b, --binary
-t, --text

where two lines are obviously incompatible. You can't write it using brace
notation; you need *four* lines! like

(-b --binary -t)--text...
(-b --binary --text)-t...

O.K. it could be shrtened to two lines but then quoting becomes unnatural
and too error prone.

>
> >Another thing you might want to think about is if your XML format can
> >somehow be flexible enough to include support for multiple languages.
>
> That's a great idea.  And this is one of the places where the XML would
> actually contain information you can't easily place in _arguments call.
>

Why? Assuming that main code for _arguments is in C (as I believe is the
case) it makes it quite easy to add call to gettext there. Current message
strings will the serve as keys to message catalog. Actually, GNU gettext
makes such transition transition natural.

We stil have strings that are not passed to C code, of course. But I doubt
that XML will help here. The problem is not to generate them but rather to
select at run time.
-
andrej

Re: Generating completion functions from XML

[Sven Wischnowsky]

Andrej Borsenkow wrote:

> ...
> 
> I really wish we had different notation for equivalent flags. Exclusion
> lists at this point are already overloaded and loko ugly (most important,
> tgey force us to repeat the whole line). Consider
> 
> -b, --binary
> -t, --text
> 
> where two lines are obviously incompatible. You can't write it using brace
> notation; you need *four* lines! like
> 
> (-b --binary -t)--text...
> (-b --binary --text)-t...
> 
> O.K. it could be shrtened to two lines but then quoting becomes unnatural
> and too error prone.

In this case '(-b --binary -t --text)'-{-{text,binary},b,t}'...' would work.

Remember, an option without the `*' flag is mutually exclusive to itself
(internally it really just gets added to its exclusion list, so one
could just add it by hand).


Bye
  Sven


-- 
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de