Re: New configuration -- should we...

Re: New configuration -- should we...

[Sven Wischnowsky]

Bart Schaefer wrote:

> ...
> 
> There's still the issue of parameterizing the action, which is what the
> "style" part of the tag syntax accomplishes.  Your second suggested
> implementation makes this explicit, which I like (though prefixing them
> with hyphens to look like options may not be necessary).

I'll say something about this below...

> One further thought, at the risk of distraction:  Aren't these tags just
> describing more contexts?  That is, what we're really trying to do is
> split the whole system into stages: (1) determine context, (2) generate
> appropriate matches, (3) organize matches for display.  The farther apart
> we can drive these, the better; maybe there's even some way to cause the
> tags to become context-keys for _normal, or the equivalent?

I'm not entirely sure what you mean by (3). For now I'll take it as
`say how the matches should be generated'. I.e. things like `always',
`only if a certain prefix is on the line' `with descriptions if
possible' (maybe you also meant `how they are sorted in the list' --
group-wise sorted, that is?). Anyway, this is the stuff I suggested
the styles for. See below.

About the `context-keys for _normal': that's coming back to the very
beginning of the discussion for the new completion system. And, yes,
this has been on my mind a lot lately (and your remark about wishing
that there was a better correspondance between tag-names and function-
names in a previous mail is connected to this, isn't it?).

I can only repeat what I said then: for some of these we already have
special functions (and since some time I've been trying to generate
more of these -- that's why `_pids' appeared some time ago, for
example). However, many of the sub-contexts in completion functions
are so specialised that it doesn't seem to make sense to add separate
functions for them.

But... for some... yes. Maybe we should also try to think more in this 
direction. Probably add a sub-dispatcher (in the way `_complete' and
`_normal' are global dispatchers now) which automatically calls the
right function for a certain context. Maybe some help from our friend
`compdef' for this.

I'll first think some more about the other stuff mentioned below and
once we have that, try to think more about this (please someone
remember me if I forget). I think this is possible because I currently 
can only think about such a dispatcher which could easily be build on
top of the basic tag stuff.

> ...
> 
> } There are two ways how we could make that function be called. Either
> } we do it only on the first call to `_tags' or on every call. The
> } latter would probably be more general but the first one presumably
> } would be easier when writing such a config function.
> 
> I would think this really boils down to a computation-vs-memory tradeoff.
> That is, do you want to compute the mapping once and store it forever, or
> recompute it every time to save the storage?  The function itself should
> not need to be significantly different either way, should it?

If it is called for each set, it would have to check which set is
wanted. Something like `case $attempt in 1) ...;; 2) ....;; esac'.

> } For the way how the config function says which tags are to be used I
> } currently see two possibilities (well, more than two, but these are
> } the easiest I see). In both cases the config function is only called
> } when `_tags' is called with the tags for the current state as
> } arguments, not when it is called to get the next set of tags to try.
> 
> Thus separating "determine context and organize display" from "generate
> matches."  Can you see a way to separate the first two, as well?

Again, see below.

> ...
> 
> I like this so much better that I've ignored your first suggestion.

Good. Just the way I feel myself.

> }    There could also be support to easliy set the pure config-tags,
> }    i.e. the pseudo tags we add as a replacement for the config keys we 
> }    have now. I haven't yet thought too much about that part.
> 
> Config keys / pseudo tags are the user's expression of generic behavioral
> preference.  Maybe the right way to think of them is as global defaults
> for the comptry arguments, which can then be overridden locally by each
> config function?  (Your follow-up in 8626 makes me think you're thinking
> this way, too.)

[ below:    # I'm a goto-label ]

If I'm right in thinking that `organize for display' is what the
styles are used for, then, yes, I had already realised that we would
want to be able to define them separately. In my `conf2' version I had 
added some extra syntax for giving global definitions for style.
Something like (this is just text, not actual `conf2'-syntax): for
tag `options' always use the `description' style. That effectively
defined a global default value. That was one of the things that made
me try to look at the whole thing again.

(I like the word `style' for this even if there may be better words
for it in your language, so I'll continue to use it; hope that isn't
too irritating...)

Maybe we should separate the management for the styles from the tags
stuff (at least in concept). On the completion function side, btw, it
is already this way: they say which tags they can offer, get the sets
and before trying a certain tag look at the styles set for it.

However, it would certainly make sense to be able to define styles on
the narrower context `tag so-and-so'. I mean: as you pointed out, tags 
are really a bit of a narrower context-description, or more precisely, 
the combination of completion-context (function and context-in-that
function, the latter being what I had called only `context' in
previous mails) and tag gives a narrower context. I have this image of 
a one-column table for the overall completion contexts (`-foo-' and
commands), another on-column table for the tags, separate from the
first one. And then there is a third table for the styles, containing
things like `show descriptions or not', `show a common prefix or not'.

(Btw, I think all (or most of) this is just repeating what you said,
I'm just typing-while-thinking, so please forgive me for this...)

Now, the `_tags' mechanism connects the first two tables. And since I
think the third one is more general than even the overall-context-table,
I think we could or should have a way to connect it to the first two
tables. I.e. lets define a way to say how styles should be set for a
certain command-context *or* for a certain tag *or* the narrower
context of tag-for-command-context both. That plus global defaults --
for `_tags' there is a already a way to define such global defaults by
simply using the `*)' in the examples I wrote. 

For setting global defaults and styles for certain contexts it seems
to be enough to add some builtin or function that is called from the
user's init-files. However, for the tag-plus-command-context thing it
might still be convenient to be able to just set them in the config
function where one has to work with the contexts anyway. But that
means only (and would be documented that way) that the tag-mechanism
is a friend of the style-mechanism and is able to (locally or
temporarily) register such links in the style-mechanism. It would also 
be possible for users to globally define such mappings from exact-
context-even-with-tag to styles.

Before I got your reply I had already started to think in this
direction, mainly caused by the option-syntax I suggested. I was
thinking about somthing like `global options'. And I thought that this 
might even be a another reason for using the option-syntax. Because
you could then use a global `-describe=yes' (or just `-describe') and
override it for a certain context with `-describe=no' (or
`-no-describe'). That would probably look familiar for many users.

But I still have to think some more about this separation to be able
to see if a static definition is enough or if we should try to make
this use some user-defined function, too. The latter seems cleaner
when we use this approach everwhere else. I don't know about the
how, where, when, etc. yet, though. But I've become rather cautious
(or timid) after the shock yesterday.


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: New configuration -- should we...

[Bart Schaefer]

On Nov 11,  3:12pm, Sven Wischnowsky wrote:
} Subject: Re: New configuration -- should we...
}
} 
} Bart Schaefer wrote:
} 
} > One further thought, at the risk of distraction:  Aren't these tags just
} > describing more contexts?  That is, what we're really trying to do is
} > split the whole system into stages: (1) determine context, (2) generate
} > appropriate matches, (3) organize matches for display.  The farther apart
} > we can drive these, the better; maybe there's even some way to cause the
} > tags to become context-keys for _normal, or the equivalent?
} 
} I'm not entirely sure what you mean by (3). For now I'll take it as
} `say how the matches should be generated'. I.e. things like `always',
} `only if a certain prefix is on the line' `with descriptions if
} possible' (maybe you also meant `how they are sorted in the list' --
} group-wise sorted, that is?). Anyway, this is the stuff I suggested
} the styles for.

For (3) I thought almost entirely of `how groups are sorted in the list'
and `order by rows or by columns' and maybe even `use menu-completion or
menu-selection' and that kind of thing.  `always' or `only if a certain
prefix is on the line' are part of (1); `with descriptions if possible'
is a part of (2).

} [...] for some of these we already have
} special functions (and since some time I've been trying to generate
} more of these -- that's why `_pids' appeared some time ago, for
} example). However, many of the sub-contexts in completion functions
} are so specialised that it doesn't seem to make sense to add separate
} functions for them.

I quote that slightly out of context so that I can say:

The view I was attempting to take in the previous message is that every
context has can be thought of as identifying an action.  If you treat
the action as a function (even if it's not complex enough to warrant one
in actual implementation), you can then think of the style as the set of
arguments to be passed to that function to specify/control its behavior.

Except that really the styles are collections of arguments for something
that might best be described by more than one function call, each of which
uses a different subset of those arguments.

} (I like the word `style' for this even if there may be better words
} for it in your language, so I'll continue to use it; hope that isn't
} too irritating...)

No, that's fine.  This relates directly to what I said about the user's
understanding/expression of his preference being different from how zsh
needs to process it.  The style is a user-expression concept.

} > } There are two ways how we could make that function be called. Either
} > } we do it only on the first call to `_tags' or on every call. The
} > } latter would probably be more general but the first one presumably
} > } would be easier when writing such a config function.
} > 
} > I would think this really boils down to a computation-vs-memory tradeoff.
} > That is, do you want to compute the mapping once and store it forever, or
} > recompute it every time to save the storage?  The function itself should
} > not need to be significantly different either way, should it?
} 
} If it is called for each set, it would have to check which set is
} wanted. Something like `case $attempt in 1) ...;; 2) ....;; esac'.

Oh, I see.  No, I wouldn't recommend calling it for each set.  I was
thinking of something else, which in retrospect wouldn't work anyway.

The rest of what you said makes sense; I don't have any specifc remarks.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: New configuration -- should we...

[Sven Wischnowsky]

[ I wanted to reply to this on Friday and then forgot... sorry. ]

Bart Schaefer wrote:

> } > One further thought, at the risk of distraction:  Aren't these tags just
> } > describing more contexts?  That is, what we're really trying to do is
> } > split the whole system into stages: (1) determine context, (2) generate
> } > appropriate matches, (3) organize matches for display.  The farther apart
> } > we can drive these, the better; maybe there's even some way to cause the
> } > tags to become context-keys for _normal, or the equivalent?
> } 
> } I'm not entirely sure what you mean by (3). For now I'll take it as
> } `say how the matches should be generated'. I.e. things like `always',
> } `only if a certain prefix is on the line' `with descriptions if
> } possible' (maybe you also meant `how they are sorted in the list' --
> } group-wise sorted, that is?). Anyway, this is the stuff I suggested
> } the styles for.
> 
> For (3) I thought almost entirely of `how groups are sorted in the list'
> and `order by rows or by columns' and maybe even `use menu-completion or
> menu-selection' and that kind of thing.  `always' or `only if a certain
> prefix is on the line' are part of (1); `with descriptions if possible'
> is a part of (2).

First aside: This reminded me that I wanted to make the options affecting
completion be controllable via $compstate. I had forgotten to do that
for the new list options.

Second aside: Hm, wouldn't users probably think that `with descriptions 
if possible' is part of (3), even if it is part of (2) for the
completion function writer?


We could make things like these configurable via styles, too. I.e. I
don't see a reason for using a different configuration mechanism for
things like that. It would then affect groups of matches as a whole
which I like anyway. But that would also mean that a) it is even more
important that everyone uses `_description' whenever adding matches
and b) that we should always use grouping (otherwise we can't really
control such things on a per-group basis).

We could then even implement styles that allow users to say if they
want the group sorted and/or uniquified. And we could make the new
listing-options be settable per-group and let users define which types 
of matches they want packed.

Some of the things (like menu-completion), however, can obviously not
be controlled on a per-group basis. Only in the sense of `if there
*are* any matches in this group (meaning: for this tag)', then use
menu-completion or -selection or whatever.

Another thing would be the naming of groups. The completion system
heads into the direction of having a one-to-one relationship between
tags and groups. We could make the group names (the ones used
internally) be configurable, defaulting to the tag-name. Users could
then specify if they want to put matches for different tags into the
same group by specifying mappings from tags to group names.

The `how groups are sorted in the list' finally is probably the most
interesting one but also the most complicated. The problem is that the 
completion code just displays the groups in the order in which they
were added. So we would either have to implement a way so that
completion functions can change that order before the completion code
starts listing the stuff or we would have to make the completion
system actually add the groups in a different order. For that we would 
have to change the tag-loops we have now into two nested loops.

Hm. The first one reminds me of the access-to-match-information stuff
I've been talking about sometimes. Maybe if we had a special array
containing the names of the groups added (or an association mapping
group names to some information) we could allow users to modify that
array and the completion code would then use the groups in the order
in which they appear in the array after the widget has returned. I
can't think of a good way to configure this, though.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: New configuration -- should we...

[Bart Schaefer]

On Nov 11, 11:24am, Sven Wischnowsky wrote:
} Subject: New configuration -- should we...
}
} [Maybe you'll all think that I have now gone completely mad...]

Was that a pun?

} Bart once said that if you have a programming language, don't implement 
} a second one -- and somehow the new completion system came into
} existence.

Apparently I need to listen to myself more often.

} The completion system finds out the context in which completion is
} attempted. This is a process in multiple steps, the overall context is 
} found using the `#compdef' stuff. The function thus then probably
} knows about other finer-grained contexts. In each of these contexts
} one or more types of matches can be generated. This is the place where 
} `_tags' is used.

Here's a viewpoint that just occurred to me, and that might be helpful:

The shell itself knows syntactic contexts.  These are represented by
the special `-context-' tokens.  The shell also knows that when the
`-command-' context is "filled in" (needs no further completion), it then
implies a particular semantic context (completing the arguments of the
specific command).  #compdef maps the contexts the shell knows about to
the functions that generate matches for those contexts.

Superficially, every such function is different, because the semantics
of every `-context-' and every command differ from the others (if they
didn't, you wouldn't need to handle them individually).  However, the
very nature of interactions with filesystems and environments means that
there are lots of overlaps in those semantics.  (Otherwise the old
compctl system, and completion in other shells, wouldn't work at all.)
The idea behind _tags is to give names to those overlapping regions, and
to organize them when more than one applies in any given situation.

Now:
 
} Instead of using some internally held configuration state we could
} simply make `_tags' call a user-supplied function which gets the tags
} supported in the current context and then has to `sort' them.

I like this approach because it separates the "naming" function of _tags
from the "ordering" function.  That is, the current syntax for a tag
represents a data structure that simultaneously maps a name to an action
and a name to a priority.  This new suggestion allows the tag syntax to
bear only one of those burdens -- mapping name to action -- and leaves
it up to the user-supplied function to do the ordering.

There's still the issue of parameterizing the action, which is what the
"style" part of the tag syntax accomplishes.  Your second suggested
implementation makes this explicit, which I like (though prefixing them
with hyphens to look like options may not be necessary).

One further thought, at the risk of distraction:  Aren't these tags just
describing more contexts?  That is, what we're really trying to do is
split the whole system into stages: (1) determine context, (2) generate
appropriate matches, (3) organize matches for display.  The farther apart
we can drive these, the better; maybe there's even some way to cause the
tags to become context-keys for _normal, or the equivalent?

The trick is to balance the way zsh wants to organize things -- as the
three stages above, each separately parameterized -- with the way users
want to understand/express it, which is [*] first by generic behavioral
preference for each "overlapping region" and second in large chunks by
outermost context (i.e. per command).

[*] As I see it.

} There are two ways how we could make that function be called. Either
} we do it only on the first call to `_tags' or on every call. The
} latter would probably be more general but the first one presumably
} would be easier when writing such a config function.

I would think this really boils down to a computation-vs-memory tradeoff.
That is, do you want to compute the mapping once and store it forever, or
recompute it every time to save the storage?  The function itself should
not need to be significantly different either way, should it?

} For the way how the config function says which tags are to be used I
} currently see two possibilities (well, more than two, but these are
} the easiest I see). In both cases the config function is only called
} when `_tags' is called with the tags for the current state as
} arguments, not when it is called to get the next set of tags to try.

Thus separating "determine context and organize display" from "generate
matches."  Can you see a way to separate the first two, as well?

} 2) The second way is based on support from `computil' (in a first
}    implemention-to-play-with it could be entirely in shell code).

I like this so much better that I've ignored your first suggestion.

}    There could also be support to easliy set the pure config-tags,
}    i.e. the pseudo tags we add as a replacement for the config keys we 
}    have now. I haven't yet thought too much about that part.

Config keys / pseudo tags are the user's expression of generic behavioral
preference.  Maybe the right way to think of them is as global defaults
for the comptry arguments, which can then be overridden locally by each
config function?  (Your follow-up in 8626 makes me think you're thinking
this way, too.)

}    Only so
}    much: with support by C-code it would almost certainly be easy
}    enough to allow really arbitrary types for them. E.g. we could make 
}    them be reported as strings, arrays, and assocs. With that it would
}    even be possible to use the same mechanism to replace some (or
}    all?) of the parameters we use now. This is especially interesting
}    for things like the `COMMAND_hosts_ports_user' because for the
}    `COMMAND' part we could use the context-switching this
}    configuration stuff is all about anyway. I /think/ this would make
}    it easier for users.

I think this is looking in the right direction, but I can't think deeply
enough into it at this point to be sure.

} Ok, the more I think about this, the more I like it.

Yes.

} Also, someone please tell me if this is too complicated for the average 
} user. Is something like `conf2' really easier to understand?

See my "The trick is ..." paragraph above.

} Or does anyone have suggestions for, e.g. builtins or helper functions 
} that would allow us to make the config funcs better readable or
} understandable?

I don't yet ...

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: New configuration -- should we...

[Sven Wischnowsky]

I forgot to mention:

One of the things I want all this new configu stuff for is to be able
to control more detailed what matches will be generated form the
*outside* of the (real) completion system. I.e. functions that use the 
normal completion context but restrict/change what would normally be
added.

In my previous suggestions I had the `override_tags' assoc which was
so ugly that I already removed it in my private version, in the hope
that I could find something better.

When we the function approach this would be extremly simple. `_tags'
checks if a parameter with a certain name is set and non-empty. If so, 
it uses that value as the name of the config function to
call. Functions that which to override the normal settings would then
just use their own config function, basta.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

New configuration -- should we...

[Sven Wischnowsky]

[Maybe you'll all think that I have now gone competely mad...]

I was hacking away happily on the new configuration stuff yesterday
(namely my version of conf2) and thinking about how all this could be
improved when we move it into C-code and make it read from a
file/stdin. That made me think about things like quoting which in turn 
made me think about how we could use existing code for that. Then I
had the feeling that this somehow went wrong, looked again at the
examples for calling that configuration function and  then came to a
sudden halt. It just sounded far too familiar.

Bart once said that if you have a programming language, don't inplment 
a second one -- and somehow the new completion system came into
existence.

Well, the configuration stuff is already quite complicated (a.k.a.
powerful). And what's worse, the internals are quite messy.

So, for the moment forget everything I ever said about the new
configuration stuff... (some of the following should probably be clear 
by now, I just want to give the full picture again).


The completion system finds out the context in which completion is
attempted. This is a process in multiple steps, the overall context is 
found using the `#compdef' stuff. The function thus then probably
knows about other finer-grained contexts. In each of these contexts
one or more types of matches can be generated. This is the place where 
`_tags' is used. The function calls it to say which types of matches
(represented by the tags) are sensible in the current context. After
that, `_tags' is called repeatedly and says which tags are to be used
first, second, etc.

Instead of using some internally held configuration state we could
simply make `_tags' call a user-supplied function which gets the tags
supported in the current context and then has to `sort' them. I.e. it
says things like `first try glob, then paths, then files'. So `_tags'
would only be a wrapper around this function, setting up some
parameters (and probably other stuff?) that makes the user's life
easier when (s)he writes h(er|is) own config function.
There are two ways how we could make that function be called. Either
we do it only on the first call to `_tags' or on every call. The
latter would probably be more general but the first one presumably
would be easier when writing such a config function.

A default config function would be supplied by us, setting those
things up that we think are good ideas. Users could, of course, then
just copy and modify that function once they feel the need. Obviously
we could also build a default implementation for the function that
works together with something like `conf2' but I'm not particularly
happy with that idea.

For the way how the config function says which tags are to be used I
currently see two possibilities (well, more than two, but these are
the easiest I see). In both cases the config function is only called
when `_tags' is called with the tags for the current state as
arguments, not when it is called to get the next set of tags to try.

1) The config function sets up a bunch of arrays containing the tags
   to try. E.g.:

    compconfig() {
      tags1=( arguments 'values[description]')
      tags2=( 'options[description,hidden-prefix]')

      case "$command" in
      *dvi*)
        tags3=( glob paths )
	tags4=( files )
	tags5=( "$@" )
        ;;
      *)
        tags3=( glob )
        tags4=( paths )
        tags5=( files )
	tags6=( "$@" )
	;;
      esac
    }

   `$command' is set up by `_tags' and so is `$context' (the latter
   for the context in the current completion function. The names for
   the arrays could probably be better. Assigning `( "$@" )' at the
   end ensures that all tags not explicitly sorted are used anyway,
   but only after all the named tags are tried. `_tags' could easily
   remove the tags that were not requested from the arrays.

   After calling `compconfig', `_tags' would get all those `tags*'
   arrays and store them internally (before calling `compconfig' there 
   were cleared).
   Later, when `_tags' is called to retrieve the tags to try, it would 
   use the contents of the arrays one after another.

   There are some uglinesses with this, though. the first one is that
   this requires many array-copies. The second one is that we can't
   move much of it into C-code, trying to make things faster.

2) The second way is based on support from `computil' (in a first
   implemention-to-play-with it could be entirely in shell code).
   Instead of setting up arrays, the config function calls a builtin
   multiple times. E.g.:

    compconfig() {
      comptry arguments value -description
      comptry options -description -hidden-prefix

      case "$command" in
      *dvi*)
        comptry glob paths
	comptry files
	;;
      *)
        comptry glob
        comptry paths
        comptry files
	;;
      esac

      comptry "$@"
    }

   (Note that I made styles look a bit like options -- another syntax
   would be possible -- again I don't know what users would like to
   have, but maybe making styles look like options for tags isn't that 
   bad a way to think about them?).

   `comptry' (or should it be `comptags'?) would store the tag-sets
   (one set per call) internally and functions like `_requested' would
   use it with options saying how they want to access the data stored
   for the previous call to `_tags'. The "$@" and the comment about
   tags which are set but not really used in this context: same as in
   the first example.

   There could also be support to easliy set the pure config-tags,
   i.e. the pseudo tags we add as a replacement for the config keys we 
   have now. I haven't yet thought too much about that part. Only so
   much: with support by C-code it would almost certainly be easy
   enough to allow really arbitrary types for them. E.g. we could make 
   them be reported as strings, arrays, and assocs. With that it would
   even be possible to use the same mechanism to replace some (or
   all?) of the parameters we use now. This is especially interesting
   for things like the `COMMAND_hosts_ports_user' because for the
   `COMMAND' part we could use the context-switching this
   configuration stuff is all about anyway. I /think/ this would make
   it easier for users.

   Since much of the stuff could be written in C, performance
   shouldn't be too bad anyway, but doing it as in the example would
   also allow us to use aggressive caching.


Ok, the more I think about this, the more I like it. It is much
cleaner than my preevious suggestions, I think. It is arbitrarily
extensible if need be. Together with some C-code magic it should be
fast and it allows user not only to say what they want to have
completed, but also on what this decision should be based (i.e. what
they want to test). So, if you want to stop me, please do it now ;-)

Also, someone please tell me if this is too complicated for the average 
user. Is something like `conf2' really easier to understand?

Or does anyone have suggestions for, e.g. builtins or helper functions 
that would allow us to make the config funcs better readable or
understandable?


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de