User-defined zle widgets and built-in widget failure

User-defined zle widgets and built-in widget failure

[Bart Schaefer]

I'd like to write a widget that modifies BUFFER, calls a built-in widget,
and then if that built-in widget fails, restores the previous BUFFER and
CURSOR.

Unfortunately, failure of "zle whatever" isn't detectable; "zle" always
sets $? to zero.  What I'd prefer is something like, whenever a feep() is
emitted, then "zle" has a nonzero value, so you can write stuff like

savebuffer=$BUFFER
BUFFER=$searchstring
CURSOR=$#searchstring
zle history-beginning-search-backward || BUFFER=$savebuffer

While I'm on the topic, it seems to me that the current/first/last history
numbers would be useful bits of information for zle widgets.  Then you
could do things like

histno=$HISTORYNUMBER
zle history-beginning-search-backward
[[ $histno != $HISTNUMBER ]] || BUFFER=$savebuffer

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: User-defined zle widgets and built-in widget failure

[Zefram]

Bart Schaefer wrote:
>I'd like to write a widget that modifies BUFFER, calls a built-in widget,
>and then if that built-in widget fails, restores the previous BUFFER and
>CURSOR.
>
>Unfortunately, failure of "zle whatever" isn't detectable; "zle" always
>sets $? to zero.

Failure internally is recorded by the feep flag.  If that is set at
the end of the main loop, a beep is emitted and the flag reset.  All we
need to do is make that flag visible to widgets.  Like a lot of other
ZLE things, this is one of the things I had a fairly concrete plan for
when my spare time suddenly disappeared last year.

The mechanism I planned is that each thingy will include a single-bit
flag, which can be tested and modified using the zle builtin.  Then all
you do to test for error is "if zle -F feep; ...", where the value of
"F" has yet to be determined.  Other thingies would have flags attached
similarly.  Or a more general possibility would be thingy array variables
accessible via thingy names, or possibly by more magic parameter names
(though I'd want to do a proper namespace separation, e.g., ${zle.feep}).

That reminds me, more generally I'd like to move the zsh-specific
parameter names to completely separate names that can't conflict
with POSIX parameter names.  If we have ${zsh.module_path} instead of
$module_path, then this can be available regardless of the name used to
invoke zsh.  What do people think?

-zefram

Re: User-defined zle widgets and built-in widget failure

[Bart Schaefer]

On Jun 9,  7:01pm, Zefram wrote:
} Subject: Re: User-defined zle widgets and built-in widget failure
}
} The mechanism I planned is that each thingy will include a single-bit
} flag, which can be tested and modified using the zle builtin.  Then all
} you do to test for error is "if zle -F feep; ...", where the value of
} "F" has yet to be determined.

I think we should tie the most common cases to standard scripting concepts;
e.g., failure should be a nonzero $?, and the numeric value of $? should
indicate the reason for failure.  It shouldn't be necessary to invoke "zle"
multiple times to find out this kind of basic information; further, it
makes it obvious how a widget writer sets the value, i.e., with "return".

} Or a more general possibility would be thingy array variables
} accessible via thingy names

Just how many thingys are we talking about here?

I suggest defining one array variable $ZLE (or whatever name you want),
and define the meanings of the array positions.  E.g., ZLE could be
equivlent to ($BUFFER $CURSOR $LBUFFER $RBUFFER) in the current scheme;
instead of BUFFER="the text here" you'd say ZLE[1]="the text here".
(I don't think you should literally do that, because the buffer chunks
are best accessed directly as scalars, but for purposes of example ...)

} (though I'd want to do a proper namespace separation, e.g., ${zle.feep}).

You're creeping dangerously close to Perl associative arrays, there.
Unless we're talking about a huge amount of data, just use numbered
positions in an array with a "sufficiently unique" name.

} That reminds me, more generally I'd like to move the zsh-specific
} parameter names to completely separate names that can't conflict
} with POSIX parameter names.  If we have ${zsh.module_path} instead of
} $module_path, then this can be available regardless of the name used to
} invoke zsh.  What do people think?

I think I'd rather not have to rewrite my .z* scripts again.  How much
"moving" are we talking about?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: User-defined zle widgets and built-in widget failure

[Zefram]

Bart Schaefer wrote:
>I think we should tie the most common cases to standard scripting concepts;
>e.g., failure should be a nonzero $?, and the numeric value of $? should
>indicate the reason for failure.  It shouldn't be necessary to invoke "zle"
>multiple times to find out this kind of basic information; further, it
>makes it obvious how a widget writer sets the value, i.e., with "return".

Good points.  I think that's worth handling as a special case, even
though there are many other bits of state that will need a more
complete mechanism.  This special case can be defined pretty well:
zle returns non-zero iff the feep flag is set at the time of return,
and the feep flag is set on return from a user-defined widget iff the
function returns non-zero.

>} Or a more general possibility would be thingy array variables
>} accessible via thingy names
>
>Just how many thingys are we talking about here?

I was thinking about a general array mechanism, using thingies to make
pushing them onto the zle stack particularly efficient.  However, it
would essentially be duplicating the existing array parameter mechanism,
so I'd rather use what we've already got.

>I suggest defining one array variable $ZLE (or whatever name you want),
>and define the meanings of the array positions.  E.g., ZLE could be
>equivlent to ($BUFFER $CURSOR $LBUFFER $RBUFFER) in the current scheme;
>instead of BUFFER="the text here" you'd say ZLE[1]="the text here".

Yuck.  This arbitrary index issue is one of the things I dislike about
Perl (the stat() interface, specifically).  Let's stick to names.  And,
of course, separate parameters is the only way to handle array values,
since we don't have nested arrays.

>} (though I'd want to do a proper namespace separation, e.g., ${zle.feep}).
>
>You're creeping dangerously close to Perl associative arrays, there.

I don't see that as an associative array.  It's just a structured
namespace for parameters.

>Unless we're talking about a huge amount of data, just use numbered
>positions in an array with a "sufficiently unique" name.

Really, we're talking about *arbitrary* amounts of data.  Non-builtin
widgets whould be able to store their own data in the same manner as
the standard widgets, and ideally be able to keep that data out of the
way of normal user variables.

>I think I'd rather not have to rewrite my .z* scripts again.  How much
>"moving" are we talking about?

Actually, I don't anticipate permanently removing names established
in 3.0.  An option, on by default in zsh mode, would enable the current
names.  Those that take the time to rewrite their .z* scripts can then
disable the current names, and get a parameter namespace as clean as
possible.  Of course, the POSIX-mandated parameters have to retain their
current names in any case, but all the zsh extensions could be renamed.

If we *don't* introduce a proper namespace scheme, I hate to think what
will happen when modules gain the ability to define special parameters.


This is probably a good time to talk about what I've been doing for the
past year.  (Standard disclaimer applies.)  I work for Tao Systems Ltd.,
which is in the business of writing a rather interesting portable OS
called Elate.  One of the main features of this OS is that each utility
and each library function is dynamically loaded, in a shareable manner
-- the equivalent under Unix would be to put every library function in
a separate shared library, though of course it's much more efficient
in Elate.

Among other things, I was assigned to design and write the standard
Elate shell.  (There are many zsh fans here.)  I have had the opportunity
to write a shell that has really clean syntax and consistent behaviour,
initially without any pressure to make the shell practically usable.
(It finished up pretty usable *and* clean, because I had the time to
achieve both aims.)  I also had the opportunity to establish many of
the conventions for this shell environment.  Having experimented with
a lot of my shell ideas on this project, I think some of them are worth
trying to get into zsh in some form.

One of the conventions I established was that all environment variables,
except for strictly user-generated ones, have structured names.
For example, the equivalent of the Unix "PATH" variable is called
"shell.path".  The equivalent of "LOCALE" is called "user.locale".
This means that scripts, or the user at the shell prompt, can use *any*
alphanumeric variable name, without any risk of colliding with anyone
else's variables, now or in the future.  And utility writers have a
guaranteed way to define variables special to a particular utility
without clashing with anyone else's variables, so there's none of this
"MAILFILE" nonsense.

What I like most about this convention is that, as the shell writer, I
can invent new special variables without colliding with anyone else.  If I
add a feature like zsh's $WATCH, I can call the parameter "shell.watch",
and there is no problem even if an existing script happened to call a
variable "watch", which is not an unlikely occurrence.

Another feature worth examining is the design of the command line editor.
I actually designed it to be much like I wanted ZLE to be.  The key
feature is that the editor is almost entirely written in shell code --
the only builtin code written for it is (1) the display updating code and
(2) an interface to the parser (for completion).  The buffer is actually
stored in shell variables ("sle.lbuffer" et al), and so is trivially
manipulable by user-defined editor functions.  In fact, user-defined
editing functions are precisely on level footing with the standard
functions, because *all* of them are simply shell functions.

I'm not suggesting that we go to that extreme with ZLE -- there is an
enormous performance hit in doing the simplest editing actions in shell
functions, and completion suffers somewhat -- but I do want to achieve
three basic aims:

	(1) as little as possible should be built into the base zle module

	(2) built-in widgets should use the same interfaces as
	    user-defined ones, so that they behave *as if* they were
	    shell functions

	(3) as much of the ZLE state as possible should be directly
	    manipulable by user-defined widgets

My experience with the Elate shell has shown that building in nothing
(even the ability to run the various types of commands is dynamically
loaded), and doing as much as possible without inventing new mechanisms,
makes a shell very easily extensible, and very radically configurable
for any specific purpose.

-zefram

Variable namespaces, goals for ZLE, etc.

[Bart Schaefer]

On Jun 9,  9:02pm, Zefram wrote:
} Subject: Re: User-defined zle widgets and built-in widget failure
}
} Bart Schaefer wrote:
} >I think I'd rather not have to rewrite my .z* scripts again.  How much
} >"moving" are we talking about?
} 
} Actually, I don't anticipate permanently removing names established
} in 3.0.  An option, on by default in zsh mode, would enable the current
} names.  Those that take the time to rewrite their .z* scripts can then
} disable the current names, and get a parameter namespace as clean as
} possible.

What we need is a way to link variable names to one another, with a few
options for how the linkage is accomplished.  The options would include
having the names share a value, automatic array-to-colon-list conversion
(and back), etc.  It'd also be useful to be able to define the way that
import and export are performed (in which case the imported and exported
values might be separate).

} If we *don't* introduce a proper namespace scheme, I hate to think what
} will happen when modules gain the ability to define special parameters.

I don't expect it'd be any worse than the current situation with external
programs having to come up with names for their environment variables.
If you can explain why a namespace like "zsh." is better than a simple
convention like "ZSH_", I'm willing to be persuaded.  However, the only
reason for introducing naming hierarchy (even if only one level of it as
you've so far proposed) is to be able to manipulate entire hierarchies
as single entities.  If the name of the space is not useful except as a
differentiating prefix, then it's better simply to use a prefix.

} This is probably a good time to talk about what I've been doing for the
} past year.

Sounds like a pretty fun job.

} three basic aims:
} 
} 	(1) as little as possible should be built into the base zle module
} 
} 	(2) built-in widgets should use the same interfaces as
} 	    user-defined ones, so that they behave *as if* they were
} 	    shell functions
} 
} 	(3) as much of the ZLE state as possible should be directly
} 	    manipulable by user-defined widgets

I don't disagree with any of those.  However, don't go overboard with (1).
An example of a case where I think that idea was carried too far is with
the removal of so many of the default compctls in 3.1.4.  It seems quite
contradictory to me to turn on by default every flashy zle option, but to
turn off all the even-more-useful completion defaults. 

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Associative arrays, structured namespaces

[Bruce Stephens]

Zefram <zefram@tao.co.uk> writes:

> One of the conventions I established was that all environment
> variables, except for strictly user-generated ones, have structured
> names.  For example, the equivalent of the Unix "PATH" variable is
> called "shell.path".  The equivalent of "LOCALE" is called
> "user.locale".  This means that scripts, or the user at the shell
> prompt, can use *any* alphanumeric variable name, without any risk
> of colliding with anyone else's variables, now or in the future.
> And utility writers have a guaranteed way to define variables
> special to a particular utility without clashing with anyone else's
> variables, so there's none of this "MAILFILE" nonsense.

This strikes me as an excellent idea.  Presumably the implementation
is similar to that of associative arrays, though?  

If you're going to have structured namespaces, I'd like to be able to
use them to do things I'd otherwise use associative arrays for, I
think, so I'd like ways of asking for all variables like "shell.*" and
stuff.  Oh, maybe there are quoting issues I haven't thought of.
Anyway, this looks quite clean.

What's then needed is some way of tying these names to other names, so
that PATH connects correctly to shell.path and so on.  This would be a
good opportunity to generalise things a bit, and allow arbitrary shell
functions to be called when a variable gets set or read, and stuff
like that.

This is all in ksh93, isn't it?  Another reason for doing it.

Re: Associative arrays, structured namespaces

[Zefram]

Bruce Stephens wrote:
>This strikes me as an excellent idea.  Presumably the implementation
>is similar to that of associative arrays, though?  

No.  The way it's done in Elate, the shell just treats the structured
variable name as an opaque string.  It's only convention that interprets
the name as having structure.  Associative arrays is a completely
separate issue.

>What's then needed is some way of tying these names to other names, so
>that PATH connects correctly to shell.path and so on.

That's relatively easy.  We already have some support for linked
parameters.

>                                                       This would be a
>good opportunity to generalise things a bit, and allow arbitrary shell
>functions to be called when a variable gets set or read, and stuff
>like that.

I'd like to do that too.

-zefram

Re: Variable namespaces, goals for ZLE, etc.

[Zefram]

Bart Schaefer wrote:
>
>} If we *don't* introduce a proper namespace scheme, I hate to think what
>} will happen when modules gain the ability to define special parameters.
>
>I don't expect it'd be any worse than the current situation with external
>programs having to come up with names for their environment variables.
>If you can explain why a namespace like "zsh." is better than a simple
>convention like "ZSH_", I'm willing to be persuaded.

The advantage of a "zsh." prefix is that *nothing* currently relies
on such a variable name existing or behaving in any particular manner,
whereas "ZSH_" names are defined by POSIX to have no special semantics.
I think it is reasonable for a script, for any shell, to be able to define
parameters with any name, including "_" characters, without having to
worry about whether (a future version of) the shell will attach unwanted
semantics to the variable name.  Of course, we can't do anything about
"PATH", etc., but we *can* get zsh-specific variables out of the way,
making zsh more reliably usable as a scripting language.

-zefram

Re: Associative arrays, structured namespaces

[Bruce Stephens]

Zefram <zefram@tao.co.uk> writes:

> Bruce Stephens wrote:
> >This strikes me as an excellent idea.  Presumably the implementation
> >is similar to that of associative arrays, though?  
> 
> No.  The way it's done in Elate, the shell just treats the structured
> variable name as an opaque string.  It's only convention that interprets
> the name as having structure.  Associative arrays is a completely
> separate issue.

Yes, I wasn't thinking clearly.  I was thinking that adding a SET
function to shell.path (which would be called shell.SET, I suppose)
and all that kind of functionality would be harder without regarding
things as separate associated arrays, but I'm clearly wrong: it's
still dead easy with a single namespace, because you know exactly what
you're looking for.

Supposing zsh has a hash table of parameter names which pointed at
values; couldn't the hash just be replaced by a btree or something?
Wouldn't that provide enough to reasonably efficiently produce all
variables beginning with some prefix?  

Probably you're right, though: associative arrays ought to be done
specially, if they're done at all.  Come to think of it, having an
associative array very_long_prefix.put_keys_next.* would probably be
too expensive done as a set of variables.  Yeah, you're definitely
right: keep things simple.

Re: Associative arrays, structured namespaces

[Bruce Stephens]

Bruce Stephens <b.stephens@isode.com> writes:

> Yes, I wasn't thinking clearly.  I was thinking that adding a SET
> function to shell.path (which would be called shell.SET, I suppose)

Slow brain strikes again.  That should presumably be "shell.path.SET".

Re: Variable namespaces, goals for ZLE, etc.

[Bart Schaefer]

On Jun 10, 11:55am, Zefram wrote:
} Subject: Re: Variable namespaces, goals for ZLE, etc.
}
} The advantage of a "zsh." prefix is that *nothing* currently relies
} on such a variable name existing or behaving in any particular manner,
} whereas "ZSH_" names are defined by POSIX to have no special semantics.

Are there any "available" names in the POSIX space?

The problem with introducing an incompatible syntax in order to escape
POSIX is that you can't reliably export variables with the new names.
I've used versions of sh that would mangle the environment and/or issue
warning messages if there were any strings present where the stuff to
the left of the `=' contained characters other than [A-Z][a-z][0-9]_.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Variable namespaces, goals for ZLE, etc.

[Zefram]

Bart Schaefer wrote:
>Are there any "available" names in the POSIX space?

Depends what you mean by "available".  POSIX lists "the special
parameters" as being "*", "@", "#", "?", "-", "$", "!" and "0"; by
implication, all other parameters have the semantics of normal variables.
It lists "IFS", "PATH", "PPID" et al as normal variables that are either
set or read by the shell, but does not permit them to have any weird
semantics beyond those it defines.  It is arguable that the shell can
modify its behaviour based on the values of other variables, but it
certainly isn't permitted to make variables with normal names behave as
special variables.

>The problem with introducing an incompatible syntax in order to escape
>POSIX is that you can't reliably export variables with the new names.

You can export anything.  Whether other utilities accept anything unusual
is, admittedly, a more complicated issue.

>I've used versions of sh that would mangle the environment and/or issue
>warning messages if there were any strings present where the stuff to
>the left of the `=' contained characters other than [A-Z][a-z][0-9]_.

We could do a configure test for that.  Perhaps if the local sh does
complain, we could import/export using munged names starting with
"__ZSH__" or something like that.  Within zsh, the variable name "__ZSH__"
would still be legal, it would just be exported under a different name.
Alternatively, we keep things simple, and just don't import/export
special variables on platforms that really can't handle it -- there
can't be many of them, surely?

-zefram

Re: Variable namespaces, goals for ZLE, etc.

[Bart Schaefer]

On Jun 10,  7:01pm, Zefram wrote:
} Subject: Re: Variable namespaces, goals for ZLE, etc.
}
} Bart Schaefer wrote:
} >I've used versions of sh that would mangle the environment and/or issue
} >warning messages if there were any strings present where the stuff to
} >the left of the `=' contained characters other than [A-Z][a-z][0-9]_.
} 
} We could do a configure test for that.  Perhaps if the local sh does
} complain, we could import/export using munged names starting with
} "__ZSH__" or something like that.

It'd be much better to do the same thing everywhere.  Consider a user who
can't change his shell to be zsh, but who wants to export the ZDOTDIR
variable from his "regular" shell before exec'ing zsh.  It'd be nice if
zsh always imported it from the same name, regardless of platform and
regardless of what it later called it internally.

} Alternatively, we keep things simple, and just don't import/export
} special variables on platforms that really can't handle it -- there
} can't be many of them, surely?

Depends on what you mean by "many."  IIRC, the sh on a Sun platform was
among those that issued the complaints.  (I forget now what OS version.)
It could be a "few platforms, but many installations" kind of thing.

Now, for truly "special" parameters it may not make any difference whether
they can be exported (and if it doesn't, again we should be consistent and
never export them on any platform).  But for "ordinary" parameters that
happen to be used by zsh in a predefined way, lack of export may be quite
a drawback.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com