a plan for ZLE extendability

a plan for ZLE extendability

[Zefram]

There is a need for ZLE to be extendable without having to write a
binary module.  Any system of extension has to allow

1. execution of arbitrary zsh commands from within ZLE

2. execution of arbitrary ZLE commands from within zsh commands

3. arbitrary modification of the ZLE buffer and minibuffer from within
   zsh commands

To this end, I have devised the following system.

It will be possible for the user to register a shell function as a
ZLE command.  (Internally, this can be done using a slightly modified
addzlefunction().)  This would look like

zle -f new-zle-command shell_function

and the shell_function argument could default to being the same as
the ZLE command name.  Such a ZLE function could be similarly deleted
(using deletezlefunction()).  These added ZLE functions can, of course, be
bound to keys, and executed in all the ways normal ZLE functions can be.
This meets the first requirement.

The third requirement (modification of buffers) can be met most generally
by adding a special parameter -- perhaps $ZLE_BUFFER -- that can be
assigned to.  $ZLE_MINIBUFFER would also be required.  To control
the cursor position, $ZLE_BUFFER could be split into two parameters,
representing the parts of the buffer before and after the cursor.
An alternative technique would be a numeric special parameter containing
the cursor offset.  Both methods could be used, even intermixed.
Obviously, these parameters could only be special while ZLE is active.

For the second requirement, execution of ZLE commands from zsh, a
more complex solution is required, to handle repeat counts and other
additional arguments.  I envisage a system in which ZLE maintains a
stack of what (for the want of a better name) I shall call "thingies".
A thingy is normally a string, but for efficiency we might want to handle
ZLE function names specially.

(Btw, it's easier to think of the thingy stack as a queue, but with
thingies being added at the head of the queue rather than the tail.)
There is a fundamental ZLE operation to pop a thingy off the stack and
execute it as a ZLE function.  This would be the normal way to execute
ZLE functions:

zle up-line-or-history vi-first-non-blank

would push the specified thingies onto the stack in reverse order,
and then execute until they had all been popped off.  An option would
be required to push a thingy without executing:

zle -p "this is also a thingy!"

Another option to zle would execute the top thingy from the stack:

zle -p pound-insert
zle -x

Arguments to ZLE functions are passed as thingies on the stack:

zle vi-replace-chars "%"

vi-replace-chars internally reads the next thingy on the stack and
pops it off, so this command line would execute only one command,
after which the "%" would no longer be on the stack to be executed.
User-written ZLE functions could do these things explicitly:

function replace-buffer {
  ZLE_BUFFER=$zle_stack[1]
  zle -s
  ZLE_CURSORPOS=0
}

(The stack should probably be a read-only special parameter, but arbitrary
modification *could* be permitted in the interests of permissivity.
In any case, modification via the zle builtin would be more efficient.)

A few new ZLE functions are required to do input.  For example,
vi-replace-chars internally calls vigetkey(), which in this scheme would
become a ZLE function vi-get-key.  This function would read a key, in the
vi way, and push it onto the stack as a thingy.  To allow functions to
either read arguments that way or use arguments that were pre-existing
on the stack, vi-get-key and so on would only actually perform input if
the stack is currently empty.  Thus a limited version of vi-replace-chars
could be written thus:

function my-replace-char {
  zle vi-get-key
  if [[ -z "$zle_stack[1]" ]]; then
    zle undefined-key # feep
  else
    ZLE_BUFFER2=$zle_stack[1]${ZLE_BUFFER2#?}
  fi
  zle -s
}

Another important input function would be get-key-cmd (getkeycmd()), which
reads a key sequence and pushes the corresponding function name from the
current keymap.  The main loop of ZLE would be equivalent to the commands

while (( ! $ZLE_ACCEPTED )); do
  zle get-key-cmd
  zle -x
done

($ZLE_ACCEPTED should be another (modifiable) special parameter.
The current repeat count and vi buffer selection should be similarly
available.)  There is a need to be able to perform actual input even
when the stack is non-empty.  Probably the neatest way to do this is to
have a special thingy that (when at the top of the stack) will cause
the input functions to perform input anyway.  However, making this
thingy distinct from all other possible thingies could be difficult.
Ignoring that difficulty, I'll spell this thingy as "@":

function my-execute-command {
  zle -p @
  zle get-key-cmd
  zle -x
  zle -s
}

Another possible way to handle this would be to have the special thingy
`hide' any thingies below it, not just for the input functions.  It would
be pushed by a special option to zle, and popped by another option
(possibly the same one that executes the top thingy).  The main loop
would pop any such thingies left on the stack by unruly user-written
functions, possibly issuing a warning.

The "send string" functionality can be handled in one of two ways.
The first possibility is to treat it as it is now, as a special case
in getkeycmd().  The second possibility, that I generally prefer, is to
make it into a normal ZLE function, taking an argument, and allow keys
to be mapped to any sequence of thingies.

Comments are welcome.  I have three or four patches planned on which this
system would depend, so it'll be a little while before I get around to
implementing it.

-zefram

Re: a plan for ZLE extendability

[Peter Stephenson]

Zefram wrote:
> There is a need for ZLE to be extendable without having to write a
> binary module.
> 
> To this end, I have devised the following system.

this sounds fine... someone ought to make a few comments...

> The third requirement (modification of buffers) can be met most generally
> by adding a special parameter -- perhaps $ZLE_BUFFER -- that can be
> assigned to.  $ZLE_MINIBUFFER would also be required.  To control
> the cursor position, $ZLE_BUFFER could be split into two parameters,
> representing the parts of the buffer before and after the cursor.

You can avoid $ZLE_BUFFER by doing like what completion does and
passing the buffer as arguments $1 and $2 of the function.  I think
the drawback is you need some kind of arrangement with the function
call mechanism to get the returned values of $1 and $2; maybe nested
functions are less intuitive and less efficient this way, too.

> There is a need to be able to perform actual input even
> when the stack is non-empty.  Probably the neatest way to do this is to
> have a special thingy that (when at the top of the stack) will cause
> the input functions to perform input anyway.

What's the objection to e.g. having a special option to zle that
causes the arguments to be executed immediately?  Or one that causes
just the top N thingies on the stack to be executed (or the possibility
of both)?  That would avoid having special thingies.

> The "send string" functionality can be handled in one of two ways.
> The first possibility is to treat it as it is now, as a special case
> in getkeycmd().  The second possibility, that I generally prefer, is to
> make it into a normal ZLE function, taking an argument, and allow keys
> to be mapped to any sequence of thingies.

One thing that's missing at the moment is being able to bind to
sequences of functions, instead of just a rather opaque string of
characters which may or may not be bound to the desired functions, so
the second alternative is definitely desirable.  I think in any case
bindkey -s can easily be rescued to store an appropriate sequence.

I didn't quite get whether thingies are typed: are (1) thingies
interpreted as functions when first popped and only as strings when
required as an argument, or (2) either pushed on the stack as
functions or as strings?  If (2) then a string encountered as a
function can simply be sent without needing any specific send-string
function.

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutches Electronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: a plan for ZLE extendability

[Zefram]

>this sounds fine... someone ought to make a few comments...

That's what I thought.  I'm surprised no one has responded up to now.

>You can avoid $ZLE_BUFFER by doing like what completion does and
>passing the buffer as arguments $1 and $2 of the function.  I think
>the drawback is you need some kind of arrangement with the function
>call mechanism to get the returned values of $1 and $2; maybe nested
>functions are less intuitive and less efficient this way, too.

I considered using the positional parameters -- actually I would have
used them for the thingy stack -- but I decided that the hiding of
these parameters when another function is called would make it
inappropriate.  I don't like this aspect of completion functions,
partly for this reason.  It is also aesthetically displeasing, and
illogical, to make changes to the buffer by assigning to the function's
arguments.

>What's the objection to e.g. having a special option to zle that
>causes the arguments to be executed immediately?  Or one that causes
>just the top N thingies on the stack to be executed (or the possibility
>of both)?  That would avoid having special thingies.

The effect of such an option would be equivalent to pushing one of the
special thingies, and then the thingies specified.  There would in
practice still be a need for a special thingy.  (There's no overhead in
that; there are other things that require thingies to have flags too.)
The syntax I am currently leaning towards would have

zle -h

to push a special thingy, {h}iding the rest of the stack; normal
thingies could also be specified, so

zle -h get-key

would push a special thingy and then push and execute get-key.  The
result would be the key read as the only visible thingy on the stack;
after that had been popped, the special thingy could be popped.  Is
this close enough to what you had in mind?

>                                                 I think in any case
>bindkey -s can easily be rescued to store an appropriate sequence.

Of course, the existing semantics of bindkey -s would be emulated.

>I didn't quite get whether thingies are typed: are (1) thingies
>interpreted as functions when first popped and only as strings when
>required as an argument, or (2) either pushed on the stack as
>functions or as strings?  If (2) then a string encountered as a
>function can simply be sent without needing any specific send-string
>function.

(1).  Each thingy (apart from the special hiding thingy) is just a
string, as far as the user is concerned.  (Internally the thingy also
has a couple of flags.)  When a function execution is requested, the
top thingy on the stack is popped and looked up as a function name, and
that function is then executed.  In any case, automatically pushing
executed strings back onto the input would be inappropriate.  Tty input
is not fundamental to this model of ZLE; it's just something handled by
the get-key function (with which send-string collaborates).  A user
could even redefine get-key and send-string, to take input from
somewhere else.

-zefram

Re: a plan for ZLE extendability

[Peter Stephenson]

Zefram wrote:
> The syntax I am currently leaning towards would have
> 
> zle -h
> 
> to push a special thingy, {h}iding the rest of the stack; normal
> thingies could also be specified, so
> 
> zle -h get-key
> 
> would push a special thingy and then push and execute get-key.  The
> result would be the key read as the only visible thingy on the stack;
> after that had been popped, the special thingy could be popped.  Is
> this close enough to what you had in mind?

Yup, from the user point of view that's pretty much how I was
thinking.

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutches Electronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.