Using zle outside zsh

Using zle outside zsh

[DervishD]

    Hi all :)

    First of all, I've changed my email address, although I can still
be contacted on the old address, just in case someone in the list
has it.

    That said, let's go to the matter ;)

    No, I'm not talking about getting zle out of zsh and reusing it
as a library. I asked that in the past and is very difficult, so I've
thought about other solution.

    The scenario: we have a nice program we use a lot, and it's very
well coded, indeed. It is interactive, and takes its input from stdin
and outputs to stdout/stderr, but it lacks command line editing. It
uses the old stdio to read the user input, and we would like to have
command line editing, a fancy prompt, some kind of completion, a
couple of keyboard shorcuts or macros, a way of configuring
keybindings, etc...

    A solution is to hack the program and write a nice command line
interpreter for it, or use libedit, libreadline or try to get the
functionality of zle and put in the program. But this requires time,
coding, etc. and maybe forking the project (assuming it is free
software, of course).

    What I'm thinking is: will it be difficult to use zsh as the
command line interpreter? I mean, you run the program as:

    program < fifoin > fifoout 2> fifoerr

    and you configure zsh so it takes the command line (zle, full
edition, etc.) and instead of running it, writes it to 'fifoin',
while reads 'fifoout' and 'fifoerr' and dumps them on the screen.

    I haven't though about how implementing all that, it's just an
idea that popped out a minute ago, when I needed to use a poor-man's
ftp client without command line editing because 'zftp' doesn't have a
'STAT' command (yes, I know, it has 'quote', I discovered it too
late)...

    Thanks for your attention :) If anyone has ideas about a
solution, they're welcome. Of course, I'm don't want anybody to
implement this, I'll do that, I just want to know ideas, drawbacks,
etc.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[DervishD]

    Hi all :)

    My excuses for replying myself.

 * DervishD <disposable1@telefonica.net> dixit:
>     program < fifoin > fifoout 2> fifoerr
> 
>     and you configure zsh so it takes the command line (zle, full
> edition, etc.) and instead of running it, writes it to 'fifoin',
> while reads 'fifoout' and 'fifoerr' and dumps them on the screen.

    If just 'preexec' could access BUFFER, writing the edited command
line could be written anywhere, but since BUFFER is only accessible
inside completion widgets, this should be done with a widget,
replacing 'accept-line' or something like that. Am I completely lost?

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[Bart Schaefer]

On Mon, 23 Aug 2004, DervishD wrote:

>     The scenario: we have a nice program we use a lot, and it's very
> well coded, indeed. It is interactive, and takes its input from stdin
> and outputs to stdout/stderr, but it lacks command line editing.
> 
>     What I'm thinking is: will it be difficult to use zsh as the
> command line interpreter?

This is actually relatively easy to do.  For a working example, see
Functions/Misc/nslookup in the zsh distribution.

Re: Using zle outside zsh

[DervishD]

    Hi Bart :)
 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> >     The scenario: we have a nice program we use a lot, and it's very
> > well coded, indeed. It is interactive, and takes its input from stdin
> > and outputs to stdout/stderr, but it lacks command line editing.
> >     What I'm thinking is: will it be difficult to use zsh as the
> > command line interpreter?
> This is actually relatively easy to do.  For a working example, see
> Functions/Misc/nslookup in the zsh distribution.

    That's one of the many ideas I had, using pty, but I forgot about
'vared', which does exactly what I want... Thanks a lot for your
answer, Bart :) The number of beers (sorry, soft drinks) I owe you is
far beyond the cost of my mortgage ;)) I'm very grateful with you as
always, Bart, I really mean it.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> >     What I'm thinking is: will it be difficult to use zsh as the
> > command line interpreter?
> This is actually relatively easy to do.  For a working example, see
> Functions/Misc/nslookup in the zsh distribution.

    The problem here is that if the command prints its own prompt, a
code like this:

#!/bin/zsh

emulate -L zsh

zpty telnet telnet

zpty -r telnet response 'telnet>'
print -nr $response

while line=''; vared -e line
do
    [[ "$line" == "quit" ]] && break

    zpty -w telnet "$line"
    zpty -r telnet response 'telnet>'
    print -nr $response
done

zpty -w telnet "quit"
zpty -d telnet


    doesn't work, because the prompt doesn't have a carriage return.
The first 'print -nr' doesn't print anything because the buffers
aren't flushed until a '\n' is printed (in line buffer mode, I mean).

    But the worst problem is in the inner loop. I want zpty to read
all that 'telnet' spits until it finds 'telnet>', but blocks forever.
And if I change the code to just "zpty -r telnet" (that is, no
parameter and no pattern), it spits about 700 bytes of data and
stops, waiting for a buffer to fill (which never does).

    If the command run by zpty doesn't ends its chat with end of
line, is almost impossible to use zsh as a 'frontend' (so to say...).
Am I doing anything wrong?

    Obviously I can change the code above to make it read by lines:

#!/bin/zsh

emulate -L zsh

zpty telnet telnet

zpty -r telnet response 'telnet>'
print -nr $response

while line=''; vared -e line
do
    [[ "$line" == "quit" ]] && break

    zpty -w telnet "$line"
    while zpty -r telnet response
    do
        print -nr $response
    done
done

zpty -w telnet "quit"
zpty -d telnet

    This works, but never prints the command prompt, since it doesn't
have "\n", and moreover the inner loop blocks forever. Making the
zpty non-blocking leads to a race condition (solved by adding some
delay between the writing and the reading in the inner loop) :(( I
think that this is more difficult than I thought when the command
doesn't ends with a carriage return, which is the common case since I
want to use zle as a frontend for commands that prints their own
prompt... Any suggestion?

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[Bart Schaefer]

On Mon, 23 Aug 2004, DervishD wrote:

>     The problem here is that if the command prints its own prompt, a
> code like this:
[...]
> zpty -r telnet response 'telnet>'
> print -nr $response
[...]
>     doesn't work, because the prompt doesn't have a carriage return.

That's not why it doesn't work.  Read the doc for "zpty -r" again:

     If a PATTERN is given as well, output is read until the whole
     string read matches the PATTERN, even in the non-blocking case.

Note "the whole string".  This is why nslookup always has a "*" as the
first thing in the pattern when it calls "zpty -r".  So ...

  zpty -r telnet response $'(|*\n)telnet>'

... is probably more along the lines of what you want.

>     But the worst problem is in the inner loop. I want zpty to read
> all that 'telnet' spits until it finds 'telnet>', but blocks forever.

Same problem, I think.

Re: Using zle outside zsh

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> >     The problem here is that if the command prints its own prompt, a
> > code like this:
> [...]
> > zpty -r telnet response 'telnet>'
> > print -nr $response
> [...]
> >     doesn't work, because the prompt doesn't have a carriage return.
> That's not why it doesn't work.  Read the doc for "zpty -r" again:
[...]
> Note "the whole string".

    Yes, my fault, but that doesn't solve the problem :( With that
change the first read works but the second doesn't. I've tested with
my system's telnet and ftp. The code is now:

#!/bin/zsh

emulate -L zsh

zpty -b ftp ftp

# This worked as before since the line is exactly 'ftp> '.
# Same for telnet
zpty -r ftp response '(|*\n)ftp> '
# This doesn't print anything until a '\n' is printed :((
print -nr $response

while line=''; vared -e line
do
    [[ "$line" == "quit" ]] && break

    zpty -w ftp "$line"

    #This blocks forever.
    zpty -r ftp response '(|*\n)ftp> '
    print -nr $response
done

zpty -w ftp "quit"
zpty -d ftp


    If I change the pattern in the read that blocks forever for
something like '*progress', it reads until 'progress is found', no
problem! but when I extend it to read the prompt, it blocks forever.

    My other problem is still that print -n doesn't print anything
until a '\n' is output. Can I solve this using some option or should
I mess with 'stty'?

    BTW, the problem is the same using coprocesses and read -p...

    Thanks again, Bart.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[Bart Schaefer]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1276 bytes --]


On Tue, 24 Aug 2004, DervishD wrote:

>     Yes, my fault, but that doesn't solve the problem :( With that
> change the first read works but the second doesn't.

You didn't read/copy precisely enough, Raúl.

> # This worked as before since the line is exactly 'ftp> '.
> # Same for telnet
> zpty -r ftp response '(|*\n)ftp> '

zpty -r ftp response $'(|*\n)ftp> '

Note the dollar-sign.  Without that the \n is not converted to a newline.

Also, you do NOT want "zpty -b".  It makes the PTY non-blocking in *both*
directions, which confuses ftp into insanity.  It may even have exited by
the time you first try to write to it.

As for other reads hanging forever ... well, "ftp> " isn't the only prompt
that might be printed.  It may be asking for a username or a password, for
example, and "password: " won't match the pattern.  Try something like

zpty -r ftp response $'(|*\n)(ftp> |*[[:punct:]]: |Password:)'

Finally, vared is probably printing a carriage return and thereby erasing 
the last line of the "print -nr" output, which might fool you into 
thinking the read is hung.  So pick the last line out of $response and
pass that to vared as the prompt, like so:

pmpt=${response##$'*\n'}

while line=''; vared -e -p $pmpt line
do
  ...
  pmpt=${response##$'*\n'}
done

Re: Using zle outside zsh

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> > # This worked as before since the line is exactly 'ftp> '.
> > # Same for telnet
> > zpty -r ftp response '(|*\n)ftp> '
> zpty -r ftp response $'(|*\n)ftp> '
> Note the dollar-sign.  Without that the \n is not converted to a newline.

    I see... I assumed the single quote was enough because it was
enough in a 'print' statement. Obviously, I did not remember that $''
does exactly that: interpret the string as if it were passed to the
'print' builtin...
 
> Also, you do NOT want "zpty -b".  It makes the PTY non-blocking in *both*
> directions, which confuses ftp into insanity.  It may even have exited by
> the time you first try to write to it.

    I was testing first with telnet and used '-b' for a test. And I
forgot to remove it O:) Yes, ftp gets confused (and telnet too) if
'-b' is used.

> As for other reads hanging forever ... well, "ftp> " isn't the only prompt
> that might be printed.  It may be asking for a username or a password, for
> example, and "password: " won't match the pattern.  Try something like
> 
> zpty -r ftp response $'(|*\n)(ftp> |*[[:punct:]]: |Password:)'

    Oh, don't worry, I'm just testing the read-write loop. After that
I'll check which other prompts I must take care of. For example, if
you issue 'open' in the ftp main prompt, ftp asks you the address
using '(to)' (at least the client I'm using).
 
> Finally, vared is probably printing a carriage return and thereby erasing 
> the last line of the "print -nr" output, which might fool you into 
> thinking the read is hung.

    Yes, that's exactly what is happening. If I insert a delay
between the first read and the vared call, I can see the output from
the "print -nr" command.

>  So pick the last line out of $response and
> pass that to vared as the prompt, like so:
> pmpt=${response##$'*\n'}

    I must use "${response##*$'\n'}" or it doesn't work because the
'*' is quoted, but otherwise it works perfectly. Is there a way of
making 'vared' not to print a carriage return?

    OTOH, I think that, for a program like 'ftp' that not only prints
its own set of prompts but uses some server responses as prompts, it
would be better to read line by line until no more lines are present,
then use the last line as the prompt, but that involves using a while
loop and using 'zpty -rt', and that leads to a minor race, when we
write in the pty and wait for a response. Obviously the answer
doesn't need to be instantaneous, and that makes the first 'zpty -rt'
to fail :( Using the delay doesn't solve the problem, either: the
last line (that one containing the prompt) is not returned.

    Seems that the only possible solutions is to know in advance all
possible prompts and use 'zpty -r COMMAND NAME PATTERN', or using
zselect, or even using a proxy program that translates and normalizes
all received prompts including a '\n', or something like that...

    Thanks, Bart, for you answer :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Using zle outside zsh

[Bart Schaefer]

On Tue, 24 Aug 2004, DervishD wrote:

> >  So pick the last line out of $response and
> > pass that to vared as the prompt, like so:
> > pmpt=${response##$'*\n'}
> 
>     I must use "${response##*$'\n'}" or it doesn't work because the
> '*' is quoted

Oh, yes.  Typo on my part.  Sorry.

> but otherwise it works perfectly. Is there a way of
> making 'vared' not to print a carriage return?

Not really.  'setopt nopromptcr' works initially, but the first time ZLE 
decides to redraw the whole line or minimize the number of characters used 
for a backwards motion, it emits a \r and then things get pretty garbled.  
(Maybe Wayne will see this and fix something.)

>     OTOH, I think that, for a program like 'ftp' that not only prints
> its own set of prompts but uses some server responses as prompts, it
> would be better to read line by line until no more lines are present

Unfortunately there's no way to know *when* "no more lines are present". 
It's a TTY, after all; you don't get an EOF.

>     Seems that the only possible solutions is to know in advance all
> possible prompts and use 'zpty -r COMMAND NAME PATTERN', or using
> zselect, or even using a proxy program that translates and normalizes
> all received prompts including a '\n', or something like that...

zselect might help, because you could look for input from the keyboard
and from the pty at the same time -- but then you can't use vared.

Or you could try something with "zle -F" but I don't know if that works
inside vared (I'd presume it does, but I've never tried).

Re: Using zle outside zsh

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> > >  So pick the last line out of $response and
> > > pass that to vared as the prompt, like so:
> > > pmpt=${response##$'*\n'}
> >     I must use "${response##*$'\n'}" or it doesn't work because the
> > '*' is quoted
> Oh, yes.  Typo on my part.  Sorry.

    Don't worry. My mistakes weren't typos O:)
 
> >     OTOH, I think that, for a program like 'ftp' that not only prints
> > its own set of prompts but uses some server responses as prompts, it
> > would be better to read line by line until no more lines are present
> Unfortunately there's no way to know *when* "no more lines are present". 
> It's a TTY, after all; you don't get an EOF.

    Yes, but I can use 'zpty -rt' to test if more output must be
copied. The problem with this method is that it won't read the prompt
line since it doesn't have a '\n' at the end, it's not a line!. If
the prompt was always the same, it wouldn't be a problem, but...

    It works as it is now, and the only problem is to get all the
prompts. It is a work far less complex than to write a line editor
for the ftp client ;)
 
> >     Seems that the only possible solutions is to know in advance all
> > possible prompts and use 'zpty -r COMMAND NAME PATTERN', or using
> > zselect, or even using a proxy program that translates and normalizes
> > all received prompts including a '\n', or something like that...
> zselect might help, because you could look for input from the keyboard
> and from the pty at the same time -- but then you can't use vared.

    I prefer to use vared by now, although I can change all
keybindings for this particular use inside the script and get rid of
vared. My version of zsh doesn't even have 'zselect', so...
 
> Or you could try something with "zle -F" but I don't know if that works
> inside vared (I'd presume it does, but I've never tried).

    My zsh version (4.0.9) doesn't seem to have '-F' as an option to
zle. I'll take a look in the 4.2.1 I haven't installed yet.

    Well, thanks a lot for all your help, Bart :) Now I have an
alternative to zftp with full command line editing... as soon as I do
a bit of work. Not much, fortunately :) Thanks

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/