zsh-users
 help / color / mirror / code / Atom feed
* Capturing STDOUT without subshells or file I/O
@ 2018-09-02 20:57 Ben Klein
  2018-09-03  0:40 ` Joey Pabalinas
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Ben Klein @ 2018-09-02 20:57 UTC (permalink / raw)
  To: zsh-users


[-- Attachment #1.1.1: Type: text/plain, Size: 1404 bytes --]

Hello Zshellers,

I have an interesting predicament, I with to capture the STDOUT (builtin
printf) of a ZSH function to a variable, in which the function should be
running in the current shell context.

Currently, I have this:

  p10k_render_prompt_from_spec p10k_left p10k_opts > $tmpd/prompt
  read -d $'\0' _P10K_RENDERED_OUTPUT_PROMPT < $tmpd/prompt
  p10k_render_prompt_from_spec p10k_right p10k_opts right > $tmpd/prompt
  read -d $'\0' _P10K_RENDERED_OUTPUT_RPROMPT < $tmpd/prompt

And that involves creating a temporary file (/tmp) which might be
mounted on a spinning drive, so performance might take a hit.

The other route using a pipe:

p10k_render_prompt_from_spec p10k_left p10k_opts | read -d $'\0'
_P10K_RENDERED_OUTPUT_PROMPT

Causes environment variables and changes made during the `render_prompt`
call to be ignored. (Ends up in a subshell...?)

Is there some way I can use a virtual FD, or perhaps Zsh provides some
kind of buffer I could use instead?

In the end, I'm looking for a way to connect STDOUT to STDIN between ZSH
functions/builtins without any subshells or file I/O. (It's fine if it
gets buffered until close-of-stream.)

-- 
*\Ben Klein*
Founder and Owner of Robosane, robobenklein@robosane.net
You can find me elsewhere online as 'robobenklein'.
If you need to contact me securely, I am also reachable via GPG, or on
Keybase.

[-- Attachment #1.1.2: Type: text/html, Size: 2005 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-02 20:57 Capturing STDOUT without subshells or file I/O Ben Klein
@ 2018-09-03  0:40 ` Joey Pabalinas
  2018-09-03 14:02 ` Daniel Shahaf
  2018-09-18  5:12 ` Sebastian Gniazdowski
  2 siblings, 0 replies; 12+ messages in thread
From: Joey Pabalinas @ 2018-09-03  0:40 UTC (permalink / raw)
  To: Ben Klein; +Cc: zsh-users, Joey Pabalinas

[-- Attachment #1: Type: text/plain, Size: 3680 bytes --]

On Sun, Sep 02, 2018 at 04:57:49PM -0400, Ben Klein wrote:
> Currently, I have this:
> 
>   p10k_render_prompt_from_spec p10k_left p10k_opts > $tmpd/prompt
>   read -d $'\0' _P10K_RENDERED_OUTPUT_PROMPT < $tmpd/prompt
>   p10k_render_prompt_from_spec p10k_right p10k_opts right > $tmpd/prompt
>   read -d $'\0' _P10K_RENDERED_OUTPUT_RPROMPT < $tmpd/prompt
> 
> And that involves creating a temporary file (/tmp) which might be
> mounted on a spinning drive, so performance might take a hit.
> 
> The other route using a pipe:
> 
> p10k_render_prompt_from_spec p10k_left p10k_opts | read -d $'\0'
> _P10K_RENDERED_OUTPUT_PROMPT
> 
> Causes environment variables and changes made during the `render_prompt`
> call to be ignored. (Ends up in a subshell...?)
> 
> Is there some way I can use a virtual FD, or perhaps Zsh provides some
> kind of buffer I could use instead?
> 
> In the end, I'm looking for a way to connect STDOUT to STDIN between ZSH
> functions/builtins without any subshells or file I/O. (It's fine if it
> gets buffered until close-of-stream.)

You can sort of use the fact that if you delete the filesystem
name of a temporary file (which would make the hard link count 0),
as long as there still exists open file descriptors referring to
it, you are still able to read/write to it.

Let's make an example function that changes $FOO from 0 to 42, then
prints some output, sort of like your function:

> % FOO=0
> % fun() { FOO=42; print -r - "output of $funcstack[1]()"; }
> % print -r - "FOO=$FOO"
> FOO=0

Now we will use mktemp(1) to create a file in a tmpfs (usually /tmp)
directory, which a swap-backed ramdisk, then redirects an arbitrary
read/write fd to it and deletes the file:

> % tmp=$(mktemp)
> % exec {FD}<>$tmp
> % rm -fv -- $tmp
> removed '/tmp/tmp.sS1G9F6OjR'

The redirected file descriptor is still open however (if you have
colors in your ls you will see it in red because the file no longer
exists), and you can write to it by redirecting stdout:

> % ls -li /proc/$$/fd
> 29987340 lrwx------ 1 jp jp 64 Sep  2 14:36 0 -> /dev/pts/2
> 29986634 lrwx------ 1 jp jp 64 Sep  2 14:36 1 -> /dev/pts/2
> 29986638 lrwx------ 1 jp jp 64 Sep  2 14:36 12 -> /tmp/tmp.dneCPiOqS5
> 29986635 lrwx------ 1 jp jp 64 Sep  2 14:36 2 -> /dev/pts/2
> % fun >&$FD

Reading back your output, however, is a little weirder. The r/w
file descriptor file position is set past your output; reading
from it normally would give us back our output. You also can't
reopen it (file is gone), nor is there a mechanism in zsh to
call something like lseek(2) to rewind file descriptors.

However, what we *can* do it (ab)use the fact that subshells
will rewind the file descriptors since they are duplicated
during fork(). So we can make a subshell, redirect our fd to
stdin and read back what we have written! Like so:

> hobbes% output=$((</dev/stdin)<&$FD)
> hobbes% print -lr - "FOO=$FOO" "output=\"$output\""
> FOO=42
> output="output of fun()"

And of course, it's always good practice to close your
file descriptors in the main shell instance (subshell
file descriptors are implicitly closed when the subshell
exits) when you are finished with them:

> hobbes% exec {FD}>&-

Of course, realize this is kind of a hacky solution, and the better
way is to just have your function either not play with the current
environment (does something like a prompt generation script really
need to change the environment?) or have it set `PROMPT` and `RPROMPT`
itself and avoid worrying stdout at all.

But hey, we all need to have a little fun sometimes :)

-- 
Cheers,
Joey Pabalinas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-02 20:57 Capturing STDOUT without subshells or file I/O Ben Klein
  2018-09-03  0:40 ` Joey Pabalinas
@ 2018-09-03 14:02 ` Daniel Shahaf
  2018-09-03 18:43   ` Joey Pabalinas
  2018-09-18  5:12 ` Sebastian Gniazdowski
  2 siblings, 1 reply; 12+ messages in thread
From: Daniel Shahaf @ 2018-09-03 14:02 UTC (permalink / raw)
  To: Ben Klein; +Cc: zsh-users

Ben Klein wrote on Sun, Sep 02, 2018 at 16:57:49 -0400:
> I have an interesting predicament, I with to capture the STDOUT (builtin
> printf) of a ZSH function to a variable, in which the function should be
> running in the current shell context.

Just pass -v to the builtin printf, e.g.,

% printf -v foo 'a%03d' 42
% typeset -p foo
a042
% 


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-03 14:02 ` Daniel Shahaf
@ 2018-09-03 18:43   ` Joey Pabalinas
  2018-09-17 20:13     ` Ben Klein
  0 siblings, 1 reply; 12+ messages in thread
From: Joey Pabalinas @ 2018-09-03 18:43 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: Ben Klein, zsh-users, Joey Pabalinas

[-- Attachment #1: Type: text/plain, Size: 291 bytes --]

On Mon, Sep 03, 2018 at 02:02:14PM +0000, Daniel Shahaf wrote:
> Just pass -v to the builtin printf, e.g.,
> 
> % printf -v foo 'a%03d' 42
> % typeset -p foo
> a042
> % 

Much nicer, wow. Looks like I'll be revising quite a few of my scripts
today.

-- 
Cheers,
Joey Pabalinas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-03 18:43   ` Joey Pabalinas
@ 2018-09-17 20:13     ` Ben Klein
  2018-09-17 20:46       ` Daniel Shahaf
  2018-09-17 21:19       ` dana
  0 siblings, 2 replies; 12+ messages in thread
From: Ben Klein @ 2018-09-17 20:13 UTC (permalink / raw)
  To: zsh-users; +Cc: Daniel Shahaf, Joey Pabalinas


[-- Attachment #1.1.1: Type: text/plain, Size: 1185 bytes --]

So I just rewrote a large portion of a codebase on being able to use
`printf -v`, but it looks like this support was added after ZSH 5.1,
which I intend to support. (Ubuntu 16.04 still has it.)

So I guess the question is up again, how should I capture the `printf`
output without the `-v` option, no subshells, and no file I/O? Is there
a different method for ZSH v5.1?

I would like to do `printf '%.2f' "3.4" | read var` but it appears that
the command before the pipe causes a subshell to be opened.

Or potentially, is there some way I can make a wrapper that will use
`printf -v` when available, but falls back to another method?


On 09/03/2018 02:43 PM, Joey Pabalinas wrote:
> On Mon, Sep 03, 2018 at 02:02:14PM +0000, Daniel Shahaf wrote:
>> Just pass -v to the builtin printf, e.g.,
>>
>> % printf -v foo 'a%03d' 42
>> % typeset -p foo
>> a042
>> % 
> Much nicer, wow. Looks like I'll be revising quite a few of my scripts
> today.
>

-- 
*\Ben Klein*
Founder and Owner of Robosane, robobenklein@robosane.net
You can find me elsewhere online as 'robobenklein'.
If you need to contact me securely, I am also reachable via GPG, or on
Keybase.

[-- Attachment #1.1.2: Type: text/html, Size: 1991 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-17 20:13     ` Ben Klein
@ 2018-09-17 20:46       ` Daniel Shahaf
  2018-09-17 21:01         ` Mikael Magnusson
  2018-09-17 21:19       ` dana
  1 sibling, 1 reply; 12+ messages in thread
From: Daniel Shahaf @ 2018-09-17 20:46 UTC (permalink / raw)
  To: Ben Klein, zsh-users

Ben Klein wrote on Mon, 17 Sep 2018 16:13 -0400:
> So I guess the question is up again, how should I capture the `printf`
> output without the `-v` option, no subshells, and no file I/O? Is there
> a different method for ZSH v5.1?
> 
> I would like to do `printf '%.2f' "3.4" | read var` but it appears that
> the command before the pipe causes a subshell to be opened.
> 

If you could call pipe(2), you'd be able to do
.
    printf foo >&$w
    read bar <&$r
.
without forking (where $r,$w were returned by pipe(2)).  However, I
don't see any suitable callsite in the 5.6 codebase.  (Don't have 5.1
handy, sorry.)

Maybe we should expose pipe(2) in zsh/system?

> Or potentially, is there some way I can make a wrapper that will use
> `printf -v` when available, but falls back to another method?

Should be possible, yes.  You can tell the two cases apart by doing:

    if printf -v >/dev/null ; then

Cheers,

Daniel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-17 20:46       ` Daniel Shahaf
@ 2018-09-17 21:01         ` Mikael Magnusson
  2018-09-17 21:20           ` Bart Schaefer
  0 siblings, 1 reply; 12+ messages in thread
From: Mikael Magnusson @ 2018-09-17 21:01 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: Ben Klein, Zsh Users

On Mon, Sep 17, 2018 at 10:46 PM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Ben Klein wrote on Mon, 17 Sep 2018 16:13 -0400:
>> So I guess the question is up again, how should I capture the `printf`
>> output without the `-v` option, no subshells, and no file I/O? Is there
>> a different method for ZSH v5.1?
>>
>> I would like to do `printf '%.2f' "3.4" | read var` but it appears that
>> the command before the pipe causes a subshell to be opened.
>>
>
> If you could call pipe(2), you'd be able to do
> .
>     printf foo >&$w
>     read bar <&$r
> .
> without forking (where $r,$w were returned by pipe(2)).  However, I
> don't see any suitable callsite in the 5.6 codebase.  (Don't have 5.1
> handy, sorry.)
>
> Maybe we should expose pipe(2) in zsh/system?

That's just a deadlock-by-design.

-- 
Mikael Magnusson

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-17 20:13     ` Ben Klein
  2018-09-17 20:46       ` Daniel Shahaf
@ 2018-09-17 21:19       ` dana
  2018-09-17 21:33         ` Bart Schaefer
  1 sibling, 1 reply; 12+ messages in thread
From: dana @ 2018-09-17 21:19 UTC (permalink / raw)
  To: Ben Klein; +Cc: zsh-users, Daniel Shahaf, Joey Pabalinas

On 17 Sep 2018, at 15:13, Ben Klein <robobenklein@gmail.com> wrote:
>So I guess the question is up again, how should I capture the `printf` output
>without the `-v` option, no subshells, and no file I/O? Is there a different
>method for ZSH v5.1?

Mikael once showed me this trick:

  print -zf '%.2f' 3.4
  read -rz var
  echo $var # 3.40

Not sure if there are any situations where it would break.

dana


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-17 21:01         ` Mikael Magnusson
@ 2018-09-17 21:20           ` Bart Schaefer
  0 siblings, 0 replies; 12+ messages in thread
From: Bart Schaefer @ 2018-09-17 21:20 UTC (permalink / raw)
  To: Zsh Users

On Mon, Sep 17, 2018 at 2:01 PM, Mikael Magnusson <mikachu@gmail.com> wrote:
> On Mon, Sep 17, 2018 at 10:46 PM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>>
>> If you could call pipe(2), you'd be able to do
>> .
>>     printf foo >&$w
>>     read bar <&$r
>> .
>> without forking (where $r,$w were returned by pipe(2)).
>
> That's just a deadlock-by-design.

To expand upon that a bit, the pipe created by the system has a
limited (and generally indeterminate) amount of buffer space, so if
the print side ever sends more than that amount of data at once, it
will block forever, and the read side will never get a chance to
execute.  This is WHY (printf '%.2f' "3.4" | read var) creates a
subshell on the left, so that the read and write are simultaneous and
independent.

You could safely do it if you only wrote and read one byte at a time,
but you're not going to pull that off with printf and formatting.

If zsh exposed a way to make the writes non-blocking, you could do it
without deadlock but with possible data loss, but that's not going to
help you with zsh 5.1 and earlier.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-17 21:19       ` dana
@ 2018-09-17 21:33         ` Bart Schaefer
  0 siblings, 0 replies; 12+ messages in thread
From: Bart Schaefer @ 2018-09-17 21:33 UTC (permalink / raw)
  To: dana; +Cc: Ben Klein, Zsh Users, Daniel Shahaf, Joey Pabalinas

On Mon, Sep 17, 2018 at 2:19 PM, dana <dana@dana.is> wrote:
>
> Mikael once showed me this trick:
>
>   print -zf '%.2f' 3.4
>   read -rz var
>   echo $var # 3.40

That's pretty close to exactly what "printf -v" does underneath.  It
just requires that the buffer stack be present, so it means the zle
module has to have been loaded -- but that happens even in
non-interactive shells unless you do something to prevent it, so it
should work almost all the time.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-02 20:57 Capturing STDOUT without subshells or file I/O Ben Klein
  2018-09-03  0:40 ` Joey Pabalinas
  2018-09-03 14:02 ` Daniel Shahaf
@ 2018-09-18  5:12 ` Sebastian Gniazdowski
  2018-09-21  6:58   ` Oliver Kiddle
  2 siblings, 1 reply; 12+ messages in thread
From: Sebastian Gniazdowski @ 2018-09-18  5:12 UTC (permalink / raw)
  To: robobenklein; +Cc: Zsh Users

On Sun, 2 Sep 2018 at 22:59, Ben Klein <robobenklein@gmail.com> wrote:
>
> Hello Zshellers,
>
> I have an interesting predicament, I with to capture the STDOUT (builtin printf) of a ZSH function to a variable, in which the function should be running in the current shell context.

mksh can do this with:

${ foo;}

foo runs in current shell and the ${ ... } is substituted with its
output. One more cool thing mksh has - ${|foo;} which is replaced by
the value of REPLY after running foo in current shell. I think that
each time a serious, optimized coding occurs (like when creating an
advanced theme), those will be very handy.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Capturing STDOUT without subshells or file I/O
  2018-09-18  5:12 ` Sebastian Gniazdowski
@ 2018-09-21  6:58   ` Oliver Kiddle
  0 siblings, 0 replies; 12+ messages in thread
From: Oliver Kiddle @ 2018-09-21  6:58 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: robobenklein, Zsh Users

On 18 Sep, Sebastian Gniazdowski wrote:
> > I have an interesting predicament, I with to capture the STDOUT (builtin printf) of a ZSH function to a variable, in which the function should be running in the current shell context.
> mksh can do this with:
>
> ${ foo;}
>
> foo runs in current shell and the ${ ... } is substituted with its
> output.

I was under the impression that ksh93 does this with the basic $( ... )
syntax meaning that that syntax does not inherently imply the use of
a subshell. It depends on whether there are pipes, external commands or
only builtins there.

For ksh93, they had their own I/O abstraction – sfio – which was a
complete replacement for stdio but which made it easy to capture the
output instead of writing it. Using sprintf everywhere would be a
very invasive change but the open_memstream/temporary file approach
used for print -v would work. And in any case, there would be other,
bigger, problems to solve.

Personally, I can't claim to be have ever been especially concerned
about the cost of the fork and was not especially enthused by the
addition of -v to print.

$(...) substitutions are also run sequentially:

  echo $SECONDS ; echo $(sleep 1;echo $SECONDS) $(sleep 1;echo $SECONDS)

Perhaps I'm forgetting something obvious but at the moment, I can't
think why they couldn't be run in parallel - if performance is the
concern.

Oliver

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2018-09-21  6:59 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-02 20:57 Capturing STDOUT without subshells or file I/O Ben Klein
2018-09-03  0:40 ` Joey Pabalinas
2018-09-03 14:02 ` Daniel Shahaf
2018-09-03 18:43   ` Joey Pabalinas
2018-09-17 20:13     ` Ben Klein
2018-09-17 20:46       ` Daniel Shahaf
2018-09-17 21:01         ` Mikael Magnusson
2018-09-17 21:20           ` Bart Schaefer
2018-09-17 21:19       ` dana
2018-09-17 21:33         ` Bart Schaefer
2018-09-18  5:12 ` Sebastian Gniazdowski
2018-09-21  6:58   ` Oliver Kiddle

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).