* stdbuf -o0 -i0 via a Zsh native interface? @ 2023-05-25 11:21 Sebastian Gniazdowski 2023-05-25 14:16 ` Peter Stephenson 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-25 11:21 UTC (permalink / raw) To: Zsh hackers list Hi I'm using coproc to continuously read command's output, to have the effect of `CMD | fzf` – i.e. of instantly available contents while reading all of it eventually, However, I've had problem with buffering – no output from the CMD in `coproc CMD` was available for a long time, and I've found that `coproc { stdbuf -o0 -i0 CMD; }` helps. However, this is sub-optimal solution because I have to hideously prepend the stdbuf command before each user CMD, meaning that entering builtin commands will not work (e.g.: `stdbuf -i0 -o0 print smthg`). I wonder if the infamous buffering problem, solved by the hacky ld-preload stdbuf program could be fixed on the level of Zsh? Like, e.g.: special array, say: zsh_buffer=( 0 0 0 )? For stdin, stdout and stderr buffers. -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 11:21 stdbuf -o0 -i0 via a Zsh native interface? Sebastian Gniazdowski @ 2023-05-25 14:16 ` Peter Stephenson 2023-05-25 18:35 ` Bart Schaefer 0 siblings, 1 reply; 13+ messages in thread From: Peter Stephenson @ 2023-05-25 14:16 UTC (permalink / raw) To: Zsh hackers list > On 25/05/2023 12:21 Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > I'm using coproc to continuously read command's output, to have the > effect of `CMD | fzf` – i.e. of instantly available contents while > reading all of it eventually, However, I've had problem with buffering > – no output from the CMD in `coproc CMD` was available for a long > time, and I've found that `coproc { stdbuf -o0 -i0 CMD; }` helps. > > However, this is sub-optimal solution because I have to hideously > prepend the stdbuf command before each user CMD, meaning that entering > builtin commands will not work (e.g.: `stdbuf -i0 -o0 print smthg`). > > I wonder if the infamous buffering problem, solved by the hacky > ld-preload stdbuf program could be fixed on the level of Zsh? Like, > e.g.: special array, say: zsh_buffer=( 0 0 0 )? For stdin, stdout and > stderr buffers. Something like this? It looks like a good fit for zsystem. This is deliberately fairly restricted functionality; anything more would be starting to be a support headache, but up to this point it ought to be very much WYSIWYG. pws diff --git a/Doc/Zsh/mod_system.yo b/Doc/Zsh/mod_system.yo index e25201faa..73611540f 100644 --- a/Doc/Zsh/mod_system.yo +++ b/Doc/Zsh/mod_system.yo @@ -228,6 +228,13 @@ If the option tt(-r) is given, the lock is only for reading, otherwise it is for reading and writing. The file descriptor is opened accordingly. ) +item(tt(zsystem setbuffer) var(fd) tt(L)var(|)tt(N))( +Set the file descriptor var(fd), which may be 0, 1 or 2, to +either line buffering (tt(L)) or no buffering (tt(N)). It is +not possible to set an explicit buffer. Caution should be +exercised as the resulting mode may not be entirely compatible +with normal shell operation. +) item(tt(zsystem supports) var(subcommand))( The builtin tt(zsystem)'s subcommand tt(supports) tests whether a given subcommand is supported. It returns status 0 if so, else diff --git a/Src/Modules/system.c b/Src/Modules/system.c index 929a8b002..033c84d27 100644 --- a/Src/Modules/system.c +++ b/Src/Modules/system.c @@ -772,6 +772,53 @@ bin_zsystem_flock(char *nam, char **args, UNUSED(Options ops), UNUSED(int func)) } +/* + * Set up buffering on a given file descriptor. + */ +/**/ +static int +bin_zsystem_setbuffer(char *nam, char **args, + UNUSED(Options ops), UNUSED(int func)) +{ + int fd; + FILE *strm; + + if (!args[0] || !args[1]) { + zwarnnam(nam, "setbuffer: not enough arguemnts"); + return 1; + } + if (args[2]) { + zwarnnam(nam, "setbuffer: too many arguments"); + return 1; + } + + fd = getposint(args[0], nam); + if (fd < 0) + return 1; + if (fd > 2) { + zwarnnam(nam, "setbuffer: file descriptor 0, 1 or 2 expected"); + return 1; + } + strm = (fd == 0) ? stdin : (fd == 1) ? stdout : stderr; + + if (!strcmp(args[1], "L")) { + if (setvbuf(strm, NULL, _IOLBF, 0)) { + zwarnnam(nam, "setting line buffer failed: %e"); + return 1; + } + } else if (!strcmp(args[1], "N")) { + if (setvbuf(strm, NULL, _IONBF, 0)) { + zwarnnam(nam, "setting no buffer failed: %e"); + return 1; + } + } else { + zwarnanm(nam, "setbuffer: argument L or N expected"); + return 1; + } + + return 0; +} + /* * Return status zero if the zsystem feature is supported, else 1. * Operates silently for future-proofing. @@ -808,6 +855,8 @@ bin_zsystem(char *nam, char **args, Options ops, int func) /* If more commands are implemented, this can be more sophisticated */ if (!strcmp(*args, "flock")) { return bin_zsystem_flock(nam, args+1, ops, func); + } else if (!strcmp(*args, "setbuffer")) { + return bin_zsystem_setbuffer(nam, args+1, ops, func); } else if (!strcmp(*args, "supports")) { return bin_zsystem_supports(nam, args+1, ops, func); } ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 14:16 ` Peter Stephenson @ 2023-05-25 18:35 ` Bart Schaefer 2023-05-25 19:02 ` Sebastian Gniazdowski 2023-05-30 11:17 ` Peter Stephenson 0 siblings, 2 replies; 13+ messages in thread From: Bart Schaefer @ 2023-05-25 18:35 UTC (permalink / raw) To: Peter Stephenson; +Cc: Zsh hackers list On Thu, May 25, 2023 at 7:16 AM Peter Stephenson <p.w.stephenson@ntlworld.com> wrote: > > > On 25/05/2023 12:21 Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > However, this is sub-optimal solution because I have to hideously > > prepend the stdbuf command before each user CMD, meaning that entering > > builtin commands will not work (e.g.: `stdbuf -i0 -o0 print smthg`). > > > > I wonder if the infamous buffering problem, solved by the hacky > > ld-preload stdbuf program could be fixed on the level of Zsh? > > Something like this? It looks like a good fit for zsystem. That's a start, but it doesn't solve Sebastian's problem of having to hack the stdbuf prefix into the command string for external commands. We'd need something akin to the STTY parameter to specify a prefix for external commands. That might have its own security concerns. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 18:35 ` Bart Schaefer @ 2023-05-25 19:02 ` Sebastian Gniazdowski 2023-05-25 19:09 ` Bart Schaefer 2023-05-30 11:17 ` Peter Stephenson 1 sibling, 1 reply; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-25 19:02 UTC (permalink / raw) To: Bart Schaefer; +Cc: Peter Stephenson, Zsh hackers list On Thu, 25 May 2023 at 18:36, Bart Schaefer <schaefer@brasslantern.com> wrote: > > On Thu, May 25, 2023 at 7:16 AM Peter Stephenson > <p.w.stephenson@ntlworld.com> wrote: > > > > > On 25/05/2023 12:21 Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > > > Something like this? It looks like a good fit for zsystem. > > That's a start, but it doesn't solve Sebastian's problem of having to > hack the stdbuf prefix into the command string for external commands. I think that it does solve the problem, if the buffer settings are applied to external commands. I would say that they are, but your answer gives doubts… are they applied? Also, I wonder why the buffers can be set only to 0, 1 and 2 and not to every descriptor? This might be an effect of suggestion from my original post, but the mechanism of a builtin command obtaining file descriptor opens my eyes that buffers can be changed on every descriptor (or not?). > We'd need something akin to the STTY parameter to specify a prefix for > external commands. That might have its own security concerns. That's an interesting approach, however IMHO it would make the mechanism's interface script-targeted, and not foreground, interactive work (because of the need of prepend command each time with a var export). -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 19:02 ` Sebastian Gniazdowski @ 2023-05-25 19:09 ` Bart Schaefer 2023-05-25 19:28 ` Sebastian Gniazdowski 0 siblings, 1 reply; 13+ messages in thread From: Bart Schaefer @ 2023-05-25 19:09 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Peter Stephenson, Zsh hackers list On Thu, May 25, 2023 at 12:02 PM Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > I think that it does solve the problem, if the buffer settings are > applied to external commands. I would say that they are, but your > answer gives doubts… are they applied? Once you fork/exec, the initialization of stdio buffers etc. is all redone inside the new process, as far as I know. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 19:09 ` Bart Schaefer @ 2023-05-25 19:28 ` Sebastian Gniazdowski 2023-05-25 21:21 ` Bart Schaefer 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-25 19:28 UTC (permalink / raw) To: Bart Schaefer; +Cc: Peter Stephenson, Zsh hackers list Ahso, that's why stdbuf uses ld-preload-like tricks to enforce the buffer size… That makes me look more tolerant to your STTY-way approach, i.e.: I suspect that ld-tricks are out of reach and I would accept even STTY-like solution if only it would be implemented… On Thu, 25 May 2023 at 19:09, Bart Schaefer <schaefer@brasslantern.com> wrote: > > On Thu, May 25, 2023 at 12:02 PM Sebastian Gniazdowski > <sgniazdowski@gmail.com> wrote: > > > > I think that it does solve the problem, if the buffer settings are > > applied to external commands. I would say that they are, but your > > answer gives doubts… are they applied? > > Once you fork/exec, the initialization of stdio buffers etc. is all > redone inside the new process, as far as I know. -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 19:28 ` Sebastian Gniazdowski @ 2023-05-25 21:21 ` Bart Schaefer 2023-05-26 2:17 ` Sebastian Gniazdowski 0 siblings, 1 reply; 13+ messages in thread From: Bart Schaefer @ 2023-05-25 21:21 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Peter Stephenson, Zsh hackers list On Thu, May 25, 2023 at 12:28 PM Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > Ahso, that's why stdbuf uses ld-preload-like tricks to enforce the > buffer size… That makes me look more tolerant to your STTY-way > approach, i.e.: I suspect that ld-tricks are out of reach and I would > accept even STTY-like solution if only it would be implemented… Use with great caution, but to test the concept: TRAPDEBUG() { setopt err_exit; stdbuf -o0 -i0 ${(z)ZSH_DEBUG_CMD} } (assumes default setting of DEBUG_BEFORE_CMD option) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 21:21 ` Bart Schaefer @ 2023-05-26 2:17 ` Sebastian Gniazdowski 2023-05-26 10:23 ` Sebastian Gniazdowski 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-26 2:17 UTC (permalink / raw) To: Bart Schaefer; +Cc: Peter Stephenson, Zsh hackers list On Thu, 25 May 2023 at 21:21, Bart Schaefer <schaefer@brasslantern.com> wrote: > > Use with great caution, but to test the concept: > > TRAPDEBUG() { setopt err_exit; stdbuf -o0 -i0 ${(z)ZSH_DEBUG_CMD} } Interesting, I've never touched debug trap, this gives me a glimpse of how it is working. And as for the topic in general, I've had a moment of understanding that what's needed is basically: a `stdbuf` command that's *built-in*, and this comes down to your idea of `STTY=… my-cmd` -like inline-export variable, which is a nicer, more integrated version of a stdbuf-clone builtin (and could be made non-only-inline-export, to set once and forget). Now, who has time to look at stdbuf.c? :) To grasp what the LD-trick used there is? -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-26 2:17 ` Sebastian Gniazdowski @ 2023-05-26 10:23 ` Sebastian Gniazdowski 2023-05-26 9:30 ` Roman Perepelitsa 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-26 10:23 UTC (permalink / raw) To: Bart Schaefer; +Cc: Peter Stephenson, Zsh hackers list I've took a look, and it seems to be easy: 1. Export LD_PRELOAD=/usr/lib/zsh/5.7/libzstdbuf.so - https://github.com/coreutils/coreutils/blob/master/src/stdbuf.c#L196 - https://github.com/coreutils/coreutils/blob/master/src/stdbuf.c#L259 2. Provide stdbuf function in libzstdbuf.so: - https://github.com/coreutils/coreutils/blob/master/src/libstdbuf.c#L139 Seems fairly simple… On Fri, 26 May 2023 at 02:17, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > On Thu, 25 May 2023 at 21:21, Bart Schaefer <schaefer@brasslantern.com> wrote: > > > > Use with great caution, but to test the concept: > > > > TRAPDEBUG() { setopt err_exit; stdbuf -o0 -i0 ${(z)ZSH_DEBUG_CMD} } > > Interesting, I've never touched debug trap, this gives me a glimpse of > how it is working. > > And as for the topic in general, I've had a moment of understanding > that what's needed is basically: a `stdbuf` command that's *built-in*, > and this comes down to your idea of `STTY=… my-cmd` -like > inline-export variable, which is a nicer, more integrated version of a > stdbuf-clone builtin (and could be made non-only-inline-export, to set > once and forget). Now, who has time to look at stdbuf.c? :) To grasp > what the LD-trick used there is? > > -- > Best regards, > Sebastian Gniazdowski -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-26 10:23 ` Sebastian Gniazdowski @ 2023-05-26 9:30 ` Roman Perepelitsa 2023-05-26 20:52 ` Bart Schaefer 0 siblings, 1 reply; 13+ messages in thread From: Roman Perepelitsa @ 2023-05-26 9:30 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Bart Schaefer, Peter Stephenson, Zsh hackers list On Fri, May 26, 2023 at 11:24 AM Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > I've took a look, and it seems to be easy: > 1. Export LD_PRELOAD=/usr/lib/zsh/5.7/libzstdbuf.so > - https://github.com/coreutils/coreutils/blob/master/src/stdbuf.c#L196 > - https://github.com/coreutils/coreutils/blob/master/src/stdbuf.c#L259 > 2. Provide stdbuf function in libzstdbuf.so: > - https://github.com/coreutils/coreutils/blob/master/src/libstdbuf.c#L139 > > Seems fairly simple… This won't work reliably. E.g., it may not work on binaries that are linked against a different libc. A more reliable solution for your problem is something like `unbuffer` from `expect`. This tool creates a TTY and takes advantage of the convention that stdout to TTY is not buffered. Better yet, you can do this with zpty without external tools. Roman. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-26 9:30 ` Roman Perepelitsa @ 2023-05-26 20:52 ` Bart Schaefer 0 siblings, 0 replies; 13+ messages in thread From: Bart Schaefer @ 2023-05-26 20:52 UTC (permalink / raw) To: Zsh hackers list On Fri, May 26, 2023 at 2:30 AM Roman Perepelitsa <roman.perepelitsa@gmail.com> wrote: > > This won't work reliably. E.g., it may not work on binaries that are > linked against a different libc. Exactly, hence my suggestion of a mechanism for prefixing external commands with the name of a local "helper" that would work with the current platform's libraries/linker. There's already a fork happening so it doesn't seem worthwhile to try to build that functionality into the shell itself. > A more reliable solution for your > problem is something like `unbuffer` from `expect`. This tool creates > a TTY Indeed, using unbuffer as the prefix command could be a fallback for cases where there is no stdbuf equivalent. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-25 18:35 ` Bart Schaefer 2023-05-25 19:02 ` Sebastian Gniazdowski @ 2023-05-30 11:17 ` Peter Stephenson 2023-05-31 12:38 ` Sebastian Gniazdowski 1 sibling, 1 reply; 13+ messages in thread From: Peter Stephenson @ 2023-05-30 11:17 UTC (permalink / raw) To: Zsh hackers list > On 25/05/2023 19:35 Bart Schaefer <schaefer@brasslantern.com> wrote: > On Thu, May 25, 2023 at 7:16 AM Peter Stephenson > <p.w.stephenson@ntlworld.com> wrote: > > > > > On 25/05/2023 12:21 Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > > However, this is sub-optimal solution because I have to hideously > > > prepend the stdbuf command before each user CMD, meaning that entering > > > builtin commands will not work (e.g.: `stdbuf -i0 -o0 print smthg`). > > > > > > I wonder if the infamous buffering problem, solved by the hacky > > > ld-preload stdbuf program could be fixed on the level of Zsh? > > > > Something like this? It looks like a good fit for zsystem. > > That's a start, but it doesn't solve Sebastian's problem of having to > hack the stdbuf prefix into the command string for external commands. > > We'd need something akin to the STTY parameter to specify a prefix for > external commands. That might have its own security concerns. Yes, so in fact Sebastian's got the bit I originally added covered with external commands already; they don't need to be built into the shell, just executed appropriately. Can't we simply add a hook at the point where STTY is used? What this doesn't do is the equivalent sanity after the event --- in that case triggering TTY restoration if we detect a case where it might have been messed up. ut that should be less important here as the changes affect the exec'd command's environment, not the whole tty. Probably would need carefully documenting as to limitations, but a little simple IO manipulation ought to be OK. pws diff --git a/Src/exec.c b/Src/exec.c index 8f9d5a885..af33397f4 100644 --- a/Src/exec.c +++ b/Src/exec.c @@ -741,6 +741,8 @@ execute(LinkList args, int flags, int defpath) zsfree(s); } + callhookfunc("zshexternal", NULL, 1, NULL); + /* If ARGV0 is in the commands environment, we use * * that as argv[0] for this external command */ if (unset(RESTRICTED) && (z = zgetenv("ARGV0"))) { ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: stdbuf -o0 -i0 via a Zsh native interface? 2023-05-30 11:17 ` Peter Stephenson @ 2023-05-31 12:38 ` Sebastian Gniazdowski 0 siblings, 0 replies; 13+ messages in thread From: Sebastian Gniazdowski @ 2023-05-31 12:38 UTC (permalink / raw) To: Peter Stephenson; +Cc: Zsh hackers list The problem with unbuffer is that it doesn't work… It's just a simple Tcl script which aims at providing .dev.tty via a high level Tcl calls, which however turns out to work like: - catch some output ~500 lines and output it, - catch everything else and output it at the end. Proof: https://asciinema.org/a/FrONqnnrdjHLn6Lx2xhMZBNIz On Tue, 30 May 2023 at 11:18, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote: > > > On 25/05/2023 19:35 Bart Schaefer <schaefer@brasslantern.com> wrote: > > On Thu, May 25, 2023 at 7:16 AM Peter Stephenson > > <p.w.stephenson@ntlworld.com> wrote: > > > > > > > On 25/05/2023 12:21 Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote: > > > > However, this is sub-optimal solution because I have to hideously > > > > prepend the stdbuf command before each user CMD, meaning that entering > > > > builtin commands will not work (e.g.: `stdbuf -i0 -o0 print smthg`). > > > > > > > > I wonder if the infamous buffering problem, solved by the hacky > > > > ld-preload stdbuf program could be fixed on the level of Zsh? > > > > > > Something like this? It looks like a good fit for zsystem. > > > > That's a start, but it doesn't solve Sebastian's problem of having to > > hack the stdbuf prefix into the command string for external commands. > > > > We'd need something akin to the STTY parameter to specify a prefix for > > external commands. That might have its own security concerns. > > Yes, so in fact Sebastian's got the bit I originally added covered with > external commands already; they don't need to be built into the shell, > just executed appropriately. > > Can't we simply add a hook at the point where STTY is used? What > this doesn't do is the equivalent sanity after the event --- in that > case triggering TTY restoration if we detect a case where it might have > been messed up. ut that should be less important here as the changes > affect the exec'd command's environment, not the whole tty. > > Probably would need carefully documenting as to limitations, but > a little simple IO manipulation ought to be OK. > > pws > > diff --git a/Src/exec.c b/Src/exec.c > index 8f9d5a885..af33397f4 100644 > --- a/Src/exec.c > +++ b/Src/exec.c > @@ -741,6 +741,8 @@ execute(LinkList args, int flags, int defpath) > zsfree(s); > } > > + callhookfunc("zshexternal", NULL, 1, NULL); > + > /* If ARGV0 is in the commands environment, we use * > * that as argv[0] for this external command */ > if (unset(RESTRICTED) && (z = zgetenv("ARGV0"))) { > -- Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-05-31 12:39 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-05-25 11:21 stdbuf -o0 -i0 via a Zsh native interface? Sebastian Gniazdowski 2023-05-25 14:16 ` Peter Stephenson 2023-05-25 18:35 ` Bart Schaefer 2023-05-25 19:02 ` Sebastian Gniazdowski 2023-05-25 19:09 ` Bart Schaefer 2023-05-25 19:28 ` Sebastian Gniazdowski 2023-05-25 21:21 ` Bart Schaefer 2023-05-26 2:17 ` Sebastian Gniazdowski 2023-05-26 10:23 ` Sebastian Gniazdowski 2023-05-26 9:30 ` Roman Perepelitsa 2023-05-26 20:52 ` Bart Schaefer 2023-05-30 11:17 ` Peter Stephenson 2023-05-31 12:38 ` Sebastian Gniazdowski
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).