stream buffering

stream buffering

[Rory Mulvaney]

I'm wondering if the reason that some I/O intensive zsh shell scripts 
aren't fast is mostly only because the I/O is (I think) line-buffered. 
Is there a way to do full buffering of streams, or what would be the best 
way to provide it?  Maybe a module similar to zsh/net/socket, perhaps 
using setvbuf(3)?

Also, thanks for the big help on my coproc question a little while ago.

Thanks in advance,
Rory

Re: stream buffering

[Peter Stephenson]

On Thu, 17 Feb 2011 19:40:20 -0600 (CST)
Rory Mulvaney <rorymulv@gmail.com> wrote:
> I'm wondering if the reason that some I/O intensive zsh shell scripts 
> aren't fast is mostly only because the I/O is (I think) line-buffered. 
> Is there a way to do full buffering of streams, or what would be the best 
> way to provide it?  Maybe a module similar to zsh/net/socket, perhaps 
> using setvbuf(3)?

What I/O are you referring to?  The shell's own input for a script needs
to be parsed and intepreted, which will be much more of a restriction
than buffering except for scripts with really silly amounts of embedded
text like here documents.  Standard input to the shell via the read
builtin etc. similarly tends to be in short chunks, likewise output via
print.  Input to other commands run from scripts makes its own
arrangements for buffering.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: stream buffering

[Peter Stephenson]

On Fri, 18 Feb 2011 17:56:10 -0600
Rory Mulvaney <rorymulv@gmail.com> wrote:
> Sorry, I'm definitely referring to the type of I/O using read and
> print (or some other builtin created by a module).  And I don't
> necessarily mean in a script; it could just be from a loop on the
> command line where the loop reads or prints all lines in a file.
> 
> while read var ; do 
>   -something- 
> done

There are sysread and syswrite commands in the zsh/system module,
allowing you to read much larger chunks in one go.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: stream buffering

[Rory Mulvaney]

On Fri, 18 Feb 2011, Peter Stephenson wrote:

> On Thu, 17 Feb 2011 19:40:20 -0600 (CST)
> Rory Mulvaney <rorymulv@xxxxxxxxx> wrote:
> 
> > I'm wondering if the reason that some I/O intensive zsh shell scripts 
> > aren't fast is mostly only because the I/O is (I think) line-buffered. 
> > Is there a way to do full buffering of streams, or what would be the 
> > best way to provide it?  Maybe a module similar to zsh/net/socket, 
> > perhaps using setvbuf(3)?
> 
> What I/O are you referring to?  The shell's own input for a script needs 
> to be parsed and intepreted, which will be much more of a restriction than 
> buffering except for scripts with really silly amounts of embedded text 
> like here documents.  Standard input to the shell via the read builtin 
> etc. similarly tends to be in short chunks, likewise output via print.  
> Input to other commands run from scripts makes its own arrangements for 
> buffering.

Sorry, I'm definitely referring to the type of I/O using read and print 
(or some other builtin created by a module).  And I don't necessarily mean 
in a script; it could just be from a loop on the command line where the 
loop reads or prints all lines in a file.

while read var ; do 
  -something- 
done

-Rory 

[sorry if there are some incorrect threading headers, since I just 
subscribed now]