zsh-users
 help / Atom feed
* zsh poor performances while reading and testing ?
@ 2019-07-03 13:58 Marc Chantreux
  0 siblings, 0 replies; 9+ messages in thread
From: Marc Chantreux @ 2019-07-03 13:58 UTC (permalink / raw)
  To: Zsh Users

hello people,

i recently made a benchmark to emphasize the gain of speed
people can have using filters instead of pure shell loops.
however i was surprised to see how slow zsh is compared to
other shells when it comes to read data.

the interesting part of the benchmark is:

  for it (bash zsh ksh) {
    TIMEFMT=": $it %U %S %E"
    time $it  -c 'while read it; do : ; done < x > /dev/null'
  }

: bash 4,95s 1,21s 6,18s
: zsh 12,65s 28,12s 40,82s
: ksh 9,87s 26,52s 36,42s

is there any obvious reason for that? is there a way to make
it faster without diving in the C code?

regards,
marc

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
       [not found] <20190703135824.GA19289__20170.6622539618$1562162400$gmane$org@prometheus.u-strasbg.fr>
@ 2019-07-03 14:28 ` Stephane Chazelas
  2019-07-05 13:28   ` Marc Chantreux
       [not found]   ` <20190705132842.GA21074__38577.2207098611$1562333415$gmane$org@prometheus.u-strasbg.fr>
  0 siblings, 2 replies; 9+ messages in thread
From: Stephane Chazelas @ 2019-07-03 14:28 UTC (permalink / raw)
  To: Marc Chantreux; +Cc: Zsh Users

2019-07-03 15:58:24 +0200, Marc Chantreux:
[...]
> i recently made a benchmark to emphasize the gain of speed
> people can have using filters instead of pure shell loops.
> however i was surprised to see how slow zsh is compared to
> other shells when it comes to read data.
> 
> the interesting part of the benchmark is:
> 
>   for it (bash zsh ksh) {
>     TIMEFMT=": $it %U %S %E"
>     time $it  -c 'while read it; do : ; done < x > /dev/null'
>   }
> 
> : bash 4,95s 1,21s 6,18s
> : zsh 12,65s 28,12s 40,82s
> : ksh 9,87s 26,52s 36,42s
> 
> is there any obvious reason for that? is there a way to make
> it faster without diving in the C code?
[...]

read reads one byte at a time so as not to read past the newline
character. On seekable files bash and ksh implement an
optimisation whereby they read more than one byte (128 bytes in
bash IIRC) but seek back to the last newline.

ksh93 goes even further in that it remembers the excess bytes it
has read and reuses them for the next builtin commands, causing
this kind of bug: https://github.com/att/ast/issues/15

You'll probably find that they are all as inefficient for
non-seekable non-peekable input like pipes.

IIRC, ksh93 implements "|" with socketpair() instead of pipe()
on Linux so it can "peek" data to do this kind of optimisation

$ strace  ksh -c 'seq 20 | read'
[...]
socketpair(AF_UNIX, SOCK_STREAM, 0, [3, 4]) = 0
shutdown(4, SHUT_RD)                    = 0
fchmod(4, 0200)                         = 0
shutdown(3, SHUT_WR)                    = 0
fchmod(3, 0400)                         = 0
[...]
fcntl(3, F_DUPFD, 0)                    = 0
[...]
recvfrom(0, "1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14"..., 65536, MSG_PEEK, NULL, NULL) = 51
read(0, "1\n", 2)                       = 2
[...]

Again that kind of optimisation can backfire and be invalid if
there's more than one reader to the pipe (not to mention the
problems on linux where /dev/fd/x doesn't work socketpairs).

By the way, the syntax to read a line is

IFS= read -r line

not

read line

even in zsh.

-- 
Stephane

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-03 14:28 ` Stephane Chazelas
@ 2019-07-05 13:28   ` Marc Chantreux
  2019-07-05 13:44     ` Peter Stephenson
  2019-07-05 13:44     ` Mikael Magnusson
       [not found]   ` <20190705132842.GA21074__38577.2207098611$1562333415$gmane$org@prometheus.u-strasbg.fr>
  1 sibling, 2 replies; 9+ messages in thread
From: Marc Chantreux @ 2019-07-05 13:28 UTC (permalink / raw)
  To: Marc Chantreux, Zsh Users

hello,

> You'll probably find that they are all as inefficient for
> non-seekable non-peekable input like pipes.

actually my point making this bench was: don't use shell to write
serious filters. however i really appreciate knowing why this difference
exists. thanks a lot.

> IFS= read -r line

ok for -r but as long as i use only one variable, why is it important to
use IFS= ?

regards
marc

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-05 13:28   ` Marc Chantreux
@ 2019-07-05 13:44     ` Peter Stephenson
  2019-07-05 14:21       ` Marc Chantreux
  2019-07-05 13:44     ` Mikael Magnusson
  1 sibling, 1 reply; 9+ messages in thread
From: Peter Stephenson @ 2019-07-05 13:44 UTC (permalink / raw)
  To: zsh-users

On Fri, 2019-07-05 at 15:28 +0200, Marc Chantreux wrote: 
> > You'll probably find that they are all as inefficient for
> > non-seekable non-peekable input like pipes.
> actually my point making this bench was: don't use shell to write
> serious filters. however i really appreciate knowing why this difference
> exists. thanks a lot.
> 
> > 
> > IFS= read -r line
> ok for -r but as long as i use only one variable, why is it important to
> use IFS= ?

As Look and Learn magazine used to say, Find Out By Doing.


% read -r line
  full line  
           ^^ spaces here
% print -r -- "'$line'"
'  full line  '
% read -r line
  full line  
           ^^ spaces here
% print -r -- "'$line'"
'full line'


pws


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-05 13:28   ` Marc Chantreux
  2019-07-05 13:44     ` Peter Stephenson
@ 2019-07-05 13:44     ` Mikael Magnusson
  1 sibling, 0 replies; 9+ messages in thread
From: Mikael Magnusson @ 2019-07-05 13:44 UTC (permalink / raw)
  To: Marc Chantreux; +Cc: Zsh Users

On 7/5/19, Marc Chantreux <eiro@phear.org> wrote:
> hello,
>
>> You'll probably find that they are all as inefficient for
>> non-seekable non-peekable input like pipes.
>
> actually my point making this bench was: don't use shell to write
> serious filters. however i really appreciate knowing why this difference
> exists. thanks a lot.
>
>> IFS= read -r line
>
> ok for -r but as long as i use only one variable, why is it important to
> use IFS= ?

% echo '  hello ' | IFS= read -r one; echo -E - _${one}_
_  hello _
% echo '  hello ' | read -r one; echo -E - _${one}_
_hello_


-- 
Mikael Magnusson

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-05 13:44     ` Peter Stephenson
@ 2019-07-05 14:21       ` Marc Chantreux
  2019-07-05 14:29         ` Peter Stephenson
  0 siblings, 1 reply; 9+ messages in thread
From: Marc Chantreux @ 2019-07-05 14:21 UTC (permalink / raw)
  To: Peter Stephenson; +Cc: zsh-users

hello,

thanks to you and Mikael, it's obvious now but frankly this is tricky
because very unexpected for me (i played around read for years without
notifying this).

thanks and regards,
marc

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-05 14:21       ` Marc Chantreux
@ 2019-07-05 14:29         ` Peter Stephenson
  2019-07-05 15:53           ` Marc Chantreux
  0 siblings, 1 reply; 9+ messages in thread
From: Peter Stephenson @ 2019-07-05 14:29 UTC (permalink / raw)
  To: zsh-users

> On 05 July 2019 at 15:21 Marc Chantreux <eiro@phear.org> wrote:
> thanks to you and Mikael, it's obvious now but frankly this is tricky
> because very unexpected for me (i played around read for years without
> notifying this).

There are certainly good reasons apart from inefficiency for finding
scripting languages other than the shell for implementing utilities
involving I/O :-/.

Personally, I don't use shell scripting these days for anything other
than helping with interactive use, and I don't consider myself
particularly unfamiliar with the shell.

pws

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
  2019-07-05 14:29         ` Peter Stephenson
@ 2019-07-05 15:53           ` Marc Chantreux
  0 siblings, 0 replies; 9+ messages in thread
From: Marc Chantreux @ 2019-07-05 15:53 UTC (permalink / raw)
  To: Peter Stephenson; +Cc: zsh-users

hello Peter,

> Personally, I don't use shell scripting these days for anything other
> than helping with interactive use, and I don't consider myself
> particularly unfamiliar with the shell.

doesn't seems so ;)

as i said: my fist attempt when i wrote this benchmark was to point out
how fast are filters compared to while loops. so i don't use zsh for
those kind of stuff too.

i was using zsh+(many filters) for decades now because in some cases,
it was the fastest way to get things done (even perl5 isn't even close
to the simplicity of some shell scripts). the only recent game changer
for me it perl6: for the first time of my life, i feel it's time to kiss
shell scripting goodbye.

regards,
marc

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: zsh poor performances while reading and testing ?
       [not found]   ` <20190705132842.GA21074__38577.2207098611$1562333415$gmane$org@prometheus.u-strasbg.fr>
@ 2019-07-05 18:05     ` Stephane Chazelas
  0 siblings, 0 replies; 9+ messages in thread
From: Stephane Chazelas @ 2019-07-05 18:05 UTC (permalink / raw)
  To: Marc Chantreux; +Cc: Zsh Users

2019-07-05 15:28:42 +0200, Marc Chantreux:
[...]
> > IFS= read -r line
> 
> ok for -r but as long as i use only one variable, why is it important to
> use IFS= ?
[...]

See also:

https://unix.stackexchange.com/questions/209123/understanding-ifs-read-r-line

-- 
Stephane

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, back to index

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-03 13:58 zsh poor performances while reading and testing ? Marc Chantreux
     [not found] <20190703135824.GA19289__20170.6622539618$1562162400$gmane$org@prometheus.u-strasbg.fr>
2019-07-03 14:28 ` Stephane Chazelas
2019-07-05 13:28   ` Marc Chantreux
2019-07-05 13:44     ` Peter Stephenson
2019-07-05 14:21       ` Marc Chantreux
2019-07-05 14:29         ` Peter Stephenson
2019-07-05 15:53           ` Marc Chantreux
2019-07-05 13:44     ` Mikael Magnusson
     [not found]   ` <20190705132842.GA21074__38577.2207098611$1562333415$gmane$org@prometheus.u-strasbg.fr>
2019-07-05 18:05     ` Stephane Chazelas

zsh-users

Archives are clonable: git clone --mirror http://inbox.vuxu.org/zsh-users

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.zsh.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox