The speed of zsh

The speed of zsh

[Zoltan Hidvegi]

It's rumored that zsh is faster than bash.  I made some experiments and I
found that this is not really true.  Executing the same script bash is
about as fast as zsh and both ksh and pdksh are 2-3 times faster than zsh.
This is true for executing a scipt containing only builtins as well as for
scripts calling external commands.  Somehow ksh spawns external commands
twice as fast as zsh.  I used Linux-2.0.10, ld.so.1.7.14, libc-5.3.12,
AT&T ksh 12/28/93d, pdksh-5.2.7 and bash 1.14.6 for my tests.

The patch below improves zsh preformance by 10-15%.  An other 10% speed
improvement would be possible by avoiding the
child_block()/child_unblock() calls whenever possible (other shells do not
use any system calls while executing builtin-only scritpts).

Of course using special zsh features zsh scripts for the same task may be
faster than sh/ksh scripts.

I used a modified 99 bottle of beer script with beers=99999 bottles, zsh
called as sh:

s="s"

while [ $beers -gt 0 ]; do
        : "$beers bottle$s of beer on the wall,"
        : "$beers bottle$s of beer,"
        : "take one down, pass it around,"
        let beers=beers-1
        if [ $beers -ne 0 ]; then
                test $beers -eq 1 && s=""
                : "$beers bottle$s of beer on the wall."
        else
                : "no bottles of beer on the wall."
        fi
        :
done

Execution time was 408.53s+31.71s, 99% CPU, 7:22.85 total
Note that it is more than normal because of profiling, but the important
part here is the 31.71s system time.  ksh93 execution time was
97.13s+0.08s, 99% CPU, 1:38.08 total

Flat profile for zsh with the patch below:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 35.46    122.36   122.36                             mcount
  6.12    143.48    21.12   999998     0.02     0.16  execcmd
  4.60    159.35    15.87   899991     0.02     0.04  paramsubst
  3.96    173.02    13.67 20300232     0.00     0.00  halloc
  2.70    182.34     9.32  3099986     0.00     0.01  stringsubst
  2.32    190.34     8.00   999998     0.01     0.02  execpline
  2.10    197.57     7.23  1200080     0.01     0.01  getvalue
  1.88    204.07     6.50  4199985     0.00     0.00  untokenize
  1.84    210.42     6.35  2699984     0.00     0.00  haswilds
  1.63    216.03     5.61   899994     0.01     0.07  execbuiltin
  1.62    221.61     5.58  1600000     0.00     0.04  prefork
  1.58    227.07     5.46 15000174     0.00     0.00  hcalloc
  1.58    232.52     5.45  4899955     0.00     0.00  dupstruct2
  1.54    237.84     5.32 12299887     0.00     0.00  dupnode
  1.41    242.70     4.86   400004     0.01     0.02  execlist
  1.16    246.69     3.99  2300329     0.00     0.00  ztrdup
  1.12    250.54     3.85   999998     0.00     0.00  execpline2
  0.99    253.97     3.43  3099986     0.00     0.00  remnulargs
  0.94    257.21     3.24  9999950     0.00     0.00  closemn
  0.92    260.40     3.19  2900008     0.00     0.00  dupstring
  0.89    263.48     3.08  2199984     0.00     0.00  zzlex
  0.88    266.52     3.04  1800058     0.00     0.00  gethashnode
  0.88    269.56     3.04   899994     0.00     0.01  mathevall
  0.85    272.48     2.92  3599967     0.00     0.00  duplist
  0.84    275.38     2.90  3000514     0.00     0.00  hasher
  0.74    277.94     2.56   999998     0.00     0.00  deletejob
  0.70    280.36     2.42  4799981     0.00     0.00  ugetnode
  0.70    282.76     2.40   899994     0.00     0.01  mathparse
  0.61    284.88     2.12  1199992     0.00     0.00  testlex
  0.60    286.95     2.07  1099996     0.00     0.00  getstrvalue
  0.57    288.93     1.98  3099986     0.00     0.00  filesubstr
  0.57    290.91     1.98   299998     0.01     0.02  par_cond_2
  0.57    292.87     1.96   999995     0.00     0.00  fixfds
  0.57    294.82     1.95   899991     0.00     0.00  strcatsub
  0.54    296.68     1.86   899994     0.00     0.00  zstrtol
  0.52    298.49     1.81  1200153     0.00     0.00  gethashnode2
  0.52    300.30     1.81   999998     0.00     0.00  initjob
  0.50    302.02     1.72   299998     0.01     0.15  evalcond
  0.50    303.73     1.71  3200034     0.00     0.00  zsfree
  0.50    305.44     1.71  2699984     0.00     0.00  glob
  ...
  0.00    345.09     0.00        1     0.00     0.00  zexit

Replacing let beers=beers-1 with beers=`expr $beers - 1` in the script,
using beers=9999 and unprofiled (so faster) zsh the execution time is
216.34s+239.19s, 99% CPU, 7:36.86 total
ksh93 execution time is 92.37s+108.46s, 99% CPU, 3:21.31 total

Note that ksh used less than half as much system time as zsh.

Zoltan


*** Src/exec.c	1996/08/15 10:39:14	2.84
--- Src/exec.c	1996/08/24 15:46:59
***************
*** 1642,1648 ****
  #ifdef HAVE_DEV_FD
  		int i;
  
! 		for (i = 10; i < OPEN_MAX; i++)
  		    if (fdtable[i] > 1)
  			fdtable[i]++;
  #endif
--- 1642,1648 ----
  #ifdef HAVE_DEV_FD
  		int i;
  
! 		for (i = 10; i <= max_zsh_fd; i++)
  		    if (fdtable[i] > 1)
  			fdtable[i]++;
  #endif
***************
*** 1651,1657 ****
  		subsh_close = -1;
  		execshfunc(cmd, (Shfunc) hn);
  #ifdef HAVE_DEV_FD
! 		for (i = 10; i < OPEN_MAX; i++)
  		    if (fdtable[i] > 1)
  			if (--(fdtable[i]) <= 2)
  			    zclose(i);
--- 1651,1657 ----
  		subsh_close = -1;
  		execshfunc(cmd, (Shfunc) hn);
  #ifdef HAVE_DEV_FD
! 		for (i = 10; i <= max_zsh_fd; i++)
  		    if (fdtable[i] > 1)
  			if (--(fdtable[i]) <= 2)
  			    zclose(i);
***************
*** 1917,1923 ****
  {
      int i;
  
!     for (i = 10; i < OPEN_MAX; i++)
  	if (fdtable[i] && (!how || fdtable[i] == how))
  	    zclose(i);
  }
--- 1917,1923 ----
  {
      int i;
  
!     for (i = 10; i <= max_zsh_fd; i++)
  	if (fdtable[i] && (!how || fdtable[i] == how))
  	    zclose(i);
  }
*** Src/globals.h	1996/08/15 16:40:04	2.40
--- Src/globals.h	1996/08/15 16:40:04
***************
*** 394,399 ****
--- 394,403 ----
  
  EXTERN char fdtable[OPEN_MAX];
  
+ /* The highest fd that marked with nonzero in fdtable */
+ 
+ EXTERN int max_zsh_fd;
+ 
  /* input fd from the coprocess */
  
  EXTERN int coprocin;
***************
*** 555,560 ****
--- 559,572 ----
  #ifdef DEBUG
  EXTERN int alloc_stackp;
  #endif
+ 
+ /* Variables used by signal queueing */
+ 
+ EXTERN int queueing_enabled;
+ EXTERN sigset_t signal_mask_queue[MAX_QUEUE_SIZE];
+ EXTERN int signal_queue[MAX_QUEUE_SIZE];
+ EXTERN int queue_front;
+ EXTERN int queue_rear;
  
  /* 1 if aliases should not be expanded */
   
*** Src/signals.c	1996/08/12 01:36:46	2.21
--- Src/signals.c	1996/08/23 21:14:25
***************
*** 343,397 ****
      return ret;
  }
  
- 
- /* Use a circular queue to save signals caught during    *
-  * critical sections of code.  You call queue_signals to *
-  * start queueing, and unqueue_signals to process the    *
-  * queue and stop queueing.  Since the kernel doesn't    *
-  * queue signals, it is probably overkill for zsh to do  *
-  * this, but it shouldn't hurt anything to do it anyway. */
- 
- /* Right now I'm queueing all signals, but maybe we only *
-  * need to queue SIGCHLD.  Anybody know?                 */
- 
- 
- #define MAX_QUEUE_SIZE 16
- 
- static int queueing_enabled;
- static sigset_t signal_mask_queue[MAX_QUEUE_SIZE];
- static int signal_queue[MAX_QUEUE_SIZE];
- static int queue_front = 0;
- static int queue_rear = 0;
- 
- /* Start queueing up any received signals rather *
-  * than handling them.                           */
- 
- /**/
- void
- queue_signals(void)
- {
-     queueing_enabled++;        /* signals are now being queued */
- }
- 
- /* Process all queued signals */
- 
- /**/
- void
- unqueue_signals(void)
- {
-     DPUTS(!queueing_enabled, "BUG: unqueue_signals called but not queueing");
-     if (--queueing_enabled)
- 	return;
- 
-     while (queue_front != queue_rear) {            /* while signals in queue */
- 	sigset_t oset;
-         queue_front = ++queue_front % MAX_QUEUE_SIZE;
-         oset = signal_setmask(signal_mask_queue[queue_front]);
-         handler(signal_queue[queue_front]);        /* handle queued signal   */
- 	signal_setmask(oset);
-     }
- }
-  
  /* What flavor of waitpid/wait3/wait shall we use? */
   
  #ifdef HAVE_WAITPID
--- 343,348 ----
*** Src/signals.h	1996/06/28 02:05:24	2.2
--- Src/signals.h	1996/06/28 02:05:24
***************
*** 70,72 ****
--- 70,98 ----
  /* return a signal to it default action */
  #define signal_default(S)  signal(S, SIG_DFL)
  
+ /* Use a circular queue to save signals caught during    *
+  * critical sections of code.  You call queue_signals to *
+  * start queueing, and unqueue_signals to process the    *
+  * queue and stop queueing.  Since the kernel doesn't    *
+  * queue signals, it is probably overkill for zsh to do  *
+  * this, but it shouldn't hurt anything to do it anyway. */
+ 
+ /* Right now I'm queueing all signals, but maybe we only *
+  * need to queue SIGCHLD.  Anybody know?                 */
+ 
+ #define MAX_QUEUE_SIZE 16
+ 
+ #define queue_signals()    (queueing_enabled++)
+ 
+ #define unqueue_signals()  do { \
+     DPUTS(!queueing_enabled, "BUG: unqueue_signals called but not queueing"); \
+     if (!--queueing_enabled) { \
+ 	while (queue_front != queue_rear) {      /* while signals in queue */ \
+ 	    sigset_t oset; \
+ 	    queue_front = ++queue_front % MAX_QUEUE_SIZE; \
+ 	    oset = signal_setmask(signal_mask_queue[queue_front]); \
+ 	    handler(signal_queue[queue_front]);  /* handle queued signal   */ \
+ 	    signal_setmask(oset); \
+ 	} \
+     } \
+ } while (0)
*** Src/utils.c	1996/08/14 16:21:47	2.51
--- Src/utils.c	1996/08/23 22:13:59
***************
*** 868,879 ****
  #else
  	int fe = movefd(dup(fd));
  #endif
! 	close(fd);
! 	fdtable[fd] = 0;
  	fd = fe;
      }
!     if(fd != -1)
  	fdtable[fd] = 1;
      return fd;
  }
  
--- 868,881 ----
  #else
  	int fe = movefd(dup(fd));
  #endif
! 	zclose(fd);
  	fd = fe;
      }
!     if(fd != -1) {
  	fdtable[fd] = 1;
+ 	if (fd > max_zsh_fd)
+ 	    max_zsh_fd = fd;
+     }
      return fd;
  }
  
***************
*** 883,896 ****
  void
  redup(int x, int y)
  {
!     if(x < 0) {
! 	close(y);
! 	fdtable[y] = 0;
!     } else if (x != y) {
  	dup2(x, y);
! 	fdtable[y] = fdtable[x];
! 	close(x);
! 	fdtable[x] = 0;
      }
  }
  
--- 885,897 ----
  void
  redup(int x, int y)
  {
!     if(x < 0)
! 	zclose(y);
!     else if (x != y) {
  	dup2(x, y);
! 	if ((fdtable[y] = fdtable[x]) && y > max_zsh_fd)
! 	    max_zsh_fd = y;
! 	zclose(x);
      }
  }
  
***************
*** 900,907 ****
  int
  zclose(int fd)
  {
!     if (fd >= 0)
  	fdtable[fd] = 0;
      return close(fd);
  }
  
--- 901,911 ----
  int
  zclose(int fd)
  {
!     if (fd >= 0) {
  	fdtable[fd] = 0;
+ 	while (!fdtable[max_zsh_fd])
+ 	    max_zsh_fd--;
+     }
      return close(fd);
  }
  
EndOfPatch

Re: The speed of zsh

[Bart Schaefer]

On Aug 24,  9:59pm, Zoltan Hidvegi wrote:
} Subject: The speed of zsh
}
} Somehow ksh spawns external commands twice as fast as zsh.

I suspect it has something to do with zsh's use of pipes for synchronizing
parent and child processes; the zsh parent doesn't do anything until the
child finishes its entersubsh() and closes the pipe.  Ksh probably doesn't
create the pipe in the first place.

It also occurs to me that with child_block()/child_unblock() in use, zsh
probably doesn't *need* the pipe-based synchronization any longer.  Take
it out and see what happens.

} The patch below improves zsh preformance by 10-15%.  An other 10% speed
} improvement would be possible by avoiding the
} child_block()/child_unblock() calls whenever possible (other shells do not
} use any system calls while executing builtin-only scritpts).

I think I said before that we could avoid those if we aren't going to fork
and if the job table is empty.  Maybe they're expensive enough to be worth
the extra test.  On what do you base the 10% figure?

(I'm reasonably sure that with no forks and nothing in the job table, zsh
needs neither the signal blocks nor the pipe sync.)

}   %   cumulative   self              self     total           
}  time   seconds   seconds    calls  ms/call  ms/call  name    
}   4.60    159.35    15.87   899991     0.02     0.04  paramsubst
}   3.96    173.02    13.67 20300232     0.00     0.00  halloc
}   1.58    227.07     5.46 15000174     0.00     0.00  hcalloc

Wow.  That's an awful lot of allocations (no wonder the heap makes so much
difference to zsh's speed), and paramsubst() is pretty expensive per call.

-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"

Re: The speed of zsh

[Zoltan Hidvegi]

> } The patch below improves zsh preformance by 10-15%.  An other 10% speed
> } improvement would be possible by avoiding the
> } child_block()/child_unblock() calls whenever possible (other shells do not
> } use any system calls while executing builtin-only scritpts).
> 
> I think I said before that we could avoid those if we aren't going to fork
> and if the job table is empty.  Maybe they're expensive enough to be worth
> the extra test.  On what do you base the 10% figure?

Because zsh time is 26.11s+3.00s, 94% CPU, 30.659 total while ksh is
9.99s+0.00s, 93% CPU, 10.674 total for 9999 bottles of beer.

It's clear that zsh spends 10% of the time in system calls and the only
system call used by zsh while executing non-forking builtin-only scripts is
sigprocmask (which can be easily seen using strace).

Zoltan

Re: The speed of zsh

[Zoltan Hidvegi]

> } Somehow ksh spawns external commands twice as fast as zsh.
> 
> I suspect it has something to do with zsh's use of pipes for synchronizing
> parent and child processes; the zsh parent doesn't do anything until the
> child finishes its entersubsh() and closes the pipe.  Ksh probably doesn't
> create the pipe in the first place.

It the script I tried there was only a process substitution which does not
use synch pipes.  Anyway I did remove synch calls and it did make zsh a bit
faster even when it executed scripts which never before used synch perhaps
because of the better use of CPU cache.  And I now discovered the real
reason why that particular script was so slow.  Zsh executes process
substitutions after forking.  This means that the commands used in process
substitutions never get hashed in the main process so the path is searched
each time.  After adding hash expr to the script zsh become almost as fast
as ksh.  But builtin-only scripts are still much slower than in ksh :-).

Attached is the patch to remove pipe synchronization code.

Zoltan

*** Src/exec.c	1996/08/25 10:32:58	2.85
--- Src/exec.c	1996/08/25 12:05:46
***************
*** 718,731 ****
  		if ((list_pipe || last1) && !list_pipe_child &&
  		    jn->stat & STAT_STOPPED) {
  		    pid_t pid;
- 		    int synch[2];
- 
- 		    pipe(synch);
  
  		    if ((pid = fork()) == -1) {
  			trashzle();
- 			close(synch[0]);
- 			close(synch[1]);
  			putc('\n', stderr);
  			fprintf(stderr, "zsh: job can't be suspended\n");
  			fflush(stderr);
--- 718,726 ----
***************
*** 734,747 ****
  			thisjob = newjob;
  		    }
  		    else if (pid) {
- 			char dummy;
- 
  			list_pipe_pid = pid;
  			nowait = errflag = 1;
  			breaks = loops;
- 			close(synch[1]);
- 			read(synch[0], &dummy, 1);
- 			close(synch[0]);
  			jobtab[list_pipe_job].other = newjob;
  			jobtab[list_pipe_job].stat |= STAT_SUPERJOB;
  			jn->stat |= STAT_SUBJOB | STAT_NOPRINT;
--- 729,737 ----
***************
*** 750,759 ****
  			break;
  		    }
  		    else {
- 			close(synch[0]);
  			entersubsh(Z_ASYNC, 0, 0);
  			setpgrp(0L, mypgrp = jobtab[list_pipe_job].gleader);
- 			close(synch[1]);
  			kill(getpid(), SIGSTOP);
  			list_pipe = 0;
  			list_pipe_child = 1;
--- 740,747 ----
***************
*** 818,843 ****
  	 * shell command, do foo in a subshell and do the     *
  	 * rest of the pipeline in the current shell.         */
  	if (pline->left->type >= CURSH && (how & Z_SYNC)) {
- 	    int synch[2];
- 
- 	    pipe(synch);
  	    if ((pid = fork()) == -1) {
- 		close(synch[0]);
- 		close(synch[1]);
  		zerr("fork failed: %e", NULL, errno);
  	    } else if (pid) {
! 		char dummy, *text;
  
  		text = getjobtext((void *) pline->left);
  		addproc(pid, text);
- 		close(synch[1]);
- 		read(synch[0], &dummy, 1);
- 		close(synch[0]);
  	    } else {
  		zclose(pipes[0]);
- 		close(synch[0]);
  		entersubsh(how, 2, 0);
- 		close(synch[1]);
  		execcmd(pline->left, input, pipes[1], how, 0);
  		_exit(lastval);
  	    }
--- 806,821 ----
  	 * shell command, do foo in a subshell and do the     *
  	 * rest of the pipeline in the current shell.         */
  	if (pline->left->type >= CURSH && (how & Z_SYNC)) {
  	    if ((pid = fork()) == -1) {
  		zerr("fork failed: %e", NULL, errno);
  	    } else if (pid) {
! 		char *text;
  
  		text = getjobtext((void *) pline->left);
  		addproc(pid, text);
  	    } else {
  		zclose(pipes[0]);
  		entersubsh(how, 2, 0);
  		execcmd(pline->left, input, pipes[1], how, 0);
  		_exit(lastval);
  	    }
***************
*** 1411,1430 ****
          sigtrapped[SIGEXIT] || havefiles()))))) {
  
  	pid_t pid;
- 	int synch[2];
- 	char dummy;
  
  	child_block();
- 	pipe(synch);
  
  	if ((pid = zfork()) == -1) {
- 	    close(synch[0]);
- 	    close(synch[1]);
  	    return;
  	} if (pid) {
- 	    close(synch[1]);
- 	    read(synch[0], &dummy, 1);
- 	    close(synch[0]);
  #ifdef HAVE_DEV_FD
  	    closem(2);
  #endif
--- 1389,1400 ----
***************
*** 1442,1450 ****
  	    return;
  	}
  	/* pid == 0 */
- 	close(synch[0]);
  	entersubsh(how, type != SUBSH && !(how & Z_ASYNC) ? 2 : 1, 0);
- 	close(synch[1]);
  	forked = 1;
  	if (sigtrapped[SIGINT] & ZSIG_IGNORED)
  	    holdintr();
--- 1412,1418 ----

Re: The speed of zsh

[Bart Schaefer]

On Aug 25,  4:25pm, Zoltan Hidvegi wrote:
} Subject: Re: The speed of zsh
}
} > } Somehow ksh spawns external commands twice as fast as zsh.
} > 
} > I suspect it has something to do with zsh's use of pipes for synchronizing
} > parent and child processes
} 
} It the script I tried there was only a process substitution which does not
} use synch pipes.

I was never sure whether you were always talking about that script or if
you'd done some other profiling as well.  My remark wasn't specific to
the script.

} Anyway I did remove synch calls and it did make zsh a bit
} faster even when it executed scripts which never before used synch perhaps
} because of the better use of CPU cache.  [...]
} 
} Attached is the patch to remove pipe synchronization code.

This needs to be tested carefully to make sure various combinations of
piping to and from builtin loops and simple builtins still work right,
and to make sure zsh still returns the correct exit status from pipelines
where later elements exit before earlier elements are finished; especially
pipelines where a command on the rhs (or in the middle) fails to execute
and therefore exits immediately.

I forget all the reasons why the pipe-sync stuff was added long ago, but
I think those were the kinds of situations involved.


-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"