zsh-3.0-pre1: speed up the "list-choices" etc.

zsh-3.0-pre1: speed up the "list-choices" etc.

[SUZUKI Hisao]

Dear z-shell workers,

     Recently I found an inconvenience in doing "list-choices"
or "delete-char-or-list" (typically by typing Control-d) on a
character terminal connected to an i860-sysv4 box with RS-232C.
The speed of displaying completion lists is _forbiddingly_ slow
on this terminal.

     Certainly it is mainly because of the terminal and its
device interface, it is possible to speed up the "list-choices"
etc. by improving the z-shell.  It is `nicezputs' in Src/utils.c
that displays the string of completion list, and its current
implementation calls `fputs' for each character in the string.
Modifying Src/utils.c so that `nicezputs' calls `fputs' once for
all characters, the speed becomes tolerable even on the slow
terminal.

     The following is the sample fix; it defines a new function
<<static int to_nicechar(int c, char *buf)>>, which turns `c'
into a "nice string" on `buf' and returns its length as the
result value.  In the function `nicezputs', the local buffer
<<auto char buf[200]>> is used to store each nice string by the
statement <<i += to_nicechar(c, buf + i)>> for each character,
and is dumped with <<fputs(buf, stream)>> when it is filled or
all characters are stored.

     In addition, <<char *nicechar(int c)>> is redefined in
terms of `to_nicechar' to avoid duplication of the code, and
<<size_t nicestrlen(char *s)>> and <<size_t niceztrlen(char
const *s)>> are modified slightly so as to invoke `to_nicechar'
instead of `nicechar' to improve efficiency by counting the
length of strings without re-scanning them.

Regards,

SUZUKI Hisao		# `Begin at the beginning and go on till
suzuki@otsl.oki.co.jp	# you come to the end:  then stop.'

----------------------------------------------------------------
*** utils.c.orig	Fri Jun 28 13:43:51 1996
--- utils.c	Sun Jul  7 12:50:04 1996
***************
*** 148,168 ****
      return 0;
  }
  
! /* Turn a character into a visible representation thereof.  The visible *
!  * string is put together in a static buffer, and this function returns *
!  * a pointer to it.  Printable characters stand for themselves, DEL is  *
!  * represented as "^?", newline and tab are represented as "\n" and     *
!  * "\t", and normal control characters are represented in "^C" form.    *
   * Characters with bit 7 set, if unprintable, are represented as "\M-"  *
   * followed by the visible representation of the character with bit 7   *
   * stripped off.  Tokens are interpreted, rather than being treated as  *
   * literal characters.                                                  */
  
! /**/
! char *
! nicechar(int c)
  {
-     static char buf[6];
      char *s = buf;
      c &= 0xff;
      if (isprint(c))
--- 148,170 ----
      return 0;
  }
  
! /* Turn a character which is given as the first argument into a visible *
!  * representation thereof.  The visible string is put together in a     *
!  * buffer which is given as the second argument.  This function returns *
!  * the length of the string, which is less than NICECHAR_MAX.           *
!  * Printable characters stand for themselves, DEL is represented as     *
!  * "^?", newline and tab are represented as "\n" and "\t" respectively, *
!  * and normal control characters are represented in "^C" form.          *
   * Characters with bit 7 set, if unprintable, are represented as "\M-"  *
   * followed by the visible representation of the character with bit 7   *
   * stripped off.  Tokens are interpreted, rather than being treated as  *
   * literal characters.                                                  */
  
! #define NICECHAR_MAX 6
! 
! static int
! to_nicechar(int c, char *buf)
  {
      char *s = buf;
      c &= 0xff;
      if (isprint(c))
***************
*** 191,196 ****
--- 193,211 ----
      done:
      *s++ = c;
      *s = 0;
+     return s - buf;		/* length of the resulting string */
+ }
+ 
+ /* Turn a character into a visible representation thereof.  The visible *
+  * string is put together in a static buffer, and this function returns *
+  * a pointer to it.                                                     */
+ 
+ /**/
+ char *
+ nicechar(int c)
+ {
+     static char buf[NICECHAR_MAX];
+     to_nicechar(c, buf);
      return buf;
  }
  
***************
*** 212,221 ****
  size_t
  nicestrlen(char *s)
  {
      size_t l = 0;
  
      for (; *s; s++)
! 	l += strlen(nicechar(STOUC(*s)));
      return l;
  }
  
--- 227,237 ----
  size_t
  nicestrlen(char *s)
  {
+     char buf[NICECHAR_MAX];
      size_t l = 0;
  
      for (; *s; s++)
! 	l += to_nicechar(STOUC(*s), buf);
      return l;
  }
  
***************
*** 3071,3077 ****
--- 3087,3095 ----
  int
  nicezputs(char const *s, FILE *stream)
  {
+     char buf[200];
      int c;
+     int i = 0;
  
      while ((c = *s++)) {
  	if (itok(c))
***************
*** 3081,3089 ****
  		continue;
  	if (c == Meta)
  	    c = *s++ ^ 32;
! 	if(fputs(nicechar(c), stream) < 0)
! 	    return EOF;
      }
      return 0;
  }
  
--- 3099,3113 ----
  		continue;
  	if (c == Meta)
  	    c = *s++ ^ 32;
! 	i += to_nicechar(c, buf + i);
! 	if (i + NICECHAR_MAX > 200) {
! 	    if (fputs(buf, stream) < 0) return EOF;
! 	    i = 0;
! 	}
      }
+     if (i > 0) {
+ 	if (fputs(buf, stream) < 0) return EOF;
+     }
      return 0;
  }
  
***************
*** 3093,3098 ****
--- 3117,3123 ----
  size_t
  niceztrlen(char const *s)
  {
+     char buf[NICECHAR_MAX];
      size_t l = 0;
      int c;
  
***************
*** 3104,3110 ****
  		continue;
  	if (c == Meta)
  	    c = *s++ ^ 32;
! 	l += strlen(nicechar(STOUC(c)));
      }
      return l;
  }
--- 3129,3135 ----
  		continue;
  	if (c == Meta)
  	    c = *s++ ^ 32;
! 	l += to_nicechar(STOUC(c), buf);
      }
      return l;
  }

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Zoltan Hidvegi]

SUZUKI Hisao wrote:
> etc. by improving the z-shell.  It is `nicezputs' in Src/utils.c
> that displays the string of completion list, and its current
> implementation calls `fputs' for each character in the string.
> Modifying Src/utils.c so that `nicezputs' calls `fputs' once for
> all characters, the speed becomes tolerable even on the slow
> terminal.

The speed difference should not be noticeable if the output goes to a
buffered stream.  If the output is not buffered that's really a bug.

Zoltan

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Zefram]

>     Certainly it is mainly because of the terminal and its
>device interface, it is possible to speed up the "list-choices"
>etc. by improving the z-shell.  It is `nicezputs' in Src/utils.c
>that displays the string of completion list, and its current
>implementation calls `fputs' for each character in the string.

The reason it does that (and I can answer with authority here as I
originated that code) is that fputs(), being part of stdio, should
buffer output.  It is consequently not really significant how many
times it actually gets called.  Had we been dealing with unbuffered
output, I would have added some explicit buffering.

>Modifying Src/utils.c so that `nicezputs' calls `fputs' once for
>all characters, the speed becomes tolerable even on the slow
>terminal.

If that's the case, then it indicates something more fundamental is
wrong.  I suspect that it's a problem with your system.  In my
experience, zsh's completion lists appear at pretty much the speed the
terminal is set to (9600 baud for me at the moment, and that's
normal).  zsh doesn't disable buffering on the tty output stream at any
point -- it uses fflush() appropriately.

-zefram

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[SUZUKI Hisao]

Zoltan Hidvegi <hzoli@cs.elte.hu> wrote:
>SUZUKI Hisao wrote:
>> etc. by improving the z-shell.  It is `nicezputs' in Src/utils.c
>> that displays the string of completion list, and its current
>> implementation calls `fputs' for each character in the string.
>> Modifying Src/utils.c so that `nicezputs' calls `fputs' once for
>> all characters, the speed becomes tolerable even on the slow
>> terminal.
>
>The speed difference should not be noticeable if the output goes to a
>buffered stream.  If the output is not buffered that's really a bug.

Indeed the speed difference is hardly noticeable on
Sparc-SunOS4.1, but it is noticeable on both i860-SysV4.0 and
Sparc-Solaris2.3 (Solaris2 is a version of SysV4 in fact, you
know).

Certainly it is utterly intolerable only with slow I/O (and slow
CPU), the difference is _noticeable_ even if you use a
high-speed terminal (such as xterm on X11) and a not-so-slow CPU
(such as SPARCstation5 running Solaris2.3).  For example, you
can see the listing displayed character-by-character on xterm
running Sparc-Solaris2.3 z-shell:

	% cd zsh-3.0-pre1/Src
	% <^L>				[clear-screen]
	% ls <SPC><^D>			[delete-char-or-list]
	zsh: do you wish to see all 114 possibilities? y
	[...listing displayed character-by-character...]

Undoubtedly the output is not buffered on these machines, and I
doubt if it is a bug in itself.

# BTW, `Suzuki' is my family name and `Hisao' is my given name.
# In Japan, the family name comes first and the given name last,
# as in Hungary.  As you call yourself `Zoltan Hidvegi' in
# English context, I'd better have called myself `Hisao Suzuki'
# as long before...

Regards,

SUZUKI Hisao		# `Begin at the beginning and go on till
suzuki@otsl.oki.co.jp	# you come to the end:  then stop.'

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Zefram]

>Indeed the speed difference is hardly noticeable on
>Sparc-SunOS4.1, but it is noticeable on both i860-SysV4.0 and
>Sparc-Solaris2.3 (Solaris2 is a version of SysV4 in fact, you
>know).

I can't reproduce that under Solaris 2.4.

-zefram

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Bart Schaefer]

On Jul 8,  9:03am, Zefram wrote:
} Subject: Re: zsh-3.0-pre1: speed up the "list-choices" etc.
}
} >Indeed the speed difference is hardly noticeable on
} >Sparc-SunOS4.1, but it is noticeable on both i860-SysV4.0 and
} >Sparc-Solaris2.3 (Solaris2 is a version of SysV4 in fact, you
} >know).
} 
} I can't reproduce that under Solaris 2.4.

It's entirely possible that it's reproducible only under Solaris 2.x with
the Japanese language environment installed.  I know there are some pretty
radical differences there, which we had to deal with in zmail.  (However,
I don't know any of the details, as we farmed out that port.)

-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[SUZUKI Hisao]

>>Indeed the speed difference is hardly noticeable on
>>Sparc-SunOS4.1, but it is noticeable on both i860-SysV4.0 and
>>Sparc-Solaris2.3 (Solaris2 is a version of SysV4 in fact, you
>>know).
>
>I can't reproduce that under Solaris 2.4.

Surely you cannot reproduce it?  Well, then it might be a bug of
the _old_ SysV4...  In any way, it would be necessary to confirm
it more objectively.  Please do the following:

	% truss zsh
	[wait till the z-shell prompt appears]
	[then type <v> <SPC> <^D> to list the completions]

If you see something like

	write(10, " caml-light/ "... 48)	= 48

then the output is buffered.  Or if you see something like

	write(10, " c", 1)			= 1

then it is not buffered.

SUZUKI Hisao		# `Begin at the beginning and go on till
suzuki@otsl.oki.co.jp	# you come to the end:  then stop.'

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Zefram]

>	% truss zsh

Why didn't I think of that before?

I can confirm that output is UNbuffered under Solaris 2, but buffered
under SunOS 4.  How remarkable.  Does anyone know *why* it's
unbuffered?  Could this be a SysV thing?

-zefram

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[Zoltan Hidvegi]

> >	% truss zsh
> 
> Why didn't I think of that before?
> 
> I can confirm that output is UNbuffered under Solaris 2, but buffered
> under SunOS 4.  How remarkable.  Does anyone know *why* it's
> unbuffered?  Could this be a SysV thing?

Quote from the Solaris 2.4 fdopen manual:

     When opened, a stream is fully buffered if and  only  if  it
     can  be  determined  not  to refer to an interactive device.
     The error and end-of-file indicators  are  cleared  for  the
     stream.

The patch below should fix this.  Note that here setvbuff is called with
NULL buffer and if _IOFBF is not defined setbuffer is not called.  That's
because this problem seems to affect only SVR4 systems where setvbuf and
_IOFBF is always defined.  And setvbuf used to be called with NULL buffer
but the Linux libc-5.3.6 and earlier versions had a bug that setvbuf with
NULL buffer did not work on unbuffered streams.  But Linux fdopen always
gives a buffered stream so that is not a problem here and libc older than
5.3.12 is obsolate anyway :-).

Zoltan


*** Src/init.c	1996/07/08 01:56:51	2.25
--- Src/init.c	1996/07/08 14:14:18
***************
*** 365,370 ****
--- 365,373 ----
  
      /* Associate terminal file descriptor with a FILE pointer */
      shout = fdopen(SHTTY, "w");
+ #ifdef _IOFBF
+     setvbuf(shout, NULL, _IOFBF, 0);
+ #endif
    
      gettyinfo(&shttyinfo);	/* get tty state */
  #if defined(__sgi)

Re: zsh-3.0-pre1: speed up the "list-choices" etc.

[SUZUKI Hisao]

>The patch below should fix this.  Note that here setvbuff is called with
>NULL buffer and if _IOFBF is not defined setbuffer is not called.  That's
>because this problem seems to affect only SVR4 systems where setvbuf and
>_IOFBF is always defined.

The patch works finely for i860-SysV4.0, too.

Now I can continue to use the z-shell not only on Sparcs etc.
but also on this (minor :-) machine.  Thank you very much!

SUZUKI Hisao		# `Begin at the beginning and go on till
suzuki@otsl.oki.co.jp	# you come to the end:  then stop.'