ZSH's future [longish] ?

ZSH's future [longish] ?

[Mark Borges]

I know, I hate these vague and open-ended subject lines, too. This is
not really related to zsh development either. So feel free to quit
reading, if you've even gotten this far...

Anyway, I discovered this URL last night:

     http://www.research.att.com/orgs/ssr/book/reuse/

and apparently ATT is giving away free copies of ksh93 for personal
use. You have to agree to a non-exclusive license agreement, but it
appears to be fairly liberal for single user usage. Unless I'm
misunderstanding it (quite possible) and will have the ATT police
after me tomorrow.

I was curious if anyone on this list has seen it (I've appended a list
of features below) and what their reactions would be.

How do the people actively involved in ZSH development think zsh
compares with ksh93? (Windows95, ksh93. Yuck. What's the world coming
to? ;-) 

Lest I be misunderstood, let me state that I've really enjoyed having
zsh as my shell, and have been using it for the past 4 years. I like
it better than ksh88. I don't see myself using ksh anytime soon. 

But I'm wondering if people will start flocking to ksh93 (now that it
appears to be easier to get a hold of), and the incentive and impetus
for actively developing zsh will wane such that zsh will be more of a
"hobby" than it already is.

Comments? Anyone care to do a feature-by-feature comparison? (I'm not
sure what all of them mean) If anything glaring is missing in zsh,
does anyone foresee it being added in zsh-4.0?

After a quick glance, it appears ksh93 can handle arithmetic and
arrays in a more powerful way than zsh currently does, e.g.,

	$ ./ksh
	$ ((a=1.e-3+3.4/1.2))
	$ echo $a
	2.83433333333
	$ ^D
	$ ((a=1.e-3+3.4/1.2))
	zsh: bad math expression: illegal character: .

But, so what, really. If one wants to do heavy duty math in a script,
use perl, right?

I think zsh's niche is being king of the interactive features and
conveniences. Perhaps that should be stressed more in the ZSH WWW
pages? 

  -mb-

==============================================================================
KSH-93 - The KornShell Command and Programming Language

KSH-93 is the most recent version of the KornShell Language described
in "The KornShell Command and Programming Language," by Morris Bolsky
and David Korn of AT&ampT Bell Laboratories. The KornShell is a shell
programming language, which is upward compatible with "sh" (the Bourne
Shell), and is intended to conform to the IEEE P1003.2/ISO 9945.2
Shell and Utilities standard. KSH-93 provides an enhanced programming
environment in addition to the major command-entry features of the BSD
shell "csh". With KSH-93, medium-sized programming tasks can be
performed at shell-level without a significant loss in performance. In
addition, "sh" scripts can be run on KSH-93 without modification.

The code should conform to the IEEE POSIX 1003.1 standard and to the
proposed ANSI-C standard so that it should be portable to all such
systems. Like the previous version, KSH-88, it is designed to accept
eight bit character sets transparently, thereby making it
internationally compatible. It can support multi-byte characters sets
with some characteristics of the character set given at run time.

KSH-93 provides the following features, many of which were also
inherent in KSH-88:

    o Enhanced Command Re-entry Capability: The KSH-93 history
      function records commands entered at any shell level and stores
      them, up to a user-specified limit, even after you log off. This
      allows you to re-enter long commands with a few keystrokes -
      even those commands you entered yesterday. The history file
      allows for eight bit characters in commands and supports
      essentially unlimited size histories.

    o In-line Editing: In "sh", the only way to fix mistyped commands
      is to backspace or retype the line. KSH-93 allows you to edit a
      command line using a choice of EMACS-TC or "vi" functions. You
      can use the in-line editors to complete filenames as you type
      them. You may also use this editing feature when entering
      command lines from your history file. A user can capture
      keystrokes and rebind keys to customize the editing interface.

    o Extended I/O Capabilities: KSH-93 provides several I/O
      capabilities not available in "sh", including the ability to:
        * specify a file descriptor for input and output
        * start up and run co-processes
        * produce a prompt at the terminal before a read
        * easily format and interpret responses to a menu
        * echo lines exactly as output without escape processing
        * format output using printf formats.
        * read and echo lines ending in "\\".

    o Improved performance: KSH-93 executes many scripts faster than
      the System V Bourne shell. A major reason for this is that many
      of the standard utilities are built-in. To reduce the time to
      initiate a command, KSH-93 allows commands to be added as
      built-ins at run time on systems that support dynamic loading
      such as System V Release 4.

    o Arithmetic: KSH-93 allows you to do integer arithmetic in any
      base from two to sixty-four. You can also do double precision
      floating point arithmetic. Almost the complete set of C language
      operators are available with the same syntax and
      precedence. Arithmetic expressions can be used to as an argument
      expansion or as a separate command. In addition there is an
      arithmetic for command that works like the for statement in C.

    o Arrays: KSH-93 supports both indexed and associative arrays. The
      subscript for an indexed array is an arithmetic expression,
      whereas, the subscript for an associative array is a string.

    o Shell Functions and Aliases: Two mechanisms - functions and
      aliases - can be used to assign a user-selected identifier to an
      existing command or shell script. Functions allow local
      variables and provide scoping for exception handling. Functions
      can be searched for and loaded on first reference the way
      scripts are.

    o Substring Capabilities: KSH-93 allows you to create a substring
      of any given string either by specifying the starting offset and
      length, or by stripping off leading or trailing substrings
      during parameter substitution. You can also specify attributes,
      such as upper and lower case, field width, and justification to
      shell variables.

    o More pattern matching capabilities: KSH-93 allows you to specify
      extended regular expressions for file and string matches.

    o KSH-93 uses a hierarchal name space for variables. Compound
      variables can be defined and variables can be passed by
      reference. In addition, each variable can have one or more
      disciplines associated with it to intercept assignments and
      references.

    o Improved debugging: KSH-93 can generate line numbers on
      execution traces. Also, I/O redirections are now traced. There
      is a DEBUG trap that gets evaluated after each command so that
      errors can be localized.

    o Job Control: On systems that support job control, including
      System V Release 4, KSH-93 provides a job-control mechanism
      almost identical to that of the BSD "csh", version 4.1. This
      feature allows you to stop and restart programs, and to move
      programs between the foreground and the background.

    o Added security: KSH-93 can execute scripts which do not have
      read permission and scripts which have the setuid and/or setgid
      set when invoked by name, rather than as an argument to the
      shell. It is possible to log or control the execution of setuid
      and/or setgid scripts. The noclobber option prevents you from
      accidentally erasing a file by redirecting to an existing file.

    o KSH-93 can be extended by adding built-in commands at run
      time. In addition, KSH-93 can be used as a library that can be
      embedded into an application to allow scripting.

Re: ZSH's future

[Richard Coleman]

> I know, I hate these vague and open-ended subject lines, too. This is
> not really related to zsh development either. So feel free to quit
> reading, if you've even gotten this far...

Well, it's been a while since we had a `future of zsh' discussion,
so I'll jump in.

> How do the people actively involved in ZSH development think zsh
> compares with ksh93? (Windows95, ksh93. Yuck. What's the world coming
> to? ;-) 
> 
> But I'm wondering if people will start flocking to ksh93 (now that it
> appears to be easier to get a hold of), and the incentive and impetus
> for actively developing zsh will wane such that zsh will be more of a
> "hobby" than it already is.

As long as we enjoy using and developing zsh, it doesn't really matter
what everyone else does, but I predict the use of zsh will rise rather
than fall in the future.  We've made some really good progress since
zsh-2.5.03 and I think things will continue to get better.  Also, considering
the number of hours I've been devoting to zsh, I believe it's gone way beyond
a hobby for me :-)

> I think zsh's niche is being king of the interactive features and
> conveniences. Perhaps that should be stressed more in the ZSH WWW
> pages?

I think this is the key point.  From an interactive standpoint, zsh is
the most `natural' feeling shell I've used.  In particular, zsh's
completion facilities, csh-style history mechanisms, and extended
globbing facilities make it much better at interactive use than ksh.
Ksh (especially with the new features) is probably better for heavy-duty
scripts, but since Perl is widely available, I don't think this is such
a big deal.

This is not to say that we won't add any of these features to zsh.
We will probably add some of these at some time in the future.  But if we
add any of them, it should be because it is generally useful and not just
because ksh has it.

I agree that we would stress the interactive facilities of zsh in the
documentation and web pages.  I've been thinking about adding such wording
to the man pages, but couldn't decide exactly what to say.

Richard Coleman
coleman@math.gatech.edu

Re: ZSH's future [longish] ?

[Zoltan Hidvegi]

> ...
> Shell), and is intended to conform to the IEEE P1003.2/ISO 9945.2
> Shell and Utilities standard. KSH-93 provides an enhanced programming
> ...

I think zsh is mostly POSIX compatible.  I think it would not be difficult to
improve POSIX compatibility without affectin present features.  I already did
some work for it and I use zsh as /bin/sh for more than a half year now
without problems.

> environment in addition to the major command-entry features of the BSD
> shell "csh". With KSH-93, medium-sized programming tasks can be
> performed at shell-level without a significant loss in performance. In
> addition, "sh" scripts can be run on KSH-93 without modification.

I use zsh for these medium-sized programming staff.  It is very poverfull, and
it has the advantage that it is free and I can compile zsh on any machine I
have (there are 7 different unices here).

> The code should conform to the IEEE POSIX 1003.1 standard and to the
> proposed ANSI-C standard so that it should be portable to all such

Thanks to Richard, zsh is one of the most portable program I've ever seen.  I
think ksh could'n beat it.

> systems. Like the previous version, KSH-88, it is designed to accept
> eight bit character sets transparently, thereby making it
> internationally compatible. It can support multi-byte characters sets
> with some characteristics of the character set given at run time.

This is a weak point in zsh, since it has problems with character codes
between 128 and 159.  This is not really a big broblem sine this code range is
not usually used for printable characters.  Multibyte character support is
completely missing.  I may try to do something for these sometime (but I
probably won't do anything before next summer).

History features of zsh are quite good I think (except some bugs in the
baseline which I fixed long ago), and command line editing is unbeatable.
Same for the I/O capabilities.  Performance is also pretty good although I did
not compared with ksh.

The only thing missing from zsh's arithmetic capabilities is floating point
arithmetic.  The vanilla version may also have problems with indexed arrays in
arithmetic expressions and assignments inside $((...)) substitutions.

>     o Arrays: KSH-93 supports both indexed and associative arrays. The
>       subscript for an indexed array is an arithmetic expression,
>       whereas, the subscript for an associative array is a string.

I think zsh provides all of these array features.  Some of this code is broken
in the baseline version, but I fixed these problems long ago.

>     o Substring Capabilities: KSH-93 allows you to create a substring
>       of any given string either by specifying the starting offset and
>       length, or by stripping off leading or trailing substrings
>       during parameter substitution. You can also specify attributes,
>       such as upper and lower case, field width, and justification to
>       shell variables.

Zsh can handle substrings perfectly and the variable expansion features are
excelent.  Unfortunately the baseline release have some bugs here but I fixed
these bugs log ago.

>     o More pattern matching capabilities: KSH-93 allows you to specify
>       extended regular expressions for file and string matches.

Zsh glob patterns can do anything that an extended regular expression can do.
This syntax is different but I think this is not really a big problem.  A
bonus with zsh that it can use exclude patterns and file modes for globing and
supports qualifiers.

>     o KSH-93 uses a hierarchal name space for variables. Compound
>       variables can be defined and variables can be passed by
>       reference. In addition, each variable can have one or more
>       disciplines associated with it to intercept assignments and
>       references.
> 
>     o Improved debugging: KSH-93 can generate line numbers on
>       execution traces. Also, I/O redirections are now traced. There
>       is a DEBUG trap that gets evaluated after each command so that
>       errors can be localized.

These seems to be missing from zsh.  An especially week point of zsh is
special variable handling.

Job control is fully functional in zsh.

>     o Added security: KSH-93 can execute scripts which do not have
>       read permission and scripts which have the setuid and/or setgid
>       set when invoked by name, rather than as an argument to the
>       shell. It is possible to log or control the execution of setuid
>       and/or setgid scripts. The noclobber option prevents you from
>       accidentally erasing a file by redirecting to an existing file.

These features are already in zsh, or can be added if necessary.  When zsh
invoked as setuid, it disables user startup/shutdown files.  I think it would
be reasonable to source an additional suid_zprofile when invoked as a suid
script (as it is already done when invoked as sh/ksh).

>     o KSH-93 can be extended by adding built-in commands at run
>       time. In addition, KSH-93 can be used as a library that can be
>       embedded into an application to allow scripting.

That would be nice in zsh as well.  It may help to keep zsh a small shell and
load feature modules on-demand.  E.g.  I often write complicated scripts with
zsh where the whole zle staff is not needed.

To summarize my opinion, zsh is at least as good as ksh.  It has outstanding
interactive capabilities with advanced features completely missing from ksh.
But it can also be used as a powerful script language.  An example: with my
releases the expansion of ${${(f)$(</etc/group)}%%:*} gives the list of the
group names from /etc/groups.  With ksh you would have to write $(cat -d: -f1
/etc/group) which may look nicer but requires an expensive fork and exec.

With zsh using this kind of nested expansion almost anything can be done.
This, together with the extended globbing (using file type/date/owner/length
etc) makes zsh an extremely powerful script language.  The problem is that
most of the zsh users are not aware of these capabilities.  A good and
easy-to-understand zsh book would help a lot.

Cheers,
  Zoltan

Re: ZSH's future [longish] ?

[Mark Borges]

Thanks for the comments, Zoltan.

>> On Thu, 26 Oct 1995 16:13:26 +0100 (MET),
>> Zoltan Hidvegi(Z) wrote:

>> The code should conform to the IEEE POSIX 1003.1 standard and to
>> the proposed ANSI-C standard so that it should be portable to all
>> such

Z> Thanks to Richard, zsh is one of the most portable program I've
Z> ever seen.  I think ksh could'n beat it.

Yes, I agree. Richard should take a bow here ;-).

Z> and command line editing is unbeatable.

Yep. zle wins over readline in my book.

>> o Arrays: KSH-93 supports both indexed and associative arrays. The
>> subscript for an indexed array is an arithmetic expression,
>> whereas, the subscript for an associative array is a string.

Z> I think zsh provides all of these array features.  Some of this
Z> code is broken in the baseline version, but I fixed these problems
Z> long ago.

Does zsh support associative arrays?

Z> But it can also be used as a powerful script language.  An example:
Z> with my releases the expansion of ${${(f)$(</etc/group)}%%:*} gives
Z> the list of the group names from /etc/groups.

Uh-oh. If I use the (f) qualifier I only get the first line -- without
it, I get all the group names in a single line separated by
whitespace. A bug in my (old) hzoli version -- 2.6-beta11-test7-hzoli11 --
perhaps? 

Thanks again everyone for the comments. 

Unless I receive some objections, I plan to collect information from
this thread and create a summary article to put on the web site. With
proper credits/acknowledgements, of course.

  -mb-

Re: ZSH's future [longish] ?

[Zoltan Hidvegi]

> Z> But it can also be used as a powerful script language.  An example:
> Z> with my releases the expansion of ${${(f)$(</etc/group)}%%:*} gives
> Z> the list of the group names from /etc/groups.
> 
> Uh-oh. If I use the (f) qualifier I only get the first line -- without
> it, I get all the group names in a single line separated by
> whitespace. A bug in my (old) hzoli version -- 2.6-beta11-test7-hzoli11 --
> perhaps? 

No, not a bug.  I forgot to mention that it should be put inside double
quotes.  Without double quotes word splitting is done on $(</etc/group) which
removes line breaks.  Double quotes disable word splitting but the explicit
(f) qualifier forces line splitting which is later handled like an array.
Even rc_expand_param can be used, just try "/${^${(f)$(</etc/group)}%%:*}/".
And this is only a simple example...

Cheers,
  Zoltan

Re: ZSH's future - POSIX

[Geoff Wing]

:>> Zoltan Hidvegi(Z) wrote:
:>> The code should conform to the IEEE POSIX 1003.1 standard and to
:>> the proposed ANSI-C standard so that it should be portable to all
:>> such
:Z> Thanks to Richard, zsh is one of the most portable program I've
:Z> ever seen.  I think ksh could'n beat it.
:Yes, I agree. Richard should take a bow here ;-).
	
I seem to remember there was a problem with some of the stat() and associated
stuff.  Mainly because POSIX doesn't support all the S_I*() stuff used.
There would need to be a large rewrite of one part of code to make it fully
POSIX compatible.  One really big drawback with POSIX compatibility is its
lack of support for things like symbolic-links, sockets, etc.
-- 
Mason [G.C.W]  mason@werple.mira.net.au    "Hurt...Agony...Pain...LOVE-IT"

Re: ZSH's future - POSIX

[Richard Coleman]

> > The code should conform to the IEEE POSIX 1003.1 standard and to
> > the proposed ANSI-C standard so that it should be portable to all
> > such
> 	
> I seem to remember there was a problem with some of the stat() and associated
> stuff.  Mainly because POSIX doesn't support all the S_I*() stuff used.
> There would need to be a large rewrite of one part of code to make it fully
> POSIX compatible.  One really big drawback with POSIX compatibility is its
> lack of support for things like symbolic-links, sockets, etc.

There are two notions of POSIX compatibility here.

The first is whether the code itself uses only POSIX features.  Zsh is
fairly good in this respect.  This was one of the reasons I rewrote the
signals code a while back.  But I don't think zsh would compile in a strict
POSIX environment.  Most of the problems would probably be because zsh
assumes the existentence of things not covered in the POSIX standard,
but available on most all unix machines (such as symbolic links).
This could probably be easily fixed if necessary, but I don't really
see the need for this.  If every vendor supports a feature, then it is
standard enough for me.  The goal after all is portability.

The second is whether zsh comforms to the POSIX standard concerning
shell syntax.  I believe this was what the ksh literature for ksh93
was referring to.  I believe ksh93 is supposed to be a conforming
superset of the POSIX shell standard.  I assume that zsh doesn't
conform to this standard.  It might be worth investigating what
parts of zsh don't conform.

rc