bug (feature?) in zsh

bug (feature?) in zsh

[Toshiro]

I'm using Debian unstable, zsh version 4.2.2. I've noticed that, when I 
install a new package that puts some program in my path (for 
example, /usr/bin) I can't invoke the program using completion in any zsh 
shell that I have opened before installing the program. 

Just an example to clarify this (my english is very poor :))

Let's say I install the program sdiff in /usr/bin; if I have another zshells 
open before installing the program (as is always the case in my box) and try 
to invoke the program pressing sd<TAB>, sdiff does not appear in the list of 
possible commands, I have to type 'rehash' to be able to see the new command 
in zsh.

I don't know if this is a bug or this is the intended behaviour, but I don't 
really like it. Anybody knows how to fix this? 

Regards,
Toshiro.

Re: bug (feature?) in zsh

[Bart Schaefer]

On Jan 16,  2:16am, Toshiro wrote:
} Subject: bug (feature?) in zsh
}
} I'm using Debian unstable, zsh version 4.2.2. I've noticed that, when I 
} install a new package that puts some program in my path (for 
} example, /usr/bin) I can't invoke the program using completion in any zsh 
} shell that I have opened before installing the program. 

This is almost reaching FAQ status, but isn't yet in the FAQ.

The completion system (including compctl) uses the command hash table to
look up names for completion in command position, because it's expensive
to search the path for them.  That means that if you install some new
piece of software, you need to run 'rehash' before the completion system
will see it, as you discovered.

See the zsh-users thread with subject "HASH_LIST_ALL?" from October 2004
for more discussion; in particular, article 8059 presents a technique
for working around this behavior.

Re: bug (feature?) in zsh

[Toshiro]

> > } I'm using Debian unstable, zsh version 4.2.2. I've noticed that, when I
> > } install a new package that puts some program in my path (for
> > } example, /usr/bin) I can't invoke the program using completion in any
> > zsh } shell that I have opened before installing the program.
> >
> > This is almost reaching FAQ status, but isn't yet in the FAQ.
> >
> > The completion system (including compctl) uses the command hash table to
> > look up names for completion in command position, because it's expensive
> > to search the path for them.  That means that if you install some new
> > piece of software, you need to run 'rehash' before the completion system
> > will see it, as you discovered.
>
> This isn't just true with zsh, it's true with lesser shells like bash or
> ksh.

I don't know what you really mean with this statement, but I can assure you 
that  I have to type 'rehash' to be able to see the new program in zsh; in 
bash I don't have to do anything at all, the new program is available 
immediately.

Toshiro.

Re: bug (feature?) in zsh

[William H. Magill]

On 16 Jan, 2005, at 01:33, Bart Schaefer wrote:

> On Jan 16,  2:16am, Toshiro wrote:
> } Subject: bug (feature?) in zsh
> }
> } I'm using Debian unstable, zsh version 4.2.2. I've noticed that, 
> when I
> } install a new package that puts some program in my path (for
> } example, /usr/bin) I can't invoke the program using completion in 
> any zsh
> } shell that I have opened before installing the program.
>
> This is almost reaching FAQ status, but isn't yet in the FAQ.
>
> The completion system (including compctl) uses the command hash table 
> to
> look up names for completion in command position, because it's 
> expensive
> to search the path for them.  That means that if you install some new
> piece of software, you need to run 'rehash' before the completion 
> system
> will see it, as you discovered.
>
> See the zsh-users thread with subject "HASH_LIST_ALL?" from October 
> 2004
> for more discussion; in particular, article 8059 presents a technique
> for working around this behavior

The use of a hash table for command execution/completion was 
implemented with csh -- many, many years ago. Those shells which have 
descended from csh (tcsh, zsh) behave in this same fashion. It is the 
expected behavior.

Sh behavior was always different. It did not use a hash table. 
Consequently shells which descended from sh (new sh, bash, ksh) do not 
have a hash table.

This difference was mainly because the "csh" was intended as an 
"interactive" shell, and therefore included features (like command 
completion) which made usage on what at the time were the standard 
input devices -- 110 baud ttys -- much easier on the operator. By 
comparison, sh, was deemed a "batch" shell where the full command paths 
were expected to be utilized, both as "documentation" in the script and 
to assure that the "environment" used by the shell was explicitly what 
the script writer wanted, and not that of the entity using the shell.  
(Note that this is still considered "Best Practice." And that it is 
required for most implementations of "cron" in use today -- the 
environment available to the cron job is that of process 1, NOT that of 
the User.)

Note also that I use the term "descended from" quite liberally. In the 
beginning there was only Bell Labs and it was called "sh." "sh" was 
replaced by "new sh" (frequently called sh5) when it became System V. 
Meanwhile BSD created csh to run on top of sh. All Unix variants and 
Linux variants boot into some form of "sh." Today, I believe all 
variants of Unix and Linux run "new sh" for process 0, which in turn 
spawns init. Only after User processes are spawned do the /etc/shells 
options come into play.

That this is not in the FAQ is not really surprising. Interesting, but 
not surprising.

For some strange reason, throughout the history of Unix, the concept of 
"rehash" has always been a deeply guarded secret of the Unix Wizards 
... only taught to the truly deserving; neophytes who sought to follow 
the ways of the guru. Others were simply told, "Logout and log back in, 
and it will work."

For the record, the "strange reason" is the difference between the 
"System Administrator" (aka root) and the "User" -- everybody else. The 
System Administrator is technically the only one who installs (or can 
install) software available to anyone other than the person installing 
it. "Technically," the System Administrator only does such a thing on a 
"closed" system -- i.e. one not permitting user logins (/etc/nologin). 
Therefore, the SysAdmin needs to "rehash" for testing purposes, but 
everyone else will "automagically" see the new information when they 
login.

Today, there are probably far more "SysAdmins" supporting "single-user" 
boxes, especially under Linux, than there are SysAdmins supporting 
multi-user, time-sharing systems.  This is especially true with the 
widespread use of sudo as opposed to actually logging in as the root 
user.

T.T.F.N.
William H. Magill
# Beige G3 [Rev A motherboard - 300 MHz 768 Meg] OS X 10.2.8
# Flat-panel iMac (2.1) [800MHz - Super Drive - 768 Meg] OS X 10.3.7
# PWS433a [Alpha 21164 Rev 7.2 (EV56)- 64 Meg] Tru64 5.1a
# XP1000  [Alpha 21264-3 (EV6) - 256 meg] FreeBSD 5.3
# XP1000  [Alpha 21264-A (EV 6.7) - 384 meg] FreeBSD 5.3
magill@mcgillsociety.org
magill@acm.org
magill@mac.com
whmagill@gmail.com

Re: bug (feature?) in zsh

[Bart Schaefer]

On Jan 17, 10:59am, William H. Magill wrote:
} Subject: Re: bug (feature?) in zsh
}
} On 16 Jan, 2005, at 01:33, Bart Schaefer wrote:
} 
} > The completion system (including compctl) uses the command hash table 
} 
} The use of a hash table for command execution/completion was 
} implemented with csh -- many, many years ago. Those shells which have 
} descended from csh (tcsh, zsh) behave in this same fashion. It is the 
} expected behavior.

This would almost be correct, except that zsh can't really be said to
have "descended" from csh.  It was written entirely independently and
was always meant to be more like ksh than csh; Paul Falstad borrowed
some csh features that he liked, is all.

Furthermore, it's possible to turn off zsh's use of the hash table if
all you're doing is command execution -- but completion forces the hash
table to be filled and uses it exclusively.
 
} This difference was mainly because the "csh" was intended as an 
} "interactive" shell, and therefore included features (like command 
} completion) which made usage on what at the time were the standard 
} input devices -- 110 baud ttys -- much easier on the operator.

Several years passed after csh's first deployment before completion
was added.

} In the beginning there was only Bell Labs and it was called "sh."
} [...]  Meanwhile BSD created csh to run on top of sh.

"Run on top of"??  Csh is and always has been written directly in C,
not as some kind of front on sh.

} All Unix variants and Linux variants boot into some form of "sh."
} Today, I believe all variants of Unix and Linux run "new sh" for
} process 0, which in turn spawns init.

This is news to me, and to the authors of "hp-ux 11i internals" as well
(which I happen to have sitting here because the publisher sent me an
unsolicited copy hoping I'd review it on Amazon).  The boot loader loads
the kernel, and the kernel spawns init directly, not by going through
sh. (This doesn't mean the kernel doesn't also spawn shells.)

} That this is not in the FAQ is not really surprising. Interesting, but 
} not surprising.

What I meant "isn't yet in the FAQ" is the difference between completion
and execution with regard to the hash table behavior.

Re: bug (feature?) in zsh

[Stephane Chazelas]

On Mon, Jan 17, 2005 at 10:59:54AM -0500, William H. Magill wrote:
[...]
> The use of a hash table for command execution/completion was 
> implemented with csh -- many, many years ago. Those shells which have 
> descended from csh (tcsh, zsh) behave in this same fashion. It is the 
> expected behavior.
> 
> Sh behavior was always different. It did not use a hash table. 
> Consequently shells which descended from sh (new sh, bash, ksh) do not 
> have a hash table.
[...]

bash has a hash table for commands.

The difference is that, when not finding a command in its
hash table, it looks in $PATH for the command (that hash table
doesn't seem to be used for command completion, though ($PATH is
scanned by bash each time you hit tab)).

> Note also that I use the term "descended from" quite liberally. In the 
> beginning there was only Bell Labs and it was called "sh." "sh" was 
> replaced by "new sh" (frequently called sh5) when it became System V. 
> Meanwhile BSD created csh to run on top of sh. All Unix variants and 
> Linux variants boot into some form of "sh." Today, I believe all 
> variants of Unix and Linux run "new sh" for process 0, which in turn 
> spawns init. Only after User processes are spawned do the /etc/shells 
> options come into play.

modern Unices (except Tru64 and Solaris) don't have a SysV shell
anymore. Their /bin/sh is generally a POSIX conformant shell
like ksh, bash or a ash derivative.

I don't think any Unix ever started a shell on boot startup
(except maybe in some specific run-levels).

/etc/shells is a database of valid user shells queried by commands
like chsh, su or in.ftpd (through the GNU or BSD C library function
getusershell for instance), it has nothing to do with the kernel.

"user processes" are spawn from whatever process that changes
the uid (login, su, sudo, dtlogin, xdm...). Typically:

init -> getty -> login (setuid) -> user login shell
or:
init -> startup-script -> su www (setuid) -> httpd
or:
init -> dtlogin startup script -> dtlogin (setuid) -> dtsession
-> dtterm -> user shell

[...]
> For some strange reason, throughout the history of Unix, the concept of 
> "rehash" has always been a deeply guarded secret of the Unix Wizards 
> ... only taught to the truly deserving; neophytes who sought to follow 
> the ways of the guru. Others were simply told, "Logout and log back in, 
> and it will work."

In my experience rehash has always been know to csh users.

> For the record, the "strange reason" is the difference between the 
> "System Administrator" (aka root) and the "User" -- everybody else. The 
> System Administrator is technically the only one who installs (or can 
> install) software available to anyone other than the person installing 
> it. "Technically," the System Administrator only does such a thing on a 
> "closed" system -- i.e. one not permitting user logins (/etc/nologin). 
> Therefore, the SysAdmin needs to "rehash" for testing purposes, but 
> everyone else will "automagically" see the new information when they 
> login.
[...]

Unix environments have often been multiuser environments for
developers. In such contexts, many non-admin users would install
home software on their account and be confronted to the problem.

-- 
Stéphane

Re: bug (feature?) in zsh

[Seth Kurtzberg]

[-- Attachment #1: Type: text/plain, Size: 4856 bytes --]

Unfortunately, as I believe has been noted, several of the vendors 
shipping SVR4 UNIX ship with csh as the default shell for the root 
account.  However, they also ship with an equivalent account, toor (root 
spelled backwards) with either sh or ksh as the shell.

Personally, I dropped csh instantly as soon a an alternative with 
command history (ksh) (let alone completion) became available .  There 
are a few csh features that I like, but, in generally, I find it 
extremely aggregating to use.  I'm not sure why, as I've written code in 
C almost as soon as it existed, and I still use C in some cases for 
device drivers, but the shell drives me straight up the wall.

On my systems, I create a third user, zoot, which has (surprise) zsh as 
the shell.  I realize the zoot doesn't spell anything backwards, but it 
seemed like a sensible name to me for some odd reason.

Historically, it is true that csh was in no way related (in terms of 
code reuse) to sh.  csh is also, unfortunately, not fully re-entrant, 
and, for that reason, a good thing to stay away from.  In my opinion zsh 
has every useful csh feature, but that's just my opinion.  I've been 
trying to think of a time recently that I used anything other than 
!####, and I can't, but I may have forgotten a lot of csh features in 
the considerable number of years since ksh came out.

Since you can run zsh under cygwin, I haven't run any other shell for a 
long time.  I'm something of a zsh zealot, though.

Stephane Chazelas wrote:

>On Mon, Jan 17, 2005 at 10:59:54AM -0500, William H. Magill wrote:
>[...]
>  
>
>>The use of a hash table for command execution/completion was 
>>implemented with csh -- many, many years ago. Those shells which have 
>>descended from csh (tcsh, zsh) behave in this same fashion. It is the 
>>expected behavior.
>>
>>Sh behavior was always different. It did not use a hash table. 
>>Consequently shells which descended from sh (new sh, bash, ksh) do not 
>>have a hash table.
>>    
>>
>[...]
>
>bash has a hash table for commands.
>
>The difference is that, when not finding a command in its
>hash table, it looks in $PATH for the command (that hash table
>doesn't seem to be used for command completion, though ($PATH is
>scanned by bash each time you hit tab)).
>
>  
>
>>Note also that I use the term "descended from" quite liberally. In the 
>>beginning there was only Bell Labs and it was called "sh." "sh" was 
>>replaced by "new sh" (frequently called sh5) when it became System V. 
>>Meanwhile BSD created csh to run on top of sh. All Unix variants and 
>>Linux variants boot into some form of "sh." Today, I believe all 
>>variants of Unix and Linux run "new sh" for process 0, which in turn 
>>spawns init. Only after User processes are spawned do the /etc/shells 
>>options come into play.
>>    
>>
>
>modern Unices (except Tru64 and Solaris) don't have a SysV shell
>anymore. Their /bin/sh is generally a POSIX conformant shell
>like ksh, bash or a ash derivative.
>
>I don't think any Unix ever started a shell on boot startup
>(except maybe in some specific run-levels).
>
>/etc/shells is a database of valid user shells queried by commands
>like chsh, su or in.ftpd (through the GNU or BSD C library function
>getusershell for instance), it has nothing to do with the kernel.
>
>"user processes" are spawn from whatever process that changes
>the uid (login, su, sudo, dtlogin, xdm...). Typically:
>
>init -> getty -> login (setuid) -> user login shell
>or:
>init -> startup-script -> su www (setuid) -> httpd
>or:
>init -> dtlogin startup script -> dtlogin (setuid) -> dtsession
>-> dtterm -> user shell
>
>[...]
>  
>
>>For some strange reason, throughout the history of Unix, the concept of 
>>"rehash" has always been a deeply guarded secret of the Unix Wizards 
>>... only taught to the truly deserving; neophytes who sought to follow 
>>the ways of the guru. Others were simply told, "Logout and log back in, 
>>and it will work."
>>    
>>
>
>In my experience rehash has always been know to csh users.
>
>  
>
>>For the record, the "strange reason" is the difference between the 
>>"System Administrator" (aka root) and the "User" -- everybody else. The 
>>System Administrator is technically the only one who installs (or can 
>>install) software available to anyone other than the person installing 
>>it. "Technically," the System Administrator only does such a thing on a 
>>"closed" system -- i.e. one not permitting user logins (/etc/nologin). 
>>Therefore, the SysAdmin needs to "rehash" for testing purposes, but 
>>everyone else will "automagically" see the new information when they 
>>login.
>>    
>>
>[...]
>
>Unix environments have often been multiuser environments for
>developers. In such contexts, many non-admin users would install
>home software on their account and be confronted to the problem.
>
>  
>


[-- Attachment #2: Type: text/html, Size: 5615 bytes --]