[PATCH] Proposal: stat -> zstat

[PATCH] Proposal: stat -> zstat

[Phil Pennock]

Since a stat external command has become popular, perhaps it's worth zsh
4.4 migrating users towards using zstat as a builtin?  Then turn off
stat somewhere around zsh 6 ... *cough*

Also, by having two builtins which can be independently disabled, the
{...} always {...} stuff from zcalsys becomes thankfully unnecessary.

I was thinking of making stat be a function wrapper, but I suspect that
some people may have scripts using "command stat" vs "builtin stat".

Thoughts?

diff -urp zsh-head/Doc/Zsh/builtins.yo zsh-zstat/Doc/Zsh/builtins.yo
--- zsh-head/Doc/Zsh/builtins.yo	Fri Mar 23 10:14:05 2007
+++ zsh-zstat/Doc/Zsh/builtins.yo	Wed May  2 22:30:21 2007
@@ -2014,6 +2014,7 @@ module(zprof)(zsh/zprof)
 module(zpty)(zsh/zpty)
 module(zregexparse)(zsh/zutil)
 module(zsocket)(zsh/net/socket)
+module(zstat)(zsh/stat)
 module(zstyle)(zsh/zutil)
 module(ztcp)(zsh/net/tcp)
 enditem()
diff -urp zsh-head/Doc/Zsh/calsys.yo zsh-zstat/Doc/Zsh/calsys.yo
--- zsh-head/Doc/Zsh/calsys.yo	Mon Mar 26 07:33:33 2007
+++ zsh-zstat/Doc/Zsh/calsys.yo	Wed May  2 22:05:35 2007
@@ -466,8 +466,9 @@ findex(age)
 The function tt(age) can be autoloaded and use separately from
 the calendar system, although it uses the function tt(calendar_scandate)
 for date formatting.  It requires the tt(zsh/stat) builtin, which
-makes available the builtin tt(stat).  This may conflict with an
-external programme of the same name; if it does, the builtin may be
+makes available the builtin tt(zstat).   This currently still provides
+the old name tt(stat) which may conflict with an
+external programme of that same name; if it does, the builtin may be
 disabled for normal operation by including the following code in
 an initialization file:
 
diff -urp zsh-head/Doc/Zsh/mod_stat.yo zsh-zstat/Doc/Zsh/mod_stat.yo
--- zsh-head/Doc/Zsh/mod_stat.yo	Fri Apr  7 13:07:10 2000
+++ zsh-zstat/Doc/Zsh/mod_stat.yo	Wed May  2 22:29:19 2007
@@ -1,13 +1,28 @@
 COMMENT(!MOD!zsh/stat
 A builtin command interface to the tt(stat) system call.
 !MOD!)
-The tt(zsh/stat) module makes available one builtin command:
+The tt(zsh/stat) module makes available two builtin commands which are
+the same internally.
 
 startitem()
 findex(stat)
+item(tt(stat) ...)(
+This is identical to tt(zstat) but use of the name tt(stat) is deprecated.
+As the years have passed, an external command named tt(stat) has appeared
+with incompatible output.  Zsh has renamed its tt(stat) to tt(zstat) and
+is currently maintaining the old name for backwards compatibility.
+When writing scripts, please use tt(zstat).  One day, zsh will remove
+its tt(stat) builtin; in the meantime, you can disable the tt(stat) name
+whilst keeping tt(zstat) with the command:
+
+example(disable stat)
+)
+
+startitem()
+findex(zstat)
 cindex(files, listing)
 cindex(files, examining)
-item(tt(stat) [ tt(-gnNolLtTrs) ] [ tt(-f) var(fd) ] \
+item(tt(zstat) [ tt(-gnNolLtTrs) ] [ tt(-f) var(fd) ] \
     [ tt(-H) var(hash) ] [ tt(-A) var(array) ] \
     [ tt(-F) var(fmt) ] [ tt(PLUS())var(element) ] [ var(file) ... ])(
 The command acts as a front end to the tt(stat) system call (see
diff -urp zsh-head/Functions/Calendar/age zsh-zstat/Functions/Calendar/age
--- zsh-head/Functions/Calendar/age	Mon Dec  4 02:59:11 2006
+++ zsh-zstat/Functions/Calendar/age	Wed May  2 22:17:51 2007
@@ -27,24 +27,16 @@
 # any argument is passed in the first format.
 
 emulate -L zsh
-integer mystat disable_stat
+integer mystat
 
 zmodload -i zsh/stat
-# Allow the builtin stat to be hidden.
-zmodload -i zsh/parameter
-
-{
-if [[ $builtins[stat] != defined ]]; then
-  (( disable_stat = 1 ))
-  enable stat
-fi
 
 autoload -U calendar_scandate
 
 local -a vals
 
 [[ -e $REPLY ]] || return 1
-stat -A vals +mtime $REPLY || return 1
+zstat -A vals +mtime $REPLY || return 1
 
 if (( $# >= 1 )); then
   local AGEREF=$1
@@ -77,9 +69,6 @@ else
   mystat=1
 fi
 
-} always {
-# If the builtin stat was previously disabled, disable it again.
-(( disable_stat )) && disable stat
 }
 
 return $mystat
diff -urp zsh-head/Functions/Example/zls zsh-zstat/Functions/Example/zls
--- zsh-head/Functions/Example/zls	Mon Apr  2 05:29:06 2001
+++ zsh-zstat/Functions/Example/zls	Wed May  2 22:19:12 2007
@@ -49,7 +49,7 @@ fi
 
 for f in $*
 do
-    stat -s$L -H stat -F "%b %e %H:%M" - $f || continue
+    zstat -s$L -H stat -F "%b %e %H:%M" - $f || continue
     if [[ $opts != *d* && $stat[mode] == d* ]] then
 	dirs=( $dirs $f )
     elif [[ $opts == *l* ]] then
diff -urp zsh-head/Src/Modules/stat.c zsh-zstat/Src/Modules/stat.c
--- zsh-head/Src/Modules/stat.c	Tue May 30 15:35:03 2006
+++ zsh-zstat/Src/Modules/stat.c	Wed May  2 22:00:36 2007
@@ -620,6 +620,7 @@ bin_stat(char *name, char **args, Option
 }
 
 static struct builtin bintab[] = {
+    BUILTIN("zstat", 0, bin_stat, 0, -1, 0, NULL, NULL),
     BUILTIN("stat", 0, bin_stat, 0, -1, 0, NULL, NULL),
 };
 
diff -urp zsh-head/Src/Modules/stat.mdd zsh-zstat/Src/Modules/stat.mdd
--- zsh-head/Src/Modules/stat.mdd	Sun Nov 26 12:01:03 2000
+++ zsh-zstat/Src/Modules/stat.mdd	Wed May  2 22:19:56 2007
@@ -2,6 +2,6 @@ name=zsh/stat
 link=dynamic
 load=no
 
-autobins="stat"
+autobins="stat zstat"
 
 objects="stat.o"

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> Since a stat external command has become popular, perhaps it's worth
> zsh 4.4 migrating users towards using zstat as a builtin?

Adding zstat is useful.  We could possibly still make modules that only
use it internally disable stat if the module wasn't previously
loaded---otherwise removing the disabling actually makes things worse.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: [PATCH] Proposal: stat -> zstat

[Bart Schaefer]

On May 2, 10:39pm, Phil Pennock wrote:
} Subject: [PATCH] Proposal: stat -> zstat
}
} Since a stat external command has become popular, perhaps it's worth zsh
} 4.4 migrating users towards using zstat as a builtin?

Adding zstat is not a problem, but the stat command in the module is
intended to be a replacement for the external command, in the same
way that mv, cp, etc. in the zsh/files module are.  So although I'd
be in favor of some means of making the stat builtin functionality
available to modules that want it, without hiding the external stat,
I'm opposed to the idea of removing the name "stat" for the builtin.

Perhaps what we should be thinking of is a more general utility for
zmodload, along the lines of Perl's export-tag semantics, so that a
module can be asked to selectively install only those bits that the
caller needs.

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> Perhaps what we should be thinking of is a more general utility for
> zmodload, along the lines of Perl's export-tag semantics, so that a
> module can be asked to selectively install only those bits that the
> caller needs.

How about the following.  This tries quite hard not to break anything.

Each module can advertise a set of features it supports.  These
will be variables, builtins, conditions or math functions; we'll need a
convention for how to disambiguate.  I wouldn't propose to build the
convention into the API, however, just represent each feature by a string
and rely on modules to use the convention.  This makes it extensible.

At the C API, we can then add standard module functions to query and enable
or disable features.  On most systems we can just search for appropriate
functions; on AIX there's some magic I don't yet understand turning the
standard module functions into an index, so it might be harder.  Since we
need to recompile modules for each version of the shell, this only becomes
a problem with someone else's add-on that hasn't been upgraded.

We can have one function to pass up a string list of features as an array
and the current set of enabled features, and another function to pass down
the new set of enabled features.  The default is for all features to be
enabled.

The list of features is in the same internal format as a zsh array.

The set of enabled features can be an arbitrary-length array of unsigned
integers with bits giving the enabled status.  We can explicitly limit
ourselves to the first 32 bits of each integer for future proofing.
Obviously we need one bit for each feature array element.  Alternatively,
we could use an integer array of the same length as the feature list and
declare all but the bottom bits a being for future expansion.

At the shell command API,

zmodload <module>
  loads everything (as at present).  If the module is already loaded, and
  supports features, zmodload will check to see if all features are loaded.
  If they are, it's an error.  If not, it will enable them all.
  (This is necessary to fix cases like a function looking for a builtin
  it knows is supplied by a module and loading the module if it doesn't
  find the builtin.)
zmodload -i <module>
  will load the module if it's not loaded, and enable all features.

zmodload -F <module> feature1 feature2
   will load the module or cause an error if it's already loaded,
   and turn on feature1 and feature2 (only---all others are disabled).
zmodload -iF <module> -feature1 feature3
   will load the module if it's not already loaded, turn off feature1
   and turn on feature3, leaving all other features alone.
zmodload -lF <module>
   lists all enabled features of <module>, one per line.  It returns
   status 1 (printing an error message?  or not?) if <module> is
   not loaded.
zmodload -LF <module>
   lists the zmodload commands required to turn on the currently
   enabled set of features.  This might be "zmodload <module>" if
   all features are enabled.

If you try to enable or disable a feature the module doesn't say it
supports, an error message is printed and status 1 is returned (but any
other feature change requests will be processed).

New functions that just require particular features will call "zmodload -iF
features...".  Older functions can still call "zmodload -i" and rely on
all the feature being present.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: [PATCH] Proposal: stat -> zstat

[Phil Pennock]

On 2007-05-03 at 09:50 +0100, Peter Stephenson wrote:
> Adding zstat is useful.  We could possibly still make modules that only
> use it internally disable stat if the module wasn't previously
> loaded---otherwise removing the disabling actually makes things worse.

Hrm.

Situation before:
  If zsh/stat loaded and stat enabled:
    happy days
  If zsh/stat loaded and stat disabled:
    temporarily reenable, disable with exception
  If zsh/stat not loaded and user likes builtin stat:
    gloriously happy days
  If zsh/stat not loaded and user dislikes builtin stat:
    manual-page documents work-around

Situation with my patch:
  If zsh/stat loaded and zstat enabled:
    happy days, don't touch stat
  If zsh/stat loaded and zstat disabled:
    hrm.  Why would zstat be disabled?  Perhaps worth covering, but
    nothing else works around "functions loaded but disabled" so this
    would be the exception
  If zsh/stat loaded and stat disabled:
    Fine.  We leave it that way.
  If zsh/stat not loaded and user likes builtin stat:
    gloriously happy days
  If zsh/stat not loaded and user dislikes builtin stat:
    manual-page documents work-around (same)

I'm not seeing what makes things worse.  History suggests that given
that you've claimed it does then I'm wrong and just not seeing it, so
I'd appreciate a clarification.  :^)

Now, one thing which might be useful in addition to the proposed
extended syntax for zmodload would be to add a new flag BINF_DISABLED
which actually names the (1<<0) case and offers something to go in a
module, so that a builtin table entry can per definition be disabled by
default.

So the stat documentation note could state something like:

 In the next major release of zsh, stat will still be a valid loadable
 command but it will not be enabled by default.  If you need to maintain
 a script's compatibility across different versions of zsh, then a good
 idea is to _now_ add an explicit "enable stat" after "zmodload
 zsh/stat".  In this and older releases of zsh, the explicit enable is
 harmless.  In the next major version, it will allow the script to work
 unmodified.

 If you only need compatibility with version 3.1.6 or newer, then you
 can move to zstat now and avoid introducing stat unnecessarily when the
 zsh is recent enough by using this code:

   zmodload -i zsh/parameter zsh/stat
   if [[ $builtins[stat] == defined && $builtins[zstat] != defined ]]
   then
     function zstat { builtin stat "$@" }
   fi

As an aside: I explicitly check against 'defined' instead of non-zero
since this is more resilient to *cough* people who put:
  zmodload -ab -i zsh/stat stat zstat
in a startup file unguarded by version numbers as it's "harmless".
*whistles innocently*

With documentation like this, and a phased approach to any change, I
think that it'd be fair to say that zsh still takes backwards
compatibility somewhat more seriously than some other shells I could
name.

I'm not sure how to word "next major release" to cleanly convey "the
stable release directly after, not counting patch-levels, the stable
release in which this was introduced (and if you're using zsh 4.3.x then
note that this is unstable, so we're giving you more time)".  The only
way would be to commit to explicit release version numbers.  zsh 4.4
adds zstat, 4.6 disables stat by default, 4.8 removes stat?  Or does
anyone have thoughts about a zsh 5?

Personally, I think that multibyte support is significant enough to
warrant 5.0 following 4.3.x.  At times after the EU added the Euro and I
switched to UTF-8, I found missing multibyte support a reason to do some
things in other shells.  (Why use a rarely found Latin 9 when you can
bite the bullet and switch to something more global?)

-Phil

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Phil Pennock wrote:
> Situation with my patch:
>   If zsh/stat not loaded and user dislikes builtin stat:
>     manual-page documents work-around (same)

I think I see what you mean now... we can simply always leave stat
disabled if it was, because we never actually use it any more.  So the
only issue is what happens at the point a module first loads zsh/stat.
I presume you're suggesting "zmodload zsh/stat; disable stat" at start-up
is the way to fix this.

I'd still prefer a general solution along the lines of the feature
mechanism, which requires (as far as I can see) no kludging or removal
of anything, but I have had absolutely no response to that.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: [PATCH] Proposal: stat -> zstat

[Phil Pennock]

On 2007-05-06 at 12:57 +0100, Peter Stephenson wrote:
> I'd still prefer a general solution along the lines of the feature
> mechanism, which requires (as far as I can see) no kludging or removal
> of anything, but I have had absolutely no response to that.

Well, I like it.  FWIW.

Sorry, kept quiet because I couldn't see anything wrong with the idea
and I wasn't volunteering to do the coding.  ;-)

Oh, and because I've been contaminated by Perl so I'm use to:
 use Foo::bar qw/:DEFAULT :wibble -froz/;
so your proposal fits neatly with my mental models and is actually
simpler, since it doesn't have tagged groups of things to
enable/disable.  Except insofar as the features-named-with-conventions
would let you do just that.

Is it worth adding to the spec that the features will be processed
left-to-right, so that interactions between features will work?  Eg,
disable all except certain aspects?  Most modules will never use this,
thankfully.

zmodload -F zsh/pcre -:builtins :condops
zmodload -F zsh/files :files -mv :fs

(hypothetically putting "sync" in an ":fs" category)

And yes, the : as a part of the name instead of a separator is
distinctly un-shell-like, I use it here only so that those similarly
afflicted will understand what I mean.

On the other hand, arguing devil's advocate, what would the feature
infrastructure provide that's not already available via zsh/parameters
and use of enable/disable?

Perhaps, evil thought, if feature-set changes could be tied to a
function in the same way as localoptions lets you emulate ... then this
would definitely provide cleaner start-up and functions could then just
explicitly declare their dependencies .... how much overhead would there
be for this?  I'm not sure how much complexity this introduces for
little gain.

</brain-dump><sleep>

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Phil Pennock wrote:
> Is it worth adding to the spec that the features will be processed
> left-to-right, so that interactions between features will work?  Eg,
> disable all except certain aspects?  Most modules will never use this,
> thankfully.

For now, features are likely to mean individual features rather than
groups, but it certainly makes sense to specify they'll be parsed left
to right at the outset.

> zmodload -F zsh/pcre -:builtins :condops
> zmodload -F zsh/files :files -mv :fs
> 
> (hypothetically putting "sync" in an ":fs" category)
> 
> And yes, the : as a part of the name instead of a separator is
> distinctly un-shell-like, I use it here only so that those similarly
> afflicted will understand what I mean.

I was actually thinking of using ":" but in a different way: "b:" for a
builtin, "p:" for a shell parameter, "c:" for a condition, "m:" for a
math function, so you'd do

zmodload -Fi zsh/stat b:zstat

in the case in question.  The main reason for this is to disambiguate
the different namespaces.  It would be natural to allow

zmodload -mFi zsh/stat 'b:*'

which is very similar to existing syntax.  So I suppose that means we
could ensure that something like

zmodload -mFi zsh/stat 'b:*' -b:stat

would work (load all builtins but disable "stat").  However, explicitly
disabling things gets us back in the mess we've just climbed out
of---each chunk of code knows what it needs, and should be able to
request only those features, but doesn't know what the rest of the shell
needs or doesn't need, so should just leave those alone.  (Unless we can
come up with a rule where we take the net effect of a set of patterns
and then apply them, so that b:stat ends up being left alone.  I think
that's going to be too complicated.)

> On the other hand, arguing devil's advocate, what would the feature
> infrastructure provide that's not already available via zsh/parameters
> and use of enable/disable?

Natural integration of shell code using different features.  The disable
business, as you've already spotted, is not a very pleasant way of
getting rid of a bit you don't want, since someone else may want it
and not expect it to be disabled.  Explicit feature requests act
separately and fit together without any special action.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: [PATCH] Proposal: stat -> zstat

[Bart Schaefer]

On May 3,  4:46pm, Peter Stephenson wrote:
} Subject: Re: [PATCH] Proposal: stat -> zstat
}
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Perhaps what we should be thinking of is a more general utility
} 
} How about the following.  This tries quite hard not to break anything.

This sounds very much like what I was thinking.  Some comments ...
 
} Each module can advertise a set of features it supports. These will
} be variables, builtins, conditions or math functions; we'll need a
} convention for how to disambiguate. I wouldn't propose to build the
} convention into the API, however, just represent each feature by a
} string

Do I correctly understand that the naming convention has nothing to do
with what happens when one asks that feature X of module Z be loaded or
enabled?  Your suggested -F option implies as much, but presently there
are different flags for builtins/conditions/parameters/mathfuncs.

I guess the real question is, is there any explicit interaction between
-ab/-ac/-af/-ap and -F?  Could something bad happen if I ask for a
parameter feature but say it should be loaded as a builtin?

Conversely (sort of), can features be collections of several builtins/
conditions/etc.?  Hypothetically, if I ask zmodload for the "bash-regex"
feature, do I get both the =~ condtional and the BASH_REMATCH parameter?
(I know, those aren't directly linked in the current implementation, but
I needed an example.)

} The set of enabled features can be an arbitrary-length array of
} unsigned integers with bits giving the enabled status. We can
} explicitly limit ourselves to the first 32 bits of each integer for
} future proofing. Obviously we need one bit for each feature array
} element.

I'm not following how this maps to the individual modules. Is there
one integer in the array per module, so each module is limited to 32
features?  

In any event this is an implementation detail and I'm more interested
in the shell commands.

} zmodload <module>
}   loads everything (as at present).  If the module is already loaded, and
}   supports features, zmodload will check to see if all features are loaded.
}   If they are, it's an error.  If not, it will enable them all.

I've always wondered why zmodload doesn't simply behave by default as
if the -i option were provided.  What's useful about generating an
error if the module is already loaded?

} zmodload -F <module> feature1 feature2
}    will load the module or cause an error if it's already loaded,
}    and turn on feature1 and feature2 (only---all others are disabled).

How does this interact with autoloading?  Can features (or the user,
with zmodload -d) declare that they depend on other features (of the
same or other modules)?  What happens if you try to turn off a feature
that some other feature depends on?

} zmodload -iF <module> -feature1 feature3
}    will load the module if it's not already loaded, turn off feature1
}    and turn on feature3, leaving all other features alone.

I suggest that this work like options, where a "no" prefix disables the
feature.  (Hyphens and underscores ignored like options, too.)

I'm not strongly opposed to using a leading "-" but if we do I think a
leading "+" should also be allowed.  Also, what about the -u option?
Does "zmodload -uF ..." mean anything?

That about covers it ...

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> } Each module can advertise a set of features it supports. These will
> } be variables, builtins, conditions or math functions; we'll need a
> } convention for how to disambiguate. I wouldn't propose to build the
> } convention into the API, however, just represent each feature by a
> } string
>
> Do I correctly understand that the naming convention has nothing to do
> with what happens when one asks that feature X of module Z be loaded
> or enabled?  Your suggested -F option implies as much, but presently
> there are different flags for
> builtins/conditions/parameters/mathfuncs.

In my reply to Phil I suggested using a convention like b:zstat,
p:functions, f:sin, c:-pcre-match, etc., to get around namespace problems.
It's quite possible we don't have clashes at the moment, but it's entirely
conceivable a module may export both a builtin and a parameter with the
same name.  I think ":" is a good choice here---doesn't need quoting, used
in a similar way for styles---but I'd be happy to entertain alternatives.
I said m: for math functions before, but f: makes more sense
given the next point below.

This ought to be transparent to the main shell; it just passes down a
feature name to be handled by the module...

> I guess the real question is, is there any explicit interaction
> between -ab/-ac/-af/-ap and -F?  Could something bad happen if I ask
> for a parameter feature but say it should be loaded as a builtin?

...however, this is an interesting point.  It does seem a pointless extra
complexity to keep this separate from the feature stuff.

Suppose:

zmodload -ab zsh/nosh fishfingers

is defined to be equivalent to

zmodload -Fa zsh/nosh b:fishfingers

and we document that the the second form is preferred?  In other words, the
-a addition to the feature mechanism *does* require the name to be in the
form <codechar>:<featurename>, and generates an autoload request of the
appropriate type (or an error if it doesn't recognise the prefix).  This
uses the same codes, but as part of the feature name rather than an
additional option.

One extra tweak at least is needed: if we loaded a module through the new
form of autoload, we only enable any features explicitly requested in that
list.  For compatibility the old form would fully load the module (it may
well be that a function assumes that autoloading one builtin will bring in a
whole load of others).  So I think we need a list separate from the
autoload stubs put into the relevant hash table (that's just an
implementation detail; we could alternatively search for autoload stubs
when we load the module without enabling all features).

Presumably

zmodload -ab zsh/nosh b:fishfingers

should generate an error for sanity checking; since this would be
essentially a legacy interface it would probably be OK to veto any colon
in the "feature" part.

> Conversely (sort of), can features be collections of several builtins/
> conditions/etc.?  Hypothetically, if I ask zmodload for the
> "bash-regex" feature, do I get both the =~ condtional and the
> BASH_REMATCH parameter? (I know, those aren't directly linked in the
> current implementation, but I needed an example.)

Hypothetically, I think this is OK and the documentation for the module can
make clear what is actually going on; by not using the <codechar>:
prefix it should also be clear in the naming that this is an abstract
feature rather than a specific shell builtin etc.  We could probably
suggest (as a design guideline) that such features don't contain a ":" at
all.

It's then entirely up to the module what it does with such a feature.  It
could even, say, set all the options for a particular emulation.
I would propose to document zmodload to make this clear.

> } The set of enabled features can be an arbitrary-length array of
> } unsigned integers with bits giving the enabled status. We can
> } explicitly limit ourselves to the first 32 bits of each integer for
> } future proofing. Obviously we need one bit for each feature array
> } element.
> 
> I'm not following how this maps to the individual modules. Is there
> one integer in the array per module, so each module is limited to 32
> features?  

No, that's why I said "an arbitrary-length array" of integers.  However,
I'm coming round to the view that it's much simpler to have just an
array the same length as the feature list; the additional memory
requirement isn't so great when we have strings floating around anyway.

Nonetheless, if anyone can think of a neat way of not having to pass up and
down every single feature on every call that would be interesting.
I don't want to make the API too complex, however.

> } zmodload <module>
> }   loads everything (as at present).  If the module is already
> loaded, and }   supports features, zmodload will check to see if all
> features are loaded. }   If they are, it's an error.  If not, it will
> enable them all.
> 
> I've always wondered why zmodload doesn't simply behave by default as
> if the -i option were provided.  What's useful about generating an
> error if the module is already loaded?

That's a good question I can't answer.  I've simply kept this because it's
always behaved that way, but I can't think of a good reason to use it.
Unless anyone can see one, I'd be perfectly happy to make -i the default.
There's no point adding an option in the opposite sense, since if you have
to add new code to check anyway you might as well use zmodload -e.

> } zmodload -F <module> feature1 feature2
> }    will load the module or cause an error if it's already loaded,
> }    and turn on feature1 and feature2 (only---all others are
> disabled).
> 
> How does this interact with autoloading?>

Basic suggestion above, but...

> Can features (or the user,
> with zmodload -d) declare that they depend on other features (of the
> same or other modules)?  

Urk.  The current dependency mechanism is per-module and if we're going to
keep it then to do it properly we need it to be per-feature.
If each *feature* in the target module can depend on a feature in another
module it gets very hairy even to specify.  I'm not sure we want to go down
that road immediately, but something like

zsh -dF target-module feature1... -- dependency-module featureN+1...

could do it.

Although this is a significant question, I think we could implement module
features without this initially; at the moment the dependencies used by the
shell are at a fairly gross level and I don't think any of the supplied
functions specify dependencies.  Typically you need all of a support module
(e.g. zsh/zftp and zsh/net/tcp).

> What happens if you try to turn off a feature
> that some other feature depends on?

If we do something like the above, we can keep the specified line as a
database (instead of dumping the info when the target module has been
loaded, as at present).  Then, as long as we get the internal structures
right, it ought to be easy to check; e.g.

% zmodload -F dependency-module -featureN-1
zsh: zmodload: featureN-1 of dependency-module is required by target-module

We may need to improve listing of dependencies, though the basics for there
are that (running "zmodload -d target-module" will explain the error
message above).  We already have -u to remove dependencies and that can be
added to the syntax suggested above.

(Unnecessarily detailed point of implementation: by implication, the
features are defined by there position in an array returned by the module.
So we can use that index internally to refer to the feature.  This would
make it useful to have completely separate calls to the module for features
(which are read-only) and feature-enables.

  char **features_(Module m);

Returns the NULL-terminated array of features.  I've a vague feeling it
would be beneficial to have this zalloc'd for the reasons given below.

  int *feature_enables_(Module m, int *intarrptr);

If intarrptr is non-NULL, set the features from the array (*intarrptr).  In
any case, return a point to the array of current feature enables; if
intarrptr was passed, this is returned, else zalloc'd memory.

This API doesn't impose the burden on the module of a permanent array of
feature enables in the expanded form, one integer per feature.  That might
be a significant saving for a module like zsh/mathfunc with a lot of
builtins.  With a query/set, we get the benefit that the array is only
allocated once.

> } zmodload -iF <module> -feature1 feature3
> }    will load the module if it's not already loaded, turn off
> feature1 }    and turn on feature3, leaving all other features alone.
>
> I suggest that this work like options, where a "no" prefix disables
> the feature.  (Hyphens and underscores ignored like options, too.)
>
> I'm not strongly opposed to using a leading "-" but if we do I think a
> leading "+" should also be allowed.

I'm perfectly happy with allowing +.  I'm not that keen on "no" for general
feature use because of the "notify"/"nomatch" problem: we need to test
everything twice just in case.  (I don't like limitations on feature names
based on details of the English language, either.)

Ignoring punctuation is also a bit problematic in the case where the
feature maps directly to a builtin etc.; underscores are natural in builtin
names and "-" is a natural part of a condition.

I can certainly see why you'd want name munging for abstract/collective
features.  However, leaving this up the module is a bit fraught since the
main shell needs to be able to do the comparison to manipulate the feature
enables (unless we handle the feature list parsing at a lower level, which
I don't like).  So I don't see any easy way to get this unless we have some
convention for names of certain types.  Hmm... if the feature enables is
now a full integer, perhaps we could define a bit in it to say "supports
name munging"?

> Also, what about the -u option?
> Does "zmodload -uF ..." mean anything?

I don't think so (except as described for dependencies).  It doesn't make
sense to unload a module unless you're turning off all features, and we
don't need a special syntax to say that's what's happening when you do
unload a module.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

Re: [PATCH] Proposal: stat -> zstat

[Bart Schaefer]

On May 8,  3:40pm, Peter Stephenson wrote:
}
} In my reply to Phil I suggested using a convention like b:zstat,
} p:functions, f:sin, c:-pcre-match, etc., to get around namespace
} problems. It's quite possible we don't have clashes at the moment,
} but it's entirely conceivable a module may export both a builtin
} and a parameter with the same name. I think ":" is a good choice
} here---doesn't need quoting, used in a similar way for styles---but
} I'd be happy to entertain alternatives.

I agree that ":" is a good choice; "." is the other obvious possibility,
but we probably want to reserve that for ksh-style parameter namespaces.

} This ought to be transparent to the main shell; it just passes down a
} feature name to be handled by the module...

Right.

} Suppose:
} 
} zmodload -ab zsh/nosh fishfingers
} 
} is defined to be equivalent to
} 
} zmodload -Fa zsh/nosh b:fishfingers

This makes perfect sense.

} and we document that the the second form is preferred? In other words,
} the -a addition to the feature mechanism *does* require the name to
} be in the form <codechar>:<featurename>, and generates an autoload
} request of the appropriate type (or an error if it doesn't recognise
} the prefix).

This is also related to the question of whether a module feature can
represent a collection of things.  Could

  zmodload -Fa zsh/nosh turtleteeth

mean e.g. "autoload anything related to turtleteeth"?  I suppose that
would mean loading zsh/nosh immediately, but with everything disabled,
in order to find out what is contained in that feature.  (And then
unloading it again?  Hrm.)  Or else abstract features are in their
own category, but then how do you know when to autoload them?  For the
first pass, or even the second, perhaps abstract features should not
be auto-loadable.

Which brings up a point:  Previously modules were either all there or
all not, except insofar as certain bits could be hidden by explicitly
using "disable" after loading.  With the features change we can get
into the condition of the module being present, but invisible.

} One extra tweak at least is needed: if we loaded a module through the
} new form of autoload, we only enable any features explicitly requested
} in that list.  For compatibility the old form would fully load the
} module (it may well be that a function assumes that autoloading one
} builtin will bring in a whole load of others).

What happens if both -Fa b:turtleteeth and -ab turtleteeth have been
executed prior to the first attempt to find turtleteeth?  The easy way
out is to do the maximal load that has been previously requested.

Random possibly unrelated thought:
We might want to think ahead to the possibilty of module scopes (like
the local parameter scope).

} Presumably
} 
} zmodload -ab zsh/nosh b:fishfingers
} 
} should generate an error

Since ":" isn't valid in an identifier I think that's OK.  (Have we yet
discussed whether symbolic conditionals like "=~" could be autoloaded?)

} > Can features (or the user, with zmodload -d) declare that they
} > depend on other features (of the same or other modules)?
} 
} Urk. The current dependency mechanism is per-module and if we're going
} to keep it then to do it properly we need it to be per-feature. If
} each *feature* in the target module can depend on a feature in another
} module it gets very hairy even to specify.

I'm actually most concerned with the C definition of the module being
able to specify dependencies, rather than the user being able to do so
with some tangled zmodload -d syntax.

} Although this is a significant question, I think we could implement
} module features without this initially

Yes.

Re: [PATCH] Proposal: stat -> zstat

[Peter Stephenson]

Bart Schaefer wrote:
> This is also related to the question of whether a module feature can
> represent a collection of things.  Could
> 
>   zmodload -Fa zsh/nosh turtleteeth
> 
> mean e.g. "autoload anything related to turtleteeth"?  I suppose that
> would mean loading zsh/nosh immediately, but with everything disabled,
> in order to find out what is contained in that feature.  (And then
> unloading it again?  Hrm.)  Or else abstract features are in their
> own category, but then how do you know when to autoload them?  For the
> first pass, or even the second, perhaps abstract features should not
> be auto-loadable.

That last suggestion is likely to be the case.  It's hard to think of a
general mechanism where the shell would know what autoloading an abtract
feature means.  It may be best done (a little like Emacs' hooks) with
a shell function that simply sets up regular autoloads for any concrete
features.

> Which brings up a point:  Previously modules were either all there or
> all not, except insofar as certain bits could be hidden by explicitly
> using "disable" after loading.  With the features change we can get
> into the condition of the module being present, but invisible.

Yes, but there still there to zmodload, so it's detectable.  An option
to unload automatically when all features are disabled isn't difficult,
I don't think; it can be done locally in the main shell's feature
handler.  This should be transparent to future feature requests, and
easy to add later on.

> What happens if both -Fa b:turtleteeth and -ab turtleteeth have been
> executed prior to the first attempt to find turtleteeth?  The easy way
> out is to do the maximal load that has been previously requested.

Yes, that shouldn't be too hard.

> Random possibly unrelated thought:
> We might want to think ahead to the possibilty of module scopes (like
> the local parameter scope).

It's quite hairy to define what the scope is, and also pretty hairy to
implement it since builtins, math functions and conditions don't have a
scope at the moment.

> } Presumably
> } 
> } zmodload -ab zsh/nosh b:fishfingers
> } 
> } should generate an error
> 
> Since ":" isn't valid in an identifier I think that's OK.  (Have we yet
> discussed whether symbolic conditionals like "=~" could be autoloaded?)

That needs some tweaking in the main shell; I vaguely feel leaving
them in the form of -special-tests and not polluting the normal
shell syntax might be preferable.

> I'm actually most concerned with the C definition of the module being
> able to specify dependencies, rather than the user being able to do so
> with some tangled zmodload -d syntax.

Internally this is similar, but deciding what the dependencies actually
are is probably messier in general; since, as I said, we typically
need all of a service module for the next level up this doesn't seem
to arise yet.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

To get further information regarding CSR, please visit our Investor Relations page at http://ir.csr.com/csr/about/overview

4.3.17 unset RPS1 vs RPROMPT

[Phil Pennock]

zsh 4.3.17, multiple OSes.

Using "prompt walters", I get a green right-prompt.
Using "prompt pdp" to invoke my own setup, I still have it.
I debug.  I have "unset RPS1" in my setup function.
Strange, try at command-line.

 unset RPS1       # no effect
 unset RPROMPT    # works

I see Src/params.c:
IPDEF7("RPS1", &rprompt),
IPDEF7("RPROMPT", &rprompt),

and I know that the docs claim them equivalent.

So try with "zsh -f", and it works.  Some debugging later, and I find
that if I "unset RPS1" then "typeset -p RPS1 RPROMPT" shows just
RPROMPT, not both.  If I then call "prompt walters", then RPROMPT will
be set and can only be cleared with "unset RPROMPT".

If I also "unset RPROMPT", then it comes back into being as a magic
variable during prompt setup.  If I assign to RPS1, then it too comes
back as a magic variable bound to the same internal rprompt.

Is it expected that RPROMPT and RPS1 can be decoupled in this way?
Is there a correct way to clear the prompt, other than *always* having
to do "unset RPROMPT RPS1" because one or the other may have become
decoupled?

I don't see it documented and it seems very strange to me.  I'd expect
that if one is unset, the rprompt content gets unset, but assigning to
either would instantiate the other again, since it can't be reused by a
user anyway -- if they unset and then assign to a new value, it still
changes the same C level "rprompt" variable.

Thoughts?
-Phil

Re: 4.3.17 unset RPS1 vs RPROMPT

[Mikael Magnusson]

On 29/06/2012, Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> zsh 4.3.17, multiple OSes.
>
> Using "prompt walters", I get a green right-prompt.
> Using "prompt pdp" to invoke my own setup, I still have it.
> I debug.  I have "unset RPS1" in my setup function.
> Strange, try at command-line.
>
>  unset RPS1       # no effect
>  unset RPROMPT    # works
>
> I see Src/params.c:
> IPDEF7("RPS1", &rprompt),
> IPDEF7("RPROMPT", &rprompt),
>
> and I know that the docs claim them equivalent.
>
> So try with "zsh -f", and it works.  Some debugging later, and I find
> that if I "unset RPS1" then "typeset -p RPS1 RPROMPT" shows just
> RPROMPT, not both.  If I then call "prompt walters", then RPROMPT will
> be set and can only be cleared with "unset RPROMPT".
>
> If I also "unset RPROMPT", then it comes back into being as a magic
> variable during prompt setup.  If I assign to RPS1, then it too comes
> back as a magic variable bound to the same internal rprompt.
>
> Is it expected that RPROMPT and RPS1 can be decoupled in this way?
> Is there a correct way to clear the prompt, other than *always* having
> to do "unset RPROMPT RPS1" because one or the other may have become
> decoupled?
>
> I don't see it documented and it seems very strange to me.  I'd expect
> that if one is unset, the rprompt content gets unset, but assigning to
> either would instantiate the other again, since it can't be reused by a
> user anyway -- if they unset and then assign to a new value, it still
> changes the same C level "rprompt" variable.
>
> Thoughts?

Does it help if you remove the 'export's in prompt_walters_setup?

-- 
Mikael Magnusson

Re: 4.3.17 unset RPS1 vs RPROMPT

[Phil Pennock]

On 2012-06-29 at 12:02 +0200, Mikael Magnusson wrote:
> On 29/06/2012, Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> > Is it expected that RPROMPT and RPS1 can be decoupled in this way?
> > Is there a correct way to clear the prompt, other than *always* having
> > to do "unset RPROMPT RPS1" because one or the other may have become
> > decoupled?

> Does it help if you remove the 'export's in prompt_walters_setup?

No.

-Phil

Re: 4.3.17 unset RPS1 vs RPROMPT

[Bart Schaefer]

On Jun 29,  2:40am, Phil Pennock wrote:
}
} Is it expected that RPROMPT and RPS1 can be decoupled in this way?

I suspect that's an unexpected side-effect from a change made quite
some time ago to limit the "special"-ness of assorted variables, for
compatibility with emulated shells for which those variables don't
have a pre-existing definition.

} Is there a correct way to clear the prompt, other than *always* having
} to do "unset RPROMPT RPS1" because one or the other may have become
} decoupled?

Assign the empty string to either one of them instead?

The way to fix this would be to create a new GSU structure that calls a
set of function to link them, and change their IPDEF* declaration in
Src/params.c to use that GSU, probably by moving them from IPDEF7 to
IPDEF5 (now that IPDEF5 has been repaired to work correctly).

(For those wondering, IPDEF6 was a single-use macro for PWD which was
removed because PWD is no longer a special variable at all.)

Re: 4.3.17 unset RPS1 vs RPROMPT

[Peter Stephenson]

On Fri, 29 Jun 2012 02:40:46 -0700
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> Using "prompt walters", I get a green right-prompt.
> Using "prompt pdp" to invoke my own setup, I still have it.
> I debug.  I have "unset RPS1" in my setup function.
> Strange, try at command-line.
> 
>  unset RPS1       # no effect
>  unset RPROMPT    # works

I'm still not really following what you're trying to do, but my guess is
you're not actually trying to unset the variable at all, you're trying
to set it to an empty string.  They may sound similar, and they often
have the same effect, but they are different things.  If the variable is
unset, it is indeed uncoupled from anything else that's going on, which
is kind of what unset means.  So unsetting an unset variable does indeed
have no effect.

As far as I can see the prompt functions never try to unset the prompts.
I'd suggest you use "RPS1=" to make the prompt empty.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: 4.3.17 unset RPS1 vs RPROMPT

[Phil Pennock]

On 2012-06-29 at 19:52 +0100, Peter Stephenson wrote:
> I'm still not really following what you're trying to do, but my guess is
> you're not actually trying to unset the variable at all, you're trying
> to set it to an empty string.  They may sound similar, and they often
> have the same effect, but they are different things.  If the variable is
> unset, it is indeed uncoupled from anything else that's going on, which
> is kind of what unset means.  So unsetting an unset variable does indeed
> have no effect.

I'd decided it was a bug because that's what unset means, but setting it
again rebinds it to have the same meaning as before, so it can't be
meaningfully unset.  You can't rebind the variable name to a different
internal structure, so at this point the semantics become somewhat
messed up.

I think that there's a bug here, but deciding what the correct behaviour
should be is non-trivial and certainly not worth blocking 5.0.0 for.

Fundamentally, in zsh at present, it seems that magic variables can not
have their magic purged from them as the names can never truly be
unbound, just temporarily made unavailable, yet the same content can be
available under other names.  For library-level robustness against
someone having set a prompt variable (PROMPT3/PS3 for instance) but
unbound one of the names, getting back consistent state seems to require
setting both names and only caring about the value of which one is set
second, so that both names remain available thereafter.

If the name could be rebound to another variable and no longer be magic,
the current approach would be sane, but it seems that perhaps the
correct behaviour should be to ensure that whatever the state of one
name, the other name should be in the same state: set, unset, empty,
with-content, whatever.

Perhaps I've become contaminated with the Pythonic approach of being
distinct about the binding of a name to a variable versus the changing
of the value of a variable.

-Phil

Re: 4.3.17 unset RPS1 vs RPROMPT

[Bart Schaefer]

On Jun 29, 10:05pm, Phil Pennock wrote:
}
} If the name could be rebound to another variable and no longer be magic,
} the current approach would be sane, but it seems that perhaps the
} correct behaviour should be to ensure that whatever the state of one
} name, the other name should be in the same state: set, unset, empty,
} with-content, whatever.

There's some precedent for this; although $#, $@ and $* are read-only
variables, if you "unset argv" you erase $* and $@, and $# becomes 0.
On the other hand, argv has its own weird semantics (see below) which
probably exempts it from this analogy.

} Perhaps I've become contaminated with the Pythonic approach of being
} distinct about the binding of a name to a variable versus the changing
} of the value of a variable.

I think this comes down to a scoping issue.

Shell variables have a form of dynamic scoping that's somewhat different
from that of any other interpreted language.  In the first shells, all
variables were environment variables.  Then shell variables became a
separate thing and the idea of exporting them into the environment was
introduced, but there was still only one global scope.  Then local scopes
were introduced, but to retain the semantics of the former global scope,
the local scopes became dynamic, based on runtime call heirarchy rather
than code structure.

Once a shell variable has a scope, unsetting the variable does not delete
it from that scope.  If you re-use the name, it returns in the same scope
where it was originally defined, unless you are declaring it in a new
dynamic scope.  You can't change the magic binding in the original scope;
that's what it means for the variable to be "special".

But you can rebind them in any scope other than the global one, and thus
in all dynamically nested scopes, with e.g. "local -h RPROMPT".

Weird semantics:  Although argv is seems implicitly to be local in all
scopes, unsetting without first actually declaring it local causes it
to disappear in all scopes, which breaks the connection to $* et al.

torch% showargv() { print -l "argv: $argv" "*: $*" }
torch% noargv() { print -l "argv: $argv"; unset argv; print "*: $*" }
torch% showargv one two three
argv: one two three
*: one two three
torch% noargv four five six
argv: four five six
*: 
torch% showargv seven eight nine
argv: 
*: seven eight nine
torch% noargv gone gone gone
argv: 
*: gone gone gone
torch% 

Resetting it brings it back with full magic intact:

torch% argv=()
torch% showargv seven eight nine
argv: seven eight nine
*: seven eight nine
torch% 

Of course I say "seems" to be local in all scopes because really it's
the special binding to $@ and $* that makes it local; the name itself
and its properties are still defined in the global scope.

Re: 4.3.17 unset RPS1 vs RPROMPT

[Peter Stephenson]

Phil Pennock wrote:
> Fundamentally, in zsh at present, it seems that magic variables can not
> have their magic purged from them as the names can never truly be
> unbound, just temporarily made unavailable, yet the same content can be
> available under other names.

Right, this is one fundamental issue; it's always been that way.

It's not 100% clear that if someone running "unset RSP1 RPROMPT" has
really got it in mind that they never want to set a right prompt again,
though, particularly if they're used to how the shell currently works.

It would probably be neatest to make a distinction as an option either
to unset or to the shell (the latter making it easy to provide
compatibility with other shells).

Not new, so can wait for now...

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/