Creating A Clean Environment For Autoloaded Functions

Creating A Clean Environment For Autoloaded Functions

[Russell Harmon]

If I want to write a function which can be autoloaded, how can I
prevent functions defined outside of my function from being accessible
from within my function? For example, I have a function ls { gls "$@"
}, and I know of no way to prevent that function from leaking into the
definition of all the functions I've autoloaded which use ls.

Additionally, is it possible to zmodload a module which is
automatically unloaded from the environment after my function
completes _only if_ that module was not already loaded?
--
Russell Harmon

Re: Creating A Clean Environment For Autoloaded Functions

[Bart Schaefer]

On Dec 30,  4:44am, Russell Harmon wrote:
>
> If I want to write a function which can be autoloaded, how can I
> prevent functions defined outside of my function from being accessible
> from within my function? For example, I have a function ls { gls "$@"
> }, and I know of no way to prevent that function from leaking into the
> definition of all the functions I've autoloaded which use ls.

This amounts to asking how to violate a basic tenet of shell function
behavior.  They're supposed to act like commands that are always at the
front of the search path, and the search path doesn't change based on
the order in which every other command was added to it.

The way you avoid this is to write every function that explicitly does
not want this behavior so as to explicitly override it; for examply by
using "command ls" instead of "ls".

However, you could do something like this, assuming zsh/parameter is
loaded:

    ls() {
      if (( $#funcstack > 1 ))
      then command ls "$@"
      else gls "$@"
      fi
    }

This will prevent gls from being run inside any other function, although
it won't bypass the ls wrapper function entirely.

> Additionally, is it possible to zmodload a module which is
> automatically unloaded from the environment after my function
> completes _only if_ that module was not already loaded?

There isn't a property of modules that supports this, but you can do it
yourself explicitly with something like this:

    {
      zmodload -e zsh/mathfunc	# for example
      local unload_mathfunc=$?
      zmodload zsh/mathfunc

      : do what you need to with math functions

    } always {
      (( unload_mathfunc )) && zmodload -u zsh/mathfunc
    }

If the zsh/parameter module is not itself one of the ones you care about
unloading in this way, you can be more generic:

    {
      zmodload zsh/parameter
      local -a existing_modules
      existing_modules=( ${(k)modules[(R)loaded]} ) 

      : do whatever zmodloads you want ...

    } always {
      # Requires zsh 5.0, otherwise you need a loop
      zmodload -u ${(k)modules[(R)loaded]:|existing_modules}
    }

I'm not sure that zmodload is clever enough to notice that inter-module
dependencies have been satisfied by the list provided to -u and thus
unload the modules in the correct order to be sure it succeeds, so a
bit of tweaking to the above may be required for full generality.

Re: Creating A Clean Environment For Autoloaded Functions

[Russell Harmon]

On Sun, Dec 30, 2012 at 2:20 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Dec 30,  4:44am, Russell Harmon wrote:
>>
>> If I want to write a function which can be autoloaded, how can I
>> prevent functions defined outside of my function from being accessible
>> from within my function? For example, I have a function ls { gls "$@"
>> }, and I know of no way to prevent that function from leaking into the
>> definition of all the functions I've autoloaded which use ls.
>
> This amounts to asking how to violate a basic tenet of shell function
> behavior.  They're supposed to act like commands that are always at the
> front of the search path, and the search path doesn't change based on
> the order in which every other command was added to it.
>
> The way you avoid this is to write every function that explicitly does
> not want this behavior so as to explicitly override it; for examply by
> using "command ls" instead of "ls".
>
> However, you could do something like this, assuming zsh/parameter is
> loaded:
>
>     ls() {
>       if (( $#funcstack > 1 ))
>       then command ls "$@"
>       else gls "$@"
>       fi
>     }
>
> This will prevent gls from being run inside any other function, although
> it won't bypass the ls wrapper function entirely.

I guess that breaking with this particular shell idiom is what I want
to do. Otherwise, how is one supposed to be able to create
autoloadable functions with reliable and reproducible behavior? I'm
thinking to add an option to autoload to undefine all functions much
like how -U works with aliases.

Basically, I'm trying to follow what the best practices are when
writing autoloadable functions so that they will be able to run
reliably in the largest number of different environments possible.

>
>> Additionally, is it possible to zmodload a module which is
>> automatically unloaded from the environment after my function
>> completes _only if_ that module was not already loaded?
>
> There isn't a property of modules that supports this, but you can do it
> yourself explicitly with something like this:
>
>     {
>       zmodload -e zsh/mathfunc  # for example
>       local unload_mathfunc=$?
>       zmodload zsh/mathfunc
>
>       : do what you need to with math functions
>
>     } always {
>       (( unload_mathfunc )) && zmodload -u zsh/mathfunc
>     }
>
> If the zsh/parameter module is not itself one of the ones you care about
> unloading in this way, you can be more generic:
>
>     {
>       zmodload zsh/parameter
>       local -a existing_modules
>       existing_modules=( ${(k)modules[(R)loaded]} )
>
>       : do whatever zmodloads you want ...
>
>     } always {
>       # Requires zsh 5.0, otherwise you need a loop
>       zmodload -u ${(k)modules[(R)loaded]:|existing_modules}
>     }
>
> I'm not sure that zmodload is clever enough to notice that inter-module
> dependencies have been satisfied by the list provided to -u and thus
> unload the modules in the correct order to be sure it succeeds, so a
> bit of tweaking to the above may be required for full generality.

Hmm, I think I'm realizing that loading & unloading of dependent
modules every time a function is called might not be exactly what I
want. I think what I'm looking for is more like zsh modules where you
have hook functions for when the function is loaded & unloaded so that
you can unload relevant modules when no longer needed. That has it's
own difficulties however because it'd be hard to tell if some other
function loaded after you were loaded needs a module which you are
about to unload.

Re: Creating A Clean Environment For Autoloaded Functions

[Bart Schaefer]

On Dec 30,  4:02pm, Russell Harmon wrote:
}
} On Sun, Dec 30, 2012 at 2:20 PM, Bart Schaefer
} <schaefer@brasslantern.com> wrote:
} > This amounts to asking how to violate a basic tenet of shell function
} > behavior.  They're supposed to act like commands that are always at the
} > front of the search path, and the search path doesn't change based on
} > the order in which every other command was added to it.
} 
} I guess that breaking with this particular shell idiom is what I want
} to do. Otherwise, how is one supposed to be able to create
} autoloadable functions with reliable and reproducible behavior?

As I said, you do it like this:

} > The way you avoid this is to write every function that explicitly does
} > not want this behavior so as to explicitly override it; for examply by
} > using "command ls" instead of "ls".

To be completely "reliable and reproducible" you also have to set up
the emulation mode very carefully, force a fixed value of $path, etc.
In practice is this is often more than either you or the caller of your
function actually want.

The alternative is to write scripts rather than autoloaded functions,
which run in their own process and have more complete control of their
context.  Part of the reason for using functions rather than scripts
is so that they operate in the context of the running shell, which
implies that the caller is permitted to screw around with that context.

} I'm thinking to add an option to autoload to undefine all functions
} much like how -U works with aliases.

The problem with that is that -U doesn't actually undefine the aliases,
it just prevents them from being expanded at parse time.  There is no
equivalent parse-time mechanism for distinguishing an external command
from a function.

In zsh (and probably only in zsh of all shells) there actually is a way
to remove all functions if you really want to.

    the_lonely_function() {
      builtin zmodload zsh/parameter
      builtin typeset +h functions
      builtin unfunction -m \*
      #
      # There are now no functions but this function,
      # and even this function cannot call itself.
      # On return, the local parameter scope will end
      # and other function definitions will reappear.
      #
    }

There are a few obscure gotchas with this; for example, functions that
were previously autoload-able will instead become real functions that
use the "autoload -X" command to bootstrap.  That may not always have
the equivalent effect to the original autoload.
 
} Basically, I'm trying to follow what the best practices are when
} writing autoloadable functions so that they will be able to run
} reliably in the largest number of different environments possible.

I'd have to say that the *usual* practices are to worry about the
emulation mode and setopts, use globbing and parameter expansion
tricks to avoid reliance on external commands as much as possible,
and to presume "autoload -U" has eliminated parse-time surprises.
Beyond that and the judicious application of "builtin" or "command"
prefixes where necessary, you're putting in too much effort for the
incremental value.

Re: Creating A Clean Environment For Autoloaded Functions

[Han Pingtian]

On Sun, Dec 30, 2012 at 11:20:44AM -0800, Bart Schaefer wrote:
>     {
>       zmodload zsh/parameter
>       local -a existing_modules
>       existing_modules=( ${(k)modules[(R)loaded]} ) 
> 
>       : do whatever zmodloads you want ...
> 
>     } always {
>       # Requires zsh 5.0, otherwise you need a loop
>       zmodload -u ${(k)modules[(R)loaded]:|existing_modules}

How could we know that, the "(k)" will be run before ":|" here, please?
Because "k" will be applied with the subscripting, so it will be before
the ":|" ?

Thanks.

>     }

PATCH and more remarks on parameter expansion docs

[Bart Schaefer]

Collecting a few threads about parameter expansion here in order to make
some other remarks.

On Jan 1,  7:30am, Han Pingtian wrote:
} Subject: Re: Creating A Clean Environment For Autoloaded Functions
}
} >       # Requires zsh 5.0, otherwise you need a loop
} >       zmodload -u ${(k)modules[(R)loaded]:|existing_modules}
} 
} How could we know that, the "(k)" will be run before ":|" here, please?
} Because "k" will be applied with the subscripting, so it will be before
} the ":|" ?

That's correct.  I've added some mention of this in the patch below.

On Dec 27,  3:08pm, Han Pingtian wrote:
} Subject: Re: splitting in assignment forms of expansion
}
} I think we should say something like this about this feature in the man
} page:
}
}     Note during assignments with the A flag, that splitting is applied 
}     to word in the assignment forms of spec before the assignment to
}     name is performed. This will affects the result.

I added something similar in the patch.

In looking through the parameter flags to see what else may have been
overlooked in the "rules", I found no mention of the (c) and (t) flags.

The (c) flag does what Han Pingtian was asking about in the thread Re:
priority problem of ":|" and ":*" --

    % x=(a b c)
    % print "${#x}" ${(c)#x}
    % 3 5

That is, (c) joins up the array for length purposes even in places where
it would not normally be joined.  I've expanded the doc for this a little
as well, but I didn't explicitly mention it in the rules section because
it only matters in combination with "#".

The (t) flag is an oddball.  If combined with (P) [rule 4], it takes
effect after that name replacement.  However, if *not* combined with
(P), then it takes effect *before* subscripting [rule 3].  Thus:

    % x=path
    % print ${(t)x}
    scalar
    % print ${x[2,4]}
    ath
    % print ${(t)x[2,4]}
    cal
    % print ${(tP)x}
    array-unique-special

I might have expected ${(t)x[2,4]} to print the type of the variable
named "ath", but that isn't what happens.  So I haven't attempted to
work it into the rules, but it's a perfect example of how parameter
expansion syntax lacks well-defined precedence.


Index: Doc/Zsh/expn.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/expn.yo,v
retrieving revision 1.148
diff -u -r1.148 expn.yo
--- Doc/Zsh/expn.yo     19 Oct 2012 09:19:08 -0000      1.148
+++ Doc/Zsh/expn.yo     2 Jan 2013 04:57:04 -0000
@@ -748,6 +748,8 @@
 the length in characters of the result instead of
 the result itself.  If var(spec) is an array expression,
 substitute the number of elements of the result.
+This has the side-effect that joining is skipped even in quoted
+forms, which may affect other sub-expressions in var(spec).
 Note that `tt(^)', `tt(=)', and `tt(~)', below, must appear
 to the left of `tt(#)' when these forms are combined.
 )
@@ -870,11 +872,12 @@
 Create an array parameter with `tt(${)...tt(=)...tt(})',
 `tt(${)...tt(:=)...tt(})' or `tt(${)...tt(::=)...tt(})'.
 If this flag is repeated (as in `tt(AA)'), create an associative
-array parameter.  Assignment is made before sorting or padding.
-The var(name) part may be a subscripted range for ordinary
-arrays; the var(word) part em(must) be converted to an array, for
-example by using `tt(${(AA)=)var(name)tt(=)...tt(})' to activate
-field splitting, when creating an associative array.
+array parameter.  Assignment is made before sorting or padding;
+if field splitting is active, the var(word) part is split before
+assignment.  The var(name) part may be a subscripted range for
+ordinary arrays; the var(word) part em(must) be converted to
+an array, for example by using `tt(${(AA)=)var(name)tt(=)...tt(})'
+to activate field splitting, when creating an associative array.
 )
 item(tt(a))(
 Sort in array index order; when combined with `tt(O)' sort in reverse
@@ -884,7 +887,9 @@
 )
 item(tt(c))(
 With tt(${#)var(name)tt(}), count the total number of characters in an array,
-as if the elements were concatenated with spaces between them.
+as if the elements were concatenated with spaces between them.  This is not
+a true join of the array, so other expressions used with this flag may have
+an effect on the elements of the array before it is counted.
 )
 item(tt(C))(
 Capitalize the resulting words.  `Words' in this case refers to sequences
@@ -1308,7 +1313,8 @@
 subscript.  Thus if tt(var) is an array, tt(${var[1][2]}) is the second
 character of the first word, but tt(${var[2,4][2]}) is the entire third
 word (the second word of the range of words two through four of the
-original array).  Any number of subscripts may appear.
+original array).  Any number of subscripts may appear.  Flags such as
+tt((k)) and tt((v)) which alter the result of subscripting are applied.
 )
 item(tt(4.) em(Parameter name replacement))(
 The effect of any tt((P)) flag, which treats the value so far as a
@@ -1316,11 +1322,12 @@
 )
 item(tt(5.) em(Double-quoted joining))(
 If the value after this process is an array, and the substitution
-appears in double quotes, and no tt((@)) flag is present at the current
-level, the words of the value are joined with the first character of the
-parameter tt($IFS), by default a space, between each word (single word
-arrays are not modified).  If the tt((j)) flag is present, that is used for
-joining instead of tt($IFS).
+appears in double quotes, and neither an tt((@)) flag nor a tt(#)
+length operator is present at the current level, then words of the
+value are joined with the first character of the parameter tt($IFS),
+by default a space, between each word (single word arrays are not
+modified).  If the tt((j)) flag is present, that is used for joining
+instead of tt($IFS).
 )
 item(tt(6.) em(Nested subscripting))(
 Any remaining subscripts (i.e. of a nested substitution) are evaluated at

Re: PATCH and more remarks on parameter expansion docs

[Han Pingtian]

On Tue, Jan 01, 2013 at 09:15:01PM -0800, Bart Schaefer wrote:
> On Dec 27,  3:08pm, Han Pingtian wrote:
> } Subject: Re: splitting in assignment forms of expansion
> }
> } I think we should say something like this about this feature in the man
> } page:
> }
> }     Note during assignments with the A flag, that splitting is applied 
> }     to word in the assignment forms of spec before the assignment to
> }     name is performed. This will affects the result.
> 
> I added something similar in the patch.
Thanks your nice patch. But I'm still wondering why not change the
statements after "${=spec}" in man page. Or I'm misunderstanding the
original statements which has been stating this "splitting before
assignment" behavior is only related to "(A)" flag? 
> 
> 
> The (t) flag is an oddball.  If combined with (P) [rule 4], it takes
> effect after that name replacement.  However, if *not* combined with
> (P), then it takes effect *before* subscripting [rule 3].  Thus:
> 
>     % x=path
>     % print ${(t)x}
>     scalar
>     % print ${x[2,4]}
>     ath
>     % print ${(t)x[2,4]}
>     cal
>     % print ${(tP)x}
>     array-unique-special
> 
> I might have expected ${(t)x[2,4]} to print the type of the variable
> named "ath", but that isn't what happens.  So I haven't attempted to
> work it into the rules, but it's a perfect example of how parameter
> expansion syntax lacks well-defined precedence.
Yep, it looks like there are quite some special cases. 

Re: PATCH and more remarks on parameter expansion docs

[Bart Schaefer]

On Jan 2,  4:32pm, Han Pingtian wrote:
}
} Thanks your nice patch. But I'm still wondering why not change the
} statements after "${=spec}" in man page. Or I'm misunderstanding the
} original statements which has been stating this "splitting before
} assignment" behavior is only related to "(A)" flag? 

You mean this?

     Note that splitting is applied to WORD in the assignment forms of
     SPEC _before_ the assignment to NAME is performed.  This affects
     the result of array assignments with the A flag.

I must not be understanding what else you think it should say.  Is it
just that it isn't "... the A flag, among other things"?

Re: PATCH and more remarks on parameter expansion docs

[Han Pingtian]

On Wed, Jan 02, 2013 at 08:46:44AM -0800, Bart Schaefer wrote:
> On Jan 2,  4:32pm, Han Pingtian wrote:
> }
> } Thanks your nice patch. But I'm still wondering why not change the
> } statements after "${=spec}" in man page. Or I'm misunderstanding the
> } original statements which has been stating this "splitting before
> } assignment" behavior is only related to "(A)" flag? 
> 
> You mean this?
> 
>      Note that splitting is applied to WORD in the assignment forms of
>      SPEC _before_ the assignment to NAME is performed.  This affects
>      the result of array assignments with the A flag.
Yes, I mean this.

I think it doesn't say this splitting _before_ assignments is only true
when (A) is used. Looks like if (A) wasn't used, splitting will _after_
the assignments:

    % print -l ${=xx::=foo   bar} . $xx . ${(A)=xx::=foo   bar} . $xx
    foo
    bar
    .
    foo   bar
    .
    foo
    bar
    .
    foo
    bar

So I think maybe it should say this splitting before assignments is only
true when (A) is used. The original statements give me the feeling that
this splitting before assignments is peformed in any conditions. Or I am
misunderstanding the statements?

Thanks.
> 
> I must not be understanding what else you think it should say.  Is it
> just that it isn't "... the A flag, among other things"?

Re: PATCH and more remarks on parameter expansion docs

[Bart Schaefer]

On Jan 3,  7:28am, Han Pingtian wrote:
} Subject: Re: PATCH and more remarks on parameter expansion docs
}
} On Wed, Jan 02, 2013 at 08:46:44AM -0800, Bart Schaefer wrote:
} > 
} >      Note that splitting is applied to WORD in the assignment forms of
} >      SPEC _before_ the assignment to NAME is performed.  This affects
} >      the result of array assignments with the A flag.
} 
} I think it doesn't say this splitting _before_ assignments is only true
} when (A) is used. Looks like if (A) wasn't used, splitting will _after_
} the assignments

Ah, I see.  OK, I agree that it's imprecise about this.  It's imprecise
but not in the way that you think:  The splitting is in fact done before
the assignment in either case, but at that point the internals have both
the expanded string and the split result available.  The (A) flag then
determines which of those two values is used in the assignment.

I'm not sure whether in practice this can ever have a visible effect.  In
theory if there were some oddity with the syntax of WORD that caused the
splitting step to fail, then the NAME parameter would not be assigned,
but I can't trivially produce an example.

On the other hand, when does this "mistake" in the documentation matter?
Given what it says, what did you expect ${=xx::=foo   bar} to assign?

The doc needs to strike a balance between giving so much detail that it's
confusing and too little to tell the user what to expect.

Re: PATCH and more remarks on parameter expansion docs

[Han Pingtian]

On Thu, Jan 03, 2013 at 11:42:17AM -0800, Bart Schaefer wrote:
> On Jan 3,  7:28am, Han Pingtian wrote:
> } Subject: Re: PATCH and more remarks on parameter expansion docs
> }
> } On Wed, Jan 02, 2013 at 08:46:44AM -0800, Bart Schaefer wrote:
> } > 
> } >      Note that splitting is applied to WORD in the assignment forms of
> } >      SPEC _before_ the assignment to NAME is performed.  This affects
> } >      the result of array assignments with the A flag.
> } 
> } I think it doesn't say this splitting _before_ assignments is only true
> } when (A) is used. Looks like if (A) wasn't used, splitting will _after_
> } the assignments
> 
> Ah, I see.  OK, I agree that it's imprecise about this.  It's imprecise
> but not in the way that you think:  The splitting is in fact done before
> the assignment in either case, but at that point the internals have both
> the expanded string and the split result available.  The (A) flag then
> determines which of those two values is used in the assignment.
> 
> I'm not sure whether in practice this can ever have a visible effect.  In
> theory if there were some oddity with the syntax of WORD that caused the
> splitting step to fail, then the NAME parameter would not be assigned,
> but I can't trivially produce an example.
> 
> On the other hand, when does this "mistake" in the documentation matter?
> Given what it says, what did you expect ${=xx::=foo   bar} to assign?
> 

Since ${=xx::=foo   bar} would result in two items foo and bar, but xx's
value is "foo   bar", so it looks like xx is assigned value "foo   bar",
then xx's value is splitted. 

But xx is a scalar here, so we cannot assign two value foo and bar to it
at the same time. So I think the value would be joined again before the
assignment? So I expect xx's value would be "foo bar"?

Thanks.