zsh-workers
 help / color / mirror / code / Atom feed
* [BUG] zsh/param/private scoping error
@ 2021-09-01 11:49 Marlon Richert
  2021-09-01 14:09 ` Daniel Shahaf
  2021-09-01 14:39 ` Bart Schaefer
  0 siblings, 2 replies; 11+ messages in thread
From: Marlon Richert @ 2021-09-01 11:49 UTC (permalink / raw)
  To: Zsh hackers list

The following two functions are normally equivalent:

() { typeset -g tst }
() { tst= }

Both result in a global variable 'tst' being created. However when the
second function is called from a function that happens to declare a
private variable with the same name, it no longer creates a global
variable but instead aborts with an error:

% () { private tst; () { tst= } }
(anon):1: tst: attempt to assign private in nested scope
%

This is not how one would expect private variables to behave. Inside
the inner function, the private variable should be completely out of
scope and the `tst=` statement should result in the creation of a
global variable.

Note that the error above does not happens when there already exists a
global variable with the same name:

% typeset -g tst
% () { private tst; () { tst= } }
%


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 11:49 [BUG] zsh/param/private scoping error Marlon Richert
@ 2021-09-01 14:09 ` Daniel Shahaf
  2021-09-01 16:21   ` Marlon Richert
  2021-09-01 14:39 ` Bart Schaefer
  1 sibling, 1 reply; 11+ messages in thread
From: Daniel Shahaf @ 2021-09-01 14:09 UTC (permalink / raw)
  To: Marlon Richert; +Cc: Zsh hackers list

Marlon Richert wrote on Wed, Sep 01, 2021 at 14:49:58 +0300:
> This is not how one would expect private variables to behave. Inside
> the inner function, the private variable should be completely out of
> scope and the `tst=` statement should result in the creation of a
> global variable.

Makes sense to me.  A regression test (one with the 'F' flag) would be
welcome, if you've got spare tuits.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 11:49 [BUG] zsh/param/private scoping error Marlon Richert
  2021-09-01 14:09 ` Daniel Shahaf
@ 2021-09-01 14:39 ` Bart Schaefer
  2021-09-01 16:03   ` Mikael Magnusson
  1 sibling, 1 reply; 11+ messages in thread
From: Bart Schaefer @ 2021-09-01 14:39 UTC (permalink / raw)
  To: Marlon Richert; +Cc: Zsh hackers list

On Wed, Sep 1, 2021 at 4:51 AM Marlon Richert <marlon.richert@gmail.com> wrote:
>
> % () { private tst; () { tst= } }
> (anon):1: tst: attempt to assign private in nested scope
> %
>
> This is not how one would expect private variables to behave.

But it's how variable scoping behaves in the shell language.  If I do

% () { local tst; () { tst=foo } }

Then it is the tst in the surrounding scope that gets altered, not the
global one.  If the inner function is INTENDED to create a global, it
should be using "typeset -g" explicitly.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 14:39 ` Bart Schaefer
@ 2021-09-01 16:03   ` Mikael Magnusson
  2021-09-01 16:11     ` Marlon Richert
  0 siblings, 1 reply; 11+ messages in thread
From: Mikael Magnusson @ 2021-09-01 16:03 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Marlon Richert, Zsh hackers list

On 9/1/21, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Wed, Sep 1, 2021 at 4:51 AM Marlon Richert <marlon.richert@gmail.com>
> wrote:
>>
>> % () { private tst; () { tst= } }
>> (anon):1: tst: attempt to assign private in nested scope
>> %
>>
>> This is not how one would expect private variables to behave.
>
> But it's how variable scoping behaves in the shell language.  If I do
>
> % () { local tst; () { tst=foo } }
>
> Then it is the tst in the surrounding scope that gets altered, not the
> global one.  If the inner function is INTENDED to create a global, it
> should be using "typeset -g" explicitly.

This seems weird to me, since private is stated to be "used to create
parameters whose scope is limited to the current function body, and
not to other functions called  by  the current function." If the
private parameter is not in scope, then any called functions should
behave as if it is not in scope, ie automatically create a parameter
scoped to whatever level is above the one that has the private.
(technically speaking, this is probably hard). If private parameters
are only compatible with code that runs without warnings under setopt
WARN_CREATE_GLOBAL, then we should document this.

-- 
Mikael Magnusson


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 16:03   ` Mikael Magnusson
@ 2021-09-01 16:11     ` Marlon Richert
  2021-09-01 17:00       ` Mikael Magnusson
  0 siblings, 1 reply; 11+ messages in thread
From: Marlon Richert @ 2021-09-01 16:11 UTC (permalink / raw)
  To: Mikael Magnusson; +Cc: Bart Schaefer, Zsh hackers list

On Wed, Sep 1, 2021 at 7:03 PM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> On 9/1/21, Bart Schaefer <schaefer@brasslantern.com> wrote:
> > On Wed, Sep 1, 2021 at 4:51 AM Marlon Richert <marlon.richert@gmail.com>
> > wrote:
> >>
> >> % () { private tst; () { tst= } }
> >> (anon):1: tst: attempt to assign private in nested scope
> >> %
> >>
> >> This is not how one would expect private variables to behave.
> >
> > But it's how variable scoping behaves in the shell language.  If I do
> >
> > % () { local tst; () { tst=foo } }
> >
> > Then it is the tst in the surrounding scope that gets altered, not the
> > global one.  If the inner function is INTENDED to create a global, it
> > should be using "typeset -g" explicitly.
>
> This seems weird to me, since private is stated to be "used to create
> parameters whose scope is limited to the current function body, and
> not to other functions called  by  the current function." If the
> private parameter is not in scope, then any called functions should
> behave as if it is not in scope, ie automatically create a parameter
> scoped to whatever level is above the one that has the private.
> (technically speaking, this is probably hard).

Also note that when _reading_ variables, the `private` keyword already
behaves like this, that is, the private variable is out of scope for
the inner function:

% () { ; private tst=bar; () { print tst=$tst } }
tst=
%
% tst=foo; () { ; private tst=bar; () { print tst=$tst } }
tst=foo
%


> If private parameters
> are only compatible with code that runs without warnings under setopt
> WARN_CREATE_GLOBAL, then we should document this.

That would largely defeat the purpose of private variables, wouldn't it? :)


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 14:09 ` Daniel Shahaf
@ 2021-09-01 16:21   ` Marlon Richert
  0 siblings, 0 replies; 11+ messages in thread
From: Marlon Richert @ 2021-09-01 16:21 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: Zsh hackers list

On Wed, Sep 1, 2021 at 5:10 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Marlon Richert wrote on Wed, Sep 01, 2021 at 14:49:58 +0300:
> > This is not how one would expect private variables to behave. Inside
> > the inner function, the private variable should be completely out of
> > scope and the `tst=` statement should result in the creation of a
> > global variable.
>
> Makes sense to me.  A regression test (one with the 'F' flag) would be
> welcome, if you've got spare tuits.

I was about to create one in V10private.ztst, when I noticed it
appears to exist already on lines 230 - 250. :)

 typeset -A hash_test=(top level)
 () {
  local -Pa array_test=(in function)
  local -PA hash_test=($array_test)
  () {
   print X ${(kv)hash_test}
   hash_test=(even deeper)
   {
     array_test+=(${(kv)hash_test})
   } always {
     print ${array_test-array_test not set} ${(t)array_test}
   }
  }
  print Y ${(kv)hash_test} Z $array_test
 }
 print ${(kv)hash_test} ${(t)array_test}
1:privates are not visible in anonymous functions, part 3
>X top level
>array_test not set
?(anon):4: array_test: attempt to assign private in nested scope
F:future revision will create a global with this assignment


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 16:11     ` Marlon Richert
@ 2021-09-01 17:00       ` Mikael Magnusson
  2021-09-01 17:35         ` Bart Schaefer
  0 siblings, 1 reply; 11+ messages in thread
From: Mikael Magnusson @ 2021-09-01 17:00 UTC (permalink / raw)
  To: Marlon Richert; +Cc: Bart Schaefer, Zsh hackers list

On 9/1/21, Marlon Richert <marlon.richert@gmail.com> wrote:
> On Wed, Sep 1, 2021 at 7:03 PM Mikael Magnusson <mikachu@gmail.com> wrote:
>> If private parameters
>> are only compatible with code that runs without warnings under setopt
>> WARN_CREATE_GLOBAL, then we should document this.
>
> That would largely defeat the purpose of private variables, wouldn't it? :)

Ah, I misinterpreted one of the examples, I thought that this would
work as you expect:
() { private foo; () { typeset -g foo=bar } }
(while this wouldn't:
() { private foo; () { foo=bar } }
)

But in both cases it is an error (although only the latter prints a
message and aborts execution), and foo is not set globally.

In fact a typeset -g is needed on the global level before the function
using private is called, which seems impossible for the innermost
function to enforce.

-- 
Mikael Magnusson


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 17:00       ` Mikael Magnusson
@ 2021-09-01 17:35         ` Bart Schaefer
  2021-10-03 21:55           ` Bart Schaefer
  0 siblings, 1 reply; 11+ messages in thread
From: Bart Schaefer @ 2021-09-01 17:35 UTC (permalink / raw)
  To: Mikael Magnusson; +Cc: Marlon Richert, Zsh hackers list

On Wed, Sep 1, 2021 at 10:00 AM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> Ah, I misinterpreted one of the examples, I thought that this would
> work as you expect:
> () { private foo; () { typeset -g foo=bar } }
> (while this wouldn't:
> () { private foo; () { foo=bar } }
> )
>
> But in both cases it is an error (although only the latter prints a
> message and aborts execution), and foo is not set globally.

Something else has changed since the implementation of private, I
think, because this is not what I expected:

top () { private foo=top; mid }
mid () { typeset -g foo=middle; bot }
bot () { print $foo }
functions -t top

% top
+top:0> private foo=top
+top:0> mid
+mid:0> typeset -g foo=middle
+mid:0> bot
bot: foo: attempt to assign private in nested scope

What makes the expansion of $foo in bot into an assignment?

> In fact a typeset -g is needed on the global level before the function
> using private is called, which seems impossible for the innermost
> function to enforce.

As you can see from the "F:" test Marlon quoted, this is something I
eventually intended to change.  It's rather complicated because of the
way locals are implemented as a LIFO at each individual named entry in
the parameter table, rather than a LIFO of parameter tables.  Even
without private, "typeset -g" only reaches as far upwards as the most
recent local declaration of the name.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-09-01 17:35         ` Bart Schaefer
@ 2021-10-03 21:55           ` Bart Schaefer
  2021-10-04  4:54             ` Mikael Magnusson
  0 siblings, 1 reply; 11+ messages in thread
From: Bart Schaefer @ 2021-10-03 21:55 UTC (permalink / raw)
  To: Zsh hackers list

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Wed, Sep 1, 2021 at 10:35 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> Something else has changed since the implementation of private, I
> think, because this is not what I expected:
>
> top () { private foo=top; mid }
> mid () { typeset -g foo=middle; bot }
> bot () { print $foo }
> functions -t top

The attached patch addresses this, producing the output I expected:

+top:0> private foo=top
+top:0> mid
+mid:0> typeset -g foo=middle
+mid:0> bot
+bot:0> print middle
middle

It also improves the error message in the event that a typeset
attempts to change the scope or properties of a private parameter.

> Even
> without private, "typeset -g" only reaches as far upwards as the most
> recent local declaration of the name.

A documentation patch to better explain this is included.  As a bonus,
there's also an explanation of why some parameters do not appear in
"typeset -p" output.

[-- Attachment #2: private-scoping.txt --]
[-- Type: text/plain, Size: 6158 bytes --]

diff --git a/Doc/Zsh/mod_private.yo b/Doc/Zsh/mod_private.yo
index 78aee0acf..6da18f814 100644
--- a/Doc/Zsh/mod_private.yo
+++ b/Doc/Zsh/mod_private.yo
@@ -10,12 +10,16 @@ ifnzman()
 startitem()
 findex(private)
 cindex(private parameter, creating)
-item(tt(private) [ {tt(PLUS())|tt(-)}tt(AHUahlprtux) ] \
+item(tt(private) [ {tt(PLUS())|tt(-)}tt(AHUahlmrtux) ] \
 [ {tt(PLUS())|tt(-)}tt(EFLRZi) [ var(n) ] ] [ var(name)[tt(=)var(value)] ... ])(
 The tt(private) builtin accepts all the same options and arguments as tt(local)
 (ifzman(zmanref(zshbuiltins))ifnzman(noderef(Shell Builtin Commands))) except
 for the `tt(-)tt(T)' option.  Tied parameters may not be made private.
 
+The `tt(-)tt(p)' option is presently a no-op because the state of
+private parameters cannot be reliably be reloaded.  This also applies
+to printing private parameters with `tt(typeset -p)'.
+
 If used at the top level (outside a function scope), tt(private) creates a
 normal parameter in the same manner as tt(declare) or tt(typeset).  A
 warning about this is printed if tt(WARN_CREATE_GLOBAL) is set
@@ -75,7 +79,8 @@ itemiz(Within any other function called by the declaring function, the
 private parameter does em(NOT) hide other parameters of the same name, so
 for example a global parameter of the same name is visible and may be
 assigned or unset.  This includes calls to anonymous functions, although
-that may also change in the future.)
+that may also change in the future.  However, the private name may not be
+created outside the local scope when it was not previously declared.)
 itemiz(An exported private remains in the environment of inner scopes but
 appears unset for the current shell in those scopes.  Generally, exporting
 private parameters should be avoided.)
diff --git a/Doc/Zsh/params.yo b/Doc/Zsh/params.yo
index b514eb072..a88e44d4f 100644
--- a/Doc/Zsh/params.yo
+++ b/Doc/Zsh/params.yo
@@ -632,6 +632,14 @@ In the parameter lists that follow, the mark `<S>' indicates that the
 parameter is special.  `<Z>' indicates that the parameter does not exist
 when the shell initializes in tt(sh) or tt(ksh) emulation mode.
 
+The parameters `tt(!)', `tt(#)', `tt(*)', `tt(-)', `tt(?)', `tt(@)',
+`tt($)', `tt(ARGC)', `tt(HISTCMD)', `tt(LINENO)', `tt(PPID)',
+`tt(status)', `tt(TTYIDLE)', `tt(zsh_eval_context)',
+`tt(ZSH_EVAL_CONTEXT)', and `tt(ZSH_SUBSHELL)' are read-only and thus
+cannot be restored by the user, so they are not output by
+`tt(typeset -p)'.  This also applies to many read-only parameters loaded
+from modules.
+
 The following parameters are automatically set by the shell:
 
 startitem()
diff --git a/Src/Modules/param_private.c b/Src/Modules/param_private.c
index 24545819d..c53839152 100644
--- a/Src/Modules/param_private.c
+++ b/Src/Modules/param_private.c
@@ -125,7 +125,7 @@ makeprivate(HashNode hn, UNUSED(int flags))
 	    break;
 	}
 	/* PM_HIDE so new parameters in deeper scopes do not shadow */
-	pm->node.flags |= (PM_HIDE|PM_SPECIAL|PM_REMOVABLE);
+	pm->node.flags |= (PM_HIDE|PM_SPECIAL|PM_REMOVABLE|PM_RO_BY_DESIGN);
 	pm->level -= 1;
     }
 }
@@ -171,7 +171,7 @@ bin_private(char *nam, char **args, LinkList assigns, Options ops, int func)
 {
     int from_typeset = 1;
     int ofake = fakelevel;	/* paranoia in case of recursive call */
-    int hasargs = *args != NULL || (assigns && firstnode(assigns));
+    int hasargs = /* *args != NULL || */ (assigns && firstnode(assigns));
     makeprivate_error = 0;
 
     if (!OPT_ISSET(ops, 'P')) {
@@ -190,8 +190,10 @@ bin_private(char *nam, char **args, LinkList assigns, Options ops, int func)
 	return bin_typeset("private", args, assigns, ops, func);
     }
 
-    ops->ind['g'] = 2;	/* force bin_typeset() to behave as "local" */
-    if (OPT_ISSET(ops, 'p') || (!hasargs && OPT_ISSET(ops, '+'))) {
+    if (!(OPT_ISSET(ops, 'm') || OPT_ISSET(ops, '+')))
+	ops->ind['g'] = 2;	/* force bin_typeset() to behave as "local" */
+    if (OPT_ISSET(ops, 'p') || OPT_ISSET(ops, 'm') ||
+	(!hasargs && OPT_ISSET(ops, '+'))) {
 	return bin_typeset("private", args, assigns, ops, func);
     }
 
@@ -559,7 +561,7 @@ printprivatenode(HashNode hn, int printflags)
 
 static struct builtin bintab[] = {
     /* Copied from BUILTIN("local"), "P" added */
-    BUILTIN("private", BINF_PLUSOPTS | BINF_MAGICEQUALS | BINF_PSPECIAL | BINF_ASSIGN, (HandlerFunc)bin_private, 0, -1, 0, "AE:%F:%HL:%PR:%TUZ:%ahi:%lprtux", "P")
+    BUILTIN("private", BINF_PLUSOPTS | BINF_MAGICEQUALS | BINF_PSPECIAL | BINF_ASSIGN, (HandlerFunc)bin_private, 0, -1, 0, "AE:%F:%HL:%PR:%TUZ:%ahi:%lmprtux", "P")
 };
 
 static struct features module_features = {
diff --git a/Src/params.c b/Src/params.c
index 704aad588..b703a97ce 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -1013,6 +1013,11 @@ createparam(char *name, int flags)
 		(oldpm->node.flags & PM_SPECIAL) ||
 		/* POSIXBUILTINS horror: we need to retain 'export' flags */
 		(isset(POSIXBUILTINS) && (oldpm->node.flags & PM_EXPORTED))) {
+		if (oldpm->node.flags & PM_RO_BY_DESIGN) {
+		    zerr("%s: can't change parameter attribute",
+			 name);
+		    return NULL;
+		}
 		oldpm->node.flags &= ~PM_UNSET;
 		if ((oldpm->node.flags & PM_SPECIAL) && oldpm->ename) {
 		    Param altpm =
diff --git a/Test/V10private.ztst b/Test/V10private.ztst
index 03e8259d5..56ffbc5b4 100644
--- a/Test/V10private.ztst
+++ b/Test/V10private.ztst
@@ -116,14 +116,14 @@
  }
  outer () {
   local -PA hash_test=(in function)
-  typeset -p hash_test
+  private + hash_test
   inner
  }
  outer
  print ${(kv)hash_test}
 0:private hides value from surrounding scope in nested scope
 >typeset -a hash_test=( top level )
->typeset -A hash_test=( [in]=function )
+>hash_test=( [in]=function )
 >typeset -g -a hash_test=( top level )
 >array-local top level
 >top level
@@ -246,7 +246,7 @@ F:note "typeset" rather than "private" in output from outer
 1:privates are not visible in anonymous functions, part 3
 >X top level
 >array_test not set
-?(anon):4: array_test: attempt to assign private in nested scope
+?(anon):4: array_test: can't change parameter attribute
 F:future revision will create a global with this assignment
 
  typeset -a array_test

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-10-03 21:55           ` Bart Schaefer
@ 2021-10-04  4:54             ` Mikael Magnusson
  2021-10-04 15:12               ` Bart Schaefer
  0 siblings, 1 reply; 11+ messages in thread
From: Mikael Magnusson @ 2021-10-04  4:54 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh hackers list

On 10/3/21, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Wed, Sep 1, 2021 at 10:35 AM Bart Schaefer <schaefer@brasslantern.com>
> wrote:
>>
>> Something else has changed since the implementation of private, I
>> think, because this is not what I expected:
>>
>> top () { private foo=top; mid }
>> mid () { typeset -g foo=middle; bot }
>> bot () { print $foo }
>> functions -t top
>
> The attached patch addresses this, producing the output I expected:
>
> +top:0> private foo=top
> +top:0> mid
> +mid:0> typeset -g foo=middle
> +mid:0> bot
> +bot:0> print middle
> middle
>
> It also improves the error message in the event that a typeset
> attempts to change the scope or properties of a private parameter.
>
>> Even
>> without private, "typeset -g" only reaches as far upwards as the most
>> recent local declaration of the name.
>
> A documentation patch to better explain this is included.  As a bonus,
> there's also an explanation of why some parameters do not appear in
> "typeset -p" output.

+private parameters cannot be reliably be reloaded.  This also applies

This sentence has an extra "be".

-- 
Mikael Magnusson


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [BUG] zsh/param/private scoping error
  2021-10-04  4:54             ` Mikael Magnusson
@ 2021-10-04 15:12               ` Bart Schaefer
  0 siblings, 0 replies; 11+ messages in thread
From: Bart Schaefer @ 2021-10-04 15:12 UTC (permalink / raw)
  To: Mikael Magnusson; +Cc: Zsh hackers list

On Sun, Oct 3, 2021 at 9:54 PM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> This sentence has an extra "be".

So everyone seems to be telling me.  :-)  I'll fix it when committing.


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2021-10-04 15:12 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-01 11:49 [BUG] zsh/param/private scoping error Marlon Richert
2021-09-01 14:09 ` Daniel Shahaf
2021-09-01 16:21   ` Marlon Richert
2021-09-01 14:39 ` Bart Schaefer
2021-09-01 16:03   ` Mikael Magnusson
2021-09-01 16:11     ` Marlon Richert
2021-09-01 17:00       ` Mikael Magnusson
2021-09-01 17:35         ` Bart Schaefer
2021-10-03 21:55           ` Bart Schaefer
2021-10-04  4:54             ` Mikael Magnusson
2021-10-04 15:12               ` Bart Schaefer

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).