'export -p' lacks POSIX output

'export -p' lacks POSIX output

[Martijn Dekker]

The command 'export -p' always outputs 'typeset' commands. If
POSIXBUILTINS is active, it should output 'export' commands.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22_03
| When -p is specified, export shall write to the standard output the
| names and values of all exported variables, in the following format:
|
| "export %s=%s\n", <name>, <value>
|
| if name is set, and:
|
| "export %s\n", <name>
|
| if name is unset.

Thanks,

- M.

Re: 'export -p' lacks POSIX output

[Bart Schaefer]

On Oct 22,  5:02am, Martijn Dekker wrote:
} Subject: 'export -p' lacks POSIX output
}
} The command 'export -p' always outputs 'typeset' commands. If
} POSIXBUILTINS is active, it should output 'export' commands.

Would anyone object if this just happened all the time, rather than
depending on POSIXBUILTINS + "export"?

The decision of what command name to print is buried way down in the
depths of hash table scanning callbacks, which although it could test
for POSIXBUILTINS, has no knowledge of the original command name used
to invoke the scan.  (This is a bugaboo for the zsh/param/private
module, so I'm pretty intimately familiar with it at this point.)

More explicitly,

    typeset -x foo=bar
    typeset -p foo

would print

    export foo=bar

even if POSIXBUILTINS were not set.

Re: 'export -p' lacks POSIX output

[Martijn Dekker]

Op 22-10-16 om 20:24 schreef Bart Schaefer:
> Would anyone object if this just happened all the time, rather than
> depending on POSIXBUILTINS + "export"?

After the patch, variables with a non-scalar type get output such as:

export -i10 SHLVL=2

Option flags to 'export' other than -p are not POSIX, and POSIX
specifies output without any flags for 'export -p'.

Maybe that implies that commands like 'export -p' and 'trap' should
generate commands suitable for any POSIX shell. This would mean the
$'...' quoting method should also be deactivated. It may become POSIX at
some point, but it's anyone's guess how long that will take, and then
shells have to implement it. Then again, POSIX says the input should be
"suitable for reinput to the shell", not "suitable for reinput to any
POSIX shell".[*]

In any case, if the output needs to be conditional upon POSIXBUILTINS
anyway, I reckon you might as well not change the behaviour at all if
POSIXBUILTINS is not active.

- M.

[*]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22_04

Re: 'export -p' lacks POSIX output

[Bart Schaefer]

On Oct 23,  2:00pm, Martijn Dekker wrote:
} Subject: Re: 'export -p' lacks POSIX output
}
} Op 22-10-16 om 20:24 schreef Bart Schaefer:
} > Would anyone object if this just happened all the time, rather than
} > depending on POSIXBUILTINS + "export"?
} 
} After the patch, variables with a non-scalar type get output such as:
} 
} export -i10 SHLVL=2
} 
} Option flags to 'export' other than -p are not POSIX, and POSIX
} specifies output without any flags for 'export -p'.

We may have to settle for incompatibility here.  As I mentioned, there
is no way (without either a large code change or introducing yet another
global variable [*]) for the code that prints the command name to know
whether "typeset -p", "declare -p", "export -p", etc. were used.  If the
non-string (note, integer is not non-scalar) flags are eliminated, the
state won't be properly reflected if the commands are re-input.

If you're writing a POSIX-compatible script you shouldn't declare anything
integer in the first place, should you?

} In any case, if the output needs to be conditional upon POSIXBUILTINS
} anyway, I reckon you might as well not change the behaviour at all if
} POSIXBUILTINS is not active.

The selection of the command name to print and the output of the type
dinstinction flags are in separate sections of the code, so that's not
really a concern.  I think the addition of -g for typeset of non-local
parameters in function scope is useful (waiting to hear if anyone has
objections) and that doesn't depend on POSIXBUILTINS.

[*] Of course if such a variable WERE introduced I could take advantage
of it in zsh/param/private.  I was trying to avoid invasive changes for
that module but if it's needed for other reasons I'd be less shy.

Re: 'export -p' lacks POSIX output

[Peter Stephenson]

On Sat, 22 Oct 2016 11:24:23 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Oct 22,  5:02am, Martijn Dekker wrote:
> } Subject: 'export -p' lacks POSIX output
> }
> } The command 'export -p' always outputs 'typeset' commands. If
> } POSIXBUILTINS is active, it should output 'export' commands.
> 
> Would anyone object if this just happened all the time, rather than
> depending on POSIXBUILTINS + "export"?

That sounds reasonable and probably clearer; the one thing to be careful
about is that "export" is actually equivalent to "typeset -gx".

pws

exported unset variables [was: 'export -p' lacks POSIX output]

[Martijn Dekker]

Op 22-10-16 om 04:02 schreef Martijn Dekker:
> The command 'export -p' always outputs 'typeset' commands. If
> POSIXBUILTINS is active, it should output 'export' commands.
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22_03
> | When -p is specified, export shall write to the standard output the
> | names and values of all exported variables, in the following format:
> |
> | "export %s=%s\n", <name>, <value>
> |
> | if name is set, and:
> |
> | "export %s\n", <name>
> |
> | if name is unset.

That reminds me that, in POSIX, there is such a thing as a variable that
is both exported and unset; the variable name gets the 'exported' flag
and will retain it if/when it is set later. Ref. as above.

In zsh, however, variables are automatically set to the empty value upon
being exported.

Back in April 2015, in zsh-workers/34991, I reported a similar issue for
readonly variables, which Peter fixed in the following three commits:

bf258a1c07a9cf1119f83d1d15310b02b94f4d67
60c6bcdeae6d132c0ab770cc8491055b24f7670e
f855801fb9ed82d2596f52f5b64dd66c42255c5f

zsh needs a similar fix for exported unset variables, presumably also
subject to the POSIXBUILTINS option.

Thanks,

- M.

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Bart Schaefer]

On Oct 28, 10:00pm, Martijn Dekker wrote:
}
} In zsh, however, variables are automatically set to the empty value upon
} being exported.

That's no longer true subsequent to my "export -p" patch:

torch% export XYZPDQ
torch% export -p | grep XYZPDQ
export XYZPDQ
torch% printenv | grep XYZPDQ
torch% 

Unless there's some other nuance here that I'm missing?

Even before this patch "typeset" would report that the exported variable
was set to the empty string, but zsh did not modify the environment unless
an empty string were actually assigned.

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Martijn Dekker]

Op 28-10-16 om 22:31 schreef Bart Schaefer:
> On Oct 28, 10:00pm, Martijn Dekker wrote:
> }
> } In zsh, however, variables are automatically set to the empty value upon
> } being exported.
> 
> That's no longer true subsequent to my "export -p" patch:
> 
> torch% export XYZPDQ
> torch% export -p | grep XYZPDQ
> export XYZPDQ
> torch% printenv | grep XYZPDQ
> torch% 
> 
> Unless there's some other nuance here that I'm missing?

Hmm... With current git version:

$ PS1='%% ' Src/zsh -f -o posixbuiltins
% unset -v var
% export var
% echo ${var+s}  # this shows it as set (should be unset)
s
% export -p var  # but this shows it as unset
export var
% var=''
% export -p var  # 'export' knows if it's set or not, ${var+s} doesn't?
export var=''

So, currently, ${var+s} doesn't correctly represent that an exported
'var' is unset.

(Also, is it correct/expected behaviour that zsh doesn't parse comments
on the interactive command line? Blindly copying/pasting the above won't
work for that reason.)

Thanks,

- M.

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Bart Schaefer]

[It's once again taking a long time for some messages to reach the
mailing list.  My first message on this thread hasn't even shown up
there yet.]

On Oct 28, 10:48pm, Martijn Dekker wrote:
}
} $ PS1='%% ' Src/zsh -f -o posixbuiltins
} % unset -v var
} % export var
} % echo ${var+s}  # this shows it as set (should be unset)
} s
} % export -p var  # but this shows it as unset
} export var

Ah, I see.  Yes, I mentioned this in the email about the patch.  The
shell variable named "var" becomes set upon being exported, but the
environment copy remains unset until explicitly assigned.  Zsh has
always distinguished the shell variable from the exported variable,
even though the value of the exported variable (if any) is stored in
the data structure for the shell variable.

All my patch did was cause "export -p" to output the state of the
environment variable instead of the state of the shell variable.

See if the patch below will help.

} (Also, is it correct/expected behaviour that zsh doesn't parse comments
} on the interactive command line? Blindly copying/pasting the above won't
} work for that reason.)

Yes, the option INTERACTIVE_COMMENTS has to be set to recognize comments
at the command line.  Should this be on in POSIX mode?


diff --git a/Src/builtin.c b/Src/builtin.c
index 2db739f..183ed45 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2008,11 +2008,12 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
      * handled in createparam().  Here we just avoid using it for the
      * present tests if it's unset.
      *
-     * POSIXBUILTINS horror: we need to retain the 'readonly' flag
-     * of an unset parameter.
+     * POSIXBUILTINS horror: we need to retain the 'readonly' or 'export'
+     * flags of an unset parameter.
      */
     usepm = pm && (!(pm->node.flags & PM_UNSET) ||
-		   (isset(POSIXBUILTINS) && (pm->node.flags & PM_READONLY)));
+		   (isset(POSIXBUILTINS) &&
+		    (pm->node.flags & (PM_READONLY|PM_EXPORTED))));
 
     /*
      * We need to compare types with an existing pm if special,
@@ -2135,7 +2136,8 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	/*
 	 * Stricter rules about retaining readonly attribute in this case.
 	 */
-	if ((on & PM_READONLY) && (!usepm || (pm->node.flags & PM_UNSET)) &&
+	if ((on & (PM_READONLY|PM_EXPORTED)) &&
+	    (!usepm || (pm->node.flags & PM_UNSET)) &&
 	    !ASG_VALUEP(asg))
 	    on |= PM_UNSET;
 	else if (usepm && (pm->node.flags & PM_READONLY) &&

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Martijn Dekker]

Op 29-10-16 om 01:18 schreef Bart Schaefer:
[...]
> All my patch did was cause "export -p" to output the state of the
> environment variable instead of the state of the shell variable.

Understood.

> See if the patch below will help.

Seems like it's about halfway there:

$ PS1='%% ' Src/zsh -f -o posixbuiltins
% export var; echo ${var+set} ${var-unset}; typeset -p var
unset
export var
% var=; echo ${var+set} ${var-unset}; typeset -p var
set
typeset var=''

The variable name loses its export attribute upon assignment; it should
retain it.

> } (Also, is it correct/expected behaviour that zsh doesn't parse comments
> } on the interactive command line? Blindly copying/pasting the above won't
> } work for that reason.)
> 
> Yes, the option INTERACTIVE_COMMENTS has to be set to recognize comments
> at the command line.  Should this be on in POSIX mode?

I think so. All the other shells recognise comments in interactive mode
by default. The spec for comments at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_03
(#9) does not distinguish between interactive and non-interactive shells.

Thanks,

- M.

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Bart Schaefer]

On Oct 29,  9:11am, Martijn Dekker wrote:
} Subject: Re: exported unset variables [was: 'export -p' lacks POSIX output
}
} % var=; echo ${var+set} ${var-unset}; typeset -p var
} set
} typeset var=''
} 
} The variable name loses its export attribute upon assignment; it should
} retain it.

That one potentially gets rather ugly; consider:

% setopt POSIXBUILTINS
% export var
% var=()
zsh: var: attempt to assign array value to non-array

Is that OK with everyone?

As an aside, zsh currently allows the export flag to be applied to array
variables, but then will not put them in the environment.  Patch below
revises 39704 to use "typeset -xa ary" rather than "export -a ary".

Replaces 39754.


diff --git a/Src/builtin.c b/Src/builtin.c
index 2db739f..986ace2 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2008,11 +2008,12 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
      * handled in createparam().  Here we just avoid using it for the
      * present tests if it's unset.
      *
-     * POSIXBUILTINS horror: we need to retain the 'readonly' flag
-     * of an unset parameter.
+     * POSIXBUILTINS horror: we need to retain the 'readonly' or 'export'
+     * flags of an unset parameter.
      */
     usepm = pm && (!(pm->node.flags & PM_UNSET) ||
-		   (isset(POSIXBUILTINS) && (pm->node.flags & PM_READONLY)));
+		   (isset(POSIXBUILTINS) &&
+		    (pm->node.flags & (PM_READONLY|PM_EXPORTED))));
 
     /*
      * We need to compare types with an existing pm if special,
@@ -2135,7 +2136,8 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	/*
 	 * Stricter rules about retaining readonly attribute in this case.
 	 */
-	if ((on & PM_READONLY) && (!usepm || (pm->node.flags & PM_UNSET)) &&
+	if ((on & (PM_READONLY|PM_EXPORTED)) &&
+	    (!usepm || (pm->node.flags & PM_UNSET)) &&
 	    !ASG_VALUEP(asg))
 	    on |= PM_UNSET;
 	else if (usepm && (pm->node.flags & PM_READONLY) &&
@@ -2143,6 +2145,10 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	    zerr("read-only variable: %s", pm->node.nam);
 	    return NULL;
 	}
+	/* This is handled by createparam():
+	if (usepm && (pm->node.flags & PM_EXPORTED) && !(off & PM_EXPORTED))
+	    on |= PM_EXPORTED;
+	*/
     }
 
     /*
diff --git a/Src/params.c b/Src/params.c
index 3084b1f..330f22b 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -940,7 +940,10 @@ createparam(char *name, int flags)
 		zerr("%s: restricted", name);
 		return NULL;
 	    }
-	    if (!(oldpm->node.flags & PM_UNSET) || (oldpm->node.flags & PM_SPECIAL)) {
+	    if (!(oldpm->node.flags & PM_UNSET) ||
+		(oldpm->node.flags & PM_SPECIAL) ||
+		/* POSIXBUILTINS horror: we need to retain 'export' flags */
+		(isset(POSIXBUILTINS) && (oldpm->node.flags & PM_EXPORTED))) {
 		oldpm->node.flags &= ~PM_UNSET;
 		if ((oldpm->node.flags & PM_SPECIAL) && oldpm->ename) {
 		    Param altpm =
@@ -5225,10 +5228,6 @@ printparamvalue(Param p, int printflags)
 {
     char *t, **u;
 
-    if ((p->node.flags & PM_EXPORTED) && !p->env) {
-	putchar('\n');
-	return;
-    }
     if (printflags & PRINT_KV_PAIR)
 	putchar(' ');
     else
@@ -5332,7 +5331,8 @@ printparamnode(HashNode hn, int printflags)
 	}
 	if (locallevel && p->level >= locallevel) {
 	    printf("typeset ");	    /* printf("local "); */
-	} else if (p->node.flags & PM_EXPORTED) {
+	} else if ((p->node.flags & PM_EXPORTED) &&
+		   !(p->node.flags & (PM_ARRAY|PM_HASHED))) {
 	    printf("export ");
 	} else if (locallevel) {
 	    printf("typeset -g ");
@@ -5350,8 +5350,8 @@ printparamnode(HashNode hn, int printflags)
 	    if (pmptr->flags & PMTF_TEST_LEVEL) {
 		if (p->level)
 		    doprint = 1;
-	    } else if ((pmptr->binflag != PM_EXPORTED ||
-			((p->node.flags & PM_LOCAL) || p->level)) &&
+	    } else if ((pmptr->binflag != PM_EXPORTED || p->level ||
+			(p->node.flags & (PM_LOCAL|PM_ARRAY|PM_HASHED))) &&
 		       (p->node.flags & pmptr->binflag))
 		doprint = 1;
 

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Peter Stephenson]

On Sat, 29 Oct 2016 11:09:04 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> That one potentially gets rather ugly; consider:
> 
> % setopt POSIXBUILTINS
> % export var
> % var=()
> zsh: var: attempt to assign array value to non-array
> 
> Is that OK with everyone?

We've generally taken the attitude that POSIX copmliance settings aren't
intended to be useful to native zsh users, just compatible for everyone
else, so what it used to do isn't a big issue.  I think this fits in
that category.

I don't think exporting arrays would be that hard, but if you want to do
it, the right way is probably to use "typeset -T" and specify which
character you want the elements to be joined with.

pws

Re: exported unset variables [was: 'export -p' lacks POSIX output]

[Bart Schaefer]

On Oct 29,  9:11am, Martijn Dekker wrote:
}
} > } (Also, is it correct/expected behaviour that zsh doesn't parse comments
} > } on the interactive command line? Blindly copying/pasting the above won't
} > } work for that reason.)
} > 
} > Yes, the option INTERACTIVE_COMMENTS has to be set to recognize comments
} > at the command line.  Should this be on in POSIX mode?
} 
} I think so. All the other shells recognise comments in interactive mode
} by default.

Hmm, it looks like this is already the case; if I invoke zsh as "ksh":

$ setopt | grep comment
nointeractivecomments off

And indeed the source gives interactivecomments the OPT_BOURNE flag.

The only other thing might be to also give it the OPT_EMULATE flag, but
I'm uncertain about that.  In what circumstances are you encountering
difficulty with this?

interactive comments [was: exported unset variables]

[Martijn Dekker]

Op 29-10-16 om 20:05 schreef Bart Schaefer:
> The only other thing might be to also give it the OPT_EMULATE flag, but
> I'm uncertain about that.  In what circumstances are you encountering
> difficulty with this?

None really, it was just a difference that slipped under my radar until now.

Yes, zsh invoked as sh or ksh already parses comments on the interactive
command line. It would seem logical to me for 'emulate sh' and 'emulate
ksh' to enable that behaviour as well.

Thanks,

- M.

Re: interactive comments [was: exported unset variables]

[Bart Schaefer]

On Oct 29,  9:20pm, Martijn Dekker wrote:
}
} Yes, zsh invoked as sh or ksh already parses comments on the interactive
} command line. It would seem logical to me for 'emulate sh' and 'emulate
} ksh' to enable that behaviour as well.

Try "emulate -R sh" if you want this.  "emulate" without the -R option
changes only those options that are likely to affect scripting, and
deliberately avoids changing options that affect only interative use.

Hence my reluctance to ad OPT_EMULATE ...