[PATCH] typeset -p doesn't report -U (unique) attribute

[PATCH] typeset -p doesn't report -U (unique) attribute

[Stephane Chazelas]

From:
https://unix.stackexchange.com/questions/470956/how-to-print-all-the-attributes-of-a-zsh-parameter


$ typeset -U a=(x x)
$ typeset -p a
typeset -a a=( x )

Expected:

$ typeset -p a
typeset -aU a=( x )


diff --git a/Src/params.c b/Src/params.c
index a1c299f60..f7ecff32a 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -5854,20 +5854,27 @@ printparamnode(HashNode hn, int printflags)
 		if ((pmptr->flags & PMTF_USE_BASE) && p->base) {
 		    printf("%d ", p->base);
 		    doneminus = 0;
 		}
 		if ((pmptr->flags & PMTF_USE_WIDTH) && p->width) {
 		    printf("%d ", p->width);
 		    doneminus = 0;
 		}
 	    }
 	}
+	if (p->node.flags & PM_UNIQUE) {
+	    if (!doneminus) {
+	      putchar('-');
+	      doneminus = 1;
+	    }
+	    putchar('U');
+	}
 	if (doneminus)
 	    putchar(' ');
     }
 
     if ((printflags & PRINT_NAMEONLY) ||
 	((p->node.flags & PM_HIDEVAL) && !(printflags & PRINT_INCLUDEVALUE))) {
 	zputs(p->node.nam, stdout);
 	putchar('\n');
     } else {
 	if (printflags & PRINT_KV_PAIR) {

Re: [PATCH] typeset -p doesn't report -U (unique) attribute

[Daniel Shahaf]

Stephane Chazelas wrote on Mon, 24 Sep 2018 22:05 +0100:
> $ typeset -p a
> typeset -aU a=( x )

I've added a test.  Next time feel free to do the honours :)

[PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Stephane Chazelas]

2018-09-24 22:05:50 +0100, Stephane Chazelas:
> From:
> https://unix.stackexchange.com/questions/470956/how-to-print-all-the-attributes-of-a-zsh-parameter
[...]

Hi, that's a follow up on my previous patch about typeset -p
to report unique parameters.

It would be useful for "typeset -p", or "typeset +m" or
${(t)param} or $parameters[param] to report the fact that the
array is tied (and how for typeset -p) including for special
ones that are created as such (like PATH to path).

See also discussion at
https://unix.stackexchange.com/questions/470956/how-to-print-all-the-attributes-of-a-zsh-parameter#comment860654_471031

This patch attempts to address that.  While writing it, I found
a number of related "issues" or possible areas of improvement,
most of them minor.  That ends up being quite a large change.  I
didn't initially intend to spend that much effort on it, I'm not
very familiar with the source code (though a lot more now than
when I started). So even though I've done quite a bit of
testing, I think it should be reviewed by someone with more
intimate knowledge of the code.

Other issues:
- typeset -T A a; echo ${(t)a}
  scalar-tag_local

  "tag_local" is a "function" thing. Changed to "-tied"

- $ typeset -T A=a:b a +; typeset -T A a; echo $a
  a:b

  Change: force an assignment upon change to join character
  (I suspect there's a better way to do that one)

- typeset +x -T VAR var doesn't honour the +x

- $ zsh -c 'manpath=(a:b c); typeset -p manpath; (){local MANPATH}; typeset -p manpath'
  typeset -a manpath=( a:b c )
  typeset -a manpath=( a b c )
  Not addressed by this patch. More generally, in those cases (same for
  tied=() vs tied=('') which cannot be preserved upon conversion
  to colonarray) it's not always clear when and how it's going
  to be "fixed", but I think we can live with it.

-  zsh 'readonly PATH; readonly -p'
  prints nothing. That's because readonly specials are not
  output. But I think the intention was for specials that are
  read-only because they are so by design as they are only be
  meant to be set by the shell (things like $#, $?, $PID...).
  However for the ones that the user set as read-only, they
  should be output by "readonly". It's quite common to make
  specials readonly (like IFS, PATH...)

- attributes of read-only params can be changed even in POSIX
  mode like in ksh. Not changed, but documented.

- $ ARGV0=sh zsh -c 'readonly A=1; readonly -p'
  typeset -r A=1

  Changed to "readonly A=1" when POSIXBUILTIN is on and not in
  ksh emulation. That one may not be necessary. The idea is to
  be able to pass the output of "export -p" or "readonly -p" to
  other POSIX shells.

  But then note:

  $ emulate sh
  $ a=$'\x80' export -p a
  export a=$'\M-\C-@'

  POSIX is currently specifying $'...', currently including \c@,
  \x80, \200, but not \C-@ nor \M-X. IMO, these days, outputting
  \x80 here would be more useful.

  http://austingroupbugs.net/view.php?id=249#c590
  still work in progress.


diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index 0141305b4..4039595df 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -2029,6 +2029,9 @@ scalar version causes a split on all separators (which cannot be quoted).
 It is possible to apply tt(-T) to two previously tied variables but with a
 different separator character, in which case the variables remain joined
 as before but the separator is changed.
+
+When an existing scalar is tied to a new array, the value of the scalar
+is preserved but no attribute other than export will be preserved.
 )
 enditem()
 
@@ -2076,12 +2079,12 @@ flag.
 )
 item(tt(-U))(
 For arrays (but not for associative arrays), keep only the first
-occurrence of each duplicated value.  This may also be set for
-colon-separated special parameters like tt(PATH) or tt(FIGNORE), etc.
-Note the flag takes effect on assignment, and the type of the
-variable being assigned to is determinative; for variables with
-shared values it is therefore recommended to set the flag for
-all interfaces, e.g. `tt(typeset -U PATH path)'.
+occurrence of each duplicated value.  This may also be set for tied
+parameters (see tt(-T)) or colon-separated special parameters like
+tt(PATH) or tt(FIGNORE), etc.  Note the flag takes effect on assignment,
+and the type of the variable being assigned to is determinative; for
+variables with shared values it is therefore recommended to set the flag
+for all interfaces, e.g. `tt(typeset -U PATH path)'.
 
 This flag has a different meaning when used with tt(-f); see below.
 )
@@ -2174,10 +2177,17 @@ be turned off.
 If the tt(POSIX_BUILTINS) option is set, the readonly attribute is
 more restrictive: unset variables can be marked readonly and cannot then
 be set; furthermore, the readonly attribute cannot be removed from any
-variable.  Note that in zsh (unlike other shells) it is still possible
-to create a local variable of the same name as this is considered a
-different variable (though this variable, too, can be marked readonly).
-)
+variable.
+
+It is still possible to change other attributes of the variable though,
+some of which like tt(-U) or tt(-Z) would affect the value. More generally,
+the readonly attribute should not be relied on as a security mechanism.
+
+Note that in zsh (like in pdksh but unlike most other shells) it is
+still possible to create a local variable of the same name as this is
+considered a different variable (though this variable, too, can be marked
+readonly). Special variables that have been made readonly retain their value
+and readonly attribute when made local.)
 item(tt(-t))(
 Tags the named parameters.  Tags have no special meaning to the shell.
 This flag has a different meaning when used with tt(-f); see above.
diff --git a/Src/Modules/db_gdbm.c b/Src/Modules/db_gdbm.c
index ed702b912..12dd839cf 100644
--- a/Src/Modules/db_gdbm.c
+++ b/Src/Modules/db_gdbm.c
@@ -809,7 +809,7 @@ myfreeparamnode(HashNode hn)
 
     zsfree(pm->node.nam);
     /* If this variable was tied by the user, ename was ztrdup'd */
-    if (pm->node.flags & PM_TIED && pm->ename) {
+    if (!(pm->node.flags & PM_SPECIAL) && pm->ename) {
         zsfree(pm->ename);
         pm->ename = NULL;
     }
diff --git a/Src/Modules/parameter.c b/Src/Modules/parameter.c
index 783c36df3..76824cf58 100644
--- a/Src/Modules/parameter.c
+++ b/Src/Modules/parameter.c
@@ -75,6 +75,8 @@ paramtypestr(Param pm)
 	    val = dyncat(val, "-readonly");
 	if (f & PM_TAGGED)
 	    val = dyncat(val, "-tag");
+	if (f & PM_TIED)
+	    val = dyncat(val, "-tied");
 	if (f & PM_EXPORTED)
 	    val = dyncat(val, "-export");
 	if (f & PM_UNIQUE)
@@ -2194,67 +2196,67 @@ static const struct gsu_array historywords_gsu =
 static struct paramdef partab[] = {
     SPECIALPMDEF("aliases", 0,
 	    &pmraliases_gsu, getpmralias, scanpmraliases),
-    SPECIALPMDEF("builtins", PM_READONLY, NULL, getpmbuiltin, scanpmbuiltins),
+    SPECIALPMDEF("builtins", PM_READONLY_SPECIAL, NULL, getpmbuiltin, scanpmbuiltins),
     SPECIALPMDEF("commands", 0, &pmcommands_gsu, getpmcommand, scanpmcommands),
     SPECIALPMDEF("dirstack", PM_ARRAY,
 	    &dirs_gsu, NULL, NULL),
     SPECIALPMDEF("dis_aliases", 0,
 	    &pmdisraliases_gsu, getpmdisralias, scanpmdisraliases),
-    SPECIALPMDEF("dis_builtins", PM_READONLY,
+    SPECIALPMDEF("dis_builtins", PM_READONLY_SPECIAL,
 	    NULL, getpmdisbuiltin, scanpmdisbuiltins),
     SPECIALPMDEF("dis_functions", 0, 
 	    &pmdisfunctions_gsu, getpmdisfunction, scanpmdisfunctions),
-    SPECIALPMDEF("dis_functions_source", PM_READONLY, NULL,
+    SPECIALPMDEF("dis_functions_source", PM_READONLY_SPECIAL, NULL,
 		 getpmdisfunction_source, scanpmdisfunction_source),
     SPECIALPMDEF("dis_galiases", 0,
 	    &pmdisgaliases_gsu, getpmdisgalias, scanpmdisgaliases),
-    SPECIALPMDEF("dis_patchars", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("dis_patchars", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &dispatchars_gsu, NULL, NULL),
-    SPECIALPMDEF("dis_reswords", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("dis_reswords", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &disreswords_gsu, NULL, NULL),
     SPECIALPMDEF("dis_saliases", 0,
 	    &pmdissaliases_gsu, getpmdissalias, scanpmdissaliases),
-    SPECIALPMDEF("funcfiletrace", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("funcfiletrace", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &funcfiletrace_gsu, NULL, NULL),
-    SPECIALPMDEF("funcsourcetrace", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("funcsourcetrace", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &funcsourcetrace_gsu, NULL, NULL),
-    SPECIALPMDEF("funcstack", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("funcstack", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &funcstack_gsu, NULL, NULL),
     SPECIALPMDEF("functions", 0, &pmfunctions_gsu, getpmfunction,
 		 scanpmfunctions),
-    SPECIALPMDEF("functions_source", PM_READONLY, NULL,
+    SPECIALPMDEF("functions_source", PM_READONLY_SPECIAL, NULL,
 		 getpmfunction_source, scanpmfunction_source),
-    SPECIALPMDEF("functrace", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("functrace", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &functrace_gsu, NULL, NULL),
     SPECIALPMDEF("galiases", 0,
 	    &pmgaliases_gsu, getpmgalias, scanpmgaliases),
-    SPECIALPMDEF("history", PM_READONLY,
+    SPECIALPMDEF("history", PM_READONLY_SPECIAL,
 	    NULL, getpmhistory, scanpmhistory),
-    SPECIALPMDEF("historywords", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("historywords", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &historywords_gsu, NULL, NULL),
-    SPECIALPMDEF("jobdirs", PM_READONLY,
+    SPECIALPMDEF("jobdirs", PM_READONLY_SPECIAL,
 	    NULL, getpmjobdir, scanpmjobdirs),
-    SPECIALPMDEF("jobstates", PM_READONLY,
+    SPECIALPMDEF("jobstates", PM_READONLY_SPECIAL,
 	    NULL, getpmjobstate, scanpmjobstates),
-    SPECIALPMDEF("jobtexts", PM_READONLY,
+    SPECIALPMDEF("jobtexts", PM_READONLY_SPECIAL,
 	    NULL, getpmjobtext, scanpmjobtexts),
-    SPECIALPMDEF("modules", PM_READONLY,
+    SPECIALPMDEF("modules", PM_READONLY_SPECIAL,
 	    NULL, getpmmodule, scanpmmodules),
     SPECIALPMDEF("nameddirs", 0,
 	    &pmnameddirs_gsu, getpmnameddir, scanpmnameddirs),
     SPECIALPMDEF("options", 0,
 	    &pmoptions_gsu, getpmoption, scanpmoptions),
-    SPECIALPMDEF("parameters", PM_READONLY,
+    SPECIALPMDEF("parameters", PM_READONLY_SPECIAL,
 	    NULL, getpmparameter, scanpmparameters),
-    SPECIALPMDEF("patchars", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("patchars", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &patchars_gsu, NULL, NULL),
-    SPECIALPMDEF("reswords", PM_ARRAY|PM_READONLY,
+    SPECIALPMDEF("reswords", PM_ARRAY|PM_READONLY_SPECIAL,
 	    &reswords_gsu, NULL, NULL),
     SPECIALPMDEF("saliases", 0,
 	    &pmsaliases_gsu, getpmsalias, scanpmsaliases),
-    SPECIALPMDEF("userdirs", PM_READONLY,
+    SPECIALPMDEF("userdirs", PM_READONLY_SPECIAL,
 	    NULL, getpmuserdir, scanpmuserdirs),
-    SPECIALPMDEF("usergroups", PM_READONLY,
+    SPECIALPMDEF("usergroups", PM_READONLY_SPECIAL,
 	    NULL, getpmusergroups, scanpmusergroups)
 };
 
diff --git a/Src/builtin.c b/Src/builtin.c
index 4abc7da35..c5b319b68 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -64,7 +64,7 @@ static struct builtin builtins[] =
     BUILTIN("enable", 0, bin_enable, 0, -1, BIN_ENABLE, "afmprs", NULL),
     BUILTIN("eval", BINF_PSPECIAL, bin_eval, 0, -1, BIN_EVAL, NULL, NULL),
     BUILTIN("exit", BINF_PSPECIAL, bin_break, 0, 1, BIN_EXIT, NULL, NULL),
-    BUILTIN("export", BINF_PLUSOPTS | BINF_MAGICEQUALS | BINF_PSPECIAL | BINF_ASSIGN, (HandlerFunc)bin_typeset, 0, -1, 0, "E:%F:%HL:%R:%TUZ:%afhi:%lp:%rtu", "xg"),
+    BUILTIN("export", BINF_PLUSOPTS | BINF_MAGICEQUALS | BINF_PSPECIAL | BINF_ASSIGN, (HandlerFunc)bin_typeset, 0, -1, BIN_EXPORT, "E:%F:%HL:%R:%TUZ:%afhi:%lp:%rtu", "xg"),
     BUILTIN("false", 0, bin_false, 0, -1, 0, NULL, NULL),
     /*
      * We used to behave as if the argument to -e was optional.
@@ -2258,6 +2258,22 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	    } else if (pm->env && !(pm->node.flags & PM_HASHELEM))
 		delenv(pm);
 	    DPUTS(ASG_ARRAYP(asg), "BUG: typeset got array value where scalar expected");
+	    if (altpm) {
+		struct tieddata* tdp = (struct tieddata *) pm->u.data;
+		if (tdp) {
+		    if (tdp->joinchar != joinchar && !asg->value.scalar) {
+			/*
+			 * Reassign the scalar to itself to do the splitting with
+			 * the new joinchar
+			 */
+			tdp->joinchar = joinchar;
+			if (!(pm = assignsparam(pname, ztrdup(getsparam(pname)), 0)))
+			    return NULL;
+		    }
+		}
+		else
+		    DPUTS(!tdp, "BUG: no join character to update");
+	    }
 	    if (asg->value.scalar &&
 		!(pm = assignsparam(pname, ztrdup(asg->value.scalar), 0)))
 		return NULL;
@@ -2325,6 +2341,9 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	    zerrnam(cname, "%s: can only have a single instance", pname);
 	    return pm;
 	}
+
+	on |= pm->node.flags & PM_TIED;
+
 	/*
 	 * For specials, we keep the same struct but zero everything.
 	 * Maybe it would be easier to create a new struct but copy
@@ -2476,7 +2495,7 @@ typeset_single(char *cname, char *pname, Param pm, UNUSED(int func),
 	return NULL;
     }
 
-    if (altpm && PM_TYPE(pm->node.flags) == PM_SCALAR) {
+    if (altpm && PM_TYPE(pm->node.flags) == PM_SCALAR && !(pm->node.flags & PM_SPECIAL)) {
 	/*
 	 * It seems safer to set this here than in createparam(),
 	 * to make sure we only ever use the colonarr functions
@@ -2646,7 +2665,17 @@ bin_typeset(char *name, char **argv, LinkList assigns, Options ops, int func)
 
     /* Given no arguments, list whatever the options specify. */
     if (OPT_ISSET(ops,'p')) {
-	printflags |= PRINT_TYPESET;
+
+	if (isset(POSIXBUILTINS) && SHELL_EMULATION() != EMULATE_KSH) {
+	  if (func == BIN_EXPORT)
+	    printflags |= PRINT_POSIX_EXPORT;
+	  else if (func == BIN_READONLY)
+	    printflags |= PRINT_POSIX_READONLY;
+	  else
+	    printflags |= PRINT_TYPESET;
+	} else
+	    printflags |= PRINT_TYPESET;
+
 	if (OPT_HASARG(ops,'p')) {
 	    char *eptr;
 	    int pflag = (int)zstrtol(OPT_ARG(ops,'p'), &eptr, 10);
@@ -2662,13 +2691,20 @@ bin_typeset(char *name, char **argv, LinkList assigns, Options ops, int func)
     }
     hasargs = *argv != NULL || (assigns && firstnode(assigns));
     if (!hasargs) {
+	int exclude = 0;
 	if (!OPT_ISSET(ops,'p')) {
 	    if (!(on|roff))
 		printflags |= PRINT_TYPE;
 	    if (roff || OPT_ISSET(ops,'+'))
 		printflags |= PRINT_NAMEONLY;
+	} else if (printflags & (PRINT_POSIX_EXPORT|PRINT_POSIX_READONLY)) {
+	    /*
+	     * For POSIX export/readonly, exclude non-scalars unless
+	     * explicitly requested.
+	     */
+	    exclude = (PM_ARRAY|PM_HASHED) & ~(on|roff);
 	}
-	scanhashtable(paramtab, 1, on|roff, 0, paramtab->printnode, printflags);
+	scanhashtable(paramtab, 1, on|roff, exclude, paramtab->printnode, printflags);
 	unqueue_signals();
 	return 0;
     }
@@ -2683,6 +2719,7 @@ bin_typeset(char *name, char **argv, LinkList assigns, Options ops, int func)
 	struct asgment asg0, asg2;
 	char *oldval = NULL, *joinstr;
 	int joinchar, nargs;
+	int already_tied = 0;
 
 	if (OPT_ISSET(ops,'m')) {
 	    zwarnnam(name, "incompatible options for -T");
@@ -2765,47 +2802,81 @@ bin_typeset(char *name, char **argv, LinkList assigns, Options ops, int func)
 	    joinchar = joinstr[1] ^ 32;
 	else
 	    joinchar = *joinstr;
-	/*
-	 * Keep the old value of the scalar.  We need to do this
-	 * here as if it is already tied to the same array it
-	 * will be unset when we retie the array.  This is all
-	 * so that typeset -T is idempotent.
-	 *
-	 * We also need to remember here whether the damn thing is
-	 * exported and pass that along.  Isn't the world complicated?
-	 */
-	if ((pm = (Param) paramtab->getnode(paramtab, asg0.name))
-	    && !(pm->node.flags & PM_UNSET)
-	    && (locallevel == pm->level || !(on & PM_LOCAL))) {
-	    if (pm->node.flags & PM_TIED) {
+
+	pm = (Param) paramtab->getnode(paramtab, asg0.name);
+	apm = (Param) paramtab->getnode(paramtab, asg->name);
+
+	if (pm && (pm->node.flags & (PM_SPECIAL|PM_TIED)) == (PM_SPECIAL|PM_TIED)) {
+	    /*
+	     * Only allow typeset -T on special tied parameters if the tied
+	     * parameter and join char are the same
+	     */
+	    if (strcmp(pm->ename, asg->name) || !(apm->node.flags & PM_SPECIAL)) {
+		zwarnnam(name, "%s special parameter can only be tied to special parameter %s", asg0.name, pm->ename);
+		unqueue_signals();
+		return 1;
+	    }
+	    if (joinchar != ':') {
+		zwarnnam(name, "cannot change the join character of special tied parameters");
 		unqueue_signals();
-		if (PM_TYPE(pm->node.flags) != PM_SCALAR) {
-		    zwarnnam(name, "already tied as non-scalar: %s", asg0.name);
-		} else if (!strcmp(asg->name, pm->ename)) {
+		return 1;
+	    }
+	    already_tied = 1;
+	} else if (apm && (apm->node.flags & (PM_SPECIAL|PM_TIED)) == (PM_SPECIAL|PM_TIED)) {
+	    /*
+	     * For the array variable, this covers attempts to tie the
+	     * array to a different scalar or to the scalar after it has
+	     * been made non-special
+	     */
+	    zwarnnam(name, "%s special parameter can only be tied to special parameter %s", asg->name, apm->ename);
+	    unqueue_signals();
+	    return 1;
+	} else if (pm) {
+	    if (!(pm->node.flags & PM_UNSET)
+		&& (locallevel == pm->level || !(on & PM_LOCAL))) {
+		if (pm->node.flags & PM_TIED) {
+		    if (PM_TYPE(pm->node.flags) != PM_SCALAR) {
+			zwarnnam(name, "already tied as non-scalar: %s", asg0.name);
+			unqueue_signals();
+			return 1;
+		    } else if (!strcmp(asg->name, pm->ename)) {
+			already_tied = 1;
+		    } else {
+			zwarnnam(name, "can't tie already tied scalar: %s",
+				 asg0.name);
+			unqueue_signals();
+			return 1;
+		    }
+		} else {
 		    /*
-		     * Already tied in the fashion requested.
+		     * Variable already exists in the current scope but is not tied.
+		     * We're preserving its value and export attribute but no other
+		     * attributes upon converting to "tied".
 		     */
-		    struct tieddata *tdp = (struct tieddata*)pm->u.data;
-		    int flags = (asg->flags & ASG_KEY_VALUE) ?
-			ASSPM_KEY_VALUE : 0;
-		    /* Update join character */
-		    tdp->joinchar = joinchar;
-		    if (asg0.value.scalar)
-			assignsparam(asg0.name, ztrdup(asg0.value.scalar), 0);
-		    else if (asg->value.array)
-			assignaparam(
-			    asg->name, zlinklist2array(asg->value.array),flags);
-		    return 0;
-		} else {
-		    zwarnnam(name, "can't tie already tied scalar: %s",
-			    asg0.name);
+		    if (!asg0.value.scalar && !asg->value.array &&
+			!(PM_TYPE(pm->node.flags) & (PM_ARRAY|PM_HASHED)))
+			oldval = ztrdup(getsparam(asg0.name));
+		    on |= (pm->node.flags & ~roff) & PM_EXPORTED;
 		}
-		return 1;
 	    }
-	    if (!asg0.value.scalar && !asg->value.array &&
-		!(PM_TYPE(pm->node.flags) & (PM_ARRAY|PM_HASHED)))
-		oldval = ztrdup(getsparam(asg0.name));
-	    on |= (pm->node.flags & PM_EXPORTED);
+	}
+	if (already_tied) {
+	    int ret;
+	    /*
+	     * If already tied, we still need to call typeset_single on
+	     * both the array and colonarray, if only to update the attributes
+	     * of both, and of course to set the new value if one is provided
+	     * for either of them.
+	     */
+	    ret = !(typeset_single(name, asg0.name, pm,
+				   func, on, off, roff, &asg0, apm,
+				   ops, joinchar) &&
+		    typeset_single(name, asg->name, apm,
+				   func, (on | PM_ARRAY) & ~PM_EXPORTED,
+				   off & ~PM_ARRAY, roff, asg, NULL, ops, 0)
+		   );
+	    unqueue_signals();
+	    return ret;
 	}
 	/*
 	 * Create the tied array; this is normal except that
@@ -2832,9 +2903,7 @@ bin_typeset(char *name, char **argv, LinkList assigns, Options ops, int func)
 	 * Create the tied colonarray.  We make it as a normal scalar
 	 * and fix up the oddities later.
 	 */
-	if (!(pm=typeset_single(name, asg0.name,
-				(Param)paramtab->getnode(paramtab,
-							 asg0.name),
+	if (!(pm=typeset_single(name, asg0.name, pm,
 				func, on, off, roff, &asg0, apm,
 				ops, joinchar))) {
 	    if (oldval)
diff --git a/Src/hashtable.h b/Src/hashtable.h
index 21398e17c..f6778664e 100644
--- a/Src/hashtable.h
+++ b/Src/hashtable.h
@@ -63,6 +63,7 @@
 #define BIN_UNALIAS  29
 #define BIN_UNFUNCTION  30
 #define BIN_UNSET    31
+#define BIN_EXPORT   32
 
 /* These currently depend on being 0 and 1. */
 #define BIN_SETOPT    0
diff --git a/Src/params.c b/Src/params.c
index f7ecff32a..089a958ae 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -290,7 +290,7 @@ static initparam special_params[] ={
 #define GSU(X) BR((GsuScalar)(void *)(&(X)))
 #define NULL_GSU BR((GsuScalar)(void *)NULL)
 #define IPDEF1(A,B,C) {{NULL,A,PM_INTEGER|PM_SPECIAL|C},BR(NULL),GSU(B),10,0,NULL,NULL,NULL,0}
-IPDEF1("#", pound_gsu, PM_READONLY),
+IPDEF1("#", pound_gsu, PM_READONLY_SPECIAL),
 IPDEF1("ERRNO", errno_gsu, PM_UNSET),
 IPDEF1("GID", gid_gsu, PM_DONTIMPORT | PM_RESTRICTED),
 IPDEF1("EGID", egid_gsu, PM_DONTIMPORT | PM_RESTRICTED),
@@ -300,11 +300,11 @@ IPDEF1("SAVEHIST", savehist_gsu, PM_RESTRICTED),
 IPDEF1("SECONDS", intseconds_gsu, 0),
 IPDEF1("UID", uid_gsu, PM_DONTIMPORT | PM_RESTRICTED),
 IPDEF1("EUID", euid_gsu, PM_DONTIMPORT | PM_RESTRICTED),
-IPDEF1("TTYIDLE", ttyidle_gsu, PM_READONLY),
+IPDEF1("TTYIDLE", ttyidle_gsu, PM_READONLY_SPECIAL),
 
 #define IPDEF2(A,B,C) {{NULL,A,PM_SCALAR|PM_SPECIAL|C},BR(NULL),GSU(B),0,0,NULL,NULL,NULL,0}
 IPDEF2("USERNAME", username_gsu, PM_DONTIMPORT|PM_RESTRICTED),
-IPDEF2("-", dash_gsu, PM_READONLY),
+IPDEF2("-", dash_gsu, PM_READONLY_SPECIAL),
 IPDEF2("histchars", histchars_gsu, PM_DONTIMPORT),
 IPDEF2("HOME", home_gsu, PM_UNSET),
 IPDEF2("TERM", term_gsu, PM_UNSET),
@@ -337,7 +337,7 @@ LCIPDEF("LC_TIME"),
 # endif
 #endif /* USE_LOCALE */
 
-#define IPDEF4(A,B) {{NULL,A,PM_INTEGER|PM_READONLY|PM_SPECIAL},BR((void *)B),GSU(varint_readonly_gsu),10,0,NULL,NULL,NULL,0}
+#define IPDEF4(A,B) {{NULL,A,PM_INTEGER|PM_READONLY_SPECIAL},BR((void *)B),GSU(varint_readonly_gsu),10,0,NULL,NULL,NULL,0}
 IPDEF4("!", &lastpid),
 IPDEF4("$", &mypid),
 IPDEF4("?", &lastval),
@@ -377,10 +377,9 @@ IPDEF7("PS3", &prompt3),
 IPDEF7R("PS4", &prompt4),
 IPDEF7("SPROMPT", &sprompt),
 
-#define IPDEF9F(A,B,C,D) {{NULL,A,D|PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT},BR((void *)B),GSU(vararray_gsu),0,0,NULL,C,NULL,0}
-#define IPDEF9(A,B,C) IPDEF9F(A,B,C,0)
-IPDEF9F("*", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY),
-IPDEF9F("@", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY),
+#define IPDEF9(A,B,C,D) {{NULL,A,D|PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT},BR((void *)B),GSU(vararray_gsu),0,0,NULL,C,NULL,0}
+IPDEF9("*", &pparams, NULL, PM_ARRAY|PM_READONLY_SPECIAL|PM_DONTIMPORT),
+IPDEF9("@", &pparams, NULL, PM_ARRAY|PM_READONLY_SPECIAL|PM_DONTIMPORT),
 
 /*
  * This empty row indicates the end of parameters available in
@@ -389,17 +388,17 @@ IPDEF9F("@", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY),
 {{NULL,NULL,0},BR(NULL),NULL_GSU,0,0,NULL,NULL,NULL,0},
 
 #define IPDEF8(A,B,C,D) {{NULL,A,D|PM_SCALAR|PM_SPECIAL},BR((void *)B),GSU(colonarr_gsu),0,0,NULL,C,NULL,0}
-IPDEF8("CDPATH", &cdpath, "cdpath", 0),
-IPDEF8("FIGNORE", &fignore, "fignore", 0),
-IPDEF8("FPATH", &fpath, "fpath", 0),
-IPDEF8("MAILPATH", &mailpath, "mailpath", 0),
-IPDEF8("WATCH", &watch, "watch", 0),
-IPDEF8("PATH", &path, "path", PM_RESTRICTED),
-IPDEF8("PSVAR", &psvar, "psvar", 0),
-IPDEF8("ZSH_EVAL_CONTEXT", &zsh_eval_context, "zsh_eval_context", PM_READONLY),
+IPDEF8("CDPATH", &cdpath, "cdpath", PM_TIED),
+IPDEF8("FIGNORE", &fignore, "fignore", PM_TIED),
+IPDEF8("FPATH", &fpath, "fpath", PM_TIED),
+IPDEF8("MAILPATH", &mailpath, "mailpath", PM_TIED),
+IPDEF8("WATCH", &watch, "watch", PM_TIED),
+IPDEF8("PATH", &path, "path", PM_RESTRICTED|PM_TIED),
+IPDEF8("PSVAR", &psvar, "psvar", PM_TIED),
+IPDEF8("ZSH_EVAL_CONTEXT", &zsh_eval_context, "zsh_eval_context", PM_READONLY_SPECIAL|PM_TIED),
 
 /* MODULE_PATH is not imported for security reasons */
-IPDEF8("MODULE_PATH", &module_path, "module_path", PM_DONTIMPORT|PM_RESTRICTED),
+IPDEF8("MODULE_PATH", &module_path, "module_path", PM_DONTIMPORT|PM_RESTRICTED|PM_TIED),
 
 #define IPDEF10(A,B) {{NULL,A,PM_ARRAY|PM_SPECIAL},BR(NULL),GSU(B),10,0,NULL,NULL,NULL,0}
 
@@ -409,7 +408,7 @@ IPDEF8("MODULE_PATH", &module_path, "module_path", PM_DONTIMPORT|PM_RESTRICTED),
  */
 
 /* All of these have sh compatible equivalents.                */
-IPDEF1("ARGC", argc_gsu, PM_READONLY),
+IPDEF1("ARGC", argc_gsu, PM_READONLY_SPECIAL),
 IPDEF2("HISTCHARS", histchars_gsu, PM_DONTIMPORT),
 IPDEF4("status", &lastval),
 IPDEF7("prompt", &prompt),
@@ -417,20 +416,20 @@ IPDEF7("PROMPT", &prompt),
 IPDEF7("PROMPT2", &prompt2),
 IPDEF7("PROMPT3", &prompt3),
 IPDEF7("PROMPT4", &prompt4),
-IPDEF8("MANPATH", &manpath, "manpath", 0),
-IPDEF9("argv", &pparams, NULL),
-IPDEF9("fignore", &fignore, "FIGNORE"),
-IPDEF9("cdpath", &cdpath, "CDPATH"),
-IPDEF9("fpath", &fpath, "FPATH"),
-IPDEF9("mailpath", &mailpath, "MAILPATH"),
-IPDEF9("manpath", &manpath, "MANPATH"),
-IPDEF9("psvar", &psvar, "PSVAR"),
-IPDEF9("watch", &watch, "WATCH"),
+IPDEF8("MANPATH", &manpath, "manpath", PM_TIED),
+IPDEF9("argv", &pparams, NULL, 0),
+IPDEF9("fignore", &fignore, "FIGNORE", PM_TIED),
+IPDEF9("cdpath", &cdpath, "CDPATH", PM_TIED),
+IPDEF9("fpath", &fpath, "FPATH", PM_TIED),
+IPDEF9("mailpath", &mailpath, "MAILPATH", PM_TIED),
+IPDEF9("manpath", &manpath, "MANPATH", PM_TIED),
+IPDEF9("psvar", &psvar, "PSVAR", PM_TIED),
+IPDEF9("watch", &watch, "WATCH", PM_TIED),
 
-IPDEF9F("zsh_eval_context", &zsh_eval_context, "ZSH_EVAL_CONTEXT", PM_READONLY),
+IPDEF9("zsh_eval_context", &zsh_eval_context, "ZSH_EVAL_CONTEXT", PM_TIED|PM_READONLY_SPECIAL),
 
-IPDEF9F("module_path", &module_path, "MODULE_PATH", PM_RESTRICTED),
-IPDEF9F("path", &path, "PATH", PM_RESTRICTED),
+IPDEF9("module_path", &module_path, "MODULE_PATH", PM_TIED|PM_RESTRICTED),
+IPDEF9("path", &path, "PATH", PM_TIED|PM_RESTRICTED),
 
 /* These are known to zsh alone. */
 
@@ -451,7 +450,7 @@ IPDEF8("MAILPATH", &mailpath, NULL, 0),
 IPDEF8("WATCH", &watch, NULL, 0),
 IPDEF8("PATH", &path, NULL, PM_RESTRICTED),
 IPDEF8("PSVAR", &psvar, NULL, 0),
-IPDEF8("ZSH_EVAL_CONTEXT", &zsh_eval_context, NULL, PM_READONLY),
+IPDEF8("ZSH_EVAL_CONTEXT", &zsh_eval_context, NULL, PM_READONLY_SPECIAL),
 
 /* MODULE_PATH is not imported for security reasons */
 IPDEF8("MODULE_PATH", &module_path, NULL, PM_DONTIMPORT|PM_RESTRICTED),
@@ -464,7 +463,7 @@ IPDEF8("MODULE_PATH", &module_path, NULL, PM_DONTIMPORT|PM_RESTRICTED),
  * and $@, this is not readonly.  This parameter is not directly
  * visible in user space.
  */
-static initparam argvparam_pm = IPDEF9F("", &pparams, NULL, \
+static initparam argvparam_pm = IPDEF9("", &pparams, NULL, \
 				 PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT);
 
 #undef BR
@@ -5024,10 +5023,10 @@ arrfixenv(char *s, char **t)
     if (!(pm->node.flags & PM_EXPORTED))
 	return;
 
-    if (pm->node.flags & PM_TIED)
-	joinchar = STOUC(((struct tieddata *)pm->u.data)->joinchar);
-    else
+    if (pm->node.flags & PM_SPECIAL)
 	joinchar = ':';
+    else
+	joinchar = STOUC(((struct tieddata *)pm->u.data)->joinchar);
 
     addenv(pm, t ? zjoin(t, joinchar, 1) : "");
 }
@@ -5650,7 +5649,7 @@ freeparamnode(HashNode hn)
 	pm->gsu.s->unsetfn(pm, 1);
     zsfree(pm->node.nam);
     /* If this variable was tied by the user, ename was ztrdup'd */
-    if (pm->node.flags & PM_TIED)
+    if (!(pm->node.flags & PM_SPECIAL))
 	zsfree(pm->ename);
     zfree(pm, sizeof(struct param));
 }
@@ -5685,7 +5684,9 @@ static const struct paramtypes pmtypes[] = {
     { PM_UPPER, "uppercase", 'u', 0},
     { PM_READONLY, "readonly", 'r', 0},
     { PM_TAGGED, "tagged", 't', 0},
-    { PM_EXPORTED, "exported", 'x', 0}
+    { PM_EXPORTED, "exported", 'x', 0},
+    { PM_UNIQUE, "unique", 'U', 0},
+    { PM_TIED, "tied", 'T', 0}
 };
 
 #define PMTYPES_SIZE ((int)(sizeof(pmtypes)/sizeof(struct paramtypes)))
@@ -5774,10 +5775,6 @@ printparamvalue(Param p, int printflags)
 	}
 	break;
     }
-    if ((printflags & (PRINT_KV_PAIR|PRINT_LINE)) == PRINT_KV_PAIR)
-	putchar(' ');
-    else if (!(printflags & PRINT_KV_PAIR))
-	putchar('\n');
 }
 
 /**/
@@ -5785,36 +5782,41 @@ mod_export void
 printparamnode(HashNode hn, int printflags)
 {
     Param p = (Param) hn;
+    Param peer = NULL;
 
     if (p->node.flags & PM_UNSET) {
-	if (isset(POSIXBUILTINS) && (p->node.flags & PM_READONLY) &&
-	    (printflags & PRINT_TYPESET))
-	{
+	if (printflags & (PRINT_POSIX_READONLY|PRINT_POSIX_EXPORT) &&
+	    p->node.flags & (PM_READONLY|PM_EXPORTED)) {
 	    /*
-	     * Special POSIX rules: show the parameter as readonly
+	     * Special POSIX rules: show the parameter as readonly/exported
 	     * even though it's unset, but with no value.
 	     */
 	    printflags |= PRINT_NAMEONLY;
 	}
-	else if (p->node.flags & PM_EXPORTED)
-	    printflags |= PRINT_NAMEONLY;
 	else
 	    return;
     }
     if (p->node.flags & PM_AUTOLOAD)
 	printflags |= PRINT_NAMEONLY;
 
-    if (printflags & PRINT_TYPESET) {
-	if ((p->node.flags & (PM_READONLY|PM_SPECIAL)) ==
-	    (PM_READONLY|PM_SPECIAL) ||
-	    (p->node.flags & PM_AUTOLOAD)) {
+    if (printflags & (PRINT_TYPESET|PRINT_POSIX_READONLY|PRINT_POSIX_EXPORT)) {
+	if (p->node.flags & (PM_RO_BY_DESIGN|PM_AUTOLOAD)) {
 	    /*
 	     * It's not possible to restore the state of
 	     * these, so don't output.
 	     */
 	    return;
 	}
-	if (locallevel && p->level >= locallevel) {
+	/*
+	 * The zsh variants of export -p/readonly -p also report other
+	 * flags to indicate other attributes or scope. The POSIX variants
+	 * don't.
+	 */
+	if (printflags & PRINT_POSIX_EXPORT) {
+	    printf("export ");
+	} else if (printflags & PRINT_POSIX_READONLY) {
+	    printf("readonly ");
+	} else if (locallevel && p->level >= locallevel) {
 	    printf("typeset ");	    /* printf("local "); */
 	} else if ((p->node.flags & PM_EXPORTED) &&
 		   !(p->node.flags & (PM_ARRAY|PM_HASHED))) {
@@ -5861,22 +5863,48 @@ printparamnode(HashNode hn, int printflags)
 		}
 	    }
 	}
-	if (p->node.flags & PM_UNIQUE) {
-	    if (!doneminus) {
-	      putchar('-');
-	      doneminus = 1;
-	    }
-	    putchar('U');
-	}
 	if (doneminus)
 	    putchar(' ');
+
+	if (p->node.flags & PM_TIED) {
+	    /*
+	     * For scalars tied to arrays,s
+	     *   * typeset +m outputs
+	     *      array tied SCALAR array
+	     *      tied array SCALAR
+	     *   * typeset -p outputs:
+	     *      typeset -T SCALAR array  (for hidden values)
+	     *      typeset -T SCALAR array=(values)
+	     *      for both scalar and array (flags may be different)
+	     *
+	     * We choose to print the value for the array instead of the scalar
+	     * as scalars can't disambiguate between
+	     * typeset -T SCALAR array=()
+	     * and
+	     * typeset -T SCALAR array=('')
+	     * (same for (a b:c)...)
+	     */
+	    Param tmp = (Param) paramtab->getnode(paramtab, p->ename);
+
+	    /*
+	     * Swap param and tied peer for typeset -p output
+	     */
+	    if (!(printflags & PRINT_TYPESET) || (p->node.flags & PM_ARRAY))
+		peer = tmp;
+	    else {
+		peer = p;
+		p = tmp;
+	    }
+
+	    quotedzputs(peer->node.nam, stdout);
+	    putchar(' ');
+	}
     }
 
     if ((printflags & PRINT_NAMEONLY) ||
-	((p->node.flags & PM_HIDEVAL) && !(printflags & PRINT_INCLUDEVALUE))) {
-	zputs(p->node.nam, stdout);
-	putchar('\n');
-    } else {
+	((p->node.flags & PM_HIDEVAL) && !(printflags & PRINT_INCLUDEVALUE)))
+	quotedzputs(p->node.nam, stdout);
+    else {
 	if (printflags & PRINT_KV_PAIR) {
 	    if (printflags & PRINT_LINE)
 		printf("\n  ");
@@ -5888,4 +5916,22 @@ printparamnode(HashNode hn, int printflags)
 
 	printparamvalue(p, printflags);
     }
+    if (peer && (printflags & PRINT_TYPESET) && !(p->node.flags & PM_SPECIAL)) {
+	/*
+	 * append the join char for tied parameters if different from colon
+	 * for typeset -p output.
+	 */
+	unsigned char joinchar = STOUC(((struct tieddata *)peer->u.data)->joinchar);
+	if (joinchar != ':') {
+	    char buf[2];
+	    buf[0] = joinchar;
+	    buf[1] = '\0';
+	    putchar(' ');
+	    quotedzputs(buf, stdout);
+	}
+    }
+    if ((printflags & (PRINT_KV_PAIR|PRINT_LINE)) == PRINT_KV_PAIR)
+	putchar(' ');
+    else if (!(printflags & PRINT_KV_PAIR))
+	putchar('\n');
 }
diff --git a/Src/subst.c b/Src/subst.c
index c1021fbf3..c706b9688 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -2552,8 +2552,8 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 		    val = dyncat(val, "-readonly");
 		if (f & PM_TAGGED)
 		    val = dyncat(val, "-tag");
-		if (f & PM_TAGGED_LOCAL)
-		    val = dyncat(val, "-tag_local");
+		if (f & PM_TIED)
+		    val = dyncat(val, "-tied");
 		if (f & PM_EXPORTED)
 		    val = dyncat(val, "-export");
 		if (f & PM_UNIQUE)
diff --git a/Src/zsh.h b/Src/zsh.h
index b81db1527..8d39a0493 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -1886,18 +1886,21 @@ struct tieddata {
 #define PM_ANONYMOUS    (1<<20) /* (function) anonymous function            */
 #define PM_LOCAL	(1<<21) /* this parameter will be made local        */
 #define PM_SPECIAL	(1<<22) /* special builtin parameter                */
-#define PM_DONTIMPORT	(1<<23)	/* do not import this variable              */
-#define PM_RESTRICTED	(1<<24) /* cannot be changed in restricted mode     */
-#define PM_UNSET	(1<<25)	/* has null value                           */
-#define PM_REMOVABLE	(1<<26)	/* special can be removed from paramtab     */
-#define PM_AUTOLOAD	(1<<27) /* autoloaded from module                   */
-#define PM_NORESTORE	(1<<28)	/* do not restore value of local special    */
-#define PM_AUTOALL	(1<<28) /* autoload all features in module
+#define PM_RO_BY_DESIGN (1<<23) /* to distinguish from specials that can be
+				   made read-only by the user               */
+#define PM_READONLY_SPECIAL (PM_SPECIAL|PM_READONLY|PM_RO_BY_DESIGN)
+#define PM_DONTIMPORT	(1<<24)	/* do not import this variable              */
+#define PM_RESTRICTED	(1<<25) /* cannot be changed in restricted mode     */
+#define PM_UNSET	(1<<26)	/* has null value                           */
+#define PM_REMOVABLE	(1<<27)	/* special can be removed from paramtab     */
+#define PM_AUTOLOAD	(1<<28) /* autoloaded from module                   */
+#define PM_NORESTORE	(1<<29)	/* do not restore value of local special    */
+#define PM_AUTOALL	(1<<29) /* autoload all features in module
 				 * when loading: valid only if PM_AUTOLOAD
 				 * is also present.
 				 */
-#define PM_HASHELEM     (1<<29) /* is a hash-element */
-#define PM_NAMEDDIR     (1<<30) /* has a corresponding nameddirtab entry    */
+#define PM_HASHELEM     (1<<30) /* is a hash-element */
+#define PM_NAMEDDIR     (1<<31) /* has a corresponding nameddirtab entry    */
 
 /* The option string corresponds to the first of the variables above */
 #define TYPESET_OPTSTR "aiEFALRZlurtxUhHTkz"
@@ -2138,6 +2141,8 @@ typedef groupset *Groupset;
 #define PRINT_INCLUDEVALUE	(1<<4)
 #define PRINT_TYPESET		(1<<5)
 #define PRINT_LINE	        (1<<6)
+#define PRINT_POSIX_EXPORT	(1<<7)
+#define PRINT_POSIX_READONLY	(1<<8)
 
 /* flags for printing for the whence builtin */
 #define PRINT_WHENCE_CSH	(1<<7)
diff --git a/Test/B02typeset.ztst b/Test/B02typeset.ztst
index b53f42f83..ac86e0ad1 100644
--- a/Test/B02typeset.ztst
+++ b/Test/B02typeset.ztst
@@ -20,6 +20,14 @@
 # Not yet tested:
 #  Assorted illegal flag combinations
 
+# For a few tests, we include a
+#    typeset -p param
+#    typeset -m param
+#    typeset +m param
+# to test the proper output of typeset for a number of different types
+# of variables. Note that we can't use a dedicated function to factorize
+# that code, as that would affect the scoping.
+
 %prep
   ## Do not remove the next line, it's used by V10private.ztst
   # test_zsh_param_private
@@ -37,6 +45,9 @@
     typeset -a array
     array=(l o c a l)
     print $scalar $array
+    typeset -p scalar array
+    typeset -m scalar array
+    typeset +m scalar array
   }
   scope01() {
     local scalar
@@ -44,6 +55,9 @@
     local -a array
     array=(l o c a l)
     print $scalar $array
+    typeset -p scalar array
+    typeset -m scalar array
+    typeset +m scalar array
   }
   scope02() {
     declare scalar
@@ -51,10 +65,16 @@
     declare -a array
     array=(l o c a l)
     print $scalar $array
+    typeset -p scalar array
+    typeset -m scalar array
+    typeset +m scalar array
   }
   scope10() {
     export outer=outer
     /bin/sh -fc 'echo $outer'
+    typeset -p outer
+    typeset -m outer
+    typeset +m outer
   }
   scope11() {
     typeset -x outer=outer
@@ -68,6 +88,9 @@
     local -xT OUTER outer
     outer=(i n n e r)
     /bin/sh -fc 'echo $OUTER'
+    typeset -p OUTER outer
+    typeset -m OUTER outer
+    typeset +m OUTER outer
   }
 
   # Bug?  `typeset -h' complains that ! # $ * - ? @ are not identifiers.
@@ -79,8 +102,14 @@
 
 %test
 
+ typeset -p scalar array
+ typeset -m scalar array
  typeset +m scalar array
-0:Report types of parameters with typeset +m
+0:Report types for global variables
+>typeset -g scalar=scalar
+>typeset -g -a array=( a r r a y )
+>scalar=scalar
+>array=( a r r a y )
 >scalar
 >array array
 
@@ -88,18 +117,36 @@
  print $scalar $array
 0:Simple local declarations
 >local l o c a l
+>typeset scalar=local
+>typeset -a array=( l o c a l )
+>scalar=local
+>array=( l o c a l )
+>local scalar
+>array local array
 >scalar a r r a y
 
  scope01
  print $scalar $array
 0:Equivalence of local and typeset in functions
 >local l o c a l
+>typeset scalar=local
+>typeset -a array=( l o c a l )
+>scalar=local
+>array=( l o c a l )
+>local scalar
+>array local array
 >scalar a r r a y
 
  scope02
  print $scalar $array
 0:Basic equivalence of declare and typeset
 >local l o c a l
+>typeset scalar=local
+>typeset -a array=( l o c a l )
+>scalar=local
+>array=( l o c a l )
+>local scalar
+>array local array
 >scalar a r r a y
 
  declare +m scalar
@@ -110,6 +157,9 @@
  print $outer
 0:Global export
 >outer
+>export outer=outer
+>outer=outer
+>outer
 >outer
 
  scope11
@@ -130,18 +180,30 @@
  print $f
  float -F f
  print $f
+ typeset -p f
+ typeset -m f
+ typeset +m f
 0:Floating point, adding a precision, and fixed point
 >float local f
 >3.14e+00
 >3.142
+>typeset -F f=3.142
+>f=3.142
+>float local f
 
  integer i=3.141
  typeset +m i
  integer -i2 i
  print $i
+ typeset -p i
+ typeset -m i
+ typeset +m i
 0:Integer and changing the base
 >integer local i
 >2#11
+>typeset -i2 i=3
+>i=3
+>integer 2 local i
 
  float -E3 f=3.141
  typeset +m f
@@ -174,16 +236,33 @@
 
  typeset -gU array
  print $array
+ typeset -p array
+ typeset -m array
+ typeset +m array
 0:Uniquified arrays and non-local scope
 >a r y
+>typeset -g -aU array=( a r y )
+>array=( a r y )
+>array unique array
 
  typeset -T SCALAR=l:o:c:a:l array
  print $array
  typeset -U SCALAR
  print $SCALAR $array
+ typeset -p SCALAR array
+ typeset -m SCALAR array
+ typeset +m SCALAR array
+ print ${(t)SCALAR} ${(t)array}
 0:Tied parameters and uniquified colon-arrays
 >l o c a l
 >l:o:c:a l o c a
+>typeset -UT SCALAR array=( l o c a )
+>typeset -aT SCALAR array=( l o c a )
+>SCALAR=l:o:c:a
+>array=( l o c a )
+>local unique tied array SCALAR
+>array local tied SCALAR array
+>scalar-local-tied-unique array-local-tied
 
  (setopt NO_multibyte cbases
  LC_ALL=C 2>/dev/null
@@ -209,9 +288,18 @@
  typeset -T SCALAR=$'l\000o\000c\000a\000l' array $'\000'
  typeset -U SCALAR
  print $array
+ typeset -p SCALAR array
+ typeset -m SCALAR array
+ typeset +m SCALAR array
  [[ $SCALAR == $'l\000o\000c\000a' ]]
 0:Tied parameters and uniquified arrays with NUL-character as separator
 >l o c a
+>typeset -UT SCALAR array=( l o c a ) ''
+>typeset -aT SCALAR array=( l o c a ) ''
+>SCALAR=$'l\C-@o\C-@c\C-@a'
+>array=( l o c a )
+>local unique tied array SCALAR
+>array local tied SCALAR array
 
  typeset -T SCALAR array
  typeset +T SCALAR
@@ -223,15 +311,30 @@
  print $OUTER
 0:Export of tied parameters
 >i:n:n:e:r
+>typeset -xT OUTER outer=( i n n e r )
+>typeset -aT OUTER outer=( i n n e r )
+>OUTER=i:n:n:e:r
+>outer=( i n n e r )
+>local exported tied outer OUTER
+>array local tied OUTER outer
 >outer
 
  typeset -TU MORESTUFF=here-we-go-go-again morestuff '-'
  print -l $morestuff
+ typeset -p MORESTUFF morestuff
+ typeset -m MORESTUFF morestuff
+ typeset +m MORESTUFF morestuff
 0:Tied arrays with separator specified
 >here
 >we
 >go
 >again
+>typeset -UT MORESTUFF morestuff=( here we go again ) -
+>typeset -aUT MORESTUFF morestuff=( here we go again ) -
+>MORESTUFF=here-we-go-again
+>morestuff=( here we go again )
+>local unique tied morestuff MORESTUFF
+>array local unique tied MORESTUFF morestuff
 
  typeset -T THIS will not work
 1:Tied array syntax
@@ -251,13 +354,25 @@
  local b=1 ;: to stomp assoc[1] if assoc[b] is broken
  typeset assoc[1]=a assoc[b]=2 assoc[3]=c
  print $assoc[1] $assoc[b] $assoc[3]
+ typeset -p assoc
+ typeset -m assoc
+ typeset +m assoc
 0:Legal local associative array element assignment
 >a 2 c
+>typeset -A assoc=( [1]=a [3]=c [b]=2 )
+>assoc=( [1]=a [3]=c [b]=2 )
+>association local assoc
 
  local scalar scalar[1]=a scalar[2]=b scalar[3]=c
  print $scalar
+ typeset -p scalar
+ typeset -m scalar
+ typeset +m scalar
 0:Local scalar subscript assignment
 >abc
+>typeset scalar=abc
+>scalar=abc
+>local scalar
 
  typeset -L 10 fools
  for fools in "   once" "twice"  "      thrice" "   oops too long here"; do
@@ -273,11 +388,17 @@
  for foolf in 1.3 4.6 -2.987 -4.91031; do
    print "'$foolf'"
  done
+ typeset -p foolf
+ typeset -m foolf
+ typeset +m foolf
 0:Left justification of floating point
 >'1.300     '
 >'4.600     '
 >'-2.987    '
 >'-4.910    '
+>typeset -FL10 foolf=-4.910
+>foolf=-4.910
+>float local left justified 10 foolf
 
  typeset -L 10 -Z foolzs
  for foolzs in 001.3 04.6 -2.987 -04.91231; do
@@ -293,10 +414,16 @@
  for foors in short longer even-longer; do
    print "'$foors'"
  done
+ typeset -p foors
+ typeset -m foors
+ typeset +m foors
 0:Right justification of scalars
 >'     short'
 >'    longer'
 >'ven-longer'
+>typeset -R10 foors=even-longer
+>foors=even-longer
+>local right justified 10 foors
 
  typeset -Z 10 foozs
  for foozs in 42 -42 " 43" " -43"; do
@@ -436,24 +563,36 @@
  print $case1
  upper="VALUE OF \$UPPER"
  print ${(P)case1}
+ typeset -p case1
+ typeset -m case1
+ typeset +m case1
 0:Upper case conversion, does not apply to values used internally
 >UPPER
 >VALUE OF $UPPER
+>typeset -u case1=upper
+>case1=upper
+>local uppercase case1
 
  local case2=LOWER
  typeset -l case2
  print $case2
  LOWER="value of \$lower"
  print ${(P)case2}
+ typeset -p case2
+ typeset -m case2
+ typeset +m case2
 0:Lower case conversion, does not apply to values used internally
 >lower
 >value of $lower
+>typeset -l case2=LOWER
+>case2=LOWER
+>local lowercase case2
 
  typeset -a array
  array=(foo bar)
  fn() { typeset -p array nonexistent; }
  fn
-1:declare -p shouldn't create scoped values
+1:typeset -p shouldn't create scoped values
 >typeset -g -a array=( foo bar )
 ?fn:typeset: no such variable: nonexistent
 
@@ -490,7 +629,7 @@
 ?0
 ?(eval):5: read-only variable: pbro
 ?(eval):6: read-only variable: pbro
-?typeset -g -r pbro
+?readonly pbro
 ?0
 ?(eval):10: read-only variable: pbro
 
@@ -820,7 +959,145 @@
 >  [three]=''
 >)
 
- (typeset -a -U foo=(bar bar)
- typeset -p foo)
-0:typeset -p of typeset -U
->typeset -aU foo=( bar )
+ (export PATH MANPATH
+ path=(/bin)
+ MANPATH=/
+ # read-only special params like zsh_eval_context are not output by typeset -p
+ specials=(path PATH manpath MANPATH zsh_eval_context ZSH_EVAL_CONTEXT)
+ typeset -p $specials
+ typeset -m $specials
+ typeset +m $specials
+ for var ($specials) print $var: ${(Pt)var}
+ )
+0:typeset output for some special tied parameters
+>typeset -g -aT PATH path=( /bin )
+>export -T PATH path=( /bin )
+>typeset -g -aT MANPATH manpath=( / )
+>export -T MANPATH manpath=( / )
+>path=( /bin )
+>PATH=/bin
+>manpath=( / )
+>MANPATH=/
+>zsh_eval_context=( toplevel shfunc shfunc shfunc eval )
+>ZSH_EVAL_CONTEXT=toplevel:shfunc:shfunc:shfunc:eval
+>array tied PATH path
+>tied path PATH
+>array tied MANPATH manpath
+>tied manpath MANPATH
+>array readonly tied ZSH_EVAL_CONTEXT zsh_eval_context
+>readonly tied zsh_eval_context ZSH_EVAL_CONTEXT
+>path: array-tied-special
+>PATH: scalar-tied-export-special
+>manpath: array-tied-special
+>MANPATH: scalar-tied-export-special
+>zsh_eval_context: array-readonly-tied-special
+>ZSH_EVAL_CONTEXT: scalar-readonly-tied-special
+
+ typeset -T VAR var=(a b a b)
+ typeset -UuT VAR var +
+ print $VAR
+0:redeclare a tied variable with different attributes
+>A+B
+
+ typeset -T VAR=a+b var
+ typeset -T VAR var +
+ print $var
+0:colonarray re-split when changing the join character
+>a b
+
+ readonly -T VAR var=(a b)
+ readonly -T VAR var +
+1:cannot change the join character on a readonly tied variable
+?(eval):1: read-only variable: var
+
+ typeset -T FOO manpath
+1:Can't tie a special tied array to a different variable
+?(eval):typeset:1: manpath special parameter can only be tied to special parameter MANPATH
+
+ typeset -T MANPATH foo
+1:Can't tie a special tied scalar to a different variable
+?(eval):typeset:1: MANPATH special parameter can only be tied to special parameter manpath
+
+ typeset -T MANPATH manpath +
+1:Can't change the join character of a special tied variable
+?(eval):typeset:1: cannot change the join character of special tied parameters
+
+ (){
+   typeset -h path
+   typeset -T PATH path=(x)
+ }
+ (){
+   typeset -h PATH
+   typeset -T PATH path=(x)
+ }
+1:reject attempt to tie special to downgraded peer
+?(anon):typeset:2: PATH special parameter can only be tied to special parameter path
+?(anon):typeset:2: path special parameter can only be tied to special parameter PATH
+
+ typeset MANPATH
+ manpath=(/ /)
+ typeset -UT MANPATH manpath
+ print $manpath
+0:OK to run typeset -T on tied specials as long as peer and joinchar are unchanged
+>/
+
+ typeset FOO=a:b
+ export FOO
+ typeset +x -T FOO foo
+ typeset -p FOO
+0:Make sure +x is honoured when tying a parameter
+>typeset -T FOO foo=( a b )
+
+ $ZTST_testdir/../Src/zsh --emulate sh -f -c '
+ PATH=/bin; export PATH; readonly PATH
+ export -p PATH
+ typeset -p PATH
+ readonly -p'
+0: readonly/export output for exported+readonly+special when started as sh
+>export PATH=/bin
+>export -r PATH=/bin
+>readonly PATH=/bin
+
+ function {
+ emulate -L sh
+ MANPATH=/bin; export MANPATH; readonly MANPATH
+ export -p MANPATH
+ typeset -p MANPATH
+ readonly -p
+ }
+0: readonly/export output for exported+readonly+tied+special after switching to sh emulation
+>export MANPATH=/bin
+>export -rT MANPATH manpath=( /bin )
+>readonly MANPATH=/bin
+
+ function {
+   local -rax zsh_exported_readonly_array=(2)
+   local -rAx zsh_exported_readonly_hash=(3 3)
+   local -rx zsh_exported_readonly_scalar=1
+   print zsh:
+   export -p | grep zsh_exported_readonly
+   readonly -p | grep zsh_exported_readonly
+   print sh:
+   emulate -L sh
+   export -p | grep zsh_exported_readonly
+   readonly -p | grep zsh_exported_readonly
+   print still asking for arrays:
+   export -ap | grep zsh_exported_readonly
+   readonly -ap | grep zsh_exported_readonly
+ }
+0: no array/hash in POSIX export/readonly -p
+>zsh:
+>typeset -arx zsh_exported_readonly_array=( 2 )
+>typeset -Arx zsh_exported_readonly_hash=( [3]=3 )
+>typeset -rx zsh_exported_readonly_scalar=1
+>typeset -arx zsh_exported_readonly_array=( 2 )
+>typeset -Arx zsh_exported_readonly_hash=( [3]=3 )
+>typeset -rx zsh_exported_readonly_scalar=1
+>sh:
+>export zsh_exported_readonly_scalar=1
+>readonly zsh_exported_readonly_scalar=1
+>still asking for arrays:
+>export zsh_exported_readonly_array=( 2 )
+>export zsh_exported_readonly_scalar=1
+>readonly zsh_exported_readonly_array=( 2 )
+>readonly zsh_exported_readonly_scalar=1

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Sun, 2018-10-07 at 14:35 +0100, Stephane Chazelas wrote:
> 2018-09-24 22:05:50 +0100, Stephane Chazelas:
> It would be useful for "typeset -p", or "typeset +m" or
> ${(t)param} or $parameters[param] to report the fact that the
> array is tied (and how for typeset -p) including for special
> ones that are created as such (like PATH to path).
>
> This patch attempts to address that.  While writing it, I found
> a number of related "issues" or possible areas of improvement,
> most of them minor.  That ends up being quite a large change.  I
> didn't initially intend to spend that much effort on it, I'm not
> very familiar with the source code (though a lot more now than
> when I started). So even though I've done quite a bit of
> testing, I think it should be reviewed by someone with more
> intimate knowledge of the code.

Thanks, I did't see anything obviously weird and it's well tested.  I'm
inclined to commit it and see what happens.  Any problems are most
likely to be combinations / juxtapositions that nobody's thought about.

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 09:58 -0400, Vin Shelton wrote:
> I'm now seeing 2 failures on my Sparky (Debian) system.  I'm not seeing
> these failures on my Fedora28 VM.

Hmm, that looks like it's got something to do with your particular set
up, which isn't to see there's no problem in the build/test system
either.  In one case there's no good reason why you should get the wrong
status, and I don't think it's down to the basic shell internals (it
could be a knock-on effect from the test, e.g. what's in the
environment).  In the other it seems to think you're running as
root.  Is this from a clean build?

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Daniel Shahaf]

Peter Stephenson wrote on Mon, 08 Oct 2018 15:10 +0100:
> On Mon, 2018-10-08 at 09:58 -0400, Vin Shelton wrote:
> > I'm now seeing 2 failures on my Sparky (Debian) system.  I'm not seeing
> > these failures on my Fedora28 VM.
> 
> Hmm, that looks like it's got something to do with your particular set
> up, which isn't to see there's no problem in the build/test system
> either.  In one case there's no good reason why you should get the wrong
> status, and I don't think it's down to the basic shell internals (it
> could be a knock-on effect from the test, e.g. what's in the
> environment).  In the other it seems to think you're running as
> root.  Is this from a clean build?

I see a failure in a clean build on Debian stretch

[[[
Test /home/daniel/src/zsh/./Test/V10private.ztst failed: bad status 139, expected 0 from:
 if (( UID )); then
   ZTST_verbose=0 $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh private.TMP/B02
 else
   ZTST_skip="cannot re-run typeset tests when tests run as superuser"
 fi
Was testing: typeset still works with zsh/param/private module loaded
]]]

It then says "2 failures, 2 skipped" even though no other failures are
listed in the output.

This is with --enable-zsh-debug and modules compiled statically, if that matters.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 14:28 +0000, Daniel Shahaf wrote:
> Peter Stephenson wrote on Mon, 08 Oct 2018 15:10 +0100:
> I see a failure in a clean build on Debian stretch
> 
> [[[
> Test /home/daniel/src/zsh/./Test/V10private.ztst failed: bad status 139,
> expected 0 from:
>  if (( UID )); then
>    ZTST_verbose=0 $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh private.TMP/B02
>  else
>    ZTST_skip="cannot re-run typeset tests when tests run as superuser"
>  fi
> Was testing: typeset still works with zsh/param/private module loaded
> ]]]
> 
> It then says "2 failures, 2 skipped" even though no other failures are
> listed in the output.

Hmm, I'm guessing all the B02 failures are all down to being re-run from
V10?  That should be obvious from the output --- the first run of B02 is
much earlier.  Is that failing for anyone?  (This needs confirming.)

> This is with --enable-zsh-debug and modules compiled statically, if that
> matters.

Then there's even more you're not telling me --- V10 doesn't get
compiled in and would be skipped by default in this case.  However, I
updated the configuration and I'm still not seeing any problems myself.

As I'm not seeing problems, it's over to others --- include very much
fuller information if you want my help.  You'll need to narrow down
which tests are failing.  I suggest then running the code by hand ---
it's not clear from the above few lines what's failing there, so the key
may lie in private.TMP/B02.

There's a reasonable chance valgrind may help with the 139 status.  I
didn't get any hints from adding it to the combined V10 / B02 test,
however.

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Daniel Shahaf]

Peter Stephenson wrote on Mon, Oct 08, 2018 at 16:24:04 +0100:
> On Mon, 2018-10-08 at 14:28 +0000, Daniel Shahaf wrote:
> > Peter Stephenson wrote on Mon, 08 Oct 2018 15:10 +0100:
> > I see a failure in a clean build on Debian stretch
> > 
> > [[[
> > Test /home/daniel/src/zsh/./Test/V10private.ztst failed: bad status 139,
> > expected 0 from:
> >  if (( UID )); then
> >    ZTST_verbose=0 $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh private.TMP/B02
> >  else
> >    ZTST_skip="cannot re-run typeset tests when tests run as superuser"
> >  fi
> > Was testing: typeset still works with zsh/param/private module loaded
> > ]]]
> > 
> > It then says "2 failures, 2 skipped" even though no other failures are
> > listed in the output.
> 
> Hmm, I'm guessing all the B02 failures are all down to being re-run from
> V10?  That should be obvious from the output --- the first run of B02 is
> much earlier.  Is that failing for anyone?  (This needs confirming.)
> 

Yes.

% make -s check TESTNUM=B02 ZTST_verbose=2 | tail 
ZTST_test: all tests successful
**************************************
0 successful test scripts, 1 failure, 0 skipped
**************************************
Makefile:187: recipe for target 'check' failed

> > This is with --enable-zsh-debug and modules compiled statically, if that
> > matters.
> 
> Then there's even more you're not telling me --- V10 doesn't get
> compiled in and would be skipped by default in this case.  However, I
> updated the configuration and I'm still not seeing any problems myself.
> 

I don't know what you mean.  V10 normally runs and succeeds in my build.

> There's a reasonable chance valgrind may help with the 139 status.  I
> didn't get any hints from adding it to the combined V10 / B02 test,
> however.

I did.  If someone wants to look into it (not necessarly Peter), be my guest.

==25895== Invalid read of size 8
==25895==    at 0x4823AE: zshlex (lex.c:280)
==25895==    by 0x4ACBC0: parse_event (parse.c:581)
==25895==    by 0x47109D: loop (init.c:146)
==25895==    by 0x47659B: zsh_main (init.c:1757)
==25895==    by 0x420201: main (main.c:93)
==25895==  Address 0x3a is not stack'd, malloc'd or (recently) free'd
==25895== 
==25895== 
==25895== Process terminating with default action of signal 11 (SIGSEGV)
==25895==  Access not within mapped region at address 0x3A
==25895==    at 0x4823AE: zshlex (lex.c:280)
==25895==    by 0x4ACBC0: parse_event (parse.c:581)
==25895==    by 0x47109D: loop (init.c:146)
==25895==    by 0x47659B: zsh_main (init.c:1757)
==25895==    by 0x420201: main (main.c:93)
==25895==  If you believe this happened as a result of a stack
==25895==  overflow in your program's main thread (unlikely but
==25895==  possible), you can try to increase the size of the
==25895==  main thread stack using the --main-stacksize= flag.
==25895==  The main thread stack size used in this run was 8388608.
==25895== 
==25895== HEAP SUMMARY:
==25895==     in use at exit: 238,978 bytes in 8,046 blocks
==25895==   total heap usage: 229,079 allocs, 221,033 frees, 12,615,547 bytes allocated
==25895== 
==25895== LEAK SUMMARY:
==25895==    definitely lost: 0 bytes in 0 blocks
==25895==    indirectly lost: 0 bytes in 0 blocks
==25895==      possibly lost: 0 bytes in 0 blocks
==25895==    still reachable: 238,978 bytes in 8,046 blocks
==25895==         suppressed: 0 bytes in 0 blocks
==25895== Reachable blocks (those to which a pointer was found) are not shown.
==25895== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==25895== 
==25895== For counts of detected and suppressed errors, rerun with: -v
==25895== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
zsh: segmentation fault  valgrind --leak-check=full ../Src/zsh +Z -f ~/src/zsh/Test/ztst.zsh 

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Mon, Oct 8, 2018 at 8:49 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Peter Stephenson wrote on Mon, Oct 08, 2018 at 16:24:04 +0100:
> > Then there's even more you're not telling me --- V10 doesn't get
> > compiled in and would be skipped by default in this case.  However, I
> > updated the configuration and I'm still not seeing any problems myself.
>
> I don't know what you mean.  V10 normally runs and succeeds in my build.

He means that the param_private module isn't included in a default
build, so you have done something (edited config.modules, probably) to
enable it.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Daniel Shahaf]

Bart Schaefer wrote on Mon, 08 Oct 2018 08:57 -0700:
> On Mon, Oct 8, 2018 at 8:49 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> >
> > Peter Stephenson wrote on Mon, Oct 08, 2018 at 16:24:04 +0100:
> > > Then there's even more you're not telling me --- V10 doesn't get
> > > compiled in and would be skipped by default in this case.  However, I
> > > updated the configuration and I'm still not seeing any problems myself.
> >
> > I don't know what you mean.  V10 normally runs and succeeds in my build.
> 
> He means that the param_private module isn't included in a default
> build, so you have done something (edited config.modules, probably) to
> enable it.

On my machine, configure generates a link=dynamic auto=yes load=yes line
for zsh/param/private.  I do change that link to link=static auto=no, but
you seem to imply it should've generated that line with link=no?

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Mon, Oct 8, 2018 at 9:06 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Bart Schaefer wrote on Mon, 08 Oct 2018 08:57 -0700:
> > [PWS] means that the param_private module isn't included in a default
> > build, so you have done something (edited config.modules, probably) to
> > enable it.
>
> On my machine, configure generates a link=dynamic auto=yes load=yes line
> for zsh/param/private.  I do change that link to link=static auto=no, but
> you seem to imply it should've generated that line with link=no?

Well, that's what Peter was implying, I was just clarifying.  However,
I think you're right that zsh/param/private normally is available as a
dynamic module.

I think all this stuff about V10 is actually a goose chase, and it's
B02 that is the real point of failure.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 09:52 -0700, Bart Schaefer wrote:
> On Mon, Oct 8, 2018 at 9:06 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> > Bart Schaefer wrote on Mon, 08 Oct 2018 08:57 -0700:
> > > [PWS] means that the param_private module isn't included in a default
> > > build, so you have done something (edited config.modules, probably) to
> > > enable it.
> > On my machine, configure generates a link=dynamic auto=yes load=yes line
> > for zsh/param/private.  I do change that link to link=static auto=no, but
> > you seem to imply it should've generated that line with link=no?
> Well, that's what Peter was implying, I was just clarifying.  However,
> I think you're right that zsh/param/private normally is available as a
> dynamic module.

Yes, all I was saying is that you'd changed the configuration in a way
you hadn't mentioned, which you've now confirmed.  No reason to suppose
this is actually related.

> I think all this stuff about V10 is actually a goose chase, and it's
> B02 that is the real point of failure.

If that's failing on it's own, that's certainly the one to focus on
first, and highly likely the other will go away.

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Daniel Shahaf]

Daniel Shahaf wrote on Mon, 08 Oct 2018 15:49 +0000:
> I did.  If someone wants to look into it (not necessarly Peter), be my guest.

It's crashing on the 'print $manpath' line in B02.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 16:02 +0000, Daniel Shahaf wrote:
> Daniel Shahaf wrote on Mon, 08 Oct 2018 15:49 +0000:
> > 
> > I did.  If someone wants to look into it (not necessarly Peter), be my
> > guest.
> It's crashing on the 'print $manpath' line in B02.

Next step is to see if that code chunk on its own is responsible,

 typeset MANPATH
 manpath=(/ /)
 typeset -UT MANPATH manpath
 print $manpath

--- that's going to need to be within a function to have the right
effect, so

() {
  typeset MANPATH
  manpath=(/ /)
  typeset -UT MANPATH manpath
  print $manpath
}

--- or if there's some precursor, and if the latter whether $manpath is
accessible without a crash at the end of the previous test.  If yes to
both it's going to be interesting, but that seems unlikely.  If there's
a precursor hopefully it's some previous assignment to manpath or
MANPATH that can be tracked down and added to the above by hand.
Printing out "$manpath $MANPATH ${(t)manpath} ${(t)MANPATH}" at the top
of the test (before it starts crashing) might or might not help.

Presumably, then, the bin_typeset call is leaving manpath in a bad
state.  This is certainly a somewhat hybrid case as manpath is a special
while -T is for user tying --- so this could be down to consistently
making sure the specialness stays.  It's not clear from what I've seen
so far if the valgrind error is actually within the above code.

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 15:49 +0000, Daniel Shahaf wrote:
> Peter Stephenson wrote on Mon, Oct 08, 2018 at 16:24:04 +0100:
> > There's a reasonable chance valgrind may help with the 139 status.  I
> > didn't get any hints from adding it to the combined V10 / B02 test,
> > however.
> I did.  If someone wants to look into it (not necessarly Peter), be my
> guest.
> 
> ==25895== Invalid read of size 8
> ==25895==    at 0x4823AE: zshlex (lex.c:280)
> ==25895==    by 0x4ACBC0: parse_event (parse.c:581)
> ==25895==    by 0x47109D: loop (init.c:146)
> ==25895==    by 0x47659B: zsh_main (init.c:1757)
> ==25895==    by 0x420201: main (main.c:93)
> ==25895==  Address 0x3a is not stack'd, malloc'd or (recently) free'd

If your lex.c is like mine, I don't think this is telling us anything
directly relevant --- presumably the damage has been done.  That's
dereferencing the list of here documents previously built up, which is
nothing to do with the change.

pws

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Daniel Shahaf]

Peter Stephenson wrote on Mon, 08 Oct 2018 17:43 +0100:
> On Mon, 2018-10-08 at 15:49 +0000, Daniel Shahaf wrote:
> > Peter Stephenson wrote on Mon, Oct 08, 2018 at 16:24:04 +0100:
> > > There's a reasonable chance valgrind may help with the 139 status.  I
> > > didn't get any hints from adding it to the combined V10 / B02 test,
> > > however.
> > I did.  If someone wants to look into it (not necessarly Peter), be my
> > guest.
> > 
> > ==25895== Invalid read of size 8
> > ==25895==    at 0x4823AE: zshlex (lex.c:280)
> > ==25895==    by 0x4ACBC0: parse_event (parse.c:581)
> > ==25895==    by 0x47109D: loop (init.c:146)
> > ==25895==    by 0x47659B: zsh_main (init.c:1757)
> > ==25895==    by 0x420201: main (main.c:93)
> > ==25895==  Address 0x3a is not stack'd, malloc'd or (recently) free'd
> 
> If your lex.c is like mine, I don't think this is telling us anything
> directly relevant --- presumably the damage has been done.  That's
> dereferencing the list of here documents previously built up, which is
> nothing to do with the change.

So the lex.c error may be an independent issue.  The link to the typeset
change is probably the failure address, 0x3A, which is the hex value of
the ASCII colon character, which is the joinchar of $MANPATH.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Mon, Oct 8, 2018 at 9:59 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> So the lex.c error may be an independent issue.  The link to the typeset
> change is probably the failure address, 0x3A, which is the hex value of
> the ASCII colon character, which is the joinchar of $MANPATH.

This suggests that something is building $manpath by making a list of
pointers into the string for $MANPATH, and then trying to free those
pointers [possible via freearray()] when unsetting $manpath, or
something like that.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Stephane Chazelas]

Thanks everyone for looking into this. I had forgotten about
"private" and hadn't realised it was not built in by default
(not sure how it gets enabled in the debian build either).
I was not aware of the valgrind testing facilities either.

As I spent quite some time to try and grasp the typeset
framework, it's probably going to be easier for me (as it's fresh
in mind) to fix the remaining issues.

Unfortunatly I'm off on holidays on Wednesday for a week and a
half. I'm happy to look into this when I'm back and provide with
an update to the patch then, if nobody's keen to delve into this
by then. It's not like those were critical issues.

Cheers,
Stephane.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Mon, 2018-10-08 at 16:58 +0000, Daniel Shahaf wrote:
> Peter Stephenson wrote on Mon, 08 Oct 2018 17:43 +0100:
> So the lex.c error may be an independent issue.  The link to the typeset
> change is probably the failure address, 0x3A, which is the hex value of
> the ASCII colon character, which is the joinchar of $MANPATH.

That's interesting and points to a problem along the lines I suggested
--- confusion between special and non-special tied variables.  MANPATH
as a special tied variable uses colonarrsetfn() / colonarrgetfn() to set
and retrieve values, which uses a colon implicitly, not from the
parameter structure.  Non-special tied arrays have a tieddata structure
in the data.

I didn't get any reaction from the following, however.

pws

diff --git a/Src/params.c b/Src/params.c
index 089a958..b1420f7 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -4058,6 +4058,8 @@ char *
 colonarrgetfn(Param pm)
 {
     char ***dptr = (char ***)pm->u.data;
+    DPUTS(!(pm->node.flags & PM_SPECIAL),
+	  "Wrong get fn for npn-special tied array");
     return *dptr ? zjoin(*dptr, ':', 1) : "";
 }
 
@@ -4066,6 +4068,8 @@ void
 colonarrsetfn(Param pm, char *x)
 {
     char ***dptr = (char ***)pm->u.data;
+    DPUTS(!(pm->node.flags & PM_SPECIAL),
+	  "Wrong set fn for npn-special tied array");
     /*
      * We have to make sure this is never NULL, since that
      * can cause problems.
@@ -4085,6 +4089,7 @@ char *
 tiedarrgetfn(Param pm)
 {
     struct tieddata *dptr = (struct tieddata *)pm->u.data;
+    DPUTS(pm->node.flags & PM_SPECIAL, "Wrong get fn for special tied
array");
     return *dptr->arrptr ? zjoin(*dptr->arrptr, STOUC(dptr->joinchar), 1) :
"";
 }
 
@@ -4093,6 +4098,7 @@ void
 tiedarrsetfn(Param pm, char *x)
 {
     struct tieddata *dptr = (struct tieddata *)pm->u.data;
+    DPUTS(pm->node.flags & PM_SPECIAL, "Wrong set fn for special tied
array");
 
     if (*dptr->arrptr)
 	freearray(*dptr->arrptr);

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Tue, 2018-10-09 at 10:02 +0100, Peter Stephenson wrote:
> On Mon, 2018-10-08 at 16:58 +0000, Daniel Shahaf wrote:
> > Peter Stephenson wrote on Mon, 08 Oct 2018 17:43 +0100:
> > So the lex.c error may be an independent issue.  The link to the typeset
> > change is probably the failure address, 0x3A, which is the hex value of
> > the ASCII colon character, which is the joinchar of $MANPATH.
> That's interesting and points to a problem along the lines I suggested
> --- confusion between special and non-special tied variables.  MANPATH
> as a special tied variable uses colonarrsetfn() / colonarrgetfn() to set
> and retrieve values, which uses a colon implicitly, not from the
> parameter structure.  Non-special tied arrays have a tieddata structure
> in the data.

I wonder if this has anything to do with it?  I haven't definitely
verified we can get here in the case newly tested but it's not obvious
we can't.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index b81acdb..ca3ef23 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2258,7 +2258,7 @@ typeset_single(char *cname, char *pname, Param pm,
UNUSED(int func),
 	    } else if (pm->env && !(pm->node.flags & PM_HASHELEM))
 		delenv(pm);
 	    DPUTS(ASG_ARRAYP(asg), "BUG: typeset got array value where scalar
expected");
-	    if (altpm) {
+	    if (altpm && !(pm->node.flags & PM_SPECIAL)) {
 		struct tieddata* tdp = (struct tieddata *) pm->u.data;
 		if (tdp) {
 		    if (tdp->joinchar != joinchar && !asg->value.scalar) {

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Tue, Oct 9, 2018 at 2:44 AM Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> I wonder if this has anything to do with it?
>
> -           if (altpm) {
> +           if (altpm && !(pm->node.flags & PM_SPECIAL)) {

That has remediated the test failure on CentOS 6.10, at least.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

... and has fixed both of the test failures I was experiencing on Sparky
Linux.

  - Vin

On Tue, Oct 9, 2018 at 12:52 PM Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Tue, Oct 9, 2018 at 2:44 AM Peter Stephenson
> <p.stephenson@samsung.com> wrote:
> >
> > I wonder if this has anything to do with it?
> >
> > -           if (altpm) {
> > +           if (altpm && !(pm->node.flags & PM_SPECIAL)) {
>
> That has remediated the test failure on CentOS 6.10, at least.
>

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Mon, Oct 8, 2018 at 8:24 AM Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> Hmm, I'm guessing all the B02 failures are all down to being re-run from
> V10?

No, they're coming from the first pass through B02.  Succeeds for me
on MacOS, but on CentOS 6.10 it fails:

Test ./B02typeset.ztst failed: bad status 1, expected 0 from:
 function {
   local -rax zsh_exported_readonly_array=(2)
   local -rAx zsh_exported_readonly_hash=(3 3)
   local -rx zsh_exported_readonly_scalar=1
   print zsh:
   export -p | grep zsh_exported_readonly
   readonly -p | grep zsh_exported_readonly
   print sh:
   emulate -L sh
   export -p | grep zsh_exported_readonly
   readonly -p | grep zsh_exported_readonly
   print still asking for arrays:
   export -ap | grep zsh_exported_readonly
   readonly -ap | grep zsh_exported_readonly
 }
Was testing:  no array/hash in POSIX export/readonly -p

There is no other useful output, even with ZTST_VERBOSE.

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Bart Schaefer]

On Mon, Oct 8, 2018 at 8:54 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
>  function {
>    local -rax zsh_exported_readonly_array=(2)
>    local -rAx zsh_exported_readonly_hash=(3 3)
>    local -rx zsh_exported_readonly_scalar=1
>    print zsh:
>    export -p | grep zsh_exported_readonly
>    readonly -p | grep zsh_exported_readonly
>    print sh:
>    emulate -L sh
>    export -p | grep zsh_exported_readonly
>    readonly -p | grep zsh_exported_readonly
>    print still asking for arrays:
>    export -ap | grep zsh_exported_readonly
>    readonly -ap | grep zsh_exported_readonly
>  }

This runs to successful completion from an interactive Src/zsh -f.  So
do the couple of tests immediately before it.   I suspect it's some
kind of cascading effect from the series of tests in B02, but I'm not
sure how far I back one has to go.

local read-only variables (Was:[PATCH] [long] typeset doesn't report tied parameters (and related issues))

[Stephane Chazelas]

2018-10-07 14:35:46 +0100, Stephane Chazelas:
[...]
> -variable.  Note that in zsh (unlike other shells) it is still possible
                                ^^^^^^^^^^^^^^^^^^^
> -to create a local variable of the same name as this is considered a
> -different variable (though this variable, too, can be marked readonly).
> -)
> +variable.
> +
> +It is still possible to change other attributes of the variable though,
> +some of which like tt(-U) or tt(-Z) would affect the value. More generally,
> +the readonly attribute should not be relied on as a security mechanism.
> +
> +Note that in zsh (like in pdksh but unlike most other shells) it is
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +still possible to create a local variable of the same name as this is
> +considered a different variable (though this variable, too, can be marked
> +readonly). Special variables that have been made readonly retain their value
> +and readonly attribute when made local.)
[...]

Actually also yash, we might as well remove that "(unlike other
shells)". Note that pdksh has quite a few derivatives (mksh,
posh, openbsd sh (I've not checked there)).

Those shells don't make a distinction for special variables.

Actually I'm not convinced it's useful to treat them
specially in that regard. For instance, if you do "readonly
IFS", you can no longer use library of functions that do some
"local IFS=:" If I add "readonly IFS" to .zshenv,
compaudit/compinit report errors and the shell exits with my
current ~/.zshrc.

See also related POSIX discussion at
http://austingroupbugs.net/view.php?id=767
http://austingroupbugs.net/view.php?id=767#c3735

-- 
Stephane

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Oliver Kiddle]

On 7 Oct, Stephane Chazelas wrote:
> +#define PM_NAMEDDIR     (1<<31) /* has a corresponding nameddirtab entry    */

On a 32-bit system, this now results in integer overflow compiler
warnings. flags in struct hashnode is declared as being of type int.

unsigned int might have been a better choice for a variable holding bit
flags in the first place but struct hashnode is widely used so changing
the type probably is not a matter of just changing the one line. With
that in mind, should we change it straight to uint64_t and open up space
for expansion?

We don't have any existing uses of stdint.h types like uint64_t. I
wouldn't blink at using them in other codebases but do we need to add
any autoconf magic to preserve wide portability?

Oliver

Re: [PATCH] [long] typeset doesn't report tied parameters (and related issues)

[Peter Stephenson]

On Thu, 2018-10-11 at 11:45 +0200, Oliver Kiddle wrote:
> On 7 Oct, Stephane Chazelas wrote:
> > 
> > +#define PM_NAMEDDIR     (1<<31) /* has a corresponding nameddirtab entry    */
> On a 32-bit system, this now results in integer overflow compiler
> warnings. flags in struct hashnode is declared as being of type int.

Right, missed that, sorry.

We still have scope to use the trick we're already using in a number of
other cases --- overlap parameter-only and function-only flags.
Parameters never have ksh or zsh storage type, and functions are never
special or marked as local.

Bumped the dev version number as this will invalidate wordcode.

pws

diff --git a/Config/version.mk b/Config/version.mk
index 8448544..03f6cbf 100644
--- a/Config/version.mk
+++ b/Config/version.mk
@@ -27,5 +27,5 @@
 # This must also serve as a shell script, so do not add spaces around the
 # `=' signs.
 
-VERSION=5.6.2-dev-0
-VERSION_DATE='September 15, 2018'
+VERSION=5.6.2-dev-1
+VERSION_DATE='October 11, 2018'
diff --git a/Src/zsh.h b/Src/zsh.h
index 8d39a04..8941588 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -1867,7 +1867,7 @@ struct tieddata {
 /* The following are the same since they *
  * both represent -U option to typeset   */
 #define PM_UNIQUE	(1<<13)	/* remove duplicates                        */
-#define PM_UNALIASED	(1<<13)	/* do not expand aliases when autoloading   */
+#define PM_UNALIASED	(1<<13)	/* (function) do not expand aliases when autoloading   */
 
 #define PM_HIDE		(1<<14)	/* Special behaviour hidden by local        */
 #define PM_CUR_FPATH    (1<<14) /* (function): can use $fpath with filename */
@@ -1876,31 +1876,30 @@ struct tieddata {
 #define PM_TIED 	(1<<16)	/* array tied to colon-path or v.v.         */
 #define PM_TAGGED_LOCAL (1<<16) /* (function): non-recursive PM_TAGGED      */
 
-#define PM_KSHSTORED	(1<<17) /* function stored in ksh form              */
-#define PM_ZSHSTORED	(1<<18) /* function stored in zsh form              */
-
 /* Remaining flags do not correspond directly to command line arguments */
-#define PM_DONTIMPORT_SUID (1<<19) /* do not import if running setuid */
-#define PM_LOADDIR      (1<<19) /* (function) filename gives load directory */
-#define PM_SINGLE       (1<<20) /* special can only have a single instance  */
-#define PM_ANONYMOUS    (1<<20) /* (function) anonymous function            */
-#define PM_LOCAL	(1<<21) /* this parameter will be made local        */
-#define PM_SPECIAL	(1<<22) /* special builtin parameter                */
-#define PM_RO_BY_DESIGN (1<<23) /* to distinguish from specials that can be
+#define PM_DONTIMPORT_SUID (1<<17) /* do not import if running setuid */
+#define PM_LOADDIR      (1<<17) /* (function) filename gives load directory */
+#define PM_SINGLE       (1<<18) /* special can only have a single instance  */
+#define PM_ANONYMOUS    (1<<18) /* (function) anonymous function            */
+#define PM_LOCAL	(1<<19) /* this parameter will be made local        */
+#define PM_KSHSTORED	(1<<19) /* (function) stored in ksh form              */
+#define PM_SPECIAL	(1<<20) /* special builtin parameter                */
+#define PM_ZSHSTORED	(1<<20) /* (function) stored in zsh form              */
+#define PM_RO_BY_DESIGN (1<<21) /* to distinguish from specials that can be
 				   made read-only by the user               */
 #define PM_READONLY_SPECIAL (PM_SPECIAL|PM_READONLY|PM_RO_BY_DESIGN)
-#define PM_DONTIMPORT	(1<<24)	/* do not import this variable              */
-#define PM_RESTRICTED	(1<<25) /* cannot be changed in restricted mode     */
-#define PM_UNSET	(1<<26)	/* has null value                           */
-#define PM_REMOVABLE	(1<<27)	/* special can be removed from paramtab     */
-#define PM_AUTOLOAD	(1<<28) /* autoloaded from module                   */
-#define PM_NORESTORE	(1<<29)	/* do not restore value of local special    */
-#define PM_AUTOALL	(1<<29) /* autoload all features in module
+#define PM_DONTIMPORT	(1<<22)	/* do not import this variable              */
+#define PM_RESTRICTED	(1<<23) /* cannot be changed in restricted mode     */
+#define PM_UNSET	(1<<24)	/* has null value                           */
+#define PM_REMOVABLE	(1<<25)	/* special can be removed from paramtab     */
+#define PM_AUTOLOAD	(1<<26) /* autoloaded from module                   */
+#define PM_NORESTORE	(1<<27)	/* do not restore value of local special    */
+#define PM_AUTOALL	(1<<27) /* autoload all features in module
 				 * when loading: valid only if PM_AUTOLOAD
 				 * is also present.
 				 */
-#define PM_HASHELEM     (1<<30) /* is a hash-element */
-#define PM_NAMEDDIR     (1<<31) /* has a corresponding nameddirtab entry    */
+#define PM_HASHELEM     (1<<28) /* is a hash-element */
+#define PM_NAMEDDIR     (1<<29) /* has a corresponding nameddirtab entry    */
 
 /* The option string corresponds to the first of the variables above */
 #define TYPESET_OPTSTR "aiEFALRZlurtxUhHTkz"