treatment of empty strings - why is this not a bug?

treatment of empty strings - why is this not a bug?

[Greg Klanderman]

It's so nice that zsh doesn't do splitting on unquoted parameter
expansions:

[~] greg@lwm| zsh -f
lwm% function count () { echo $# }
lwm% a=( "a b" "c d" )
lwm% echo $#a[@]
2
lwm% count $a[@]
2
lwm% count "$a[@]"
2

however, it does silently drop empty strings:

lwm% a=( "" "" "" )
lwm% echo $#a[@] 
3
lwm% count $a[@]
0
lwm% count "$a[@]"
3

which means you still have to use all those quotes anyway.

I just don't understand why there should be any distinction
between the splitting and dropping of empty strings.

And even using quotes, zsh's treatment of empty strings here
seems very inconsistent:

lwm% x=::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
v= =><=
v= =><=
lwm% x=:a:b: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done 
v= =><=
v= =>a<=
v= =>b<=
v= =><=

thanks,
Greg

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

On Tue, 13 Jan 2009 02:32:34 -0500
Greg Klanderman <gak@klanderman.net> wrote:
> lwm% x=::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
> v= =><=
> v= =><=

Hmm...

% setopt rcexpandparam                                     
% x=::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
v= =><=
v= =><=
v= =><=
v= =><=

What's this got to do with RC_EXPAND_PARAM and why is the behaviour
without that option not a bug?  Er...  I'm very glad you asked.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

On Tue, 13 Jan 2009 19:24:09 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Tue, 13 Jan 2009 02:32:34 -0500
> Greg Klanderman <gak@klanderman.net> wrote:
> > lwm% x=::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
> > v= =><=
> > v= =><=
> 
> Hmm...
> 
> % setopt rcexpandparam                                     
> % x=::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
> v= =><=
> v= =><=
> v= =><=
> v= =><=
> 
> What's this got to do with RC_EXPAND_PARAM and why is the behaviour
> without that option not a bug?  Er...  I'm very glad you asked.
> 
> -- 
> Peter Stephenson <p.w.stephenson@ntlworld.com>
> Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Oh, yes.  I had to look at the code before the answer came to me,
again---I've run up against this one several times before.

It's not a bug because it's documented to do something incredibly
pointless:

  For historical reasons, the usual behaviour that empty array elements
  are retained inside double quotes is disabled for arrays generated
  by splitting; hence the following:

  example(line="one::three"
  print -l "${(s.:.)line}")

  produces two lines of output for tt(one) and tt(three) and elides the
  empty field.  To override this behaviour, supply the "(@)" flag as well,
  i.e.  tt("${(@s.:.)line}").

Not sure why RC_EXPAND_PARAM makes it work more sensibly, however.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: treatment of empty strings - why is this not a bug?

[Greg Klanderman]

>>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:

> It's not a bug because it's documented to do something incredibly
> pointless:

If the reason for this behavior is incredibly pointless, should there
be a shell option to change the default to the more reasonable thing?

>   For historical reasons, the usual behaviour that empty array elements
>   are retained inside double quotes is disabled for arrays generated
>   by splitting; hence the following:

I still do not understand why I get exactly two empty strings no
matter how many colons I have in the input string: I would expect
none based on the description you cited.

phl% x=::::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
v= =><=
v= =><=
phl% x=: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done 
v= =><=
v= =><=

thanks,
Greg

Re: treatment of empty strings - why is this not a bug?

[Greg Klanderman]

>>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:

>   For historical reasons, the usual behaviour that empty array elements
>   are retained inside double quotes is disabled for arrays generated
>   by splitting; hence the following:

Hi Peter, also could you please address my original question: when the
decision that unquoted parameter expansions in zsh should not undergo
field splitting was made, why was the behavior of dropping empty
strings retained?

thanks,
Greg

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

Greg Klanderman wrote:
> I still do not understand why I get exactly two empty strings no
> matter how many colons I have in the input string: I would expect
> none based on the description you cited.
> 
> phl% x=::::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
> v= =><=
> v= =><=
> phl% x=: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done 
> v= =><=
> v= =><=

It's to do with the code that joins parameters with what's next to them,
which is why the RC_EXPAND_PARAM option is different.  Quite why it does
what it does I don't understand either, however; but I don't really
understand this particular piece of traditional behaviour at all.
If I were changing it, rather than try to sanitize it I would simply
remove it.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

Greg Klanderman wrote:
> Hi Peter, also could you please address my original question: when the
> decision that unquoted parameter expansions in zsh should not undergo
> field splitting was made, why was the behavior of dropping empty
> strings retained?

No idea; this is very ancient history right from the shell's origins.
It's useful to be able to do this separately from splitting words, but
that's not an answer.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: treatment of empty strings - why is this not a bug?

[Bart Schaefer]

I was hoping to just let this thread go by, but maybe I'm the only one
still on list who's been around long enough to have an inkling of what
is going on here.

On Jan 13,  2:32am, Greg Klanderman wrote:
}
} I just don't understand why there should be any distinction
} between the splitting and dropping of empty strings.

It all goes back to the semantics of $@.

Here's bash:

$ set -- aa "" bb "" cc
$ echo $#
5
$ for x in $@; do echo $x; done
aa
bb
cc
$

Note that the empty elements of $@ are dropped.  Now, you might argue
that the *reason* they're dropped is because $@ has been joined into
a string and then re-split into words on $IFS, but the end result is
that empty elements disappear unless quoted.

Paul Falstad (original author of zsh) made a conscious decision that
a non-empty string, even one containing some or only characters that
appear in $IFS, was a significant item of data and should remain in
the expansion of $@.  However, he was unwilling to deviate from the
standard semantics of $@ to the point of treating empty strings as
significant.  Too many programs that deal with file names, for example,
would begin spewing errors if they received empty strings in their
argument lists when called as e.g. ls -l $@.

The point is to be minimally surprising to the person who just doesn't
want to think very hard about array and $IFS semantics, not to be
entirely logically consistent to someone who analyzes the behavior in
detail.  If you're enough of a geek to care, you're also enough of a
geek to figure out a workaround.

Everything else follows from there.  $a[@] is supposed to have the
same semantics as $@ for any array a; this must be, so that $argv[@]
is exactly equivalent to $@.

As to why things behave seemingly differently when using (s-:-) or
the like, well, in many cases zsh's parameter expansion was developed
looking only at the end results and not by paying close attention to
which "layer" of the implementation performed which operation.  As new
features were built on top of the implementation, particularly some of
the nested expansion tricks, unexpected oddities arose because that
layering became more important.  Some of those oddities became widely
enough relied on in scripts that we're now essentially stuck with them.

(Zsh is not one of "those" open source projects that is willing to
allow every new release to be "improved" by breaking everything that
went before.  For the most part, what has always worked still works,
and if it doesn't it's usually because it was never intended to work,
not because someone decided that the way it was once meant to work is
now philosophically wrong.)

The more literal explantion for your example has to do with multiple
consective separators being treated as a single word break (as happens
with, for example, multiple consecutive spaces when splitting on $IFS
in the shwordsplit case) vs. multiple separators being treated as
multiple word breaks.  However, I've forgotten whether rcexpandparam
was intentionally or accidentally given that side-effect.

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

On Thu, 15 Jan 2009 20:29:29 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Greg Klanderman wrote:
> > I still do not understand why I get exactly two empty strings no
> > matter how many colons I have in the input string: I would expect
> > none based on the description you cited.
> > 
> > phl% x=::::: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done
> > v= =><=
> > v= =><=
> > phl% x=: && for v in "${(s-:-)x}" ; do echo "v= =>$v<=" ; done 
> > v= =><=
> > v= =><=
> 
> It's to do with the code that joins parameters with what's next to them,
> which is why the RC_EXPAND_PARAM option is different.

Actually, I think I can do better than that.  It's because quoted NULLs
never disappear---the only things that disappear are the empty fields
resulting from the splitting.  You can therefore think of the key
expression in the foregoing as

  ""${(s-:-)}""

(two double-quoted NULLs bookending an expression split into an
array---this isn't how it works internally, but the effect is very similar.)
Now you can see the distinction from having no quotes in the expression
at all.  So in the case where RC_EXPAND_PARAM is unset what remains are the
NULLs at each end; with it set, those get combined with all the fields in
the middle.  It does have its own twisted logic.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: treatment of empty strings - why is this not a bug?

[Greg Klanderman]

>>>>> Bart Schaefer <schaefer@brasslantern.com> writes:

> I was hoping to just let this thread go by, but maybe I'm the only one
> still on list who's been around long enough to have an inkling of what
> is going on here.

Hi Bart, thank you for responding to this question.  I hope you and/or
Peter will entertain one more round of this thread and answer my final
two questions, then I'll happily drop it if you wish.

> Paul Falstad (original author of zsh) made a conscious decision that
> a non-empty string, even one containing some or only characters that
> appear in $IFS, was a significant item of data and should remain in
> the expansion of $@.  However, he was unwilling to deviate from the
> standard semantics of $@ to the point of treating empty strings as
> significant.

I almost think it's worse this way - you read the docs and think "Oh
great, I don't have to worry about putting all those crazy double
quotes around everything anymore to preserve my array elements",
however, you in fact do still need all those double quotes.

> Too many programs that deal with file names, for example,
> would begin spewing errors if they received empty strings in their
> argument lists when called as e.g. ls -l $@.

I don't understand how empty strings get into arrays unless one
explicitly puts them there.. and if one explicitly puts them there,
then presumably they don't want them randomly dropped.

The best reason I can come up with for this semantics is having code
like this continue to work:

    foo_arg=""
    if [[ ... ]] ; then
      foo_arg="-bar"
    fi
    foo $foo_arg

However, now that our shell has arrays, that should really be written
as:

    foo_arg=()
    if [[ ... ]] ; then
      foo_arg=("-bar")
    fi
    foo $foo_arg

and so I see no remaining need for dropping empty strings.

> The point is to be minimally surprising to the person who just doesn't
> want to think very hard about array and $IFS semantics, not to be
> entirely logically consistent to someone who analyzes the behavior in
> detail.  If you're enough of a geek to care, you're also enough of a
> geek to figure out a workaround.

Sure, I can keep using double quotes, but I don't want to given how
close zsh is to DTRT and not needing them.

You and Peter seem to be in charge here - do you both still agree that
dropping empty strings is a desirable default?

Would you guys be OK with adding an option to inhibit this?

thank you,
Greg

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

Greg Klanderman wrote:
> You and Peter seem to be in charge here - do you both still agree that
> dropping empty strings is a desirable default?
> 
> Would you guys be OK with adding an option to inhibit this?

I think that's more likely to be a source of confusion rather than a
help.  If you know about the problem you can already get round it.  If
you don't the option isn't going to help.  The option's yet another
headache for debugging.

Further, I don't think the option would be useful without syntax to
restore the current behaviour for each variable (the opposite of
double-quoting), since as Bart pointed out that's quite widely used.
That adds yet another of layer of complexity and source of bugs.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: treatment of empty strings - why is this not a bug?

[Greg Klanderman]

>>>>> Peter Stephenson <pws@csr.com> writes:

> I think that's more likely to be a source of confusion rather than a
> help.  If you know about the problem you can already get round it.  If
> you don't the option isn't going to help.  The option's yet another
> headache for debugging.

If the NO_SH_WORD_SPLIT default is only to be the source of subtle
bugs and not actually useful, then it should be removed.

> Further, I don't think the option would be useful without syntax to
> restore the current behaviour for each variable (the opposite of
> double-quoting), since as Bart pointed out that's quite widely used.
> That adds yet another of layer of complexity and source of bugs.

I cannot believe it's widely used, except in legacy scripts that
predate array parameters.  Why would you put an empty string into a
variable unless you wanted it there?  I don't see any reason you'd
ever want the current behavior in a new script, and any existing
script should just emulate to the broken behavior.

greg

Re: treatment of empty strings - why is this not a bug?

[Richard Hartmann]

On Fri, Jan 16, 2009 at 20:40, Greg Klanderman <gak@klanderman.net> wrote:

> I cannot believe it's widely used, except in legacy scripts that
> predate array parameters.  Why would you put an empty string into a
> variable unless you wanted it there?  I don't see any reason you'd
> ever want the current behavior in a new script, and any existing
> script should just emulate to the broken behavior.

Existing scripts tend to run in odd places, often without anyone
around who can still maintain them. Such a change could have very
bad effects, possibly without anyone noticing (in time).

That is a real problem and happens far too often, no need to make
it worse.


Richard

Re: treatment of empty strings - why is this not a bug?

[Bart Schaefer]

On Jan 16, 12:35pm, Greg Klanderman wrote:
}
} I don't understand how empty strings get into arrays unless one
} explicitly puts them there..

Consider an array populated by using a glob pattern, which is then
subjected to replacement using $arr:s/pat// or $arr:h or $arr:t.
You don't know at time of execution what will be matched in by the
glob, nor necessarily what will be left behind after substitution.
If the substitution results in an emtpy string, how often do you
want that to remain in the argument list of whatever action you next
apply to the contents of the array?

If what you're attempting to do is transform one set of file names
into another, which is the most common operation at the shell command
line if not in scripts, I'd argue that generally you do not.  On the
other hand, if the file name contains spaces, you don't want it to be
split.

Re: treatment of empty strings - why is this not a bug?

[Bart Schaefer]

On Jan 16,  2:40pm, Greg Klanderman wrote:
}
} If the NO_SH_WORD_SPLIT default is only to be the source of subtle
} bugs and not actually useful, then it should be removed.

I'd suggest that if the shell has been behaving this way for more than
15 years now, and a majority of its users consider the behavior to be
useful, that perhaps asserting that it is "only" a source of subtle
bugs is a tad presumptive.

} I don't see any reason you'd
} ever want the current behavior in a new script, and any existing
} script should just emulate to the broken behavior.

That's exactly backwards from the philosophy that zsh development has
always followed.

Re: treatment of empty strings - why is this not a bug?

[Greg Klanderman]

>>>>> Bart Schaefer <schaefer@brasslantern.com> writes:

> Consider an array populated by using a glob pattern, which is then
> subjected to replacement using $arr:s/pat// or $arr:h or $arr:t.
> You don't know at time of execution what will be matched in by the
> glob, nor necessarily what will be left behind after substitution.
> If the substitution results in an emtpy string, how often do you
> want that to remain in the argument list of whatever action you next
> apply to the contents of the array?

I do not agree with this reasoning, but I don't think it's worth
continuing to argue about it.

thank you,
Greg

Re: treatment of empty strings - why is this not a bug?

[Peter Stephenson]

On Sat, 17 Jan 2009 00:31:05 -0500
Greg Klanderman <gak@klanderman.net> wrote:
> >>>>> Bart Schaefer <schaefer@brasslantern.com> writes:
> 
> > Consider an array populated by using a glob pattern, which is then
> > subjected to replacement using $arr:s/pat// or $arr:h or $arr:t.
> > You don't know at time of execution what will be matched in by the
> > glob, nor necessarily what will be left behind after substitution.
> > If the substitution results in an emtpy string, how often do you
> > want that to remain in the argument list of whatever action you next
> > apply to the contents of the array?
> 
> I do not agree with this reasoning, but I don't think it's worth
> continuing to argue about it.

It's not just reasoning, it's actually used all over the place for
eliminating unwanted matches.

It's probably worth getting one think crystal clear: there is absolutely
no likelihood of changing basic syntactic defaults, which is a recipe
for disaster.  So you're right, there's not a lot point arguing.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

PATCH: make PROMPT_SP end-of-line marker configurable

[Greg Klanderman]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 176 bytes --]


Hi,

This adds a parameter, EOLMARK, which allows configuring the
end-of-line marker used by the PROMPT_SP option.

I also cleaned up the doc for PS1 slightly.

thanks,
Greg


[-- Attachment #2: zsh-EOLMARK.patch --]
[-- Type: text/plain, Size: 3955 bytes --]

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.221
diff -u -r1.221 utils.c
--- Src/utils.c	6 Apr 2009 09:06:36 -0000	1.221
+++ Src/utils.c	17 May 2009 04:31:42 -0000
@@ -1216,12 +1216,16 @@
 	 * Unfortunately it interacts badly with ZLE displaying message
 	 * when ^D has been pressed. So just disable PROMPT_SP logic in
 	 * this case */
+	char *eolmark = getsparam("EOLMARK");
 	char *str;
-	int percents = opts[PROMPTPERCENT];
+	int percents = opts[PROMPTPERCENT], w = 0;
+	if (!eolmark || !*eolmark)
+	    eolmark = "%B%S%#%s%b";
 	opts[PROMPTPERCENT] = 1;
-	str = promptexpand("%B%S%#%s%b", 0, NULL, NULL, NULL);
+	str = promptexpand(eolmark, 1, NULL, NULL, NULL);
+	countprompt(str, &w, 0, -1);
 	opts[PROMPTPERCENT] = percents;
-	fprintf(shout, "%s%*s\r", str, (int)columns - 1 - !hasxn, "");
+	fprintf(shout, "%s%*s\r", str, (int)columns - w - !hasxn, "");
 	free(str);
     }
 
Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.81
diff -u -r1.81 options.yo
--- Doc/Zsh/options.yo	6 Apr 2009 09:06:36 -0000	1.81
+++ Doc/Zsh/options.yo	17 May 2009 04:31:42 -0000
@@ -1307,16 +1307,18 @@
 item(tt(PROMPT_SP) <D>)(
 Attempt to preserve a partial line (i.e. a line that did not end with a
 newline) that would otherwise be covered up by the command prompt due to
-the PROMPT_CR option.  This works by outputting some cursor-control
+the tt(PROMPT_CR) option.  This works by outputting some cursor-control
 characters, including a series of spaces, that should make the terminal
 wrap to the next line when a partial line is present (note that this is
 only successful if your terminal has automatic margins, which is typical).
 
-When a partial line is preserved, you will see an inverse+bold character at
-the end of the partial line:  a "%" for a normal user or a "#" for root.
+When a partial line is preserved, by default you will see an inverse+bold
+character at the end of the partial line:  a "%" for a normal user or
+a "#" for root.  If set, the shell parameter tt(EOLMARK) can be used to
+customize how the end of partial lines are shown.
 
-NOTE: if the PROMPT_CR option is not set, enabling this option will have no
-effect.  This option is on by default.
+NOTE: if the tt(PROMPT_CR) option is not set, enabling this option will
+have no effect.  This option is on by default.
 )
 pindex(PROMPT_PERCENT)
 pindex(NO_PROMPT_PERCENT)
Index: Doc/Zsh/params.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/params.yo,v
retrieving revision 1.54
diff -u -r1.54 params.yo
--- Doc/Zsh/params.yo	24 Apr 2009 09:00:37 -0000	1.54
+++ Doc/Zsh/params.yo	17 May 2009 04:31:42 -0000
@@ -823,6 +823,14 @@
 arithmetic expansion before being interpreted as a pathname.  Note that
 tt(ENV) is em(not) used unless zsh is emulating bf(sh) or bf(ksh).
 )
+vindex(EOLMARK)
+item(tt(EOLMARK))(
+When the tt(PROMPT_CR) and tt(PROMPT_SP) options are set, the tt(EOLMARK)
+parameter can be used to customize how the end of partial lines are shown.
+This parameter undergoes prompt expansion, with the tt(PROMPT_PERCENT)
+option set.  If not set or empty, the default behavior is equivalent to
+the value `tt(%B%S%#%s%b)'.
+)
 vindex(FCEDIT)
 item(tt(FCEDIT))(
 The default editor for the tt(fc) builtin.  If tt(FCEDIT) is not set,
@@ -1046,10 +1054,10 @@
 vindex(PS1)
 item(tt(PS1) <S>)(
 The primary prompt string, printed before a command is read.
-the default is `tt(%m%# )'.  It undergoes a special form of expansion
+It undergoes a special form of expansion
 before being displayed; see
 ifzman(EXPANSION OF PROMPT SEQUENCES in zmanref(zshmisc))\
-ifnzman(noderef(Prompt Expansion)).
+ifnzman(noderef(Prompt Expansion)).  The default is `tt(%m%# )'.
 )
 vindex(PS2)
 item(tt(PS2) <S>)(

Re: PATCH: make PROMPT_SP end-of-line marker configurable

[Peter Stephenson]

On Sun, 17 May 2009 00:55:35 -0400
Greg Klanderman <gak@klanderman.net> wrote:
> This adds a parameter, EOLMARK, which allows configuring the
> end-of-line marker used by the PROMPT_SP option.

Thanks.  I think it might be better if it was called PROMPT_EOL_MARK.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: make PROMPT_SP end-of-line marker configurable

[Greg Klanderman]

>>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:

> Thanks.  I think it might be better if it was called PROMPT_EOL_MARK.

OK, do you want me to resubmit with that change?

greg

Re: PATCH: make PROMPT_SP end-of-line marker configurable

[Peter Stephenson]

Greg Klanderman wrote:
> 
> >>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:
> 
> > Thanks.  I think it might be better if it was called PROMPT_EOL_MARK.
> 
> OK, do you want me to resubmit with that change?

No, I've submitted it with the change.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: make PROMPT_SP end-of-line marker configurable

[Greg Klanderman]

Looks good, thank you Peter!

>>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:

> Greg Klanderman wrote:
>> 
>> >>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:
>> 
>> > Thanks.  I think it might be better if it was called PROMPT_EOL_MARK.
>> 
>> OK, do you want me to resubmit with that change?

> No, I've submitted it with the change.

> -- 
> Peter Stephenson <p.w.stephenson@ntlworld.com>
> Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: make PROMPT_SP end-of-line marker configurable

[Greg Klanderman]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

>>>>> Peter Stephenson <p.w.stephenson@ntlworld.com> writes:

> No, I've submitted it with the change.

Hi Peter, please apply this fix on top of what you committed
yesterday; my change had a slight bug..

thanks,
greg


[-- Attachment #2: further fix --]
[-- Type: text/plain, Size: 613 bytes --]

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.222
diff -u -r1.222 utils.c
--- Src/utils.c	17 May 2009 18:23:10 -0000	1.222
+++ Src/utils.c	18 May 2009 18:22:16 -0000
@@ -1225,7 +1225,10 @@
 	str = promptexpand(eolmark, 1, NULL, NULL, NULL);
 	countprompt(str, &w, 0, -1);
 	opts[PROMPTPERCENT] = percents;
-	fprintf(shout, "%s%*s\r", str, (int)columns - w - !hasxn, "");
+	zputs(str, shout);
+	for (w = (int)columns - w - !hasxn; w > 0; w--)
+	    putc(' ', shout);
+	putc('\r', shout);
 	free(str);
     }
 

bug in ztrftime(): '%e' and '%f' specifiers swapped

[Greg Klanderman]

Hi Peter,

I just tracked down a bug in 'cvs' completion, in particular the logic
in _cvs_modified_entries, and found the problem is due to your changes
in Src/utils.c, revision 1.210:

| revision 1.210
| date: 2009/03/02 10:12:17;  author: pws;  state: Exp;  lines: +56 -15
| 26614 + 26615: history -t <fmt> plus ztrftime "-" format modifier

You have inadvertently swapped the meanings of the '%e' and '%f' format
specifiers.  This patch should do the trick:

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.225
diff -u -r1.225 utils.c
--- Src/utils.c	5 Jun 2009 11:15:53 -0000	1.225
+++ Src/utils.c	26 Jun 2009 20:18:41 -0000
@@ -2489,10 +2489,10 @@
 		    *buf++ = '0' + tm->tm_mday / 10;
 		*buf++ = '0' + tm->tm_mday % 10;
 		break;
-	    case 'e':
+	    case 'f':
 		strip = 1;
 		/* FALLTHROUGH */
-	    case 'f':
+	    case 'e':
 		if (tm->tm_mday > 9)
 		    *buf++ = '0' + tm->tm_mday / 10;
 		else if (!strip)

However, I find this equivalent patch to be more readable:

Index: Src/utils.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/utils.c,v
retrieving revision 1.225
diff -u -r1.225 utils.c
--- Src/utils.c	5 Jun 2009 11:15:53 -0000	1.225
+++ Src/utils.c	26 Jun 2009 20:32:52 -0000
@@ -2490,9 +2490,8 @@
 		*buf++ = '0' + tm->tm_mday % 10;
 		break;
 	    case 'e':
-		strip = 1;
-		/* FALLTHROUGH */
 	    case 'f':
+		strip = strip || (fmt[-1] == 'f');
 		if (tm->tm_mday > 9)
 		    *buf++ = '0' + tm->tm_mday / 10;
 		else if (!strip)
@@ -2500,10 +2499,9 @@
 		*buf++ = '0' + tm->tm_mday % 10;
 		break;
 	    case 'K':
-		strip = 1;
-		/* FALLTHROUGH */
 	    case 'H':
 	    case 'k':
+		strip = strip || (fmt[-1] == 'K');
 		if (tm->tm_hour > 9)
 		    *buf++ = '0' + tm->tm_hour / 10;
 		else if (!strip) {
@@ -2515,9 +2513,8 @@
 		*buf++ = '0' + tm->tm_hour % 10;
 		break;
 	    case 'L':
-		strip = 1;
-		/* FALLTHROUGH */
 	    case 'l':
+		strip = strip || (fmt[-1] == 'L');
 		hr12 = tm->tm_hour % 12;
 		if (hr12 == 0)
 		    hr12 = 12;

Note: I didn't test either of those as I will not be able to build
until I update my autoconf.

Btw, are the zsh extension format specifiers documented?

thanks,
greg

Re: bug in ztrftime(): '%e' and '%f' specifiers swapped

[Peter Stephenson]

Greg Klanderman wrote:
> You have inadvertently swapped the meanings of the '%e' and '%f' format
> specifiers.

Thanks for tracking this down.

> However, I find this equivalent patch to be more readable:

Interesting how subjective this is, I find it significantly less so
(though they're certainly equivalent so this is all rather minor and if
you hadn't sent two patches I'd never even have thought about it):

- there is no immediate visual cue that the formats do different things
- the test for the format letter is repeated without it being obvious it
  is the same test
- it's made even less obvious because the second test has to advance
  back down the format string since the key letter isn't saved in a
  variable [we do this sort of thing all the time elsewhere, though]
- the fact that 'f' sets the "strip" feature unconditionally is obscured
- I'm not that keen on "||" and "&&" in assignments anyway [again, the
  shell is full of stuff I'm not that keen on]

so I've committed the first one.

> Btw, are the zsh extension format specifiers documented?

See the description of the %D format in the description of prompt
escapes, which is where these first occurred; feel free to add some more
cross references, I see there isn't one in the zsh/datetime description,
for example.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: bug in ztrftime(): '%e' and '%f' specifiers swapped

[Greg Klanderman]

> Interesting how subjective this is, I find it significantly less so

Yep!  Not to try to sway you, but I'll give you my reasons just for
the fun of it..

I guess the biggest is that I really don't like cases that do
something then fall through, it's just very hard to reason about, and
usually makes further modifications difficult and error prone.

Also, this style very rarely is able to extend beyond two cases - case
in point when you added the 'H' format you had to violate your first
three points below.

> - there is no immediate visual cue that the formats do different things

I would always assume if there are multiple cases sharing the same
logic that the logic itself will differentiate between them, not that
they all function identically.

> - the test for the format letter is repeated without it being obvious it
>   is the same test
> - it's made even less obvious because the second test has to advance
>   back down the format string since the key letter isn't saved in a
>   variable [we do this sort of thing all the time elsewhere, though]

I certainly would have saved the switch'd on character in a variable
if I were writing this, and probably if I'd had a working build system
I might have done more significant cleanup.  On the other hand, when
submitting patches to a project like this I usually go for the minimal
change because otherwise it's a larger burden on whoever reviews and
commits the change.  Seeing that pattern (fmt[-1] == 'x') already in
use in that function I figured it was considered not so bad.

> - the fact that 'f' sets the "strip" feature unconditionally is obscured

strip = strip || (fmtchar == 'f') is pretty clear to me :-)

> - I'm not that keen on "||" and "&&" in assignments anyway [again, the
>   shell is full of stuff I'm not that keen on]

> so I've committed the first one.

thank you!

greg

have '&' automatically disown?

[Greg Klanderman]

Just wondering here.. I almost never use job control, and so almost
always use '&!' for background commands.  I was thinking it would
be nice to not have to type all those '!'s.. but I don't see a setting
to have '&' alone default to the disown behavior.  Is there some way
to do that which I've missed?

thanks,
Greg

Re: have '&' automatically disown?

[Peter Stephenson]

Greg Klanderman wrote:
> Just wondering here.. I almost never use job control, and so almost
> always use '&!' for background commands.  I was thinking it would
> be nice to not have to type all those '!'s.. but I don't see a setting
> to have '&' alone default to the disown behavior.  Is there some way
> to do that which I've missed?

Well, if you really almost never use job control you can "unsetopt
monitor" until the occasion when you actually need it... but note that
that will unhide any job you didn't actually disown, so swapping between
the two modes isn't particularly helpful.  In other words, turning off
the option doesn't actually stop the shell recording jobs, only
reporting them.  I've a vague feeling this is supposed to be a feature.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: have '&' automatically disown?

[Peter Stephenson]

Peter Stephenson wrote:
> Well, if you really almost never use job control you can "unsetopt
> monitor" until the occasion when you actually need it... but note that
> that will unhide any job you didn't actually disown, so swapping between
> the two modes isn't particularly helpful.

I've reread this and I garbled it to the point of unintelligibility.
Sorry.

When you "unsetopt monitor" jobs are not reported.

When you "setopt monitor" again *then* jobs you didn't explicitly disown
are reported.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: have '&' automatically disown?

[Greg Klanderman]

Thanks for your response and clarification Peter,

>>>>> On February 5, 2010 Peter Stephenson <pws@csr.com> wrote:

> When you "setopt monitor" again *then* jobs you didn't explicitly disown
> are reported.

Really?  That's not what I see, patchlevel 1.4847.

It is the case that while monitor is off, 'jobs' does still list the
jobs, however 'fg' reports "fg: no job control in this shell.".  I
turned monitor back on and 'fg'd a job, but now neither C-z nor C-c
will interrupt that job:

[~] greg@lwm| zsh -f
lwm% setopt nomonitor nohup nocheckjobs
lwm% 
lwm% sleep 2000 &
lwm% sleep 3000 &
lwm% jobs
[1]  - running    sleep 2000
[2]  + running    sleep 3000
lwm% fg %1
fg: no job control in this shell.
lwm% setopt monitor
lwm% jobs
[1]  - running    sleep 2000
[2]  + running    sleep 3000
lwm% fg %1
[1]  - running    sleep 2000
^C^C^C^Z^Z^Z^Z


Greg

Re: have '&' automatically disown?

[Peter Stephenson]

On Fri, 05 Feb 2010 22:26:44 -0500
Greg Klanderman <gak@klanderman.net> wrote:
> Thanks for your response and clarification Peter,
> 
> >>>>> On February 5, 2010 Peter Stephenson <pws@csr.com> wrote:
> 
> > When you "setopt monitor" again *then* jobs you didn't explicitly disown
> > are reported.
> 
> Really?  That's not what I see, patchlevel 1.4847.

I was probably too brief: when I said "reported", I meant "reported
interactively when their state changes".  I was assuming you were trying
to ignore jobs to the extent that if you didn't look at them again, you
wouldn't notice them, which is what turning off "monitor" should do for
you, unless I've missed something.  If you start going looking for them
you'll still find them.  I don't know if that's good enough for your
needs or not.
 
> It is the case that while monitor is off, 'jobs' does still list the
> jobs, however 'fg' reports "fg: no job control in this shell.".  I
> turned monitor back on and 'fg'd a job, but now neither C-z nor C-c
> will interrupt that job:

I can well believe all of this, but it's not clear to me why its
relevant.  If it means you still need job control to work continuously,
then certainly turning off "monitor" isn't an option.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: have '&' automatically disown?

[Greg Klanderman]

>>>>> On February 7, 2010 Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:

> If you start going looking for them you'll still find them.

Certainly not the end of the world...

> I don't know if that's good enough for your needs or not.

nohup + nocheckjobs does most of what I care about I guess.
I just noticed the later before starting this thread.

> I can well believe all of this, but it's not clear to me why its
> relevant.

Because if I turn monitor back on I can completely wedge my shell.

> If it means you still need job control to work continuously, then
> certainly turning off "monitor" isn't an option.

I don't actually want to use job control on the jobs started while
monitor was off, but since they are still there, chances are I will at
some point shoot myself in the foot.

Would it make sense to just auto-disown when monitor is off?

thanks,
Greg

Re: have '&' automatically disown?

[Peter Stephenson]

Greg Klanderman wrote:
> > I can well believe all of this, but it's not clear to me why its
> > relevant.
> 
> Because if I turn monitor back on I can completely wedge my shell.

Hmm, yes, I can see that makes it relevant.  MONITOR wasn't really
designed to be dynamic, it just kind of evolved in a complex way like
everything else.

> Would it make sense to just auto-disown when monitor is off?

That *does* make a lot of sense, actually: if job control's turned off,
there's no point in half-pretending it isn't.  Disowning existing
background jobs at that point at least makes more sense than having them
come back in a zombified fashion.  I'll think about it tomorrow.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: have '&' automatically disown?

[Mikael Magnusson]

On 7 February 2010 22:06, Greg Klanderman <gak@klanderman.net> wrote:
>>>>>> On February 7, 2010 Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
>
>> If you start going looking for them you'll still find them.
>
> Certainly not the end of the world...
>
>> I don't know if that's good enough for your needs or not.
>
> nohup + nocheckjobs does most of what I care about I guess.
> I just noticed the later before starting this thread.
>
>> I can well believe all of this, but it's not clear to me why its
>> relevant.
>
> Because if I turn monitor back on I can completely wedge my shell.
>
>> If it means you still need job control to work continuously, then
>> certainly turning off "monitor" isn't an option.
>
> I don't actually want to use job control on the jobs started while
> monitor was off, but since they are still there, chances are I will at
> some point shoot myself in the foot.
>
> Would it make sense to just auto-disown when monitor is off?

This will probably get really annoying quickly...

_self_insert() { [[ ! -o monitor && $KEYS = '&' ]] &&
RBUFFER=!$RBUFFER; zle .$WIDGET; ((CURSOR++)) }

-- 
Mikael Magnusson

Re: have '&' automatically disown?

[Bart Schaefer]

On Feb 7,  9:34pm, Peter Stephenson wrote:
} Subject: Re: have '&' automatically disown?
}
} Greg Klanderman wrote:
} > Would it make sense to just auto-disown when monitor is off?
} 
} That *does* make a lot of sense, actually: if job control's turned off,
} there's no point in half-pretending it isn't.  Disowning existing
} background jobs at that point at least makes more sense than having them
} come back in a zombified fashion.  I'll think about it tomorrow.

Disowning a job makes it immune to HUP signals, etc.  That's not what
nomonitor is for -- turning off monitor is supposed to make zsh act
like the old Bourne shell, where jobs still "belonged" to the shell
and would be HUP'd if the terminal connection dropped, but the shell
had no way move them between background and foreground.

The reason you seem to hang the shell when foregrounding a job that
was started with monitor turned off, is because jobs started without
monitoring must be immune to keyboard signals other than hangup.  The
shell is not really hung, but once the process has been forked the
shell has no way to tell it to change its SIGINT/SIGTSTP behavior to
go back to being interruptible/stoppable.

Re: have '&' automatically disown?

[Greg Klanderman]

>>>>> On February 7, 2010 Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Feb 7,  9:34pm, Peter Stephenson wrote:
> } Subject: Re: have '&' automatically disown?
> }
> } Greg Klanderman wrote:
> } > Would it make sense to just auto-disown when monitor is off?
> } 
> } That *does* make a lot of sense, actually: if job control's turned off,
> } there's no point in half-pretending it isn't.  Disowning existing
> } background jobs at that point at least makes more sense than having them
> } come back in a zombified fashion.  I'll think about it tomorrow.

> Disowning a job makes it immune to HUP signals, etc.  That's not what
> nomonitor is for -- turning off monitor is supposed to make zsh act
> like the old Bourne shell, where jobs still "belonged" to the shell
> and would be HUP'd if the terminal connection dropped, but the shell
> had no way move them between background and foreground.

[sorry for neglecting to reply for so long]

Would it make sense to auto-disown if both nomonitor and nohup are set?

Greg

Re: have '&' automatically disown?

[Bart Schaefer]

On Sep 2, 10:57am, Greg Klanderman wrote:
}
} Would it make sense to auto-disown if both nomonitor and nohup are set?

After thinking about this for a while, the answer I get is "almost."

The remaining difference between "disown" and "unsetopt hup monitor" is
whether the job appears in the output of "jobs" (and, by extension, in
the $jobstates et al. variables).  Well, OK, strictly speaking there is
also the difference of what ancestor reaps the child when it finally
does exit, but that's almost invisible to an interactive shell user.

Nevertheless, one might still wish to have access to the job's state
and PID even if the only way to control it is with "kill".

I'm concluding that the only reasonable way to auto-disown, other than
the precmd hook solutions offered on zsh-users, is with a new option.
There are too many side-effects of disowning to add that semantics to
any existing set of options.

Re: have '&' automatically disown?

[Greg Klanderman]

Thanks for thinking some more about this Bart.

Greg