[9fans] mishandling empty lists - let's fix it

[9fans] mishandling empty lists - let's fix it

[Sam Watkins]

hi,

I wanted to have a whinge about one fault I find in unix: commands such as cat,
grep etc. do not handle an empty argument list correctly.  For example,

  cat

should output nothing and exit - concatenating 0 files.  Instead it copies
stdin to stdout, which is inconsistent.  This problem still exists in plan 9.

Copying stdin should be coded as:

  cat -

Such syntax would provide useful extra capability:

  find -type d | cat lib/emptydirs - | mail 9fans@9fans.net

Although this bug is not annoying for command-line work, consider a script that
processes all .c files to be found using cat or grep:

  find -name '*.c' | xargs cat | cc -     # this clever cc can handle it :)

This program works fine until there are no .c files to be found, in that case
it hangs, waiting for one on stdin!  This is a hazard to shell scripters, and a
potential source of security holes.  It's worse than cat -v.

Another example:

  8c && echo 1

should compile 0 files to give an empty object file, and echo 1.

Is there any chance we could fix these things in plan 9/inferno?
I feel that freedom from past mistakes is a key principle of plan 9.

If people support this change, I would be keen to work on it.

thanks!

Sam


(I noticed the "pike" language has a similar bug, `+(1,2,3) -> 6
but `+() -> error!  `+() should be 0 and `*() should be 1, of course.)
I don't think such bugs exist in Lisp :)

Re: [9fans] mishandling empty lists - let's fix it

[Rob Pike]

cat * /dev/null

is the recommended solution.

-rob

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

On Sat, Oct 03, 2009 at 10:01:09AM -0700, Rob Pike wrote:
> cat * /dev/null
>
> is the recommended solution.

Thanks Rob,

That works with cat, but it won't work with chmod, grep -L, ls, find, file
and many others.  I think all of the unix and plan 9 utilities that deal
with a variable number of files have this problem.
Does anyone agree with me that it needs fixing?

Sam

Re: [9fans] mishandling empty lists - let's fix it

[Rob Pike]

> Does anyone agree with me that it needs fixing?

I don't. I also think that even if it were a problem the usage is far
too ingrained to be fixable.

-rob

Re: [9fans] mishandling empty lists - let's fix it

[blstuart]

>> Does anyone agree with me that it needs fixing?
>
> I don't. I also think that even if it were a problem the usage is far
> too ingrained to be fixable.

Nor do I.  Having the no-argument case be filter behavior
(stdin/stdout) is the most elegant, consistent, and predictable
of the options I've seen.

BLS

Re: [9fans] mishandling empty lists - let's fix it

[hiro]

> Nor do I.  Having the no-argument case be filter behavior
> (stdin/stdout) is the most elegant, consistent, and predictable
> of the options I've seen.

Yeah, I always found the lone - very ugly!

If you do a fix you could also add --outputfile in order to improve
the readability :D

Re: [9fans] mishandling empty lists - let's fix it

[Steve Simon]

> Does anyone agree with me that it needs fixing?

sorry, I don't agree that it is broken so I don't thing
it need fixing.

It does occasionally annoy me that tr(1) will not take a file
as an argument but again, changing that would have implications
too wide to make it worthwhile; I try to think of it as OS patina.

-Steve

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> Does anyone agree with me that it needs fixing?

For what it's worth, I agree that it's a problem.  Maybe it's
just the fact that I'm a mathematician, but I've always felt
that any command which accepts a list of files should behave
the way I would expect when presented with an empty list.
Cat, rm, chmod, etc. do not.  At a minimum I think we should
try to avoid creating further exceptions.

Does it need fixing?  Implementing it properly would break most
shell scripts I've written.  If there were renamed versions of
cat, rm, chmod, etc. which treated empty lists in the expected
way then I would probably use them in situations were portability
is not an issue.  It would remove a frequent source of bugs and
leave me with one less thing to worry about.

Then there is the shell.  If you only use commands which behave
sanely for empty lists then the current globbing rules are a
pain.  Changing them to, for example, expand * to `' in an
empty directory breaks too many things.  I've learned to avoid
globbing as much as possible.  In any shell with command
substitution globbing is unnecessary, and is often dangerous.
I would happily use a shell with NO globbing, just to avoid
another source of worries.

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> I've learned to avoid
> globbing as much as possible.  In any shell with command
> substitution globbing is unnecessary, and is often dangerous.
> I would happily use a shell with NO globbing, just to avoid
> another source of worries.

i think this is all a bit dramatic.

there are very simple idioms one can
use in the few cases globbing may be
a problem.  others have suggested ./<pattern>
and i'd add that one can also use
	if(! ~ <pattern> '<pattern>')
		something;
but this is seldom necessary.  one
might consider having the file server
ban globbing characters in file names.

if i could roll back time, i would push
for regular expressions instead of the
goofy shell matching.  then knuth could
have to say that unix is 29 definitions of
regular expressions living under one roof.
:-)

- erik

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> i think this is all a bit dramatic.
...
> if i could roll back time, i would push
> for regular expressions instead of the
> goofy shell matching.  then knuth could
> have to say that unix is 29 definitions of
> regular expressions living under one roof.
> :-)

But you can use regular expressions instead of goofy shell
matching.  Find and ls give you lists of file names and grep
prunes them for you.  Shell command substitution then puts
them in the right place in the command, or use xargs when
appropriate.  I don't claim that globbing is a horrible evil,
merely that it is an unnecessary evil.

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[roger peppe]

2009/10/5 John Stalker <stalker@maths.tcd.ie>:
> But you can use regular expressions instead of goofy shell
> matching.  Find and ls give you lists of file names and grep
> prunes them for you.  Shell command substitution then puts
> them in the right place in the command, or use xargs when
> appropriate.  I don't claim that globbing is a horrible evil,
> merely that it is an unnecessary evil.

dodgy in unix where filenames can contain \n.
under plan 9 you have to remember to set ifs to \n only,
which is awkward and error prone.

this stuff is better designed for interactive use than scripting,
but it still does an adequate job provided you remember the pitfalls.
rc is loads better like that than shells with trad semantics.

i like the way i can parse quoted names in the inferno shell.
various p9 utils have been outfitted to print quoted names
but they're only useful interactively without support from rc.

Re: [9fans] mishandling empty lists - let's fix it

[Bakul Shah]

On Sun, 04 Oct 2009 03:03:27 +1100 Sam Watkins <sam@nipl.net>  wrote:
>
>   find -name '*.c' | xargs cat | cc -     # this clever cc can handle it :)
>
> This program works fine until there are no .c files to be found, in that case
> it hangs, waiting for one on stdin!  This is a hazard to shell scripters, and
> a potential source of security holes.

Your example doesn't hang (and if it does, your xargs is
broken).  You are thinking of something like this:

$ echo 'cat $*' > foo.sh
$ sh foo.sh

This is not elegant but a reasonable tradeoff.  A common use
of many tools is in a pipeline and having to type - every
time can get annoying.

To "fix" this you may think of changing your shell to append
/dev/null if a command is given no arguments but that will
fail in cases like `cat -v'.  In unix it is upto each command
to interprets its arguments and a shell can not assume
anything about command arguments.

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

On Sun, 04 Oct 2009 03:03:27 +1100 Sam Watkins <sam@nipl.net>  wrote:
> find -name '*.c' | xargs cat | cc -

On Sat, Oct 03, 2009 at 11:46:16AM -0700, Bakul Shah wrote:
> Your example doesn't hang (and if it does, your xargs is broken).

hm sorry, I meant:

  cat `find -name *.c` | cc -

> A common use of many tools is in a pipeline and having to type - every time
> can get annoying.

We could differentiate tools that work on stdin from tools that work on a list
of files.  Or each tool could default to one or the other.

If a tool normally works on a list of files (like cat), it can also use stdin:

  cat -

If a tool normally works on stdin (like sed), it can use a list of files:

  grep pattern : foo bar baz

By the way, grep is even more inconsistent than usual: it only adds the file:
prefix to matching lines when there are 2 or more files!

I don't see how this can be fixed in unix without breaking umpteen million
shell scripts.  But perhaps this could be corrected in plan 9 or inferno which
are still more experimental OSes; or we could keep it in mind for "plan 10".

Sam

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

> I don't see how this can be fixed in unix without breaking umpteen million
> shell scripts.

By creating new commands with distinct new names.

++L

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

I wrote:
> I don't see how this can be fixed in unix without breaking umpteen million
> shell scripts.

On Sun, Oct 04, 2009 at 06:12:15AM +0200, lucio@proxima.alt.za wrote:
> By creating new commands with distinct new names.

I thought of a better way.  We can fix the commands without breaking
compatibility, using `--'.  `--' in unix normally excludes further options.
My idea is that `--' should force multi-file mode, excluding different
behaviour in case of an empty list or a single-member list.

  `--' ends options, forces consistent multi-file mode, allows empty lists

This gives the best of both worlds - existing usage is still valid, but people
can choose to use new syntax when dealing with lists of files.
It is consistent with the current meaning of `--' in unix.

Examples of new usage (using rc this time):

  # wrap stdin with a header and footer
  cat header - footer

  fn find { du -a $* | awk '{print $2}' }    # du/find are not fixed yet

  # cat all C files, ok if none
  cat -- `{find . | grep '\.c$'}

  # same thing, with grep instead, always outputs file:matching-line
  grep malloc -- `{find . | grep '\.c$'}

Down with DWIM!!

What do you think of this proposal?
Can you think of a real example where it would be incompatible?


Sam

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

> What do you think of this proposal?

I think the saying is "the game is not worth the candle".  I doubt
anyone should invest the time and effort to implement this, specially
as few will adopt its use.  Of course, fifty years from now everyone
would be savvy to it, but by then one expects a different environment
and this type of consistency will have lost meaning.

Plus, by adding this feature you're not achieving the objective I
thought you sought, which was to make the commands inherently
consistent: you're retaining the inconsistency, but candy-coating it.
My long-ago mentor would suggest you have not yet read Catch-22.

++L

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

> you're retaining the inconsistency, but candy-coating it.

No, I'm offering a simple syntax using which one can avoid the inconsistency.
I'm retaining the option to have inconsistent behaviour, for backward
compatibility, and because some people seem to like it for command-line use.

  cat *      # still ok, will break if no files match, or a file called -v etc
  cat -- *   # more reliable  (so long as * returns empty if it fails to match)

The new syntax comes for free with `--', which is the standard syntax used by
almost all programs that take options and also process one or more files named
on the command line.  `--' is needed so that these programs can cope with
filenames starting with a dash like `-README-' for example, and not confuse
them with options.  All serious scripters should be using this `--' already,
especially if they are doing sysadmin work, otherwise their scripts may break
or go on a rampage deleting stuff when some user makes a file called `-rf'.

This fix (to support zero-length file lists when using `--') would not only
keep compatibility, it would also fix a whole lot of buggy scripts that are
using `--' but not checking for the empty list.

> I doubt anyone should invest the time and effort to implement this

I've added support for this feature into my get_options function, which I use
in my own programs.  It took me about 1 minute to do that.  I expect a similar
amount of time would be needed to fix each major tool having the problem.
There might be 10 or 20 such tools that are commonly used in plan 9.
So it would be less than a couple hours work.

> cat -
> cat /fd/0
> cat

There's no longer any need to implement support for `cat -' and similar to
avoid the problem of mishandling empty file lists, although it might be nice.

I'm sick of this topic like everyone else must be so that's all from me.


Sam

Re: [9fans] mishandling empty lists - let's fix it

[Federico G. Benavento]

On Mon, Oct 5, 2009 at 3:20 AM, Sam Watkins <sam@nipl.net> wrote:
>> you're retaining the inconsistency, but candy-coating it.
>
> No, I'm offering a simple syntax using which one can avoid the inconsistency.
> I'm retaining the option to have inconsistent behaviour, for backward
> compatibility, and because some people seem to like it for command-line use.
>
>  cat *      # still ok, will break if no files match, or a file called -v etc
>  cat -- *   # more reliable  (so long as * returns empty if it fails to match)
>
> The new syntax comes for free with `--', which is the standard syntax used by
> almost all programs that take options and also process one or more files named
> on the command line.  `--' is needed so that these programs can cope with
> filenames starting with a dash like `-README-' for example, and not confuse
> them with options.  All serious scripters should be using this `--' already,
> especially if they are doing sysadmin work, otherwise their scripts may break
> or go on a rampage deleting stuff when some user makes a file called `-rf'.
>

the way you usually deal with this is by prefixing the files with "./"
as in rm -fr ./-fr

if that doesn't work you can always use basename(1) or even pwd(1)...

-- 
Federico G. Benavento

Re: [9fans] mishandling empty lists - let's fix it

[Richard Miller]

>   `--' ends options, forces consistent multi-file mode, allows empty lists

Hey, yeah, great idea.  And why not use '---' to force switching to
Linux compatible mode, and '----' to switch to BSD syntax, and '-----'
for System V syntax (5 hyphens, System 5, easy to remember).  Oh, and we
might need a way to switch back to Plan 9 syntax.  That better be ''
(0 hyphens).

Example:

  cat ---- -s ----- -s '' -v

prints the contents of file named '-v', squeezing multiple empty lines
(BSD -s) and silencing error message (System V -s).

Re: [9fans] mishandling empty lists - let's fix it

[Uriel]

On Sun, Oct 4, 2009 at 12:46 PM, Richard Miller <9fans@hamnavoe.com> wrote:
>>   `--' ends options, forces consistent multi-file mode, allows empty lists
>
> Hey, yeah, great idea.  And why not use '---' to force switching to
> Linux compatible mode, and '----' to switch to BSD syntax, and '-----'
> for System V syntax (5 hyphens, System 5, easy to remember).  Oh, and we
> might need a way to switch back to Plan 9 syntax.  That better be ''
> (0 hyphens).
>
> Example:
>
>  cat ---- -s ----- -s '' -v
>
> prints the contents of file named '-v', squeezing multiple empty lines
> (BSD -s) and silencing error message (System V -s).

I'm sure the people in charge of writing the next version of the PoSix
standard will be very happy to adopt your proposal!

That is, if they have not independently 'discovered' this fantastic
solution to this horrible 'problem' already on their own.

Of course, they could also move with the times, and adopt an XML
format for command arguments which would be much more extensible and
Web 2.0 compliant.

That way all commands could share the same dynamically linked parser,
and one could write scripts with their favorite XML-editor!

uriel

Re: [9fans] mishandling empty lists - let's fix it

[Jacob Todd]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

On Tue, Oct 06, 2009 at 08:59:32AM +0200, Uriel wrote:
> I'm sure the people in charge of writing the next version of the PoSix
> standard will be very happy to adopt your proposal!
> 
> That is, if they have not independently 'discovered' this fantastic
> solution to this horrible 'problem' already on their own.
> 
> Of course, they could also move with the times, and adopt an XML
> format for command arguments which would be much more extensible and
> Web 2.0 compliant.
> 
> That way all commands could share the same dynamically linked parser,
> and one could write scripts with their favorite XML-editor!
> 
> uriel
> 

Breaking news: bash now uses mysql to store your history,  environmental
variables, et cetera.

Don't forget to tune in at 7 to discuss Emacs newest feature,a Virtual Machine
that runs in the editor.

-- 
Jake Todd
// If it isn't broke, tweak it!

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: [9fans] *Suspect* Re: mishandling empty lists - let's fix it

[W B Hacker]

Jacob Todd wrote:
> On Tue, Oct 06, 2009 at 08:59:32AM +0200, Uriel wrote:
>> I'm sure the people in charge of writing the next version of the PoSix
>> standard will be very happy to adopt your proposal!
>>
>> That is, if they have not independently 'discovered' this fantastic
>> solution to this horrible 'problem' already on their own.
>>
>> Of course, they could also move with the times, and adopt an XML
>> format for command arguments which would be much more extensible and
>> Web 2.0 compliant.
>>
>> That way all commands could share the same dynamically linked parser,
>> and one could write scripts with their favorite XML-editor!
>>
>> uriel
>>
>
> Breaking news: bash now uses mysql to store your history,  environmental
> variables, et cetera.
>
> Don't forget to tune in at 7 to discuss Emacs newest feature,a Virtual Machine
> that runs in the editor.
>

That second part ain't new.

But it's a virtual machine that mostly runs its *mouth*.

Bill

Re: [9fans] mishandling empty lists - let's fix it

[sqweek]

2009/10/4 Sam Watkins <sam@nipl.net>:
> I wrote:
>> I don't see how this can be fixed in unix without breaking umpteen million
>> shell scripts.
>
> On Sun, Oct 04, 2009 at 06:12:15AM +0200, lucio@proxima.alt.za wrote:
>> By creating new commands with distinct new names.
>
> I thought of a better way.  We can fix the commands without breaking
> compatibility, using `--'.  `--' in unix normally excludes further options.

 It seems to me the obvious way to gain consistency is to do the list
parsing in one place only:

fn apply {
    cmd=$1
    shift
    while(! test $1 = :) {
        cmd=($cmd $1)
        shift
    }
    for(i in $*) $cmd <$i
}
# eg: apply grep foo : *.c

 Of course, this is likely to run you into other problems with the way
some programs want to use their arguments, which is why unix/plan 9
adopt the pragmatic approach.

 PS. excuse the test usage, feel free to point out the ~ idioms - if I
see them often enough they might stick in my memory :)
 PPS. i didn't actually check the semantics of for on an empty list,
maybe a conditional is required

-sqweek

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

sqweek:
> It seems to me the obvious way to gain consistency is to do the list parsing
> in one place only:

hi sqweek,

Thanks for the thoughtful response.  You are right, it could be fixed with
another tool like xargs.  I wrote a similar tool "modify" which I use to modify
files in place with standard tools, like `modify nl : a b c` to number lines in
three files.  Given that nearly all the tools do already handle lists of
arguments (wrongly), I saw a need to correct them.

Plan 9 tools that accept options already do accept `--' to mark the end of
options and the start of proper arguments, it is a necessary feature.

I will show what sort of change would be needed for grep.  I have not looked at
the source for plan 9 grep, so this is just an example.

If the existing code was:

	char *pattern;
	int use_stdin, prefix_filename;
	...
		if strcmp(argv[i], "--") == 0
			...
	...
	pattern = argv[0]; ++argv; --argc;
	use_stdin = !argc;
	prefix_filename = argc > 1;

The changed code would be:

	char *pattern;
	int dwim, use_stdin, prefix_filename;
	dwim = 1;
	...
		if strcmp(argv[i], "--") == 0
			dwim = 0
			...
	...
	pattern = argv[0]; ++argv; --argc;
	use_stdin = dwim && !argc;
	prefix_filename = !dwim || argc > 1;

The difference from current logic is very slight.  It's not rocket science.

Then something like:

  grep foo -- `{find ...}

would work correctly and consistently.

  grep foo -- *

will still not work because rc's globbing also fears zero and returns the
pattern instead of an empty list if there are no matches.

Sam

Re: [9fans] mishandling empty lists - let's fix it

[matt]

sqweek wrote:

>
>fn apply {
>    cmd=$1
>    shift
>    while(! test $1 = :) {
>        cmd=($cmd $1)
>        shift
>    }
>    for(i in $*) $cmd <$i
>}
># eg: apply grep foo : *.c
>
>
>

that's the beauty of the rc shell, you can define what you like

fn cat {
    if(~ $#* 0)
        /bin/cat /dev/null
    if not {
       as = ()
       for(a in $*) {
          if(~ $a -)
             as = ($as /fd/0)
          if not
             as = ($as $a)
       }
       /bin/cat $as
    }
}

now
% echo test | cat
% echo test | cat -
test
%  cat  /lib/words | sed 1q
AAA
%

If your shell programs are defeated by unexpected empty lists I suggest
you write better shell programs :)

Re: [9fans] mishandling empty lists - let's fix it

[Charles Forsyth]

>  cat -

cat /fd/0

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

>>  cat -
>
> cat /fd/0

cat

:-)

++L

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

>   cat ---- -s ----- -s '' -v
>
> prints the contents of file named '-v', squeezing multiple empty lines
> (BSD -s) and silencing error message (System V -s).

classic!

- erik

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

>   grep foo -- *
>
> will still not work because rc's globbing also fears zero and returns the
> pattern instead of an empty list if there are no matches.

i used to think that was a misfeature.
in college i wrote a shell that elided
failed globs in the argument list.

the problem is that people assume that
failed globs generate an error, e.g.
	cat «glob pattern» >[2=] || echo cat failed
i didn't redesign the system, so that would
hang with my shell.

if we follow your redesign of unix, we
would never get an error because
	cat *
does nothing and wouldn't be an error
at all.  one would need to write
	x = *
	if(~ $#* 0)
		echo cat failed
	if not
		cat $x
that seems less attractive to me.

- erik

Re: [9fans] mishandling empty lists - let's fix it

[Russ Cox]

>         cat *
> does nothing and wouldn't be an error
> at all.

worse, if you follow this to its logical conclusion,
any word that isn't an actual file name must be
a failed glob expression and should be discarded.
so grep -n would be the same as grep, unless
you have a file named -n.

to the original poster: there's definitely a logical
consistency to your original proposal.  others have
pointed out specific flaws, but the fundamental issue
is that it's just not the same system anymore.

russ

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

John Stalker:
> For what it's worth, I agree that it's a problem.

hooray!  thanks :)

> Does it need fixing?  Implementing it properly would break most shell scripts
> I've written.

Well, I have a fix that should not break any real scripts, I repeat it below.

Russ Cox:
> to the original poster: there's definitely a logical consistency to your
> original proposal.  others have pointed out specific flaws, but the
> fundamental issue is that it's just not the same system anymore.

Thanks.  I've already shown how to fix the problem by adding a meaning to `--':
if `--' is seen, do not do anything dwimmy in the case of 0 or 1 files in the
list, treat them like any longer list.  The system is very much the same.

My solution fixes the problem and I claim it would not break any existing real
shell scripts, or upset interactive use, however it would remove many bugs.


Now, about the globbing...

erik:
> if we follow your redesign of unix, we would never get an error because
> cat * does nothing and wouldn't be an error at all.

"redesign of unix" is overstating it.

There should not be an error when a glob or pattern matches nothing.
"none" is a perfectly valid reply to the query *.c - what .c files are here?

For interactive use, an environment variable WARNIFEMPTY or something could
request warnings (or ERRORIFEMPTY for errors) from the shell if a glob expands
to nothing; or from commands if a command is run with no files to operate on.

Or perhaps we could have a different globber that does the right thing,
such as `{glob *} - that would be okay for scripts, and the status quo is
tolerable for interactive use.

Maybe the regular globbing can't be fixed without breaking stuff - but I don't
know of any program that does depend on this bogus misfeature, and I'm sure I
could find plenty of scripts that break because of it... and it's no good to
blame the writers of those scripts, it is a clear fault in the system.

I don't admire the various god-awful hacks proposed to avoid pitfalls stemming
these bugs.


Sam

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> For interactive use, an environment variable WARNIFEMPTY or something could
> request warnings (or ERRORIFEMPTY for errors) from the shell if a glob expands
> to nothing; or from commands if a command is run with no files to operate on.
>
> Or perhaps we could have a different globber that does the right thing,
> such as `{glob *} - that would be okay for scripts, and the status quo is
> tolerable for interactive use.

i'm pretty sure that modal stuff like this would be worse
than the status quo.

> Maybe the regular globbing can't be fixed without breaking stuff - but I don't
> know of any program that does depend on this bogus misfeature,

now who's overstating.  :-)  i highly suggest you implement your
suggestions.  i look forward to seeing a paper or wip at iwp9 in
2010.

- erik

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

I think that you're right to complain, but that you aren't quite
complaining about the right thing and are therefore offering the
wrong solution.  The problem is utilities which take list arguments,
but which treat empty lists differently.  Shell globbing rules can
make this better or worse, depending on context, but the problem
is in the semantics of our favourite utilities.  In a consistent
world

`cat' would concatenate 0 files, i.e. ouptut nothing,
`chmod 755' would set the permissions of no files to 755,
`cp foo/' would move no files to the directory foo,
`df' would show free disk space for no filesystems,
`kill' would stop no processes,
`ln foo/' would hardlink no files into the directory foo,
`ls' would list no files (you would use `ls .' for the usual case),
`mkdir' would create no directories,
`mv foo/' would move no files into the directory foo,
`ps -U' would show no users' processes,
`rm' would remove no files,
and `sh' would execute no scripts in the Bourne shell.

None of these would produce errors.  These are UNIX examples, but
Plan 9 is not much different in this regard.  A quick glance shows
that somewhere between a third and a half of all the basic utilities
have this property.  Nothing you do to the shell will change this
much.  A quick look through shell scripts I've written recently
convinces me that almost all of them would break if the basic
utilities were modified to treat empty lists consistently, even
though only one uses file name globbing in any way.

If you really want to fix the problem then the sensible thing
to do would be to write new versions of many utilities, with new
names, and then write a shell without globbing.  For new scripts
you would use the new utilities and shell and leave everything
else as it is.  Is it worth the effort?  That's a question which
can only be answered by the person who would be doing the work.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[Jason Catena]

> If you really want to fix the problem then the sensible thing
> to do would be to write new versions of many utilities, with new
> names, and then write a shell without globbing.  For new scripts
> you would use the new utilities and shell and leave everything
> else as it is.  Is it worth the effort?  That's a question which
> can only be answered by the person who would be doing the work.

I see the point of having utilities handle no-input without failing
and without error output (other than maybe a status to test the case),
when their output is used as input to other utilities, especially as
embedded commands on the command line.  I often use the backtick style
to output into an assignment to *, to separate with $1 etc.  Currently
if a program fails, then instead of no output (maybe ok, depending on
the program) it delivers a noisy error message and an exit status.

This style is not the most common use case: most people mean to change
the state of the system when they run a command, and want to see error
messages when a list turns up empty.  So I'd welcome commands which
behave the new way, and I wouldn't have a problem calling them by
other names.  (My first thought was the suffix 0. You might want to
avoid the prefix g.)

> John Stalker

Jason Catena

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

> So I'd welcome commands which
> behave the new way, and I wouldn't have a problem calling them by
> other names.

My thinking is that a shell with slightly different command execution
semantics may be required, together with commands and/or built-in
commands and functions that behave consistently with the shell's
philosophical design.

In addition, I don't think an empty list should ever be implied, or at
minimum ought to be able to be specified explicitly, () would suit me
fine, '' may or may not be a viable alternative.

But we've drifted off a critique of conventional usage to the realms
of philosophical design and the terms of discussion are now very
different.

++L

Re: [9fans] mishandling empty lists - let's fix it

[ron minnich]

On Mon, Oct 5, 2009 at 2:04 PM, John Stalker <stalker@maths.tcd.ie> wrote:

> `cat' would concatenate 0 files, i.e. ouptut nothing,

cat would copy stdin to stdout

> `chmod 755' would set the permissions of no files to 755,

would read a list of files from stdin and change the modes

> `cp foo/' would move no files to the directory foo,
would read a list of files from stdin and copy them.

> `df' would show free disk space for no filesystems,
would read a list of file system names and df them.

> `kill' would stop no processes,
would read a list of pids in and stop them.

> `ln foo/' would hardlink no files into the directory foo,
would read a list of filenames in and ln them

> `ls' would list no files (you would use `ls .' for the usual case),
would read a list of file names in and ls them.

> `mkdir' would create no directories,
would read a list of file names in and mkdir them (I could really use this one)

> `mv foo/' would move no files into the directory foo,
would read a list of file names in and mv them

> `ps -U' would show no users' processes,
would read a list of user names in and ps them

> `rm' would remove no files,
would read a list of file names in and rm them

> and `sh' would execute no scripts in the Bourne shell.

would read a set of commands from stdin and ... oh wait it does this.

I prefer my version to your versions. I can see uses for them already.

ron

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> > `ln foo/' would hardlink no files into the directory foo,
> would read a list of filenames in and ln them

better: ln: '/bin/ln' directory entry not found

- erik

Re: [9fans] mishandling empty lists - let's fix it

[ron minnich]

On Tue, Oct 6, 2009 at 4:37 AM, erik quanstrom <quanstro@coraid.com> wrote:
>> > `ln foo/' would hardlink no files into the directory foo,
>> would read a list of filenames in and ln them
>
> better: ln: '/bin/ln' directory entry not found

well, yea, but I didn't want to get into that one :-)

ron

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> I prefer my version to your versions. I can see uses for them already.
>
> ron

So do I, though I'm not sure I would have called the latter "my"
versions.  That's not the point however.  Since I don't seem to
been have sufficiently clear, I'll reword it a bit:

 There are a lot basic UNIX/p9 utilities which treat empty lists
 inconsistently.  It would be nice if there were fewer, but you
 can't arrange that without breaking many things.

The list I gave was what you would get by looking through /bin on
a BSD system and making empty lists behave like other lists, just
empty.  Most of your list would, in fact, be more useful, but of
of course break even more existing scripts.

I think you got one wrong, however.
>> `cat' would concatenate 0 files, i.e. ouptut nothing,
>cat would copy stdin to stdout
How do you perform the original, concatenating, function of
cat with this version?  With "my" version I can type `cat -'
or at worst `cat /dev/fd/0' to replicate the behaviour of
"your" version.  An idea more along the lines of your others
would be that cat reads a list of files on stdin and
concatentes their contents to stdout.

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[Brian L. Stuart]

> > I prefer my version to your
> versions. I can see uses for them already.
> > 
> > ron
> 
> So do I, though I'm not sure I would have called the latter
> "my"
> versions.  That's not the point however.  Since I
> don't seem to
> been have sufficiently clear, I'll reword it a bit:
> 
>  There are a lot basic UNIX/p9 utilities which treat empty
> lists
>  inconsistently.  It would be nice if there were
> fewer, but you
>  can't arrange that without breaking many things.

Ultimately it comes down to a question of perspective and
aesthetics.  If you view these tools as primarily programs
that act on files listed on the command line, then you would
prefer some of the proposals for change that have been made.
If you view these tools as primarily filters to be connected
by pipes, then the command line arguments are just conveniences
so as not to type < and > as often.  Then you prefer the tools
as they have been.  Neither perspective is a bug, for each
fulfills it's requirements.

Let each code and maintain what fits their preference and
aesthetic.  If others like it, they will use if.  If they
don't they won't.

Now let's make room for a discussion on ARM ports.

BLS

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> If you view these tools as primarily programs that act on files
> listed on the command line, then you would prefer some of the
> proposals for change that have been made.  If you view these tools
> as primarily filters to be connected by pipes, then the command
> line arguments are just conveniences so as not to type < and > as
> often.  Then you prefer the tools as they have been.

I don't want to extend further a discussion which is clearly
already past its ``sell by'' date, but didn't you mean the
other way around?  The inconsistencies in current UNIX/p9
seem to cater to simple interactive use at the expense
both of scripts and of more complex interactive use.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[ron minnich]

one last note on this because it is kind of interesting.

Consistency is good. Sometimes it goes too far. The RCA computer OS
guys decided that CR and LF were not really command delimiters, that
rather ETX made the most sense. So to end a command you had to
remember to type ^C.

Consistent, totally. Conformant with the intent of ASCII, probably.
Kind of interesting, easy to embed CR and LF in your commands. And
utterly horrible to use.

There are many examples of this kind on computing, where consistency
was taken just a bit too far and led into a nasty thicket.

I think that in limited situations, inconsistency is a good thing.
And, conversely, one can take consistency as an absolute good, and run
off the road.

Some people argue that Plan 9 has run off the road for just that
reason. Many of us would beg to disagree.

That's where the judgement lies, I guess.

ron

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> one last note on this because it is kind of interesting.
...
> I think that in limited situations, inconsistency is a good thing.
> And, conversely, one can take consistency as an absolute good, and run
> off the road.
...
> ron

I agree with most of this, and with most of what I've edited out.
But I think that in this case we can distinguish two types of
utilities: those that behave inconsistently for a well thought out
and useful reason, and those which are inconsistent merely because
someone wasn't thinking properly.

Cat, for example, is inconsistent for sensible reasons.  In a world
where it has options, and p9 cat still doesn't, the convention of
assuming stdin in the absence of arguments is mildly useful.  It's
also easy to add /dev/null to the list if you want to avoid that
behaviour.  Globbing can be a problem here, which is what the OP was
complaining about, but that is, if anything, a problem with the shell.

Rm is, I think, an example of someone who simply wasn't thinking
properly.  In

	rm `{ complicated pipeline }

I almost certainly want the exit status to reflect whether all the
selected files were deleted, even if there weren't any, so I want
rm without arguments to succeed silently.  There is no real advantage
to the present behaviour.  Yes, it warns you that you might not have
removed what you wanted to, but that's not the way UNIX and it's
descendants are supposed to behave.  After all, if I type

	rm -rf $jnukdir/*

then I don't expect a warning that $jnukdir is not defined while
$junkdir is, and that perhaps what I really meant was

	rm -rf $junkdir

Everything can be carried to extreme.  Even ancestor worship.  The
basic utilities were written by smart people who didn't always fully
understand what they were doing.  There is nothing radical about
suggesting that we try not to repeat their mistakes.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[roger peppe]

2009/10/7 John Stalker <stalker@maths.tcd.ie>:
> Rm is, I think, an example of someone who simply wasn't thinking
> properly.  In
>
>        rm `{ complicated pipeline }
>
> I almost certainly want the exit status to reflect whether all the
> selected files were deleted, even if there weren't any, so I want
> rm without arguments to succeed silently.

it does under plan 9.

FWIW, my experimental "alphabet" shell formalised the types of command line
arguments and checked usage before invoking the actual command.
each option took a fixed number of arguments; options were followed
by zero or more fixed arguments, optionally followed by a repeated
argument of the same type. this was sufficient for many of the conventional
commands, but not, for instance, mv which has a fixed argument at the
end.

as with any language, static typing is an advantage (it avoids some errors)
and a disadvantage (it's hard to do some things because they don't fit
within the type system). mileage varies.

Re: [9fans] mishandling empty lists - let's fix it

[matt]

You can build this whole dream using plan9 and show the world, you don't
have to rewrite any of the utilities, just shape the calling convention
to your liking.
I bet you could even find the pattern and wrap the whole lot in a meta
script, maybe even utilise an FS


fn mkdir {
    if(! ~ $#* 0)
       /bin/mkdir $*
ifs = '
'
    m = `{read}
     while(! ~ $#m 0) {
          ifs = ' '
          p = `{echo -n $m}
         /bin/mkdir $p
          ifs='
'
          m = `{read}
    }
}

with no newlines in directory names

>*On Mon, Oct 5, 2009 at 2:04 PM, John Stalker <stalker@maths.tcd.ie> wrote:
>
>
>
>>`cat' would concatenate 0 files, i.e. ouptut nothing,
>>
>>
>
>cat would copy stdin to stdout
>
>
>
>>`chmod 755' would set the permissions of no files to 755,
>>
>>
>
>would read a list of files from stdin and change the modes
>
>
>
>>`cp foo/' would move no files to the directory foo,
>>
>>
>would read a list of files from stdin and copy them.
>
>
>
>>`df' would show free disk space for no filesystems,
>>
>>
>would read a list of file system names and df them.
>
>
>
>>`kill' would stop no processes,
>>
>>
>would read a list of pids in and stop them.
>
>
>
>>`ln foo/' would hardlink no files into the directory foo,
>>
>>
>would read a list of filenames in and ln them
>
>
>
>>`ls' would list no files (you would use `ls .' for the usual case),
>>
>>
>would read a list of file names in and ls them.
>
>
>
>>`mkdir' would create no directories,
>>
>>
>would read a list of file names in and mkdir them (I could really use this one)
>
>
>
>>`mv foo/' would move no files into the directory foo,
>>
>>
>would read a list of file names in and mv them
>
>
>
>>`ps -U' would show no users' processes,
>>
>>
>would read a list of user names in and ps them
>
>
>
>>`rm' would remove no files,
>>
>>
>would read a list of file names in and rm them
>
>
>
>>and `sh' would execute no scripts in the Bourne shell.
>>
>>
>
>would read a set of commands from stdin and ... oh wait it does this.
>
>I prefer my version to your versions. I can see uses for them already.
>
>ron
>
>
>

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

I've been away for a while and I see that the list has become
troll-infested in my absence.  Not that there aren't still plenty
of intelligent people around, but the signal to noise ratio has
passed a certain critical threshold.  In general I prefer places
where most of the people who reply to posts have read them, and
perhaps even attempted to think about them.  Some of you may
remember when 9fans used to work like that.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[Jason Catena]

The pupils of the Tendai school used to study meditation before Zen
entered Japan. Four of them who were intimate friends promised one
another to observe seven days of silence.

On the first day all were silent. Their meditation had begun
auspiciously, but when night came and the oil lamps were growing dim
one of the pupils could not help exclaiming to a servant: "Fix those
lamps."

The second pupil was surprised to hear the first one talk. "We are not
supposed to say a word," he remarked.

"You two are stupid. Why did you talk?" asked the third.

"I am the only one who has not talked," concluded the fourth pupil.

Re: [9fans] mishandling empty lists - let's fix it

[Tim Newsham]

> You can build this whole dream using plan9 and show the world, you don't have
> to rewrite any of the utilities, just shape the calling convention to your
> liking.
[...]
> fn mkdir {

just write a single combinator that when applied to a command
makes it behave in the "no non-flag arguments == dont run" way.
Then its just:

     l0 cat $args
     l0 chmod +x $args
     ...

Tim Newsham
http://www.thenewsh.com/~newsham/

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> just write a single combinator that when applied to a command
> makes it behave in the "no non-flag arguments == dont run" way.
> Then its just:
>
>      l0 cat $args
>      l0 chmod +x $args

> Tim Newsham

This mostly works, but misses a few cases.  For example ln, mv, cp
a possibly empty list of files to a directory.  If someone really
wanted to implement this, and I don't, in case that wasn't obvious,
then the most efficient way to do it would be to use your combinator
for most of the cases and handle the others individually.

There is also a problem with identifying non-flag arguments.  In

	foo -x bar

is bar a non-flag argument or not?  You really have to read the
man page of foo to find out whether you are dealing with something
like

 	foo [-x user ] file ...

or something more like

	for [-x] file ...

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[Jason Catena]

On Tue, Oct 6, 2009 at 14:27, John Stalker <stalker@maths.tcd.ie> wrote:
>> just write a single combinator that when applied to a command
>> makes it behave in the "no non-flag arguments == dont run" way.
>> Then its just:
>>
>>      l0 cat $args
>>      l0 chmod +x $args
>
>> Tim Newsham
>
> There is also a problem with identifying non-flag arguments.

For the original post's purpose ("commands such as cat, grep etc. do
not handle an empty argument list correctly") we might not have to go
this far.  If these new semantics are meant to process command line
arguments, then they should not be used with the l0 combinator to
process pipe input, which means they won't be called without some
attempt to supply them with arguments after options.  Common cases
where the attempt might provide no arguments would be shell variables
or backticked commands, which are both code that gets further
processed.  We can use this to delay the processing into the
combinator, and not execute the utility command if we have nothing to
do.

So, it should be sufficient to either (1) quote the argument-providing
code in the command line, to delay evaluating it until immediately
before the utility command in the combinator, and if non-nil apply its
output to the utility command; or (2) add some null argument like
/dev/null, depending on the command, in the combinator after any other
arguments, to make it quietly do nothing if the argument-providing
code evaluates to nil in the calling script.

Seems to me we have three different command sets now: one with the
current, traditional semantics which mix argument and pipe input, and
handle missing input inconsistently; a set (listed by John) which
handle no command-line arguments consistently. generally doing
nothing; and a set (listed by Ron) which handle standard input
consistently.  Of the three, Ron's set seems most interesting to me:
(1) if your tools can handle their arguments on standard input, they
ought to cleanly handle the null case too, (2) pipe is a cleaner
notation for passing output along than backtick within backtick within
backtick, and (3) pipes commonly pass multiple lines, not just
separated fields.

> John Stalker

Jason Catena

Re: [9fans] mishandling empty lists - let's fix it

[Russ Cox]

> if `--' is seen, do not do anything dwimmy in the case of 0 or 1 files in the
> list, treat them like any longer list.  The system is very much the same.

No, it's more complicated, because
it is two systems with a mode bit
to choose between them.

Russ

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

>  There are a lot basic UNIX/p9 utilities which treat empty lists
>  inconsistently.  It would be nice if there were fewer, but you
>  can't arrange that without breaking many things.

the beauty is that these utilities do inconsistent things,
and therefore are not easily confused.

we're arguing that birds, mammals and fish should all have
the same number of legs.  let's hope there are no centipeds.

- erik

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> Everything can be carried to extreme.  Even ancestor worship.  The
> basic utilities were written by smart people who didn't always fully
> understand what they were doing.  There is nothing radical about
> suggesting that we try not to repeat their mistakes.

well why not take your own advice and implement this.
live with it.  let us know how it goes.

- erik

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> > Everything can be carried to extreme.  Even ancestor worship.  The
> > basic utilities were written by smart people who didn't always fully
> > understand what they were doing.  There is nothing radical about
> > suggesting that we try not to repeat their mistakes.
>
> well why not take your own advice and implement this.
> live with it.  let us know how it goes.

> - erik

Would you please start reading posts before replying to them?  The
only thing I have proposed is that people stop gratituously creating
programs which treat empty lists exceptionally.  If that is what you
mean by the paragraph above then I do try to implement it, and am
quite happy with the results.  But I think you mean something else.
You mean that I want to replace cat, rm, chmod etc. by programs which
adhere to this principle.  I've said several times that I am not
going to do this and that I think it would break too many things,
including most of my own shell scripts.
--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> Would you please start reading posts before replying to them?  The
> only thing I have proposed is that people stop gratituously creating
> programs which treat empty lists exceptionally.  If that is what you
> mean by the paragraph above then I do try to implement it, and am
> quite happy with the results.  But I think you mean something else.
> You mean that I want to replace cat, rm, chmod etc. by programs which
> adhere to this principle.  I've said several times that I am not
> going to do this and that I think it would break too many things,
> including most of my own shell scripts.

i have read every message in this thread.  why accuse me
of something you can't possibly know?

i thought you were changing your position.  i guess that was
wrong.  but can you blame me?  this thread goes around in
circles with no possibility of any code being written or results.
i was trying to gently suggest that this thread be
suspended until somebody does something.

- erik

Re: [9fans] mishandling empty lists - let's fix it

[W B Hacker]

erik quanstrom wrote:
>> Would you please start reading posts before replying to them?  The
>> only thing I have proposed is that people stop gratituously creating
>> programs which treat empty lists exceptionally.  If that is what you
>> mean by the paragraph above then I do try to implement it, and am
>> quite happy with the results.  But I think you mean something else.
>> You mean that I want to replace cat, rm, chmod etc. by programs which
>> adhere to this principle.  I've said several times that I am not
>> going to do this and that I think it would break too many things,
>> including most of my own shell scripts.
>
> i have read every message in this thread.  why accuse me
> of something you can't possibly know?
>
> i thought you were changing your position.  i guess that was
> wrong.  but can you blame me?  this thread goes around in
> circles with no possibility of any code being written or results.
> i was trying to gently suggest that this thread be
> suspended until somebody does something.
>
> - erik
>

Look at John's initial post on the thread, wherein he gave fair warning of his
intent and his target victim pool.

Took him just one day - despite time-zone differences - to demonstrate he could
do it - smart folks and vacuum-tube-CPU-age argument not withstanding.

Don't blame him.

We played his game.

Bill Hacker
(Fourth Buddhist...)