[9fans] mishandling empty lists - let's fix it

[9fans] mishandling empty lists - let's fix it

[Sam Watkins]

hi,

I wanted to have a whinge about one fault I find in unix: commands such as cat,
grep etc. do not handle an empty argument list correctly.  For example,

  cat

should output nothing and exit - concatenating 0 files.  Instead it copies
stdin to stdout, which is inconsistent.  This problem still exists in plan 9.

Copying stdin should be coded as:

  cat -

Such syntax would provide useful extra capability:

  find -type d | cat lib/emptydirs - | mail 9fans@9fans.net

Although this bug is not annoying for command-line work, consider a script that
processes all .c files to be found using cat or grep:

  find -name '*.c' | xargs cat | cc -     # this clever cc can handle it :)

This program works fine until there are no .c files to be found, in that case
it hangs, waiting for one on stdin!  This is a hazard to shell scripters, and a
potential source of security holes.  It's worse than cat -v.

Another example:

  8c && echo 1

should compile 0 files to give an empty object file, and echo 1.

Is there any chance we could fix these things in plan 9/inferno?
I feel that freedom from past mistakes is a key principle of plan 9.

If people support this change, I would be keen to work on it.

thanks!

Sam


(I noticed the "pike" language has a similar bug, `+(1,2,3) -> 6
but `+() -> error!  `+() should be 0 and `*() should be 1, of course.)
I don't think such bugs exist in Lisp :)

Re: [9fans] mishandling empty lists - let's fix it

[Rob Pike]

cat * /dev/null

is the recommended solution.

-rob

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

On Sat, Oct 03, 2009 at 10:01:09AM -0700, Rob Pike wrote:
> cat * /dev/null
>
> is the recommended solution.

Thanks Rob,

That works with cat, but it won't work with chmod, grep -L, ls, find, file
and many others.  I think all of the unix and plan 9 utilities that deal
with a variable number of files have this problem.
Does anyone agree with me that it needs fixing?

Sam

Re: [9fans] mishandling empty lists - let's fix it

[Bakul Shah]

On Sun, 04 Oct 2009 03:03:27 +1100 Sam Watkins <sam@nipl.net>  wrote:
>
>   find -name '*.c' | xargs cat | cc -     # this clever cc can handle it :)
>
> This program works fine until there are no .c files to be found, in that case
> it hangs, waiting for one on stdin!  This is a hazard to shell scripters, and
> a potential source of security holes.

Your example doesn't hang (and if it does, your xargs is
broken).  You are thinking of something like this:

$ echo 'cat $*' > foo.sh
$ sh foo.sh

This is not elegant but a reasonable tradeoff.  A common use
of many tools is in a pipeline and having to type - every
time can get annoying.

To "fix" this you may think of changing your shell to append
/dev/null if a command is given no arguments but that will
fail in cases like `cat -v'.  In unix it is upto each command
to interprets its arguments and a shell can not assume
anything about command arguments.

Re: [9fans] mishandling empty lists - let's fix it

[Rob Pike]

> Does anyone agree with me that it needs fixing?

I don't. I also think that even if it were a problem the usage is far
too ingrained to be fixable.

-rob

Re: [9fans] mishandling empty lists - let's fix it

[blstuart]

>> Does anyone agree with me that it needs fixing?
>
> I don't. I also think that even if it were a problem the usage is far
> too ingrained to be fixable.

Nor do I.  Having the no-argument case be filter behavior
(stdin/stdout) is the most elegant, consistent, and predictable
of the options I've seen.

BLS

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

On Sun, 04 Oct 2009 03:03:27 +1100 Sam Watkins <sam@nipl.net>  wrote:
> find -name '*.c' | xargs cat | cc -

On Sat, Oct 03, 2009 at 11:46:16AM -0700, Bakul Shah wrote:
> Your example doesn't hang (and if it does, your xargs is broken).

hm sorry, I meant:

  cat `find -name *.c` | cc -

> A common use of many tools is in a pipeline and having to type - every time
> can get annoying.

We could differentiate tools that work on stdin from tools that work on a list
of files.  Or each tool could default to one or the other.

If a tool normally works on a list of files (like cat), it can also use stdin:

  cat -

If a tool normally works on stdin (like sed), it can use a list of files:

  grep pattern : foo bar baz

By the way, grep is even more inconsistent than usual: it only adds the file:
prefix to matching lines when there are 2 or more files!

I don't see how this can be fixed in unix without breaking umpteen million
shell scripts.  But perhaps this could be corrected in plan 9 or inferno which
are still more experimental OSes; or we could keep it in mind for "plan 10".

Sam

Re: [9fans] mishandling empty lists - let's fix it

[hiro]

> Nor do I.  Having the no-argument case be filter behavior
> (stdin/stdout) is the most elegant, consistent, and predictable
> of the options I've seen.

Yeah, I always found the lone - very ugly!

If you do a fix you could also add --outputfile in order to improve
the readability :D

Re: [9fans] mishandling empty lists - let's fix it

[Steve Simon]

> Does anyone agree with me that it needs fixing?

sorry, I don't agree that it is broken so I don't thing
it need fixing.

It does occasionally annoy me that tr(1) will not take a file
as an argument but again, changing that would have implications
too wide to make it worthwhile; I try to think of it as OS patina.

-Steve

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

> I don't see how this can be fixed in unix without breaking umpteen million
> shell scripts.

By creating new commands with distinct new names.

++L

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

I wrote:
> I don't see how this can be fixed in unix without breaking umpteen million
> shell scripts.

On Sun, Oct 04, 2009 at 06:12:15AM +0200, lucio@proxima.alt.za wrote:
> By creating new commands with distinct new names.

I thought of a better way.  We can fix the commands without breaking
compatibility, using `--'.  `--' in unix normally excludes further options.
My idea is that `--' should force multi-file mode, excluding different
behaviour in case of an empty list or a single-member list.

  `--' ends options, forces consistent multi-file mode, allows empty lists

This gives the best of both worlds - existing usage is still valid, but people
can choose to use new syntax when dealing with lists of files.
It is consistent with the current meaning of `--' in unix.

Examples of new usage (using rc this time):

  # wrap stdin with a header and footer
  cat header - footer

  fn find { du -a $* | awk '{print $2}' }    # du/find are not fixed yet

  # cat all C files, ok if none
  cat -- `{find . | grep '\.c$'}

  # same thing, with grep instead, always outputs file:matching-line
  grep malloc -- `{find . | grep '\.c$'}

Down with DWIM!!

What do you think of this proposal?
Can you think of a real example where it would be incompatible?


Sam

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

> What do you think of this proposal?

I think the saying is "the game is not worth the candle".  I doubt
anyone should invest the time and effort to implement this, specially
as few will adopt its use.  Of course, fifty years from now everyone
would be savvy to it, but by then one expects a different environment
and this type of consistency will have lost meaning.

Plus, by adding this feature you're not achieving the objective I
thought you sought, which was to make the commands inherently
consistent: you're retaining the inconsistency, but candy-coating it.
My long-ago mentor would suggest you have not yet read Catch-22.

++L

Re: [9fans] mishandling empty lists - let's fix it

[Charles Forsyth]

>  cat -

cat /fd/0

Re: [9fans] mishandling empty lists - let's fix it

[lucio]

>>  cat -
>
> cat /fd/0

cat

:-)

++L

Re: [9fans] mishandling empty lists - let's fix it

[Richard Miller]

>   `--' ends options, forces consistent multi-file mode, allows empty lists

Hey, yeah, great idea.  And why not use '---' to force switching to
Linux compatible mode, and '----' to switch to BSD syntax, and '-----'
for System V syntax (5 hyphens, System 5, easy to remember).  Oh, and we
might need a way to switch back to Plan 9 syntax.  That better be ''
(0 hyphens).

Example:

  cat ---- -s ----- -s '' -v

prints the contents of file named '-v', squeezing multiple empty lines
(BSD -s) and silencing error message (System V -s).

Re: [9fans] mishandling empty lists - let's fix it

[sqweek]

2009/10/4 Sam Watkins <sam@nipl.net>:
> I wrote:
>> I don't see how this can be fixed in unix without breaking umpteen million
>> shell scripts.
>
> On Sun, Oct 04, 2009 at 06:12:15AM +0200, lucio@proxima.alt.za wrote:
>> By creating new commands with distinct new names.
>
> I thought of a better way.  We can fix the commands without breaking
> compatibility, using `--'.  `--' in unix normally excludes further options.

 It seems to me the obvious way to gain consistency is to do the list
parsing in one place only:

fn apply {
    cmd=$1
    shift
    while(! test $1 = :) {
        cmd=($cmd $1)
        shift
    }
    for(i in $*) $cmd <$i
}
# eg: apply grep foo : *.c

 Of course, this is likely to run you into other problems with the way
some programs want to use their arguments, which is why unix/plan 9
adopt the pragmatic approach.

 PS. excuse the test usage, feel free to point out the ~ idioms - if I
see them often enough they might stick in my memory :)
 PPS. i didn't actually check the semantics of for on an empty list,
maybe a conditional is required

-sqweek

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

sqweek:
> It seems to me the obvious way to gain consistency is to do the list parsing
> in one place only:

hi sqweek,

Thanks for the thoughtful response.  You are right, it could be fixed with
another tool like xargs.  I wrote a similar tool "modify" which I use to modify
files in place with standard tools, like `modify nl : a b c` to number lines in
three files.  Given that nearly all the tools do already handle lists of
arguments (wrongly), I saw a need to correct them.

Plan 9 tools that accept options already do accept `--' to mark the end of
options and the start of proper arguments, it is a necessary feature.

I will show what sort of change would be needed for grep.  I have not looked at
the source for plan 9 grep, so this is just an example.

If the existing code was:

	char *pattern;
	int use_stdin, prefix_filename;
	...
		if strcmp(argv[i], "--") == 0
			...
	...
	pattern = argv[0]; ++argv; --argc;
	use_stdin = !argc;
	prefix_filename = argc > 1;

The changed code would be:

	char *pattern;
	int dwim, use_stdin, prefix_filename;
	dwim = 1;
	...
		if strcmp(argv[i], "--") == 0
			dwim = 0
			...
	...
	pattern = argv[0]; ++argv; --argc;
	use_stdin = dwim && !argc;
	prefix_filename = !dwim || argc > 1;

The difference from current logic is very slight.  It's not rocket science.

Then something like:

  grep foo -- `{find ...}

would work correctly and consistently.

  grep foo -- *

will still not work because rc's globbing also fears zero and returns the
pattern instead of an empty list if there are no matches.

Sam

Re: [9fans] mishandling empty lists - let's fix it

[Sam Watkins]

> you're retaining the inconsistency, but candy-coating it.

No, I'm offering a simple syntax using which one can avoid the inconsistency.
I'm retaining the option to have inconsistent behaviour, for backward
compatibility, and because some people seem to like it for command-line use.

  cat *      # still ok, will break if no files match, or a file called -v etc
  cat -- *   # more reliable  (so long as * returns empty if it fails to match)

The new syntax comes for free with `--', which is the standard syntax used by
almost all programs that take options and also process one or more files named
on the command line.  `--' is needed so that these programs can cope with
filenames starting with a dash like `-README-' for example, and not confuse
them with options.  All serious scripters should be using this `--' already,
especially if they are doing sysadmin work, otherwise their scripts may break
or go on a rampage deleting stuff when some user makes a file called `-rf'.

This fix (to support zero-length file lists when using `--') would not only
keep compatibility, it would also fix a whole lot of buggy scripts that are
using `--' but not checking for the empty list.

> I doubt anyone should invest the time and effort to implement this

I've added support for this feature into my get_options function, which I use
in my own programs.  It took me about 1 minute to do that.  I expect a similar
amount of time would be needed to fix each major tool having the problem.
There might be 10 or 20 such tools that are commonly used in plan 9.
So it would be less than a couple hours work.

> cat -
> cat /fd/0
> cat

There's no longer any need to implement support for `cat -' and similar to
avoid the problem of mishandling empty file lists, although it might be nice.

I'm sick of this topic like everyone else must be so that's all from me.


Sam

Re: [9fans] mishandling empty lists - let's fix it

[Federico G. Benavento]

On Mon, Oct 5, 2009 at 3:20 AM, Sam Watkins <sam@nipl.net> wrote:
>> you're retaining the inconsistency, but candy-coating it.
>
> No, I'm offering a simple syntax using which one can avoid the inconsistency.
> I'm retaining the option to have inconsistent behaviour, for backward
> compatibility, and because some people seem to like it for command-line use.
>
>  cat *      # still ok, will break if no files match, or a file called -v etc
>  cat -- *   # more reliable  (so long as * returns empty if it fails to match)
>
> The new syntax comes for free with `--', which is the standard syntax used by
> almost all programs that take options and also process one or more files named
> on the command line.  `--' is needed so that these programs can cope with
> filenames starting with a dash like `-README-' for example, and not confuse
> them with options.  All serious scripters should be using this `--' already,
> especially if they are doing sysadmin work, otherwise their scripts may break
> or go on a rampage deleting stuff when some user makes a file called `-rf'.
>

the way you usually deal with this is by prefixing the files with "./"
as in rm -fr ./-fr

if that doesn't work you can always use basename(1) or even pwd(1)...

-- 
Federico G. Benavento

Re: [9fans] mishandling empty lists - let's fix it

[matt]

sqweek wrote:

>
>fn apply {
>    cmd=$1
>    shift
>    while(! test $1 = :) {
>        cmd=($cmd $1)
>        shift
>    }
>    for(i in $*) $cmd <$i
>}
># eg: apply grep foo : *.c
>
>
>

that's the beauty of the rc shell, you can define what you like

fn cat {
    if(~ $#* 0)
        /bin/cat /dev/null
    if not {
       as = ()
       for(a in $*) {
          if(~ $a -)
             as = ($as /fd/0)
          if not
             as = ($as $a)
       }
       /bin/cat $as
    }
}

now
% echo test | cat
% echo test | cat -
test
%  cat  /lib/words | sed 1q
AAA
%

If your shell programs are defeated by unexpected empty lists I suggest
you write better shell programs :)

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> Does anyone agree with me that it needs fixing?

For what it's worth, I agree that it's a problem.  Maybe it's
just the fact that I'm a mathematician, but I've always felt
that any command which accepts a list of files should behave
the way I would expect when presented with an empty list.
Cat, rm, chmod, etc. do not.  At a minimum I think we should
try to avoid creating further exceptions.

Does it need fixing?  Implementing it properly would break most
shell scripts I've written.  If there were renamed versions of
cat, rm, chmod, etc. which treated empty lists in the expected
way then I would probably use them in situations were portability
is not an issue.  It would remove a frequent source of bugs and
leave me with one less thing to worry about.

Then there is the shell.  If you only use commands which behave
sanely for empty lists then the current globbing rules are a
pain.  Changing them to, for example, expand * to `' in an
empty directory breaks too many things.  I've learned to avoid
globbing as much as possible.  In any shell with command
substitution globbing is unnecessary, and is often dangerous.
I would happily use a shell with NO globbing, just to avoid
another source of worries.

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[erik quanstrom]

> I've learned to avoid
> globbing as much as possible.  In any shell with command
> substitution globbing is unnecessary, and is often dangerous.
> I would happily use a shell with NO globbing, just to avoid
> another source of worries.

i think this is all a bit dramatic.

there are very simple idioms one can
use in the few cases globbing may be
a problem.  others have suggested ./<pattern>
and i'd add that one can also use
	if(! ~ <pattern> '<pattern>')
		something;
but this is seldom necessary.  one
might consider having the file server
ban globbing characters in file names.

if i could roll back time, i would push
for regular expressions instead of the
goofy shell matching.  then knuth could
have to say that unix is 29 definitions of
regular expressions living under one roof.
:-)

- erik

Re: [9fans] mishandling empty lists - let's fix it

[John Stalker]

> i think this is all a bit dramatic.
...
> if i could roll back time, i would push
> for regular expressions instead of the
> goofy shell matching.  then knuth could
> have to say that unix is 29 definitions of
> regular expressions living under one roof.
> :-)

But you can use regular expressions instead of goofy shell
matching.  Find and ls give you lists of file names and grep
prunes them for you.  Shell command substitution then puts
them in the right place in the command, or use xargs when
appropriate.  I don't claim that globbing is a horrible evil,
merely that it is an unnecessary evil.

--
John Stalker
School of Mathematics
Trinity College Dublin
tel +353 1 896 1983
fax +353 1 896 2282

Re: [9fans] mishandling empty lists - let's fix it

[roger peppe]

2009/10/5 John Stalker <stalker@maths.tcd.ie>:
> But you can use regular expressions instead of goofy shell
> matching.  Find and ls give you lists of file names and grep
> prunes them for you.  Shell command substitution then puts
> them in the right place in the command, or use xargs when
> appropriate.  I don't claim that globbing is a horrible evil,
> merely that it is an unnecessary evil.

dodgy in unix where filenames can contain \n.
under plan 9 you have to remember to set ifs to \n only,
which is awkward and error prone.

this stuff is better designed for interactive use than scripting,
but it still does an adequate job provided you remember the pitfalls.
rc is loads better like that than shells with trad semantics.

i like the way i can parse quoted names in the inferno shell.
various p9 utils have been outfitted to print quoted names
but they're only useful interactively without support from rc.

Re: [9fans] mishandling empty lists - let's fix it

[Uriel]

On Sun, Oct 4, 2009 at 12:46 PM, Richard Miller <9fans@hamnavoe.com> wrote:
>>   `--' ends options, forces consistent multi-file mode, allows empty lists
>
> Hey, yeah, great idea.  And why not use '---' to force switching to
> Linux compatible mode, and '----' to switch to BSD syntax, and '-----'
> for System V syntax (5 hyphens, System 5, easy to remember).  Oh, and we
> might need a way to switch back to Plan 9 syntax.  That better be ''
> (0 hyphens).
>
> Example:
>
>  cat ---- -s ----- -s '' -v
>
> prints the contents of file named '-v', squeezing multiple empty lines
> (BSD -s) and silencing error message (System V -s).

I'm sure the people in charge of writing the next version of the PoSix
standard will be very happy to adopt your proposal!

That is, if they have not independently 'discovered' this fantastic
solution to this horrible 'problem' already on their own.

Of course, they could also move with the times, and adopt an XML
format for command arguments which would be much more extensible and
Web 2.0 compliant.

That way all commands could share the same dynamically linked parser,
and one could write scripts with their favorite XML-editor!

uriel

Re: [9fans] mishandling empty lists - let's fix it

[Jacob Todd]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

On Tue, Oct 06, 2009 at 08:59:32AM +0200, Uriel wrote:
> I'm sure the people in charge of writing the next version of the PoSix
> standard will be very happy to adopt your proposal!
> 
> That is, if they have not independently 'discovered' this fantastic
> solution to this horrible 'problem' already on their own.
> 
> Of course, they could also move with the times, and adopt an XML
> format for command arguments which would be much more extensible and
> Web 2.0 compliant.
> 
> That way all commands could share the same dynamically linked parser,
> and one could write scripts with their favorite XML-editor!
> 
> uriel
> 

Breaking news: bash now uses mysql to store your history,  environmental
variables, et cetera.

Don't forget to tune in at 7 to discuss Emacs newest feature,a Virtual Machine
that runs in the editor.

-- 
Jake Todd
// If it isn't broke, tweak it!

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: [9fans] *Suspect* Re: mishandling empty lists - let's fix it

[W B Hacker]

Jacob Todd wrote:
> On Tue, Oct 06, 2009 at 08:59:32AM +0200, Uriel wrote:
>> I'm sure the people in charge of writing the next version of the PoSix
>> standard will be very happy to adopt your proposal!
>>
>> That is, if they have not independently 'discovered' this fantastic
>> solution to this horrible 'problem' already on their own.
>>
>> Of course, they could also move with the times, and adopt an XML
>> format for command arguments which would be much more extensible and
>> Web 2.0 compliant.
>>
>> That way all commands could share the same dynamically linked parser,
>> and one could write scripts with their favorite XML-editor!
>>
>> uriel
>>
>
> Breaking news: bash now uses mysql to store your history,  environmental
> variables, et cetera.
>
> Don't forget to tune in at 7 to discuss Emacs newest feature,a Virtual Machine
> that runs in the editor.
>

That second part ain't new.

But it's a virtual machine that mostly runs its *mouth*.

Bill