Documentation about Multios is misleading, and perhaps untrue

Documentation about Multios is misleading, and perhaps untrue

[Tom Boyd]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

In the section of the manual about Multios, the manual states:

If the user tries to open a file descriptor for reading more than once, the
shell opens the file descriptor as a pipe to a process that copies all the
specified inputs to its output in the order specified, similar to cat,
provided the MULTIOS option is set. Thus

sort <foo <fubar

or even

sort <f{oo,ubar}

is equivalent to ‘cat foo fubar | sort’.


This last statement seems to be untrue. The apparent behavior I see in zsh
is that for commands of the form:

commandword < file1 < file2

file1 and file2 must be successfully opened by the shell *before* the
command is even executed. if the opening of any of the listed input files
fails or blocks, then execution of the command also fails or blocks until
all openings are unblocked.

Conversely, the command:

cat file1 file2

causes the shell to immediately execute the cat command, which then
*itself* the assumes responsibility of opening the files listed as
arguments, in order. cat has different semantics than multios: cat
*immediately* begins processing the files listed as arguments from left to
right, stopping only when it reaches the first one that blocks and resuming
when it finishes open. Also if cat encounters a file that fails to open, it
merely skips it and continues down the argument list.

myprocess < file1 < namedpipe &

will block entirely until namedpipe is opened on the sender side and only
then will it start proccessing  file1.

Conversely

cat file1 namedpipe | myprocess

will immediately start processing file1, block and wait for namedpipe to
open on the sender side, and then continue processing its fed data.

What should be done about this ?
-

tom

Re: Documentation about Multios is misleading, and perhaps untrue

[dana]

On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote:
>What should be done about this ?

The patch below documents the error for <file, which is consistent with the
documentation for the other redirection operators.

Not sure if there's really an issue with the rest; i've never assumed from that
description that it means it works *exactly* like cat, just that the effect is
similar in the general case. But i guess the examples could be qualified as
'roughly' equivalent or whatever, idk

dana


diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo
index c793638b7..833241574 100644
--- a/Doc/Zsh/redirect.yo
+++ b/Doc/Zsh/redirect.yo
@@ -26,6 +26,7 @@ separate filename in turn.
 startitem()
 item(tt(<) var(word))(
 Open file var(word) for reading as standard input.
+It is an error to open a file in this fashion if it does not exist.
 )
 item(tt(<>) var(word))(
 Open file var(word) for reading and writing as standard input.

Re: Documentation about Multios is misleading, and perhaps untrue

[Bart Schaefer]

On Wed, Oct 10, 2018 at 9:05 PM dana <dana@dana.is> wrote:
>
> On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote:
> >What should be done about this ?
>
> Not sure if there's really an issue with the rest

A bit of common sense has to be applied here.  These are redirection
operators and are going to behave first like redirection operators,
which means that the shell is going to open the file descriptors
before executing any of the commands, and then pass those descriptors
around.  The semantics of redirections demands this.  An external
command like "cat" receives a list of names in its argument list and
processes the names one by one, so it can completely open and close
the file with each name before moving on to the next one, but the
shell can't do that and would be broken in other cases if it tried.

It's pointless to try to call call out every possible instance where
the fundamental semantics of shell operations affect a particular use
of the syntax.

Re: Documentation about Multios is misleading, and perhaps untrue

[Daniel Shahaf]

Bart Schaefer wrote on Thu, 11 Oct 2018 12:12 -0700:
> On Wed, Oct 10, 2018 at 9:05 PM dana <dana@dana.is> wrote:
> >
> > On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote:
> > >What should be done about this ?
> >
> > Not sure if there's really an issue with the rest
> 
> A bit of common sense has to be applied here.  These are redirection
> operators and are going to behave first like redirection operators,
> which means that the shell is going to open the file descriptors
> before executing any of the commands, and then pass those descriptors
> around.  The semantics of redirections demands this.  An external
> command like "cat" receives a list of names in its argument list and
> processes the names one by one, so it can completely open and close
> the file with each name before moving on to the next one, but the
> shell can't do that and would be broken in other cases if it tried.
> 
> It's pointless to try to call call out every possible instance where
> the fundamental semantics of shell operations affect a particular use
> of the syntax.

I won't disagree with such a broad statement, but on the other hand, I
don't think "We shouldn't document this because this is how the OS
limitations force us to implement the shell" is a useful stance, either.
Readers of zshall(1) should not be assumed to be familiar with the C
syscalls and library interfaces' limitations.

Re: Documentation about Multios is misleading, and perhaps untrue

[Bart Schaefer]

On Thu, Oct 11, 2018 at 1:35 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Bart Schaefer wrote on Thu, 11 Oct 2018 12:12 -0700:
> >
> > It's pointless to try to call call out every possible instance where
> > the fundamental semantics of shell operations affect a particular use
> > of the syntax.
>
> I won't disagree with such a broad statement, but on the other hand, I
> don't think "We shouldn't document this because this is how the OS
> limitations force us to implement the shell" is a useful stance, either.

I won't disagree with your statement, either, but I don't think it
actually applies here.  This doesn't have anything to do with OS
limitations, it's just the way shells work.  It's the way redirection
works in ALL shells.  This is application documentation, not a
UNIX/Linux/HURD/Darwin tutorial, and I think it's a waste of our time
to attempt to be the latter.

Furthermore, I think this IS documented, and was even before
workers/43672, if one considers the examples in context rather than
taking them literally.

Re: Documentation about Multios is misleading, and perhaps untrue

[Peter Stephenson]

I don't see why we shouldn't at least be a bit more careful...

pws

diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo
index c793638..fc52c77 100644
--- a/Doc/Zsh/redirect.yo
+++ b/Doc/Zsh/redirect.yo
@@ -268,9 +268,10 @@ example(echo exit 0 >> *.sh)
 
 If the user tries to open a file descriptor for reading more than once,
 the shell opens the file descriptor as a pipe to a process that copies
-all the specified inputs to its output in the order
-specified, similar to bf(cat),
-provided the tt(MULTIOS) option is set.  Thus
+all the specified inputs to its output in the order specified, provided
+the tt(MULTIOS) option is set.  This is roughly similar to bf(cat) but
+note that shell redirection syntax implies differences in detailed
+behaviour.  Thus
 
 example(sort <foo <fubar)
 

Re: Documentation about Multios is misleading, and perhaps untrue

[Tom Boyd]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

 I think this section should just breifly describe that the file
descriptors are opened before the command is executed, and that

command < file1 < file2

is *similar* to

cat file1 file2 | command

except that the first waits for all files to be opened *before* command
executes and completely fails if any file opening fails, where as cat x y z
| command immediately starts processing files and simply skips ones that
fail.

On a more general philosophical note though, I get that context is
important, but man pages and software references should **never** contain
objectively false statements. It's not ok to say something that factually
incorrect and justify it by assuming the reader will have enough "common
sense" to determine what parts are correct and which arent. These
references are the single souce of truth for a lot of readers.



On Fri, Oct 12, 2018 at 4:31 AM Peter Stephenson <p.stephenson@samsung.com>
wrote:

> I don't see why we shouldn't at least be a bit more careful...
>
> pws
>
> diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo
> index c793638..fc52c77 100644
> --- a/Doc/Zsh/redirect.yo
> +++ b/Doc/Zsh/redirect.yo
> @@ -268,9 +268,10 @@ example(echo exit 0 >> *.sh)
>
>  If the user tries to open a file descriptor for reading more than once,
>  the shell opens the file descriptor as a pipe to a process that copies
> -all the specified inputs to its output in the order
> -specified, similar to bf(cat),
> -provided the tt(MULTIOS) option is set.  Thus
> +all the specified inputs to its output in the order specified, provided
> +the tt(MULTIOS) option is set.  This is roughly similar to bf(cat) but
> +note that shell redirection syntax implies differences in detailed
> +behaviour.  Thus
>
>  example(sort <foo <fubar)
>
>
>

-- 
Thomas Boyd

Re: Documentation about Multios is misleading, and perhaps untrue

[Daniel Shahaf]

Tom Boyd wrote on Mon, 15 Oct 2018 20:45 -0400:
>  I think this section should just breifly describe that the file
> descriptors are opened before the command is executed, and that
> 
> command < file1 < file2
> 
> is *similar* to
> 
> cat file1 file2 | command
> 
> except that the first waits for all files to be opened *before* command
> executes and completely fails if any file opening fails, where as cat x y z
> | command immediately starts processing files and simply skips ones that
> fail.

It's really not zsh's job to document cat(1).

However, I agree with your overall point, that the statement about
"equivalence" in the manual is incorrect.  At the same time, I think a
different fix would be better:

1. Ensure that it's documented that all redirections (input and output)
   are opened before the command is even exec'd.  This is true for all
   redirections, not just for MULTIOS syntaxes.

2. In the section that gives analogies to cat(1) and sort(1), simply
   state that the examples assume that all dirent names are ordinary,
   readable files.

Makes sense?  Anybody volunteering to write the patch (not me)?

> On a more general philosophical note though, I get that context is
> important, but man pages and software references should **never** contain
> objectively false statements. It's not ok to say something that factually
> incorrect and justify it by assuming the reader will have enough "common
> sense" to determine what parts are correct and which arent. These
> references are the single souce of truth for a lot of readers.

Cheers,

Daniel

Re: Documentation about Multios is misleading, and perhaps untrue

[Peter Stephenson]

On Tue, 2018-10-16 at 02:07 +0000, Daniel Shahaf wrote:
> However, I agree with your overall point, that the statement about
> "equivalence" in the manual is incorrect.  At the same time, I think a
> different fix would be better:
> 
> 1. Ensure that it's documented that all redirections (input and output)
>    are opened before the command is even exec'd.  This is true for all
>    redirections, not just for MULTIOS syntaxes.
> 
> 2. In the section that gives analogies to cat(1) and sort(1), simply
>    state that the examples assume that all dirent names are ordinary,
>    readable files.
> 
> Makes sense?  Anybody volunteering to write the patch (not me)?

Not really.  This is the same point as Bart's --- that this is
documentation for zsh, not a primer on how shells work.
I won't labour the point, but this isn't a good place to document how a
shell normally does redirection.

Where I think there is a point here is for multios, where there is an
extra process hidden away to concatenate the files (and it's a forked
shell, not a "cat").  The fact that *that* process opens files
immediately isn't obvious and isn't standard.  So this could probably do
with a comment somewhere.  I've added this below (this updates my
previous comparison with cat).

Qualifying the documentation to say if this isn't an ordinary file or
isn't readable it might not behave in an ordinary way or you might not
be able to read it is also verbiage that's trying to cover for ignorance
beyond what the shell manual can deal with.  We could just as well
document any example in the manual that uses files to say you need to be
able to read them or you'll get an error, or that if it's a special file
it might be special.  Our job is instead to document, for example, the
way the nature of an error might differ from what a shell user might
expect.

One of our big problems is people finding the existing documentation too
thick to get through.  We can go on making it wordier and harder to get
through till the cows come home and it still won't make it a basic shell
primer.

What's needed here is not zsh documentation, it's something properly
written in its own right that introduces what a redirection (and,
indeed, everything else) is.  This would give proper examples and
context and comparisons and further reading and stuff you just don't get
in user manuals for tools, any tool.  Such things no doubt exist and it
would be better to refer to here rather than do a half-baked job
ourselves.

I have to say, even there, that if you look at my user guide (now rather
old, but basic syntax hasn't changed):

http://zsh.sourceforge.net/Guide/zshguide03.html#l60

which is very much more verbose, I think I assume even there that it's
obvious that in general (and ignoring the special nature of multios) a
redirection happens before the command is run.  But that's the sort of
place to mention this sort of thing.

pws

diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo
index c793638..2c406b6 100644
--- a/Doc/Zsh/redirect.yo
+++ b/Doc/Zsh/redirect.yo
@@ -234,6 +234,9 @@ example(date >foo | cat)
 
 writes the date to the file `tt(foo)', and also pipes it to cat.
 
+Note that the shell opens all the files to be used in the multio process
+immediately, not at the point they are about to be written.
+
 Note also that redirections are always expanded in order.  This happens
 regardless of the setting of the tt(MULTIOS) option, but with the option
 in effect there are additional consequences. For example,
@@ -268,9 +271,13 @@ example(echo exit 0 >> *.sh)
 
 If the user tries to open a file descriptor for reading more than once,
 the shell opens the file descriptor as a pipe to a process that copies
-all the specified inputs to its output in the order
-specified, similar to bf(cat),
-provided the tt(MULTIOS) option is set.  Thus
+all the specified inputs to its output in the order specified, provided
+the tt(MULTIOS) option is set.  It should be noted that each file is
+opened immediately, not at the point where it is about to be read:
+this behaviour differs from tt(cat), so if strictly standard behaviour
+is needed, tt(cat) should be used instead.
+
+Thus
 
 example(sort <foo <fubar)