* Documentation about Multios is misleading, and perhaps untrue @ 2018-10-11 2:33 Tom Boyd 2018-10-11 4:05 ` dana 0 siblings, 1 reply; 9+ messages in thread From: Tom Boyd @ 2018-10-11 2:33 UTC (permalink / raw) To: zsh-workers [-- Attachment #1: Type: text/plain, Size: 1761 bytes --] In the section of the manual about Multios, the manual states: If the user tries to open a file descriptor for reading more than once, the shell opens the file descriptor as a pipe to a process that copies all the specified inputs to its output in the order specified, similar to cat, provided the MULTIOS option is set. Thus sort <foo <fubar or even sort <f{oo,ubar} is equivalent to ‘cat foo fubar | sort’. This last statement seems to be untrue. The apparent behavior I see in zsh is that for commands of the form: commandword < file1 < file2 file1 and file2 must be successfully opened by the shell *before* the command is even executed. if the opening of any of the listed input files fails or blocks, then execution of the command also fails or blocks until all openings are unblocked. Conversely, the command: cat file1 file2 causes the shell to immediately execute the cat command, which then *itself* the assumes responsibility of opening the files listed as arguments, in order. cat has different semantics than multios: cat *immediately* begins processing the files listed as arguments from left to right, stopping only when it reaches the first one that blocks and resuming when it finishes open. Also if cat encounters a file that fails to open, it merely skips it and continues down the argument list. myprocess < file1 < namedpipe & will block entirely until namedpipe is opened on the sender side and only then will it start proccessing file1. Conversely cat file1 namedpipe | myprocess will immediately start processing file1, block and wait for namedpipe to open on the sender side, and then continue processing its fed data. What should be done about this ? - tom ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-11 2:33 Documentation about Multios is misleading, and perhaps untrue Tom Boyd @ 2018-10-11 4:05 ` dana 2018-10-11 19:12 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: dana @ 2018-10-11 4:05 UTC (permalink / raw) To: Tom Boyd; +Cc: zsh-workers On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote: >What should be done about this ? The patch below documents the error for <file, which is consistent with the documentation for the other redirection operators. Not sure if there's really an issue with the rest; i've never assumed from that description that it means it works *exactly* like cat, just that the effect is similar in the general case. But i guess the examples could be qualified as 'roughly' equivalent or whatever, idk dana diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo index c793638b7..833241574 100644 --- a/Doc/Zsh/redirect.yo +++ b/Doc/Zsh/redirect.yo @@ -26,6 +26,7 @@ separate filename in turn. startitem() item(tt(<) var(word))( Open file var(word) for reading as standard input. +It is an error to open a file in this fashion if it does not exist. ) item(tt(<>) var(word))( Open file var(word) for reading and writing as standard input. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-11 4:05 ` dana @ 2018-10-11 19:12 ` Bart Schaefer 2018-10-11 20:35 ` Daniel Shahaf 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2018-10-11 19:12 UTC (permalink / raw) To: zsh-workers; +Cc: tvboyd23 On Wed, Oct 10, 2018 at 9:05 PM dana <dana@dana.is> wrote: > > On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote: > >What should be done about this ? > > Not sure if there's really an issue with the rest A bit of common sense has to be applied here. These are redirection operators and are going to behave first like redirection operators, which means that the shell is going to open the file descriptors before executing any of the commands, and then pass those descriptors around. The semantics of redirections demands this. An external command like "cat" receives a list of names in its argument list and processes the names one by one, so it can completely open and close the file with each name before moving on to the next one, but the shell can't do that and would be broken in other cases if it tried. It's pointless to try to call call out every possible instance where the fundamental semantics of shell operations affect a particular use of the syntax. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-11 19:12 ` Bart Schaefer @ 2018-10-11 20:35 ` Daniel Shahaf 2018-10-11 20:59 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Daniel Shahaf @ 2018-10-11 20:35 UTC (permalink / raw) To: Bart Schaefer, zsh-workers; +Cc: tvboyd23 Bart Schaefer wrote on Thu, 11 Oct 2018 12:12 -0700: > On Wed, Oct 10, 2018 at 9:05 PM dana <dana@dana.is> wrote: > > > > On 10 Oct 2018, at 21:33, Tom Boyd <tvboyd23@gmail.com> wrote: > > >What should be done about this ? > > > > Not sure if there's really an issue with the rest > > A bit of common sense has to be applied here. These are redirection > operators and are going to behave first like redirection operators, > which means that the shell is going to open the file descriptors > before executing any of the commands, and then pass those descriptors > around. The semantics of redirections demands this. An external > command like "cat" receives a list of names in its argument list and > processes the names one by one, so it can completely open and close > the file with each name before moving on to the next one, but the > shell can't do that and would be broken in other cases if it tried. > > It's pointless to try to call call out every possible instance where > the fundamental semantics of shell operations affect a particular use > of the syntax. I won't disagree with such a broad statement, but on the other hand, I don't think "We shouldn't document this because this is how the OS limitations force us to implement the shell" is a useful stance, either. Readers of zshall(1) should not be assumed to be familiar with the C syscalls and library interfaces' limitations. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-11 20:35 ` Daniel Shahaf @ 2018-10-11 20:59 ` Bart Schaefer 2018-10-12 8:30 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2018-10-11 20:59 UTC (permalink / raw) To: zsh-workers; +Cc: tvboyd23 On Thu, Oct 11, 2018 at 1:35 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote: > > Bart Schaefer wrote on Thu, 11 Oct 2018 12:12 -0700: > > > > It's pointless to try to call call out every possible instance where > > the fundamental semantics of shell operations affect a particular use > > of the syntax. > > I won't disagree with such a broad statement, but on the other hand, I > don't think "We shouldn't document this because this is how the OS > limitations force us to implement the shell" is a useful stance, either. I won't disagree with your statement, either, but I don't think it actually applies here. This doesn't have anything to do with OS limitations, it's just the way shells work. It's the way redirection works in ALL shells. This is application documentation, not a UNIX/Linux/HURD/Darwin tutorial, and I think it's a waste of our time to attempt to be the latter. Furthermore, I think this IS documented, and was even before workers/43672, if one considers the examples in context rather than taking them literally. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-11 20:59 ` Bart Schaefer @ 2018-10-12 8:30 ` Peter Stephenson 2018-10-16 0:45 ` Tom Boyd 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2018-10-12 8:30 UTC (permalink / raw) To: zsh-workers I don't see why we shouldn't at least be a bit more careful... pws diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo index c793638..fc52c77 100644 --- a/Doc/Zsh/redirect.yo +++ b/Doc/Zsh/redirect.yo @@ -268,9 +268,10 @@ example(echo exit 0 >> *.sh) If the user tries to open a file descriptor for reading more than once, the shell opens the file descriptor as a pipe to a process that copies -all the specified inputs to its output in the order -specified, similar to bf(cat), -provided the tt(MULTIOS) option is set. Thus +all the specified inputs to its output in the order specified, provided +the tt(MULTIOS) option is set. This is roughly similar to bf(cat) but +note that shell redirection syntax implies differences in detailed +behaviour. Thus example(sort <foo <fubar) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-12 8:30 ` Peter Stephenson @ 2018-10-16 0:45 ` Tom Boyd 2018-10-16 2:07 ` Daniel Shahaf 0 siblings, 1 reply; 9+ messages in thread From: Tom Boyd @ 2018-10-16 0:45 UTC (permalink / raw) To: p.stephenson; +Cc: zsh-workers [-- Attachment #1: Type: text/plain, Size: 1785 bytes --] I think this section should just breifly describe that the file descriptors are opened before the command is executed, and that command < file1 < file2 is *similar* to cat file1 file2 | command except that the first waits for all files to be opened *before* command executes and completely fails if any file opening fails, where as cat x y z | command immediately starts processing files and simply skips ones that fail. On a more general philosophical note though, I get that context is important, but man pages and software references should **never** contain objectively false statements. It's not ok to say something that factually incorrect and justify it by assuming the reader will have enough "common sense" to determine what parts are correct and which arent. These references are the single souce of truth for a lot of readers. On Fri, Oct 12, 2018 at 4:31 AM Peter Stephenson <p.stephenson@samsung.com> wrote: > I don't see why we shouldn't at least be a bit more careful... > > pws > > diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo > index c793638..fc52c77 100644 > --- a/Doc/Zsh/redirect.yo > +++ b/Doc/Zsh/redirect.yo > @@ -268,9 +268,10 @@ example(echo exit 0 >> *.sh) > > If the user tries to open a file descriptor for reading more than once, > the shell opens the file descriptor as a pipe to a process that copies > -all the specified inputs to its output in the order > -specified, similar to bf(cat), > -provided the tt(MULTIOS) option is set. Thus > +all the specified inputs to its output in the order specified, provided > +the tt(MULTIOS) option is set. This is roughly similar to bf(cat) but > +note that shell redirection syntax implies differences in detailed > +behaviour. Thus > > example(sort <foo <fubar) > > > -- Thomas Boyd ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-16 0:45 ` Tom Boyd @ 2018-10-16 2:07 ` Daniel Shahaf 2018-10-16 9:33 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Daniel Shahaf @ 2018-10-16 2:07 UTC (permalink / raw) To: zsh-workers Tom Boyd wrote on Mon, 15 Oct 2018 20:45 -0400: > I think this section should just breifly describe that the file > descriptors are opened before the command is executed, and that > > command < file1 < file2 > > is *similar* to > > cat file1 file2 | command > > except that the first waits for all files to be opened *before* command > executes and completely fails if any file opening fails, where as cat x y z > | command immediately starts processing files and simply skips ones that > fail. It's really not zsh's job to document cat(1). However, I agree with your overall point, that the statement about "equivalence" in the manual is incorrect. At the same time, I think a different fix would be better: 1. Ensure that it's documented that all redirections (input and output) are opened before the command is even exec'd. This is true for all redirections, not just for MULTIOS syntaxes. 2. In the section that gives analogies to cat(1) and sort(1), simply state that the examples assume that all dirent names are ordinary, readable files. Makes sense? Anybody volunteering to write the patch (not me)? > On a more general philosophical note though, I get that context is > important, but man pages and software references should **never** contain > objectively false statements. It's not ok to say something that factually > incorrect and justify it by assuming the reader will have enough "common > sense" to determine what parts are correct and which arent. These > references are the single souce of truth for a lot of readers. Cheers, Daniel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Documentation about Multios is misleading, and perhaps untrue 2018-10-16 2:07 ` Daniel Shahaf @ 2018-10-16 9:33 ` Peter Stephenson 0 siblings, 0 replies; 9+ messages in thread From: Peter Stephenson @ 2018-10-16 9:33 UTC (permalink / raw) To: zsh-workers On Tue, 2018-10-16 at 02:07 +0000, Daniel Shahaf wrote: > However, I agree with your overall point, that the statement about > "equivalence" in the manual is incorrect. At the same time, I think a > different fix would be better: > > 1. Ensure that it's documented that all redirections (input and output) > are opened before the command is even exec'd. This is true for all > redirections, not just for MULTIOS syntaxes. > > 2. In the section that gives analogies to cat(1) and sort(1), simply > state that the examples assume that all dirent names are ordinary, > readable files. > > Makes sense? Anybody volunteering to write the patch (not me)? Not really. This is the same point as Bart's --- that this is documentation for zsh, not a primer on how shells work. I won't labour the point, but this isn't a good place to document how a shell normally does redirection. Where I think there is a point here is for multios, where there is an extra process hidden away to concatenate the files (and it's a forked shell, not a "cat"). The fact that *that* process opens files immediately isn't obvious and isn't standard. So this could probably do with a comment somewhere. I've added this below (this updates my previous comparison with cat). Qualifying the documentation to say if this isn't an ordinary file or isn't readable it might not behave in an ordinary way or you might not be able to read it is also verbiage that's trying to cover for ignorance beyond what the shell manual can deal with. We could just as well document any example in the manual that uses files to say you need to be able to read them or you'll get an error, or that if it's a special file it might be special. Our job is instead to document, for example, the way the nature of an error might differ from what a shell user might expect. One of our big problems is people finding the existing documentation too thick to get through. We can go on making it wordier and harder to get through till the cows come home and it still won't make it a basic shell primer. What's needed here is not zsh documentation, it's something properly written in its own right that introduces what a redirection (and, indeed, everything else) is. This would give proper examples and context and comparisons and further reading and stuff you just don't get in user manuals for tools, any tool. Such things no doubt exist and it would be better to refer to here rather than do a half-baked job ourselves. I have to say, even there, that if you look at my user guide (now rather old, but basic syntax hasn't changed): http://zsh.sourceforge.net/Guide/zshguide03.html#l60 which is very much more verbose, I think I assume even there that it's obvious that in general (and ignoring the special nature of multios) a redirection happens before the command is run. But that's the sort of place to mention this sort of thing. pws diff --git a/Doc/Zsh/redirect.yo b/Doc/Zsh/redirect.yo index c793638..2c406b6 100644 --- a/Doc/Zsh/redirect.yo +++ b/Doc/Zsh/redirect.yo @@ -234,6 +234,9 @@ example(date >foo | cat) writes the date to the file `tt(foo)', and also pipes it to cat. +Note that the shell opens all the files to be used in the multio process +immediately, not at the point they are about to be written. + Note also that redirections are always expanded in order. This happens regardless of the setting of the tt(MULTIOS) option, but with the option in effect there are additional consequences. For example, @@ -268,9 +271,13 @@ example(echo exit 0 >> *.sh) If the user tries to open a file descriptor for reading more than once, the shell opens the file descriptor as a pipe to a process that copies -all the specified inputs to its output in the order -specified, similar to bf(cat), -provided the tt(MULTIOS) option is set. Thus +all the specified inputs to its output in the order specified, provided +the tt(MULTIOS) option is set. It should be noted that each file is +opened immediately, not at the point where it is about to be read: +this behaviour differs from tt(cat), so if strictly standard behaviour +is needed, tt(cat) should be used instead. + +Thus example(sort <foo <fubar) ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-10-16 9:42 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-11 2:33 Documentation about Multios is misleading, and perhaps untrue Tom Boyd 2018-10-11 4:05 ` dana 2018-10-11 19:12 ` Bart Schaefer 2018-10-11 20:35 ` Daniel Shahaf 2018-10-11 20:59 ` Bart Schaefer 2018-10-12 8:30 ` Peter Stephenson 2018-10-16 0:45 ` Tom Boyd 2018-10-16 2:07 ` Daniel Shahaf 2018-10-16 9:33 ` Peter Stephenson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).