implementing a control for completing filenames with a defined list of tokens

implementing a control for completing filenames with a defined list of tokens

[Eric Smith]

Wise zsh'lers,

I would like to define a completion widget to complete filenames with a defined
list of tokens.  The widget would look up in a list the tokens and suggest
these (ideally in order) to the user.

As an example;

$ cat tokenfile
token1
token2
token3

User has the binding control-k to execute the completion function;

mv <somefile> <c-k>
Then the widget would supply

mv <somefile> token1
token1
token2
token3

How would I best implement this?

-- 
Eric Smith

Re: implementing a control for completing filenames with a defined list of tokens

[Bart Schaefer]

On Dec 2,  3:26pm, Eric Smith wrote:
} Subject: implementing a control for completing filenames with a defined li
}
} I would like to define a completion widget to complete filenames with
} a defined list of tokens. The widget would look up in a list the
} tokens and suggest these (ideally in order) to the user.

This one is about as straightforward as it's possible to get.

First you need a function that reads tokenfile and passes the lines
therein as arguments to the "compadd" builtin.

    _tokens() { compadd ${(f)"$(<tokenfile)"} }

Next you need a completion widget.  There's a ready-made function for
creating completion widgets, called _generic.

    zle -C token-completion complete-word _generic
    bindkey ^K token-completion

(^K is normally kill-line so you might want to pick another binding.)

Finally tell the completion system that when the token-completer is
invoked, it should use the _tokens function to supply the matches:

    zstyle ':completion:token-completion:*' completer _tokens

And you're done.  If you eventually want it to complete other things, you
can append more functions to the zstyle.

Another way to do this is to create a file in your $fpath having a name
starting with underscore, and begin that file with a "#compdef -k ..."
line.  That could be as simple as this:

---- 8< ---- snip ---- 8< ----
#compdef -k complete-word ^K

_tokens() { compadd ${(f)"$(<tokenfile)"} }
_generic _tokens
---- 8< ---- snip ---- 8< ----

However, that will redefine the _tokens function on each completion.  A
slightly better formulation would be to have the file redefine the same
function as its file name.  If the file is named "_token_completion":

---- 8< ---- snip ---- 8< ----
#compdef -k complete-word ^K

_tokens() { compadd ${(f)"$(<tokenfile)"} }
_token_completion() { _generic _tokens }
_token_completion
---- 8< ---- snip ---- 8< ----

It's probably not necessary to go this far when _tokens is so simple,
but if you eventually write something more involved it's a good pattern.

Note that with the file you don't need the zstyle: passing _tokens as an
argument to _generic has the equivalent effect.  You could omit _tokens
there and instead have the style, just replace "token-completion" in the
style pattern with the name of the file.

Re: implementing a control for completing filenames with a defined list of tokens

[Eric Smith]

Thank you Bart.

I want to use this completion facility at the start of file (a
dir) names and also *in the middle* of the name. 
So the tags may appear anywhere in the filename.
So I could type foobar__^K
for example and the function will complete with my tokens which all begin with a double underscore.

$ cat tokenfile 
__job__1403-whatever__
__article__
__finance__
__contract__
__published__
__project-foo__ 
__job__1402-visit__

How may I adapt your suggestions for this?

Thanks a lot.

Eric

Bart Schaefer wrote on Mon-02-Dec 13  4:58PM
> On Dec 2,  3:26pm, Eric Smith wrote:
> } Subject: implementing a control for completing filenames with a defined li
> }
> } I would like to define a completion widget to complete filenames with
> } a defined list of tokens. The widget would look up in a list the
> } tokens and suggest these (ideally in order) to the user.
> 
> This one is about as straightforward as it's possible to get.
> 
> First you need a function that reads tokenfile and passes the lines
> therein as arguments to the "compadd" builtin.
> 
>     _tokens() { compadd ${(f)"$(<tokenfile)"} }
> 
> Next you need a completion widget.  There's a ready-made function for
> creating completion widgets, called _generic.
> 
>     zle -C token-completion complete-word _generic
>     bindkey ^K token-completion
> 
> (^K is normally kill-line so you might want to pick another binding.)
> 
> Finally tell the completion system that when the token-completer is
> invoked, it should use the _tokens function to supply the matches:
> 
>     zstyle ':completion:token-completion:*' completer _tokens
> 
> And you're done.  If you eventually want it to complete other things, you
> can append more functions to the zstyle.
> 
> Another way to do this is to create a file in your $fpath having a name
> starting with underscore, and begin that file with a "#compdef -k ..."
> line.  That could be as simple as this:
> 
> ---- 8< ---- snip ---- 8< ----
> #compdef -k complete-word ^K
> 
> _tokens() { compadd ${(f)"$(<tokenfile)"} }
> _generic _tokens
> ---- 8< ---- snip ---- 8< ----
> 
> However, that will redefine the _tokens function on each completion.  A
> slightly better formulation would be to have the file redefine the same
> function as its file name.  If the file is named "_token_completion":
> 
> ---- 8< ---- snip ---- 8< ----
> #compdef -k complete-word ^K
> 
> _tokens() { compadd ${(f)"$(<tokenfile)"} }
> _token_completion() { _generic _tokens }
> _token_completion
> ---- 8< ---- snip ---- 8< ----
> 
> It's probably not necessary to go this far when _tokens is so simple,
> but if you eventually write something more involved it's a good pattern.
> 
> Note that with the file you don't need the zstyle: passing _tokens as an
> argument to _generic has the equivalent effect.  You could omit _tokens
> there and instead have the style, just replace "token-completion" in the
> style pattern with the name of the file.

-- 
Eric Smith

Re: implementing a control for completing filenames with a defined list of tokens

[Bart Schaefer]

On Mar 16,  3:13pm, Eric Smith wrote:
}
} I want to use this completion facility at the start of file (a
} dir) names and also *in the middle* of the name. 
} So the tags may appear anywhere in the filename.
} So I could type foobar__^K
} for example and the function will complete with my tokens which all
} begin with a double underscore.

Hmm, that's a somewhat different question, and the answer depends on
whether you want to complete existing directory names (limit the result
to those that match your tokens) or whether you are trying to construct
new names from the input line plus the tokens.

Limiting to existing names is quite a bit easier; instead of defining a
completer for the token-completion generic widget, we just define a
file-patterns style:

    zle -C token-completion complete-word _generic
    bindkey ^K token-completion
    zstyle -e ':completion:token-completion:*' file-patterns \
      'reply=( "*(${(j:|:)${(qf)"$(<tokenfile)"}})*(/)" )'

I've used zstyle -e there so that tokenfile is re-read every time the
patterns are needed, so you can update the tokens without having to
reload the completion.  Replace tokenfile with a full path unless you
want the tokens to be relative to the current directory.

What happens there is that the pattern is used to create a list of all
possible directories (because of the trailing "(/)" qualifier) that
match your tokens, and then the completion system will automatically
narrow that list to the ones that also match the current input line.

To build up a word for a file that doesn't exist, you need a completer
as in the original scheme, but you'll have to paste the prefix/suffix
from the command line around the tokens yourself.  Something like:

    _tokens() {
      if [[ $PREFIX$SUFFIX = *__* ]]; then
        compadd ${PREFIX%_#}${^${(f)"$(<tokenfile)"}}${SUFFIX#_#}
      else
        return 1
      fi
    }

where ${PREFIX%_#} means the prefix with any trailing underscores
removed, and similarly ${SUFFIX#_#} removes leading underscores.

And then you need "setopt complete_in_word" to have this in the middle
of the word.

If you want to combine both schemes, e.g., complete words that match a
file name but offer words that don't if there are no such files, you
need something like

    zle -C token-completion complete-word _generic
    bindkey ^K token-completion
    zstyle -e ':completion:token-completion:*' file-patterns \
      'reply=( "*(${(j:|:)${(qf)"$(<tokenfile)"}})*(/)" )'
    zstyle ':completion:token-completion:*' completer _files _tokens

(with _tokens as above).

Re: implementing a control for completing filenames with a defined list of tokens

[Bart Schaefer]

On Mar 16, 12:27pm, Bart Schaefer wrote:
}
}     _tokens() {
}       if [[ $PREFIX$SUFFIX = *__* ]]; then
}         compadd ${PREFIX%_#}${^${(f)"$(<tokenfile)"}}${SUFFIX#_#}

Apologies, that needs to be ${PREFIX%%_#} and ${SUFFIX##_#} to get all
the underscores.

set -F kills read -t

[Ray Andrews]

All:

I'm trying to get some functions to accept piped input:

func ()
{
...
read -t input
echo "$input to a summer's day?"
...
}

$ echo "Shall I compare thee" | func

Shall I compare thee to a summer's day?

...
All good. However if 'set -F' is active before 'func' is called  or even 
it it's set just before the 'read -t input'
then nothing is readed. Why is that? I thought 'set -F' was just to 
prevent glob expansions. Can it be fixed?

Tx.

Re: set -F kills read -t

[Bart Schaefer]

On Mar 17,  8:10pm, Ray Andrews wrote:
} Subject: set -F kills read -t
}
} func ()
} {
} ...
} read -t input
} echo "$input to a summer's day?"
} ...
} }
} 
} $ echo "Shall I compare thee" | func
} 
} Shall I compare thee to a summer's day?
} 
} ...
} All good. However if 'set -F' is active before 'func' is called  or even 
} it it's set just before the 'read -t input'
} then nothing is readed. Why is that? I thought 'set -F' was just to 
} prevent glob expansions. Can it be fixed?

I'm not able to reproduce this:

zsh -f
torch% func() { set -F; read -t input; print "$input to a summer's day?" }
torch% echo "Shall I compare thee" | func
Shall I compare thee to a summer's day?
torch% 

Also the stuff that you've elided with "..." that comes before "read"
may be important.  When you pipe to a function, the default standard
input of every command in that function is the same as the input of
the function itself, so something else could be consuming the input
before "read" gets a shot at it.

Further, "read -t" means to fail immediately if input is not ready
when "read" begins executing.  Because zsh forks to the left, there
is an inherent race condition in having "read -t" on the right side
of a pipeline; the "echo" in the forked subshell may not yet have
had a chance to do anything by the time that the "read" in the parent
shell examines standard input.

Try examining $? after "read -t input" finishes.  If it's 1, then the
read timed out.

If you change to "read -t 1 input" you may find the problem disappears.

Re: set -F kills read -t

[Ray Andrews]

On 03/17/2014 11:50 PM, Bart Schaefer wrote:


Bart:

Confusions within Confuzzlements.
> I'm not able to reproduce this:
>
> zsh -f
> torch% func() { set -F; read -t input; print "$input to a summer's day?" }
> torch% echo "Shall I compare thee" | func
> Shall I compare thee to a summer's day?
> torch%
Yeah, that works, but your further comments expose what seems to me to 
be a bug.
> Further, "read -t" means to fail immediately if input is not ready
> when "read" begins executing.  Because zsh forks to the left, there
> is an inherent race condition in having "read -t" on the right side
> of a pipeline; the "echo" in the forked subshell may not yet have
> had a chance to do anything by the time that the "read" in the parent
> shell examines standard input.
>
> Try examining $? after "read -t input" finishes.  If it's 1, then the
> read timed out.
>
> If you change to "read -t 1 input" you may find the problem disappears.

Your code:

   func0() { set -F; read -t input; print "$input to a summer's day?" }

And this run:

   $ s="lowercase s"; S=UPPERCASE S"

   $ echo "$s $S"
   lowercase s UPPERCASE S

   $ echo $S | func0
   UPPERCASE S to a summer's day?

   $ echo $s | func0
   lowercase s to a summer's day?

   $ echo $S | func0
   lowercase s to a summer's day?   << WRONG!

   $ echo "$s $S"
   lowercase s UPPERCASE S

   $ echo $S | func0
   UPPERCASE S to a summer's day?  << THAT'S BETTER

... How can such a thing ever be permitted? That's just plain broken.
But, from your comments I tried this: (it seems the 'set -F' thing is a 
red herring)

   func1() { read -t 1 input; print "$input to a summer's day?" }

... And the same run of tests is fine.  I have no idea how this 'race 
condition' stuff
works but surely, whatever "read -t 1" has that "read -t" lacks should 
be automatic?
When/where would my first run above ever be acceptable? It should print 
the correct
variable, or maybe it should fail completely, but printing the wrong 
variable is just a felony, no?
Nothing is more important than predictability. In a pipe situation, 
'read' shouldn't leave
the station until all the passengers are on board, but even if it does, 
it shouldn't give my
seat to someone else and then call them me.  Or so it looks to this 
grasshopper.

Re: set -F kills read -t

[Peter Stephenson]

On Tue, 18 Mar 2014 09:22:22 -0700
Ray Andrews <rayandrews@eastlink.ca> wrote:
> I have no idea how this 'race 
> condition' stuff
> works but surely, whatever "read -t 1" has that "read -t" lacks should 
> be automatic?

No, that's the whole point of the difference!

"read -t" says "test the input state *right now*.  Trust me, I really do
mean 'right now'.  I am over 18".

"read -t 1" says "wait at most second to see if some input turns up".

When you start a pipeline,

  do-stuff-with-some-output | do-stuff-with-some-input

it's inevitable that what happens on the left and the right is going to
take some time.  The two processes are asynchronous by design.  The only
thing the shell guarantees you is that the pipeline itself will be set
up (there is some internal synchronisation to ensure there's something
running on the left and something running on the right).  When input and
output are taking place on the processes on the left and right are down
to details of those processes that the shell framework for pipelines
simply can't worry about.

So to ensure synchronisation of input and output you need to do extra
work.  The "wait a second" option is the easiest way of doing that,
although it's not a guarantee. However, there may be better ways, e.g.
you can also tell it "read 1 character, waiting until that arrives".  It
depends what you're ultimate object is.

Welcome to "non-blocking I/O"; you've now been through the first
lesson, "don't use non-blocking I/O unless you need to".

pws

Re: set -F kills read -t

[Bart Schaefer]

To add to what PWS just said ...

On Mar 18,  9:22am, Ray Andrews wrote:
}
} ... And the same run of tests is fine. I have no idea how this 'race
} condition' stuff works

"Race condition" is a term referring to what happens when two (or more)
tasks both wish to use the same resource at the same time, but the
resource is only available to one task at a time.  Think of restroom
stalls at a sports stadium at half time; someone ends up waiting, but
you can't tell in advance who will get there first.

In your case the writer is a janitor arriving to refill the paper towel
dispenser, and the reader has just used the sink and need to dry his
hands.  The -t option tells the reader that if the towel dispenser is
empty, he should immediately go away with his hands still wet.  If you
instead use -t 1, he waits 1 second to give the janitor a chance to do
his job.

}    $ echo $s | func0
}    lowercase s to a summer's day?
} 
}    $ echo $S | func0
}    lowercase s to a summer's day?   << WRONG!

The "problem" here is that $input is a global variable, so it remained
set across runs of func0.  In the second case "read -t" failed (if we
had bothered to test $? it would have been 1) so nothing new was put
into $input and the old value persisted.

You could fix this by declaring "local input;" which would cause input
to be reset every time the function is called, but that depends on
whether you want to be able to refer to $input after the function is
finished.

(One could argue that "read" should always erase the parameter to which
it was told to write, no matter whether the action of reading succeeds;
but that's a different conversation.)

} Nothing is more important than predictability.

Not always true.  The point of the -t option is to tell "read" that it
is in fact more important not to wait than it is to be predictable.

I suspect that what you really want is the answer to the question "is
my standard input a pipe?" and to go do something else if it is not.
Using "read -t" gives you an unpredictable answer to that question.

Without more context of what your function is meant to do when the
input is NOT a pipe, we can't tell you the best way to answer that
question, or even whether it's the right question in the first place.

-- 
Barton E. Schaefer

Re: set -F kills read -t

[Ray Andrews]

On 03/18/2014 10:45 AM, Bart Schaefer wrote:


Peter, Bart:

Thanks, now I at least know what was busted.  I must tread lightly on 
this point because zsh
has it's own culture, but from the perspective of my C brain, " read -t 
"  ... maybe this, maybe
that ... with exactly the same input is hard to accept.  It's not very 
robust.

> (One could argue that "read" should always erase the parameter to which
> it was told to write, no matter whether the action of reading succeeds;
> but that's a different conversation.)
I'd say it's almost the nub of this conversation.  If, as Peter says, 
zsh is asynchronous,
and that means that process one might or might not be finished before 
process two, then it
seems to me that if there is a failure of some sort, then that should be 
manifested.  Identical runs
of identical code should produce identical results, no? Or at least warn 
you if that isn't going to
happen. I appeal to the doctrine of least surprise. It should null the 
variable first, then a twit like
me would at least have some warning that something is amiss. Or it could 
print " (null) " or
something else helpful.
> } Nothing is more important than predictability.
>
> Not always true.  The point of the -t option is to tell "read" that it
> is in fact more important not to wait than it is to be predictable.
Ok, but in the context of a pipe can't we have 'wait for the input that 
IS coming. Don't wait
one second or ten seconds or no seconds, wait until the input arrives. 
Wait until the first 'echo' has done its
thing. Ain't that intuitive? When would one ever want 'read -t' to maybe 
capture input or
maybe not, depending on something unpredictable?

echo "a string" | func

should send "a string" to func absolutely every time. The very existence 
of the pipe
symbol should say 'wait for it'. Wait for 'echo' to return.
> I suspect that what you really want is the answer to the question "is
> my standard input a pipe?" and to go do something else if it is not.
> Using "read -t" gives you an unpredictable answer to that question.
>
> Without more context of what your function is meant to do when the
> input is NOT a pipe, we can't tell you the best way to answer that
> question, or even whether it's the right question in the first place.
It's just a wrapper function. In this test case, around 'grep'. I use my 
wrapper directly
but I thought it may as well be able to accept input from a pipe too. 
But this is a
matter of principal. Asynchronous piping seems almost a contradiction. 
Surely each
stage of a chain of pipes has a right to expect linear travel of data.  
This problem seems
never to happen with piped binaries, they don't seem to need to wait 
some arbitrary
number of seconds for input, nor should it happen with functions. Or 
maybe that just
can't  be done, I don't know.

Anyway, in practical terms 'read -t 1' does the trick. Most of the time. 
Not on Tuesdays
nor when I have a process running in the background that slows things 
down, but mostly
it works ;-) Not meaning to rock the boat of course, zsh does what it does.

Re: set -F kills read -t

[Jan Larres]

On 19/03/14 11:08, Ray Andrews wrote:
> On 03/18/2014 10:45 AM, Bart Schaefer wrote:
>> Not always true.  The point of the -t option is to tell "read" that it
>> is in fact more important not to wait than it is to be predictable.
>
> Ok, but in the context of a pipe can't we have 'wait for the input
> that IS coming. Don't wait one second or ten seconds or no seconds,
> wait until the input arrives. Wait until the first 'echo' has done its
> thing. Ain't that intuitive? When would one ever want 'read -t' to
> maybe capture input or maybe not, depending on something
> unpredictable?
>
> echo "a string" | func
>
> should send "a string" to func absolutely every time. The very
> existence of the pipe symbol should say 'wait for it'. Wait for 'echo'
> to return.

As Peter said this is just normal non-blocking I/O, which is not at all
shell-specific let alone zsh-specific. Since you clearly seem to want
blocking I/O, why are you using the '-t' argument to begin with? If you
just wrote 'read input' it should do exactly what you want.

-Jan

Re: set -F kills read -t

[Ray Andrews]

On 03/18/2014 04:12 PM, Jan Larres wrote:
> On 19/03/14 11:08, Ray Andrews wrote:
> As Peter said this is just normal non-blocking I/O, which is not at 
> all shell-specific let alone zsh-specific. Since you clearly seem to 
> want blocking I/O, why are you using the '-t' argument to begin with? 
> If you just wrote 'read input' it should do exactly what you want. -Jan 
Only because sometimes the input to the function is via arguments, not 
via pipe. I'm trying to emulate what (say) grep does:

ls *.txt | grep some_file

vs.

grep "grep" *

... and only the '-t' switch seems to work with both forms. Really 'read 
-t 1' seems perfectly functional,
I just wonder about the arbitrariness of a time limit vs. a more robust 
situation, since in a pipe it seems
to me that there is no doubt that input is expected. One should be able 
to insure that the flow of the
code is what one would expect without a 'dumb' pause, I'd say.

Re: set -F kills read -t

[Jan Larres]

On 19/03/14 17:06, Ray Andrews wrote:
> On 03/18/2014 04:12 PM, Jan Larres wrote:
>> On 19/03/14 11:08, Ray Andrews wrote:
>> As Peter said this is just normal non-blocking I/O, which is not at
>> all shell-specific let alone zsh-specific. Since you clearly seem to
>> want blocking I/O, why are you using the '-t' argument to begin with?
>> If you just wrote 'read input' it should do exactly what you want.
>
> Only because sometimes the input to the function is via arguments, not
> via pipe. I'm trying to emulate what (say) grep does:
>
> ls *.txt | grep some_file
>
> vs.
>
> grep "grep" *

The best way to do that, and presumably the way grep itself does it, is
to test whether stdin is connected to a terminal:

mygrep() {
    if [ -t 0 ]; then
        echo terminal
    else
        read input
        echo $input input
    fi
}

$ mygrep
terminal
$ echo foo | mygrep
foo input

-Jan

Re: set -F kills read -t

[Ray Andrews]

On 03/18/2014 10:30 PM, Jan Larres wrote:
> The best way to do that, and presumably the way grep itself does it, is
> to test whether stdin is connected to a terminal:
>
> mygrep() {
>      if [ -t 0 ]; then
>          echo terminal
>      else
>          read input
>          echo $input input
>      fi
> }
>
> $ mygrep
> terminal
> $ echo foo | mygrep
> foo input
>
Bloody marvellous! That scratches my itch just perfectly, thanks Jan.

I wish there was some web site: "101 things you still don't know you can 
do with
zsh, and why you want to know them".

Re: set -F kills read -t

[Bart Schaefer]

On Mar 19,  8:23am, Ray Andrews wrote:
} Subject: Re: set -F kills read -t
}
} On 03/18/2014 10:30 PM, Jan Larres wrote:
} > The best way to do that, and presumably the way grep itself does it, is

Grep doesn't care if the input is a terminal, it only cares whether it
has a file name in its argument list or not.

} I wish there was some web site: "101 things you still don't know you
} can do with zsh, and why you want to know them".

http://www.amazon.com/Bash-Shell-Conquering-Command-Line/dp/1590593766/ref=sr_1_1?s=books&ie=UTF8&qid=1395259194&sr=1-1&keywords=from+bash+to+z+shell

:-)

Re: set -F kills read -t

[Ray Andrews]

On 03/19/2014 01:00 PM, Bart Schaefer wrote:
> On Mar 19,  8:23am, Ray Andrews wrote:
> } Subject: Re: set -F kills read -t
> }
> } On 03/18/2014 10:30 PM, Jan Larres wrote:
> } > The best way to do that, and presumably the way grep itself does it, is
>
> Grep doesn't care if the input is a terminal, it only cares whether it
> has a file name in its argument list or not.
>
> } I wish there was some web site: "101 things you still don't know you
> } can do with zsh, and why you want to know them".
>
> http://www.amazon.com/Bash-Shell-Conquering-Command-Line/dp/1590593766/ref=sr_1_1?s=books&ie=UTF8&qid=1395259194&sr=1-1&keywords=from+bash+to+z+shell
>
> :-)
>
I'm on chapter two ;-)

Re: set -F kills read -t

[Bart Schaefer]

Pardon me while I provide further evidence in support of the theorem that
the best way to get the correct answer from the internet is to post the
wrong answer.

On Mar 18,  3:08pm, Ray Andrews wrote:
}
} Thanks, now I at least know what was busted. I must tread lightly on
} this point because zsh has it's own culture, but from the perspective
} of my C brain, " read -t " ... maybe this, maybe that ... with exactly
} the same input is hard to accept. It's not very robust.

"read -t input" in zsh is pretty nearly equivalent to this C code:

   char input[1024];
   fcntl(0, F_SETFL, O_NONBLOCK);
   read(0, input, 1024);

(except of course that the size of the input is not predetermined).  If
you cojole that C into a runnable program and try it, you'll find that
it behaves just the way "read -t" does.  If you don't want that fcntl()
in there, don't pass the -t option.
 
} > (One could argue that "read" should always erase the parameter to which
} > it was told to write, no matter whether the action of reading succeeds;
} > but that's a different conversation.)

} I'd say it's almost the nub of this conversation. If, as Peter says,
} zsh is asynchronous, and that means that process one might or might
} not be finished before process two, then it seems to me that if there
} is a failure of some sort, then that should be manifested.

It *IS* manifested ... as the return value of "read" ($?).  Which your
function ignored ...  The full situation goes something like this:

  input=START
  if read -t input
  then print "I definitely read input: $input"
  elif (( $#input ))
  then print "Error, input unchanged: $input"
  else print "End of file: $input"
  fi

In most cases you don't care:

  if read -t input
  then print "got \$input: $input"
  else print "got nothing, do not use \$input"
  fi

} Identical runs of identical code should produce identical results, no?

No.  Consider for example:

   for (( count=1; count <= 10000; count++ )) do
     if read -t something
     then print "GOT: $something"; break
     fi
     print $count
   done

This happily counts to 10000, stopping as soon as it gets input.  Exactly
how many numbers are printed before it stops depends on when the input
arrives.  Identical runs of identical code, but not identical results.
If the most important thing is watching it count, then this is exactly
what "read -t" is for.  If the important thing is "GOT: $something",
then don't use -t.

} Or at least warn you if that isn't going to happen. I appeal to the
} doctrine of least surprise.

I appeal to the documentation:

    -t [ NUM ]
          Test if input is available BEFORE ATTEMPTING TO READ.  ...
	  If no input is available, return status 1 AND DO NOT SET
	  ANY VARIABLES.

(emphasis mine, obviously).  What is it that made you believe you need
the -t option in the first place?

} Ok, but in the context of a pipe can't we have 'wait for the input
} that IS coming. Don't wait one second or ten seconds or no seconds,
} wait until the input arrives.

Well, yes.  That's what "read" *without* the -t option does.  That's why
I said:

} > I suspect that what you really want is the answer to the question "is
} > my standard input a pipe?" and to go do something else if it is not.

What is it about piped input that requires different behavior from your
function?

} echo "a string" | func
} 
} should send "a string" to func absolutely every time.

It does send it.  Whether "func" consumes that input is up to the code
inside of "func".

} The very existence of the pipe symbol should say 'wait for it'. Wait
} for 'echo' to return.

But that's not what the pipe symbol means.  It means only "connect the
standard output of (stuff on the left) to the standard input of (stuff
on the right)."

Consider what would happen if "echo" produced a gigabyte of output, or
a terabyte, or a petabyte.  Where is all of that supposed to go while
waiting for echo to return?  Do you always expect your web browser to
download an entire video before beginning to play it?

} It's just a wrapper function. In this test case, around 'grep'.

OK, but the wrapper must be doing *something*.  I mean, this is a
wrapper around grep:

  mygrep() {
    print -u2 "Hi, I'm going to grep now."
    grep "$@"
  }

This will perfectly happily read from a pipe, a file, or a file named
in the arguments, without needing "read" at all.  So what exactly is
your wrapper doing, such that it needs to consume standard input before
grep does?

} But this is a matter of principal. Asynchronous piping seems almost a
} contradiction.

Every shell since the 1970s has worked this way, so you may want to
reconsider which principle is involved. :-)

} Surely each stage of a chain of pipes has a right to expect linear
} travel of data.

If you think of a pipeline as establishing "direction", then yes, the
chain has the right to expect that the data will always flow in the
same direction and (if there is only a single producer) in a fixed
order; but it does not have the right to expect that it will always
start flowing at a particular time and flow at a fixed rate.

Re: set -F kills read -t

[Ray Andrews]

On 03/18/2014 06:17 PM, Bart Schaefer wrote:

Bart,
> Pardon me while I provide further evidence in support of the theorem that
> the best way to get the correct answer from the internet is to post the
> wrong answer.
Who's answer was wrong?
> (except of course that the size of the input is not predetermined).  If
> you cojole that C into a runnable program and try it, you'll find that
> it behaves just the way "read -t" does.  If you don't want that fcntl()
> in there, don't pass the -t option.
>   
Ok, then how else would one write a function that could use arguments 
*or* piped input? grep
does it, and doesn't need an arbitrary wait. Again, in *practice* the " 
-t 1 " seems perfectly good
enough, I don't want to belabour this, it's a theoretical/philosophical 
point only.
>
>    input=START
>    if read -t input
>    then print "I definitely read input: $input"
>    elif (( $#input ))
>    then print "Error, input unchanged: $input"
>    else print "End of file: $input"
>    fi
>
Ok, that  at least covers the bases if the read failed

> } Identical runs of identical code should produce identical results, no?
>
> No.  Consider for example:
>
>     for (( count=1; count <= 10000; count++ )) do
>       if read -t something
>       then print "GOT: $something"; break
>       fi
>       print $count
>     done
>
> This happily counts to 10000, stopping as soon as it gets input.  Exactly
> how many numbers are printed before it stops depends on when the input
> arrives.  Identical runs of identical code, but not identical results.
> If the most important thing is watching it count, then this is exactly
> what "read -t" is for.  If the important thing is "GOT: $something",
> then don't use -t.
Point made. That's well and good, I can see that " -t " would be exactly 
right in that situation, but in my
pipe situation, it's not exactly right.
> } Or at least warn you if that isn't going to happen. I appeal to the
> } doctrine of least surprise.
>
> I appeal to the documentation:
>
>      -t [ NUM ]
>            Test if input is available BEFORE ATTEMPTING TO READ.  ...
> 	  If no input is available, return status 1 AND DO NOT SET
> 	  ANY VARIABLES.
Yeah ... I guess in my still synchronous and 'blocking' mind, when 
there's a pipe there is
*always* input so always 'available'.  It will take some getting used to 
the idea that the
various steps in a long sequence of piped functions can quit any time 
they like in any
order that happens to happen.
>
> Consider what would happen if "echo" produced a gigabyte of output, or
> a terabyte, or a petabyte.  Where is all of that supposed to go while
> waiting for echo to return?  Do you always expect your web browser to
> download an entire video before beginning to play it?
A valid point. I have no issue that " -t [seconds to wait] " is 
available when needed, but in
the case of:

echo "a string" | func

I hardly think that func should not politely wait for "a string", and as 
my tests showed,
sometimes it didn't. I dunno, maybe there is no way that 'read' can be 
aware that
something to the left of it in a pipe situation has returned, so what 
I'm wanting might
be impossible. If it was possible it would be the '-p' switch: 'pipe 
mode' ... wait for piped
input to finish.
> } It's just a wrapper function. In this test case, around 'grep'.
>
> OK, but the wrapper must be doing *something*.  I mean, this is a
> wrapper around grep:
>
>    mygrep() {
>      print -u2 "Hi, I'm going to grep now."
>      grep "$@"
>    }
>
> This will perfectly happily read from a pipe, a file, or a file named
> in the arguments, without needing "read" at all.  So what exactly is
> your wrapper doing, such that it needs to consume standard input before
> grep does?
Yabut, in the pipe situation you don't supply a filespec ....

Shoot.  All this 'read' stuff has been a colossal mistake. Damn, 
everything I read on the
internet said 'read' was the way to go. HAAA, which get's back to your 
theorem! I see
now how I've been barking up the wrong tree. It never even occurred to 
me that " $@ "
would soak up piped input, I thought " $@ " stuff had to be arguments 
after the command <:-(

Sorry Bart, I'm a long study.

Re: set -F kills read -t

[Bart Schaefer]

On Mar 18, 10:00pm, Ray Andrews wrote:
}
} Yeah ... I guess in my still synchronous and 'blocking' mind, when 
} there's a pipe there is *always* input so always 'available'.

Consider this silly example:

    cat /dev/null | func

There will be no output from cat, so as seen by [the commands in the
definition of] func, there's a pipe, but there is no input.

} ... in the case of:
} 
} echo "a string" | func
} 
} I hardly think that func should not politely wait for "a string"

Shell functions are little more than names for the set of commands
inside them; it's nonsensical to say that "func should politely wait".
What will politely wait (or not) is the first command inside func
that attempts to read standard input.  If there is no such command,
nothing waits.

} >    mygrep() {
} >      print -u2 "Hi, I'm going to grep now."
} >      grep "$@"
} >    }
} 
} ... I see now how I've been barking up the wrong tree. It never even
} occurred to me that " $@ " would soak up piped input, I thought " $@ "
} stuff had to be arguments after the command <:-(

It's not actually the case that $@ "soaks up" anything.

   mygrep pattern file

expands to

  {
   print -u2 "Hi, I'm going to grep now."
   grep "pattern" "file"
  }

so grep reads from the file named "file", whereas

   mygrep pattern

expands to

  {
   print -u2 "Hi, I'm going to grep now."
   grep "pattern"
  }

and then grep, noting that it has only a pattern and no file name, reads
from standard input just like it always does.  If there's any magic here
at all, it's that the standard input of mygrep and the standard input of
grep are the same pipe.

Re: set -F kills read -t

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 3013 bytes --]

On 03/18/2014 11:37 PM, Bart Schaefer wrote:
>      cat /dev/null | func
>
> There will be no output from cat, so as seen by [the commands in the
> definition of] func, there's a pipe, but there is no input.
Yabut 'cat' still returns, so I'd use that as the terminator.

      echo "" | func

... would 'know' that echo is 'finished' and that's that. Can't one 
process know that
another process is finished? Anyway it doesn't matter, Jan's solution is 
crisp,
understandable, robust and has no ifs, buts or maybes. As you said, this 
asynchronous
stuff is bedrock to shell and it ain't going to change, I hafta get used 
to it.
>
> Shell functions are little more than names for the set of commands
> inside them; it's nonsensical to say that "func should politely wait".
> What will politely wait (or not) is the first command inside func
> that attempts to read standard input.  If there is no such command,
> nothing waits.
Of course. One can't presume that the function has some behaviours apart 
from
the behaviours of it's contents. But in this case we're referring to 
'read' and  I
had speculated that 'read' might be able to have the option of being 
asked to wait
for some previous command to finish. Come to think of it, what it would 
do is
take Jan's code and more or less bundle that into an option to 'read'. 
Nevermind:

     if [ ! -t 0 ]; then read input; fi

... isn't exactly hard to type. I yield the point.
>
> It's not actually the case that $@ "soaks up" anything.
>
> and then grep, noting that it has only a pattern and no file name, reads
> from standard input just like it always does.
I see. Thanks for pointing that out, I was about to replace ignorance 
with false understanding.
I thought I was expecting this to work:

      $ y () { echo "$1, $2, $3 "; }

      $ y one two three
     one, two, three

     $ echo "one two three" | y
     , ,                                                  << :-(


So grep is doing the 'switcheroo' here--going from reading arguments, to 
reading stdin, and
the pipe is the de facto 'stdin', and piped input does NOT become 
arguments :-)

Got it.

I'm doing this sort of thing now, and I'm as happy as a clam:

    function y ()
    {
         pipeinput='(nothing in the pipe)'
         terminalinput='(nothing from the terminal)'
         if [ ! -t 0 ];  then read pipeinput; fi
         if [ -n "$1" ]; then terminalinput="$@"; fi
         echo "$pipeinput $terminalinput"
    }

    $ y "from an antique land"
    (nothing in the pipe) from an antique land

    $ echo "I met a traveller" | y
    I met a traveller (nothing from the terminal)

    $ echo "I met a traveller" | y "from an antique land"
    I met a traveller from an antique land

    $ echo "I met a traveller" | y "from an antique land" | y "\nWho
    said: \"Two vast and trunkless legs of stone"
    I met a traveller from an antique land
    Who said: "Two vast and trunkless legs of stone


...

Who could ask for more transparent code? Thanks all for your patience.

Re: set -F kills read -t

[Roman Neuhauser]

# rayandrews@eastlink.ca / 2014-03-19 10:08:44 -0700:
> I thought I was expecting this to work:
> 
>       $ y () { echo "$1, $2, $3 "; }
> 
>       $ y one two three
>      one, two, three
> 
>      $ echo "one two three" | y
>      , ,                                                  << :-(

what would you expect to happen in

     $ echo "one two three" | y foo bar baz
 
> So grep is doing the 'switcheroo' here--going from reading arguments, to 
> reading stdin, and the pipe is the de facto 'stdin', and piped input
> does NOT become arguments :-)

  de facto?  stdin is stdin no matter what 'file' the descriptor names.

-- 
roman

Re: set -F kills read -t

[Bart Schaefer]

On Mar 19, 10:08am, Ray Andrews wrote:
} Subject: Re: set -F kills read -t
}
} 
} On 03/18/2014 11:37 PM, Bart Schaefer wrote:
} >      cat /dev/null | func
} >
} > There will be no output from cat, so as seen by [the commands in the
} > definition of] func, there's a pipe, but there is no input.
} 
} Yabut 'cat' still returns, so I'd use that as the terminator.

In the general case, though, you don't know *when* the thing on the left
will be done.  Yes, end-of-file can be detected on the pipe, but:

  { sleep $((RANDOM * 1000)); cat /dev/null } | func

The default behavior of just about everything *except* "read -t" will
be for func to wait forever to for the pipe to close.

I'm still curious what put you on to "-t" in the first place.

} But in this case we're referring to 'read' and I had speculated that
} 'read' might be able to have the option of being asked to wait for
} some previous command to finish.

Another thing you may be missing here is that "read" consumes ONE LINE
of text:  A string ending in "\n" (or end of file).  If you do not use
"-t", then it waits as long as it must in order to gobble up one line.
But it won't wait for a second line.

(Of course you can tell it that something other than a "\n" should be
used as a the line ending, in which case it may very well swallow
everything up to end-of-file on the pipe, but that requires even more
fooling around and -t has you quite well enough confused already.)

}     function y ()
}     {
}          pipeinput='(nothing in the pipe)'
}          terminalinput='(nothing from the terminal)'
}          if [ ! -t 0 ];  then read pipeinput; fi
}          if [ -n "$1" ]; then terminalinput="$@"; fi
}          echo "$pipeinput $terminalinput"
}     }

It's a little odd to call $@ the "terminal input" -- you can have stdin
come from a tty device the same as from any other file.  All that the
above has said is that you never want to read from a tty, but you're
willing to read exactly one line from anywhere else.  Consider:

 $ y 'I met a man with seven wives' <<<'As I was going to St Ives'

As long as your clams are happy, though ...

Re: set -F kills read -t

[Ray Andrews]

On 03/19/2014 03:21 PM, Bart Schaefer wrote:
> I'm still curious what put you on to "-t" in the first place.
I was just Googling for advice on how to get a function to accept piped 
input and 'read -t'
came up several times. Monkey see, monkey do.
> Another thing you may be missing here is that "read" consumes ONE LINE
> of text:  A string ending in "\n" (or end of file).  If you do not use
> "-t", then it waits as long as it must in order to gobble up one line.
> But it won't wait for a second line.
Yes, I know that. But it's a place to start.
> (Of course you can tell it that something other than a "\n" should be
> used as a the line ending, in which case it may very well swallow
> everything up to end-of-file on the pipe, but that requires even more
> fooling around and -t has you quite well enough confused already.)
>
> }     function y ()
> }     {
> }          pipeinput='(nothing in the pipe)'
> }          terminalinput='(nothing from the terminal)'
> }          if [ ! -t 0 ];  then read pipeinput; fi
> }          if [ -n "$1" ]; then terminalinput="$@"; fi
> }          echo "$pipeinput $terminalinput"
> }     }
>
> It's a little odd to call $@ the "terminal input" -- you can have stdin
> come from a tty device the same as from any other file.  All that the
> above has said is that you never want to read from a tty, but you're
> willing to read exactly one line from anywhere else.  Consider:
>
>   $ y 'I met a man with seven wives' <<<'As I was going to St Ives'
>
> As long as your clams are happy, though ...
Now don't go confusin' me again ;-)

Hmmm ... no, you're right, it should be able to accept any volume of 
input as a matter of principal.
Ok Bart, how would you write it? The above  is a bridgehead in as much 
as it accepts input from
either end, but  how could it be improved? I tried 'cat' with no luck. 
I've already done some chains of
pipes (using binaries) up to maybe half a dozen steps, and you just 
don't have to worry about time-outs,
or single lines or anything at all, it just works, even on Tuesday.

Re: set -F kills read -t

[Bart Schaefer]

On Mar 19,  6:46pm, Ray Andrews wrote:
}
} Ok Bart, how would you write it? The above  is a bridgehead in as much 
} as it accepts input from either end 

Command line arguments are a fundamentally different kind of "input".
Files, pipes, and terminals are often called "I/O streams" because
data flows from or to them like water (same analogy from which "pipes"
is derived).  Command line arguments are fixed locations in memory
(the argv[] array in C programs) that may be populated at the time
the program starts; nothing "flows" in the arguments.

So it's better not to think of this as "input from either end."  They
are entirely distinct.  Because the command line arguments are fixed
at program start, they're often used to control the behavior.  Using
the example of "grep" (and temporarily ignoring option flags), the
first argument is the pattern to look for, and *if* there is a second
(and so on) argument, it's the name of the file to look in, so grep
opens that file.  Otherwise it uses stdin, but either way it has an
I/O stream in which to look for the pattern.  The arguments were only
used to control the choice of stream.

Grep would be much less useful if treated command line arguments as if
they were file *contents* instead of file *names*.  That's not to say
one never treats a command line argument as content, but it's not the
typical usage.

} but  how could it be improved? I tried 'cat' with no luck. 

I'm still not sure I understand exactly what you want to do, but let's
look at a fragment of "man cat" as an example:

	cat [OPTION] [FILE]...

	Concatenate FILE(s), or standard input, to standard output.

	With no FILE, or when FILE is -, read standard input.

(This manpage uses FILE as both the name of the file and as shorthand
for "the contents of FILE", which is common.)

Note that "cat" can actually read standard input MORE THAN ONCE!  It
tries again every time it finds "-" as a file name.  Most of the time
you'll just get end of file after the first try, but some special
kinds of streams (which includes terminals) can be closed and then
opened again "where they left off" (so to speak).

The equivalent documentation for your "y" function with [[ ! -t 0 ]]
might read:

	y [STRING]...

	If the input is not a terminal, copy one LINE from standard
	input, otherwise LINE is empty.

	Concatenate LINE and STRING(s), to standard output.

If that's how you meant for it to work, I can't improve on what you
have, except for some cosmetics (for example, you should not test
[ -n "$1" ] because it's possible for $1 to be empty and $2 to have
something in it).

If you want to copy the entire standard input and then append the
STRING(s) at the end of it, it should be fine to do:

    function y ()
    {
      if [ ! -t 0 ]; then cat
      else echo '(nothing in the pipe)'; fi
      if [ $# -gt 0 ]; then print -r -- "$*"
      else echo '(nothing in the args)'; fi
    }

Not *exactly* as before since I've assumed that if you'd be copying
multiple lines from the input then it's OK to put a newline after the
'(nothing in the pipe)' string.  The "print -r --" is more cosmetics,
so that "y -n" will output the -n instead of suppressing the newline
from echo.  If you meant for "y" to accept the options of echo, by
all means change that back.

} I've already done some chains of pipes (using binaries) up to maybe
} half a dozen steps, and you just don't have to worry about time-outs,
} or single lines or anything at all, it just works, even on Tuesday.

Yes, but have you noticed that if you just type at the command line

 % cat

(or pretty much anything else in your chain of pipes) that it will just
sit there forever waiting for something?  It's reading the terminal,
because that's what the shell tells it is the standard input.  That's
what all those programs that "just work" doing: an unconditional,
blocking-forever, read from somewhere, stdin by default.

As soon as you throw in the bit about not waiting on the terminal when
there is no pipe on the left, you have to start worrying about a lot
of the stuff that you didn't worry about before.

Re: set -F kills read -t

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 2633 bytes --]

On 03/19/2014 09:21 PM, Bart Schaefer wrote:
> On Mar 19,  6:46pm, Ray Andrews wrote:
> }
> } Ok Bart, how would you write it? The above  is a bridgehead in as much
> } as it accepts input from either end
>
> Command line arguments are a fundamentally different kind of "input".

Yeah, I know. I'm speaking in a sloppy way because I'm focusing in on 
the 'grep'
situation where the command can be fed from either end. But of course it 
seems
that that is particular to grep, not zsh syntax itself, so you're right 
to suspect my
understanding.
>
> I'm still not sure I understand exactly what you want to do,
This! :

     function y ()
     {
       if [ ! -t 0 ]; then cat
       else echo '(nothing in the pipe)'; fi
       if [ $# -gt 0 ]; then print -r -- "$*"
       else echo '(nothing in the args)'; fi
     }


Except that we've come in a circle. Try calling it from this alias:

        alias y='set -F; _y'

     function _y ()
     {
       if [ ! -t 0 ]; then cat
       else echo '(nothing in the pipe)'; fi
       if [ $# -gt 0 ]; then print -r -- "$*"
       else echo '(nothing in the args)'; fi
     }

.... back to 'set -F' killing the pipe :-(

    $ echo "My love is" | _y "as a fever"
    My love is
    as a fever

    $ echo "My love is" | y "as a fever"
    (nothing in the pipe)
    as a fever



> If that's how you meant for it to work, I can't improve on what you
> have, except for some cosmetics (for example, you should not test
> [ -n "$1" ] because it's possible for $1 to be empty and $2 to have
> something in it).
That's a rude shock but I just tested it, and right you are.
Anyway '[ $# -gt 0 ]' is obviously robust, so I like it.

> (or pretty much anything else in your chain of pipes) that it will just
> sit there forever waiting for something?  It's reading the terminal,
> because that's what the shell tells it is the standard input.  That's
> what all those programs that "just work" doing: an unconditional,
> blocking-forever, read from somewhere, stdin by default.
Yes! And that's how God meant it to be ;-)
> As soon as you throw in the bit about not waiting on the terminal when
> there is no pipe on the left, you have to start worrying about a lot
> of the stuff that you didn't worry about before.
Misericordia

Bart, I'm taking too  much of your time. The list functions at a higher 
level than my
novice problems. I'm dabbling in sacred mysteries that are above me. Let 
me get
back to Peter's book and throw myself at this stuff when I know a bit 
more Coptic. C
is meant to be understandable and predictable, the shells are meant to 
be like playing
Dungeons and Dragons.

Re: set -F kills read -t

[Bart Schaefer]

On Mar 20,  8:49am, Ray Andrews wrote:
}
} Except that we've come in a circle. Try calling it from this alias:
} 
}         alias y='set -F; _y'
} 
} .... back to 'set -F' killing the pipe :-(

It's not the "set" that's killing the pipe, it's the semicolon.  Didn't
we go through this once before?
   
}     $ echo "My love is" | y "as a fever"

The alias expands that into

	echo "My love is" | set -F; _y "as a fever"

so the semicolon ends the pipeline.  Aliases are textual replacements,
not semantic units.  Also with that alias the -F setting will persist
after _y is finished, which I'm sure you didn't intend.

Try it this way:

   alias y='noglob _y'

Re: set -F kills read -t

[Ray Andrews]

On 03/20/2014 09:08 AM, Bart Schaefer wrote:
> 	echo "My love is" | set -F; _y "as a fever"
>
> so the semicolon ends the pipeline.

Yup, I should have figgered that myself. Red herrings! a dude can be 
following the wrong scent
completely.
> Aliases are textual replacements,
> not semantic units.  Also with that alias the -F setting will persist
> after _y is finished, which I'm sure you didn't intend.
No, it gets turned off automatically elsewhere in my Grand Unified 
Wrapper System.
>
> Try it this way:
>
>     alias y='noglob _y'
Seems fine! Thanks Bart, you're a patient man. No more torments for a while
anyway.

Re: set -F kills read -t

[Roman Neuhauser]

# rayandrews@eastlink.ca / 2014-03-18 22:00:49 -0700:
> Ok, then how else would one write a function that could use arguments 
> *or* piped input?

repeating another post on this thread:

mygrep() {                                                                               
    if [ -t 0 ]; then                                                 
        echo terminal                                                                                             
    else                                                                      
        read input                                                  
        echo $input input                                         
    fi                                                                
}                                                                     

-- 
roman