coloring STDERR to terminal

coloring STDERR to terminal

[Atom 'Smasher']

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

i'm not sure if this is possible without patching some sources, but here's 
what i want to do... every time STDERR is directed to a terminal, i want 
to filter it through _something_like_ this:
 	awk '{print "^[[91m"$0"^[[0m"}'

i want my STDERR to be a different color than my STDOUT (when STDERR is 
being directed to my terminal).

is there a way to do that in my dot-zshrc?

it would be neat if i could do some tricks with exec and shuffle STDERR 
through a filter like that... i'm not sure if it would be better to invoke 
`awk` once per shell, or every time STDERR is directed to a terminal... 
both ways have drawbacks. i'm not sure that's it's even possible to do it 
that way... or, if done that way, if it's possible to differentiate 
between output going to a terminal or not.

ideally, i think this would require a new environment variable... maybe 
something like STDERR_COLOR, and can probably just take an integer as a 
value and fill in the control sequences around it... among other 
advantages, there'd be no dependence on an external application. also, 
that's probably the only way to accurately deal with situations when 
STDERR isn't followed by a newline before STDOUT comes along.

if i was any good at coding in C i'd write a patch... but C is still on my 
list of things to master.

thanks...

         ...atom

  _________________________________________
  PGP key - http://atom.smasher.org/pgp.txt
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"The conservation movement is a breeding ground of Communists
 	 and other subversives.  We intend to clean them out, even if
 	 it means rounding up every bird watcher in the country."
 		-- John Mitchell, US Atty. General 1969-1972
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.6 (FreeBSD)
Comment: What is this gibberish?
Comment: http://atom.smasher.org/links/#digital_signatures

iEYEARECAAYFAkDfVrIACgkQnCgLvz19QeOxOQCdHBPSy1F/k2Sr69kPCN0XCBEo
OxoAn3y5XBWYEq8qAm8sMKduHpsfy6RS
=csXO
-----END PGP SIGNATURE-----

Re: coloring STDERR to terminal

[ari]

atom@suspicious.org mentioned:

> i'm not sure if this is possible without patching some sources, but here's 
> what i want to do... every time STDERR is directed to a terminal, i want 
> to filter it through _something_like_ this:
> 	awk '{print "^[[91m"$0"^[[0m"}'

Perhaps not what you're looking for, since the effect is asynchronous,
but you _could_ do this with a variation of:

  ERRFIFO=$HOME/.stderr-$TTY.fifo
  [ -p $ERRFIFO ] || mkfifo $ERRFIFO
  awk '{print "^[[91m"$0"^[[0m"}' <$ERRFIFO 1>&2 &

  exec 2>$ERRFIFO

Execution of this segment would, of course, need to be restricted to
once per terminal session.

ari

Re: coloring STDERR to terminal

[Atom 'Smasher']

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 27 Jun 2004, ari wrote:

> atom@suspicious.org mentioned:
>
>> i'm not sure if this is possible without patching some sources, but here's
>> what i want to do... every time STDERR is directed to a terminal, i want
>> to filter it through _something_like_ this:
>> 	awk '{print "^[[91m"$0"^[[0m"}'
>
> Perhaps not what you're looking for, since the effect is asynchronous,
> but you _could_ do this with a variation of:
>
>  ERRFIFO=$HOME/.stderr-$TTY.fifo
>  [ -p $ERRFIFO ] || mkfifo $ERRFIFO
>  awk '{print "^[[91m"$0"^[[0m"}' <$ERRFIFO 1>&2 &
>
>  exec 2>$ERRFIFO
>
> Execution of this segment would, of course, need to be restricted to
> once per terminal session.
===================

i guess this demonstrates what you mean by "asynchronous":
 	{ echo -n a ; echo -n b >&2 ; echo c ; echo d >&2 ; }

(on freeBSD, "echo -n" suppresses the newline. i don't think that option 
is portable across all flavors of *nix)

the results are ~not~ what i would've though  ;)

it also doesn't do well at all with this command:
 	gpg < clearsigned_file
which outputs some lines as STDERR, other lines as STDOUT.

and this isn't working anything the way i'd expect it to:
 	time date > file_stdout 2> file_stderr

it seems that i can't redirect STDERR unless i run the command in 
curly-braces or parenthesis...
 	{ time date ; } > file_stdout 2> file_stderr

back-quotes don't even work.


  	...atom

  _________________________________________
  PGP key - http://atom.smasher.org/pgp.txt
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"Cryptography is like literacy in the Dark Ages. Infinitely
 	 potent, for good and ill... yet basically an intellectual
 	 construct, an idea, which by its nature will resist efforts
 	 to restrict it to bureaucrats and others who deem only
 	 themselves worthy of such Privilege."
 		-- Vin McLellan,
 		A Thinking Man's Creed for Crypto
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.6 (FreeBSD)
Comment: What is this gibberish?
Comment: http://atom.smasher.org/links/#digital_signatures

iEYEARECAAYFAkDfaBMACgkQnCgLvz19QeN4+ACfQoj6iDKdInD31W7FnsAvPRlx
9EsAn0FNB0lBJqoHeUKnqrU/y7jceQDd
=X7Y4
-----END PGP SIGNATURE-----

Re: coloring STDERR to terminal

[Bart Schaefer]

On Sun, 27 Jun 2004, Atom 'Smasher' wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> i'm not sure if this is possible without patching some sources, but here's
> what i want to do... every time STDERR is directed to a terminal, i want to
> filter it through _something_like_ this:
> 	awk '{print "^[[91m"$0"^[[0m"}'

This sort of thing is the reason that zsh's coprocess hangs around after 
its input and output are closed, rather than exiting the way ksh coprocs
do.  Assuming you're not using the coprocess for anything else (hardly
anyone does) you can do this:

 coproc while read line; print '\e[91m'${(q)line}'\e[0m' > /dev/tty
 exec 2>&p

(Oh, that also assumes that the "shortloops" option is set.  Insert a "do"
and end with ";done" if you don't have that setopt.)

It's better to use a zsh while-loop than something like awk because most 
filter programs will wait for a full input buffer before processing and 
for a full output buffer before printing, but here you want each line to 
be processed and printed immediately.

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-06-29 08:43:50 -0700, Bart Schaefer wrote:
> This sort of thing is the reason that zsh's coprocess hangs around after 
> its input and output are closed, rather than exiting the way ksh coprocs
> do.  Assuming you're not using the coprocess for anything else (hardly
> anyone does) you can do this:
> 
>  coproc while read line; print '\e[91m'${(q)line}'\e[0m' > /dev/tty
>  exec 2>&p

It doesn't work very well. Normally, if I do a Ctrl-d when there are
running jobs, I get:

greux:~>
zsh: you have running jobs.
greux:~>

With the coprocess, I get:

greux:~>
greux:~> zsh: you have running jobs.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Bart Schaefer]

On Tue, 29 Jun 2004, Vincent Lefevre wrote:

> On 2004-06-29 08:43:50 -0700, Bart Schaefer wrote:

> >  coproc while read line; print '\e[91m'${(q)line}'\e[0m' > /dev/tty
> >  exec 2>&p
> 
> It doesn't work very well.

Humph.  Depends on your definition of "very well."  Everything sent to the 
standard error is printed in red, isn't it?

> Normally, if I do a Ctrl-d when there are
> running jobs, I get:
> 
> greux:~>
> zsh: you have running jobs.
> greux:~>

Which is exactly what I get even with the coproc, so there's something 
system-specific going on here (race condition resolved differently by the 
kernel process scheduler, perhaps).

> With the coprocess, I get:
> 
> greux:~>
> greux:~> zsh: you have running jobs.

If you want something fancier, have the coproc print a single byte to its
stdout every time around the read loop, and put something like

 while read -t -p -k 1; do :; done

in the precmd() function to force the main shell to synchronize with the
coproc before it prints the prompt.  However, if you produce more lines of 
output between commands than there are bytes of space in the socket buffer
you'll block the coproc and potentially deadlock everything (which is the
trouble with filtering the output of programs that don't expect to have
their output filtered, but that wasn't the question).

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-06-29 10:14:13 -0700, Bart Schaefer wrote:
> If you want something fancier, have the coproc print a single byte to its
> stdout every time around the read loop, and put something like
> 
>  while read -t -p -k 1; do :; done
> 
> in the precmd() function to force the main shell to synchronize with the
> coproc before it prints the prompt.

Thanks.

> However, if you produce more lines of output between commands than
> there are bytes of space in the socket buffer you'll block the
> coproc and potentially deadlock everything (which is the trouble
> with filtering the output of programs that don't expect to have
> their output filtered, but that wasn't the question).

I don't understand how this can happen. Otherwise this would mean that
bytes from stderr could be discarded, which would also be very bad.

Also, in the coprocess (while read line; ...), the colored data will
be output only after a \n. However, one generally wants stderr to be
output immediately (contrary to stdout), i.e. unbuffered instead of
line buffered. I assume that -k 1 would fix that, but it would be
inefficient. One could probably use "read -t -k 1" in a loop until
it fails or return a \n. Wouldn't it be fine to have a read option
(e.g. -T) that does this, i.e. read what is requested (i.e. until \n
or num characters if -k is used) or return as soon as nothing else
is available?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Bart Schaefer]

On Wed, 30 Jun 2004, Vincent Lefevre wrote:

> On 2004-06-29 10:14:13 -0700, Bart Schaefer wrote:
> > If you want something fancier, have the coproc print a single byte to 
> > its stdout every time around the read loop

To clarify, by this I mean to its original standard output, not to the 
redirected stdout which is going to /dev/tty (otherwise the parent zsh 
can't see it).  E.g. instead of

 coproc while read line; print '\e[91m'${(q)line}'\e[0m' > /dev/tty

You need

 coproc while read line; do
          print '\e[91m'${(q)line}'\e[0m' > /dev/tty
          print -n $'\0'
        done

>[...] 
> > However, if you produce more lines of output between commands than
> > there are bytes of space in the socket buffer you'll block the
> > coproc and potentially deadlock everything
> 
> I don't understand how this can happen.

The situation is thus: The parent zsh (Z) holds the write-end (W) of the 
standard input of coprocess (C), and the read end (R) of the standard 
output of the coprocess.  Within the coprocess, the first print command 
has its standard output redirected to the terminal (T), but the second is 
still writing on R.

Z passes a copy of W to new job (J) as its stderr, then waits for J.

C is thus reading from W and writing to both T and to R, but Z is not 
reading from R (because Z is waiting), so C can only execute as many loops 
as there are bytes in the buffer for R before C blocks.  If C blocks, it 
stops reading from W, which means that eventually the buffer for W will 
fill up and also block J.  Deadlock.

This happens even faster when Z and J are the same process (a built-in
command or shell function).  The whole thing operates much better if C
is independent of Z, and you live with the race condition that means you
may sometimes get a prompt in the middle of your error output.

Which is, in part, why I said whether it "doesn't work very well" depends 
on your definition of "very well."  It works as well as it can, given the
original premise.

Using "read -t" in C's while loop test doesn't help with this, because the 
gating factor is Z waiting on J.  If you put J in the background (so Z is 
not waiting on it), the gating factor becomes Z waiting on input from the 
terminal at the prompt, but that you can solve with a "zle -F" handler (in 
4.2.1 or later).

> Wouldn't it be fine to have a read option (e.g. -T) that does this, i.e. 
> read what is requested (i.e. until \n or num characters if -k is used) 
> or return as soon as nothing else is available?

See "sysread" (and "syswrite") in the zsh/system module.

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-06-30 03:52:46 -0700, Bart Schaefer wrote:
> To clarify, by this I mean to its original standard output, not to the 
> redirected stdout which is going to /dev/tty (otherwise the parent zsh 
> can't see it).  E.g. instead of
> 
>  coproc while read line; print '\e[91m'${(q)line}'\e[0m' > /dev/tty
> 
> You need
> 
>  coproc while read line; do
>           print '\e[91m'${(q)line}'\e[0m' > /dev/tty
>           print -n $'\0'
>         done
[...]
> The situation is thus: The parent zsh (Z) holds the write-end (W) of the 
> standard input of coprocess (C), and the read end (R) of the standard 
> output of the coprocess.  Within the coprocess, the first print command 
> has its standard output redirected to the terminal (T), but the second is 
> still writing on R.
[...]

Thanks for the explanation. But in fact, I didn't add the

  print -n $'\0'

line, so there is no R problem in my case. So, is this line really
useful?

Also, when I quit zsh, the coprocess is not killed (the NO_HUP option
is set, but a coprocess can be seen as a particular case). Is it a bug?

And is it possible to hide the coprocess from the jobs table?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-06-30 13:43:41 +0200, Vincent Lefevre wrote:
> Thanks for the explanation. But in fact, I didn't add the
> 
>   print -n $'\0'
> 
> line, so there is no R problem in my case. So, is this line really
> useful?

Hmm... I now see. It seems I got confused by the zshbuiltins man page,
which says:

              -p     Input is read from the coprocess.

Is it
  1) from the coprocess output, or
  2) sharing the coprocess input?

First I supposed (2); I think that this is really what I want.

But looking at the explanations, it seems to be (1); however, in this
case, there seems to be a race condition anyway, as the coprocess may
be too late to write its input to the tty, i.e. the following is
possible, isn't it?

1) I type Ctrl-d.
2) "zsh: you have running jobs." is written by zsh to stderr (will be
   read by the coprocess).
3) precmd() is called (but there are still no bytes sent by the
   coprocess).
4) The new prompt is displayed (sent to the tty).
5) The coprocess reads a line.
6) The coprocess sends "zsh: you have running jobs." to the tty.
7) The coprocess sends a byte to stdout (too late!).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Bart Schaefer]

On Wed, 30 Jun 2004, Vincent Lefevre wrote:

> On 2004-06-30 03:52:46 -0700, Bart Schaefer wrote:
> > 
> >  coproc while read line; do
> >           print '\e[91m'${(q)line}'\e[0m' > /dev/tty
> >           print -n $'\0'
> >         done
> 
> Thanks for the explanation. But in fact, I didn't add the
> 
>   print -n $'\0'
> 
> line, so there is no R problem in my case. So, is this line really
> useful?

(See next message excerpted below.)

> Also, when I quit zsh, the coprocess is not killed (the NO_HUP option
> is set, but a coprocess can be seen as a particular case). Is it a bug?

Possibly.

> And is it possible to hide the coprocess from the jobs table?

Older versions of zsh did that -- the coprocess was not managed as a job 
at all, in fact -- but it was changed because (with the old behavior) 
there was no clean way to send termination signals etc. to the coprocess.

So at present, no, it's not possible to hide the coprocess.

It just occurred to me that

 exec 2>>( while ... & )

accomplishes the same thing without using the coprocess at all.  However, 
it doesn't give you a handle on the stdout of the "colorizer" so it's more 
difficult to synchronize it with the printing of the prompt.

On Wed, 30 Jun 2004, Vincent Lefevre wrote:
> 
> Hmm... I now see. It seems I got confused by the zshbuiltins man page,
> which says:
> 
>               -p     Input is read from the coprocess.
> 
> Is it
>   1) from the coprocess output, or
>   2) sharing the coprocess input?

It's (1).

> First I supposed (2); I think that this is really what I want.

There's no way for two processes to read from the same descriptor without 
stealing bytes from each other (or synchronizing in some out-of-band way 
so that they don't do so).  Zsh multios work by creating an intervening
process that reads from one descriptor and writes duplicate bytes on two
(or more) other descriptors, for just this reason.

> But looking at the explanations, it seems to be (1); however, in this
> case, there seems to be a race condition anyway, as the coprocess may
> be too late to write its input to the tty, i.e. the following is
> possible, isn't it?
> 
> 1) I type Ctrl-d.
> 2) "zsh: you have running jobs." is written by zsh to stderr (will be
>    read by the coprocess).
> 3) precmd() is called (but there are still no bytes sent by the
>    coprocess).

At this point the "read -p" in precmd() will block, because the coproc
has not yet executed "print -n $'\0'".  So your step (4) won't happen:

> 4) The new prompt is displayed (sent to the tty).

And instead you go directly here:

> 5) The coprocess reads a line.
> 6) The coprocess sends "zsh: you have running jobs." to the tty.
> 7) The coprocess sends a byte to stdout (too late!).

At this point "read -p" gets that byte and precmd() proceeds and the 
prompt is printed.

There _is_ a race condition when, at step (2), _more_than_one_ line is 
written to stderr, because then the coprocess may "print -n" one or more 
bytes "too soon".

You might be able to get away with something involving having preexec
start up a process and then precmd wait for it and kill it off.

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-06-30 09:56:37 -0700, Bart Schaefer wrote:
> On Wed, 30 Jun 2004, Vincent Lefevre wrote:
> > And is it possible to hide the coprocess from the jobs table?
> 
> Older versions of zsh did that -- the coprocess was not managed as a job 
> at all, in fact -- but it was changed because (with the old behavior) 
> there was no clean way to send termination signals etc. to the coprocess.
> 
> So at present, no, it's not possible to hide the coprocess.

Couldn't the coprocess be marked in a special way, to send termination
signals, etc.?

> > 1) I type Ctrl-d.
> > 2) "zsh: you have running jobs." is written by zsh to stderr (will be
> >    read by the coprocess).
> > 3) precmd() is called (but there are still no bytes sent by the
> >    coprocess).
> 
> At this point the "read -p" in precmd() will block, because the coproc
> has not yet executed "print -n $'\0'".  So your step (4) won't happen:
> 
> > 4) The new prompt is displayed (sent to the tty).

I don't understand why it will block. The line you gave was:

  while read -t -p -k 1; do :; done

and the zshbuiltins(1) man page says:

     -t [ num ]
            Test if input is available before attempting to read.  If
            num  is  present,  it must begin with a digit and will be
            evaluated to give a number of seconds,  which  may  be  a
            floating point number; in this case the read times out if
            input is not available within this time.  If num  is  not
            present,  it  is  taken  to be zero, so that read returns
            immediately if no input is available.   If  no  input  is
            available,  return status 1 and do not set any variables.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Bart Schaefer]

On Thu, 1 Jul 2004, Vincent Lefevre wrote:

> I don't understand why it will block. The line you gave was:
> 
>   while read -t -p -k 1; do :; done

Ah.  You're right; I had only a half-formed thought, there.

I originally wrote (before sending my message, i.e. during the editing 
process) "read -p -k 1" (no loop), implying always read one byte from the 
coprocess.  But in re-reading I realized that blocks forever when there is 
no stderr for the coproc to colorize.  So I changed it to the "while" 
loop, which would do nothing if the coprocess produced no output, and read 
the byte otherwise.  But I failed to notice that this also means it does 
nothing if the coproc has not written the byte *yet*, which is what you 
were pointing out.

So I'm forced to conclude that there really is no way to avoid the race 
condition that you originally complained about; my first solution is as 
good as it gets.  You might get close if you used "read -t 1 ..." to make 
the parent shell sleep so the kernel can schedule the coprocess first, but 
then you'd have an annoying 1-second delay on almost every prompt.  Sorry.

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-07-01 17:11:08 -0700, Bart Schaefer wrote:
> So I'm forced to conclude that there really is no way to avoid the
> race condition that you originally complained about; my first
> solution is as good as it gets. You might get close if you used
> "read -t 1 ..." to make the parent shell sleep so the kernel can
> schedule the coprocess first, but then you'd have an annoying
> 1-second delay on almost every prompt. Sorry.

I have a better solution. stderr is redirected with

  exec 2>>(while read line; do
    print '\e[91m'${(q)line}'\e[0m' > /dev/tty; done &)

as you suggested (this avoids the coprocess in the jobs table).

Now, in precmd(), adding the following line solves the problem:

  sleep 0

Of course, this won't completely avoid the race condition, but it
improves things in pratice. I assume that this forces the scheduler
to switch to another process, and in practice, the coprocess is
scheduled before precmd() is resumed.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Bart Schaefer]

On Fri, 2 Jul 2004, Vincent Lefevre wrote:

> Now, in precmd(), adding the following line solves the problem:
> 
>   sleep 0
> 
> Of course, this won't completely avoid the race condition, but it
> improves things in pratice. I assume that this forces the scheduler
> to switch to another process, and in practice, the coprocess is
> scheduled before precmd() is resumed.

As sleep is an external command rather than a builtin (cf. zselect), this 
forces zsh to perform a wait() system call [more or less], which in turn 
forces the scheduler to let the sleep run to completion before zsh wakes 
up again.  That it's specifically "sleep 0" is probably irrelevant; any 
external command is likely to have the same effect.

Re: coloring STDERR to terminal

[Atom 'Smasher']

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

as i suspected, there doesn't seem to be a really good way to do this... 
although there are several ways to ~almost~ do it well.

that said, i'd like to request a feature for coloring STDERR. i think the 
best way to do it would be an environment variable, something like this:
 	STDERR_COL=91,107

which would wrap STDERR in:
 	^[[91m^[[107m{output}^[[0m

maybe a STDIN_COL and STDOUT_COL could also be handy?

that seems to be the only way to ~properly~ deal with the synchronization 
and buffer issues.


  	...atom

  _________________________________________
  PGP key - http://atom.smasher.org/pgp.txt
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"Do not wait; the time will never be 'just right.'
 	 Start where you stand, and work with whatever
 	 tools you may have at your command, and better
 	 tools will be found as you go along."
 		-- Napoleon Hill
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.6 (FreeBSD)
Comment: What is this gibberish?
Comment: http://atom.smasher.org/links/#digital_signatures

iQEcBAEBCAAGBQJA/OGWAAoJEAx/d+cTpVcizhYH/2gi5w9cjys16gr3+49wUdoE
llxeoAIBoyjd/MrZE7OLniL5OYrYgPjuus5SpoXoO5x4UY7azVWbLggIEVQGs0+9
Jyl8aJvhpIFTTTKqCVoUTgdbBPXbA57OWQ63FSqmPS8c3kRifi8NF3oHqHXVk2hu
/mUJJx6ROxWcf2S3LArOmzHxpAo+f4sASL8MRoGmrl46n38isYh4VAr61Tino5yu
OsnAnkOdgSqreXLP10LwRYnd84khn1tDZVqY+6CeEW3W7of/8sNxp6qDhO6tttFA
uPMvcY5COXZbsiGRR8LjHaWrGrpVLqU0h3zkT5H2rXenxsyMF5VffRtprfDJ2pk=
=8KIU
-----END PGP SIGNATURE-----

Re: coloring STDERR to terminal

[Bart Schaefer]

On Tue, 20 Jul 2004, Atom 'Smasher' wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> as i suspected, there doesn't seem to be a really good way to do this...
> although there are several ways to ~almost~ do it well.
> 
> that said, i'd like to request a feature for coloring STDERR.

Your first sentence is quite literally true, which makes your feature 
request a little pointless.  It doesn't matter by whom [*] the coloring is 
done; all the problems so far discussed are still present, with the 
possible exception of zsh being able to avoid printing a prompt in the 
middle of the error output.

Even if you were to create a single process that could monitor both the 
stdout and stderr of every command run by zsh, and properly merge those 
streams so that the colors look right on the terminal, you can't avoid the 
issue that the stdio library changes its internal buffering behavior when 
the output file descriptors are not connected to TTY devices; so you'd 
have to use pseudo-TTYs instead of simple pipes.  Even that is still 
subject to race conditions that can't be guaranteed to resolve the way
they would if the two streams were going directly to the same TTY.

On top of that, you need a scheme for dealing with interactive programs 
(or anything that does screen redraws involving cursor movement) such as 
text editors, that need their own direct and unmunged access to the actual 
terminal device.  You'd have to create some sort of registry for them, 
like the "needsterminal" attribute in mailcap files, and deal with the 
headache of keeping that up to date for every new progam.

The cumulative complexity of implementation is not worth the benefit.

[*] If the kernel drivers could do it between the process output and the
terminal display, you might get something close to what you want, but in
that case you're on the wrong mailing list.

> maybe a STDIN_COL and STDOUT_COL could also be handy?

You don't really mean "stdin color" you mean "the ZLE editor buffer is a 
different color than the ZLE prompts."  I hope.  Otherwise you're asking
for another process between the terminal and the input of every command,
which process subsumes the role of the terminal driver in echoing back the
characters typed.  There are programs that do this (run zsh in an emacs
buffer, for example) but they have their own set of quirks.

For the ZLE buffer, you can color that with some careful settings of the
various prompt variables.

Re: coloring STDERR to terminal

[Atom 'Smasher']

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, 20 Jul 2004, Bart Schaefer wrote:
> On Tue, 20 Jul 2004, Atom 'Smasher' wrote:

>> as i suspected, there doesn't seem to be a really good way to do this...
>> although there are several ways to ~almost~ do it well.
>>
>> that said, i'd like to request a feature for coloring STDERR.
>
> Your first sentence is quite literally true, which makes your feature
> request a little pointless.  It doesn't matter by whom [*] the coloring is
> done; all the problems so far discussed are still present, with the
> possible exception of zsh being able to avoid printing a prompt in the
> middle of the error output.
<<snip>>
================

you raise some intersting issues... and not being much of a C programmer 
myself i'm not sure how to address them... my logic is that if the 
coloring of STDOUT, STDERR and echoing STDIN is handled ~internally~ by 
the shell, then everything else should (FTMP) work itself out.


         ...atom

  _________________________________________
  PGP key - http://atom.smasher.org/pgp.txt
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"`Energize the demolition beams.' Light poured out into the
 	 hatchways. `I don't know,' said the voice on the PA,
 	 `apathetic bloody planet, I've no sympathy at all.'
 	 It cut off.
 	 There was a terrible ghastly silence.
 	 There was a terrible ghastly noise.
 	 There was a terrible ghastly silence.
 		-- Douglas Adams, Hitchhikers Guide to the Galaxy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.6 (FreeBSD)
Comment: What is this gibberish?
Comment: http://atom.smasher.org/links/#digital_signatures

iQEcBAEBCAAGBQJA/XJAAAoJEAx/d+cTpVciMkoIAIeAXCFMjV0q2hKeldkNt3qS
aXVOVbGLD1Y83GyaK2yXOQftxPlI3HI/5sMF+DkwvyJ6BW2gZWpx4zEkweE7Z3jy
z6X+NPSdwEkRzQl61e31v48hcxqVCijskRju2Tubbsk9D/3os7YorkO4qqbH00Dk
Jc1nugT8mkep+XaZrtlzTQAzAgtaSKlPeeFeJtToV0sVQvTsI15XTkWdPWYr70X6
bxypD0FGAvdkuUIpT3Rj2WfXRBhgZ/5iutfSs9krXkeH0tqJhHM4x90KPPEMgUMR
MdaVrPGfF8N5EQi+FdkjdwwKAWjd4LQJ10UMfKuXHfkR4MrpHfbfgsnSIFaZJ+U=
=LQxS
-----END PGP SIGNATURE-----

Re: coloring STDERR to terminal

[Bart Schaefer]

On Tue, 20 Jul 2004, Atom 'Smasher' wrote:

> you raise some intersting issues... and not being much of a C programmer 
> myself i'm not sure how to address them... my logic is that if the 
> coloring of STDOUT, STDERR and echoing STDIN is handled ~internally~ by 
> the shell, then everything else should (FTMP) work itself out.

What you're missing is that the shell is not involved in the processing of 
these streams except insofar as it sets up the file descriptors used by 
the commands it spawns.  All the shell does or can do is start the new 
process and then get out of the way until it is finished.

The exception, in the case of zsh, is multios, where the shell explicitly 
sets up each of those file descriptors to point to a special subprocess 
that copies its single input to its multiple outputs as fast as it can.  

What you're asking for is something to copy multiple inputs to a single 
output, simultaneously adding characters (the color codes) to those inputs 
yet without mixing the result any differently than it would have been 
mixed if the writers of all those inputs had instead written directly to 
that single output.

This is, pretty much by definition, impossible, except (sometimes) at the 
level of a kernel driver.

Re: coloring STDERR to terminal

[Wayne Davison]

On Tue, Jul 20, 2004 at 02:15:07PM -0700, Bart Schaefer wrote:
> What you're asking for is something to copy multiple inputs to a single 
> output, simultaneously adding characters (the color codes) to those inputs 
> yet without mixing the result any differently than it would have been 
> mixed if the writers of all those inputs had instead written directly to 
> that single output.
> 
> This is, pretty much by definition, impossible, except (sometimes) at the 
> level of a kernel driver.

It's impossible with a single tty (or pty), but if stdout and stderr
each had their own pty, then a program could merge the two ptys to the
real terminal, coloring the stderr one.

Here's a link to a proof-of-concept C program that you can try out:

    http://www.blorf.net/color-my-world.c

It spawns a shell and colors all the data sent to stderr.  I've only
tested it just a little bit, so I don't know if having a separate pty
for stderr will cause any weird problems or not.

..wayne..

Re: coloring STDERR to terminal

[Bart Schaefer]

On Tue, 20 Jul 2004, Wayne Davison wrote:

> On Tue, Jul 20, 2004 at 02:15:07PM -0700, Bart Schaefer wrote:
> > yet without mixing the result any differently than it would have been 
> > mixed if the writers of all those inputs had instead written directly 
> > to that single output.
> 
> It's impossible with a single tty (or pty), but if stdout and stderr
> each had their own pty, then a program could merge the two ptys to the
> real terminal, coloring the stderr one.

Yes, I said something about that before.  This is as close as you can get 
to what was asked for, but (a) it's still not precisely the same [*] as if 
both streams went directly to a single TTY, and (b) you can't invoke it 
blindly/automatically on every spawned command, because it will play havoc 
with any text editor or other fullscreen UI.

[*] E.g. your sample program always writes stdout first and then stderr, 
even when both are found to be available by the same call to select().

Re: coloring STDERR to terminal

[Wayne Davison]

On Tue, Jul 20, 2004 at 08:15:26PM -0700, Bart Schaefer wrote:
> you can't invoke it blindly/automatically on every spawned command,
> because it will play havoc with any text editor or other fullscreen
> UI.

This works the other way around -- you invoke a shell inside it and it
colors all the stderr I/O of everything the shell runs.  I ran an editor
just fine (vim) from a zsh running under it without any havoc.

> E.g. your sample program always writes stdout first and then stderr, 
> even when both are found to be available by the same call to select().

Yeah, I thought of that too while I was eating.  I have them reversed in
the latest code on my website.  I also enhanced the code to forward the
SIGWINCH signal (so that the shell resizes properly) and I moved some of
the code into functions (to make it easier to read).  It seems to work
adequately in my limited testing, but it's probably not super portable.

Compiling note -- under Linux you need to specify -lutil, like this:

    gcc -o color-my-world color-my-world.c -lutil

..wayne..

Re: coloring STDERR to terminal

[Bart Schaefer]

On Tue, 20 Jul 2004, Wayne Davison wrote:

> On Tue, Jul 20, 2004 at 08:15:26PM -0700, Bart Schaefer wrote:
> > you can't invoke it blindly/automatically on every spawned command,
> > because it will play havoc with any text editor or other fullscreen
> > UI.
> 
> This works the other way around -- you invoke a shell inside it and it
> colors all the stderr I/O of everything the shell runs.  I ran an editor
> just fine (vim) from a zsh running under it without any havoc.

Try running, say, "less" circa version 346 under it.  Or the original "vi" 
... or anything that opens either FD 2 or /dev/tty to read interactive 
input because stdin may have been redirected or piped ...

Re: coloring STDERR to terminal

[Vincent Lefevre]

On 2004-07-20 23:23:56 -0700, Wayne Davison wrote:
[color-my-world.c]

I haven't tried, but what about if stderr contains escape sequences?
IMHO, coloring should only be done at the terminal level, after the
escape sequences have been interpreted. So, the terminal should be
able to read from two input streams. I don't know if this possible,
but definitely not standard and not related to zsh.

Also, I've noticed that in practice, programs use stderr for other
things than error/warning messages, and sometimes not in a consistent
way.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: coloring STDERR to terminal

[Andy Spiegl]

Hi everyone,

I'd followed the discussion closely because I also thought coloring of
stderr to be a great thing to have.  I ended up using Vincents way:

>   exec 2>>(while read line; do
>     print '\e[91m'${(q)line}'\e[0m' > /dev/tty; done &)

Which seems to work nicely, well except for the reordering of lines
sometimes, but I could live with that.

However there is one situation where output not only is out of order but
doesn't appear at all:
 condor% echo $ZSH_VERSION
 4.2.0
 condor% touch foo bar
 condor% cp -i foo bar
 cp: overwrite `bar'? y
 condor% exec 2>>(while read line; do print '\e[91m'${(q)line}'\e[0m' > /dev/tty; done &)
 condor% cp -i foo bar                                                                   
 y

As you can see the question doesn't appear at all!
But when I then do this, it suddenly appears:
 condor% echo test >&2
 condor% cp: overwrite `bar'? test

So it seems that zsh doesn't flush some buffer?
Is that a zsh-bug or a problem with this exec-trick?

Chau,
 Andy.

-- 
                              o      _     _         _
  ------- __o       __o      /\_   _ \\o  (_)\__/o  (_)          -o)
  ----- _`\<,_    _`\<,_    _>(_) (_)/<_    \_| \   _|/' \/       /\\
  ---- (_)/ (_)  (_)/ (_)  (_)        (_)   (_)    (_)'  _\o_    _\_v
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The brain is a wonderful organ; it starts working the moment you get up
 in the morning, and does not stop until you get to work.

Re: coloring STDERR to terminal

[Vincent Lefevre]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On 2004-07-30 13:50:24 +0200, Andy Spiegl wrote:
> As you can see the question doesn't appear at all!

AFAIK, this is due to the "read line", that waits for a whole line.
When I reimplemented stderr coloring, I fixed that (and other problems
too) by writing a small C program "colorize" (attached) instead of zsh
code.

So, you should do the following:

  exec 2>>(colorize `tput bold; tput setaf 1` `tput sgr0` > /dev/tty &)

for instance, by putting that in your zshrc.

But I no longer use it since too many programs use stderr for things
other than error or warning messages, and sometimes not in a consistent
way. I don't know why, perhaps because developpers think that stderr
means tty (in case stdout is redirected); but if they want to write to
the tty, they should really do it instead of using stderr.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

[-- Attachment #2: colorize.c --]
[-- Type: text/plain, Size: 2687 bytes --]

/* $Id: colorize.c 3816 2004-07-03 17:01:32Z lefevre $
 *
 * Colorize the standard input. Written for zsh stderr coloring.
 */

#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/select.h>
#include <sys/types.h>

#define BUFFSIZE 512

static volatile sig_atomic_t usr1;

static void sigusr1(int sig)
{
  usr1 = 1;
}

static void writepid(char *tmpfile)
{
  FILE *f;

  f = fopen(tmpfile, "w");
  if (f == NULL)
    {
      perror("colorize (fopen)");
      exit(EXIT_FAILURE);
    }
  fprintf(f, "%ld\n", (long) getpid());
  if (fclose(f) != 0)
    {
      perror("colorize (fclose)");
      exit(EXIT_FAILURE);
    }
}

int main(int argc, char **argv)
{
  pid_t zshpid = 0;
  char *begstr, *endstr;
  fd_set rfds;
  int ret;

  if (argc != 3 && argc != 5)
    {
      fprintf(stderr,
              "Usage: colorize <begstr> <endstr> [ <zshpid> <tmpfile> ]\n");
      exit(EXIT_FAILURE);
    }

  /* Assume that the arguments are correct. Anyway, it is not possible
     to check them entirely. */
  begstr = argv[1];
  endstr = argv[2];
  if (argc == 5)
    {
      /* To do the synchronization with the zsh prompt output...
         Seems to be useless in practice, hence the argc == 3 case. */
      zshpid = atol(argv[3]);
      signal(SIGUSR1, sigusr1);
      writepid(argv[4]);
    }

  fcntl(0, F_SETFL, fcntl(0, F_GETFL) | O_NONBLOCK);

  /* To watch stdin (fd 0). */
  FD_ZERO(&rfds);
  FD_SET(0, &rfds);

  for (;;)
    {
      ret = select(1, &rfds, NULL, NULL, NULL);

      if (ret < 0 && errno != EINTR)
        {
          perror("colorize (pselect)");
          exit(EXIT_FAILURE);
        }

      if (ret > 0)
        {
          static unsigned char buffer[BUFFSIZE];
          static int dontcol = 0;
          ssize_t n;

          while ((n = read(0, buffer, BUFFSIZE)) >= 0)
            {
              ssize_t i;

              if (n == 0)
                return 0;  /* stdin has been closed */
              for (i = 0; i < n; i++)
                {
                  if (buffer[i] == 27)
                    dontcol = 1;
                  if (buffer[i] == '\n')
                    dontcol = 0;
                  if (!dontcol)
                    fputs(begstr, stdout);
                  putchar(buffer[i]);
                  if (!dontcol)
                    fputs(endstr, stdout);
                }
            }
          fflush(stdout);
        }

      if (usr1)
        {
          usr1 = 0;
          if (kill(zshpid, SIGUSR1) != 0)
            {
              perror("colorize (kill)");
              exit(EXIT_FAILURE);
            }
        }
    }
}