Re: Logging in a web server context

[Steve Litt <slitt@troubleshooters.com>]

On Sat, 13 Jun 2020 10:41:24 +0200
Carl Winbäck <c@tunnel53.net> wrote:

> Hi folks!
> 
> Is it possible when using s6-svscan/s6-supervise to somehow arrange so
> that a daemon’s stdout is sent to one logdir and stderr to another
> logdir?

Hi Carl,

Many people have given excellent answers, some of which were over my
head. I personally think it's possible, with or without s6-rc, and will
do my best to explain that in this email.

Before I'd ever used s6 or runit, I used daemontools. With daemontools,
and therefore I assume with everything, if you didn't use one of those
log-handling daemons like syslog, your service directory acquired a
directory called "main", and inside "main" was your log file, nicely
rotated by daemontools (I assume). So like somebody said, you don't get
the fan-in fan-out, you don't need to write the daemon in the message,
and you don't need complex parsing to extract messages from a specific
daemon.

I'd imagine that's possible in all daemontools-inspired process
supervisors such as s6.

Nowadays I run one of those log gathering programs, at the suggestion
of the experts on the Void Linux IRC channel, but maybe I'll switch
back if possible.

As I remember from daemontools, it captured only stdout for logging.
So, if you wanted it to record both, you did a little fake to get it to 
record both:

=======================
exec 2>&1
exec nginx <arglist>
=======================

But the preceding is exactly what you don't want: You don't want to mix
stderr and stdout. So perhaps something like the following:

=======================
exec 2>main/stderr.log
exec nginx <arglist>
=======================

I *think* the preceding would work for you, but I don't think you'd get
log rotation, a timestamp and maybe some other needed stuff. So I think
you could do the following:

=======================
exec 2>stderr.intermediate | stderrhandler.sh main
exec nginx <arglist>
=======================

Sorry for the contrived nature of line 1 of the preceding. I'm sure
there must be something more direct that doesn't require an
intermediate file, but I couldn't quickly find it using either
/bin/sh or execline.

Anyway, perhaps stderrhandler.sh could be made to do log rotation too,
as well as tacking on a timestamp.

If I'm correct about this, the supervisor itself puts the stdout log in
./main/whatever.log and rotates the log. Meanwhile, stderrhandler.sh
puts stderr into a different log, and perhaps stderrhandler.sh could
also log rotate, although I have a feeling there would be race
conditions doing so.

SteveT

Steve Litt 
May 2020 featured book: Troubleshooting Techniques
     of the Successful Technologist
http://www.troubleshooters.com/techniques