Re: Logging in a web server context

[Colin Booth <colin@heliocat.net>]

On Sat, Jun 13, 2020 at 10:41:24AM +0200, Carl Winbäck wrote:
> Hi folks!
> 
> Is it possible when using s6-svscan/s6-supervise to somehow arrange so
> that a daemon’s stdout is sent to one logdir and stderr to another
> logdir?
> 
It's doable if you can prefix the messages with something unique to each
channel and then write an s6-log logging script to detangle the two.
However web servers don't need this. The proper solution is easier with
apache but you can do something similar with nginx.
>
> Or must I use s6-rc in order to achieve this?
>
With apache no, with nginx also no but s6-rc will handle some things for
you that otherwise will be a pain in the butt.
> 
> My premises are the following:
> 
> I want to run nginx under s6, on a mainstream Linux distro such as
> Debian/Ubuntu/CentOS. My plan is currently to use s6 only for nginx
> and not any other daemons.
> 
> I am very interested to hear from others on the list what your
> thoughts are about this scenario.
> 
Web servers will only send messages along their internal logging
channels if they don't have a better place to send them. With apache
this is controlled via the CustomLog and ErrorLog directives, and with
nginx via error_log and access_log.

For apache, you can declare a pipe logger
(CustomLog="|/path/to/logger/prog options" format) and apache itself
will supervise the access and error loggers directly. It is very
convenient and assuming the loggers do the right thing on recept of EOF
you never end up with stale logging endpoints.

With nginx you can do something similar but have to use named pipes. The
documentation doesn't go into detail about it, but if you have a fifo
and tell nginx to use that as its logging target it will log down that
pipe and into the pipes reader. Since nginx doesn't know it's writing to
a fifo you need to make sure it exists before starting nginx.

The supervision way of doing this would be to:
create two services that each guarantee the existence of their fifo,
    (creating it if necessary and using an already existing fifo if it's
    there) then redirecting the read end of that fifo into a copy of
    s6-log
create a service to check for the presence of the fifos and, assuming
    they are there, exec into nginx

I don't believe a service error logger (nginx/log) is necessary because
the error_log directive should handle all of it but if you want to deal
with logs from catestrophic events (script errors, nginx crashes, etc)
then you probably want it.

You can do all of that sequencing manually or you can have s6-rc do it.
>
> The reason I’m weary about s6-rc is that I think it might not play so
> well with the distros I mentioned earlier. If this concern is not
> correct, please let me know.
> 
Speaking of s6-rc, s6-rc can manage a subordinate supervision tree
without touching the rest of the system. To handle what I described
above, you'll need to install nginx, install s6 and s6-rc, disable
nginx, create an s6-rc bundle that contains nginx and the two output
logging services, and then create a service unit of Type=oneshot for
systemd that runs s6-rc-init and s6-rc -u change nginx once the
supervision tree is fired up. The naming scheme I would use us:
nginx-srv (main nginx service, producer-for nginx-log, depends
    nginx-access and nginx-error)
nginx-log (catestrophic event logger, consumer-for nginx-srv,
    pipeline-name nginx)
nginx-access (nginx access logger, MUST SIGNAL READINESS)
nginx-error (nginx error logger, MUST SIGNAL READINESS)

Assuming the dependencies are set correctly and s6-log is triggering
readiness notifications via the -d option, telling s6-rc to start the
nginx implicit bundle (defined in nginx-log) will bring up the primary
loggers (due to nginx-srv needing them) and nginx-log, then finally
nginx-srv itself, guaranteeing that the fifos exist and have a reader
before trying to start the web server itself. In this setup you can skip
the bail-out check for the fifos in nginx-srv, but it's a pretty cheap
check so you might as well keep it as an extra safety.
> 
> [1] An alternative to collecting access logs could be to use Google
> Analytics instead, but I want to avoid that since I don’t feel
> comfortable about handing over my visitors’ personal data over to
> Google.
> 
> [2] I.e. lines like this '10.131.214.101 - - [20/Nov/2017:18:52:17
> +0000] "GET / HTTP/1.1" 401 188 "-" "Mozilla/5.0 (X11; Ubuntu; Linux
> x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"'

-- 
Colin Booth