supervision - discussion about system services, daemon supervision, init, runlevel management, and tools such as s6 and runit
 help / color / mirror / Atom feed
* Understanding the syslogd-linux Service Script
@ 2020-09-08 16:53 Scott Colby
  2020-09-08 18:15 ` Colin Booth
  2020-09-08 18:43 ` Colin Booth
  0 siblings, 2 replies; 8+ messages in thread
From: Scott Colby @ 2020-09-08 16:53 UTC (permalink / raw)
  To: supervision

Hello,

I am faced with running a program in a container that will only log
to syslog and cannot be configured otherwise. I am looking to using
s6 within the container to supervise this program and some
implementation of syslog. I thought that there must be something
simpler than rsyslog or syslog-ng, and my investigations led me to
the s6/examples/syslogd-linux service directory.

I am only slightly experienced with writing execline scripts and
would like to better understand exactly what each line in the example
run script is doing. Here it is, annotated with my understanding
and questions.

#!/command/execlineb -P
# Redirects stderr to stdout, but why is this necessary?
fdmove -c 2 1
# Clears the environment, I assume for general
# security/isolation/cleanliness reasons?
exec -c
# Prepares for setting uid/gid later
s6-envuidgid nobody
# Redirects stdout to fd 3, I think because s6-ipcserver closes fd
# 1; what happens to things sent to fd 3?
# Also, why is the -c option not used here?
fdmove 1 3
# Listens on /dev/log, this makes sense to me
s6-ipcserver -U -1 -- /dev/log
# Redirects stdout to stderr, because this is where log messages
# are expected to go
fdmove -c 1 2
# writes stdin to stdout with the values of the remote UID and GID
# prepended, plus whatever other functionality of ucspidlogd
ucspilogd IPCREMOTEEUID IPCREMOTEEGID

Please let me know if I have made any mistakes in my annotation and
what the answers to my questions are.

Thank you,
Scott

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-08 16:53 Understanding the syslogd-linux Service Script Scott Colby
@ 2020-09-08 18:15 ` Colin Booth
  2020-09-08 18:43 ` Colin Booth
  1 sibling, 0 replies; 8+ messages in thread
From: Colin Booth @ 2020-09-08 18:15 UTC (permalink / raw)
  To: Scott Colby; +Cc: supervision

On Tue, Sep 08, 2020 at 12:53:37PM -0400, Scott Colby wrote:
> Hello,
> 
> I am faced with running a program in a container that will only log
> to syslog and cannot be configured otherwise. I am looking to using
> s6 within the container to supervise this program and some
> implementation of syslog. I thought that there must be something
> simpler than rsyslog or syslog-ng, and my investigations led me to
> the s6/examples/syslogd-linux service directory.
> 
> I am only slightly experienced with writing execline scripts and
> would like to better understand exactly what each line in the example
> run script is doing. Here it is, annotated with my understanding
> and questions.
>
Responses inline.
> 
> #!/command/execlineb -P
> # Redirects stderr to stdout, but why is this necessary?
> fdmove -c 2 1
This is necessary because by default only stdout goes to the appendant
logger managed by s6-supervise. stderr is forwarded up to s6-svscan (and
the catch-all logger). It is also needed to prepare for stuff later on
in the script. Semanticaly this is actually pointing fd2 at the target
of fd1.
> # Clears the environment, I assume for general
> # security/isolation/cleanliness reasons?
> exec -c
Yup. Mostly cleanup, a little security.
> # Prepares for setting uid/gid later
> s6-envuidgid nobody
> # Redirects stdout to fd 3, I think because s6-ipcserver closes fd
> # 1; what happens to things sent to fd 3?
> # Also, why is the -c option not used here?
> fdmove 1 3
Welcome to Being Confused By Shell Redirection (don't worry, it happens
to everyone). fdmove 1 3 does not redirecct stdout to fd3, it points fd1
to fd3's target. Omitting the -c means that fd3 is closed after we
rewrite fd1's target. We close fd3 after this point because we don't
need it anymore.
> # Listens on /dev/log, this makes sense to me
> s6-ipcserver -U -1 -- /dev/log
> # Redirects stdout to stderr, because this is where log messages
> # are expected to go
> fdmove -c 1 2
s6-ipcserver -1 signals readyness on fd 1 and then closes it. This
re-opens fd1 (pointed at fd2's target, which was fd1's original target
at the start of the script) before launching ucspilogd log handlers.
This is needed because ucspilogd reads syslog type messages on stdin
(fd0) and writes the processed output on stdout (fd1). Note that this
happens individually for every new message flow established by
s6-ipcserver and not within the main control flow of the program, though
this doesn't matter here since the last thing the main program does is
establish an s6-ipcserver listener on /dev/log.
> # writes stdin to stdout with the values of the remote UID and GID
> # prepended, plus whatever other functionality of ucspidlogd
> ucspilogd IPCREMOTEEUID IPCREMOTEEGID
> 
> Please let me know if I have made any mistakes in my annotation and
> what the answers to my questions are.
>
The fully annotated flow of the script is:
start:
fd1 open, pointed at s6-log; fd2 open, pointed at catch-all; fd3 open,
pointed at s6-supervise readyness reader
fdmove -c 2 1:
fd1 open, pointed at s6-log; fd2 open, pointed at s6-log; fd3 open,
pointed at s6-supervise readyness reader
fdmove 1 3:
fd1 open, pointed at s6-supervise readyness reader; fd2 open, pointed at
s6-log; fd3 closed
s6-ipcserver -U -1 -- /dev/log
fd1 signaled ready and then closed; fd2 open, pointed at s6-log; fd3 closed
--- for each new sender attached to /dev/log ---
fd0 open, reading from /dev/log, fd1 open, writing to s6-log, fd2 open, writing to s6-log, fd3 closed.

Cheers!
-- 
Colin Booth

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-08 16:53 Understanding the syslogd-linux Service Script Scott Colby
  2020-09-08 18:15 ` Colin Booth
@ 2020-09-08 18:43 ` Colin Booth
  2020-09-09  8:01   ` Laurent Bercot
  1 sibling, 1 reply; 8+ messages in thread
From: Colin Booth @ 2020-09-08 18:43 UTC (permalink / raw)
  To: Scott Colby; +Cc: supervision

On Tue, Sep 08, 2020 at 12:53:37PM -0400, Scott Colby wrote:
> # Listens on /dev/log, this makes sense to me
> s6-ipcserver -U -1 -- /dev/log
>
One thing that I forgot to mention here is that depending on the age of
the libc this might not work properly as written. Newer libc's use
datagram connections for syslog and because of that you need to
hand-write the entire s6-ipcserver chain (since the s6-ipcserver
entrypoint doesn't currently support passing the SOCK_DGRAM option to
s6-ipcserver-socketbinder.

If you need to write it by hand, you should change that line to:
s6-ipcserver-socetbinder -m -b0 /dev/log
s6-applyuidgid -U -z
s6-ipcserverd -1 

then the rest of the program (the fdmove and ucspilogd commands).

-- 
Colin Booth

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-08 18:43 ` Colin Booth
@ 2020-09-09  8:01   ` Laurent Bercot
  2020-09-12  6:42     ` Scott Colby
  0 siblings, 1 reply; 8+ messages in thread
From: Laurent Bercot @ 2020-09-09  8:01 UTC (permalink / raw)
  To: supervision

>One thing that I forgot to mention here is that depending on the age of
>the libc this might not work properly as written. Newer libc's use
>datagram connections for syslog and because of that you need to
>hand-write the entire s6-ipcserver chain (since the s6-ipcserver
>entrypoint doesn't currently support passing the SOCK_DGRAM option to
>s6-ipcserver-socketbinder.

  The syslogd-linux example is a pretty old one, I should probably
rewrite it. It will work as is with glibc, because glibc's syslog()
has a SOCK_STREAM fallback, so the s6-ipcserver chain is fine.

  It will *not* work as is with musl, because musl's syslog() only
opens a SOCK_DGRAM socket. In that case, I recommend using socklog
instead: http://smarden.org/socklog/
and the run script would just be: "fdmove -c 2 1 fdclose 0 socklog"
(you lose readiness notification but that's not a big problem for
socklog, which listens to /dev/log very fast, and the worst case is
the first log lines are sent to /dev/console)

  Ultimately the syslogd service just needs to listen to the /dev/log
socket (as SOCK_DGRAM, as socklog does, or SOCK_STREAM, as s6-ipcserver
does) and translate the loglevel/priority in the log lines to human-
readable text and print them to stdout (as socklog or ucspilogd does).
The real work is done by the logger part, which is a s6-log process
that reads the translated log lines, matches them against some regexes
and logs them to various logdirs depending on what they match. The
s6-log script is the equivalent of syslogd.conf: that's where you
configure how to dispatch the syslog into different places.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-09  8:01   ` Laurent Bercot
@ 2020-09-12  6:42     ` Scott Colby
  2020-09-12 10:08       ` Laurent Bercot
  2020-09-12 10:14       ` Laurent Bercot
  0 siblings, 2 replies; 8+ messages in thread
From: Scott Colby @ 2020-09-12  6:42 UTC (permalink / raw)
  To: supervision

Dear Colin and Laurent,

Thank you for the thorough explanations it took me a minute with a
pen and paper to work out the fd movements, but it makes sense to
me now.

My particular environment uses glibc, so the old version of the
example seems to work well. I omit the syslog service's log directory
and the log lines appear on my container's stdout, which was my
goal.

I tried out Colin's SOCK_DGRAM version, but this was unsuccessful;
I received this line at a very high rate (20k lines in just a second
or two) on my stdout:

s6-ipcserverd: warning: unable to accept: Operation not supported

The s6-ipcserverd docs specifically state that it expects a bound
and listening SOCK_STREAM socket, and this led me to the `ipc_accept()`
call that the program makes. Out of curiosity, is there another
s6-family program that handles SOCK_DGRAM sockets? Otherwise, I'll
look in to socklog as Laurent suggested.

Thanks,
Scott

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-12  6:42     ` Scott Colby
@ 2020-09-12 10:08       ` Laurent Bercot
  2020-09-12 10:14       ` Laurent Bercot
  1 sibling, 0 replies; 8+ messages in thread
From: Laurent Bercot @ 2020-09-12 10:08 UTC (permalink / raw)
  To: supervision


>The s6-ipcserverd docs specifically state that it expects a bound
>and listening SOCK_STREAM socket, and this led me to the `ipc_accept()`
>call that the program makes. Out of curiosity, is there another
>s6-family program that handles SOCK_DGRAM sockets? Otherwise, I'll
>look in to socklog as Laurent suggested.

  There is not, because the UCSPI model - the one that s6-ipcserver
implements - is about handling data *streams*, and forking a child to
handle each separate client (as the venerable inetd does).

  There are workarounds and hacks to handle datagram sequences with a
similar architecture, but it's never perfect: clients have to connect
(which an additional restriction), you lose natural message boundaries,
and you don't have an explicit EOF so the child needs to die on a
timeout - which makes the thing awkward and actually *adds* complexity,
when the whole point of UCSPI is to make every component as simple as
possible.

  When you need to listen to a /dev/log datagram socket, socklog is
definitely the simplest, and correct, approach.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-12  6:42     ` Scott Colby
  2020-09-12 10:08       ` Laurent Bercot
@ 2020-09-12 10:14       ` Laurent Bercot
  2020-09-12 17:59         ` Scott Colby
  1 sibling, 1 reply; 8+ messages in thread
From: Laurent Bercot @ 2020-09-12 10:14 UTC (permalink / raw)
  To: supervision

>My particular environment uses glibc, so the old version of the
>example seems to work well. I omit the syslog service's log directory
>and the log lines appear on my container's stdout, which was my
>goal.

  If your goal is to have syslog messages appear on your container's
stdout, then you should actually be able to get away with not having
any syslogd service at all, which is always the preferred solution!

  When syslog() can't find a server that listens on /dev/log, it writes
its message to /dev/console. In a container environment, /dev/console
by definition should be your container's stdout. So, not handling
syslog messages in your container at all should naturally make them
fall through to where you want them.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Understanding the syslogd-linux Service Script
  2020-09-12 10:14       ` Laurent Bercot
@ 2020-09-12 17:59         ` Scott Colby
  0 siblings, 0 replies; 8+ messages in thread
From: Scott Colby @ 2020-09-12 17:59 UTC (permalink / raw)
  To: supervision

>   There is not, because the UCSPI model - the one that s6-ipcserver
implements - is about handling data *streams*, and forking a child to
handle each separate client (as the venerable inetd does).

That makes sense to me.

>   If your goal is to have syslog messages appear on your container's
> stdout, then you should actually be able to get away with not having
> any syslogd service at all, which is always the preferred solution!

That is my goal and it's interesting that I didn't observe the logs
ending up on the container's stdout. I was considering that the
program I want to run (OpenDNSSEC, to be exact) does something funny
that would prevent this when I came across an undocumented configuration
option that allows logging to a file or stdout instead of using
syslog. I have to sigh, for this is not the first time I've had to
read the source of this program in order to figure out what is going
on.

In any case, thank you for all the explanation. I'm certain my
improved understanding of both s6 and syslog will come in handy in
the future.

Scott

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2020-09-12 18:00 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-08 16:53 Understanding the syslogd-linux Service Script Scott Colby
2020-09-08 18:15 ` Colin Booth
2020-09-08 18:43 ` Colin Booth
2020-09-09  8:01   ` Laurent Bercot
2020-09-12  6:42     ` Scott Colby
2020-09-12 10:08       ` Laurent Bercot
2020-09-12 10:14       ` Laurent Bercot
2020-09-12 17:59         ` Scott Colby

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).