supervision - discussion about system services, daemon supervision, init, runlevel management, and tools such as s6 and runit
 help / color / mirror / Atom feed
From: Ben Franksen <ben.franksen@online.de>
To: supervision@list.skarnet.org
Subject: Re: logging services with shell interaction
Date: Wed, 20 Oct 2021 09:53:58 +0200	[thread overview]
Message-ID: <skohum$a4l$1@ciao.gmane.io> (raw)
In-Reply-To: <emba464994-2a6f-4713-af9e-68f4e93b785a@elzian>

Am 20.10.21 um 01:27 schrieb Laurent Bercot:
>> we have a fair number of services which allow (and occasionally 
>> require) user interaction via a (built-in) shell. All the shell 
>> interaction is supposed to be logged, in addition to all the messages 
>> that are issued spontaneously by the process. So we cannot directly 
>> use a logger attached to the stdout/stderr of the process.
> 
>   I don't understand the consequence relationship here.
> >   - If you control your services / builtin shells, the services could
> have an option to log the IO of their shells to stderr, as well as
> their own messages.

We do have control over them, theoretically, but adding this 
functionality seems impractical. This is a complex piece of software, 
built from multiple components maintained by different parties. There is 
some sort of common framework for issuing messages but none of the 
components strictly adhere to it. In other words, they use things like 
printf all over the place. The only way I see to reliably get all the IO 
for logging is to delegate this to an external process.

>   - Even if you cannot make the services log the shell IO, you can add
> a small data dumper in front of the service's shell, which transmits
> full-duplex everything it gets but also writes it to its own stdout or
> stderr; if that stdout/err is the same pipe as the stdout/err of your
> service, then all the IO from the shell will be logged to the same place
> (and log lines won't be mixed unless they're more than PIPE_BUF bytes
> long, which shouldn't happen in practice). So with that solution you
> could definitely make your services log to multilog.

Yes, that would be possible. More or less what procServ does minus the 
supervision aspect.

>> IOC=$1
>>
>> /usr/bin/procServ -f -L- --logstamp --timefmt="$TIMEFMT" \
>>  -q -n %i --ignore=^D^C^] -P "unix:$RUNDIR/$IOC" -c "$BOOTDIR" 
>> "./$STCMD" \
>>  | /usr/bin/multilog "s$LOGSIZE" "n$LOGNUM" "$LOGDIR/$IOC"
>> ```
>>
>> So far this seems to do the job, but I have two questions:
>>
>> 1. Is there anything "bad" about this approach? Most supervision tools 
>> have this sort of thing as a built-in feature and I suspect there may 
>> be a reason for that other than mere convenience.
> 
>   It's not *bad*, it's just not as airtight as supervision suites make
> it. The reasons why it's a built-in feature in daemontools/runit/s6/others
> are:
>   - it allows the logger process to be supervised as well
>   - it maintains open the pipe to the logger, so service and logger can
> be restarted independently at will, without risk of losing logs.
> 
>   As is, you can't send signals to multilog (useful if you want to force
> a rotation) without knowing its pid. And if multilog dies, it broken
> pipes procServ, and it (and your service) is probably forced to restart,
> and you lose the data that it wanted to write.
>   A supervision architecture with integrated logging protects from this.

Thanks, this answers my question perfectly.

>> 2. Do any of the existing process supervision tools support what 
>> procServ gives us wrt interactive shell access from outside?
> 
>   Not that I know of, because that need is pretty specific to your
> service architecture.

It sure is.

>   However, unless there are more details you have omitted, I still
> believe you could obtain the same functionality with a daemontools/etc.
> infrastructure and a program recording the IO from/to the shell. Since
> you don't seem opposed to using old djb programs, you could probably
> even directly reuse "recordio" from ucspi-tcp for this. :)

Interesting, I didn't know about recordio, will take a look.

Again, thanks a lot for the detailed response!

Cheers
Ben
-- 
I would rather have questions that cannot be answered, than answers that
cannot be questioned.  -- Richard Feynman



  reply	other threads:[~2021-10-20  7:54 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-10-19  8:59 Ben Franksen
2021-10-19 23:27 ` Laurent Bercot
2021-10-20  7:53   ` Ben Franksen [this message]
2021-10-20 18:01     ` Casper Ti. Vector
2021-10-23 15:48       ` Ben Franksen
2021-10-23 16:40         ` Casper Ti. Vector
2021-10-24 20:36           ` Ben Franksen
2022-04-22 10:40             ` Casper Ti. Vector
2023-06-22 17:16       ` Casper Ti. Vector
2023-06-23 11:48         ` Ben Franksen
2023-06-23 12:30           ` Casper Ti. Vector

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='skohum$a4l$1@ciao.gmane.io' \
    --to=ben.franksen@online.de \
    --cc=supervision@list.skarnet.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).