From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 22258 invoked from network); 19 Oct 2021 23:27:50 -0000 Received: from alyss.skarnet.org (95.142.172.232) by inbox.vuxu.org with ESMTPUTF8; 19 Oct 2021 23:27:50 -0000 Received: (qmail 31722 invoked by uid 89); 19 Oct 2021 23:28:14 -0000 Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm Sender: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Received: (qmail 31715 invoked from network); 19 Oct 2021 23:28:14 -0000 From: "Laurent Bercot" To: "Ben Franksen" , supervision@list.skarnet.org Subject: Re: logging services with shell interaction Date: Tue, 19 Oct 2021 23:27:47 +0000 Message-Id: In-Reply-To: References: Reply-To: "Laurent Bercot" User-Agent: eM_Client/8.2.1659.0 Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: -100 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvtddrvddvfedgudelucetufdoteggodftvfcurfhrohhfihhlvgemucfpfgfogfftkfevteeunffgpdfqfgfvnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvffufffkjghfrhgfgggtgfesthhqredttderjeenucfhrhhomhepfdfnrghurhgvnhhtuceuvghrtghothdfuceoshhkrgdqshhuphgvrhhvihhsihhonhesshhkrghrnhgvthdrohhrgheqnecuggftrfgrthhtvghrnhepvdfgveffueelgedvkedtffetgedvieeifeektefgueehffehleehjefhveeuieejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmohguvgepshhmthhpohhuth >we have a fair number of services which allow (and occasionally require) u= ser interaction via a (built-in) shell. All the shell interaction is suppos= ed to be logged, in addition to all the messages that are issued spontaneou= sly by the process. So we cannot directly use a logger attached to the stdo= ut/stderr of the process. I don't understand the consequence relationship here. - If you control your services / builtin shells, the services could have an option to log the IO of their shells to stderr, as well as their own messages. - Even if you cannot make the services log the shell IO, you can add a small data dumper in front of the service's shell, which transmits full-duplex everything it gets but also writes it to its own stdout or stderr; if that stdout/err is the same pipe as the stdout/err of your service, then all the IO from the shell will be logged to the same place (and log lines won't be mixed unless they're more than PIPE_BUF bytes long, which shouldn't happen in practice). So with that solution you could definitely make your services log to multilog. >procServ is a process supervisor adapted to such situations. It allows an= external process (conserver in our case) to attach to the service's shell v= ia a TCP or UNIX domain socket. procServ supports logging everything it see= s (input and output) to a file or stdout. That works too. >IOC=3D$1 > >/usr/bin/procServ -f -L- --logstamp --timefmt=3D"$TIMEFMT" \ > -q -n %i --ignore=3D^D^C^] -P "unix:$RUNDIR/$IOC" -c "$BOOTDIR" "./$STCM= D" \ > | /usr/bin/multilog "s$LOGSIZE" "n$LOGNUM" "$LOGDIR/$IOC" >``` > >So far this seems to do the job, but I have two questions: > >1. Is there anything "bad" about this approach? Most supervision tools hav= e this sort of thing as a built-in feature and I suspect there may be a rea= son for that other than mere convenience. It's not *bad*, it's just not as airtight as supervision suites make it. The reasons why it's a built-in feature in=20 daemontools/runit/s6/others are: - it allows the logger process to be supervised as well - it maintains open the pipe to the logger, so service and logger can be restarted independently at will, without risk of losing logs. As is, you can't send signals to multilog (useful if you want to force a rotation) without knowing its pid. And if multilog dies, it broken pipes procServ, and it (and your service) is probably forced to restart, and you lose the data that it wanted to write. A supervision architecture with integrated logging protects from this. >2. Do any of the existing process supervision tools support what procServ= gives us wrt interactive shell access from outside? Not that I know of, because that need is pretty specific to your service architecture. However, unless there are more details you have omitted, I still believe you could obtain the same functionality with a daemontools/etc. infrastructure and a program recording the IO from/to the shell. Since you don't seem opposed to using old djb programs, you could probably even directly reuse "recordio" from ucspi-tcp for this. :) -- Laurent