Update early logger logging path after remounting root as rw

Update early logger logging path after remounting root as rw

[Rio Liu]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hello people

I'm configuring my linux to use s6-rc. It works fairly well so far. One
thing I want to improve though, is that the early logger logs to
/run/uncaught-logs. It's nice to have a log file during early boot and it
helped my debugging easier. But it would be cool to able to change the
location to a permanent location like /var/log after the root has been
remounted as read-write.

Is it possible to update the log path of early/catch-all s6-log process to
a new location, and perhaps copying the early logs there as well? Or if
not, is it possible to spawn a new s6-log process that acts as a catch-all
logger?

Thanks

BTW, is there a command to restart service managed by s6-rc? I've been
using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there
might be something simpler.

Re: Update early logger logging path after remounting root as rw

[Laurent Bercot]

>I'm configuring my linux to use s6-rc. It works fairly well so far. One
>thing I want to improve though, is that the early logger logs to
>/run/uncaught-logs. It's nice to have a log file during early boot and it
>helped my debugging easier. But it would be cool to able to change the
>location to a permanent location like /var/log after the root has been
>remounted as read-write.

  Well the catch-all logger is supposed to be just that: a catch-all
logger. That means that ideally, every service should have its own
dedicated logger, typically writing to /var/log/something, and the
catch-all logger is only used for exceptional stuff such as error
messages from the supervision tree, so it doesn't matter that its logs
are stored under /run.


>Is it possible to update the log path of early/catch-all s6-log process to
>a new location, and perhaps copying the early logs there as well? Or if
>not, is it possible to spawn a new s6-log process that acts as a catch-all
>logger?

  It's difficult to do. It's possible in theory: you could have a oneshot
that modifies /run/service/s6-svscan-log/run, replacing the
"/run/uncaught-logs" string with the new location you want, then
copies /run/uncaught-logs into that new location and restarting the
s6-svscan-log service. But in practice you would add complexity to the
log infrastructure, and you need to pay close attention to all the
failure cases because you don't want to have a broken catch-all logger
for any reason at all. I don't think the benefits are worth the
additional effort; but feel free to disagree and write such a "log
mover" service.


>BTW, is there a command to restart service managed by s6-rc? I've been
>using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there
>might be something simpler.

  For oneshots, no - but you rarely want to "restart" oneshots.
  For longruns, unless you want to also restart everything that depends
on the service, you can just bypass the s6-rc layer and directly tell s6
to restart your process: s6-svc -r /run/service/$sv

  Hope this helps,

--
  Laurent

Re: Update early logger logging path after remounting root as rw

[Rio Liu]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

>
> Well the catch-all logger is supposed to be just that: a catch-all
> logger. That means that ideally, every service should have its own
> dedicated logger, typically writing to /var/log/something, and the
> catch-all logger is only used for exceptional stuff such as error
> messages from the supervision tree, so it doesn't matter that its logs
> are stored under /run.
>
So the best practice is that every (long run) services should have a
"producer-for" and an extra "service-log" service? Sounds like a lot of
repeating works... perhaps I can use some symblink to do that.

It's difficult to do. It's possible in theory: you could have a oneshot
> that modifies /run/service/s6-svscan-log/run, replacing the
> "/run/uncaught-logs" string with the new location you want, then
> copies /run/uncaught-logs into that new location and restarting the
> s6-svscan-log service.
>

Yeah I don't think I'll go that route. Maybe it'll be easier to somehow
mount root as rw before the logger starts (if it's actually easier), or
just boot with rw root if I really need those logs to be saved.

For longruns, unless you want to also restart everything that depends
> on the service, you can just bypass the s6-rc layer and directly tell s6
> to restart your process: s6-svc -r /run/service/$sv
>

Thanks! I forgot that s6-rc actually runs on top of s6-supervice :)

Re: Update early logger logging path after remounting root as rw

[Samuel Holland]

On 9/5/20 3:26 PM, Rio Liu wrote:
>>
>> Well the catch-all logger is supposed to be just that: a catch-all
>> logger. That means that ideally, every service should have its own
>> dedicated logger, typically writing to /var/log/something, and the
>> catch-all logger is only used for exceptional stuff such as error
>> messages from the supervision tree, so it doesn't matter that its logs
>> are stored under /run.
>>
> So the best practice is that every (long run) services should have a
> "producer-for" and an extra "service-log" service? Sounds like a lot of
> repeating works... perhaps I can use some symblink to do that.

Yes, or you can preprocess your service directories. For example, the script I
use looks for a "logdir" file containing the log directory path, and uses it to
generate the pipeline and logger service:

https://github.com/smaeul/rc/blob/113e9623/update#L48

>> It's difficult to do. It's possible in theory: you could have a oneshot
>> that modifies /run/service/s6-svscan-log/run, replacing the
>> "/run/uncaught-logs" string with the new location you want, then
>> copies /run/uncaught-logs into that new location and restarting the
>> s6-svscan-log service.
>>
> 
> Yeah I don't think I'll go that route. Maybe it'll be easier to somehow
> mount root as rw before the logger starts (if it's actually easier), or
> just boot with rw root if I really need those logs to be saved.

There are a couple of other options I can think of:

1) Bring up a second service reading from the same
/run/service/s6-svscan-log/fifo pipe, and once the new logger is ready, bring
down the old s6-svscan-log service (from a oneshot depending on the new logger).
During the transition, log lines could go to either service, but nothing would
be lost.

2) Keep logging to /run/uncaught-logs, but also copy the logs to /var in a way
that doesn't interfere with the s6-svscan-log service. This could be running
`tail -f` on /run/uncaught-logs/current, or periodically `rsync`ing the whole
directory to /var.

Re: Update early logger logging path after remounting root as rw

[yianiris]

The way I understand the importance of the catch-all log is to be able
to debug the system that fails to boot.  It is writing a log in /run
even before a real filesystem is mounted.  At this point the system
should give access to this file even if root is not mounted as rw yet.
There may be many ways to have this copied to a root fs, which means it
is already mounted rw, and by this time it seems as the system doesn't
have a problem booting, so this log is of little use to getting it to boot.

In 66 this is written as /run/66/log/0/current, and I have user/root
copy it after a shell is executed, this helps when I would change
something and it is not booting right on the next boot, I can compare
the before after.  Part of 66 booting procedure is to activate tty12 as
early as possible, instead of the insecure sulogin that appears in
runit.  From tty12 you can read that log and mount things manually and
change/fix what is wrong.  You can also decrease/increase verbosity in
that log.  tty12 is a special setup in 66 where root can't login, only a
user can, a security measure I find as a great idea.

If tty12 isn't accessible, that means the uncaught-log isn't, and I
don't think there is anything in the system that could have copied that
log into something else.  So, if you can have it copied and saved is
usually when you don't need it, but when you do need it it can't be
copied.  I hope this makes sense to someone other than me :)

Re: Update early logger logging path after remounting root as rw

[Rio Liu]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

Thanks for all the replies :)


> Your requirements of saving logs from /run/uncaught-logs and (mentioned
> later) batch-producing `producer-for'/`consumer-for' files can be
> automated using slew: <https://gitea.com/CasperVector/slew>.  (Other
> encapsulations of s6/s6-rc also exist, but I am the author of slew ;)
>

Yeah I'll need to look around and take inspirations from all the s6-rc
wrapper/scripts. Thanks for sharing.

1) Bring up a second service reading from the same
> /run/service/s6-svscan-log/fifo pipe, and once the new logger is ready,
> bring
> down the old s6-svscan-log service (from a oneshot depending on the new
> logger).
> During the transition, log lines could go to either service, but nothing
> would
> be lost.


I like the idea of reading /run/services/s6-svscan-log/fifo from a new
logger instance, I might try it and see how it works.

In 66 this is written as /run/66/log/0/current, and I have user/root
> copy it after a shell is executed, this helps when I would change
> something and it is not booting right on the next boot, I can compare
> the before after.  Part of 66 booting procedure is to activate tty12 as
> early as possible, instead of the insecure sulogin that appears in
> runit.  From tty12 you can read that log and mount things manually and
> change/fix what is wrong.  You can also decrease/increase verbosity in
> that log.  tty12 is a special setup in 66 where root can't login, only a
> user can, a security measure I find as a great idea.
>

Have verbose log in tty12 is a great idea, I'll see how 66 does it and
maybe implement it in my setup.

Re: Update early logger logging path after remounting root as rw

[Casper Ti. Vector]

On Sat, Sep 05, 2020 at 11:46:44AM -0400, Rio Liu wrote:
> I'm configuring my linux to use s6-rc. It works fairly well so far. One
> thing I want to improve though, is that the early logger logs to
> /run/uncaught-logs. It's nice to have a log file during early boot and it
> helped my debugging easier. But it would be cool to able to change the
> location to a permanent location like /var/log after the root has been
> remounted as read-write.
> 
> Is it possible to update the log path of early/catch-all s6-log process to
> a new location, and perhaps copying the early logs there as well? Or if
> not, is it possible to spawn a new s6-log process that acts as a catch-all
> logger?

Your requirements of saving logs from /run/uncaught-logs and (mentioned
later) batch-producing `producer-for'/`consumer-for' files can be
automated using slew: <https://gitea.com/CasperVector/slew>.  (Other
encapsulations of s6/s6-rc also exist, but I am the author of slew ;)

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2022.09.20)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C