[-- Attachment #1: Type: text/plain, Size: 805 bytes --] Hello people I'm configuring my linux to use s6-rc. It works fairly well so far. One thing I want to improve though, is that the early logger logs to /run/uncaught-logs. It's nice to have a log file during early boot and it helped my debugging easier. But it would be cool to able to change the location to a permanent location like /var/log after the root has been remounted as read-write. Is it possible to update the log path of early/catch-all s6-log process to a new location, and perhaps copying the early logs there as well? Or if not, is it possible to spawn a new s6-log process that acts as a catch-all logger? Thanks BTW, is there a command to restart service managed by s6-rc? I've been using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there might be something simpler.
>I'm configuring my linux to use s6-rc. It works fairly well so far. One >thing I want to improve though, is that the early logger logs to >/run/uncaught-logs. It's nice to have a log file during early boot and it >helped my debugging easier. But it would be cool to able to change the >location to a permanent location like /var/log after the root has been >remounted as read-write. Well the catch-all logger is supposed to be just that: a catch-all logger. That means that ideally, every service should have its own dedicated logger, typically writing to /var/log/something, and the catch-all logger is only used for exceptional stuff such as error messages from the supervision tree, so it doesn't matter that its logs are stored under /run. >Is it possible to update the log path of early/catch-all s6-log process to >a new location, and perhaps copying the early logs there as well? Or if >not, is it possible to spawn a new s6-log process that acts as a catch-all >logger? It's difficult to do. It's possible in theory: you could have a oneshot that modifies /run/service/s6-svscan-log/run, replacing the "/run/uncaught-logs" string with the new location you want, then copies /run/uncaught-logs into that new location and restarting the s6-svscan-log service. But in practice you would add complexity to the log infrastructure, and you need to pay close attention to all the failure cases because you don't want to have a broken catch-all logger for any reason at all. I don't think the benefits are worth the additional effort; but feel free to disagree and write such a "log mover" service. >BTW, is there a command to restart service managed by s6-rc? I've been >using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there >might be something simpler. For oneshots, no - but you rarely want to "restart" oneshots. For longruns, unless you want to also restart everything that depends on the service, you can just bypass the s6-rc layer and directly tell s6 to restart your process: s6-svc -r /run/service/$sv Hope this helps, -- Laurent
[-- Attachment #1: Type: text/plain, Size: 1360 bytes --] > > Well the catch-all logger is supposed to be just that: a catch-all > logger. That means that ideally, every service should have its own > dedicated logger, typically writing to /var/log/something, and the > catch-all logger is only used for exceptional stuff such as error > messages from the supervision tree, so it doesn't matter that its logs > are stored under /run. > So the best practice is that every (long run) services should have a "producer-for" and an extra "service-log" service? Sounds like a lot of repeating works... perhaps I can use some symblink to do that. It's difficult to do. It's possible in theory: you could have a oneshot > that modifies /run/service/s6-svscan-log/run, replacing the > "/run/uncaught-logs" string with the new location you want, then > copies /run/uncaught-logs into that new location and restarting the > s6-svscan-log service. > Yeah I don't think I'll go that route. Maybe it'll be easier to somehow mount root as rw before the logger starts (if it's actually easier), or just boot with rw root if I really need those logs to be saved. For longruns, unless you want to also restart everything that depends > on the service, you can just bypass the s6-rc layer and directly tell s6 > to restart your process: s6-svc -r /run/service/$sv > Thanks! I forgot that s6-rc actually runs on top of s6-supervice :)
On Sat, Sep 05, 2020 at 11:46:44AM -0400, Rio Liu wrote: > I'm configuring my linux to use s6-rc. It works fairly well so far. One > thing I want to improve though, is that the early logger logs to > /run/uncaught-logs. It's nice to have a log file during early boot and it > helped my debugging easier. But it would be cool to able to change the > location to a permanent location like /var/log after the root has been > remounted as read-write. > > Is it possible to update the log path of early/catch-all s6-log process to > a new location, and perhaps copying the early logs there as well? Or if > not, is it possible to spawn a new s6-log process that acts as a catch-all > logger? Your requirements of saving logs from /run/uncaught-logs and (mentioned later) batch-producing `producer-for'/`consumer-for' files can be automated using slew: <https://gitea.com/CasperVector/slew>. (Other encapsulations of s6/s6-rc also exist, but I am the author of slew ;) -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2022.09.20) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
On 9/5/20 3:26 PM, Rio Liu wrote: >> >> Well the catch-all logger is supposed to be just that: a catch-all >> logger. That means that ideally, every service should have its own >> dedicated logger, typically writing to /var/log/something, and the >> catch-all logger is only used for exceptional stuff such as error >> messages from the supervision tree, so it doesn't matter that its logs >> are stored under /run. >> > So the best practice is that every (long run) services should have a > "producer-for" and an extra "service-log" service? Sounds like a lot of > repeating works... perhaps I can use some symblink to do that. Yes, or you can preprocess your service directories. For example, the script I use looks for a "logdir" file containing the log directory path, and uses it to generate the pipeline and logger service: https://github.com/smaeul/rc/blob/113e9623/update#L48 >> It's difficult to do. It's possible in theory: you could have a oneshot >> that modifies /run/service/s6-svscan-log/run, replacing the >> "/run/uncaught-logs" string with the new location you want, then >> copies /run/uncaught-logs into that new location and restarting the >> s6-svscan-log service. >> > > Yeah I don't think I'll go that route. Maybe it'll be easier to somehow > mount root as rw before the logger starts (if it's actually easier), or > just boot with rw root if I really need those logs to be saved. There are a couple of other options I can think of: 1) Bring up a second service reading from the same /run/service/s6-svscan-log/fifo pipe, and once the new logger is ready, bring down the old s6-svscan-log service (from a oneshot depending on the new logger). During the transition, log lines could go to either service, but nothing would be lost. 2) Keep logging to /run/uncaught-logs, but also copy the logs to /var in a way that doesn't interfere with the s6-svscan-log service. This could be running `tail -f` on /run/uncaught-logs/current, or periodically `rsync`ing the whole directory to /var.
The way I understand the importance of the catch-all log is to be able to debug the system that fails to boot. It is writing a log in /run even before a real filesystem is mounted. At this point the system should give access to this file even if root is not mounted as rw yet. There may be many ways to have this copied to a root fs, which means it is already mounted rw, and by this time it seems as the system doesn't have a problem booting, so this log is of little use to getting it to boot. In 66 this is written as /run/66/log/0/current, and I have user/root copy it after a shell is executed, this helps when I would change something and it is not booting right on the next boot, I can compare the before after. Part of 66 booting procedure is to activate tty12 as early as possible, instead of the insecure sulogin that appears in runit. From tty12 you can read that log and mount things manually and change/fix what is wrong. You can also decrease/increase verbosity in that log. tty12 is a special setup in 66 where root can't login, only a user can, a security measure I find as a great idea. If tty12 isn't accessible, that means the uncaught-log isn't, and I don't think there is anything in the system that could have copied that log into something else. So, if you can have it copied and saved is usually when you don't need it, but when you do need it it can't be copied. I hope this makes sense to someone other than me :)
[-- Attachment #1: Type: text/plain, Size: 1603 bytes --] Thanks for all the replies :) > Your requirements of saving logs from /run/uncaught-logs and (mentioned > later) batch-producing `producer-for'/`consumer-for' files can be > automated using slew: <https://gitea.com/CasperVector/slew>. (Other > encapsulations of s6/s6-rc also exist, but I am the author of slew ;) > Yeah I'll need to look around and take inspirations from all the s6-rc wrapper/scripts. Thanks for sharing. 1) Bring up a second service reading from the same > /run/service/s6-svscan-log/fifo pipe, and once the new logger is ready, > bring > down the old s6-svscan-log service (from a oneshot depending on the new > logger). > During the transition, log lines could go to either service, but nothing > would > be lost. I like the idea of reading /run/services/s6-svscan-log/fifo from a new logger instance, I might try it and see how it works. In 66 this is written as /run/66/log/0/current, and I have user/root > copy it after a shell is executed, this helps when I would change > something and it is not booting right on the next boot, I can compare > the before after. Part of 66 booting procedure is to activate tty12 as > early as possible, instead of the insecure sulogin that appears in > runit. From tty12 you can read that log and mount things manually and > change/fix what is wrong. You can also decrease/increase verbosity in > that log. tty12 is a special setup in 66 where root can't login, only a > user can, a security measure I find as a great idea. > Have verbose log in tty12 is a great idea, I'll see how 66 does it and maybe implement it in my setup.