* Following log directories @ 2020-06-25 9:27 Carl Winbäck 2020-06-25 13:20 ` Guillermo 0 siblings, 1 reply; 14+ messages in thread From: Carl Winbäck @ 2020-06-25 9:27 UTC (permalink / raw) To: supervision Hi folks! Does anyone know if the follow-log-directories tool[1] from the nosh suite is compatible with logs produced by s6-log? Best regards, Carl Winbäck a.k.a. cpach [1] https://jdebp.eu/Softwares/nosh/guide/commands/follow-log-directories.xml ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 9:27 Following log directories Carl Winbäck @ 2020-06-25 13:20 ` Guillermo 2020-06-25 14:52 ` Guillermo 0 siblings, 1 reply; 14+ messages in thread From: Guillermo @ 2020-06-25 13:20 UTC (permalink / raw) To: Supervision Hi, > Does anyone know if the follow-log-directories tool[1] from the nosh > suite is compatible with logs produced by s6-log? Let's find out. I'll test that. G. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 13:20 ` Guillermo @ 2020-06-25 14:52 ` Guillermo 2020-06-25 22:08 ` Carl Winbäck 0 siblings, 1 reply; 14+ messages in thread From: Guillermo @ 2020-06-25 14:52 UTC (permalink / raw) To: Supervision El jue., 25 jun. 2020 a las 10:20, Guillermo escribió: > > El jue., 25 jun. 2020 a las 6:27, Carl Winbäck escribió: > > > Does anyone know if the follow-log-directories tool[1] from the nosh > > suite is compatible with logs produced by s6-log? > > Let's find out. I'll test that. And it appears to be compatible, indeed, as long as s6-log is invoked with the 't' directive (prepend timestamp in TAI64N format), to match the logging file format expected by follow-log-directories, as stated in its documentation. Also, note that follow-log-directories is a daemon, so it will keep running and outputting log lines until killed, once it catches up with the logging directory. TAI64N timestamps are not decoded, so if you want them in human-readable format, pipe its standard output to s6-tai64nlocal. You might also want to separate follow-log-directories's standard output stream from its standard error stream, since the program is somewhat chatty during catch-ups and during logdir rotations. G. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 14:52 ` Guillermo @ 2020-06-25 22:08 ` Carl Winbäck 2020-06-25 22:32 ` John W Higgins 0 siblings, 1 reply; 14+ messages in thread From: Carl Winbäck @ 2020-06-25 22:08 UTC (permalink / raw) To: Guillermo; +Cc: Supervision On Thu, 25 Jun 2020 at 17:00, Guillermo wrote: > And it appears to be compatible, indeed, as long as s6-log is invoked > with the 't' directive (prepend timestamp in TAI64N format), to match > the logging file format expected by follow-log-directories, as stated > in its documentation. Neat! Thank you Guillermo for this information. Best regards, Carl ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 22:08 ` Carl Winbäck @ 2020-06-25 22:32 ` John W Higgins 2020-06-26 0:09 ` Laurent Bercot 2020-06-27 7:33 ` Jonathan de Boyne Pollard 0 siblings, 2 replies; 14+ messages in thread From: John W Higgins @ 2020-06-25 22:32 UTC (permalink / raw) To: Supervision [-- Attachment #1: Type: text/plain, Size: 953 bytes --] One thing to remember about tools like this is that s6 is very precise with it's implemenation of TAI64N - it uses leap seconds. I use lnav [1] for a similar concept and it parses TAI64N but it does not deal with leap seconds so my timestamps were off by like 20 seconds (or something similar). I had to modify the code to handle the leap seconds. So just make sure you check that the timestamps are what you expect them to be with non s6 tools. John [1] - http://lnav.org/ On Thu, Jun 25, 2020 at 3:08 PM Carl Winbäck <c@tunnel53.net> wrote: > On Thu, 25 Jun 2020 at 17:00, Guillermo wrote: > > And it appears to be compatible, indeed, as long as s6-log is invoked > > with the 't' directive (prepend timestamp in TAI64N format), to match > > the logging file format expected by follow-log-directories, as stated > > in its documentation. > > Neat! Thank you Guillermo for this information. > > Best regards, > Carl > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 22:32 ` John W Higgins @ 2020-06-26 0:09 ` Laurent Bercot 2020-06-26 0:49 ` John W Higgins 2020-06-27 7:33 ` Jonathan de Boyne Pollard 1 sibling, 1 reply; 14+ messages in thread From: Laurent Bercot @ 2020-06-26 0:09 UTC (permalink / raw) To: John W Higgins, Supervision >I use lnav [1] for a similar concept and it parses TAI64N but it does not >deal with leap seconds so my timestamps were off by like 20 seconds (or >something similar). I had to modify the code to handle the leap seconds. TAI64N labels represent TAI time, which is linear; when you need a leap second table is when you want to convert TAI labels to UTC or local time. My guess is that lnav just prints the TAI time as if it were UTC, which indeed will print a time that, in 2020, is too high by 27 seconds. If that is the case, it's a bug that would be worth reporting to lnav. -- Laurent ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 0:09 ` Laurent Bercot @ 2020-06-26 0:49 ` John W Higgins 2020-06-26 9:24 ` Jan Braun ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: John W Higgins @ 2020-06-26 0:49 UTC (permalink / raw) To: Supervision [-- Attachment #1: Type: text/plain, Size: 974 bytes --] On Thu, Jun 25, 2020 at 5:09 PM Laurent Bercot <ska-supervision@skarnet.org> wrote: > > My guess is that lnav just prints the TAI time as if it were UTC, > which indeed will print a time that, in 2020, is too high by 27 seconds. > If that is the case, it's a bug that would be worth reporting to lnav. > > I meant nothing towards s6 - but daemontools does not deal with leap seconds (or at least it cetainly looks that way from my foolish viewpoint). So the problem for something like lnav is that it gets stuck between daemontools and other tools like s6 which implement something "differently" - be that right or wrong or anything else under the Sun, It looks like lnav took the concept from daemontools and ran with it - far worse decisions have been made by a tool trying to accomodate users. I was just pointing out that there is a second check which may need to be applied to make sure the viewer is providing the information one believes they are seeing. John ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 0:49 ` John W Higgins @ 2020-06-26 9:24 ` Jan Braun 2020-06-26 9:27 ` Laurent Bercot 2020-06-27 7:33 ` Jonathan de Boyne Pollard 2 siblings, 0 replies; 14+ messages in thread From: Jan Braun @ 2020-06-26 9:24 UTC (permalink / raw) To: John W Higgins; +Cc: Supervision [-- Attachment #1: Type: text/plain, Size: 1656 bytes --] John W Higgins schrob: > I meant nothing towards s6 - but daemontools does not deal with leap > seconds (or at least it cetainly looks that way from my foolish viewpoint). Daemontools assumes that time(3) returns TAI-like time (seconds actually elapsed since the epoch). That's a rather sane approach that allows to do time calculations with complete disregard for leap seconds and time zones. And for communication with the user, you consult tables listing the effects of the erratic movements of the earth and politicians, respectively. These tables are usually shipped in the "right/" tzdata directory, used by setting e.g. TZ=right/Europe/Lisbon . Unfortunately, POSIX mandates that time(3) returns UTC-like time instead (arithmetic difference between the epoch and now, calculated as if leap seconds didn't exist). That's what most systems do, also because the NTP protocol runs on UTC, and NTP clients by default set computer's clocks to UTC-like timestamps. The obvious disadvantage is the need to keep leap seconds in mind when doing any calculation, or to accept being wrong by up to 27 (as of now) seconds. And then consult tables for politician effects only, e.g. TZ=Europe/Lisbon or equivalently TZ=posix/Europe/Lisbon . So yes, daemontools does not deal with leap seconds, because that's the smart thing to do. ;) At least if you can configure your NTP client to support that scenario. > I was just pointing out that there is a second check which may need to be > applied to make sure the viewer is providing the information one believes > they are seeing. Yes. I hope I could provice some useful/interesting background. Cheers, Jan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 0:49 ` John W Higgins 2020-06-26 9:24 ` Jan Braun @ 2020-06-26 9:27 ` Laurent Bercot 2020-06-26 19:52 ` John W Higgins 2020-06-27 7:33 ` Jonathan de Boyne Pollard 2 siblings, 1 reply; 14+ messages in thread From: Laurent Bercot @ 2020-06-26 9:27 UTC (permalink / raw) To: John W Higgins, Supervision >I meant nothing towards s6 - but daemontools does not deal with leap >seconds (or at least it cetainly looks that way from my foolish viewpoint). Indeed, the tai64nlocal program from daemontools assumes that the system clock follows TAI-10, which is a smart choice, but unfortunately not the one that has been made by POSIX - which mandates the system clock to follow UTC. Assuming TAI-10 was a reasonable choice in 1998 (when daemontools was written) because POSIX was not as commonly followed a standard as it became several years later. In 2020, however, TAI-10 is a niche setting that can only be done by systems where all components are aware of it. Most software today assumes UTC, and cannot be blamed for it because it just follows the standard; daemontools' tai64nlocal is now the odd one out and does not display accurate time on POSIX-compliant systems. In the case of the s6-tai64nlocal program, which is s6's version of its venerable daemontools counterpart, this is configurable at skalibs compilation time: the default assumption is UTC, but the --enable-tai-clock configure option will tell skalibs that the system clock follows TAI-10. Please bear in mind that this only impacts the time *display*, i.e. when you want to print the current time to users, typically in a broken-down fashion. The whole point of using TAI in the first place is that time computations are kept simple and exact, and do not need leap second awareness or clock torture techniques such as leap smear (only Google is evil enough to have invented that). TAI is still the correct way to represent time internally, and difficulties such as "need to look at a leap second table" only arise when you want to convert it to something that humans traditionally use, such as UTC or local time, in order to display it in a form that is suitable for human consumption. >It looks like lnav took the concept from daemontools and ran with it - far >worse decisions have been made by a tool trying to accomodate users. It is definitely not a critical bug, but it is a bug nonetheless, that shows a lack of understanding of the historical context, the use cases of TAI and UTC, and the relationship between the two. It would be worth reporting it to lnav, so it can print accurate information. -- Laurent ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 9:27 ` Laurent Bercot @ 2020-06-26 19:52 ` John W Higgins 2020-06-27 6:12 ` Laurent Bercot 0 siblings, 1 reply; 14+ messages in thread From: John W Higgins @ 2020-06-26 19:52 UTC (permalink / raw) To: Supervision [-- Attachment #1: Type: text/plain, Size: 2691 bytes --] On Fri, Jun 26, 2020 at 2:27 AM Laurent Bercot <ska-supervision@skarnet.org> wrote: > > > Please bear in mind that this only impacts the time *display*, i.e. > when you want to print the current time to users, typically in a > broken-down fashion. The whole point of using TAI in the first place > is that time computations are kept simple and exact, and do not need > leap second awareness or clock torture techniques such as leap smear > (only Google is evil enough to have invented that). TAI is still the > correct way to represent time internally, and difficulties such as > "need to look at a leap second table" only arise when you want to > convert it to something that humans traditionally use, such as UTC or > local time, in order to display it in a form that is suitable for > human consumption. > > > >It looks like lnav took the concept from daemontools and ran with it - far > >worse decisions have been made by a tool trying to accomodate users. > > It is definitely not a critical bug, but it is a bug nonetheless, that > shows a lack of understanding of the historical context, the use cases > of TAI and UTC, and the relationship between the two. It would be worth > reporting it to lnav, so it can print accurate information. > > Except it's not quite that simple either. As an example echo "s6" | s6-log -t /tmp/a && echo "djb" | multilog t /tmp/a && echo "s6" | s6-log -t /tmp/a Produces the following /tmp/a/current file on my system, roughly as a write this. @400000005ef64dd2075013a2 s6 @400000005ef64db707675514 djb @400000005ef64dd2077fa679 s6 There is a large problem there for someone building a tool to show timestamps that begin as TAI. Either the middle one is 27 seconds in the past or the first and last are 27 seconds in the future. If one wants to support djb tooling - then leap seconds are off the table - if one wishes to support s6 - leap seconds come on the table. What nosh supports is unknown to me. When lnav implemented the support - they did exactly what sounds reasonable - they took the tool they wished to support (multilog I would presume as it mentionds cr.yp.to) and implemented the code required to produce a view of that tool's logs. To say that is a lack of understanding is rather harsh here. John P.S. I totally appreciate s6 and all its beauty - I really just wanted to point out to the original poster that they need to double check because the worst thing one can do is miss something like this and then wonder why their logs are off when they go and look at a production system. 27 seconds seems obvious to catch - but in development one doesn't always focus enough - it's not like 2 hours or something more obvious. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 19:52 ` John W Higgins @ 2020-06-27 6:12 ` Laurent Bercot 0 siblings, 0 replies; 14+ messages in thread From: Laurent Bercot @ 2020-06-27 6:12 UTC (permalink / raw) To: John W Higgins, Supervision >When lnav implemented the support - they did exactly what sounds reasonable >- they took the tool they wished to support (multilog I would presume as it >mentionds cr.yp.to) and implemented the code required to produce a view of >that tool's logs. To say that is a lack of understanding is rather harsh >here. Right. I was being charitable; it's indeed also possible that they *intentionally* did the wrong thing, which is obviously much worse. :P -- Laurent ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-26 0:49 ` John W Higgins 2020-06-26 9:24 ` Jan Braun 2020-06-26 9:27 ` Laurent Bercot @ 2020-06-27 7:33 ` Jonathan de Boyne Pollard 2020-06-28 19:38 ` Guillermo 2 siblings, 1 reply; 14+ messages in thread From: Jonathan de Boyne Pollard @ 2020-06-27 7:33 UTC (permalink / raw) To: Supervision John W Higgins wrote: > I meant nothing towards s6 - but daemontools does not deal with leap > seconds (or at least it cetainly looks that way from my foolish > viewpoint). > daemontools very much *does* deal in leap seconds. The little-known requirement for daemontools, which *is* in its manual and has been all along albeit that many people have overlooked this, is that one use one of the Olson "right" timezones. * http://jdebp.uk./Softwares/nosh/guide/timestamps.html * http://cr.yp.to/daemontools/tai64nlocal.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-27 7:33 ` Jonathan de Boyne Pollard @ 2020-06-28 19:38 ` Guillermo 0 siblings, 0 replies; 14+ messages in thread From: Guillermo @ 2020-06-28 19:38 UTC (permalink / raw) To: Supervision El sáb., 27 jun. 2020 a las 4:59, Jonathan de Boyne Pollard escribió: > > John W Higgins wrote: > > > I meant nothing towards s6 - but daemontools does not deal with leap > > seconds (or at least it cetainly looks that way from my foolish > > viewpoint). > > > daemontools very much *does* deal in leap seconds. The little-known > requirement for daemontools, which *is* in its manual and has been all > along albeit that many people have overlooked this, is that one use one > of the Olson "right" timezones. If anyone is interested, decoding the TAI64N timestamps produced by s6-log with nosh's version of tai64nlocal produces the same result as doing so with s6-tai64nlocal, and decoding the TAI64N timestamps produced by nosh's cyclog with s6-tai64nlocal produces the same result as doing so with nosh's tai64nlocal. At least on my system without a "right/" timezone. On the other hand, I just realized that cyclog does not seem to properly finalize the 'current' file (including the setting of the execute-by-owner permission) when it exits normally. Perhaps the destructor of class 'logger' should call logger::flush_and_close() rather than logger::close()? G. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Following log directories 2020-06-25 22:32 ` John W Higgins 2020-06-26 0:09 ` Laurent Bercot @ 2020-06-27 7:33 ` Jonathan de Boyne Pollard 1 sibling, 0 replies; 14+ messages in thread From: Jonathan de Boyne Pollard @ 2020-06-27 7:33 UTC (permalink / raw) To: Supervision John W Higgins: > One thing to remember about tools like this is that s6 is very precise > with it's implemenation of TAI64N - it uses leap seconds. > For the case of follow-log-directories, note that it does not decode the timestamps into times. As long as they follow the simple (and obvious) constraints in its manual, the tool is happy. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-06-28 19:39 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-25 9:27 Following log directories Carl Winbäck 2020-06-25 13:20 ` Guillermo 2020-06-25 14:52 ` Guillermo 2020-06-25 22:08 ` Carl Winbäck 2020-06-25 22:32 ` John W Higgins 2020-06-26 0:09 ` Laurent Bercot 2020-06-26 0:49 ` John W Higgins 2020-06-26 9:24 ` Jan Braun 2020-06-26 9:27 ` Laurent Bercot 2020-06-26 19:52 ` John W Higgins 2020-06-27 6:12 ` Laurent Bercot 2020-06-27 7:33 ` Jonathan de Boyne Pollard 2020-06-28 19:38 ` Guillermo 2020-06-27 7:33 ` Jonathan de Boyne Pollard
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).