svlogd localtime option

svlogd localtime option

[Jared Rhine]

In the spirit of making svlogd output useful to humans, svlogd has the
-tt switch to output in human-readable UTC.  Would there be resistance
to having it be really useful by supporting a -ttt which uses
localtime?  We're really talking about having using localtime instead
of gmtime in one line in fmt_ptime.c, though option processing and
such would need to change.

I appreciate that UTC-based servers are useful in a global setting,
but it's a hindrance to good systems administration in many contexts.
And there's always the internal politics, where coworkers complain.

I couldn't find any references to this enhancement request in the
mailing list archives.  Apologies if this has been raised and rejected
in the past.

It may be the case that there's already an (undocumented?) -ttt
option, if I'm reading svlogd.c right.  It looks like that option
truncates the sub-second portion.  Perhaps -tttt for localtime
instead?

PS: #define USAGE in svlogd.c is out of sync with svlogd.8, with
respect to -tt.

-- jared@wordzoo.com

"A black hole is where God is dividing by zero."
        -- attributed to Roger Smith

Re: svlogd localtime option

[Gerrit Pape]

On Thu, Jul 01, 2004 at 12:53:15PM -0700, Jared Rhine wrote:
> In the spirit of making svlogd output useful to humans, svlogd has the
> -tt switch to output in human-readable UTC.  Would there be resistance
> to having it be really useful by supporting a -ttt which uses
> localtime?  We're really talking about having using localtime instead
> of gmtime in one line in fmt_ptime.c, though option processing and
> such would need to change.
> 
> I appreciate that UTC-based servers are useful in a global setting,
> but it's a hindrance to good systems administration in many contexts.
> And there's always the internal politics, where coworkers complain.

Using TAI or UTC has the great advantage that logs are sortable after
merging, this is not guaranteed when using localtime.

I think it's better to let the pager used to display logs do such jobs.
I've been thinking about a "svlogp" program (svlog processor, or pager)
for some time, which should be able to select log message by pattern or
time, to merge logs from multiple log directories, and maybe to
transform timestamps and more.

There have been feature requests for a tai64nlocal alike program.

> It may be the case that there's already an (undocumented?) -ttt
> option, if I'm reading svlogd.c right.  It looks like that option
> truncates the sub-second portion.

Yes, but please don't use it, it's undocumented...

Regards, Gerrit.

Re: svlogd localtime option

[Charlie Brady]

On Sun, 4 Jul 2004, Gerrit Pape wrote:

> On Thu, Jul 01, 2004 at 12:53:15PM -0700, Jared Rhine wrote:
> > 
> > I appreciate that UTC-based servers are useful in a global setting,
> > but it's a hindrance to good systems administration in many contexts.
> > And there's always the internal politics, where coworkers complain.
> 
> Using TAI or UTC has the great advantage that logs are sortable after
> merging, this is not guaranteed when using localtime.

And the reason they aren't sortable is that times can be ambiguous in 
local time - one hour occurs twice each year (in many places). Ambiguous 
times in log files are not such a good idea.

> There have been feature requests for a tai64nlocal alike program.

Has anyone knocked one of these together?

---
Charlie

svlogd localtime option

[Jared Rhine]

  Jared> Would there be resistance to having [svlogd] be really useful by
  Jared> supporting a -ttt which uses localtime? [...] I appreciate
  Jared> that UTC-based servers are useful...

  Gerrit> Using TAI or UTC has the great advantage that logs are
  Gerrit> sortable after merging, this is not guaranteed when using
  Gerrit> localtime.

As mentioned, I understand that UTC has advantages.  With mutual
respect as professionals, I disagree with that characterization of
"great advantage" though, especially in comparsion to the
disadvantages.  The disadvantages hit me every single day, whereas
I've never once wanted to do a simple log merge in 10 years across
dozens (hundreds?) of servers.  When I do, it's always script-mediated
and my localtime can easily be transformed back into UTC or TAI64n
specifically for that purpose (though that's not how I do it).

Why is UTC there at all?  TAI64n has some minor advantages over even
UTC, so why don't we force everyone to pipe everytime?  Presumably to
save some humans some time.

It simply seems wise to support the common use case, which is maybe
95% browsing, 4% archiving, and 1% merging.  Instead, I should need to
incorporate a pipeline for 99+% of the times I use the file, instead
of the <1% of the time I need to merge?  And cause extra work every
time I'm grepping, opening in an editor, etc.  Of course I can
construct command lines pipes and aliases to work around the problem.
I just don't see why I'm working around it all.

  Gerrit> I think it's better to let the pager used to display logs do
  Gerrit> such jobs.

Fair enough.  But reasonable people can disagree on methods and
practices, and not everyone uses a pager for browsing.  This one
single issue is the sole gripe about the functionality of runit.

It's trivial for me to substitute gmtime with localtime, and my latest
installs are working great like that.  But now I'm maintaining a fork,
which seems silly for such a small (and useful!) change.  But if such
an enhancement would be rejected on design grounds, it's important
enough for me to do so.  I'd rather put that time into a patch that
would be accepted; 

  Charlie> Ambiguous times in log files are not such a good idea.

I'm a big boy; I'd prefer the rope to hang myself.  I'm just asking
for a switch that's maybe a couple lines of code, not to bring down
the entire sysadmin community with my wicked ways :)

Anyway, that's my pitch.  I tried to state it as strongly as I
politely can.  Still your call on the design/patch acceptance issue of
course.

-- jared@wordzoo.com

War is God's way of teaching Americans geography. -Ambrose Bierce

Re: svlogd localtime option

[Ian Stokes-Rees]

Hi,

FWIW, I agree 100% with Jared.  I *only* look at runit output on its 
own, I *never* do merging, and if I *did* do merging, it would only be 
with all the other log file outputs which I have which use localtime.  I 
often want to cut and paste blocks of runit output into emails to 
demonstrate certain bits of behaviour, and I constantly have to explain 
to people that it is GMT, not localtime.  At least here in France that 
is only a one or two hour difference (depending on time of year), 
whereas for others it could be very annoying to constantly delete 11 
hours (say).

Is it really that aweful to give users this common sense and common use 
case option by, as Jared suggested, adding a -ttt option?  If they 
really want GMT, then -tt is fine.

I also agree that runit is amazing, and I am astonished that more people 
don't use it.  I've started publicising it quite widely among the grid 
computing community I'm involved with, and have referenced it as being 
critical to the fault tolerance and robustness of the grid 
infrastructure I've been developing.

Cheers,

Ian.

Jared Rhine wrote:
>   Jared> Would there be resistance to having [svlogd] be really useful by
>   Jared> supporting a -ttt which uses localtime? [...] I appreciate
>   Jared> that UTC-based servers are useful...
> 
>   Gerrit> Using TAI or UTC has the great advantage that logs are
>   Gerrit> sortable after merging, this is not guaranteed when using
>   Gerrit> localtime.
> 
> As mentioned, I understand that UTC has advantages.  With mutual
> respect as professionals, I disagree with that characterization of
> "great advantage" though, especially in comparsion to the
> disadvantages.  The disadvantages hit me every single day, whereas
> I've never once wanted to do a simple log merge in 10 years across
> dozens (hundreds?) of servers.  When I do, it's always script-mediated
> and my localtime can easily be transformed back into UTC or TAI64n
> specifically for that purpose (though that's not how I do it).
> 
> Why is UTC there at all?  TAI64n has some minor advantages over even
> UTC, so why don't we force everyone to pipe everytime?  Presumably to
> save some humans some time.
> 
> It simply seems wise to support the common use case, which is maybe
> 95% browsing, 4% archiving, and 1% merging.  Instead, I should need to
> incorporate a pipeline for 99+% of the times I use the file, instead
> of the <1% of the time I need to merge?  And cause extra work every
> time I'm grepping, opening in an editor, etc.  Of course I can
> construct command lines pipes and aliases to work around the problem.
> I just don't see why I'm working around it all.
> 
>   Gerrit> I think it's better to let the pager used to display logs do
>   Gerrit> such jobs.
> 
> Fair enough.  But reasonable people can disagree on methods and
> practices, and not everyone uses a pager for browsing.  This one
> single issue is the sole gripe about the functionality of runit.
> 
> It's trivial for me to substitute gmtime with localtime, and my latest
> installs are working great like that.  But now I'm maintaining a fork,
> which seems silly for such a small (and useful!) change.  But if such
> an enhancement would be rejected on design grounds, it's important
> enough for me to do so.  I'd rather put that time into a patch that
> would be accepted; 
> 
>   Charlie> Ambiguous times in log files are not such a good idea.
> 
> I'm a big boy; I'd prefer the rope to hang myself.  I'm just asking
> for a switch that's maybe a couple lines of code, not to bring down
> the entire sysadmin community with my wicked ways :)
> 
> Anyway, that's my pitch.  I tried to state it as strongly as I
> politely can.  Still your call on the design/patch acceptance issue of
> course.
> 
> -- jared@wordzoo.com
> 
> War is God's way of teaching Americans geography. -Ambrose Bierce

-- 
Ian Stokes-Rees                 i.stokes-rees1@physics.ox.ac.uk
Particle Physics, Oxford        http://www-pnp.physics.ox.ac.uk/~stokes

Re: svlogd localtime option

[Uffe Jakobsen]

Hi Jared etc.,

Jared Rhine <jared@wordzoo.com> wrote:
>
> It's trivial for me to substitute gmtime with localtime, and my latest
> installs are working great like that.

First I fully agree on your arguments for a "-ttt" option - that would at
least give individuals the choise between UTC and localtime logging...

Another thing is the overall svlogd performance:

Last year I did some performance measurements on some *VERY* busy Solaris
and HP-UX systems - using a "localtime" modified svlogd...  :-)

The tests showed that calling localtime is about 25 times slower than
calling gmtime - unfortunately I never did the same measurement for
Linux-variants... (I might add that calling localtime_r is even slower than
localtime  ;-)

Switching back to gmtime saved me from upgrading the boxes - but I realize
that the above scenario is a somewhat extreme situation - and I still I
think that having the '-ttt' would be good.

kind regards Uffe

Re: svlogd localtime option

[Charlie Brady]

On Fri, 9 Jul 2004, Jared Rhine wrote:

> dozens (hundreds?) of servers.  When I do, it's always script-mediated
> and my localtime can easily be transformed back into UTC or TAI64n

This is only true if your localtime has no "daylight savings" or "summer" 
time changes.

> War is God's way of teaching Americans geography. -Ambrose Bierce

A cool idea, but doesn't seem to have happened yet. Nor has enough history 
been instilled.

---
Charlie

Re: svlogd localtime option

[Gerrit Pape]

On Fri, Jul 09, 2004 at 01:24:03AM -0700, Jared Rhine wrote:
>   Gerrit> Using TAI or UTC has the great advantage that logs are
>   Gerrit> sortable after merging, this is not guaranteed when using
>   Gerrit> localtime.

>   Gerrit> I think it's better to let the pager used to display logs do
>   Gerrit> such jobs.
> 
> Fair enough.  But reasonable people can disagree on methods and
> practices, and not everyone uses a pager for browsing.  This one
> single issue is the sole gripe about the functionality of runit.
> 
> It's trivial for me to substitute gmtime with localtime, and my latest
> installs are working great like that.  But now I'm maintaining a fork,
> which seems silly for such a small (and useful!) change.  But if such
> an enhancement would be rejected on design grounds, it's important
> enough for me to do so.  I'd rather put that time into a patch that
> would be accepted; 

> Anyway, that's my pitch.  I tried to state it as strongly as I
> politely can.  Still your call on the design/patch acceptance issue of
> course.

I definitely prefer UTC over localtime, and have my system's clocks set
to UTC anyway, so I'm personally not that much interested.  I cannot
promise acceptance of a patch, but if this feature is added to svlogd,
it should be provided through the -ttt option I think; the documentation
should prominently mention the drawbacks of localtime timestamps.

I very appreciate contribution to my open projects
 http://smarden.org/pape/#contribution

Thanks, Gerrit.