svlogd -tt

svlogd -tt

[Henrik Heil]

Hello,

i read the thread about the svlogd -t switches some weeks ago and though 
  i am using -tt (UTC) happily i am missing the feature to have the 
rotated logfiles with UTC-names instead of @TAI-names.

Is there a drawback to make this an option?
The order of the logfiles would be preserved, selecting logfiles with 
wildcards would be easy.

I cannot comment on pro/contra TAI itsself but i think one who decides 
not to have TAI-timestamps in the logfiles would prefer UTC-filenames 
for the rotated files too.

What do you think?

Best regards,
Henrik

-- 
Henrik Heil, zweipol Coy & Heil GbR
http://www.zweipol.net/

Re: svlogd -tt

[Gerrit Pape]

On Tue, Jul 20, 2004 at 09:00:06PM +0200, Henrik Heil wrote:
> i read the thread about the svlogd -t switches some weeks ago and though 
>  i am using -tt (UTC) happily i am missing the feature to have the 
> rotated logfiles with UTC-names instead of @TAI-names.
> 
> Is there a drawback to make this an option?
> The order of the logfiles would be preserved, selecting logfiles with 
> wildcards would be easy.
> 
> I cannot comment on pro/contra TAI itsself but i think one who decides 
> not to have TAI-timestamps in the logfiles would prefer UTC-filenames 
> for the rotated files too.
> 
> What do you think?

When making it a runtime option, there must be a decision whether svlogd
shall maintain two different sets of files in the log directory, or
filename conversion should be done.

I'm not eager to add this option to svlogd, but would be interested in
the opinion of other people, and maybe a patch, though.

Regards, Gerrit.

Re: svlogd -tt

[Ian Stokes-Rees]

Ian.

It is a bit hard to see at a glance what the file names mean for log 
files.  Maybe a sym link to the existing files with the sym link having 
a more descriptive name?

Ian

Gerrit Pape wrote:
> On Tue, Jul 20, 2004 at 09:00:06PM +0200, Henrik Heil wrote:
> 
>>i read the thread about the svlogd -t switches some weeks ago and though 
>> i am using -tt (UTC) happily i am missing the feature to have the 
>>rotated logfiles with UTC-names instead of @TAI-names.
>>
>>Is there a drawback to make this an option?
>>The order of the logfiles would be preserved, selecting logfiles with 
>>wildcards would be easy.
>>
>>I cannot comment on pro/contra TAI itsself but i think one who decides 
>>not to have TAI-timestamps in the logfiles would prefer UTC-filenames 
>>for the rotated files too.
>>
>>What do you think?
> 
> 
> When making it a runtime option, there must be a decision whether svlogd
> shall maintain two different sets of files in the log directory, or
> filename conversion should be done.
> 
> I'm not eager to add this option to svlogd, but would be interested in
> the opinion of other people, and maybe a patch, though.
> 
> Regards, Gerrit.

-- 
Ian Stokes-Rees                 i.stokes-rees1@physics.ox.ac.uk
Particle Physics, Oxford        http://www-pnp.physics.ox.ac.uk/~stokes

Re: svlogd -tt

[Henrik Heil]

>>i read the thread about the svlogd -t switches some weeks ago and though 
>> i am using -tt (UTC) happily i am missing the feature to have the 
>>rotated logfiles with UTC-names instead of @TAI-names.
>>
>>Is there a drawback to make this an option?
>>The order of the logfiles would be preserved, selecting logfiles with 
>>wildcards would be easy.
>>
>>I cannot comment on pro/contra TAI itsself but i think one who decides 
>>not to have TAI-timestamps in the logfiles would prefer UTC-filenames 
>>for the rotated files too.
>>
>>What do you think?
> 
> 
> When making it a runtime option, there must be a decision whether svlogd
> shall maintain two different sets of files in the log directory, or
> filename conversion should be done.

Thinking about these alternatives i must admit that both seem to have a 
drawback -- and that i underestimated the complexity of switching the 
formats.

Different sets (if i understood that right) means that it is not 
intuitive what the maximum number of old log files from the config 
means. It cannot mean num=TAI+UTC because in this case svlogd could not 
rotate if the log directory would contain num logfiles of the other 
format. If it means num*TAI + num*UTF the straightforward method to 
calculate the maximum size of the log directory no longer works and 
changing the format without moving the old files leads to a log 
directory that occupies twice the disk space.

With filename conversion i am not shure if you mean renaming the 
existing logfiles or properly calculating the oldest file from both 
formats before deleting it. Unfortunatly i have no idea how complex any 
of these two would be -- probably it's not worth the effort.

What do you think about a runtime option for UTC-filenames that only 
comes into effect if there are no TAI-filenames in the log directory and 
vice versa?
I guess this would require a small patch and almost no overhead when 
rotating the logfiles. I think there is nothing to gain by leaving the 
old files in the log directory when switching the format.

Best regards,
Henrik

-- 
Henrik Heil, zweipol Coy & Heil GbR
http://www.zweipol.net/

Re: svlogd -tt

[Gerrit Pape]

On Mon, Jul 26, 2004 at 08:39:39PM +0200, Henrik Heil wrote:
> >When making it a runtime option, there must be a decision whether svlogd
> >shall maintain two different sets of files in the log directory, or
> >filename conversion should be done.
> 
> Thinking about these alternatives i must admit that both seem to have a 
> drawback -- and that i underestimated the complexity of switching the 
> formats.
> 
> Different sets (if i understood that right) means that it is not 
> intuitive what the maximum number of old log files from the config 
> means. It cannot mean num=TAI+UTC because in this case svlogd could not 

Yes.  But even so this looks like the best way to do it if at all.
svlogd would maintain two different sets of log files (one set with
TAI-filenames, one with UTC-filenames) independently, according to its
configuration.  When changing the configuration, and "current" gets big
enough, svlogd adds it to the currently configured set, leaving the
other set of log files alone.  If desired, the log files could be
converted from one file name to the other manually using a separate tool
(this again raises the request for a tai64nlocal alike program), or
removed.

But I'm not convinced that it's worth the trouble to implement this or
similar.  There still is the idea of a log processor program (svlogp)
which could be also used to list the log files in a directory with human
readable names, or gives convenient access to the logs by other means.

> What do you think about a runtime option for UTC-filenames that only 
> comes into effect if there are no TAI-filenames in the log directory and 
> vice versa?

I'm not sure; svlogd at least should print a warning in this case, which
then ends up in the readproctitle log.  It's not that intuitive to
users, and susceptible to misunderstandings in my opinion.

Regards, Gerrit.

Re: svlogd -tt

[Henrik Heil]

Gerrit Pape wrote:
> But I'm not convinced that it's worth the trouble to implement this or
agreed.

> similar.  There still is the idea of a log processor program (svlogp)
> which could be also used to list the log files in a directory with human
> readable names, or gives convenient access to the logs by other means.
I see -- this would be the cleanest solution.

I thought the UTC-filenames would be quite trivial to implement because
the timestamps already exist for use with the -tt option but i didn't 
look at the consequences for the log directory.

My C programming skills are quite limited -- but i can offer to test 
early versions of svlogp if you decide to work on it.

>>What do you think about a runtime option for UTC-filenames that only 
>>comes into effect if there are no TAI-filenames in the log directory and 
>>vice versa?
> 
> I'm not sure; svlogd at least should print a warning in this case, which
> then ends up in the readproctitle log.  It's not that intuitive to
> users, and susceptible to misunderstandings in my opinion.
true -- was not a good idea.

Best regards,
Henrik

-- 
Henrik Heil, zweipol Coy & Heil GbR
http://www.zweipol.net/

Re: svlogd -tt

[Alex Efros]

Hi!

You all discuss very complex things... patches... :) probably I misunderstand
this issue, but I think we already have everything we need in file timestamp,
and there is no need to convert logfile names from TAI.

If you want to select logfile name based on localtime just list directory:

home /var/log/qmail/qmail-send # ls -o
total 17494
-rw-r--r--    1 qmaill        411 Feb 21 17:57 @4000000040378031026453dc.u
-rw-r--r--    1 qmaill     173932 Feb 23 11:02 @400000004039c4611fadac2c.u
-rw-r--r--    1 qmaill    1024279 Mar  2 10:32 @400000004044a95f0f120e1c.u
...skip....
-rw-r--r--    1 qmaill        542 Jul  8 18:39 @4000000040ed6b2734067d34.u
-rw-r--r--    1 qmaill     741472 Jul 23 00:45 @40000000410039f317fab1dc.u
-rwxr--r--    1 qmaill    2499038 Jul 26 21:07 @400000004105486f30840564.s
-rw-r--r--    1 root           12 Feb 20 18:00 config
-rw-r--r--    1 qmaill     683653 Jul 29 01:45 current
-rw-------    1 qmaill          0 Feb 16 03:04 lock
-rw-r--r--    1 qmaill          0 Feb 16 03:04 state
home /var/log/qmail/qmail-send #

and then copy/paste needed logfile name using mouse.

On Tue, Jul 20, 2004 at 09:00:06PM +0200, Henrik Heil wrote:
> i read the thread about the svlogd -t switches some weeks ago and though 
>  i am using -tt (UTC) happily i am missing the feature to have the 
> rotated logfiles with UTC-names instead of @TAI-names.
> 
> Is there a drawback to make this an option?
> The order of the logfiles would be preserved, selecting logfiles with 
> wildcards would be easy.

If you want to select group of logfiles based on localtime you can do a
number of tricks, for example:

home /var/log/qmail/qmail-send # find -name '@*' -mtime -7 -daystart -exec ls -o {} \;
-rw-r--r--    1 qmaill     741472 Jul 23 00:45 ./@40000000410039f317fab1dc.u
-rwxr--r--    1 qmaill    2499038 Jul 26 21:07 ./@400000004105486f30840564.s
home /var/log/qmail/qmail-send #

you can use any sort of command instead of 'ls -o' for apply to all logfiles
for a last week (-mtime -7).

If you really-really need to have localtime 'wildcards' you do something like
this:

home /var/log/qmail/qmail-send # ls -o | grep 'Jun 15 19:'
-rw-r--r--    1 qmaill      19540 Jun 15 19:27 @4000000040cf23f91d3bf0ac.u
-rw-r--r--    1 qmaill       4334 Jun 15 19:45 @4000000040cf2a0f19b638d4.u
-rw-r--r--    1 qmaill         57 Jun 15 19:55 @4000000040cf2ad33559015c.u
home /var/log/qmail/qmail-send # ls -o | grep 'Jun 15 19:' | cut -b 48-
@4000000040cf23f91d3bf0ac.u
@4000000040cf2a0f19b638d4.u
@4000000040cf2ad33559015c.u
home /var/log/qmail/qmail-send #

Is there tricks help?

-- 
			WBR, Alex.

Re: svlogd -tt

[Henrik Heil]

Alex Efros wrote:
> You all discuss very complex things... patches... :) probably I misunderstand
> this issue, but I think we already have everything we need in file timestamp,
> and there is no need to convert logfile names from TAI.
I know that i can use the mtime to select the logfiles.
Thanks for the examples anyway. I still think that this is kind of a 
workaround.
With the mtime approach you could argue that there is no need for a 
timestamp in the filenames at all. Even without mtime you could do a 
script that looks in all the files to select ranges.
In my opinion a log processor program would the preferable solution and 
fit well in the runit-suite.

Best regards,
Henrik

-- 
Henrik Heil, zweipol Coy & Heil GbR
http://www.zweipol.net/

Re: svlogd -tt

[Alex Efros]

Hi!

On Thu, Jul 29, 2004 at 12:12:59PM +0200, Henrik Heil wrote:
> I know that i can use the mtime to select the logfiles.
> Thanks for the examples anyway. I still think that this is kind of a 
> workaround.
> With the mtime approach you could argue that there is no need for a 
> timestamp in the filenames at all. Even without mtime you could do a 
> script that looks in all the files to select ranges.
> In my opinion a log processor program would the preferable solution and 
> fit well in the runit-suite.

Hmm... how about this idea for log processor program:
1) svlogd execute processor which just fork in background and return log file
   without changes back to svlogd
2) svlogd remove old TAI-logfile and create new TAI-logfile
3) processor (forked in background) notice log rotation is done (how?) and then:
    1) execute 'ln newTAIlogfile newLOCALTIMElogfile' to create hard linked file
    2) search for some oldLOCALTIMElogfile with only one hardlink to it (because
       oldTAIlogfile which was hardlinked to it was just removed by svlogd) and
       remove it too
    3) exit

Maybe it's better to do this work in cron-script instead of log processor.

-- 
			WBR, Alex.

Re: svlogd -tt

[Henrik Heil]

>>I know that i can use the mtime to select the logfiles.
>>Thanks for the examples anyway. I still think that this is kind of a 
>>workaround.
>>With the mtime approach you could argue that there is no need for a 
>>timestamp in the filenames at all. Even without mtime you could do a 
>>script that looks in all the files to select ranges.
>>In my opinion a log processor program would the preferable solution and 
>>fit well in the runit-suite.
> 
> Hmm... how about this idea for log processor program:
> 1) svlogd execute processor which just fork in background and return log file
>    without changes back to svlogd
> 2) svlogd remove old TAI-logfile and create new TAI-logfile
> 3) processor (forked in background) notice log rotation is done (how?) and then:
>     1) execute 'ln newTAIlogfile newLOCALTIMElogfile' to create hard linked file
>     2) search for some oldLOCALTIMElogfile with only one hardlink to it (because
>        oldTAIlogfile which was hardlinked to it was just removed by svlogd) and
>        remove it too
>     3) exit
> 
> Maybe it's better to do this work in cron-script instead of log processor.

Sorry for my ambiguous writing -- i meant the "svlogp" program that 
Gerrit mentioned and described like this:

> I've been thinking about a "svlogp" program (svlog processor, or pager)
> for some time, which should be able to select log message by pattern or
> time, to merge logs from multiple log directories, and maybe to
> transform timestamps and more.

With svlogp the format of filenames and timestamps wouldn't matter 
anymore. You could for example [my interpretation] specify one or more 
logdir(s), your favorite timeformat, from-date and to-date (in that 
timeformat) on the command line and get the merged logmessages for that 
range on stdout, lines prefixed with a timestamp in the given format.

Best regards,
Henrik

-- 
Henrik Heil, zweipol Coy & Heil GbR
http://www.zweipol.net/