monitoring svlogd-produced logfiles

monitoring svlogd-produced logfiles

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]

Quick question here... is anyone using anything like swatch or SEC to
parse/monitor svlogd-produced logfiles?  I'm having some issues with
tail where it won't follow ./current when it rotates, which is causing
me some grief.  I ended up solving it to some extent using the perl
File::Tail module with swatch, but I'm not overly satisfied with the
results, because swatch will now only let me monitor a single logfile
rather than two (my catchall log and the kernel log from socklog), and
on my loghost I need it to look at probably a dozen logs.

So I'm looking for a good alternative that properly follows ./current
(or reloads it on rotation) and can handle multiple logfiles.

There's a whole bunch of different programs out there so I'm hoping that
someone can share what they're using to save me some time.  I'm
currently looking at SEC, and it seems quite powerful.

The other thing I was considering was to see if I could make svlogd
write to one end of a named pipe and have the log watcher on the end,
avoiding disk writes completely (and let the log watcher write
"summarized" logs).  This would be a benefit when having two "*.*"
services, one writing the logs and the other writing them to the named
pipe.  But I'm not sure if named pipes are exclusive, i.e. can I have
multiple svlogd processes writing to the same named pipe at the same
time?

Ideally, the log watcher should be able to run supervised as well
because although I'd like to watch it realtime (I like swatch's
colorized output), I'd like it to also write the "summary" log as well,
which would be retained for a longer period of time than the standard
logs.

Any suggestions?

Thanks in advance.

-- 
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}
mysql> SELECT * FROM users WHERE clue > 0;
Empty set (0.00sec)
:: Annvix - Secure Linux Server: http://annvix.org/ ::

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: monitoring svlogd-produced logfiles

[Alex Efros]

Hi!

On Wed, Jun 14, 2006 at 05:35:08PM -0600, Vincent Danen wrote:
> Quick question here... is anyone using anything like swatch or SEC to
> parse/monitor svlogd-produced logfiles?  I'm having some issues with
> tail where it won't follow ./current when it rotates, which is causing

Tail can follow ./current after rotate:
    tail -F

-- 
			WBR, Alex.

Re: monitoring svlogd-produced logfiles

[George Georgalis]

On Thu, Jun 15, 2006 at 02:40:04AM +0300, Alex Efros wrote:
>
>Tail can follow ./current after rotate:
>    tail -F
>

Unfortunatly that is not universal...
I use a few different alias in my bourne profile


OS=$(uname)
case $OS in
	NetBSD|FreeBSD|Dragonfly|Darwin)
		alias t='tail -F'
	;;
	Linux)
		alias t='tail --follow=name'
	;;
	OpenBSD)
		alias t='tail -f'
	;;
esac


Cheers,
// George

-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: monitoring svlogd-produced logfiles

[Alex Efros]

Hi!

On Fri, Jun 16, 2006 at 07:03:03AM -0400, George Georgalis wrote:
> OS=$(uname)
> case $OS in
> 	NetBSD|FreeBSD|Dragonfly|Darwin)
> 		alias t='tail -F'
> 	;;
> 	Linux)
> 		alias t='tail --follow=name'
> 	;;
> 	OpenBSD)
> 		alias t='tail -f'
> 	;;
> esac

Linux understand -F too.

-- 
			WBR, Alex.

Re: monitoring svlogd-produced logfiles

[George Georgalis]

On Fri, Jun 16, 2006 at 02:08:06PM +0300, Alex Efros wrote:
>Hi!
>
>On Fri, Jun 16, 2006 at 07:03:03AM -0400, George Georgalis wrote:
>> OS=$(uname)
>> case $OS in
>> 	NetBSD|FreeBSD|Dragonfly|Darwin)
>> 		alias t='tail -F'
>> 	;;
>> 	Linux)
>> 		alias t='tail --follow=name'
>> 	;;
>> 	OpenBSD)
>> 		alias t='tail -f'
>> 	;;
>> esac
>
>Linux understand -F too.

*sigh* not my linux

tail -F /var/log/socklog/main/current 
tail: invalid option -- F
Try `tail --help' for more information.

// George

-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: monitoring svlogd-produced logfiles

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

* George Georgalis <george@galis.org> [2006-06-16 09:18:07 -0400]:

> >> OS=$(uname)
> >> case $OS in
> >> 	NetBSD|FreeBSD|Dragonfly|Darwin)
> >> 		alias t='tail -F'
> >> 	;;
> >> 	Linux)
> >> 		alias t='tail --follow=name'
> >> 	;;
> >> 	OpenBSD)
> >> 		alias t='tail -f'
> >> 	;;
> >> esac
> >
> >Linux understand -F too.
> 
> *sigh* not my linux
> 
> tail -F /var/log/socklog/main/current 
> tail: invalid option -- F
> Try `tail --help' for more information.

Interesting.  Why distro and version of coreutils?  -F is the equivalent
to --follow=name in newer coreutils; I don't know when it was added, but
I've been seeing it for a while.

-- 
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}
mysql> SELECT * FROM users WHERE clue > 0;
Empty set (0.00sec)
:: Annvix - Secure Linux Server: http://annvix.org/ ::

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: monitoring svlogd-produced logfiles

[George Georgalis]

On Fri, Jun 16, 2006 at 01:08:26PM -0600, Vincent Danen wrote:
>* George Georgalis <george@galis.org> [2006-06-16 09:18:07 -0400]:
>
>> >> OS=$(uname)
>> >> case $OS in
>> >> 	NetBSD|FreeBSD|Dragonfly|Darwin)
>> >> 		alias t='tail -F'
>> >> 	;;
>> >> 	Linux)
>> >> 		alias t='tail --follow=name'
>> >> 	;;
>> >> 	OpenBSD)
>> >> 		alias t='tail -f'
>> >> 	;;
>> >> esac
>> >
>> >Linux understand -F too.
>> 
>> *sigh* not my linux
>> 
>> tail -F /var/log/socklog/main/current 
>> tail: invalid option -- F
>> Try `tail --help' for more information.
>
>Interesting.  Why distro and version of coreutils?  -F is the equivalent
>to --follow=name in newer coreutils; I don't know when it was added, but
>I've been seeing it for a while.


Debian Woody (internal... migrating to BSD)

 $ tail --version
tail (GNU textutils) 2.0

// George


-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: monitoring svlogd-produced logfiles

[Alex Efros]

Hi!

On Sat, Jun 17, 2006 at 10:48:52PM -0400, George Georgalis wrote:
> Debian Woody (internal... migrating to BSD)
> 
>  $ tail --version
> tail (GNU textutils) 2.0

:D

textutils-2.0 was released in Aug 1999! 7 years! cool... :)

(Gentoo)
$ tail --version
tail (GNU coreutils) 5.94
Copyright (C) 2006 Free Software Foundation, Inc.

-- 
			WBR, Alex.

Re: monitoring svlogd-produced logfiles

[Alex Efros]

Hi!

There a lot of ways to do what you need and I think you overcomplicate.

If you don't need svlogd writing to disc and wish to redirect it output
to some 'summarize' program, then why not use that program instead of
svlogd in ./log/run? Main svlogd task is reliable _writing_to_disc_!

But you can configure svlogd to duplicate your logs (optionally filtered)
to STDERR (in addition to writing to disc). In degenerate case you even
can configure svlogd to write nothing to disc and send all log lines to
STDERR, but I don't understand why you need svlogd in this case. :)

Then, in ./log/run you redirect svlogd's STDERR to some FIFO file:
    svlogd /var/log/something 2>/var/log/MYFIFO
(you can configure a lot of different svlogd to output into single FIFO)
and setup special service which will fetch data from /var/log/MYFIFO,
summarize it, print colored to /dev/tty12 or everything - ./run example:
    exec my_cool_summarizer <>/var/log/MYFIFO

Only one important note about FIFOs - if your 'my_cool_summarizer'
service will be down and will not read from /var/log/MYFIFO all other
svlogd which write into FIFO will also stop soon (after they fill kernel
buffers) and in turn will stop services which they logging.

-- 
			WBR, Alex.

Re: monitoring svlogd-produced logfiles

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 2863 bytes --]

* Alex Efros <powerman@powerman.asdfGroup.com> [2006-06-15 02:59:09 +0300]:

> There a lot of ways to do what you need and I think you overcomplicate.

Very possible... =)

> If you don't need svlogd writing to disc and wish to redirect it output
> to some 'summarize' program, then why not use that program instead of
> svlogd in ./log/run? Main svlogd task is reliable _writing_to_disc_!

No, I do need svlogd to write to disk.  Well, maybe not.  I'm using
socklog for my logging so, for instance, I have /var/log/system/all
which has a config of "*.*".  I can duplicate that to something else,
but svlogd is handling all of the logging from socklog.  So I do need
svlogd to do it's thing (unless I setup another socklog service to read
/dev/log... can I have two socklog processes doing that?)

> But you can configure svlogd to duplicate your logs (optionally filtered)
> to STDERR (in addition to writing to disc). In degenerate case you even
> can configure svlogd to write nothing to disc and send all log lines to
> STDERR, but I don't understand why you need svlogd in this case. :)

No, I need svlogd to write the logs to disk.  In addition to the on-disk
logs, I want something to process those logs in realtime... to make a
summary log and something that I can have sitting open writing matched
entries to STDOUT so I can view it over ssh on my desktop... in
realtime.

> Then, in ./log/run you redirect svlogd's STDERR to some FIFO file:
>     svlogd /var/log/something 2>/var/log/MYFIFO
> (you can configure a lot of different svlogd to output into single FIFO)
> and setup special service which will fetch data from /var/log/MYFIFO,
> summarize it, print colored to /dev/tty12 or everything - ./run example:
>     exec my_cool_summarizer <>/var/log/MYFIFO

What does <> do?  I've never seen that notation before.

> Only one important note about FIFOs - if your 'my_cool_summarizer'
> service will be down and will not read from /var/log/MYFIFO all other
> svlogd which write into FIFO will also stop soon (after they fill kernel
> buffers) and in turn will stop services which they logging.

Right... if there's nothing to empty the buffer, it'll block svlogd.
That's fine... I can make sure, via dependency handling, that the
summarizing service is started first.

Some ideas... =)

BTW, I think I sent the response on the "tail -F" thing to you and not
the list... to summ, I did try "-n 0 -F" and it didn't follow.  Could be
a swatch thing tho... I'll have to try it again and see if it still
doesn't work.  If it doesn't, I still need to find an alternative to it,
so any suggestions are welcome.

-- 
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}
mysql> SELECT * FROM users WHERE clue > 0;
Empty set (0.00sec)
:: Annvix - Secure Linux Server: http://annvix.org/ ::

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: monitoring svlogd-produced logfiles

[Alex Efros]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

Hi!

On Wed, Jun 14, 2006 at 06:10:05PM -0600, Vincent Danen wrote:
> No, I do need svlogd to write to disk.  Well, maybe not.  I'm using

You'd better clarify itself, to write or not to write. ;-)

> /dev/log... can I have two socklog processes doing that?)

You don't need two socklog reading from /dev/log. ;) Anyway, it's unix
socket, not a fifo... I don't think you can have two listeners on single
unix socket.

> No, I need svlogd to write the logs to disk.  In addition to the on-disk
> logs, I want something to process those logs in realtime... to make a
> summary log and something that I can have sitting open writing matched
> entries to STDOUT so I can view it over ssh on my desktop... in
> realtime.

Duplicating messages to STDERR from svlogd will allow you to receive them
in realtime.

> What does <> do?  I've never seen that notation before.

man bash.
Open read/write to workaround case there nobody open this fifo for writing yet.

> BTW, I think I sent the response on the "tail -F" thing to you and not
> the list... to summ, I did try "-n 0 -F" and it didn't follow.  Could be
> a swatch thing tho... I'll have to try it again and see if it still
> doesn't work.  If it doesn't, I still need to find an alternative to it,
> so any suggestions are welcome.

Yep. To summ my reply :) I suggest using script-wrapper around tail to
a) force needed params for tail
b) debug which param swatch try to use when running tail (without hard
things like strace)

-- 
			WBR, Alex.

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: monitoring svlogd-produced logfiles

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

* Alex Efros <powerman@powerman.asdfGroup.com> [2006-06-15 03:25:26 +0300]:

> On Wed, Jun 14, 2006 at 06:10:05PM -0600, Vincent Danen wrote:
> > No, I do need svlogd to write to disk.  Well, maybe not.  I'm using
> 
> You'd better clarify itself, to write or not to write. ;-)

Sorry... I write like I think, which is "out loud".. =)

> > /dev/log... can I have two socklog processes doing that?)
> 
> You don't need two socklog reading from /dev/log. ;) Anyway, it's unix
> socket, not a fifo... I don't think you can have two listeners on single
> unix socket.

Oh yes, right, sorry.  duh.  nevermind... been a long day... =)

> > No, I need svlogd to write the logs to disk.  In addition to the on-disk
> > logs, I want something to process those logs in realtime... to make a
> > summary log and something that I can have sitting open writing matched
> > entries to STDOUT so I can view it over ssh on my desktop... in
> > realtime.
> 
> Duplicating messages to STDERR from svlogd will allow you to receive them
> in realtime.

I didn't realize svlogd could do that.  Very good to know.

> > What does <> do?  I've never seen that notation before.
> 
> man bash.
> Open read/write to workaround case there nobody open this fifo for writing yet.

Cool, thanks.  Good to know.

> > BTW, I think I sent the response on the "tail -F" thing to you and not
> > the list... to summ, I did try "-n 0 -F" and it didn't follow.  Could be
> > a swatch thing tho... I'll have to try it again and see if it still
> > doesn't work.  If it doesn't, I still need to find an alternative to it,
> > so any suggestions are welcome.
> 
> Yep. To summ my reply :) I suggest using script-wrapper around tail to
> a) force needed params for tail
> b) debug which param swatch try to use when running tail (without hard
> things like strace)

Well, I can see what swatch is doing just by executing ps... I can see
that it calls it with the right args.  I'll see tomorrow if it picks up
the new ./current file after it rotates (about another 300k left before
it rotates).

Thanks for the brain prods... =)


-- 
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}
mysql> SELECT * FROM users WHERE clue > 0;
Empty set (0.00sec)
:: Annvix - Secure Linux Server: http://annvix.org/ ::

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: monitoring svlogd-produced logfiles

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

* Vincent Danen <vdanen@linsec.ca> [2006-06-14 21:32:04 -0600]:

> > > BTW, I think I sent the response on the "tail -F" thing to you and not
> > > the list... to summ, I did try "-n 0 -F" and it didn't follow.  Could be
> > > a swatch thing tho... I'll have to try it again and see if it still
> > > doesn't work.  If it doesn't, I still need to find an alternative to it,
> > > so any suggestions are welcome.
> > 
> > Yep. To summ my reply :) I suggest using script-wrapper around tail to
> > a) force needed params for tail
> > b) debug which param swatch try to use when running tail (without hard
> > things like strace)
> 
> Well, I can see what swatch is doing just by executing ps... I can see
> that it calls it with the right args.  I'll see tomorrow if it picks up
> the new ./current file after it rotates (about another 300k left before
> it rotates).

well, using "tail -n 0 -F" seems to work.  So it must have been the
other option "--max-unchanged-stats=3" that was causing grief.

Thanks again for the mental kicks.

-- 
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}
mysql> SELECT * FROM users WHERE clue > 0;
Empty set (0.00sec)
:: Annvix - Secure Linux Server: http://annvix.org/ ::

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]