multiple log streams

multiple log streams

[Joan Picanyol i Puig]

Hi,

For a system I'm working on, I'd like to reliably store three different
log streams from a single process. Since runsv only sets up a single
pipe from a service/ to its log/, I believe I should demultiplex the
streams in log/run and feed them there to three diferent svlogd
processes, but I'm not quite clear on how to go about it. Any
suggestions?

tks
-- 
pica

Re: multiple log streams

[Alex Efros]

Hi!

On Thu, Apr 02, 2009 at 10:47:42AM +0200, Joan Picanyol i Puig wrote:
> For a system I'm working on, I'd like to reliably store three different
> log streams from a single process. Since runsv only sets up a single
> pipe from a service/ to its log/, I believe I should demultiplex the
> streams in log/run and feed them there to three diferent svlogd
> processes, but I'm not quite clear on how to go about it. Any
> suggestions?

a) use single stream, and then use svlogd features to filter messages and
store in different log files

b) run 3 socklog processes as separate services which will provide you
with 3 unix sockets, and manually log (using syslog syscalls or library)
from your application different streams to these 3 sockets

Actually a) is most natural way.

-- 
			WBR, Alex.

Re: multiple log streams

[Joan Picanyol i Puig]

* Alex Efros <powerman@powerman.name> [20090402 10:59]:
> Hi!
> 
> On Thu, Apr 02, 2009 at 10:47:42AM +0200, Joan Picanyol i Puig wrote:
> > For a system I'm working on, I'd like to reliably store three different
> > log streams from a single process. Since runsv only sets up a single
> > pipe from a service/ to its log/, I believe I should demultiplex the
> > streams in log/run and feed them there to three diferent svlogd
> > processes, but I'm not quite clear on how to go about it. Any
> > suggestions?
> 
> a) use single stream, and then use svlogd features to filter messages and
> store in different log files

I need to read from log/current in real-time to check for alarm
conditions which I can't express using svlogd pattern matching.

> b) run 3 socklog processes as separate services which will provide you
> with 3 unix sockets, and manually log (using syslog syscalls or library)
> from your application different streams to these 3 sockets

That sounds feasible. Maybe multitee is an option too...

tks
-- 
pica

Re: multiple log streams

[Alex Efros]

Hi!

On Thu, Apr 02, 2009 at 11:24:04AM +0200, Joan Picanyol i Puig wrote:
> > a) use single stream, and then use svlogd features to filter messages and
> > store in different log files
> I need to read from log/current in real-time to check for alarm
> conditions which I can't express using svlogd pattern matching.

Why do you think so? It works in real-time. You probably think about log
rotation, which isn't real-time, but it has nothing with using multiple
log files and filters.

-- 
			WBR, Alex.

Re: multiple log streams

[Joan Picanyol i Puig]

* Alex Efros <powerman@powerman.name> [20090402 11:49]:
> Hi!
> 
> On Thu, Apr 02, 2009 at 11:24:04AM +0200, Joan Picanyol i Puig wrote:
> > > a) use single stream, and then use svlogd features to filter messages and
> > > store in different log files
> > I need to read from log/current in real-time to check for alarm
> > conditions which I can't express using svlogd pattern matching.
> 
> Why do you think so? It works in real-time. You probably think about log
> rotation, which isn't real-time, but it has nothing with using multiple
> log files and filters.

Rereading the docs, I noticed that svlogd supports multiple log
directories (I still use daemontools...), which does solve my issue. If
only I could specify an action instead of stderr printing with e/E I'd
be a happy puppy.

tks
--
pica

Re: multiple log streams

[Laurent Bercot]

> Rereading the docs, I noticed that svlogd supports multiple log
> directories (I still use daemontools...), which does solve my issue. If
> only I could specify an action instead of stderr printing with e/E I'd
> be a happy puppy.

 multilog supports multiple log directories too.
 However, maybe multilog's pattern-matching abilities aren't powerful
enough for your needs, in which case there's no out-of-the-box solution
and you'll have to do more work (like writing a program able to decide
in real-time where to send the logs).

-- 
 Laurent

Re: multiple log streams

[Joan Picanyol i Puig]

Hi,

6 months later... Please bear with the extra-quoting to include context
from an old thread.

* Joan Picanyol i Puig <lists-supervision@biaix.org> [20090402 10:47]:
> For a system I'm working on, I'd like to reliably store three different
> log streams from a single process.

* Alex Efros <powerman@powerman.name> [20090402 10:59]:
> 
> On Thu, Apr 02, 2009 at 10:47:42AM +0200, Joan Picanyol i Puig wrote:
> > For a system I'm working on, I'd like to reliably store three different
> > log streams from a single process. Since runsv only sets up a single
> > pipe from a service/ to its log/, I believe I should demultiplex the
> > streams in log/run and feed them there to three diferent svlogd
> > processes, but I'm not quite clear on how to go about it. Any
> > suggestions?
> 
> a) use single stream, and then use svlogd features to filter messages and
> store in different log files
> 
> > On Thu, Apr 02, 2009 at 10:47:42AM +0200, Joan Picanyol i Puig wrote:
> I need to read from log/current in real-time to check for alarm
> conditions which I can't express using svlogd pattern matching.

* Alex Efros <powerman@powerman.name> [20090402 11:49]:
 
> Why do you think so? It works in real-time. You probably think about log
> rotation, which isn't real-time, but it has nothing with using multiple
> log files and filters.
> 
* Joan Picanyol i Puig <lists-supervision@biaix.org> [20090402 15:47]:
 
> Rereading the docs, I noticed that svlogd supports multiple log
> directories (I still use daemontools...), which does solve my issue. If
> only I could specify an action instead of stderr printing with e/E I'd
> be a happy puppy.
> 
* Laurent Bercot <ska-supervision@skarnet.org> [20090403 10:20]:
>  multilog supports multiple log directories too.
>  However, maybe multilog's pattern-matching abilities aren't powerful
> enough for your needs, in which case there's no out-of-the-box solution
> and you'll have to do more work (like writing a program able to decide
> in real-time where to send the logs).

And now I'm here (having chosing to write a real-time
application-spefic log filtering progrem) full of FUD:

Assuming producer & consumer are supervise/runsv managed services, how
do I reliably handle producer's log rotation?

My consumer opens /service/producer/log/main/current and gets signaled
by producer/log upon rotation. My consumer main loop yields every
iteration (say, by sleeping every 10ms), and its signal handler for log
rotation sets a global flag. When set, it reads until EOF on
log/main/current 's fd, closes it and reopens it. So far so good (please
correct me if I'm wrong). 

What happens if I get a signal while reading current until EOF? Could I
mess a whole logfile? Is this just a "your producer rotates too fast for
your consumer and you are not blocking" issue? I see two options:

1. in my signal handler shut down my producer's log, re-signal it up
   when done with current log
2. do a careful scheduling analysis (which I should, since I'm talking
    "real-time") and hope for the best

Anything I might have overlooked? Ideas? suggestions?

tks
--
pica

Re: multiple log streams

[Laurent Bercot]

> Assuming producer & consumer are supervise/runsv managed services, how
> do I reliably handle producer's log rotation?

 Why do you even use multilog/svlogd, since you have written your own
logging facility ?
 Make your consumer just read from stdin and log reliably to the
multiple places you need; and have it run as a logger for the producer,
in /service/producer/log/run. You won't have to deal with multilog/
svlogd's rotation.
 Of course, your consumer should implement automatic rotation itself,
but that's a whole different thing from what you are asking.


> My consumer opens /service/producer/log/main/current and gets signaled
> by producer/log upon rotation. My consumer main loop yields every
> iteration (say, by sleeping every 10ms), and its signal handler for log
> rotation sets a global flag. When set, it reads until EOF on
> log/main/current 's fd, closes it and reopens it. So far so good (please
> correct me if I'm wrong). 

 Um, I don't quite get what you're doing, but it sounds like your consumer
is performing active waiting. Don't: use a select()/poll()/iopause() loop
listening on stdin, with a long - perhaps infinite - timeout.
 If you need to trap signals, use the self-pipe trick.
 http://www.skarnet.org/skalibs/selfpipe.html

-- 
 Laurent

Re: multiple log streams

[Charlie Brady]

On Thu, 3 Sep 2009, Laurent Bercot wrote:

> If you need to trap signals, use the self-pipe trick.
> http://www.skarnet.org/skalibs/selfpipe.html

Correct URL is:

http://www.skarnet.org/software/skalibs/selfpipe.html

Re: multiple log streams

[Laurent Bercot]

> Correct URL is:
> http://www.skarnet.org/software/skalibs/selfpipe.html

 Whoops, indeed.

-- 
 Laurent

Re: multiple log streams

[Joan Picanyol i Puig]

* Laurent Bercot <ska-supervision@skarnet.org> [20090903 23:33]:
> > Assuming producer & consumer are supervise/runsv managed services, how
> > do I reliably handle producer's log rotation?
> 
>  Why do you even use multilog/svlogd, since you have written your own
> logging facility ?
>  Make your consumer just read from stdin and log reliably to the
> multiple places you need; and have it run as a logger for the producer,
> in /service/producer/log/run. You won't have to deal with multilog/
> svlogd's rotation.

I have multiple consumers, some real-time and some batch. I need to be
able to easily adapt to more consumers appearing. Batch consumers are
handled at rotation time, real-time must feed from current. In case of >
1 real-time consumers, I should be able to get by with multitee.

> > My consumer opens /service/producer/log/main/current and gets signaled
> > by producer/log upon rotation. My consumer main loop yields every
> > iteration (say, by sleeping every 10ms), and its signal handler for log
> > rotation sets a global flag. When set, it reads until EOF on
> > log/main/current 's fd, closes it and reopens it. So far so good (please
> > correct me if I'm wrong). 
> 
>  Um, I don't quite get what you're doing, but it sounds like your consumer
> is performing active waiting.

I do usleep() every iteration, which does not count as active waiting in
my book. I'm not handling EINTR (I just keep reading until EOF &&
signaled).

> Don't: use a select()/poll()/iopause() loop
> listening on stdin, with a long - perhaps infinite - timeout.
>  If you need to trap signals, use the self-pipe trick.
>  http://www.skarnet.org/skalibs/selfpipe.html

Yep, that's definitely The Right Way.

tks
--
pica

Re: multiple log streams

[Laurent Bercot]

> I have multiple consumers, some real-time and some batch. I need to be
> able to easily adapt to more consumers appearing. Batch consumers are
> handled at rotation time, real-time must feed from current. In case of >
> 1 real-time consumers, I should be able to get by with multitee.

 The problem with multitee is that it doesn't dynamically adapt. You need
to write a couple specific programs so that your requirements are exactly
met.

 If I understand your requirements correctly, I suggest the following
architecture:

 - /service/producer: your producer

 -  /service/producer/log: a specific program (most likely written in C or
perl) that
   * opens a Unix domain socket, binds it to /service/producer/log/s and
listens to it, accepting connections from real-time consumers' uids-gids
   * reads from stdin and reliably writes to every client connection it has.
This is the equivalent of multitee, but it can dynamically adapt to more
or less real-time consumers. You will always have at least one client,
which is...

 - /service/prodlogger: actually just "ipcclient /service/producer/log/s
fdmove 0 6 multilog blahblah" in the run file.
This is your main real-time consumer ; its purpose is just to feed a copy of
the stream to a multilog or svlogd process that reliably saves it to the
filesystem. The logging script is designed to trigger batch consumers.

 - "ipcclient /service/producer/log/s fdmove 0 6 realtimeconsumerprog" can
be how you generally run your real-time consumers, if realtimeconsumerprog
reads the stream on stdin.


> I do usleep() every iteration, which does not count as active waiting in
> my book.

 Sorry, I said "active waiting" but I meant "polling".
 Using notification mechanisms instead of polling mechanisms whenever you
can is clever use of resources. :)

-- 
 Laurent