svlogd research

svlogd research

[Larry Doolittle]

Gerrit and friends -

I'm trying to create a tricky (for me) setup for svlogd, and have
some questions.  I want both a reliable local log, and an encrypted
(via stunnel) remote log, where I can't control the reliability
of the remote logging machine (or the network path to it).  My
environment is Debian sid amd64, currently runit 1.8.0-1.

So my /etc/runit/proggie/log/run file must look something like

#!/bin/sh
exec svlogd -t /var/log/proggie_local /var/log/proggie_remote

where /var/log/proggie_local is familiar territory.
/var/log/proggie_remote doesn't look too bad.  Every couple of
minutes I want to dump the accumulated logs (if any) to the remote
server, so I have lines in that directory's config file like

t120
! /usr/sbin/stunnel blah blah blah -r syslog.example.org:5140

My core question is what happens when stunnel fails or stalls.
The svlogd man page says

  A  processor  is  run  in  the background.  If svlogd sees a previously
  started processor still running when trying to start a new one for  the
  same  log, it blocks until the currently running processor has finished
  successfully.  Only the HUP signal works in that situation.  Note that
  this may block any program feeding its log data to svlogd.

That would be bad, since the same svlogd process is also responsible for
the local log.  Network stalls shouldn't affect proggie or its local logging.
When I look in the code for how this is supposed to work, I see the check
for a still-running processor for the previous data set

  if (ld->ppid) {
    warnx("processor already running", ld->name);
    return(0);
  }

but no caller to this routine checks for an exit code, and nothing
retries the call to processorstart().  So it looks like if one processor
run stalls indefinitely, no additional data will ever get sent to the
processor, and the svlogd process never stalls.  This doesn't sound like
what the documentation says.  Is my analysis correct?  Am I on the right
track in general?

  - Larry

Re: svlogd research

[Charlie Brady]

On Fri, 16 Nov 2007, Larry Doolittle wrote:

> I want both a reliable local log, and an encrypted (via stunnel) remote 
> log, where I can't control the reliability of the remote logging machine 
> (or the network path to it).

My recommendation is that you log locally and periodically (often) mirror 
the log directory remotely. The mirror algorithm could use the fact that 
'current' can only grow or be renamed. IIUC, all you need to do is send 
any log file not already on the remote server and then send either all of 
current, or the balance of current, depending on whether rotation has 
occurred since the last mirror cycle.

Or you could just rsync, which will be a little more wasteful.

Re: svlogd research

[Larry Doolittle]

Friends -

On Fri, Nov 16, 2007 at 03:10:44PM -0500, Charlie Brady wrote:
> On Fri, 16 Nov 2007, Larry Doolittle wrote:
>
>> I want both a reliable local log, and an encrypted (via stunnel) remote 
>> log, where I can't control the reliability of the remote logging machine 
>> (or the network path to it).
>
> My recommendation is that you log locally and periodically (often) mirror 
> the log directory remotely.

Interesting suggestion.  I can't easily use it, because I also don't
control the software running on the remote logging machine -- it is
syslogd wrapped in SSL, non-negotiable.

I just found Gerrit's tryto(1).  I think that practically gets rid of
my possible problem, although it would still be nice to resolve my question
about syslogd's behavior when its processor gets stuck.

   - Larry

Re: svlogd research

[George Georgalis]

On Fri, Nov 16, 2007 at 12:23:13PM -0800, Larry Doolittle wrote:
>Friends -
>
>On Fri, Nov 16, 2007 at 03:10:44PM -0500, Charlie Brady wrote:
>> On Fri, 16 Nov 2007, Larry Doolittle wrote:
>>
>>> I want both a reliable local log, and an encrypted (via stunnel) remote 
>>> log, where I can't control the reliability of the remote logging machine 
>>> (or the network path to it).
>>
>> My recommendation is that you log locally and periodically (often) mirror 
>> the log directory remotely.
>
>Interesting suggestion.  I can't easily use it, because I also don't
>control the software running on the remote logging machine -- it is
>syslogd wrapped in SSL, non-negotiable.

in addition to your local log dir, you could also write to a fifo
where another program reads and writes to your remote syslogd.
Of course if you loose the remote connection there could be a
blocking issue.

another way is to use a processor to send log entries at rotation
time, and sent the alarm signal to the log daemon at what ever
interval you need to keep the remote updated. again, if you cannot
connect there will be a block, when you try to alarm the next time.

yet another way is to use a processor to signal another program to
start the remote connection and dump the new logfile (by alarm if
necessary). that could be done any number of ways and wouldn't block
if it failed, but watch out that svlogd doesn't rotate away log files
that haven't been transferred yet.

// George


-- 
George Georgalis, information system scientist <IXOYE><