svlogd as a heavy-duty logger

svlogd as a heavy-duty logger

[Dean Hall]

I work for a company that has a dire need for a syslog replacement, and
as a long-time daemontools and runit user, I'd like to propose a svlogd
(or similar) solution for us.

The main reservation I have is the robustness/scalability of svlogd.
Right now at our largest data center we're getting pretty constant log
input from around 40 machines. We're using syslog-ng, and it suffers
from some rather fatal problems when coupled with the traditional
logrotate program. I don't want to reintroduce stability issues with a
different technology, and my experience with svlogd is limited to
running a syslog-ish server with it (and socklog) on my personal,
three-computer network. Has anyone load-tested svlogd, or does anyone
have experience with svlogd's behavior under a heavy load?


d

Re: svlogd as a heavy-duty logger

[Enrico Scholz]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

deanpence@gmail.com (Dean Hall) writes:

> I work for a company that has a dire need for a syslog replacement, and
> as a long-time daemontools and runit user, I'd like to propose a svlogd
> (or similar) solution for us.
>
> The main reservation I have is the robustness/scalability of svlogd.
> ...
> Has anyone load-tested svlogd, or does anyone have experience with
> svlogd's behavior under a heavy load?

I played some time with svlogd in an environment (Linux-VServer) where
multiple svlogd daemons (>20) run on the same machine. But this solution
showed too bloated because svlogd is too minimalistic and does not support
basic functions:

* encryption of logstream is possible by an external program only which
  must be spawned at each logrotation; because of SSL key exchange this
  is a very heavyweight operation

* logrotation is very unflexible; low-volume logs will be transmitted very
  seldom and svlogd does not support a way to rotate them periodically. You
  will need external programs started e.g. by cronjobs which send SIGALRM
  to svlogd. But this will have bad sideeffects as it rotates all logfiles
  but not only the low-volume ones.


At the end, I switched to syslog-ng. It offers more flexibility and is
more lightweighted.




Enrico

[-- Attachment #2: Type: application/pgp-signature, Size: 480 bytes --]

Re: svlogd as a heavy-duty logger

[Gerrit Pape]

On Thu, May 19, 2005 at 12:21:17AM +0200, Enrico Scholz wrote:
> * logrotation is very unflexible; low-volume logs will be transmitted very
>   seldom and svlogd does not support a way to rotate them periodically. You
>   will need external programs started e.g. by cronjobs which send SIGALRM
>   to svlogd. But this will have bad sideeffects as it rotates all logfiles
>   but not only the low-volume ones.

Hi Enrico, svlogd supports the t configuration option since version
1.1.0 as per your suggestion in this thread last year

 http://thread.gmane.org/gmane.comp.sysutils.supervision.general/569
 http://article.gmane.org/gmane.comp.sysutils.supervision.general/607
 http://article.gmane.org/gmane.comp.sysutils.supervision.general/619

Regards, Gerrit.

Re: svlogd as a heavy-duty logger

[Dean Hall]

Enrico,

> > Has anyone load-tested svlogd, or does anyone have experience with
> > svlogd's behavior under a heavy load?
> 
> I played some time with svlogd in an environment (Linux-VServer) where
> multiple svlogd daemons (>20) run on the same machine. But this solution
> showed too bloated because svlogd is too minimalistic and does not support
> basic functions:
> 
> * encryption of logstream is possible by an external program only which
>   must be spawned at each logrotation; because of SSL key exchange this
>   is a very heavyweight operation

I don't need to encrypt anything. Also, we'd probably only use one
svlogd process as a child of socklog, which would then, I assume,
spawn an svlogd child for each log directory. Maybe someone can
correct me here.

> * logrotation is very unflexible; low-volume logs will be transmitted very
>   seldom and svlogd does not support a way to rotate them periodically. You
>   will need external programs started e.g. by cronjobs which send SIGALRM
>   to svlogd. But this will have bad sideeffects as it rotates all logfiles
>   but not only the low-volume ones.

I'm not sure what you mean. You can use the "s" option in the config
file to specify a size at which to rotate, and, as Gerrit pointed out,
you can now use the "t" option to specify the age in seconds at which
to rotate. So if you want small logs rotated or not, you can do it.

None of this is an issue for me anyway.

> At the end, I switched to syslog-ng. It offers more flexibility and is
> more lightweighted.

I'm not sure what kind of flexibility syslog-ng has. AFAIK, svlogd (+
socklog) can do everything (plus some) that syslog-ng can. Like I
said, we're using syslog-ng, and it's not good enough for us.


Dean