apache2 logs -> svlogd

apache2 logs -> svlogd

[Alex Efros]

Hi!

I've just got an idea how to send all apache2 logs to svlogd, and at
first glance this idea looks not so ugly as all other ways I've seen
before to accomplish this task. :-)

Usually apache2 configured in way similar to this: there exists directory
/var/log/apache2/ and it write logs into number of different files (I've
seven: access_log, deflate_log, error_log, rewrite.log, ssl_access_log,
ssl_error_log, ssl_request_log). My idea is replace these files with FIFO,
create svlogd directory for each of these FIFO and create separate
services for each log file with ./run files like this:

---[ /service/apache2_access_log/run ]---
#!/bin/sh
exec >&2
exec svlogd /var/log/apache2/access/   <>/var/log/apache2/access_log
---[ /service/apache2_access_log/run ]---

Also these FIFO must have 0600 permissions, or some users may partially
intercept log messages with simple: cat /var/log/apache2/access_log

This will work without any changes in apache's configuration, but it may
has sense to change log files format, for ex., to replace apache's
datetime with svlogd's.

There should be no problem in splitting these logs 'per virtual host'
using svlogd's ./config into different directories and giving to different
users permissions to read logs only for their virtual hosts.

Because of bash's '<>' svlogd will get in STDIN file (FIFO) opened
both for read&write, so it will not block waiting for other FIFO's side;
also if you restart apache svlogd will not die because it never get EOF.

What you think about this idea? Any flaws?

I don't know what happens if apache will try to write in these FIFO while
log service(s) isn't running. In my quick tests looks like apache just
drop lines which it unable to write in log if I 'sv down' log services.
But if log services not running before apache will start - apache will not
start at all, of course (probably can be solved by same '<>' trick in
apache's ./run file, but I don't think this is really needed).

-- 
			WBR, Alex.

Re: apache2 logs -> svlogd

[Torne Wuff]

On Mon, Feb 13 06 at  7:38:13PM +0200, Alex Efros wrote:
> I've just got an idea how to send all apache2 logs to svlogd, and at
> first glance this idea looks not so ugly as all other ways I've seen
> before to accomplish this task. :-)

Apache can do this without any external help from FIFOs etc, just use a
configuration like this:

ErrorLog "|/usr/bin/chpst -ulogger /usr/bin/svlogd /var/log/apache2/error"
CustomLog "|/usr/bin/chpst -ulogger /usr/bin/svlogd /var/log/apache2/accesslog" combined

and so on for however many logs you want in different places. These
commands are run as the user that started apache (usually root) not the
user that apache setuid()s to, so you can call chpst and change user.

Any log string that starts with a pipe is assumed to be a binary to run.

-- 
Torne

Re: apache2 logs -> svlogd

[Alex Efros]

Hi!

On Mon, Feb 13, 2006 at 06:04:02PM +0000, Torne Wuff wrote:
> Apache can do this without any external help from FIFOs etc, just use a
> configuration like this:

I know about this. But AFAIK there some problems with this way, for ex.:

    ----- Forwarded message from Joshua N Pritikin <jpritikin@pobox.com> -----
    Date: Wed, 7 Dec 2005 17:35:42 +0530
    From: Joshua N Pritikin <jpritikin@pobox.com>
    To: supervision@list.skarnet.org
    Subject: svlogd choaking on lock

    Hi!

    I'm using svlogd for apache2 access logs.  However, this is not working
    too well.  The problem is that when apache restarts, say a graceful USR1
    restart, then it kills the old svlogd and starts a new one fast enough
    that I almost always get:

    svlogd: warning: unable to lock directory: /var/log/apache2/access:
    temporary failure
    svlogd: fatal: no functional log directories.

    The error log was exhibiting the same problem but then I figured out how
    to route the ErrorLog through /dev/fd/3 and then runsv took care of not
    restarting svlogd.  So only the access log problem remains.

    Suggestions?

    -- 
    Make April 15 just another day, visit http://fairtax.org

    ----- End forwarded message -----

Recommended solution was put 'sleep 1' before svlogd, but for me it's a
little ugly and unreliable.

-- 
			WBR, Alex.

Re: apache2 logs -> svlogd

[George Georgalis]

On Mon, Feb 13, 2006 at 08:25:20PM +0200, Alex Efros wrote:
>
>    ----- Forwarded message from Joshua N Pritikin <jpritikin@pobox.com> -----
>    Date: Wed, 7 Dec 2005 17:35:42 +0530
>    From: Joshua N Pritikin <jpritikin@pobox.com>
>    To: supervision@list.skarnet.org
>    Subject: svlogd choaking on lock
>
>    Hi!
>
>    I'm using svlogd for apache2 access logs.  However, this is not working
>    too well.  The problem is that when apache restarts, say a graceful USR1
>    restart, then it kills the old svlogd and starts a new one fast enough
>    that I almost always get:
>
>    svlogd: warning: unable to lock directory: /var/log/apache2/access:
>    temporary failure
>    svlogd: fatal: no functional log directories.
>
>    The error log was exhibiting the same problem but then I figured out how
>    to route the ErrorLog through /dev/fd/3 and then runsv took care of not
>    restarting svlogd.  So only the access log problem remains.
>
>    Suggestions?
>
>    -- 
>    Make April 15 just another day, visit http://fairtax.org
>
>    ----- End forwarded message -----
>
>Recommended solution was put 'sleep 1' before svlogd, but for me it's a
>little ugly and unreliable.

I just checked that thread on gmane.

----------------------------------------
Date: 2005-12-07 16:04:02 GMT
From: Paul Jarc <prj <at> po.cwru.edu>
Subject: Re: svlogd choaking on lock
Newsgroups: gmane.comp.sysutils.supervision.general

Joshua N Pritikin <jpritikin <at> pobox.com> wrote:
> The problem is that when apache restarts, say a graceful USR1
> restart, then it kills the old svlogd and starts a new one fast enough
> that I almost always get:
>
> svlogd: warning: unable to lock directory: /var/log/apache2/access: temporary failure
> svlogd: fatal: no functional log directories.

You could change your svlogd invocation to "sleep 1 && svlogd ...".

paul

----------------------------------------


Since I only use svlogd for ErrorLog and one process for each
virtual domain, I'm interested in making this approach work. But
I don't completely understand what the problem is; or the downside
of using "sleep 1 && svlogd ..." -- I understand it would use
ulimit resources (filling the pipe) till svlogd starts, but so
what? Something ugly _might_ be apache pipe exec... something more
reliable might be a loop to restart svlogd if it fails. Thoughts?

// George


-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: apache2 logs -> svlogd

[Joshua N Pritikin]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On Tue, Feb 14, 2006 at 09:25:32AM -0500, George Georgalis wrote:
> >Recommended solution was put 'sleep 1' before svlogd, but for me it's a
> >little ugly and unreliable.

What isn't ugly about creating named pipes?  I don't think the named
pipe solution wins the beauty contest either.

Ultimately we may need to go beat up apache developers.  ;-)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: apache2 logs -> svlogd

[Alex Efros]

Hi!

On Tue, Feb 14, 2006 at 08:17:06PM +0530, Joshua N Pritikin wrote:
> On Tue, Feb 14, 2006 at 09:25:32AM -0500, George Georgalis wrote:
> > >Recommended solution was put 'sleep 1' before svlogd, but for me it's a
> > >little ugly and unreliable.
> 
> What isn't ugly about creating named pipes?  I don't think the named
> pipe solution wins the beauty contest either.

Named pipes... In short, I think named pipes is better because:
- this is universal solution which can be used for any daemon which prefer
  to write log files itself, for example: Xorg :) , xdm, ntpd, emerge :) ,
  iptraf, mysql, oops (webproxy), samba... maybe some of them can be
  configured to use svlogd, but not all (so SINGLE UNIVERSAL UGLY solution
  is much better than number of different custom more-or-less ugly
  solutions)
- svlogd for named pipe supervised by runsv and won't restart with it
  service - no problems with sleep 1 and so
- svlogd can be reconfigured&restarted without restarting service(apache)
- you can have many services sending their logs into same named pipe
  (can't imagine how this can be used in real world yet :))
- ... anything else?

> Ultimately we may need to go beat up apache developers.  ;-)

YEAH! :-)

-- 
			WBR, Alex.

Re: apache2 logs -> svlogd

[George Georgalis]

On Tue, Feb 14, 2006 at 05:58:38PM +0200, Alex Efros wrote:
>On Tue, Feb 14, 2006 at 08:17:06PM +0530, Joshua N Pritikin wrote:
>> On Tue, Feb 14, 2006 at 09:25:32AM -0500, George Georgalis wrote:
>> > >Recommended solution was put 'sleep 1' before svlogd, but for me it's a
>> > >little ugly and unreliable.

I quoted that; I didn't say it, though I was quoted as doing so a
few messages back.  It was the context of my question. 'sleep 1'
does seem reasonable to me, though not correct and therefore not
reliable... the respawn svlogd, as required, approach seems hardly
justified though.

Yet, I don't completely understand why it is proported to work --
part of the reason I quoted it. (I have an unrelated problem below.)

>> What isn't ugly about creating named pipes?  I don't think the named
>> pipe solution wins the beauty contest either.

I'm curious about Alex's runscript:

---[ /service/apache2_access_log/run ]---                                                                           
#!/bin/sh                                                                                                           
exec >&2                                                                                                            
exec svlogd /var/log/apache2/access/   <>/var/log/apache2/access_log                                                
---[ /service/apache2_access_log/run ]---    

because I don't understand the use of '<>'. Any comments or links
on this are welcome -- though it doesn't address my needs, I use a
standard access logfile for each virtual domain, and would like
to use a svlogd for each domain's error log; both cases to manage
rotation. Not sure an access (or error) service will help me,
besides the apache logfile is being written (in addition) which is
what I thought was being avoided by using svlogd...

>Named pipes... In short, I think named pipes is better because:
>- this is universal solution which can be used for any daemon which prefer
>  to write log files itself, for example: Xorg :) , xdm, ntpd, emerge :) ,
>  iptraf, mysql, oops (webproxy), samba... maybe some of them can be
>  configured to use svlogd, but not all (so SINGLE UNIVERSAL UGLY solution
>  is much better than number of different custom more-or-less ugly
>  solutions)
>- svlogd for named pipe supervised by runsv and won't restart with it
>  service - no problems with sleep 1 and so
>- svlogd can be reconfigured&restarted without restarting service(apache)
>- you can have many services sending their logs into same named pipe
>  (can't imagine how this can be used in real world yet :))
>- ... anything else?

...I think we have a different set of problems and/or concerns...

in any event, I won't be doing any apache2 svlogd in the near future,
per my unanswered post (log to pipe not working in pkgsrc apache2) on
the pkgsrc list, below.

>> Ultimately we may need to go beat up apache developers.  ;-)

Count me in!

Cheers,
// George


----- Forwarded message from George Georgalis <george@galis.org> -----

Date: Tue, 14 Feb 2006 13:21:25 -0500
Message-ID: <20060214182125.GD1378@sta.duo>
Return-Path: bounces-tech-pkg-owner-george=galis.org@NetBSD.org
From: George Georgalis <george@galis.org>
To: tech-pkg@netbsd.org
Subject: www/apache2 not accepting command for ErrorLog

The pkgsrc apache2 is not happy with a command for ErrorLog,
as documented here

http://httpd.apache.org/docs/2.2/mod/core.html#errorlog

It surprises me that this would be broken. Is there something wrong
with the package? With the original source? Or my execution?

In this httpd.conf fragment, the former works

#ErrorLog /tmp/apache2.ErrorLog
ErrorLog "|/bin/cat >>/tmp/apache2.ErrorLog"

but the latter causes apache to exit with the error:

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

I'm quite certain I have free space and inodes, seems like apache2
is trying to do locking on the exec? but what does mod_rewrite
have to do with it? I don't load it.  My configuration is generic
accept for the invocation

/usr/pkg/sbin/httpd -DNO_DETACH -f /usr/local/etc/apache2/httpd.conf -e debug -E /tmp/apache.start.err

and a very standard NameVirtualHost.

// George


-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org


----- End forwarded message -----

-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: apache2 logs -> svlogd

[Dean Hall]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

On Feb 15, 2006, at 10:09 AM, George Georgalis wrote:

<snip>

> I use a
> standard access logfile for each virtual domain, and would like
> to use a svlogd for each domain's error log; both cases to manage
> rotation. Not sure an access (or error) service will help me,
> besides the apache logfile is being written (in addition) which is
> what I thought was being avoided by using svlogd...

I think the named pipes discussion is appropriate to your problem,  
actually. In this situation, I would probably:

1. Create a log directory per-virtual host.
2. Create "error" and "access" subdirectories of all of those.
3. In each "error" and "access" subdirectory, I would create a fifo.
4. For each virtual host, I would configure apache to write access  
logs to the "access" fifo in the appropriate directory; likewise with  
error logs, except to "error".
5. For each fifo, create a service that reads from it (exec < / 
whatever/access/fifo) and execs svlogd in that directory.

The only problem with this is that apache might block until the  
svlogd service is reading from the fifo. The log services aren't  
really "log services" in the runsv sense, so you'd need to handle  
dependencies manually such that they start before apache does. (Some  
thing like <http://sysinit.sourceforge.net> might work, but I haven't  
touched it in a while, and I need to make some bugfixes.)

>>> Ultimately we may need to go beat up apache developers.  ;-)

For more reasons than this, definitely. I'll be so happy when I can  
move to lighttpd.


--
Dean Hall
http://deanpence.com

The Cornhole Show
http://thecornholeshow.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 155 bytes --]