svlogd: blocking on ENOSPC

svlogd: blocking on ENOSPC

[Gerrit Pape]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

Hi, svlogd, just as multilog, sleeps and retries in case it's unable to
write to the current logfile in a log directory and the system is ``out
of disk space''.  This usually makes the service block because svlogd
stops reading from the pipe.  Even though the space for logs can be
calculated properly, if a log partition fills up by accident, it can
bring the system in a troublesome state.

Here's a patch that makes svlogd remove the oldest of rotated log files
in the log directory in case the disk is full.  In the worst case only
the current logfile will be left.  I would be interested in your
opinions.

Regards, Gerrit.

[-- Attachment #2: diff --]
[-- Type: text/plain, Size: 1676 bytes --]

diff -u -r1.7 -r1.8
--- src/svlogd.c	7 Oct 2004 07:56:29 -0000	1.7
+++ src/svlogd.c	7 Oct 2004 14:40:56 -0000	1.8
@@ -287,8 +287,41 @@
     if (len > ((dir +n)->sizemax -(dir +n)->size))
       len =(dir +n)->sizemax -(dir +n)->size;
   }
-  while ((i =write((dir +n)->fdcur, s, len)) == -1)
+  while ((i =write((dir +n)->fdcur, s, len)) == -1) {
+    if (errno == ENOSPC) {
+      DIR *d;
+      direntry *f;
+      char oldest[FMT_PTIME];
+
+      while (fchdir((dir +n)->fddir) == -1)
+        pause2("unable to change directory, want remove old logfile",
+               (dir +n)->name);
+      oldest[0] ='A'; oldest[1] =oldest[27] =0;
+      while (! (d =opendir(".")))
+        pause2("unable to open directory, want remove old logfile",
+               (dir +n)->name);
+      errno =0;
+      while ((f =readdir(d)))
+        if ((f->d_name[0] == '@') && (str_len(f->d_name) == 27))
+          if (str_diff(f->d_name, oldest) < 0) byte_copy(oldest, 27, f->d_name);
+      if (errno) warn2("unable to read directory, want remove old logfile",
+                       (dir +n)->name);
+      closedir(d);
+      if (*oldest == '@') {
+        strerr_warn5(WARNING, "out of disk space, delete: ", (dir +n)->name,
+                     "/", oldest, 0);
+        if (unlink(oldest) == -1)
+          warn2("unable to unlink oldest logfile", (dir +n)->name);
+        else
+          i =1;
+      }
+      while (fchdir(fdwdir) == -1)
+        pause1("unable to change to initial working directory");
+      if (i) continue;
+    }
     pause2("unable to write to current", (dir +n)->name);
+  }
+
   (dir +n)->size +=i;
   if ((dir +n)->sizemax)
     if (s[i -1] == '\n')

Re: svlogd: blocking on ENOSPC

[Csillag Tamás]

On 10/18, Gerrit Pape wrote:
> Hi, svlogd, just as multilog, sleeps and retries in case it's unable to
> write to the current logfile in a log directory and the system is ``out
> of disk space''.  This usually makes the service block because svlogd
> stops reading from the pipe.  Even though the space for logs can be
> calculated properly, if a log partition fills up by accident, it can
> bring the system in a troublesome state.
> 
> Here's a patch that makes svlogd remove the oldest of rotated log files
> in the log directory in case the disk is full.  In the worst case only
> the current logfile will be left.  I would be interested in your
> opinions.
Interesting, It would be nice as a command line switch like '-d'
(or maybe even better another line in config?)
So the admin can decide what is more important to keep the service
always up or to be sure the logs are preserved.

If you turn on debugging on one service (e.g. openldap) it is better to remove only
it's logs and not the others (e.g. auth logs).

What do you think?

-- 
cstamas

Re: svlogd: blocking on ENOSPC

[Gerrit Pape]

On Mon, Nov 01, 2004 at 11:00:29PM +0100, Csillag Tam?s wrote:
> On 10/18, Gerrit Pape wrote:
> > Here's a patch that makes svlogd remove the oldest of rotated log
> > files in the log directory in case the disk is full.  In the worst
> > case only the current logfile will be left.  I would be interested
> > in your opinions.
> Interesting, It would be nice as a command line switch like '-d' (or
> maybe even better another line in config?) So the admin can decide
> what is more important to keep the service always up or to be sure the
> logs are preserved.
> 
> If you turn on debugging on one service (e.g. openldap) it is better
> to remove only it's logs and not the others (e.g. auth logs).
> 
> What do you think?

I think it should be a line starting with ``N'' in the config file to
optionally specify the minimum number of older log files to be kept in
case the filesystem is full.  By default it's equal to the maximum
number of files.

Regards, Gerrit.

Re: svlogd: blocking on ENOSPC

[Csillag Tamas]

On 11/02, Gerrit Pape wrote:
> ...
> I think it should be a line starting with ``N'' in the config file to
> optionally specify the minimum number of older log files to be kept in
> case the filesystem is full.  By default it's equal to the maximum
> number of files.
Nice ;-)

Thanks.
-- 
cstamas