9front - general discussion about 9front
 help / color / mirror / Atom feed
* [9front] Dealing with log files…
@ 2021-10-16 20:06 igor
  2021-10-16 20:52 ` umbraticus
                   ` (4 more replies)
  0 siblings, 5 replies; 15+ messages in thread
From: igor @ 2021-10-16 20:06 UTC (permalink / raw)
  To: 9front; +Cc: igor

What is the Plan9 way of dealing with growing log files such as
/sys/log/www and others?  Is there something similar to multilog¹ or
logrotate²?

Thanks in advance for any advice.

Cheers,
Igor

¹…(https://cr.yp.to/daemontools/multilog.html)
²…(https://github.com/logrotate/logrotate)


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Dealing with log files…
  2021-10-16 20:06 [9front] Dealing with log files… igor
@ 2021-10-16 20:52 ` umbraticus
  2021-10-16 21:01 ` [9front] " Stanley Lieber
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 15+ messages in thread
From: umbraticus @ 2021-10-16 20:52 UTC (permalink / raw)
  To: 9front

> What is the Plan9 way of dealing with growing log files

I just blast them away every now and then...
You could rig something with cron.

umbraticus

^ permalink raw reply	[flat|nested] 15+ messages in thread

* [9front] Re: [9front] Dealing with log files…
  2021-10-16 20:06 [9front] Dealing with log files… igor
  2021-10-16 20:52 ` umbraticus
@ 2021-10-16 21:01 ` Stanley Lieber
  2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 15+ messages in thread
From: Stanley Lieber @ 2021-10-16 21:01 UTC (permalink / raw)
  To: 9front

On October 16, 2021 8:06:24 PM UTC, igor@9lab.org wrote:
>What is the Plan9 way of dealing with growing log files such as
>/sys/log/www and others?  Is there something similar to multilog¹ or
>logrotate²?
>
>Thanks in advance for any advice.
>
>Cheers,
>Igor
>
>¹…(https://cr.yp.to/daemontools/multilog.html)
>²…(https://github.com/logrotate/logrotate)
>
>

nothing exists. the last time i brought this up there were no takers. trivial to script, although i usually just >file the large files.

sl

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Dealing with log files…
  2021-10-16 20:06 [9front] Dealing with log files… igor
  2021-10-16 20:52 ` umbraticus
  2021-10-16 21:01 ` [9front] " Stanley Lieber
@ 2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
  2021-10-17  9:06   ` umbraticus
                     ` (2 more replies)
  2021-10-17 12:29 ` Rodrigo G. López
  2021-10-17 18:15 ` smj
  4 siblings, 3 replies; 15+ messages in thread
From: Lyndon Nerenberg (VE7TFX/VE6BBM) @ 2021-10-17  8:29 UTC (permalink / raw)
  To: 9front, igor

igor@9lab.org writes:
> What is the Plan9 way of dealing with growing log files such as
> /sys/log/www and others?

Most people just leave them alone. I tend to rotate mine by hand
about once per year.  Remember, once the files make it onto the worm
there's no reclaiming the space anyway.  And if you rename + gzip
them you end up burning up even more space, since you still have
the uncompressed version in the worm.  Even though the old file
is gone from /sys/log, it lives on forever in /n/dump.

The only reason I rotate mine is to speed up grep searches. And acme
gets grumpy when presented with gigabyte-sized text files ...

--lyndon

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Dealing with log files…
  2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
@ 2021-10-17  9:06   ` umbraticus
  2021-10-17 10:30   ` igor
  2021-10-17 14:19   ` [9front] " Stanley Lieber
  2 siblings, 0 replies; 15+ messages in thread
From: umbraticus @ 2021-10-17  9:06 UTC (permalink / raw)
  To: 9front

> Remember, once the files make it onto the worm
> there's no reclaiming the space anyway

oh, yeah, forgot to mention you may want to chmod +t them

umbraticus

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Dealing with log files…
  2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
  2021-10-17  9:06   ` umbraticus
@ 2021-10-17 10:30   ` igor
  2021-10-17 14:19   ` [9front] " Stanley Lieber
  2 siblings, 0 replies; 15+ messages in thread
From: igor @ 2021-10-17 10:30 UTC (permalink / raw)
  To: 9front; +Cc: igor, lyndon

Quoth Lyndon Nerenberg (VE7TFX/VE6BBM) <lyndon@orthanc.ca>:
> igor@9lab.org writes:
> > What is the Plan9 way of dealing with growing log files such as
> > /sys/log/www and others?
> 
> Most people just leave them alone. I tend to rotate mine by hand
> about once per year.  Remember, once the files make it onto the worm
> there's no reclaiming the space anyway.  And if you rename + gzip
> them you end up burning up even more space, since you still have
> the uncompressed version in the worm.  Even though the old file
> is gone from /sys/log, it lives on forever in /n/dump.

Good point. For services with lots of log traffic (e.g. httpd) I mark
log files as temporary to avoid bothering the worm:

% ls -T /sys/log | grep '^t'
t /sys/log/auth
t /sys/log/www

…as temporary files are not backed up.

> The only reason I rotate mine is to speed up grep searches. And acme
> gets grumpy when presented with gigabyte-sized text files ...

Yeah, I have had a similar experience.

 From the feedback so far it seems that general practice is to roll
your own solution. It would be nice to have a simple tool taking
care of that chore. This could be an interesting project, a tool
with a similar feature set as multilog¹ designed the Plan9 way.

I have been looking for a project that is small and self contained…

Cheers,
Igor

¹…(https://cr.yp.to/daemontools/multilog.html)


^ permalink raw reply	[flat|nested] 15+ messages in thread

* [9front] Re: [9front] Dealing with log files…
  2021-10-16 20:06 [9front] Dealing with log files… igor
                   ` (2 preceding siblings ...)
  2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
@ 2021-10-17 12:29 ` Rodrigo G. López
  2021-10-17 13:50   ` [9front] " hiro
  2021-10-17 18:15 ` smj
  4 siblings, 1 reply; 15+ messages in thread
From: Rodrigo G. López @ 2021-10-17 12:29 UTC (permalink / raw)
  To: 9front

as others mentioned, chmod +t and a cron task with your own rotation would do it. if you want them stored along with the worm, with cwfs dumps usually happen at 5am, so you could compress, move and truncate them, just before this happens, and with whatever resolution you want— daily, weekly, etc.—for the .log.gz files.

i used to do this on an rc-httpd serving machine, but i no longer keep any logs.

something like:
#!/bin/rc

mv /sys/log/^(www www.wip)
>/sys/log/www
chmod +t /sys/log/www
</sys/log/www.wip gzip -9 >/sys/log/archive/www.`{date -n}.gz
rm /sys/log/www.wip


-rodri

On October 16, 2021 8:06:24 PM UTC, igor@9lab.org wrote:
>What is the Plan9 way of dealing with growing log files such as
>/sys/log/www and others?  Is there something similar to multilog¹ or
>logrotate²?
>
>Thanks in advance for any advice.
>
>Cheers,
>Igor
>
>¹…(https://cr.yp.to/daemontools/multilog.html)
>²…(https://github.com/logrotate/logrotate)
>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* [9front] Re: [9front] Re: [9front] Dealing with log files…
  2021-10-17 12:29 ` Rodrigo G. López
@ 2021-10-17 13:50   ` hiro
  2021-10-18  7:03     ` rgl
  0 siblings, 1 reply; 15+ messages in thread
From: hiro @ 2021-10-17 13:50 UTC (permalink / raw)
  To: 9front

what happens to a process that is still having file A open and is
gonna write() again to it after you renamed A into B?

On 10/17/21, Rodrigo G. López <rgl@antares-labs.eu> wrote:
> as others mentioned, chmod +t and a cron task with your own rotation would
> do it. if you want them stored along with the worm, with cwfs dumps usually
> happen at 5am, so you could compress, move and truncate them, just before
> this happens, and with whatever resolution you want— daily, weekly, etc.—for
> the .log.gz files.
>
> i used to do this on an rc-httpd serving machine, but i no longer keep any
> logs.
>
> something like:
> #!/bin/rc
>
> mv /sys/log/^(www www.wip)
>>/sys/log/www
> chmod +t /sys/log/www
> </sys/log/www.wip gzip -9 >/sys/log/archive/www.`{date -n}.gz
> rm /sys/log/www.wip
>
>
> -rodri
>
> On October 16, 2021 8:06:24 PM UTC, igor@9lab.org wrote:
>>What is the Plan9 way of dealing with growing log files such as
>>/sys/log/www and others?  Is there something similar to multilog¹ or
>>logrotate²?
>>
>>Thanks in advance for any advice.
>>
>>Cheers,
>>Igor
>>
>>¹…(https://cr.yp.to/daemontools/multilog.html)
>>²…(https://github.com/logrotate/logrotate)
>>
>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* [9front] Re: [9front] Dealing with log files…
  2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
  2021-10-17  9:06   ` umbraticus
  2021-10-17 10:30   ` igor
@ 2021-10-17 14:19   ` Stanley Lieber
  2 siblings, 0 replies; 15+ messages in thread
From: Stanley Lieber @ 2021-10-17 14:19 UTC (permalink / raw)
  To: 9front

fortunately, the dump must be manually triggered in hjfs, and cwfs can be set up without it.

sl

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Dealing with log files…
  2021-10-16 20:06 [9front] Dealing with log files… igor
                   ` (3 preceding siblings ...)
  2021-10-17 12:29 ` Rodrigo G. López
@ 2021-10-17 18:15 ` smj
  2021-10-18  7:35   ` [9front] " Silas McCroskey
  4 siblings, 1 reply; 15+ messages in thread
From: smj @ 2021-10-17 18:15 UTC (permalink / raw)
  To: 9front

Quoth igor@9lab.org:
> What is the Plan9 way of dealing with growing log files such as
> /sys/log/www and others?  Is there something similar to multilog¹ or
> logrotate²?

I have a caveman multi-gen logrotate that I'm working on
at the moment.  I put it on shithub if you'd like to try:

http://shithub.us/smj/rc/HEAD/info.html

It needs some refinement but it works (don't sue me)


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Re: [9front] Re: [9front] Dealing with log files…
  2021-10-17 13:50   ` [9front] " hiro
@ 2021-10-18  7:03     ` rgl
  2021-10-18  8:51       ` sirjofri
  0 siblings, 1 reply; 15+ messages in thread
From: rgl @ 2021-10-18  7:03 UTC (permalink / raw)
  To: 9front

>what happens to a process that is still having file A open and is
>gonna write() again to it after you renamed A into B?

that's a good question to keep in mind if you have heavy logging services.
the process will keep growing the file, unless you chmod +l it. but then
you'll lose information.

i guess you'll need to restart the service, or force it to reopen the old
path through some note, although that will require patching.

at work we used to have a mysqld with this problem. somebody enabled the
debugging flag and it started throwing logs at a 40GB/day rate or so.
at first we restarted the service during midnight, or rather, stopped it,
truncated the file, then started it back up. in the end we decided it
wasn't worth it since nobody was looking at the logs anyway, so we just
disabled the debugging output.

if you know of a better procedure on plan 9 to handle such a sceneario,
i'd like to learn about it.


-rodri


^ permalink raw reply	[flat|nested] 15+ messages in thread

* [9front] Re: [9front] Dealing with log files…
  2021-10-17 18:15 ` smj
@ 2021-10-18  7:35   ` Silas McCroskey
  0 siblings, 0 replies; 15+ messages in thread
From: Silas McCroskey @ 2021-10-18  7:35 UTC (permalink / raw)
  To: 9front, 23hiro

On Sun, Oct 17, 2021 at 7:19 AM hiro <23hiro@gmail.com> wrote:
>
> what happens to a process that is still having file A open and is
> gonna write() again to it after you renamed A into B?
>

Seems like a pretty textbook case for a fileserver -- /sys/log/xyz is
synthetic and is backed by /sys/log/xyz.1,2.gz,etc. on-disk.

- Silas

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Re: [9front] Re: [9front] Dealing with log files…
  2021-10-18  7:03     ` rgl
@ 2021-10-18  8:51       ` sirjofri
  2021-10-18 18:49         ` Steve Simon
  0 siblings, 1 reply; 15+ messages in thread
From: sirjofri @ 2021-10-18  8:51 UTC (permalink / raw)
  To: 9front

I can imagine some mq-like fileserver mounted at /sys/log, which can have 
some ctl file for settings like rotation, as well as dumping some 
configured logs to actual files for long-time storage, maybe including 
filters.

To explain this idea a little more:

Imagine a smb service with this log fileserver. In /sys/log you can see 
the full log of the server, but only the full log of the day. After 
mounting the full service (mount /srv/logsrv /mnt/logsrv) you could 
access the full log, but in a filtered form, e.g. only failed logins to 
the smb server. The log in /sys/log only exists in memory, (fast writing, 
...) The log in /mnt/logsrv is backed by actual files you can dump to a 
venti or something like that.

The logsrv fileserver could also contain some internal log rotation with 
multiple files and auto-deletion, the interface for the services wouldn't 
change at all (they just write to an mq-like file in /sys/log).

Of course a heavy tool like that makes sense only in a larger network, 
but it's interesting to play around with that idea.

Feel free to implement something like that.

sirjofri

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Re: [9front] Re: [9front] Dealing with log files…
  2021-10-18  8:51       ` sirjofri
@ 2021-10-18 18:49         ` Steve Simon
  2021-10-18 19:40           ` cinap_lenrek
  0 siblings, 1 reply; 15+ messages in thread
From: Steve Simon @ 2021-10-18 18:49 UTC (permalink / raw)
  To: 9front


fileserver ?

i dont get this, plan9 services all (to my knowledge) open and close the logfile on each message so you can just rename the logfile and touch and chmod a new one. there is a window where you could lose a message but i am willing to risk it.

this is what i have been doing for years on my servers. if you really don't want to lose any messages - then don't chmod -t and let venti/cwfs take care of the file.

- Steve


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [9front] Re: [9front] Re: [9front] Dealing with log files…
  2021-10-18 18:49         ` Steve Simon
@ 2021-10-18 19:40           ` cinap_lenrek
  0 siblings, 0 replies; 15+ messages in thread
From: cinap_lenrek @ 2021-10-18 19:40 UTC (permalink / raw)
  To: 9front

> i dont get this, plan9 services all (to my knowledge) open and
> close the logfile on each message so you can just rename the logfile
> and touch and chmod a new one.

interesting point.

according to the code, it keeps the fd around, but it checks 
for each message if the file name or qid changed... but apparently
only for some kind of race condition when the fd got closed and
reopand after a rfork(RFCFD)

if the dirstat fails, it will also reopen, but leak the old file
descriptors (as it assumes it is the fd realloc race).

the result is indeed that the file gets reopend when you rename
the old file and create a new one.

--
cinap

PS: /sys/src/libc/9sys/syslog.c

/*
 * Print
 *  sysname: time: mesg
 * on /sys/log/logname.
 * If cons or log file can't be opened, print on the system console, too.
 */
void
syslog(int cons, char *logname, char *fmt, ...)
{
	char buf[1024];
	char *ctim, *p;
	va_list arg;
	int n;
	Dir *d;
	char err[ERRMAX];

	err[0] = '\0';
	errstr(err, sizeof err);
	lock(&sl);

	/*
	 *  paranoia makes us stat to make sure a fork+close
	 *  hasn't broken our fd's
	 */
	d = dirfstat(sl.fd);
	if(sl.fd < 0 || sl.name == nil || strcmp(sl.name, logname) != 0 ||
	   !eqdirdev(d, sl.d)){
		free(sl.name);
		sl.name = strdup(logname);
		if(sl.name == nil)
			cons = 1;
		else{
			free(sl.d);
			sl.d = nil;
			_syslogopen();
			if(sl.fd < 0)
				cons = 1;
			else
				sl.d = dirfstat(sl.fd);
		}
	}
	free(d);
	if(cons){
		d = dirfstat(sl.consfd);
		if(sl.consfd < 0 || !eqdirdev(d, sl.consd)){
			free(sl.consd);
			sl.consd = nil;
			sl.consfd = open("#c/cons", OWRITE|OCEXEC);
			if(sl.consfd >= 0)
				sl.consd = dirfstat(sl.consfd);
		}
		free(d);
	}

	if(fmt == nil){
		unlock(&sl);
		return;
	}

	ctim = ctime(time(0));
	p = buf + snprint(buf, sizeof(buf)-1, "%s ", sysname());
	strncpy(p, ctim+4, 15);
	p += 15;
	*p++ = ' ';
	errstr(err, sizeof err);
	va_start(arg, fmt);
	p = vseprint(p, buf+sizeof(buf)-1, fmt, arg);
	va_end(arg);
	*p++ = '\n';
	n = p - buf;

	if(sl.fd >= 0){
		seek(sl.fd, 0, 2);
		write(sl.fd, buf, n);
	}

	if(cons && sl.consfd >=0)
		write(sl.consfd, buf, n);

	unlock(&sl);
}

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2021-10-19  9:18 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-16 20:06 [9front] Dealing with log files… igor
2021-10-16 20:52 ` umbraticus
2021-10-16 21:01 ` [9front] " Stanley Lieber
2021-10-17  8:29 ` Lyndon Nerenberg (VE7TFX/VE6BBM)
2021-10-17  9:06   ` umbraticus
2021-10-17 10:30   ` igor
2021-10-17 14:19   ` [9front] " Stanley Lieber
2021-10-17 12:29 ` Rodrigo G. López
2021-10-17 13:50   ` [9front] " hiro
2021-10-18  7:03     ` rgl
2021-10-18  8:51       ` sirjofri
2021-10-18 18:49         ` Steve Simon
2021-10-18 19:40           ` cinap_lenrek
2021-10-17 18:15 ` smj
2021-10-18  7:35   ` [9front] " Silas McCroskey

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).