From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.sysutils.supervision.general/1996 Path: news.gmane.org!not-for-mail From: Laurent Bercot Newsgroups: gmane.comp.sysutils.supervision.general Subject: Re: multiple log streams Date: Fri, 25 Sep 2009 22:45:27 +0200 Message-ID: <20090925204527.GC30223@skarnet.org> References: <20090402094758.GE32102@home.power> <20090402084742.GC28473@grummit.biaix.org> <20090402085726.GC32102@home.power> <20090402092404.GB30096@grummit.biaix.org> <20090402084742.GC28473@grummit.biaix.org> <20090402085726.GC32102@home.power> <20090402084742.GC28473@grummit.biaix.org> <20090903090201.GD59829@grummit.biaix.org> <20090903213444.GA25163@skarnet.org> <20090923142018.GE34137@grummit.biaix.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1253911440 5353 80.91.229.12 (25 Sep 2009 20:44:00 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 25 Sep 2009 20:44:00 +0000 (UTC) To: supervision@list.skarnet.org Original-X-From: supervision-return-2231-gcsg-supervision=m.gmane.org@list.skarnet.org Fri Sep 25 22:43:53 2009 Return-path: Envelope-to: gcsg-supervision@lo.gmane.org Original-Received: from antah.skarnet.org ([212.85.147.14]) by lo.gmane.org with smtp (Exim 4.50) id 1MrHdw-0007gr-0u for gcsg-supervision@lo.gmane.org; Fri, 25 Sep 2009 22:43:52 +0200 Original-Received: (qmail 10552 invoked by uid 76); 25 Sep 2009 20:45:27 -0000 Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Archive: Original-Received: (qmail 10544 invoked by uid 1000); 25 Sep 2009 20:45:27 -0000 Mail-Followup-To: supervision@list.skarnet.org Content-Disposition: inline In-Reply-To: <20090923142018.GE34137@grummit.biaix.org> User-Agent: Mutt/1.4i Xref: news.gmane.org gmane.comp.sysutils.supervision.general:1996 Archived-At: > I have multiple consumers, some real-time and some batch. I need to be > able to easily adapt to more consumers appearing. Batch consumers are > handled at rotation time, real-time must feed from current. In case of > > 1 real-time consumers, I should be able to get by with multitee. The problem with multitee is that it doesn't dynamically adapt. You need to write a couple specific programs so that your requirements are exactly met. If I understand your requirements correctly, I suggest the following architecture: - /service/producer: your producer - /service/producer/log: a specific program (most likely written in C or perl) that * opens a Unix domain socket, binds it to /service/producer/log/s and listens to it, accepting connections from real-time consumers' uids-gids * reads from stdin and reliably writes to every client connection it has. This is the equivalent of multitee, but it can dynamically adapt to more or less real-time consumers. You will always have at least one client, which is... - /service/prodlogger: actually just "ipcclient /service/producer/log/s fdmove 0 6 multilog blahblah" in the run file. This is your main real-time consumer ; its purpose is just to feed a copy of the stream to a multilog or svlogd process that reliably saves it to the filesystem. The logging script is designed to trigger batch consumers. - "ipcclient /service/producer/log/s fdmove 0 6 realtimeconsumerprog" can be how you generally run your real-time consumers, if realtimeconsumerprog reads the stream on stdin. > I do usleep() every iteration, which does not count as active waiting in > my book. Sorry, I said "active waiting" but I meant "polling". Using notification mechanisms instead of polling mechanisms whenever you can is clever use of resources. :) -- Laurent