From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <zsh-users-return-14198-mason-zsh=primenet.com.au@sunsite.dk>
Received: (qmail 2000 invoked from network); 16 Jun 2009 20:30:45 -0000
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on f.primenet.com.au
X-Spam-Level: 
X-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00 autolearn=ham
	version=3.2.5
Received: from new-brage.dotsrc.org (HELO a.mx.sunsite.dk) (130.225.254.104)
  by ns1.primenet.com.au with SMTP; 16 Jun 2009 20:30:45 -0000
Received-SPF: none (ns1.primenet.com.au: domain at sunsite.dk does not designate permitted sender hosts)
Received: (qmail 74919 invoked from network); 16 Jun 2009 20:30:23 -0000
Received: from sunsite.dk (130.225.247.90)
  by a.mx.sunsite.dk with SMTP; 16 Jun 2009 20:30:23 -0000
Received: (qmail 25381 invoked by alias); 16 Jun 2009 20:29:47 -0000
Mailing-List: contact zsh-users-help@sunsite.dk; run by ezmlm
Precedence: bulk
X-No-Archive: yes
X-Seq: 14198
Received: (qmail 25353 invoked from network); 16 Jun 2009 20:29:45 -0000
Received: from bifrost.dotsrc.org (130.225.254.106)
  by sunsite.dk with SMTP; 16 Jun 2009 20:29:45 -0000
Received: from mtaout03-winn.ispmail.ntl.com (mtaout03-winn.ispmail.ntl.com [81.103.221.49])
	by bifrost.dotsrc.org (Postfix) with ESMTP id AE97B8027106
	for <zsh-users@sunsite.dk>; Tue, 16 Jun 2009 22:29:42 +0200 (CEST)
Received: from aamtaout04-winn.ispmail.ntl.com ([81.103.221.35])
          by mtaout03-winn.ispmail.ntl.com
          (InterMail vM.7.08.04.00 201-2186-134-20080326) with ESMTP
          id <20090616202941.DPVY5579.mtaout03-winn.ispmail.ntl.com@aamtaout04-winn.ispmail.ntl.com>
          for <zsh-users@sunsite.dk>; Tue, 16 Jun 2009 21:29:41 +0100
Received: from pws-pc.ntlworld.com ([81.107.42.185])
          by aamtaout04-winn.ispmail.ntl.com
          (InterMail vG.2.02.00.01 201-2161-120-102-20060912) with ESMTP
          id <20090616202936.BOUS22934.aamtaout04-winn.ispmail.ntl.com@pws-pc.ntlworld.com>
          for <zsh-users@sunsite.dk>; Tue, 16 Jun 2009 21:29:36 +0100
Received: from pws-pc (pws-pc [127.0.0.1])
	by pws-pc.ntlworld.com (8.14.3/8.14.2) with ESMTP id n5GKTPxO003997
	for <zsh-users@sunsite.dk>; Tue, 16 Jun 2009 21:29:25 +0100
Message-Id: <200906162029.n5GKTPxO003997@pws-pc.ntlworld.com>
From: Peter Stephenson <p.w.stephenson@ntlworld.com>
To: zsh-users@sunsite.dk
Subject: Re: serverizing a fat process with named pipes
In-Reply-To: Message from Alexy Khrabrov <deliverable@gmail.com>
   of "Mon, 15 Jun 2009 23:17:00 EDT." <7c737f300906152017m7776d596he8a1251c9e1279b8@mail.gmail.com>
Date: Tue, 16 Jun 2009 21:29:25 +0100
X-Cloudmark-Analysis: v=1.0 c=1 a=CiLKsCEehxgA:10 a=NLZqzBF-AAAA:8 a=_jFDL6ORTFG_gJUneJYA:9 a=NPYNXX0fwQVBeEvshsw2DOlBBNcA:4 a=_dQi-Dcv4p4A:10
X-Virus-Scanned: ClamAV 0.94.2/9472/Tue Jun 16 19:24:06 2009 on bifrost
X-Virus-Status: Clean

Alexy Khrabrov wrote:
> I have a heavy process, an English parser loading megabytes of models,
> and then reading stdin, sentence per line, outputting the parsed text
> to the stdout.  How do I properly serverize it -- running in the
> background with <p1 >p2, those created with mkfifo p1 p2?  Looks like
> unless I wrap that in a kind of a loop, the processes exit at once.
> First I did this simulation:
> 
> mkfifo p1 p1
> cat <p1 >p2 &
> cat p2 &
> echo hi > p1
> 
> This does print "hi" and then both backgrounds exit.  If I wrap them
> in while true; do ... done & (& from the original goes outside), the
> simple one looks running continuously, but the parser takes a while
> starting up every time, apparently...

I think your problem's fairly straightforward:  the parser just needs to
open the file, read from it, and when it gets EOF on the input, close it
again.  It can then jump back to trying to open the file again.  It's
not really a shell problem, it's internal to the parser program.
Schematically (not complete code)...

for (;;) {
   FILE *file = fopen(p1);
   while (fread(file, ....) > 0) {
      /* parse and output */
   }
   fclose(file);
}

The programme will block at the fopen() until somebody writes to the
other side of the pipe.

You can do this sort of thing within the shell, if you need...  there's a
trick with a variable to get a file descriptor (this *is* working code
this time):

# initialize here
while true; do
   exec {fd}<p1  # script blocks here until someone writes to p1
   # something more exciting goes here...
   cat <&$fd
   exec {fd}<&- 
   print Restarting...
done

That's not quite the same as the loop you originally tried because
you've now got control of the file descriptor at the point where
the file opens and you hang on to it until you explicitly close it,
instead of relying on a process in the middle doing that for you.

If you're using a pipe on the output end you'll need to watch out for
the server process getting SIGPIPE if the reading process exits.  You
can simply ignore that ("trap '' PIPE" in the shell) and rely on a write
failing to terminate the current iteration and go back and wait to be
able to open the output again (but you do either need to handle the
SIGPIPE or the failing write).

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/