From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.tex.context/46344 Path: news.gmane.org!not-for-mail From: Hans Hagen Newsgroups: gmane.comp.tex.context Subject: Re: distributed / parallel TeX? Date: Tue, 16 Dec 2008 10:07:39 +0100 Message-ID: <49476FDB.80309@wxs.nl> References: <4946E2E2.1050108@sil.org> Reply-To: mailing list for ConTeXt users NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1229418620 14107 80.91.229.12 (16 Dec 2008 09:10:20 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 16 Dec 2008 09:10:20 +0000 (UTC) To: mailing list for ConTeXt users Original-X-From: ntg-context-bounces@ntg.nl Tue Dec 16 10:11:25 2008 Return-path: Envelope-to: gctc-ntg-context-518@m.gmane.org Original-Received: from ronja.vet.uu.nl ([131.211.172.88] helo=ronja.ntg.nl) by lo.gmane.org with esmtp (Exim 4.50) id 1LCVxQ-0006qg-2i for gctc-ntg-context-518@m.gmane.org; Tue, 16 Dec 2008 10:11:12 +0100 Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 16CCA1FC11; Tue, 16 Dec 2008 10:09:57 +0100 (CET) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 13771-02-6; Tue, 16 Dec 2008 10:08:43 +0100 (CET) Original-Received: from ronja.vet.uu.nl (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 5C56A1FC14; Tue, 16 Dec 2008 10:08:43 +0100 (CET) Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 8F2C61FC14 for ; Tue, 16 Dec 2008 10:08:41 +0100 (CET) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 13849-08 for ; Tue, 16 Dec 2008 10:08:05 +0100 (CET) Original-Received: from mail.pragma-ade.net (dsl-083-247-100-017.solcon.nl [83.247.100.17]) by ronja.ntg.nl (Postfix) with ESMTP id 72F521FBDB for ; Tue, 16 Dec 2008 10:07:42 +0100 (CET) Original-Received: from [10.100.1.100] (unverified [10.100.1.100]) by controller-1 (SurgeMail 3.9e) with ESMTP id 17819-1840426 for ; Tue, 16 Dec 2008 10:07:40 +0100 User-Agent: Thunderbird 2.0.0.18 (Windows/20081105) In-Reply-To: <4946E2E2.1050108@sil.org> X-Originating-IP: 10.100.1.100 X-Authenticated-User: hagen@controller-1 X-Virus-Scanned: amavisd-new at ntg.nl X-BeenThere: ntg-context@ntg.nl X-Mailman-Version: 2.1.11 Precedence: list List-Id: mailing list for ConTeXt users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: ntg-context-bounces@ntg.nl Errors-To: ntg-context-bounces@ntg.nl X-Virus-Scanned: amavisd-new at ntg.nl Xref: news.gmane.org gmane.comp.tex.context:46344 Archived-At: Lars Huttar wrote: > Hello, > > We've been using TeX to typeset a 1200-page book, and at that size, the > time it takes to run becomes a big issue (especially with multiple > passes... about 8 on average). It takes us anywhere from 80 minutes on > our fastest machine, to 9 hours on our slowest laptop. often there are ways to optimize the document style btw, in my laptop mk.tex which loads a bunch of real big fonts (esp punk slows down) does 10 pages/second (242 pages currently) so your setu opif probably not that efficient > So the question comes up, can TeX runs take advantage of parallelized or > distributed processing? As many in the computer industries are aware, > processor speed (clock rate) has plateaued; it is not going to continue > rising at the rate it had been. Hence the common move to dual-core, > quad-core, etc. machines. But applications in general cannot take > advantage of multiple cores to speed their work unless they are > architected to take advantage of them. > > We googled around a bit but were surprised not to find any real > references to efforts at running TeX in parallel or on distributed > networks or clusters. Wouldn't this be something that a lot of people > would find useful? Or does everyone only use TeX for typesetting short > papers? it all depends on what you process; for simple docs tex is rather fast: \starttext \setuplayout[page] \dorecurse{1000}{\null\page} \stoptext such a run takes in mkiv: 5.944 seconds, 1000 processed pages, 1000 shipped pages, 168.237 pages/second > Sure, you can use manual tricks to speed up TeX processing. > You can "comment out" sections of a document, or select them via modes. > But then you have to remember where you did the commenting out, so you > can reverse it. And you have no guarantees as to whether the > inclusion/exclusion of section B will affect the layout of section C or not. often it's inefficient font calls that slow down the job (or big graphics that one can skip including in a first pass) > Wouldn't it be nice if TeX (or a TeX wrapper, or macro package, or > "typesetting system") could take care of this for you? mayne in the future we can do some parallization also, i'm thinking of 'one run with prerolls' but it has no high priority (maybe i'll do it when i need it for a project) > What if you had a language -- or a few extensions to existing languages > -- to give your typesetting engine hints or commands about where to > split up your long document into fairly-independent chunks? What if you > designed your document specifically to be typeset in independent, > parallel pieces so that you could guarantee that you would get the same > result for section B whether or not you were typesetting the whole book > at the same time? there are quite some dependencies of pages on each other (take cross refs and how they might influence a next run) > What if the typesetting system automated the stitching-back-together > process of the chunks, gathering page reference info from each chunk to > inform the next iteration of typesetting the other chunks? this is an option when you have to reprocess parts of the documents often > Has anyone been working on this already? It seems like it must have been > discussed, but I don't know where to go to look for that discussion. if i were you i'd first look into optimizing the style Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl ----------------------------------------------------------------- ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________