From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (from majordomo@localhost) by pauillac.inria.fr (8.7.6/8.7.3) id WAA05493; Sun, 5 Sep 2004 22:12:36 +0200 (MET DST) X-Authentication-Warning: pauillac.inria.fr: majordomo set sender to owner-caml-list@pauillac.inria.fr using -f Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by pauillac.inria.fr (8.7.6/8.7.3) with ESMTP id WAA03279 for ; Sun, 5 Sep 2004 22:12:34 +0200 (MET DST) Received: from smtp3.adl2.internode.on.net (smtp3.adl2.internode.on.net [203.16.214.203]) by concorde.inria.fr (8.13.0/8.13.0) with ESMTP id i85KCVhx007478 for ; Sun, 5 Sep 2004 22:12:33 +0200 Received: from [192.168.1.200] (ppp210-32.lns2.syd3.internode.on.net [203.122.210.32]) by smtp3.adl2.internode.on.net (8.12.9/8.12.9) with ESMTP id i85KCRHY056466; Mon, 6 Sep 2004 05:42:28 +0930 (CST) Subject: Re: [Caml-list] Announcing the OMake build system version 0.9.1 From: skaller Reply-To: skaller@users.sourceforge.net To: "Marcin 'Qrczak' Kowalczyk" Cc: caml-list In-Reply-To: <87zn44v9k1.fsf@qrnik.zagroda> References: <4139ECD3.1050708@cs.caltech.edu> <001e01c492a6$872c7280$19b0e152@warp> <413A0921.7030607@ntlworld.com> <413A1E89.10806@libertysurf.fr> <1094361655.3352.476.camel@pelican.wigram> <87hdqc3kfr.fsf@qrnik.zagroda> <1094396893.3352.635.camel@pelican.wigram> <87sm9w4tfk.fsf@qrnik.zagroda> <1094403880.3352.751.camel@pelican.wigram> <87zn44v9k1.fsf@qrnik.zagroda> Content-Type: text/plain Message-Id: <1094415146.3352.941.camel@pelican.wigram> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.2 (1.2.2-4) Date: 06 Sep 2004 06:12:27 +1000 Content-Transfer-Encoding: 7bit X-Miltered: at concorde with ID 413B732F.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Loop: caml-list@inria.fr X-Spam: no; 0.00; caml-list:01 sourceforge:01 2004:99 marcin:01 'qrczak':01 kowalczyk:01 sourceforge:01 outputs:01 dependencies:01 interscript:01 fixpoint:01 interscript:01 python:01 fixpoint:01 dependencies:01 Sender: owner-caml-list@pauillac.inria.fr Precedence: bulk On Mon, 2004-09-06 at 04:45, Marcin 'Qrczak' Kowalczyk wrote: > skaller writes: > > > together these two functions conspire to determine which > > input files generate which outputs. > > So I must either explicitly tell the dependencies, or run a script > which computes them. Same as in make. This is getting too confusing. For interscript itself, the answer is no: it automatically extracts all sources no matter what you change. No dependency information is required, and no targets, and no makefile. It does this by fixpoint iteration -- the interscript program knows nothing about the source you're generating. It does know (a) the set of LP sources, because you give it the first one on the command line and it follows includes. It also knows every file you generated, because you called it to open the file and it keeps track. What I'm suggesting is that this process is quite general. It isn't restricted to 'extract source from LP source'. In particular I leverage that in interscript itself, by allowing the client to generate sources any way they think fit using embedded Python. This idea extends to any kind of build processes. There's no difference in principle between extracting sources from a tarball, running interscript, or compiling a C program to an object file. All such process will build by fixpoint iteration. Clearly the exact build commands must be given to build the system -- whether that is done the hard way or the commands are generated doesn't matter, since the fixpoint concept can rebuild the command script just like any other output. Clearly if you do NOT encode some dependency information, the process will be inefficient. If you lie, it may fail. If you provide partial information it speeds up, the more the better. > Assume that I forget that compilation of a particular file also reads > another included file, i.e. forget one dependency link. Then after > changing the included file, the target will not be rebuilt (I don't > believe that it will, because then all other targets are in the same > situation and everything would be rebuilt, which is impractical). > So *all* dependencies must be specified. Same as in make. You are missing the idea -- all the things are not only rebuilt every time (in principle) they're rebuilt AT LEAST TWICE. There is no way to check convergence in one pass. So in the first instance the dependency information is used to optimise the build order. This reduces the number of passes required. For example in worst case to build 26 files A-Z, where each is dependent on the previous one alphabetically, if the build order is ZY .. CBA then it will take 26 passes to build, and 27 to verify convergence. That's not 26 compiles, which you think is too slow -- its 26 SQUARED compiles. If you reorder the list using partial dependency information, that reduces the number of passes -- NOT the number of compiles per pass. If you have enough information to order the compiles ABCDE ... Z then it takes one pass to build (and one to verify convergence). Eliminating some of those compiles is a related but distinct problem. You may well need full dependency information to eliminate every eliminable compilation. As mentioned I personally don't bother with that in Felix package, I just use a hand coded linear order (one pass always converges) and check each object file against its *.ml file -- and then compile everything thereafter -- that always works (because it accounts for all the dependencies, since they've already been checked). This algorithm doesn't do the minimum compiles, but ocamlopt.opt is so damn fast it just doesn't matter: in practice it reduces my compile time by 5-10 times (since most work is on the backend implementation). So actually -- I'm not using the fixpoint iteration at all to compile ocaml code. Its main use in my package is to handle configuration (which can require three passes since there's stuff that generates stuff that generates stuff which the first generator needs .. :) In practice the fixpoint stuff is quite useful: first I naturally write things in dependency order anyhow, and secondly all those passes aren't done every time -- I just run one pass usually. Remember that the system uses a persistent store, so that one pass is probably pass 457 or something :) I try to strike a balance between reasonable build speed, and reasonable amount of script to specify the actual build -- that's important for platform independence and robustness -- remember the key use of the build process isn't me building the package, its for my clients to build the package -- and they really do have to compile everything :) > > If you try to include a file that is generated, > > all is well even if it is generated *after* you > > include it. > > This implies that a compile error doesn't abort the compilation Yes. More correctly an error in the build process doesn't stop the build continuing on (in general). If I want this, I have to actively program it. > (because the error might result from lack of source which will be made > later). So if I made a fatal error in a C header file included in many > C sources, the time to rebuild all of them and detect the error > multiple times will be wasted. Yes. As I have said, the fixpoint idea works, but is not automatically efficient -- you still need to do some work to make it efficient. However unlike make, you can do it as your project grows, and you can use arbitrary executable script. > 'Make' would > know that it makes no sense to compile B before C. Make is brain dead, it only know what you tell it. It can't compute anything you can't do in Python in a roughly similar number of lines of code. The converse is not true -- Python is much more expressive than make. In both cases if you want a completely optimal build order you have to maintain a complete dependency graph. > > As a counter-example: latex doesn't always converge. > > It's the only counter-example I know. You will discover another source of recusive build dependencies if you try to bootstrap your language -- and especially if you *also* try to write the build tool that manages that in your language too .. :)) -- John Skaller, mailto:skaller@users.sf.net voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net ------------------- To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners