From mboxrd@z Thu Jan 1 00:00:00 1970 MIME-Version: 1.0 In-Reply-To: References: Date: Mon, 17 Jan 2011 19:31:41 +0200 Message-ID: From: Ciprian Dorin Craciun To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [9fans] `mk` (from Plan9 ports) efficiency related issue Topicbox-Message-UUID: 9d6a41a0-ead6-11e9-9d60-3106f5b1d025 On Mon, Jan 17, 2011 at 17:53, Robert Raschke wro= te: > > On Mon, Jan 17, 2011 at 3:33 PM, Ciprian Dorin Craciun > wrote: >> >> On Mon, Jan 17, 2011 at 17:00, Robert Raschke >> wrote: >> > Your email also doesn't explain why you cannot generate a "normal" >> > mk file. >> >> =C2=A0 =C2=A0I'm afraid I don't understand the question. What do you mea= n by >> "generating a normal mk file"? >> =C2=A0 =C2=A0A) Do you mean why am I using a generator that writes the `= mk` >> script instead of writing the `mk` script myself by hand? The answer >> to this is complexity: writing `mk` is Ok when you have a simple >> application to build, but as the application grows larger so does the >> make script. (And using meta rules is not always possible.) >> =C2=A0 =C2=A0B) Why isn't the output script a "normal" `mk` script? Actu= ally is >> a very simple script (no meta-rules, no shell expansion, etc.). It's >> just big. :) >> > > Sorry, I meant an idiomatic mk file, in the sense as they are used within > the Plan 9 distribution. Have a look at "Plan 9 Mkfiles" > (http://www.cs.bell-labs.com/sys/doc/mkfiles.html) and "Maintaining Files= on > Plan 9 with Mk" (http://www.cs.bell-labs.com/sys/doc/mk.html), if you > haven't already done so. > > I think by listing all your dependencies one by one, step by step, you ar= e > bypassing a lot of the strengths of a make system. I would expect your > generator to produce a mk include file with the meta rules plus the mk fi= le > itself which lists file dependencies in a concise manner. > > Robby On Mon, Jan 17, 2011 at 17:51, Federico G. Benavento wrote: > > a normal mkfile does have meta-rules and if you have so many targets > wouldn't it make sense to have more mkfiles? > > -- > Federico G. Benavento I'll respond to both Robert and Federico in the same email, as their observations an suggestions are on the same topic. So for starters I've read both mentioned papers "Plan9 Mkfiles" and "Maintaining Files on Plan9 with Mk", and I have the following observations: * the second paper "Maintaining Files on ..." is more a general guide that describes the semantic and syntax of `mk` files; * the first paper "Plan9 Mkfiles" focuses entirely and exclusively on writing `mk` files for building Plan9 native applications; that is it describes the general rules on how to write short and simple `mk` files that take advantage on the existing Plan9 build infrastructure; * as a consequence both of them seem to suggest writing small `mk` scripts that individually build each application or library; * unfortunately what they don't deal with is inter-dependencies between multiple projects or libraries each with it's own `mk` script; * thus if looking into the `plan9port` source code, inside the `src/mkfile` I see the following snippet (and I count about 50 other similar instances): ~~~~ libs-%:V: for i in $LIBDIRS do (cd $i; echo cd `pwd`';' mk $MKFLAGS $stem; mk $MKFLAGS $st= em) done ~~~~ But after reading the paper "Recursive Make Considered Harmful", I see that this approach poses at least the following problems: * when developing and updating a single library, there is no way in which I can instruct `mk` to rebuild all dependent applications -- unless I know about them and enumerate them one by one; as a consequence -- unless I know very well the real dependency graph -- `mk` doesn't help me at all and I have to `mk clean all`; * second, there is a performance bottleneck as now libraries can't be built in parallel; (this is fine if a library is quite big, but if I have libraries with a number of files on par with the number of processors I have wasted time;) * third, it's almost impossible to make a dependency between one library to another; the dependency is encoded in their order in the variables like `LIBDIRS`; I hope that these observations answer the first question of why don't I just create a number of separate files one for each project or library. (The cited paper is at http://aegis.sourceforge.net/auug97.pdf .) For the second question about why no "meta-rules", the answer is somehow trickier, thus I'll give a few problems which arise when using meta-rules: * not all files of the same type are built by using the same rule; (for example for some files I want to enable debugging, for others not); thus I don't see how a meta-rule would solve this problem; (unless I create separate `mk` files or I resort to filename tags -- for example I would call `*.debug.c` all C files that I want to be debugged, and `*.release.c` for those I don't); * second, each file has a unique dependency graph -- for example one `*.c` file might include other headers than another `*.c` file in the same project; thus when updating a single header I want to be able to build only those `*.c` files that actually depend on it; (this observation is less important for C applications, but for other type of programs -- like Java -- this fine grained dependency tracking means a lot of saved computing power); * third, in my brief experience with make files, meta-rules are quite hard to get right... furthermore it is impossible to have two patterns like: `%.%.x`; just imagine I have two types of images: background and foreground and I want to superimpose them, then I would like to be able to write `%{foreground}.%{background}.jpg : %{foreground}.jpg %{background}.jpg ..."; (I know that I can resort here to "<| generating command" but it seems just plain wrong...) Now I hope nobody was offended by my observations regarding the way `mk` scripts are currently written. I am sure that for the purpose of building Plan9 applications this way is the best trade-off. But applying the same techniques to more complex programming languages or other systems is quite hard... Thus I just wanted to use the good `mk` tool as a backend for a build script generator that is more intimately acquainted with what it tries to build. (I want to extend my generator to Python -- as a replacement for `setup.py` -- and Java -- as a replacement for all the awful tools that exist in that ecosystem, especially Ant.) Ciprian.