Re: [9fans] `mk` (from Plan9 ports) efficiency related issue

[Ciprian Dorin Craciun <ciprian.craciun@gmail.com>]

On Mon, Jan 17, 2011 at 17:53, Robert Raschke <rtrlists@googlemail.com> wrote:
>
> On Mon, Jan 17, 2011 at 3:33 PM, Ciprian Dorin Craciun
> <ciprian.craciun@gmail.com> wrote:
>>
>> On Mon, Jan 17, 2011 at 17:00, Robert Raschke <rtrlists@googlemail.com>
>> wrote:
>> > Your email also doesn't explain why you cannot generate a "normal"
>> > mk file.
>>
>>    I'm afraid I don't understand the question. What do you mean by
>> "generating a normal mk file"?
>>    A) Do you mean why am I using a generator that writes the `mk`
>> script instead of writing the `mk` script myself by hand? The answer
>> to this is complexity: writing `mk` is Ok when you have a simple
>> application to build, but as the application grows larger so does the
>> make script. (And using meta rules is not always possible.)
>>    B) Why isn't the output script a "normal" `mk` script? Actually is
>> a very simple script (no meta-rules, no shell expansion, etc.). It's
>> just big. :)
>>
>
> Sorry, I meant an idiomatic mk file, in the sense as they are used within
> the Plan 9 distribution. Have a look at "Plan 9 Mkfiles"
> (http://www.cs.bell-labs.com/sys/doc/mkfiles.html) and "Maintaining Files on
> Plan 9 with Mk" (http://www.cs.bell-labs.com/sys/doc/mk.html), if you
> haven't already done so.
>
> I think by listing all your dependencies one by one, step by step, you are
> bypassing a lot of the strengths of a make system. I would expect your
> generator to produce a mk include file with the meta rules plus the mk file
> itself which lists file dependencies in a concise manner.
>
> Robby

On Mon, Jan 17, 2011 at 17:51, Federico G. Benavento
<benavento@gmail.com> wrote:
>
> a normal mkfile does have meta-rules and if you have so many targets
> wouldn't it make sense to have more mkfiles?
>
> --
> Federico G. Benavento


    I'll respond to both Robert and Federico in the same email, as
their observations an suggestions are on the same topic.

    So for starters I've read both mentioned papers "Plan9 Mkfiles"
and "Maintaining Files on Plan9 with Mk", and I have the following
observations:
    * the second paper "Maintaining Files on ..." is more a general
guide that describes the semantic and syntax of `mk` files;
    * the first paper "Plan9 Mkfiles" focuses entirely and exclusively
on writing `mk` files for building Plan9 native applications; that is
it describes the general rules on how to write short and simple `mk`
files that take advantage on the existing Plan9 build infrastructure;
    * as a consequence both of them seem to suggest writing small `mk`
scripts that individually build each application or library;
    * unfortunately what they don't deal with is inter-dependencies
between multiple projects or libraries each with it's own `mk` script;
    * thus if looking into the `plan9port` source code, inside the
`src/mkfile` I see the following snippet (and I count about 50 other
similar instances):
~~~~
libs-%:V:
        for i in $LIBDIRS
        do
                (cd $i; echo cd `pwd`';' mk $MKFLAGS $stem; mk $MKFLAGS $stem)
        done
~~~~

    But after reading the paper "Recursive Make Considered Harmful", I
see that this approach poses at least the following problems:
    * when developing and updating a single library, there is no way
in which I can instruct `mk` to rebuild all dependent applications --
unless I know about them and enumerate them one by one; as a
consequence -- unless I know very well the real dependency graph --
`mk` doesn't help me at all and I have to `mk clean all`;
    * second, there is a performance bottleneck as now libraries can't
be built in parallel; (this is fine if a library is quite big, but if
I have libraries with a number of files on par with the number of
processors I have wasted time;)
    * third, it's almost impossible to make a dependency between one
library to another; the dependency is encoded in their order in the
variables like `LIBDIRS`;

    I hope that these observations answer the first question of why
don't I just create a number of separate files one for each project or
library.

    (The cited paper is at http://aegis.sourceforge.net/auug97.pdf .)

    For the second question about why no "meta-rules", the answer is
somehow trickier, thus I'll give a few problems which arise when using
meta-rules:
    * not all files of the same type are built by using the same rule;
(for example for some files I want to enable debugging, for others
not); thus I don't see how a meta-rule would solve this problem;
(unless I create separate `mk` files or I resort to filename tags --
for example I would call `*.debug.c` all C files that I want to be
debugged, and `*.release.c` for those I don't);
    * second, each file has a unique dependency graph -- for example
one `*.c` file might include other headers than another `*.c` file in
the same project; thus when updating a single header I want to be able
to build only those `*.c` files that actually depend on it; (this
observation is less important for C applications, but for other type
of programs -- like Java -- this fine grained dependency tracking
means a lot of saved computing power);
    * third, in my brief experience with make files, meta-rules are
quite hard to get right... furthermore it is impossible to have two
patterns like: `%.%.x`; just imagine I have two types of images:
background and foreground and I want to superimpose them, then I would
like to be able to write `%{foreground}.%{background}.jpg :
%{foreground}.jpg %{background}.jpg ..."; (I know that I can resort
here to "<| generating command" but it seems just plain wrong...)

    Now I hope nobody was offended by my observations regarding the
way `mk` scripts are currently written. I am sure that for the purpose
of building Plan9 applications this way is the best trade-off. But
applying the same techniques to more complex programming languages or
other systems is quite hard... Thus I just wanted to use the good `mk`
tool as a backend for a build script generator that is more intimately
acquainted with what it tries to build. (I want to extend my generator
to Python -- as a replacement for `setup.py` -- and Java -- as a
replacement for all the awful tools that exist in that ecosystem,
especially Ant.)

    Ciprian.