Re: [9fans] `mk` (from Plan9 ports) efficiency related issue

[Ciprian Dorin Craciun <ciprian.craciun@gmail.com>]

On Mon, Jan 17, 2011 at 17:00, Robert Raschke <rtrlists@googlemail.com> wrote:
> Terribly sorry, my email won't help you much, apart from going "Wow, a 4000
> link mk file!" and "Hmm, I wouldn't start from here if you want to go
> there."

    No problem. Any feedback is welcomed, as I try to understand what
I'm not doing properly.


> Your email also doesn't explain why you cannot generate a "normal"
> mk file.

    I'm afraid I don't understand the question. What do you mean by
"generating a normal mk file"?
    A) Do you mean why am I using a generator that writes the `mk`
script instead of writing the `mk` script myself by hand? The answer
to this is complexity: writing `mk` is Ok when you have a simple
application to build, but as the application grows larger so does the
make script. (And using meta rules is not always possible.)
    B) Why isn't the output script a "normal" `mk` script? Actually is
a very simple script (no meta-rules, no shell expansion, etc.). It's
just big. :)


> If you want to stick with your approach, it almost looks like you may be
> better off to generate a shell script that explicitly checks and runs
> everything you need. (And yes, essentially make your "generator" be a make
> in it's own right. Another one won't hurt.)

    I favored the idea of using another make tool because of
portability and simplicity. The target is that I only need to generate
the make script once and distribute it with the source code, thus it
could be built with an already existing make tool (currently only `mk`
from Plan9 is supported, but I plan to also add GNU/BSD make support.)

    The idea of generating shell scripts doesn't seem so tempting, as
I know that scripting -- at least in Bash -- is not very reliable or
easy...

    (BTW I've chosen `mk` over `make` because it's rules (syntax and
semantic) seems much simpler and saner than the ones with GNU make...
(A lot of automagic happens in that realm...) But if I'm not able to
fix this I'll have to resort back to make...) :(


> But it's cool to see someone else who uses Erlang and RabbitMQ hanging out
> on this list. :-)
>
> Robby

    Glad to see another erlanger. :)

    Ciprian.


> On Mon, Jan 17, 2011 at 2:47 PM, Ciprian Dorin Craciun
> <ciprian.craciun@gmail.com> wrote:
>>
>>    Hello all!
>>
>>    Sorry for interrupting again, but I've stumbled on an `mk` issue...
>>
>>    I've written a little Scheme application that generates `mk`
>> scripts for building Erlang applications. (See below an extract of one
>> of my previous emails describing just the generator part; the thread
>> had the subject: <<mk (from plan9ports) modification time resolution
>> issue?>>.)
>>
>>    Now the problem is that the generated script (attached to this
>> email) has about 2671 prerequisites (` target :: prerequisite `), and
>> about 684 actual targets with recipes. (As I've explained below I'm
>> not using any meta-rules, and I'm explicit about each resulting file
>> and it's dependencies.)
>>
>>    The problem is that the time needed to run the script has
>> extremely increased, and the processor is 100% eaten by `mk`.
>>
>>    For example:
>>    * just running the `mk` script with the `-n` option takes about 14
>> seconds.
>>    * using the commands from the `mk -n` takes about 1 minute and 36
>> seconds;
>>    * then running `mk` takes another 14 seconds; (as it has nothing);
>>    * but after cleaning and running `mk` (which I've left running for
>> about 5 minutes and still didn't finished) it seems that between each
>> target (or batch of targets?) it stays about 14 seconds;
>>
>>    But what is strange is that if instead to build the default target
>> that builds everything I start building little by little independent
>> parts, it works without that great delay...
>>
>>    Any ideas what could cause this?
>>
>>    Thanks,
>>    Ciprian.
>>
>>
>> ----------
>> [[ Extract from the previous email. ]]
>> ----------
>>
>>   BTW... People might wonder how come I have 367 targets (with 1221
>> prerequisites) for such a small project? :) The answers is I don't
>> write the `mk` script by hand, but I've written a small Scheme
>> application that just generates the `mk` script based on descriptions
>> like the following. (Thus the resulting `mk` script is quite
>> exhaustive with quite tight dependencies and doesn't use
>> meta-rules...) :)
>>
>>   So just out of curiosity are there any `mk` script generators out there?
>>
>>   Ciprian.
>>
>>
>> ~~~~
>> (vbs:require-erlang)
>>
>> (vbs:define-erlang-application 'rabbit
>>       erl: "\\./(rabbitmq-server--latest/src|generated)/.*\\.erl"
>>       hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl"
>>       additional-ebin: "\\./generated/rabbit\\.app")
>>
>> (vbs:define-erlang-application 'rabbit_common
>>       erl:
>> "\\./(rabbitmq-server--latest/src|generated)/(rabbit_writer|rabbit_reader|rabbit_framing_amqp_0_8|rabbit_framing_amqp_0_9_1|rabbit_framing_channel|rabbit_basic|rabbit_binary_generator|rabbit_binary_parser|rabbit_channel|rabbit_exchange_type|rabbit_misc|rabbit_net|rabbit_heartbeat|rabbit_msg_store_index|gen_server2|priority_queue|supervisor2)\\.erl"
>>       hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl"
>>       additional-ebin: "\\./generated/rabbit_common\\.app")
>>
>> (vbs:define-erlang-application 'amqp_client
>>       dependencies: 'rabbit_common
>>       erl: "\\./rabbitmq-erlang-client--latest/src/.*\\.erl"
>>       hrl: "\\./rabbitmq-erlang-client--latest/include/.*\\.hrl"
>>       additional-ebin: "\\./generated/amqp_client\\.app")
>> ~~~~