[9fans] mk (from plan9ports) modification time resolution issue?

[9fans] mk (from plan9ports) modification time resolution issue?

[Ciprian Dorin Craciun]

    Hello all!

    I've played today with mk (from plan9ports), and I thin I've
stumbled upon the following issue: if both the build of a prerequisite
and the target itself is less than a second (the same second), then mk
believes it must remake the target when invoked a second time.

    See below as each time I rerun mk it rebuilds again one less target...

    I believe this is from the fact that mk uses as a time resolution
seconds (as provided for example by `utime`) and not finer grained
resolution (as provided for example by `utimes`).

    Now my question is: I understand that mk must be portable and have
as few operating-system dependent code, but wouldn't it be better to:
    * either implement the usage of `utimens` in plan9ports;
    * or use `<=` as a comparison instead of `<`;

    Thanks,
    Ciprian.


~~~~
# this time is ok to build targets
$ touch ./vbs.scm
$ mk -f ./make.mk -e all
./.outputs/vbs.scm(1294390872) < ./vbs.scm(1294391133)
cp -T -- ./vbs.scm ./.outputs/vbs.scm
./.outputs/vbs.c(1294390872) < ./.outputs/vbs.scm(1294391151)
( cd ./.outputs && csc -t -J -o ./vbs.c ./vbs.scm ; )
./.outputs/vbs.o(1294390872) < ./.outputs/vbs.c(1294391151)
gcc -c -o ./.outputs/vbs.o   ./.outputs/vbs.c
./.outputs/vbs.elf(1294390872) < ./.outputs/vbs.o(1294391151)
gcc -o ./.outputs/vbs.elf -lchicken  ./.outputs/vbs.o
./.outputs/vbs-app.o ./.outputs/vbs-context.o
./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o
./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o
./.outputs/environments.o

~~~~
# starting from now we just rebuild targets because of the resolution
$ mk -f ./make.mk -e all
./.outputs/vbs.c(1294391151) < ./.outputs/vbs.scm(1294391151)
( cd ./.outputs && csc -t -J -o ./vbs.c ./vbs.scm ; )
./.outputs/vbs.o(1294391151) < ./.outputs/vbs.c(1294391175)
gcc -c -o ./.outputs/vbs.o   ./.outputs/vbs.c
./.outputs/vbs.elf(1294391151) < ./.outputs/vbs.o(1294391175)
gcc -o ./.outputs/vbs.elf -lchicken  ./.outputs/vbs.o
./.outputs/vbs-app.o ./.outputs/vbs-context.o
./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o
./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o
./.outputs/environments.o
~~~~
$ mk -f ./make.mk -e all
./.outputs/vbs.o(1294391175) < ./.outputs/vbs.c(1294391175)
gcc -c -o ./.outputs/vbs.o   ./.outputs/vbs.c
./.outputs/vbs.elf(1294391175) < ./.outputs/vbs.o(1294391184)
gcc -o ./.outputs/vbs.elf -lchicken  ./.outputs/vbs.o
./.outputs/vbs-app.o ./.outputs/vbs-context.o
./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o
./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o
./.outputs/environments.o
~~~~
$ mk -f ./make.mk -e all
./.outputs/vbs.elf(1294391184) < ./.outputs/vbs.o(1294391184)
gcc -o ./.outputs/vbs.elf -lchicken  ./.outputs/vbs.o
./.outputs/vbs-app.o ./.outputs/vbs-context.o
./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o
./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o
./.outputs/environments.o
~~~~
$ mk -f ./make.mk -e all
mk: 'all' is up to date
~~~~

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[erik quanstrom]

the code has its reasons.  from mk.c:/^outofdate

		/*
		 * Treat equal times as out-of-date.
		 * It's a race, and the safer option is to do
		 * extra building rather than not enough.
	 	 */
		return node->time <= arc->n->time;

- erik

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[Ciprian Dorin Craciun]

On Fri, Jan 7, 2011 at 11:25, erik quanstrom <quanstro@quanstro.net> wrote:
> the code has its reasons.  from mk.c:/^outofdate
>
>                /*
>                 * Treat equal times as out-of-date.
>                 * It's a race, and the safer option is to do
>                 * extra building rather than not enough.
>                 */
>                return node->time <= arc->n->time;
>
> - erik


    :) I've kind of feared that this is the reason... :)

    But still how do people handle the issue?

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[Henning Schild]

On Fri, 7 Jan 2011 10:51:56 +0100
Ciprian Dorin Craciun <ciprian.craciun@gmail.com> wrote:

>     :) I've kind of feared that this is the reason... :)
>
>     But still how do people handle the issue?


I guess in most cases it is ok to ignore the slight waste of CPU-time.
And i guess people just ignore it. After all it costs less than a
second for each of these targets. If your project it full of them and
they are deps for bigger targets you may want to add "sleep x.y" to the
rules. That way it will maybe take more time but will idle instead of
wasting CPU-time. When using the sleeps adjusting NPROC might help to do
something useful while sleeping.

Another way i could think of is adjusting the mtimes using touch. So
touch the sources -1s or the target +1s after creation of the target.
Depending on the graph that might cause other trouble i guess.

Henning

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[Ciprian Dorin Craciun]

On Fri, Jan 7, 2011 at 12:40, Henning Schild
<henning@plan9.bell-labs.com> wrote:
> On Fri, 7 Jan 2011 10:51:56 +0100
> Ciprian Dorin Craciun <ciprian.craciun@gmail.com> wrote:
>
>>     :) I've kind of feared that this is the reason... :)
>>
>>     But still how do people handle the issue?
>
>
> I guess in most cases it is ok to ignore the slight waste of CPU-time.
> And i guess people just ignore it. After all it costs less than a
> second for each of these targets. If your project it full of them and
> they are deps for bigger targets you may want to add "sleep x.y" to the
> rules. That way it will maybe take more time but will idle instead of
> wasting CPU-time. When using the sleeps adjusting NPROC might help to do
> something useful while sleeping.
>
> Another way i could think of is adjusting the mtimes using touch. So
> touch the sources -1s or the target +1s after creation of the target.
> Depending on the graph that might cause other trouble i guess.
>
> Henning


    Thank you all for the feedback!

    Indeed having some very quick targets rebuilt again isn't that a
big of an issue, nor is waiting 1 second at the end of each recipe.
(The solution with touching the files out of recipes is not applicable
as it interferes with the dependency graph...)

    But the above statement is true unless you have about 367 targets
(for quite a small project -- only 2 tiny and 1 larger Erlang
applications), which when built takes about 45 seconds (with
NPROC=16), and the second time (without touching a single file in the
source directory) takes about exactly the same amount (still 45
seconds), the third time goes down to 44 seconds, then 43 seconds,
then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds,
then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again
0.2, and finally after ONLY 13 builds it finds out that nothing is to
be made...

    So unfortunately neither of the solutions (both waiting, or
touching) are quite reasonable for me...

    As a consequence I've "hacked" `mk` to take into consideration
sub-second timestamps... (And believe me the term "hacked" is most
appropriated as my solution isn't quite that clean.) :) (For those
interested I've put the patch at the end. It seems that I needed only
to hack the `mkmtime` function in `unix.c` to do the following: I
assume that the oldest file that I care to take into account is not
older than 24 hours from now, any file older than that is capped at 24
hours ago; I also assume that any file newer than one hour from now
(any over is capped again); thus I use the signed long not as seconds
from the epoch but as milliseconds starting from 24 hours ago... The
patch works only on Linux, by using the non portable (??)
(undocumented) `st_mtim` (and not `st_mtime`) `stat` field... Any one
knows a portable Unix way of doing this?)

    BTW... People might wonder how come I have 367 targets (with 1221
prerequisites) for such a small project? :) The answers is I don't
write the `mk` script by hand, but I've written a small Scheme
application that just generates the `mk` script based on descriptions
like the following. (Thus the resulting `mk` script is quite
exhaustive with quite tight dependencies and doesn't use
meta-rules...) :)

    So just out of curiosity are there any `mk` script generators out there?

    Ciprian.


~~~~
(vbs:require-erlang)

(vbs:define-erlang-application 'rabbit
	erl: "\\./(rabbitmq-server--latest/src|generated)/.*\\.erl"
	hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl"
	additional-ebin: "\\./generated/rabbit\\.app")

(vbs:define-erlang-application 'rabbit_common
	erl: "\\./(rabbitmq-server--latest/src|generated)/(rabbit_writer|rabbit_reader|rabbit_framing_amqp_0_8|rabbit_framing_amqp_0_9_1|rabbit_framing_channel|rabbit_basic|rabbit_binary_generator|rabbit_binary_parser|rabbit_channel|rabbit_exchange_type|rabbit_misc|rabbit_net|rabbit_heartbeat|rabbit_msg_store_index|gen_server2|priority_queue|supervisor2)\\.erl"
	hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl"
	additional-ebin: "\\./generated/rabbit_common\\.app")

(vbs:define-erlang-application 'amqp_client
	dependencies: 'rabbit_common
	erl: "\\./rabbitmq-erlang-client--latest/src/.*\\.erl"
	hrl: "\\./rabbitmq-erlang-client--latest/include/.*\\.hrl"
	additional-ebin: "\\./generated/amqp_client\\.app")
~~~~


~~~~
diff -u ./mk-orig/unix.c ./mk-timeres/unix.c
--- ./mk-orig/unix.c	2011-01-02 22:52:50.000000000 +0200
+++ ./mk-timeres/unix.c	2011-01-09 16:43:32.150837213 +0200
@@ -329,13 +329,33 @@
 	}
 }

+#define TIMERES_SEC ((unsigned long) 1000)
+#define TIMERES_NANO ((unsigned long) 1000 * 1000)
+#define TIMEBASE ((unsigned long) 1024)
+#define TIMESPAN_SEC ((((((unsigned long) 1) << 31) - 1 - TIMEBASE -
1) / TIMERES_SEC / 3600) * 3600)
+
 unsigned long
 mkmtime(char *name)
 {
 	struct stat st;
+	static unsigned long long ref_time = 0;
+	unsigned long long abs_time;
+	unsigned long rel_time;

 	if(stat(name, &st) < 0)
 		return 0;

-	return st.st_mtime;
+	abs_time = ((unsigned long long) (st.st_mtim.tv_sec)) * TIMERES_SEC
+ (st.st_mtim.tv_nsec / TIMERES_NANO);
+
+	if (ref_time == 0)
+		ref_time = ((unsigned long long) (time((void*)0) - TIMESPAN_SEC)) *
TIMERES_SEC;
+
+	if (abs_time <= ref_time)
+		rel_time = TIMEBASE;
+	else if (abs_time >= (ref_time + (TIMESPAN_SEC * TIMERES_SEC)))
+		rel_time = TIMEBASE + (TIMESPAN_SEC * TIMERES_SEC);
+	else
+		rel_time = TIMEBASE + (abs_time - ref_time);
+
+	return rel_time;
 }
~~~~

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[Russ Cox]

>    But the above statement is true unless you have about 367 targets
> (for quite a small project -- only 2 tiny and 1 larger Erlang
> applications), which when built takes about 45 seconds (with
> NPROC=16), and the second time (without touching a single file in the
> source directory) takes about exactly the same amount (still 45
> seconds), the third time goes down to 44 seconds, then 43 seconds,
> then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds,
> then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again
> 0.2, and finally after ONLY 13 builds it finds out that nothing is to
> be made...

sleep 1; mk

Russ

Re: [9fans] mk (from plan9ports) modification time resolution issue?

[Ciprian Dorin Craciun]

On Tue, Jan 18, 2011 at 18:28, Russ Cox <rsc@swtch.com> wrote:
>>    But the above statement is true unless you have about 367 targets
>> (for quite a small project -- only 2 tiny and 1 larger Erlang
>> applications), which when built takes about 45 seconds (with
>> NPROC=16), and the second time (without touching a single file in the
>> source directory) takes about exactly the same amount (still 45
>> seconds), the third time goes down to 44 seconds, then 43 seconds,
>> then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds,
>> then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again
>> 0.2, and finally after ONLY 13 builds it finds out that nothing is to
>> be made...
>
> sleep 1; mk
>
> Russ


    `sleep 1 ; mk` doesn't work.

    For example (I put the name of the target and the timestamp in
brackets as in: `target(timestamp)` I assume that each target takes
under one second).

~~~~
initial timestamps: a(2), b(1), c(1), d(1)
rules: b : a | c : b | d : c
or the graph: a -> b -> c -> d

first time I call mk:
    a(2), b(x), c(x), d(x)

second time I call mk after a sleep > 0:
    a(2), b(x), c(x+t1), d(x+t1)

third time I call mk after another sleep > 0:
    a(2), b(x), c(x+t1), d(x+t1+t2)
~~~~

    As a consequence I have to run `sleep 1 ; mk` a number of times
comparable with the longest path from input to output.

    Ciprian.

    P.S.: Even if for example `c -> d` takes 1 minute it is still
going to be runned twice until timestamp(c) > timestamp(b) (which is
only after the first sleep).