* [9fans] mk (from plan9ports) modification time resolution issue? @ 2011-01-07 9:13 Ciprian Dorin Craciun 2011-01-07 9:25 ` erik quanstrom 0 siblings, 1 reply; 7+ messages in thread From: Ciprian Dorin Craciun @ 2011-01-07 9:13 UTC (permalink / raw) To: 9fans Hello all! I've played today with mk (from plan9ports), and I thin I've stumbled upon the following issue: if both the build of a prerequisite and the target itself is less than a second (the same second), then mk believes it must remake the target when invoked a second time. See below as each time I rerun mk it rebuilds again one less target... I believe this is from the fact that mk uses as a time resolution seconds (as provided for example by `utime`) and not finer grained resolution (as provided for example by `utimes`). Now my question is: I understand that mk must be portable and have as few operating-system dependent code, but wouldn't it be better to: * either implement the usage of `utimens` in plan9ports; * or use `<=` as a comparison instead of `<`; Thanks, Ciprian. ~~~~ # this time is ok to build targets $ touch ./vbs.scm $ mk -f ./make.mk -e all ./.outputs/vbs.scm(1294390872) < ./vbs.scm(1294391133) cp -T -- ./vbs.scm ./.outputs/vbs.scm ./.outputs/vbs.c(1294390872) < ./.outputs/vbs.scm(1294391151) ( cd ./.outputs && csc -t -J -o ./vbs.c ./vbs.scm ; ) ./.outputs/vbs.o(1294390872) < ./.outputs/vbs.c(1294391151) gcc -c -o ./.outputs/vbs.o ./.outputs/vbs.c ./.outputs/vbs.elf(1294390872) < ./.outputs/vbs.o(1294391151) gcc -o ./.outputs/vbs.elf -lchicken ./.outputs/vbs.o ./.outputs/vbs-app.o ./.outputs/vbs-context.o ./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o ./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o ./.outputs/environments.o ~~~~ # starting from now we just rebuild targets because of the resolution $ mk -f ./make.mk -e all ./.outputs/vbs.c(1294391151) < ./.outputs/vbs.scm(1294391151) ( cd ./.outputs && csc -t -J -o ./vbs.c ./vbs.scm ; ) ./.outputs/vbs.o(1294391151) < ./.outputs/vbs.c(1294391175) gcc -c -o ./.outputs/vbs.o ./.outputs/vbs.c ./.outputs/vbs.elf(1294391151) < ./.outputs/vbs.o(1294391175) gcc -o ./.outputs/vbs.elf -lchicken ./.outputs/vbs.o ./.outputs/vbs-app.o ./.outputs/vbs-context.o ./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o ./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o ./.outputs/environments.o ~~~~ $ mk -f ./make.mk -e all ./.outputs/vbs.o(1294391175) < ./.outputs/vbs.c(1294391175) gcc -c -o ./.outputs/vbs.o ./.outputs/vbs.c ./.outputs/vbs.elf(1294391175) < ./.outputs/vbs.o(1294391184) gcc -o ./.outputs/vbs.elf -lchicken ./.outputs/vbs.o ./.outputs/vbs-app.o ./.outputs/vbs-context.o ./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o ./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o ./.outputs/environments.o ~~~~ $ mk -f ./make.mk -e all ./.outputs/vbs.elf(1294391184) < ./.outputs/vbs.o(1294391184) gcc -o ./.outputs/vbs.elf -lchicken ./.outputs/vbs.o ./.outputs/vbs-app.o ./.outputs/vbs-context.o ./.outputs/vbs-environment.o ./.outputs/vbs-mk-builder.o ./.outputs/vbs-transcript.o ./.outputs/vbs-fs-tools.o ./.outputs/environments.o ~~~~ $ mk -f ./make.mk -e all mk: 'all' is up to date ~~~~ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-07 9:13 [9fans] mk (from plan9ports) modification time resolution issue? Ciprian Dorin Craciun @ 2011-01-07 9:25 ` erik quanstrom 2011-01-07 9:51 ` Ciprian Dorin Craciun 0 siblings, 1 reply; 7+ messages in thread From: erik quanstrom @ 2011-01-07 9:25 UTC (permalink / raw) To: 9fans the code has its reasons. from mk.c:/^outofdate /* * Treat equal times as out-of-date. * It's a race, and the safer option is to do * extra building rather than not enough. */ return node->time <= arc->n->time; - erik ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-07 9:25 ` erik quanstrom @ 2011-01-07 9:51 ` Ciprian Dorin Craciun 2011-01-07 10:40 ` Henning Schild 0 siblings, 1 reply; 7+ messages in thread From: Ciprian Dorin Craciun @ 2011-01-07 9:51 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Fri, Jan 7, 2011 at 11:25, erik quanstrom <quanstro@quanstro.net> wrote: > the code has its reasons. from mk.c:/^outofdate > > /* > * Treat equal times as out-of-date. > * It's a race, and the safer option is to do > * extra building rather than not enough. > */ > return node->time <= arc->n->time; > > - erik :) I've kind of feared that this is the reason... :) But still how do people handle the issue? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-07 9:51 ` Ciprian Dorin Craciun @ 2011-01-07 10:40 ` Henning Schild 2011-01-13 19:36 ` Ciprian Dorin Craciun 0 siblings, 1 reply; 7+ messages in thread From: Henning Schild @ 2011-01-07 10:40 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Fri, 7 Jan 2011 10:51:56 +0100 Ciprian Dorin Craciun <ciprian.craciun@gmail.com> wrote: > :) I've kind of feared that this is the reason... :) > > But still how do people handle the issue? I guess in most cases it is ok to ignore the slight waste of CPU-time. And i guess people just ignore it. After all it costs less than a second for each of these targets. If your project it full of them and they are deps for bigger targets you may want to add "sleep x.y" to the rules. That way it will maybe take more time but will idle instead of wasting CPU-time. When using the sleeps adjusting NPROC might help to do something useful while sleeping. Another way i could think of is adjusting the mtimes using touch. So touch the sources -1s or the target +1s after creation of the target. Depending on the graph that might cause other trouble i guess. Henning ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-07 10:40 ` Henning Schild @ 2011-01-13 19:36 ` Ciprian Dorin Craciun 2011-01-18 16:28 ` Russ Cox 0 siblings, 1 reply; 7+ messages in thread From: Ciprian Dorin Craciun @ 2011-01-13 19:36 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Fri, Jan 7, 2011 at 12:40, Henning Schild <henning@plan9.bell-labs.com> wrote: > On Fri, 7 Jan 2011 10:51:56 +0100 > Ciprian Dorin Craciun <ciprian.craciun@gmail.com> wrote: > >> :) I've kind of feared that this is the reason... :) >> >> But still how do people handle the issue? > > > I guess in most cases it is ok to ignore the slight waste of CPU-time. > And i guess people just ignore it. After all it costs less than a > second for each of these targets. If your project it full of them and > they are deps for bigger targets you may want to add "sleep x.y" to the > rules. That way it will maybe take more time but will idle instead of > wasting CPU-time. When using the sleeps adjusting NPROC might help to do > something useful while sleeping. > > Another way i could think of is adjusting the mtimes using touch. So > touch the sources -1s or the target +1s after creation of the target. > Depending on the graph that might cause other trouble i guess. > > Henning Thank you all for the feedback! Indeed having some very quick targets rebuilt again isn't that a big of an issue, nor is waiting 1 second at the end of each recipe. (The solution with touching the files out of recipes is not applicable as it interferes with the dependency graph...) But the above statement is true unless you have about 367 targets (for quite a small project -- only 2 tiny and 1 larger Erlang applications), which when built takes about 45 seconds (with NPROC=16), and the second time (without touching a single file in the source directory) takes about exactly the same amount (still 45 seconds), the third time goes down to 44 seconds, then 43 seconds, then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds, then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again 0.2, and finally after ONLY 13 builds it finds out that nothing is to be made... So unfortunately neither of the solutions (both waiting, or touching) are quite reasonable for me... As a consequence I've "hacked" `mk` to take into consideration sub-second timestamps... (And believe me the term "hacked" is most appropriated as my solution isn't quite that clean.) :) (For those interested I've put the patch at the end. It seems that I needed only to hack the `mkmtime` function in `unix.c` to do the following: I assume that the oldest file that I care to take into account is not older than 24 hours from now, any file older than that is capped at 24 hours ago; I also assume that any file newer than one hour from now (any over is capped again); thus I use the signed long not as seconds from the epoch but as milliseconds starting from 24 hours ago... The patch works only on Linux, by using the non portable (??) (undocumented) `st_mtim` (and not `st_mtime`) `stat` field... Any one knows a portable Unix way of doing this?) BTW... People might wonder how come I have 367 targets (with 1221 prerequisites) for such a small project? :) The answers is I don't write the `mk` script by hand, but I've written a small Scheme application that just generates the `mk` script based on descriptions like the following. (Thus the resulting `mk` script is quite exhaustive with quite tight dependencies and doesn't use meta-rules...) :) So just out of curiosity are there any `mk` script generators out there? Ciprian. ~~~~ (vbs:require-erlang) (vbs:define-erlang-application 'rabbit erl: "\\./(rabbitmq-server--latest/src|generated)/.*\\.erl" hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl" additional-ebin: "\\./generated/rabbit\\.app") (vbs:define-erlang-application 'rabbit_common erl: "\\./(rabbitmq-server--latest/src|generated)/(rabbit_writer|rabbit_reader|rabbit_framing_amqp_0_8|rabbit_framing_amqp_0_9_1|rabbit_framing_channel|rabbit_basic|rabbit_binary_generator|rabbit_binary_parser|rabbit_channel|rabbit_exchange_type|rabbit_misc|rabbit_net|rabbit_heartbeat|rabbit_msg_store_index|gen_server2|priority_queue|supervisor2)\\.erl" hrl: "\\./(rabbitmq-server--latest/include|generated)/.*\\.hrl" additional-ebin: "\\./generated/rabbit_common\\.app") (vbs:define-erlang-application 'amqp_client dependencies: 'rabbit_common erl: "\\./rabbitmq-erlang-client--latest/src/.*\\.erl" hrl: "\\./rabbitmq-erlang-client--latest/include/.*\\.hrl" additional-ebin: "\\./generated/amqp_client\\.app") ~~~~ ~~~~ diff -u ./mk-orig/unix.c ./mk-timeres/unix.c --- ./mk-orig/unix.c 2011-01-02 22:52:50.000000000 +0200 +++ ./mk-timeres/unix.c 2011-01-09 16:43:32.150837213 +0200 @@ -329,13 +329,33 @@ } } +#define TIMERES_SEC ((unsigned long) 1000) +#define TIMERES_NANO ((unsigned long) 1000 * 1000) +#define TIMEBASE ((unsigned long) 1024) +#define TIMESPAN_SEC ((((((unsigned long) 1) << 31) - 1 - TIMEBASE - 1) / TIMERES_SEC / 3600) * 3600) + unsigned long mkmtime(char *name) { struct stat st; + static unsigned long long ref_time = 0; + unsigned long long abs_time; + unsigned long rel_time; if(stat(name, &st) < 0) return 0; - return st.st_mtime; + abs_time = ((unsigned long long) (st.st_mtim.tv_sec)) * TIMERES_SEC + (st.st_mtim.tv_nsec / TIMERES_NANO); + + if (ref_time == 0) + ref_time = ((unsigned long long) (time((void*)0) - TIMESPAN_SEC)) * TIMERES_SEC; + + if (abs_time <= ref_time) + rel_time = TIMEBASE; + else if (abs_time >= (ref_time + (TIMESPAN_SEC * TIMERES_SEC))) + rel_time = TIMEBASE + (TIMESPAN_SEC * TIMERES_SEC); + else + rel_time = TIMEBASE + (abs_time - ref_time); + + return rel_time; } ~~~~ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-13 19:36 ` Ciprian Dorin Craciun @ 2011-01-18 16:28 ` Russ Cox 2011-01-18 18:24 ` Ciprian Dorin Craciun 0 siblings, 1 reply; 7+ messages in thread From: Russ Cox @ 2011-01-18 16:28 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > But the above statement is true unless you have about 367 targets > (for quite a small project -- only 2 tiny and 1 larger Erlang > applications), which when built takes about 45 seconds (with > NPROC=16), and the second time (without touching a single file in the > source directory) takes about exactly the same amount (still 45 > seconds), the third time goes down to 44 seconds, then 43 seconds, > then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds, > then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again > 0.2, and finally after ONLY 13 builds it finds out that nothing is to > be made... sleep 1; mk Russ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [9fans] mk (from plan9ports) modification time resolution issue? 2011-01-18 16:28 ` Russ Cox @ 2011-01-18 18:24 ` Ciprian Dorin Craciun 0 siblings, 0 replies; 7+ messages in thread From: Ciprian Dorin Craciun @ 2011-01-18 18:24 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, Jan 18, 2011 at 18:28, Russ Cox <rsc@swtch.com> wrote: >> But the above statement is true unless you have about 367 targets >> (for quite a small project -- only 2 tiny and 1 larger Erlang >> applications), which when built takes about 45 seconds (with >> NPROC=16), and the second time (without touching a single file in the >> source directory) takes about exactly the same amount (still 45 >> seconds), the third time goes down to 44 seconds, then 43 seconds, >> then 7 seconds, then 1 second, then 0.6 seconds, then 0.2 seconds, >> then again 0.2 seconds, again 0.2 seconds, again 0.2, again 0.2, again >> 0.2, and finally after ONLY 13 builds it finds out that nothing is to >> be made... > > sleep 1; mk > > Russ `sleep 1 ; mk` doesn't work. For example (I put the name of the target and the timestamp in brackets as in: `target(timestamp)` I assume that each target takes under one second). ~~~~ initial timestamps: a(2), b(1), c(1), d(1) rules: b : a | c : b | d : c or the graph: a -> b -> c -> d first time I call mk: a(2), b(x), c(x), d(x) second time I call mk after a sleep > 0: a(2), b(x), c(x+t1), d(x+t1) third time I call mk after another sleep > 0: a(2), b(x), c(x+t1), d(x+t1+t2) ~~~~ As a consequence I have to run `sleep 1 ; mk` a number of times comparable with the longest path from input to output. Ciprian. P.S.: Even if for example `c -> d` takes 1 minute it is still going to be runned twice until timestamp(c) > timestamp(b) (which is only after the first sleep). ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-01-18 18:24 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-01-07 9:13 [9fans] mk (from plan9ports) modification time resolution issue? Ciprian Dorin Craciun 2011-01-07 9:25 ` erik quanstrom 2011-01-07 9:51 ` Ciprian Dorin Craciun 2011-01-07 10:40 ` Henning Schild 2011-01-13 19:36 ` Ciprian Dorin Craciun 2011-01-18 16:28 ` Russ Cox 2011-01-18 18:24 ` Ciprian Dorin Craciun
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).