ocamlbuild & deps

ocamlbuild & deps

[Daniel Bünzli]

Am I right in thinking that in rule specifications we could get rid of  
the ~dep(s) parameter of rules and have all deps be specified/ 
discovered dynamically via the 'build' argument ? Otherwise stated is  
~dep(s) just an optimization ?

Out of curiosity any idea in the cost of suppressing these arguments  
(i.e. was that road actually followed at some point) ?

If the answer to the first question is yes. Then I think the  
documentation could be made clearer by stating that what is asked to  
be built by the 'build' argument is considered as dependencies.  
However if you know some deps statically you can specify them as  
dep(s) argument this will just implicitely add  them to the list given  
to the 'build' argument.

Best,

Daniel

Re: [Caml-list] ocamlbuild & deps

[Romain Bardou]

> Am I right in thinking that in rule specifications we could get rid of 
> the ~dep(s) parameter of rules and have all deps be specified/discovered 
> dynamically via the 'build' argument ? Otherwise stated is ~dep(s) just 
> an optimization ?
> 
> Out of curiosity any idea in the cost of suppressing these arguments 
> (i.e. was that road actually followed at some point) ?
> 
> If the answer to the first question is yes. Then I think the 
> documentation could be made clearer by stating that what is asked to be 
> built by the 'build' argument is considered as dependencies. However if 
> you know some deps statically you can specify them as dep(s) argument 
> this will just implicitely add  them to the list given to the 'build' 
> argument.

I think there is a difference. It is indeed an optimization issue but 
not at the level of Ocamlbuild itself : it is as the level of your 
compilation process. If A *dynamically* depends on B, and your whole 
project (say, 10 hours of compilation) depends on A, but you have no way 
to build B, then Ocamlbuild will start to compile your project until it 
finds out that A cannot be built (maybe several hours later). If B had 
been put as a ~dep, then Ocamlbuild would not even had started building 
the project in the first place, saving you a lot of time.

Well, at least that's how I understand it ;)

-- 
Romain Bardou

Re: [Caml-list] ocamlbuild & deps

[Daniel Bünzli]

Le 20 févr. 09 à 16:39, Romain Bardou a écrit :

> I think there is a difference. It is indeed an optimization issue  
> but not at the level of Ocamlbuild itself : it is as the level of  
> your compilation process. If A *dynamically* depends on B, and your  
> whole project (say, 10 hours of compilation) depends on A, but you  
> have no way to build B, then Ocamlbuild will start to compile your  
> project until it finds out that A cannot be built (maybe several  
> hours later). If B had been put as a ~dep, then Ocamlbuild would not  
> even had started building the project in the first place, saving you  
> a lot of time.

Heu no. If B cannot be built then the compilation of A stops and the  
compilation of your project stops.

It is however true that if A has a dependency on a heavy C in parallel  
to B you'll have to wait for the end of C. But even in this case, it's  
just a matter of calling 'build' with B and C in a sensible order (and  
not in parallel).

Best,

Daniel

Re: [Caml-list] ocamlbuild & deps

[Romain Bardou]

Daniel Bünzli wrote:
> 
> Le 20 févr. 09 à 16:39, Romain Bardou a écrit :
> 
>> I think there is a difference. It is indeed an optimization issue but 
>> not at the level of Ocamlbuild itself : it is as the level of your 
>> compilation process. If A *dynamically* depends on B, and your whole 
>> project (say, 10 hours of compilation) depends on A, but you have no 
>> way to build B, then Ocamlbuild will start to compile your project 
>> until it finds out that A cannot be built (maybe several hours later). 
>> If B had been put as a ~dep, then Ocamlbuild would not even had 
>> started building the project in the first place, saving you a lot of 
>> time.
> 
> Heu no. If B cannot be built then the compilation of A stops and the 
> compilation of your project stops.
> 
> It is however true that if A has a dependency on a heavy C in parallel 
> to B you'll have to wait for the end of C. But even in this case, it's 
> just a matter of calling 'build' with B and C in a sensible order (and 
> not in parallel).

Ok I think I did not explain correctly. Let's say that :
- A dynamically depends on B
- A depends on C
- B cannot be built
Now if I try to build A, C will be built, then the rule to build A will 
be applied and the dependency on B will be discovered. But B cannot be 
built so the compilation fails.

Now if the compilation of C takes 10 hours, you will have to wait 10 
hours no matter what. "Damn, I waited 10 hours just to discover that it 
cannot compile!" If the dependency on B was not dynamic, C would not 
have been built at all.

So, there is indeed a difference between dynamic and static 
dependencies, with or without parallelism. In fact, dynamic dependencies 
also tend to break parallelism as dynamic dependencies cannot be built 
at the same time as static dependencies. In my example, C and B cannot 
be built in parallel. This is another difference.

You cannot, in general, run the rule to discover dynamic dependencies 
before building static dependencies. Indeed, what if the rule needs to 
read one of the static dependencies to discover other dependencies?

-- 
Romain Bardou

Re: [Caml-list] ocamlbuild & deps

[Daniel Bünzli]

Le 21 févr. 09 à 19:53, Romain Bardou a écrit :

> Ok I think I did not explain correctly. Let's say that :
[...]

I think you are not getting my point. What I meant in the first place  
is that I see strictly no difference between :

> rule ~deps ~prod r

and

> rule ~prod begin fun env build ->
>   let static_deps = build (List.map (fun d -> env d) deps) in
>   List.iter Outcome.ignore_good static_deps;
>   r env build
> end

Strictly speaking this means that all dependencies can be resolved by  
the 'build' argument. However it is sometimes useful to introduce a  
partial order among them. Now your distinction dynamic/static is just  
a limited way of specifiying a partial order among dependencies. The  
generalized way of introducing that order is just to have calls to the  
'build' argument in the order you want inside your rule.

If the above is true then I think it would be usefull to mention this  
in the documentation about the 'build' argument.

Best,

Daniel

Re: [Caml-list] ocamlbuild & deps

[Daniel Bünzli]

Sorry, in my preceding message the line

>  let static_deps = build (List.map (fun d -> env d) deps) in


Should read

>  let static_deps = build (List.map (fun d -> [env d]) deps) in

Daniel

Re: [Caml-list] ocamlbuild & deps

[Daniel Bünzli]

Le 21 févr. 09 à 21:19, Daniel Bünzli a écrit :

> If the above is true then I think it would be usefull to mention  
> this in the documentation about the 'build' argument.

It seems to be false. Suppose we have the following dependencies ("->"  
is "depends")

A -> {B, C}
B -> {B1, B2}
C -> {C1, C2}

The idea is that ocamlbuild will take the static ~deps of the rules  
and construct a first tree of dependencies. Given this tree,  
ocamlbuild will start to build the dependencies from the leaves; I  
wasn't able to get precise information about how the nodes of this  
tree are followed but what seems to be sure is that it will try to  
build all the leaves first, thus some order exists. This means that  
for the example above if all dependencies are known via ~deps  
arguments the build order is going to be something like this (at least  
that would be the most sensible way of doing it, breadth-first order  
reversed).

(B1 || B2 || C1 || C2); (C || B); A

where "||" is parallel, and ";" is sequencing.

Now if you try to simulate the static dependencies ~deps as I  
suggested in my last email. The build order you will get is :

(((B1 || B2); B) || ((C1 || C2); C)); A

and this order will give you different results when B can be built but  
C cannot (or vice-versa) as in that case you will build B (or C)  
whereas it won't be with static dependencies. Hence the conclusion is  
that you cannot simulate the ~deps argument as I suggested in my last  
email.

Best,

Daniel

Re: [Caml-list] ocamlbuild & deps

[Daniel Bünzli]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]


Nicolas told me that the actual order for static dependencies in that  
example would also be that one :

> (((B1 || B2); B) || ((C1 || C2); C)); A

However he points out that there's still a difference between static  
and dynamic dependencies : static dependencies can control the way  
rules are selected. If two rules can build a target but the first rule  
has a static dependency that cannot be built then the second rule will  
be tried. Dynamic dependencies cannot do that since once a rule is  
invoked it is considered to be definitively selected.

Best,

Daniel


[-- Attachment #2: Type: text/html, Size: 898 bytes --]