RE: [Caml-list] The DLL-hell of O'Caml

RE: [Caml-list] The DLL-hell of O'Caml

[Dave Berry]

I don't have figures for this, unfortunately.  I believe the TIL folks
use this technique to keep the size of their type information
manageable; possibly they have some measurements.

I can offer two anecdotal examples.  Way back when the Edinburgh CS
department did all its teaching on one multi-user machine with about 8M
RAM, the Edinburgh ML implementation was quite unusable because the
top-level rapidly grew too large.  Hash-consing made it manageable, with
no appreciable loss of performance.

I believe Harlequin's implementation of Dylan also introduced
hash-consing to reduce the size of the program databases (among other
tricks).  This increased performance by reducing thrashing.

It's always hard to predict how much effect a given technique will have
in a given situation.  In practice someone would have to implement it
and measure the effect.



-----Original Message-----
From: Xavier Leroy [mailto:xavier.leroy@inria.fr]
Sent: 21 March 2002 18:13
To: Christopher Quinn
Cc: caml-list@inria.fr
Subject: Re: [Caml-list] The DLL-hell of O'Caml


> Any idea how much improvement would come from hash-consing, 
> as Dave Berry suggested? And only that part of a module's 
> signature which pertained to actual usage would need 
> inclusion, no? And only of those modules externally provided 
> upon which one's project depends, ie. the standard library, 
> not the internal project modules?

No idea.  It's certainly possible to reduce the size of the signature
information that has to be retained; how much and at what cost, I
don't know.

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives:
http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ:
http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Dave Berry]

FWIW, we (KAL) maintain backwards compatibility in a whole host of DLLs
and OCXs on Windows NT, 2000, and XP.  You can drop a version of our
utilities DLL compiled last night into a distribution from two years
ago, and your distribution will carry on working as if nothing had
changed.  So it is possible, even on Microsoft systems, given sufficient
care on the part of the designers.

What would be nice would be a type system that could ensure this for us.
There's some opportunities for academic research here, surely?

-----Original Message-----
From: Xavier Leroy [mailto:xavier.leroy@inria.fr]
Sent: 21 March 2002 18:10
To: Jeff Henrikson
Cc: caml-list@inria.fr
Subject: Re: [Caml-list] The DLL-hell of O'Caml

What I think is futile is to expect users to pick independently-
compiled binary distributions of libraries and dynamically-linked
executables
left and right, and expect that the resulting system will work.  Try
to do this with a Linux system and it will break very quickly.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Warp]

> Xavier Leroy wrote :
>
> Again, I don't have anything against binary distributions, as long as
> there's an underlying source distribution and recompilation manager
> that lets me bring things up to date if a binary incompatibility
> arises.  RPMs get close to this but lack a "rebuild and install what
> needs to be recompiled" command.  Maybe Debian's packages or BSD ports
> are better in this respect.

Are you expecting all the OCaml developpers to distribute full source code
with each of the binary distribution they're making ? That's perhaps
realistic from a Linux/FSF point of view, but not from a Windows one.... To
enable the widespread of such a good language, the dev community needs solid
and useful libraries to build up a complete dev environment - including DB
access, Network , mathematics, drawing, etc... This work has already start
but some won't release their sources because they're in businness or because
sources are not "clean" enough and releasing them in a proper way would cost
a lot of time (and money)... or many many many reasons which have made lots
of commonly C/C++ used libraries been closed source.

I have worked a lot with Dynlink, and currently the main lack is to enable
"dynamic-link at load-time". That is, adding a keyword ( you're writing for
example "external" instead of "open" ) and then ocamlrun will load the CMA
at startup, check the interface, and do the linking.

If you combine such a feature with the ability to check only the equality
between the sub-interface of the module your program is using , then you're
not breaking type-safe at all, and you allow binary-only-distribution.

With a little bit of versionning, that'll be perfect :)

Nicolas Cannasse
the only window ocaml user (?) :-)

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Brian Rogoff]

On Thu, 21 Mar 2002, Xavier Leroy wrote:
> > Someone suggested the need of "closed source" as an argument
> > for binary distribution. I can make an argument that relies
> > on less.  Assume ala mathematical induction OCaml
> > proliferates.  Foobar Corp does their bookkeeping in ocaml
> > and has 2 million lines and needs to make a bugfix to their
> > SQL wrapper way down at the bottom of the heap.  Do you tell
> > them they must wait a couple of days do get their system
> > back up?
>
> This is a strawman argument in many respects:

In fact, lot's of commercial entities do provide upgrades in one swell
foop, rather than providing lots of little patches. I wasn't sure about
the history, but I asked the guy who supports all of our CAD software this
morning and he wrote

    EDA companies used to provide software patches
    which were a few files.  These days, most of
    them provide monolithic releases which are
    generally self-sufficient, and incorporate multiple
    bug fixes, in order to minimize the number of
    custom versions they have to support.

I actually expected the use of shared libraries to be increasing over
time. Oh well, that just shows you how closely expectations and reality
conform.

Anyways, this isn't an issue that's ever bothered any of the OCaml fans
where I work. Jeff, if you want to cause trouble, suggest that optimized
code and bytecode should work together seamlessly in all combinations.
That's very desirable, very hard to do, and there are no theory problems
there, just lots of grunge work :-).

-- Brian
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Jeff Henrikson]

> You're confusing whole programs with libraries.
> A program can be
> distributed as a statically-linked executable . . .

Ideally, I'd like to confuse them as much as possible.  From
the libtool manual: "libraries are programs with multiple
entry points, and
more formally defined interfaces."

I think a good rule of thumb for a rational developer
prospectively looking at using a new language is "How easy
is <insert here> compared to C?"  Insert your favorite term
[package management/system integration/dynamic library
futzing].  I claim our choice in package management should
make sense to a good but non-ocaml-indoctrinated programmer.
Right now I don't think the proposals fall along those
lines.

Ideally, multiple ocaml libraries loaded into the same
process could do all the same things as C:

- share (ocaml) runtimes
- call each other
- be loaded and unloaded
- call and be called by other languages without much
programmer effort

and at build time:

- if no interfaces are changed rebuild only one shared lib
and rebuild only that.  (In fact newer Linux can even reload
such a recompiled lib while the process is still running,
hence you can upgrade libc while still running, etc.)

Currently, certain tools to make this so are missing.  (For
example both forward and reverse stub generators smarter
than camlidl and camoflage.)  But we should not make package
management design choices that preclude these possibilities
unless there's a good reason to.

> This is a strawman argument in many respects:
>
> - The OCaml distribution is 100 KLOC and recompiles in a
few minutes
>   on a 1000 EUR machine.  So, 2MLOC takes less than one
hour, not two
> days.  . . .

I am admittedly offering arbitrary unchecked numbers.  But I
don't think the argument changes, it's analagous to an order
of growth argument.  If build times don't matter, then why
does "make" exist?  I for one am quite impatient to build
caml on MPW with a broken make tool (or makefiles? I haven't
quite taken the time to understand.) which recompile the
world upon one file touch.  I heard Gerry Sussman (scheme
coinventor) quip something once along the lines of "Software
is like some funny kind of gas.  It takes up all the space
you give it."  I think this applies to build times as well
as run times.

> > To change the subject slightly, there seem to
> be some other
> > dangerous related "purity before progress" idea bugs
> > floating around, like
>
> And there seems to be some dangerous "let's hack
> something before
> thinking over the consequences" ideas in your message.

Admittedly.  I would probably think differently if I
everything were up to me, but since I know there are enough
people of the opposite thinking to keep me in check, I feel
it's most useful for me to err on this side.  In groups with
abundance of hacker mentality I am quite happy to play the
other role.  Sorry if anyone is offended.

Regards,


Jeff Henrikson





-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[james woodyatt]

On Thursday, March 21, 2002, at 10:10 AM, Xavier Leroy wrote:
>
> [...] You're confusing whole programs with libraries.  [...] Only 
> developers (= users of third-party libraries) have to deal with code 
> compatibility issues.

I'm a practitioner and not a researcher, so I share many of Mr. 
Henrikson's concerns, but I think I see your point.

Most of my concerns revolve around the problems I expect to face in 
bundling an application that comprises many little programs that share 
one or more large libraries.  I'd like to see better support for dynamic 
loading, á la Scaml, but I can see that a policy of "source-only 
distribution" for libraries between developers is a completely different 
matter.

While the "source-only distribution" policy doesn't put a clamp on my 
personal plans, I do think that this decision will have to be revisited 
before Ocaml will make a successful transition from academia into 
industry.

One of the main attractions for industry to the C and Java languages are 
their crude-yet-functional support for "closed-source distribution" of 
libraries.  Without trying to make a political statement, I would 
suggest that Ocaml could make a *much* bigger dent in the side of the 
world, if the Caml team were to reconsider the concerns of industry in 
this matter.

I've already posted my opinion about application packaging systems, but 
it's not a religious issue for me, so I'll just let it go.


--
j h woodyatt <jhw@wetware.com>

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Sven]

On Wed, Mar 20, 2002 at 09:39:25PM +0100, Xavier Leroy wrote:
> But A.cmo does not contain SIGB -- only a mention of B and a hash of
> SIGB.  The reason is quite simple: A.cmo would be *huge* if it
> included a copy of the signatures of every module it refers, either
> directly or indirectly.  (Think several megabytes.)

Just curious, ...

would it not be enough if A.cmo had the type information of the part of B it
is going to use, and forget about the rest of it ? I remember some papers on
type discipline doing this kind of stuff.

> Having only hashes of signatures, all the linker can check is
> hash(SIGB) = hash(SIGB'), that is SIGB = SIGB'.
> 
> > Anyway, only being able to add functions is not sufficient. There are
> > many possible changes of modules that would not break "source-code 
> > compatibility", but would certainly break binary compatibility, e.g.
> > new optional arguments, new variants, new methods, etc.
> 
> This is another good point: even if we allowed SIGB' <: SIGB at
> link-time instead of requiring SIGB = SIGB', there are many
> "reasonable" changes in a signature that would break subtyping, like
> those you mention above.  And this is inevitable: these changes could
> break type-safety of existing programs.
> 
> To summarize: I believe binary compatibility doesn't really work in a
> strongly, statically-typed language.  It sometimes works for
> dynamically-typed or weakly-typed languages (Perl, C), although
> run-time failures are not unheard of.  Copious link-time
> re-type-checking as in Java makes it work sometimes too, although on
> close examination Java's binary compatibility rules are quite complex
> and restrictive.
> 
> Conclusion for OCaml?  Forget about binary compatibility, binary-only
> distributions, versioning, and all that.  All we need for OCaml
> libraries is source distribution, a standardized re-building procedure,
> and a place to record cross-library dependencies, so that everything
> that needs to be recompiled when something changes can be recompiled
> automatically.  Something like a source RPM or Debian package, or a
> BSD port.

Even for that, it is nice to have propper versioning support, right now, the
policy for debian packagers, is once the ocaml package changes (new upstream
release,, which more often than not breaks binary compatibility, but sometimes
only partially)., then all packagers need to recompile all the ocaml
libraries, and just to be sure, it is best to rebuild the program packages
also. But this needs still human intervention on the part of the developpers
to adapt the package if needed, and a lot of work for all the 8 arches
which will be supported by woody, without even counting all those not yet
fully supported arches which don't have autobuilders.

I would like a lot to have more information on versioning available.

> I believe this is a much more reasonable objective than dreaming about
> binary compatibility and whatnot.  Still, it's not trivial:
> the authors of existing OCaml libraries (myself included) have some
> work to do to implement fully automatic build of their libraries...

:)))

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Sven]

On Thu, Mar 21, 2002 at 07:10:28PM +0100, Xavier Leroy wrote:
> > Ugh.  I wholeheartedly disagree with source-only
> > distribution.  Such a choice condemns ocaml and all of its
> > applications to remain merely "hacker tools".  And all the
> > worse to mandate a "all downstream dependencies recompile"
> > policy.  I for one need to be able to someday hope that
> > "regular users" will use my program.  Every ignorant user
> > should _not_ be subjected to building and rebuild his own
> > large system because of some deep semantics issue which
> > researchers haven't quite cracked.
> 
> You're confusing whole programs with libraries.  A program can be
> distributed as a statically-linked executable (ocamlopt -ccopt -static
> or ocamlc -custom -ccopt -static), which will run happily on many,

Unless they are not running i386 that is, ...

> 
> This is not what I'm proposing.  It is perfectly feasible to make a
> binary OCaml "distribution" (in the sense of Debian and RedHat being
> "distributions") such as the CDK.  Someone (the distributor) just did
> all the work of building a large collection of sources for you.
> 
> Indeed, a truly automated build procedure like I propose would make
> the work of the CDK maintainers much easier.
> 
> What I think is futile is to expect users to pick independently-
> compiled binary distributions of libraries and dynamically-linked executables
> left and right, and expect that the resulting system will work.  Try
> to do this with a Linux system and it will break very quickly.
> 
> > (I don't think anybody has been so deluded to suggest this in a
> > large system.)  Now suppose you write device drivers.
> > Device driver programmers were frustrated enough before the
> > linux kernel was modularized so that they could load and
> > unload them without relinking the kernel and rebooting.  You
> > mean to tell them now that they must recompile not only the
> > kernel, but therefore the entire machine every time they
> > make a one line bugfix?  What kind of use would a machine
> > like that be?
> 
> More rethorical questions that don't make much sense to me.
> Loading/unloading kernel modules during kernel development is at best
> a dynamic loading issue.  That they might have to recompile their
> driver when the kernel changes doesn't bother kernel developers in the
> least.
> 
> > As a historical reference, the Debian project, having the
> > hacker mentality they do, apparently also fanatasized about
>                                             ^^^^^^^^^^^

This must have been a long time ago, today there is no quation on going away
with biunary package, altough there still exist some exception (like pine i
think, where only source distribution is authorized).

But then again, we distribute various rather slow arches (m68k and arm), and
there may be user on slow i386 systems out there also, i don't think they
would like to rebuild everything everytime something changed.

Binary distribution are the key of the success of GNU/Linux, without it, it
never would have attained the widespread use it has today.

> Again, I don't have anything against binary distributions, as long as
> there's an underlying source distribution and recompilation manager
> that lets me bring things up to date if a binary incompatibility
> arises.  RPMs get close to this but lack a "rebuild and install what
> needs to be recompiled" command.  Maybe Debian's packages or BSD ports
> are better in this respect.

on debian you can just do :

apt-get source -b ocaml

and it will fetch the ocaml source package and build it, you still need to
install it yourself.

There are build dependencies in package, it is possible to check for them, to
have them automatically installed when building a package, and other such
stuff, but i am not very familiar with them, it being a long time since i did
porting work (mostly m68k and powerpc).

We also have autobuilders, which build stuff for all the majors arches.

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Xavier Leroy]

> Any idea how much improvement would come from hash-consing, 
> as Dave Berry suggested? And only that part of a module's 
> signature which pertained to actual usage would need 
> inclusion, no? And only of those modules externally provided 
> upon which one's project depends, ie. the standard library, 
> not the internal project modules?

No idea.  It's certainly possible to reduce the size of the signature
information that has to be retained; how much and at what cost, I
don't know.

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Xavier Leroy]

> Ugh.  I wholeheartedly disagree with source-only
> distribution.  Such a choice condemns ocaml and all of its
> applications to remain merely "hacker tools".  And all the
> worse to mandate a "all downstream dependencies recompile"
> policy.  I for one need to be able to someday hope that
> "regular users" will use my program.  Every ignorant user
> should _not_ be subjected to building and rebuild his own
> large system because of some deep semantics issue which
> researchers haven't quite cracked.

You're confusing whole programs with libraries.  A program can be
distributed as a statically-linked executable (ocamlopt -ccopt -static
or ocamlc -custom -ccopt -static), which will run happily on many,
many systems, and users will never have to worry about binary
compatibility and whatnot.  Only developers (= users of third-party
libraries) have to deal with code compatibility issues.

> Someone suggested the need of "closed source" as an argument
> for binary distribution. I can make an argument that relies
> on less.  Assume ala mathematical induction OCaml
> proliferates.  Foobar Corp does their bookkeeping in ocaml
> and has 2 million lines and needs to make a bugfix to their
> SQL wrapper way down at the bottom of the heap.  Do you tell
> them they must wait a couple of days do get their system
> back up?

This is a strawman argument in many respects:

- The OCaml distribution is 100 KLOC and recompiles in a few minutes
  on a 1000 EUR machine.  So, 2MLOC takes less than one hour, not two days.  
  Chances are that if you're developing a program of this size,
  you'll invest in a multiprocessor or compilation farm, bringing
  the rebuild time even lower.  (Say, just long enough to have coffee,
  chat with your coworkers, and check Slashdot.  Isn't programmer's life
  sweet? :-)

- If I were to design an application of this size, I'll try very hard
  not to make it into one executable, but rather decompose it into
  several executables communicating via IPC.  Hence, only the parts
  that use the SQL wrapper have to be recompiled.

- Even if the program consists of one executable, it is easy to recompile
  only the modules that really depend on the SQL library: instead of doing
  "make clean; make all"; just generate dependencies with
  ocamldep -I/the/path/to/the/SQL/library and do "make".
  That we generally don't bother to record dependencies on
  third-party libraries in Makefiles doesn't mean it can't be done 
  if need arises.

> Imagine if the computer on your desk was built out of a (C
> based) source distro.  Not just a source distro, but one
> with the "all downstream dependencies recompile" policy.

This is not what I'm proposing.  It is perfectly feasible to make a
binary OCaml "distribution" (in the sense of Debian and RedHat being
"distributions") such as the CDK.  Someone (the distributor) just did
all the work of building a large collection of sources for you.

Indeed, a truly automated build procedure like I propose would make
the work of the CDK maintainers much easier.

What I think is futile is to expect users to pick independently-
compiled binary distributions of libraries and dynamically-linked executables
left and right, and expect that the resulting system will work.  Try
to do this with a Linux system and it will break very quickly.

> (I don't think anybody has been so deluded to suggest this in a
> large system.)  Now suppose you write device drivers.
> Device driver programmers were frustrated enough before the
> linux kernel was modularized so that they could load and
> unload them without relinking the kernel and rebooting.  You
> mean to tell them now that they must recompile not only the
> kernel, but therefore the entire machine every time they
> make a one line bugfix?  What kind of use would a machine
> like that be?

More rethorical questions that don't make much sense to me.
Loading/unloading kernel modules during kernel development is at best
a dynamic loading issue.  That they might have to recompile their
driver when the kernel changes doesn't bother kernel developers in the
least.

> As a historical reference, the Debian project, having the
> hacker mentality they do, apparently also fanatasized about
                                            ^^^^^^^^^^^
I like this new word :-)

> having a source only distribution (of course I don't imagine
> it was an "all downstream dependencies recompile" design):
> 
> >"At the very early stages of the Project, members
> considered
> >distributing source-only packages.  Each package would
> consist
> >of the upstream source code and a Debianized patch file,
> and users
> >would untar the sources, apply the patches, and compile
> binaries
> >themselves.  They soon realized, however, that some sort of
> >binary distribution scheme would be needed.  The earliest
> packaging
> >tool, written by Ian Murdock and called dpkg, created a
> package in
> >a Debian-specific binary format, and could be used later to
> unpack
> >and install the files in the package."

Again, I don't have anything against binary distributions, as long as
there's an underlying source distribution and recompilation manager
that lets me bring things up to date if a binary incompatibility
arises.  RPMs get close to this but lack a "rebuild and install what
needs to be recompiled" command.  Maybe Debian's packages or BSD ports
are better in this respect.

> To change the subject slightly, there seem to be some other
> dangerous related "purity before progress" idea bugs
> floating around, like

And there seems to be some dangerous "let's hack something before
thinking over the consequences" ideas in your message.

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Jeff Henrikson]

> Conclusion for OCaml?  Forget about binary compatibility,
binary-only
> distributions, versioning, and all that.  All we need for
OCaml
> libraries is source distribution, a standardized
re-building procedure,
> and a place to record cross-library dependencies, so that
everything
> that needs to be recompiled when something changes can be
recompiled
> automatically.  Something like a source RPM or Debian
package, or a
> BSD port.

Ugh.  I wholeheartedly disagree with source-only
distribution.  Such a choice condemns ocaml and all of its
applications to remain merely "hacker tools".  And all the
worse to mandate a "all downstream dependencies recompile"
policy.  I for one need to be able to someday hope that
"regular users" will use my program.  Every ignorant user
should _not_ be subjected to building and rebuild his own
large system because of some deep semantics issue which
researchers haven't quite cracked.  No matter how automated
recompilation could be, the (potentially repeated) time
penalty involved would be unacceptable.

Someone suggested the need of "closed source" as an argument
for binary distribution.  I can make an argument that relies
on less.  Assume ala mathematical induction OCaml
proliferates.  Foobar Corp does their bookkeeping in ocaml
and has 2 million lines and needs to make a bugfix to their
SQL wrapper way down at the bottom of the heap.  Do you tell
them they must wait a couple of days do get their system
back up?  Suppose they did one trial run and they find out
they put the wrong function call in.  Two more days, ad
nausem.

Imagine if the computer on your desk was built out of a (C
based) source distro.  Not just a source distro, but one
with the "all downstream dependencies recompile" policy.  (I
don't think anybody has been so deluded to suggest this in a
large system.)  Now suppose you write device drivers.
Device driver programmers were frustrated enough before the
linux kernel was modularized so that they could load and
unload them without relinking the kernel and rebooting.  You
mean to tell them now that they must recompile not only the
kernel, but therefore the entire machine every time they
make a one line bugfix?  What kind of use would a machine
like that be?

As a historical reference, the Debian project, having the
hacker mentality they do, apparently also fanatasized about
having a source only distribution (of course I don't imagine
it was an "all downstream dependencies recompile" design):

>"At the very early stages of the Project, members
considered
>distributing source-only packages.  Each package would
consist
>of the upstream source code and a Debianized patch file,
and users
>would untar the sources, apply the patches, and compile
binaries
>themselves.  They soon realized, however, that some sort of
>binary distribution scheme would be needed.  The earliest
packaging
>tool, written by Ian Murdock and called dpkg, created a
package in
>a Debian-specific binary format, and could be used later to
unpack
>and install the files in the package."

from:
http://www.debian.org/doc/manuals/project-history/ch4.html


To change the subject slightly, there seem to be some other
dangerous related "purity before progress" idea bugs
floating around, like

garrigue@kurims.kyoto-u.ac.jp wrote:

> The real problem is about how to check that binary (and
> semantics) compatibility is satisfied.

Exactly what sort of semantics?  Certainly the general case
is turing-complete.  What reasonable subset could you mean
by this?  And just why do we care?  Nobody's pretending type
checking makes your program correct.


>>>>I don't see this.  I can believe that consequences of
implementation
>>>>choices which have been made prohibit this.  For
hypothetical example,
>>>>inlining behavior which could not be disabled on public
interfaces would
>>>>be a problem.  (I don't think this particular thing
happens in ocaml.)
>>>>But I certainly don't see "a consequence of strict
typing."
>>>>
>>
>> I agree.  It is not true of other languages with static t
yping,
>> e.g. Mercury or C++.
>
>I don't know Mercury but C++. Actually, it does not ensure
proper
>typing between compilation units! The .o files do not
contain types,
>and it is up to the programmer to include the right
prototypes.

Yes, it's called using the right header files.  That and
version numbering your runtime libraries.  Just like the
rest of the computing world.   Apparently forced type
checking is an open problem, so lets not sit and cry over
it.  Fortunately it hasn't been solved anywhere else, so it
is a universally understood programmer's convention to
couple libraries with source code interfaces.  Are we really
so special that we must break it?

This is the scenario I envision.  I don't think the design
choices above are hard, though there still may be some
substantial work because apparently a mutually exclusive set
of assumptions has governed the evolution of the compiler up
until this point.

	- The compiler must guarentee a fixed calling convention
and representation for all concrete types for a given .mli
file.
	- Types parameterized by abstract types (eg "list", sorry
if my jargon is ignorant) must not change their compilation
based on knowledge of a new type.
	- Functors must have the analogous property
	- If the above is not generally runtime efficient, then it
must be made so by providing a "public" or "external" or
some such keyword to allow the interfaces which are exposed
to have these properties.

(corrections/elaborations to the above are welcome feedback)

There _do_ currently exist design choices in the ocaml
sources that arbitrarily for no reason violate this.  One of
them was fixed for SCaml, which was that arbitrary gensymed
numbers were being added to external function definitions
from ocamlopt.  Eg I implement:

foo.ml
	let f x = x + 1;;

and the compiler generates a .so symbol called "f17,"
whereas

foo.ml
	let f x = x + x + 1;;

might generate "f23" or some other arbitrary thing.  Merely
changing the length of procedure bodies can make a different
number of gensym requests happen and therefore the number 17
morphs into something else.  Not good for linking against.
The SCaml patch does fix this particular thing, but there
are likely to be others.

I reiterate that I haven't seen a concrete counterexample
which demonstrates that the semantics themselves make
runtime interfaces impossible.  A previous post of mine
answered with one counterexample which changed optional
arguments across a public interface.  That won't work in C++
or any other of the myriad languages which do have
deterministic runtime interfaces.  Apologies that I have not
read in detail a couple of examples using functors that came
in yesterday, but I had to finish this email sometime.

I additionally point out that on course granularity, dynamic
typing ala COM interface IDs and class IDs works pretty
well.  (These are just large 128 bit numbers based on things
like the creator's ethernet hardware address and a fine
granularity time stamp, designed to be a unique type code
for every possible interface and class).  The systems which
don't have the right library versions at least crash right
away when the module is linked rather than run for a while
and segfault when a function is utilized.  It's not perfect,
but its ubiquity demonstrates that it works in practice.

In closing, I'd like to ask a more tenuous and subjective
question.  Is there any fundamental reason why the caml
package manager can't _be_ the Debian package manager?  I
think this would be a big step in the maturity and ubiquity
of the OCaml community at large.  And I mean dependencies
_both_ ways, C calling Caml as well as Caml calling C and
other Caml.  (Not that people these days are publishing a
lot of C interfaces for their Caml libraries.  To integrate
with other programmer communities that should change but
it's another issue.)  I haven't thought about this one a
lot, but having a quirky on-the-side package manager for a
special quirky language gives to me the general impression
of obscurity by design.


Jeff Henrikson




-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Christopher Quinn]

> 
> But A.cmo does not contain SIGB -- only a mention of B and a hash of
> SIGB.  The reason is quite simple: A.cmo would be *huge* if it
> included a copy of the signatures of every module it refers, either
> directly or indirectly.  (Think several megabytes.)

Any idea how much improvement would come from hash-consing, 
as Dave Berry suggested? And only that part of a module's 
signature which pertained to actual usage would need 
inclusion, no? And only of those modules externally provided 
upon which one's project depends, ie. the standard library, 
not the internal project modules?

Inclusion of such signature fragments need only be an 
optional feature when building distributed libraries, not 
generally for development (inhabiting every .cmo), so the 
'bloat' is at least confined to where it matters, at the 
boundary.

Chris Q.

be an optional feature of libraries alone, thus confining 
any sort of bloat to distributed libs, commercial or otherwise?

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Warp]

> Conclusion for OCaml?  Forget about binary compatibility, binary-only
> distributions, versioning, and all that.  All we need for OCaml
> libraries is source distribution, a standardized re-building procedure,
> and a place to record cross-library dependencies, so that everything
> that needs to be recompiled when something changes can be recompiled
> automatically.  Something like a source RPM or Debian package, or a
> BSD port.

So, do we need to forget about any commercial usage of OCaml ?
In fact, commercial usage needs real "closed-source" libraries where only
bytecode files are distributed. Such libraries should be able to evolve , by
adding new functions into the interface, and to be loaded with signature
checking at run-time. Does the OCaml team don't plan at all to add such
feature ?

Nicolas Cannasse

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Markus Mottl]

On Wed, 20 Mar 2002, Xavier Leroy wrote:
> All we need for OCaml libraries is source distribution, a standardized
> re-building procedure,

Yes, please!

> I believe this is a much more reasonable objective than dreaming
> about binary compatibility and whatnot.  Still, it's not trivial:
> the authors of existing OCaml libraries (myself included) have some
> work to do to implement fully automatic build of their libraries...

Good to know that the INRIA-team is not neglecting this burning issue
and thinking about solutions. I really hope that OCaml libraries get the
reverse transcriptase they need to become really virulent for spreading
to all machines on this planet :-)

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Xavier Leroy]

> Oh, sorry, I was here on the wrong track. Adding a function does not
> break functor applications. Is there any case where module signatures
> must exactly match?

A manifest module type declaration in a signature must match exactly the
corresponding module type definition in the structure.

But you're correct that a structure with a given signature S can
always match a signature S' that is less restrictive, e.g. S has one
more function than S'.

Concerning the compilation of accesses inside modules: it is true that
structure components are accessed primarily by position ("fetch the
N-th component defined in this module"); access by name is only used
by ocamlopt for calls to known functions.  

This doesn't conflict with the ability to forget structure components:
at the same time that the compiler checks that a structure with
signature S can be used with signature S' :> S, it also generates
adapter code that reorders the components of the structure to match
what S' expects.

So, subtyping between module types and the associated generation of
adapter code handles perfectly well compatible extensions of an
existing module.  

Now, you will ask: why doesn't this work for multiple versions of a
compilation unit?  Why doesn't the linker check subtyping and generate
adapter code?  Well, it could work *if* every compilation unit came
with the complete signatures of all compilation units it refers.

Say, unit A references B and was compiled under the assumption B :
SIGB.  Then, we link A with another implementation of B that has
signature SIGB'.  If the linker knew both SIGB and SIGB', it could
check SIGB' <: SIGB and adapt A's references to B's components
accordingly.

But A.cmo does not contain SIGB -- only a mention of B and a hash of
SIGB.  The reason is quite simple: A.cmo would be *huge* if it
included a copy of the signatures of every module it refers, either
directly or indirectly.  (Think several megabytes.)

Having only hashes of signatures, all the linker can check is
hash(SIGB) = hash(SIGB'), that is SIGB = SIGB'.

> Anyway, only being able to add functions is not sufficient. There are
> many possible changes of modules that would not break "source-code 
> compatibility", but would certainly break binary compatibility, e.g.
> new optional arguments, new variants, new methods, etc.

This is another good point: even if we allowed SIGB' <: SIGB at
link-time instead of requiring SIGB = SIGB', there are many
"reasonable" changes in a signature that would break subtyping, like
those you mention above.  And this is inevitable: these changes could
break type-safety of existing programs.

To summarize: I believe binary compatibility doesn't really work in a
strongly, statically-typed language.  It sometimes works for
dynamically-typed or weakly-typed languages (Perl, C), although
run-time failures are not unheard of.  Copious link-time
re-type-checking as in Java makes it work sometimes too, although on
close examination Java's binary compatibility rules are quite complex
and restrictive.

Conclusion for OCaml?  Forget about binary compatibility, binary-only
distributions, versioning, and all that.  All we need for OCaml
libraries is source distribution, a standardized re-building procedure,
and a place to record cross-library dependencies, so that everything
that needs to be recompiled when something changes can be recompiled
automatically.  Something like a source RPM or Debian package, or a
BSD port.

I believe this is a much more reasonable objective than dreaming about
binary compatibility and whatnot.  Still, it's not trivial:
the authors of existing OCaml libraries (myself included) have some
work to do to implement fully automatic build of their libraries...

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Alain Frisch]

On Wed, 20 Mar 2002, Gerd Stolpmann wrote:

> Oh, sorry, I was here on the wrong track. Adding a function does not
> break functor applications. Is there any case where module signatures
> must exactly match?

I'm not sure this answer the question, but for instance, if you have:

a.ml empty

b.ml:
let f x = x +. 1.
include A

b.mli:
val f: float -> float


then you are not allowed to add:

let f x = x + 1

to a.ml


-- Alain

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Fergus Henderson]

On 20-Mar-2002, Jacques Garrigue <garrigue@kurims.kyoto-u.ac.jp> wrote:
> From: Fergus Henderson <fjh@cs.mu.OZ.AU>
> 
> > Adding new functions to a module ought not break binary backwards
> > compatibility.  If it does, then you lose many of the benefits of
> > separate compilation.
> 
> Could you specify what benefits?

For example, the ability to install a new version of a library
(with some new functions) without having to recompile everything
that references that library.

This includes the ability to use version N+1 of library `foo' with
version M of library `bar', even though the binary that you have for
version M of library `bar' was build using version N of library `foo'
rather than version N+1.

Binary distributions of libraries is very difficult if you have to
upgrade them all in lock-step anytime a base library changes.

> The current situation in OCaml is that you have to recompile all
> dependencies everytime you change anything in the interface.
> If you use a Makefile, even changing a comment will trigger a
> recompilation.

If you're sure nothing important changed, you can always use
`touch 01010101 filename' to mark that file as old.

> But, in my experience most C makefiles are written in the same way
> meaning that you have to recompile everytime a header changes.

Most C makefiles are written so that dependencies on third-party
library header files are not included in the Makefile dependencies.
(And there are good reasons for this; including such
dependencies can cause as many problems as it solves...)
That imples that if you install a new version of a third-party
library, and then rerun make, it won't recompile your sources.

Also, the issue of makefiles is irrelevant in the case of binary
library distributions, as in the example above with `foo' and `bar'.

> The real problem is about how to check that binary (and semantics)
> compatibility is satisfied.

Sure it would be *nice* if the system checked that automatically,
and handled library major/minor revision numbers accordingly.

But the programmer can also check this manually when
releasing a new version of a library, by comparing the
interface with that of the previous version (e.g. using `cvs diff').
That's what C and C++ programmers have to do, after all.

There are really two issues here.  One is that the language and/or
implementation should give guarantees about what kinds of changes
will preserve binary compatibility, including at a minimum the
requirement that adding new functions should not break binary
compatibility.  The second issue is that it's desirable to have
type-safe linking.

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Gerd Stolpmann]

Johan Georg Granström wrote:
>>>I think that rather than being a consequence of strict typing, it is a
>>>possible consequence of treating modules as more-or-less first class,
>>>if you use a representation of modules in which adding a new function
>>>does not preserve binary compatibility.  Does O'Caml do that?
>>>
>>It is also a matter of typing because even toplevel modules can be
>>used to parameterize functors. So adding a new function may break
>>functor applications.
>>
> 
> I don't understand this, can you give an example of such
> a case?

Oh, sorry, I was here on the wrong track. Adding a function does not
break functor applications. Is there any case where module signatures
must exactly match?

Anyway, only being able to add functions is not sufficient. There are
many possible changes of modules that would not break "source-code 
compatibility", but would certainly break binary compatibility, e.g.
new optional arguments, new variants, new methods, etc.

Gerd

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Johan Georg Granström]

> > I think that rather than being a consequence of strict typing, it is a
> > possible consequence of treating modules as more-or-less first class,
> > if you use a representation of modules in which adding a new function
> > does not preserve binary compatibility.  Does O'Caml do that?
>
> It is also a matter of typing because even toplevel modules can be
> used to parameterize functors. So adding a new function may break
> functor applications.

I don't understand this, can you give an example of such
a case?

Yours,

- Johan Granström

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Gerd Stolpmann]

Fergus Henderson wrote:
> On 12-Mar-2002, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> 
>>On 2002.03.12 01:19 Jeff Henrikson wrote:
>>
>>>>In O'Caml replacing library X by a newer version usually means that
>>>>all libraries Y that depend on X must be recompiled. And there is no
>>>>guarantee that Y can be compiled at all. I do not see any chance to
>>>>change this, it is a consequence of strict typing.
>>>>
>>>I don't see this.  I can believe that consequences of implementation
>>>choices which have been made prohibit this.  For hypothetical example,
>>>inlining behavior which could not be disabled on public interfaces would
>>>be a problem.  (I don't think this particular thing happens in ocaml.)
>>>But I certainly don't see "a consequence of strict typing."
>>>
> 
> I agree.  It is not true of other languages with static typing,
> e.g. Mercury or C++.

I don't know Mercury but C++. Actually, it does not ensure proper
typing between compilation units! The .o files do not contain types,
and it is up to the programmer to include the right prototypes.

> I think that rather than being a consequence of strict typing, it is a
> possible consequence of treating modules as more-or-less first class,
> if you use a representation of modules in which adding a new function
> does not preserve binary compatibility.  Does O'Caml do that?

It is also a matter of typing because even toplevel modules can be
used to parameterize functors. So adding a new function may break
functor applications.

My point is not that it is impossible to flexibilize linking, but that
it is not simple to do so. I think typing should be strict even across
compilation units, and that the programming environment must check
it. Strict typing also means to ensure compatible memory layouts but
this is not the only meaning. For example, it must be still possible to
hide the representation of types.

> Adding new functions to a module ought not break binary backwards
> compatibility.  If it does, then you lose many of the benefits of
> separate compilation.

I think you mean "independent compilation" here, i.e. the linker does
not check proper typing. It is simple to make errors that are hard to
find. You need expert knowledge to find out which changes are binary
compatible and which not. Better we don't have it.

Gerd

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Jacques Garrigue]

From: Fergus Henderson <fjh@cs.mu.OZ.AU>

> Adding new functions to a module ought not break binary backwards
> compatibility.  If it does, then you lose many of the benefits of
> separate compilation.

Could you specify what benefits?
The current situation in OCaml is that you have to recompile all
dependencies everytime you change anything in the interface.
If you use a Makefile, even changing a comment will trigger a
recompilation.
But, in my experience most C makefiles are written in the same way
meaning that you have to recompile everytime a header changes.

The real problem is about how to check that binary (and semantics)
compatibility is satisfied. Adding a function might be OK, but
changing a type is not OK (at least not always; Jun Furuse had some
work on it).
The OCaml approach being to chomp all the interface in a single MD5
value, any meaningful change (including addition of a function) will
prevent you from linking without recompiling.
There were good remarks on the list on how a progressive hashing
algorithm (allowing versioning) would be needed to improve that.

> Does adding new functions to a module actually break binary backwards
> compatibility in O'Caml?

At least it breaks for bytecode, where simple indexes are used to get
closures.
But this is not the point: the compiler will not let you link anyway.
I suppose this would be easily corrected if the semantics problem were
solved.

I have a strong feeling that what we need is a good versioning system,
at the language or (probably simpler) library level. But the problem
doesn't seem trivial, in particular if you want efficient checking for
dynamic linking.

Cheers,

Jacques Garrigue
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Fergus Henderson]

On 12-Mar-2002, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> 
> On 2002.03.12 01:19 Jeff Henrikson wrote:
> > > In O'Caml replacing library X by a newer version usually means that
> > > all libraries Y that depend on X must be recompiled. And there is no
> > > guarantee that Y can be compiled at all. I do not see any chance to
> > > change this, it is a consequence of strict typing.
> > 
> > I don't see this.  I can believe that consequences of implementation
> > choices which have been made prohibit this.  For hypothetical example,
> > inlining behavior which could not be disabled on public interfaces would
> > be a problem.  (I don't think this particular thing happens in ocaml.)
> > But I certainly don't see "a consequence of strict typing."

I agree.  It is not true of other languages with static typing,
e.g. Mercury or C++.

I think that rather than being a consequence of strict typing, it is a
possible consequence of treating modules as more-or-less first class,
if you use a representation of modules in which adding a new function
does not preserve binary compatibility.  Does O'Caml do that?

> > Can you
> > give a specific example?
> 
> Usually, a new version of a library modifies the signature. Ok, these
> are often only minor modifications: some new functions, new optional
> arguments etc., and normally the new version is "source-level" compatible
> with the old version. "Source-level" means that "normal" usage does not
> cause incompatibilities.

In Mercury and C++, often changes are not completely source-level
compatible, but nevertheless preserve binary compatibility.
For example, adding a new function may introduce new ambiguity errors in
existing source code, in case the same name occurs in a different module.
However, binaries will always refer to the fully-qualified name, so they
won't be affected by such ambiguities.

> But there are cases where the compiler indicates a typing error:
> 
> - You can pass f as such around. This makes a difference because the type of f is different
>   and the deduced types will be different, and it may happen that the deduced types cannot
>   be accepted, because sometimes the optional argument is automatically dropped and sometimes
>   not.

Well, if you changed the interface to `f', e.g. by adding an optional
argument, that is not a binary-backwards-compatible change.
The same happens in C++ if you add an optional argument.

> So one precondition of replacing the library is that the signatures are
> identical. Even small changes cannot be tolerated.

Adding new functions to a module ought not break binary backwards
compatibility.  If it does, then you lose many of the benefits of
separate compilation.

Does adding new functions to a module actually break binary backwards
compatibility in O'Caml?

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Gerd Stolpmann]

On 2002.03.12 01:19 Jeff Henrikson wrote:
> > In O'Caml replacing library X by a newer version usually means that
> > all libraries Y that depend on X must be recompiled. And there is no
> > guarantee that Y can be compiled at all. I do not see any chance to
> > change this, it is a consequence of strict typing.
> 
> I don't see this.  I can believe that consequences of implementation choices which have been made prohibit this.  For hypothetical
> example, inlining behavior which could not be disabled on public interfaces would be a problem.  (I don't think this particular
> thing happens in ocaml.)  But I certainly don't see "a consequence of strict typing."  Can you give a specific example?

Usually, a new version of a library modifies the signature. Ok, these are often only minor
modifications: some new functions, new optional arguments etc., and normally the new version
is "source-level" compatible with the old version. "Source-level" means that "normal" usage
does not cause incompatibilities.

An example: The old version defines a function 

(* OLD: *) val f : int -> int

and the new version changes the signature into

(* NEW: *) val f : ?option:bool -> int -> int

If "normal usage" means that f is only applied, the new version will be backwards compatible.
But there are cases where the compiler indicates a typing error:

- You can pass f as such around. This makes a difference because the type of f is different
  and the deduced types will be different, and it may happen that the deduced types cannot
  be accepted, because sometimes the optional argument is automatically dropped and sometimes
  not.
- The module defining f can be used as parameter of a functor. The new version has a different
  signature, and is not accepted as parameter any more.

So one precondition of replacing the library is that the signatures are identical. Even small
changes cannot be tolerated.

I am not an expert, and I do not know how the optional arguments exactly work, but it is
possible that the representation of the closure f has changed, too. In general, I expect
that "source-level" compatible modifications may change the representation of values.

Inlining is another reason (only for ocamlopt), but this can be turned off.

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Jeff Henrikson]

> In O'Caml replacing library X by a newer version usually means that
> all libraries Y that depend on X must be recompiled. And there is no
> guarantee that Y can be compiled at all. I do not see any chance to
> change this, it is a consequence of strict typing.

I don't see this.  I can believe that consequences of implementation choices which have been made prohibit this.  For hypothetical
example, inlining behavior which could not be disabled on public interfaces would be a problem.  (I don't think this particular
thing happens in ocaml.)  But I certainly don't see "a consequence of strict typing."  Can you give a specific example?


Jeff Henrikson




-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Gerd Stolpmann]

Mark D. Anderson wrote:
> If someone is going to work on this, I'd recommend they look even more deeply
> at what perl and python do. It is more than just "download over http"; there
> are also issues of versioning among others.

But there is a fundamental difference: These language don't have strict
typing, and they bind identifiers always dynamically. It is no problem
to replace a version of a library by a newer one. Of course, this can
also break functionality, but it is possible to develop module
interfaces conservatively.

In O'Caml replacing library X by a newer version usually means that
all libraries Y that depend on X must be recompiled. And there is no
guarantee that Y can be compiled at all. I do not see any chance to
change this, it is a consequence of strict typing.

So an automatic installer for O'Caml needs:

- A local source repository that stores all sources so modules can
   be recompiled if necessary

- Knowledge about the versions of modules, especially which versions
   of dependent modules work together and which don't

> Some of the things both those languages have, to greater or lesser degrees:
> 
> - there is a web site for humans with search and list, such as search.cpan.org

We have already two, but they are for humans only.

> - there is a command line which does search, list, and install, such as 
> "perl -MCPAN -e shell" or "ppm"
> (ppm is more an activestate thing; CPAN.pm is distributed)
> The install can deal with recursive dependencies if you want.

If it is only search/list/install package management is quite easy.
We need a file format that contains the instructions how to build
a package from source.

But upgrades are more complicated, as explained.

> - the install utility can handle pure language packages, mixed language packages
> with C source, and mixed language packages with pre-compiled C source
> for one architecture (ppm is less flexible but simpler).
> 
> - really obscene things can be done with the perl "Inline" module: http://search.cpan.org/search?dist=Inline
> 
> - a single language install tree can handle a mixture of binary modules that vary by
> architecture (i686-linux vs. MSWin32-x86-multi-thread) or by language version (5.005 vs. 5.6).
> pure-language modules can vary by language version.
> This is useful for multiple developers sharing a single install over NFS, or for
> a single developer that is trying out multiple configurations.

Again, this works only because the modules are loosely coupled.

> - the language runtime does best effort when a package is asked for by name,
> taking the "best-fit" one;
> this prevents you having to upgrade all packages just because the language is
> upgraded.

Maybe we are on the wrong track if we want to mimick Perl's installer.
Maybe it is better not to have a fully automatic system but a system
that is more user-friendly. I can imagine a graphic tool would be
helpful that visualizes the current state of the installation, that
explains possible operations, and that assists the user when carrying
them out.

For example, if a package is upgraded, this tool would not try to
recompile the dependent packages automatically, but it would display
that these packages are currently invalid and need to be replaced.
The user would have the chance to control the next steps. One
possibility would be to try to recompile the old version again,
another would be to get a new version from a remote repository.
The point is that it is under the control of the user, and that
the graphic visualization allows even beginners to manage
upgrades.

Gerd

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

Does CPAN also use information about installed libraries, 
for example which versions of glibc are available?

/mattias

> -----Original Message-----
> From: owner-caml-list@pauillac.inria.fr 
> [mailto:owner-caml-list@pauillac.inria.fr] On Behalf Of Mark 
> D. Anderson
> Sent: Monday, March 11, 2002 5:28 AM
> To: caml-list@inria.fr
> Subject: Re: [Caml-list] The DLL-hell of O'Caml
> 
> 
> If someone is going to work on this, I'd recommend they look 
> even more deeply at what perl and python do. It is more than 
> just "download over http"; there are also issues of 
> versioning among others.
> 
> Some of the things both those languages have, to greater or 
> lesser degrees:
> 
> - there is a web site for humans with search and list, such 
> as search.cpan.org
> 
> - there is a command line which does search, list, and 
> install, such as 
> "perl -MCPAN -e shell" or "ppm"
> (ppm is more an activestate thing; CPAN.pm is distributed)
> The install can deal with recursive dependencies if you want.
> 
> - the install utility can handle pure language packages, 
> mixed language packages with C source, and mixed language 
> packages with pre-compiled C source for one architecture (ppm 
> is less flexible but simpler).
> 
> - really obscene things can be done with the perl "Inline" 
> module: http://search.cpan.org/search?dist=Inline
> 
> - a single language install tree can handle a mixture of 
> binary modules that vary by architecture (i686-linux vs. 
> MSWin32-x86-multi-thread) or by language version (5.005 vs. 
> 5.6). pure-language modules can vary by language version. 
> This is useful for multiple developers sharing a single 
> install over NFS, or for a single developer that is trying 
> out multiple configurations.
> 
> - the language runtime does best effort when a package is 
> asked for by name, taking the "best-fit" one; this prevents 
> you having to upgrade all packages just because the language 
> is upgraded.
> 
> -mda
> 
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs
FAQ: http://caml.inria.fr/FAQ/ Beginner's list:
http://groups.yahoo.com/group/ocaml_beginners

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Mark D. Anderson]

If someone is going to work on this, I'd recommend they look even more deeply
at what perl and python do. It is more than just "download over http"; there
are also issues of versioning among others.

Some of the things both those languages have, to greater or lesser degrees:

- there is a web site for humans with search and list, such as search.cpan.org

- there is a command line which does search, list, and install, such as 
"perl -MCPAN -e shell" or "ppm"
(ppm is more an activestate thing; CPAN.pm is distributed)
The install can deal with recursive dependencies if you want.

- the install utility can handle pure language packages, mixed language packages
with C source, and mixed language packages with pre-compiled C source
for one architecture (ppm is less flexible but simpler).

- really obscene things can be done with the perl "Inline" module: http://search.cpan.org/search?dist=Inline

- a single language install tree can handle a mixture of binary modules that vary by
architecture (i686-linux vs. MSWin32-x86-multi-thread) or by language version (5.005 vs. 5.6).
pure-language modules can vary by language version.
This is useful for multiple developers sharing a single install over NFS, or for
a single developer that is trying out multiple configurations.

- the language runtime does best effort when a package is asked for by name,
taking the "best-fit" one;
this prevents you having to upgrade all packages just because the language is
upgraded.

-mda


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Jacques Garrigue]

Hi,

> On my linux-machine I use the CDK with ocaml-3.03ALPHA.

Then you should upgrade. As shown by its name, this is an alpha release,
whose compatibility with future releases is not guaranteed.
If I remember correctly, it names DLLs differently from 3.04,
and you should scrape it away before installing 3.04, lest you get
into lots of trouble.

> This is the installation history:
> 1. Needs Ocaml 3.03 -> updated CDK using CVS (there are no tarballs
> but I have a broadband-connection).
> 2. Ocamlnet missing -> downloaded ocamlnet-0.92
> 3. Failed to install ocamlnet, since the PCRE in the CDK seems to
> be to old.
> 4. Installed a new PCRE.
> 5. Tried to install ocamlnet again, however failed with 
> 	"unbound type constructor Mimestring.s_param"
> 
>    Probably something else is too old...I got bored...
>    ...never got back to try to install WDialog....

I sympathize...

> I know 3 solutions to the problem:
> 1. CDK
> 2. Gerd invented findlib to solve the above problem.
> 3. Adding good packages to the standard distribution,
>    so that mostly packages doesn't depend on other
>    packages, but on packages in the standard
>    distribution. (But very few if any new packages
>    are added to the distribution.)

Personally, I think that the combination of (1) and (2) could help a
lot. Make the powerful findlib a standard. Then it would probably end
up one day in the standard distribution...
I would add also the role of the O'Caml link database, which is
particularly nice as authors can update themselve their information.

Adding packages to the standard distribution could easily make it
huge. It is already not so small. Note also the tendancy to have n
libraries for the same goal, which just reflects how people are
interested just as much in how it is done as what is done.

(I know, I should start by using findlib for my libraries.
But like everybody else, I'm lazy...)

It will still be difficult to get things as smooth as in Perl: strong
typing is good at detecting problem early, so early that it won't let
you compile some code because of a single function you don't need
anyway.
Here is a more theoretical problem: how to determine the cause of a
compilation error, whether this is some change in the language, or in
a library, and which library.

Cheers,

Jacques Garrigue
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Sven]

On Sat, Mar 02, 2002 at 08:42:40PM +0100, Mattias Waldau wrote:
> > From: Chris Hecker [mailto:checker@d6.com] 
> > I've used CPAN a little bit, but we should do a survey of 
> > other archive 
> > managers and learn from what works and what doesn't.  Maybe 
> > there's a web 
> > page somewhere.  There's also apt-get from Debian and the 
> 
> apt-get, interesting idea. (does it work for windows?)

Mmm, i don't think so, altough i know the darwin project use a variation of
dpkg and apt-get for their package management. You are free to port it though,
if you want, it is open source anyway.

Also notice, that appart from the nice apt-get tool, much work has been done
to package the new ocaml 3.04 and to solve the problems related to shared libs
on debian systems. We even use a small tool ocaml-ldconf (quickly hacked by
me, so it is not optimal, but does its work) which uses 3 databases to store
the ld.conf files, one is the normal ld.conf from ocaml, and the other two
being the dpkg handled /var/lib/ocaml/ld.conf, and a system administrator
override in /etc/ocaml/ld.conf.

I would gladly contribute it if it seems usefull and people want it, but i am
not sure what help it would be without the integration permitted by apt-get,
dpkg and the debian packaging system (and the work of the debian/ocaml
maintainer team :)))

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Vitaly Lugovsky]

On Sat, 2 Mar 2002, Chris Hecker wrote:

> >apt-get, interesting idea. (does it work for windows?)
> 
> I doubt it, but it probably wouldn't be that hard to port (he says, 
> cluelessly).

 It's already ported to RPM (e.g. in AltLinux distribution),
and RPM works with Cygwin32, so, it's not a problem.


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Chris Hecker]

>apt-get, interesting idea. (does it work for windows?)

I doubt it, but it probably wouldn't be that hard to port (he says, 
cluelessly).

>compilation and windows is a problem.

Right, good point, I forgot about that.  Although, that does imply we're 
targeting both developers and users, which will make things harder.

 > I got the impression that the byte-code is compatible between
>different platforms (Is this true?)

I believe this is true.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Alexander V. Voinov]

Hi Markus et al.

Markus Mottl wrote:
> Some standard packaging tools + automated means of resolving dependencies,
> e.g. by downloading contributions from some central archive, and a
> documentation standard (ocamldoc?) would surely help. Any concrete plans?

What CDK is? How is it related to the 'mainstream' OCaml? Why it's
always aside and out of date w.r.t. official releases?

Alexander
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

> From: Chris Hecker [mailto:checker@d6.com] 
> I've used CPAN a little bit, but we should do a survey of 
> other archive 
> managers and learn from what works and what doesn't.  Maybe 
> there's a web 
> page somewhere.  There's also apt-get from Debian and the 

apt-get, interesting idea. (does it work for windows?)

>  >> In the Ocaml-CPAN it could either be source code, or
> > > compile binaries (I can live without native code, at
> > > least when experimenting with other peoples libraries.)
> 
> Why wouldn't it be source-only?  So people don't have to have 

compilation and windows is a problem. even if many windows users
use cygwin, I think the native port is very important. If there
would be commercial applications written in O'Caml, I think
they will be written for Windows (Anyone know any commercial
applications for *nix? :-) 

I got the impression that the byte-code is compatible between
different platforms (Is this true?), and then this would
simplify distribution, since especially windows-users wouldn't
have the compilationsproblems (unless they want to change the
code or use the native compiler).


/mattias

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Chris Hecker]

I think a CPAN-like-thing would be excellent and is sorely needed!  I'm 
super busy right now (who isn't?), but I'd be willing to commit some time 
to testing it at the least, and maybe contributing code after May.

I've used CPAN a little bit, but we should do a survey of other archive 
managers and learn from what works and what doesn't.  Maybe there's a web 
page somewhere.  There's also apt-get from Debian and the Redhat one to 
look at.

I wonder if we could just use one of the existing managers...although it 
seems like it should be written in caml for karmic reasons, at least.  :)

 >> In the Ocaml-CPAN it could either be source code, or
> > compile binaries (I can live without native code, at
> > least when experimenting with other peoples libraries.)

Why wouldn't it be source-only?  So people don't have to have the compiler 
installed?  It seems that there are two related but different projects:  an 
archive/package manager for developers, and one for users.  Maybe those can 
be the same thing, at least for bytecode.  But maybe the intermediate rep 
idea would work for native as well.  Or some server that builds all the 
supported platforms or something.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Alain Frisch]

On Sat, 2 Mar 2002, Mattias Waldau wrote:

> The above history is the reason I started using the CDK,
> in many cases using libraries that depend on other
> libraries is almost impossible without the CDK. Too much
> time is spent on downloading and compiling.

A related issue is independance w.r.t OCaml version; if you want to try a
new OCaml release or CVS version, it is necessary to rebuild all the
libraries.  For instance, I now have three OCaml versions installed (3.04,
3.04+7 and 3.04+6 polymorphic methods), and it is a pain to maintain three
library trees.

Similarly, when you upgrade a library, you also have to rebuild all
the libraries that depend on it. Some kind of automatization in the
process would be great.

> I know 3 solutions to the problem:
> 1. CDK
> 2. Gerd invented findlib to solve the above problem.
> 3. Adding good packages to the standard distribution,
>    so that mostly packages doesn't depend on other
>    packages, but on packages in the standard
>    distribution. (But very few if any new packages
>    are added to the distribution.)

The monolithic and centralized approach of CDK does not seem to scale
well. Findlib does a good job. It would be great to have such a tool
(findlib or a clone, but why not findlib ?) in the standard
distribution...


> We need a solution. Maybe a CPAN-like solution?

Indeed, the new packaging facility in OCaml (-pack option, which glues
together a set of modules and allows decent namespace management) opens
the door to this kind of project.

> In the Ocaml-CPAN it could either be source code, or
> compile binaries (I can live without native code, at
> least when experimenting with other peoples libraries.)

An intermediate solution would be to distribute so-called "lambda-code",
that is the internal representation used in OCaml compilers just before
code production. The advantage is that it is common to bytecode and
native back-end, and it should be possible to design an upward-compatible
(w.r.t OCaml version) concrete representation of this code.


-- Alain

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Markus Mottl]

On Sat, 02 Mar 2002, Mattias Waldau wrote:
> We need a solution. Maybe a CPAN-like solution?  In the Ocaml-CPAN it
> could either be source code, or compile binaries (I can live without
> native code, at least when experimenting with other peoples libraries.)

This raises the old question of "social tools" again. Given that our
community is still comparatively small, it is absolutely mandatory
that there be tools and standards that make us as productive as
possible. Not only will this allow us to use our scarce human resources
more efficiently, it is also more likely to attract more people when they
see that contributing and benefiting from other contributions is easy.

Some standard packaging tools + automated means of resolving dependencies,
e.g. by downloading contributions from some central archive, and a
documentation standard (ocamldoc?) would surely help. Any concrete plans?

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

(THIS IS NOT A BUG REPORT ON WDIALOG, WDIALOG IS JUST AN EXAMPLE!)

I wanted to try WDialog made by Gerd Stolpmann.

Gerd is one of the more productive Ocaml programmers, 
who generates good and documentated libraries.

On my linux-machine I use the CDK with ocaml-3.03ALPHA.

This is the installation history:
1. Needs Ocaml 3.03 -> updated CDK using CVS (there are no tarballs
but I have a broadband-connection).
2. Ocamlnet missing -> downloaded ocamlnet-0.92
3. Failed to install ocamlnet, since the PCRE in the CDK seems to
be to old.
4. Installed a new PCRE.
5. Tried to install ocamlnet again, however failed with 
	"unbound type constructor Mimestring.s_param"

   Probably something else is too old...I got bored...
   ...never got back to try to install WDialog....

---------

The above history is the reason I started using the CDK,
in many cases using libraries that depend on other
libraries is almost impossible without the CDK. Too much 
time is spent on downloading and compiling.

I know 3 solutions to the problem:
1. CDK
2. Gerd invented findlib to solve the above problem.
3. Adding good packages to the standard distribution,
   so that mostly packages doesn't depend on other
   packages, but on packages in the standard
   distribution. (But very few if any new packages
   are added to the distribution.)


There are probably more solutions. I would call this 
problem a show stopper, since it prevents user to use
good libraries. 

We need a solution. Maybe a CPAN-like solution? 
In the Ocaml-CPAN it could either be source code, or
compile binaries (I can live without native code, at
least when experimenting with other peoples libraries.)

/mattias


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners