Shared run-time DLLs for commerce

Shared run-time DLLs for commerce

[Jon Harrop]

I would like to commercialize packages like Smoke in compiled form as a shared 
run-time DLL for other OCaml programmers to use as a library. However, OCaml 
currently lacks this functionality.

Is there any chance this will be added in the future?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Shared run-time DLLs for commerce

[Alain Frisch]

Jon Harrop wrote:
> I would like to commercialize packages like Smoke in compiled form as a shared 
> run-time DLL for other OCaml programmers to use as a library. However, OCaml 
> currently lacks this functionality.

Which functionality do you need, exactly?

What's the advantage of shipping a shared library instead of a static
one (.cmxa/.a/*.cmi)?

The .cmxs files generated by natdynlink are actually dynamic libraries.
Other modules that rely on such a library will also need to be dynlinked.


-- Alain

Re: [Caml-list] Shared run-time DLLs for commerce

[Alain Frisch]

Jon Harrop wrote:
> We're currently distributing .cma files but the main problem is that they're 
> far too brittle

Ok, your concerns are not really about being able to distribute shared
libraries, but rather being able to distribute binary libraries that
don't depend on precise version of other dependent libraries and of
OCaml. I don't think you'll get them any time soon.

Suggestions:

1. Distribute the source code, even without an open source license. I
cannot imagine this would reduce your sales, but you know better.

2. Distribute a fully packaged OCaml distribution that includes all the
dependent libraries (your users will be free to add third party libs as
well).

3. Obfuscate the parts of the source code you want to keep secret.
Camlp4 might help here.

4. Distribute a static binary that plays the role of a server that
encapsulates all your precious trade secrets + a client library
distributed in source code.

> Ah, I hadn't noticed these .cmxs files. They look good but I assume they are 
> also brittle?

Yes, indeed.

-- Alain

Re: [Caml-list] Shared run-time DLLs for commerce

[Kuba Ober]

> 1. Distribute the source code, even without an open source license. I
> cannot imagine this would reduce your sales, but you know better.

Trolltech (of Qt fame) have been doing that since day 1 and I'm pretty sure it 
has won them more business, and saved a lot of support headaches and angry 
customer calls.

Cheers, Kuba

Re: [Caml-list] Shared run-time DLLs for commerce

[Jon Harrop]

I'd like to add that OCaml has the potential to become a valuable platform for 
commercial software if these issues are addressed.

On Monday 07 January 2008 15:54:45 Alain Frisch wrote:
> Jon Harrop wrote:
> > We're currently distributing .cma files but the main problem is that
> > they're far too brittle
>
> Ok, your concerns are not really about being able to distribute shared
> libraries, but rather being able to distribute binary libraries that
> don't depend on precise version of other dependent libraries and of
> OCaml. I don't think you'll get them any time soon.
>
> Suggestions:
>
> 1. Distribute the source code, even without an open source license. I
> cannot imagine this would reduce your sales, but you know better.

Yes. The concern here is not loss of sales but loss of competitive edge. With 
direct access to a comprehensible implementation of the complex algorithms 
inside the software, people will nick the algorithms even if they don't nick 
the source code. After 9 years of work, I'd rather not see that happen... :-)

> 2. Distribute a fully packaged OCaml distribution that includes all the
> dependent libraries (your users will be free to add third party libs as
> well).

Even with versions up for OCaml 3.09.1, 3.09.2 and 3.10.0 we're still getting 
requests for 3.09.3 which I didn't even know existed. With the number of 
requests we get for specific compiler versions, I believe customers would be 
unwilling to install a specific OCaml version just for our software.

In other words, we'd be slicing an already-small OCaml market into the tiny 
OCaml version 3.x.y market. I believe the whole OCaml market is just about 
commercially viable but that would not be.

> 3. Obfuscate the parts of the source code you want to keep secret.
> Camlp4 might help here.

This is a possibility but there is little scope for obfuscation within OCaml, 
AFAIK. I really want to distribute after pattern match compilation, for 
example.

I was thinking that externalizing an interface to the intermediate 
representation might be another solution?

> 4. Distribute a static binary that plays the role of a server that
> encapsulates all your precious trade secrets + a client library
> distributed in source code.

In many cases a separate server would work well. However, this is an 
OpenGL-based application so I think this would require one process to be able 
to render efficiently into an OpenGL context from another process. I'm not 
sure how or if that can work or whether or not it would be portable.

> > Ah, I hadn't noticed these .cmxs files. They look good but I assume they
> > are also brittle?
>
> Yes, indeed.

I'd be very happy if the brittle digest comparison were replaced with a 
structural API comparison instead. That would save me a lot of time and 
effort.

One final question: is it possible to throw money at the right people to get 
this kind of work done?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Shared run-time DLLs for commerce

[Stefano Zacchiroli]

On Mon, Jan 07, 2008 at 07:51:23PM +0000, Jon Harrop wrote:
> I'd be very happy if the brittle digest comparison were replaced with a 
> structural API comparison instead. That would save me a lot of time and 
> effort.

FWIW it would also save a lot of time and effort for maintainers of
OCaml-related software in (binary-based) GNU/Linux distributions, as
Debian and Red Hat.

+1 on this feature request.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Shared run-time DLLs for commerce

[Richard Jones]

On Mon, Jan 07, 2008 at 07:51:23PM +0000, Jon Harrop wrote:
> On Monday 07 January 2008 15:54:45 Alain Frisch wrote:
> > 1. Distribute the source code, even without an open source license. I
> > cannot imagine this would reduce your sales, but you know better.
> 
> Yes. The concern here is not loss of sales but loss of competitive edge. With 
> direct access to a comprehensible implementation of the complex algorithms 
> inside the software, people will nick the algorithms even if they don't nick 
> the source code. After 9 years of work, I'd rather not see that happen... :-)

Really?  Do you think that the generated bytecode is obscure enough to
protect this 'valuable' knowledge too?  If I was that concerned about
my cherished algorithms, I wouldn't let them go beyond a web service.

> > 3. Obfuscate the parts of the source code you want to keep secret.
> > Camlp4 might help here.
> 
> This is a possibility but there is little scope for obfuscation within OCaml, 
> AFAIK. I really want to distribute after pattern match compilation, for 
> example.

Check out this recent discussion on ocaml_beginners:

http://tech.groups.yahoo.com/group/ocaml_beginners/message/9163

> I'd be very happy if the brittle digest comparison were replaced with a 
> structural API comparison instead. That would save me a lot of time and 
> effort.

This is certainly true.  The single, brittle digest is a real problem
for Red Hat.  It should be at least possible to add additional values
and types without that causing incompatibility.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Shared run-time DLLs for commerce

[Jon Harrop]

On Tuesday 08 January 2008 16:03:51 Richard Jones wrote:
> On Mon, Jan 07, 2008 at 07:51:23PM +0000, Jon Harrop wrote:
> > On Monday 07 January 2008 15:54:45 Alain Frisch wrote:
> > > 1. Distribute the source code, even without an open source license. I
> > > cannot imagine this would reduce your sales, but you know better.
> >
> > Yes. The concern here is not loss of sales but loss of competitive edge.
> > With direct access to a comprehensible implementation of the complex
> > algorithms inside the software, people will nick the algorithms even if
> > they don't nick the source code. After 9 years of work, I'd rather not
> > see that happen... :-)
>
> Really?  Do you think that the generated bytecode is obscure enough to
> protect this 'valuable' knowledge too?

I value the little protection that it provides.

> If I was that concerned about 
> my cherished algorithms, I wouldn't let them go beyond a web service.

For implementations that can be hidden in that way, it would be fine, but this 
is high-performance OpenGL-based visualization so a web service is a no-go.

> > > 3. Obfuscate the parts of the source code you want to keep secret.
> > > Camlp4 might help here.
> >
> > This is a possibility but there is little scope for obfuscation within
> > OCaml, AFAIK. I really want to distribute after pattern match
> > compilation, for example.
>
> Check out this recent discussion on ocaml_beginners:
>
> http://tech.groups.yahoo.com/group/ocaml_beginners/message/9163

Actually that is an excellent example of source code obfuscation being 
inadequate in OCaml because you can reverse engineer the code into a 
developable form using only camlp4:

$ camlp4oof bf.ml
let (i, f) = (input_char, (open_in Sys.argv.(1)))
let rec p (c, b, k) =
  try
    p
      (match i f with
       | '+' -> ((c + 1), b, k)
       | '-' -> ((c - 1), b, k)
       | '<' ->
           (match b with
            | ([], r) -> (0, ([], (c :: r)), k)
            | (h :: l, r) -> (h, (l, (c :: r)), k))
       | '>' ->
           (match b with
            | (l, []) -> (0, ((c :: l), []), k)
            | (l, h :: r) -> (h, ((c :: l), r), k))
       | '[' ->
           if c = 0
           then (while (i f) <> ']' do () done; (c, b, k))
           else (c, b, (((pos_in f) - 1) :: k))
       | ']' ->
           (match k with
            | [] -> failwith "]"
            | h :: t -> (seek_in f h; (c, b, t)))
       | ',' -> ((input_byte stdin), b, k)
       | '.' -> (output_byte stdout c; (c, b, k))
       | _ -> (c, b, k))
  with | _ -> ()
let _ = p (0, ([], []), [])

On the other hand, reverse engineering non-trivial pattern matches after 
compilation would require substantially more work for someone with no 
relevant expertise.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Shared run-time DLLs for commerce

[Christophe TROESTLER]

On Mon, 7 Jan 2008 22:17:31 +0100, Stefano Zacchiroli wrote:
> 
> FWIW it would also save a lot of time and effort for maintainers of
> OCaml-related software in (binary-based) GNU/Linux distributions, as
> Debian and Red Hat.
>
On Tue, 8 Jan 2008 16:03:51 +0000, Richard Jones wrote:
> 
> This is certainly true.  The single, brittle digest is a real problem
> for Red Hat.  It should be at least possible to add additional values
> and types without that causing incompatibility.

Why don't you guys discuss exactly what kind of robustness you need
and introduce a feature wish in the bug tracker?

My 0.02€,
ChriS

On Tue, 8 Jan 2008 16:03:51 +0000, Richard Jones wrote:
> 
> This is certainly true.  The single, brittle digest is a real problem
> for Red Hat.  It should be at least possible to add additional values
> and types without that causing incompatibility.

Re: [Caml-list] Shared run-time DLLs for commerce

[Jon Harrop]

On Tuesday 08 January 2008 19:39:04 Christophe TROESTLER wrote:
> On Mon, 7 Jan 2008 22:17:31 +0100, Stefano Zacchiroli wrote:
> > FWIW it would also save a lot of time and effort for maintainers of
> > OCaml-related software in (binary-based) GNU/Linux distributions, as
> > Debian and Red Hat.
>
> On Tue, 8 Jan 2008 16:03:51 +0000, Richard Jones wrote:
> > This is certainly true.  The single, brittle digest is a real problem
> > for Red Hat.  It should be at least possible to add additional values
> > and types without that causing incompatibility.
>
> Why don't you guys discuss exactly what kind of robustness you need
> and introduce a feature wish in the bug tracker?

Of course, the package maintainers could always simply remove the digest test 
from their own OCaml package and hope for the best. ;-)

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Shared run-time DLLs for commerce

[Richard Jones]

On Tue, Jan 08, 2008 at 07:42:21PM +0000, Jon Harrop wrote:
> On Tuesday 08 January 2008 19:39:04 Christophe TROESTLER wrote:
> > On Mon, 7 Jan 2008 22:17:31 +0100, Stefano Zacchiroli wrote:
> > > FWIW it would also save a lot of time and effort for maintainers of
> > > OCaml-related software in (binary-based) GNU/Linux distributions, as
> > > Debian and Red Hat.
> >
> > On Tue, 8 Jan 2008 16:03:51 +0000, Richard Jones wrote:
> > > This is certainly true.  The single, brittle digest is a real problem
> > > for Red Hat.  It should be at least possible to add additional values
> > > and types without that causing incompatibility.
> >
> > Why don't you guys discuss exactly what kind of robustness you need
> > and introduce a feature wish in the bug tracker?
>
> Of course, the package maintainers could always simply remove the
> digest test from their own OCaml package and hope for the best. ;-)

The problem, though, is that you _can't_ remove that test from the
compiler[1].

OCaml modules in Fedora have deps like this:

$ rpm -q --requires ocaml-libvirt
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(VersionedDependencies) <= 3.0.3-1
libc.so.6()(64bit)  
libc.so.6(GLIBC_2.2.5)(64bit)  
libvirt.so.0()(64bit)  
ocaml(Buffer) = f6cef633ea14963b84b79c4095c63dc3
ocaml(Callback) = e5ca1fb5990fac2b7b17cbb1712cffe2
ocaml(Char) = e98bc9c9e918a84b3c1a5a122d42fac1
ocaml(Pervasives) = 8ba3d1faa24d659525c9025f41fd0c57
ocaml(String) = 2c162ab314b2f0a2cfd22d471b2e21ab
ocaml(runtime) = 3.10.0

These deps do ensure that you can't install the wrong RPM without
forcing it, but don't allow us any sort of backwards compatibility.
When Ocaml 3.10.1 comes out, we'll need to recompile everything.  More
critically our customers will have to recompile all of their code too.

A single critical library somewhere can have the same effect.

OCaml in RHEL/EPEL has to stick with the same libraries, even if they
have bugs, even if they have _security_ bugs (probably, but not hit
that one yet thankfully).

This is quite different from the way that RHEL provides guaranteed
binary compatibility for 7 years, with continual upgrades and
backporting along the way.

Rich.

[1] Well, OK I guess we could patch it out, but that'd be a really
stupid idea.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Shared run-time DLLs for commerce

[Alain Frisch]

Jon Harrop wrote:
> Yes. The concern here is not loss of sales but loss of competitive edge. With 
> direct access to a comprehensible implementation of the complex algorithms 
> inside the software, people will nick the algorithms even if they don't nick 
> the source code. After 9 years of work, I'd rather not see that happen... :-)

If your target audience is people who already use OCaml (and who come 
with request for support for specific versions of OCaml), I doubt you'll 
get direct competition on this niche, so it should be safe to ship the 
source code. If you target the widest audience of people looking for 
good visualization solutions, you might want to drop the tiny fraction 
of those who insist on using an old version of OCaml. Anyway, that's 
none of my business.

> I was thinking that externalizing an interface to the intermediate 
> representation might be another solution?

The first intermediate language where pattern matching is already 
lowered (the lambda code) already has strong dependencies to specific 
interfaces (e.g. it makes to references fields of external modules by 
position).  Moreover, there is no guarantee that the internal 
representation of this lambda remains compatible across versions of OCaml.

> In many cases a separate server would work well. However, this is an 
> OpenGL-based application so I think this would require one process to be able 
> to render efficiently into an OpenGL context from another process. I'm not 
> sure how or if that can work or whether or not it would be portable.

Cannot you put the low-level OpenGL part in the main process (shipped 
with source code) and the high valued algorithms in the server?

> I'd be very happy if the brittle digest comparison were replaced with a 
> structural API comparison instead. That would save me a lot of time and 
> effort.

Compiled code depends on specific locations for elements coming from 
external modules. Basically, if a version of OCaml adds a new field 
around the top of Pervasives, the code compiled against the old version 
becomes invalid.

What you can do is to ship your library as a single module abstracted 
over all its dependencies (that is, a functor which takes as arguments 
modules from the stdlib and other dependent libraries). You can put in 
the interface whatever you expect from these modules, so the same 
functor could be used with future versions if they only add new fields
(but not new constructors or record fields, for instance). Again, some 
Camlp4 hack could somewhat automate this.

The module still depends on a specific version of OCaml, but now, it 
could make sense to add to the compiler features to read and write
version-independent representations of .cmi and lambda-code files.
(But honestly, I don't believe this will happen in the official OCaml.)


I've a final suggestion:

- automate the compilation of your library for several versions of OCaml 
and of dependent libraries; it's not as if there were 200 versions of 
each to support, 5-6 should be enough, right?  A single command could be 
enough to compile and package your library for all the possible 
combinations. If you write a generic and robust tool for this task, you 
might even be able to sell it.  Now, your customers would get a small 
installer that checks their installed versions and download the relevant 
version of your library.


-- Alain