[Caml-list] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Caml-list] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Jordan W]

I'm cross posting my post from the opam-devel list:

Since there is a lot of interest in OCaml from web frontend communities, I
thought it would be useful to imagine what the ideal development flow for
this audience would look like. I know there's a ton of progress being made
on documentation, and build systems, but I thought I would explore the
problem from the perspective of a frontend developer, which means starting
with the tooling that they are familiar with. One common tool is
`CommonJS/package.json`, which is a way to model and organize dependencies
using a single JSON file per package. The npm command line tools allow you
to install files to disk based purely on these package.json files.

I created a proof of concept called CommonML, which lets developers use
their familiar CommonJS workflow with OCaml:
https://github.com/jordwalke/CommonML

I also took used it as an opportunity to explore what can be done when
there are opinionated conventions in place. If you have a predictable
project structure, how can that benefit us? In this case, I created an
automatic docs builder (with nice styling) and also automatically generate
IDE autocomplete support for all your dependencies (and your project's
internal modules).


I hope there is at least something we can take away from it that helps
inform the design of OPAM and related tools.


One nice aspect is that with `CommonJS`, there needn't be an authoritative
package service. Your package.json file can point to arbitrary git URLs if
you like.  (Note: The npm command line tool is *not* the npm package
service - they are made by the same organization but one may be used
without the other). However, this prototype I've built does allow you to
host OCaml code on npm and depend on it.

By far the nicest thing about developing with `CommonJS` is that you don't
have to think about module namespace collisions. There is Just One Way to
namespace modules/packages. This prototype automatically sets up a similar
namespacing convention for OCaml modules. It's not flexible, and you can't
customize it, but it always works. It uses module aliases (thank you to Leo
White for helping me come up with the build conventions).

Another thing I like about the `CommonJS` workflow is that developing
packages locally is virtually the same as developing against remote
dependencies. (`npm link` is much like `opam pin` I'm told). When you `npm
install` dependencies, everything is pulled down into a local
sandbox(node_modules directory) instead of being installed globally by
default. If you want to see what versions your local package is seeing,
just traverse the file system! If you want to reinstall, just delete the
node_modules directory and then `npm install` again. I believe there is a
way to get it to use a global package cache so the node_modules might
contain symlinks to those shared packages - but that's just an
optimization. There isn't any notion of building in `npm`, so there
wouldn't be a build cache I believe.

In my quick prototype, every dependency must be compiled at least once for
the root level project that you are building. This ends up being nice in
cases where the build flags (such as -g) must be in effect for the
compilation of all my dependencies - relying on the build flags that you
*installed* the package with will bite you. But of course, the rebuilding
approach can end up being super slow. Still, the incremental build times
are *totally* reasonable since it does try to do some basic incremental
compilation. I would have used ocamlbuild which probably does a much better
job, but I needed to write my own totally custom operations in order to get
the auto-namespacing (with the help of Leo White). I wasn't sure how to do
that with ocamlbuild, but if I could, I imagine the incremental compilation
times would be way better.

Either way, for most of the work I do (developing libraries with many other
small libraries as dependencies) - I could see a development flow like this
being a worthwhile goal, especially if it makes OCaml much more comfortable
for a *huge* set of developers. `CommonJS` is likely becoming the most
popular development flow. It's just a hacky proof of concept, but it was
fun.

Re: [Caml-list] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Andrew Herron]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

I've been thinking about this a bit, as a JS developer who is experimenting
with OCaml (in my non-existant free time). I don't use the CommonJS flow in
my day job although we do have many projects which our internal
build/dependency system combines with similar results.


On Wed, Feb 25, 2015 at 6:26 AM, Jordan W <jordojw@gmail.com> wrote:

>
> I created a proof of concept called CommonML, which lets developers use
> their familiar CommonJS workflow with OCaml:
> https://github.com/jordwalke/CommonML


My initial thoughts were to stick with the OPAM repositories; even if I
have to make a local OPAM repo server it seems like leveraging the existing
ecosystem is a good idea. My experiments use a dedicated `opam switch` for
the project, enforced by the Makefile (which might not scale to multiple
projects, but I haven't tried that yet).

I can think of one good reason to share OCaml and JS dependencies on the
same server though, and that's including a JS library in a js_of_ocaml
project. Certainly using a single package.json file to specify both JS and
OCaml dependencies is an interesting idea (even if under the hood we
eventually find a way to use OPAM for the OCaml projects).

Cheers,
Andy

[-- Attachment #2: Type: text/html, Size: 1703 bytes --]

Re: [Caml-list] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Jordan W]

There's certainly a lot of ideas/experiments in CommonML all rolled
into one. The overarching goal was to just explore the ideal developer
flow for people who are either used to CommonJS or who appreciate a
highly sandboxed development model without globals. There's some
"pointless" goodies thrown in like zero effort merlin support and
automatic doc generation (which is now being solved in a *much* better
way by other people more knowledgeable than myself).

CommonML is just a hacky script, but I find it useful to build
developer tools by starting with the ideal end developer experience
and performing whatever atrocious hacks I have to in order to create
it. After I get feedback, I can explore doing it the right way.
Obviously OPAM is the real deal, and I hope some people can help me
achieve much of this with OPAM (as it is, or in future versions).

But the most valuable part of the CommonML experiment, actually has
nothing to do with package managers/sandboxes at all - it's the
automatic namespacing of dependencies (using module aliases) and
automatic generation of merlin completion files. You literally only
have to list a new dependency in your `package.json`, then it will
resolve all naming conflicts, build correctly, and allow you to
immediately start autocompleting using merlin. I suspect such a
seamless experience is only feasible when there are opinionated
conventions in place. I feel like most people would be satisfied with
the ones explored in CommonML:
- namespaced dependencies, non-namespaced internal modules
(ModuleInsideMyPackage.x, YourPackage.ModuleInsideYourPackage.y) -
powered by module aliases.
- "exports" field in `package.json` that determines which modules
should be publicly available to other packages that depend on you
(nothing is visible to other packages by default).

I don't know enough about the OCaml build toolchain to solve this in a
scalable way so I made it work generally enough for a couple of the
projects that I'm prototyping. I hope the same can be achieved in some
other more sophisticated build tool. Has anyone seen anything like the
automatic namespacing I've prototyped here? I couldn't imagine
developing without it, personally.


On Tue, Feb 24, 2015 at 7:12 PM, Andrew Herron <thespyder@programmer.net> wrote:
> I've been thinking about this a bit, as a JS developer who is experimenting
> with OCaml (in my non-existant free time). I don't use the CommonJS flow in
> my day job although we do have many projects which our internal
> build/dependency system combines with similar results.
>
>
> On Wed, Feb 25, 2015 at 6:26 AM, Jordan W <jordojw@gmail.com> wrote:
>>
>>
>> I created a proof of concept called CommonML, which lets developers use
>> their familiar CommonJS workflow with OCaml:
>> https://github.com/jordwalke/CommonML
>
>
> My initial thoughts were to stick with the OPAM repositories; even if I have
> to make a local OPAM repo server it seems like leveraging the existing
> ecosystem is a good idea. My experiments use a dedicated `opam switch` for
> the project, enforced by the Makefile (which might not scale to multiple
> projects, but I haven't tried that yet).
>
> I can think of one good reason to share OCaml and JS dependencies on the
> same server though, and that's including a JS library in a js_of_ocaml
> project. Certainly using a single package.json file to specify both JS and
> OCaml dependencies is an interesting idea (even if under the hood we
> eventually find a way to use OPAM for the OCaml projects).
>
> Cheers,
> Andy

Re: [Caml-list] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Andrew Herron]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

On Wed, Feb 25, 2015 at 3:58 PM, Jordan W <jordojw@gmail.com> wrote:

> The overarching goal was to just explore the ideal developer
> flow for people who are either used to CommonJS or who appreciate a
> highly sandboxed development model without globals.


This is something I'm keen to see as well - extend the opam switch concept
to per project. Forcing a switch during build is just a poor imitation of
that.


I don't know enough about the OCaml build toolchain to solve this in a
> scalable way so I made it work generally enough for a couple of the
> projects that I'm prototyping. I hope the same can be achieved in some
> other more sophisticated build tool. Has anyone seen anything like the
> automatic namespacing I've prototyped here? I couldn't imagine
> developing without it, personally.


+1

I'll end up developing something custom (or templates like we use in JS) if
need be to suit our development style - which sounds similar to yours - a
bunch of small internal library projects all composed in a tree of
dependencies, culminating in a single high level integration / package
project for the application.

Cheers,
Andy

[-- Attachment #2: Type: text/html, Size: 1726 bytes --]