OCaml Weekly News

I'm happy to announce that liquidsoap 1.4.3 is out at: https://github.com/savonet/liquidsoap/releases/tag/v1.4.3

This is the 3rd bugfix release for the 1.4.x branch. It contains important fixes and a couple of new minor features. Update is recommended and should be fairly safe.

Along we this release, we have now added builds for arm64 debian packages and docker-ready production images for amd64 and arm64 architectures available at: https://hub.docker.com/repository/docker/savonet/liquidsoap

Again, we would like to warmly thank all users, contributors and reporters for helping us bring liquidsoap to the next step!

Also, please note that a couple of issues had to be left out to make sure that the release comes out on time. These are listed here and will be tackled as soon as possible.

Next for liquidsoap, we will focus on getting the current 2.x branch finalized and polished. We already have support for encoded content and ffmpeg raw frames. We need to write a couple of inline encoders and decoders and we'll have 90% of the features ready. This will be a major update for us!

This is to announce Simple63, an opam package for compression of integer sequences; similar to Anh and Moffat's Simple-8b. More details found in:

• github: https://github.com/barko/simple63
• documentation: https://barko.github.io/simple63/

Feedback and bug reports welcome.

This is to announce bentov, a opam package that implements a 1D histogram-sketching algorithm. For more details:

• github: https://github.com/barko/bentov
• documentation: https://barko.github.io/bentov

Feedback and bug reports welcome.

On behalf of the opam maintainers, I'd like to announce the first release of opam-compiler, a plugin to work with compiler variants, branches and forks.

This can cover a pretty wide range of use cases, so the first version is starting small with a single command to create a switch from a branch or github PR:

% opam compiler create '#9921'
Opam plugin "compiler" is not installed. Install it on the current switch? [Y/n] y

...

<><> Carrying on to "opam compiler create #9921" ><><><><><><><><><><><><><><><>

[ocaml-variants.4.12.0+trunk+no-flat-float-array] synchronised from
git+https://github.com/gasche/ocaml#Atomic.create
ocaml-variants is now pinned to git+https://github.com/gasche/ocaml#Atomic.create (version
4.12.0+trunk)
% opam switch
...
→  ocaml-ocaml-9921
          [opam-compiler] ocaml/ocaml#9921 - stdlib: rename Atomic.make into Atomic.create

You can also override the arguments passed to --configure.

As you can see in the above snippet, it's an opam plugin so it will auto-install if needed (assuming you ran opam update recently) and will be available across all switches. Its sources and issue tracker are available here.

For the next releases, our plan is to add a user-friendly way to setup a switch based on a local git clone, so that it's easy to test your compiler fork with opam packages. You can find the other features we'd like to add in the future in the relevant part of the opam roadmap.

Thanks and have fun compiling compilers!

I've release lua_parser which is now up on opam. It is a parser and pretty-printer for lua 5.2. Actually it was developed with luajit in mind which is lua 5.1 plus goto/labels (which syntactically for the purposes of parsing and pretty-printing is lua 5.2).

I am glad to announce, on behalf of the Merlin team, the release of Merlin 3.4.0 which brings some major changes in the way configuration is handled.

As you might know, Merlin reads its configuration from the closest .merlin file to the source file being edited. These files tell merlin where to find other source files and build artifacts, but also which flags should be passed to the compiler, which syntax extensions are enabled and which packages are used by the project.

In this setting the configuration is the same for all the source files of a folder, regardless of their specificities. In other words, the configuration loaded for a single source file contains the union of the dependencies of this file and of all its siblings which is not an optimal behavior.

Starting with version 3.4.0 merlin will ship with two packages: merlin and dot-merlin-reader which, as the name suggests, reads configuration from .merlin files. Both are necessary for proper function.

When a .merlin file is present in the source folder the Merlin server will start a dot-merlin-reader process and communicate with it via standard input and output following a simple protocol. These processes are halted with the server.

This change should not have any visible impact on users' workflows as long as the dot-merlin-reader binary is correctly installed and in the path. (which should be the case in opam-based setups)

This change in itself will not solve the granularity problem mentioned earlier, but it paves the way for such improvements: in a near-future Dune will stop generating .merlin files and Merlin will obtain file-based configuration directly from the build system using the same protocol as the one used by dot-merlin-reader.

TL;DR https://github.com/blandinw/ocaml-grpc-envoy/

Hey, I'm new to OCaml after writing some Clojure, C++ and Haskell in various contexts, including working at FB (relevant below).

After browsing this forum and Reddit for a bit, the assumption seems to be that OCaml is not a good fit for gRPC, since there's no pure implementation today. Now, this is something I have experience with, so I thought I'd try and challenge this assumption.

As you may know, services inside FB use Thrift (both the format and protocol) to communicate. The Thrift team worked primarily in C++ (for good reasons), causing support for other languages to lag behind despite their best efforts. Now, the interchange format (equivalent to Protobuf) does not change very often so it's fine to have a per-language implementation, but the client and server (equivalent to HTTP2 + gRPC) frequently receive new features, optimizations and fixes. After a valiant and continued effort to support most languages used internally, the Thrift team came up with an idea. Instead of maintaining multiple implementations and dealing with obscure FFI bugs, ~FingerprintTrustManagerFactory~s and whatnot, they would focus solely on the C++ implementation and provide a daemon to be ran alongside whatever code you were trying to run. You could then use simple IPC to exchange Thrift (the format) messages with that daemon, and it would handle all the nitty-gritty of running a service at scale (load balancing, connection pooling, service discovery, security, retries, timeouts, network stats, hot restarts, etc.). Needless to say, it worked remarkably well even at very high scale and everybody was much happier.

I wanted to replicate this idea with OCaml and gRPC. We already have support for protobuf thanks to the excellent ocaml-protoc. All we need is a way to exchange protobuf messages reliably on the wire. Instead of having an OCaml implementation that will have to stay up-to-date and have its own set of bugs (the official grpc/grpc-java repo has 4450 commits and 2400 issues at the moment), can we reuse existing infra with already massive support and production time?

Fortunately, the people at Lyft built just that, open-sourced it and contributed it to the Cloud Native Computing Foundation in late 2017. It is called Envoy and it is bliss.

I demonstrate how to fit these pieces together at blandinw/ocaml-grpc-envoy to build a simple KV store, including a gRPC client and server in 200 lines of OCaml code. The idea is to spawn an Envoy process that will handle all gRPC communication for our OCaml code. We use HTTP/1.1 to exchange Protobuf messages with it, using for example httpaf and Lwt. This solution has the added benefit that it is highly scalable from the start, allowing you for instance to spawn one OCaml process per core and load balance between them. You can also use Envoy (with proper config!) as your web reverse proxy instead of say, nginx.

At the very least, this solution allows us to start writing gRPC code today, and gracefully evolve towards HTTP/2, Multicore and maybe a native OCaml implementation later.

I'm curious to hear your perspective on the future of building services with OCaml, or your past experience like what went well, what was missing, etc.

Fantastic idea. So if I understand correctly, the only thing that Envoy (server-side) is doing is translating the Protobuf from gRPC HTTP2 transport to HTTP1, and forwarding these Protobuf objects over HTTP1 to the OCaml server? Envoy doesn't know to know about the actual gRPC schema, because it doesn't touch the Protobuf objects themselves, right?

That's correct. Envoy is only concerned with transporting bytes (along with load balancing, routing, etc, etc). Only OCaml knows about the Protobuf schemas.

In the OCaml server case, Envoy listens for HTTP/2 gRPC requests, accesses the bytes payload with no knowledge of the actual schema/layout and repackages these same bytes in a HTTP/1.1 request that OCaml can process. OCaml then responds with bytes (an encoded Protobuf response message) that Envoy sends back on the original HTTP2 connection.

The project is hosted here for those who are interested.There's also some excellent docs