OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of May 12 to 19, 2020.

Table of Contents

ocamlformat 0.14.2

Guillaume Petiot announced

We are pleased to announce the release of ocamlformat 0.14.2. This minor release improves the recent 0.14.0 and 0.14.1 releases regarding the doc-comments option.

How to migrate from 0.13.0

Here are the changes of the doc-comments options compared to ocamlformat 0.13.0:

  • after has been renamed to after-when-possible to take into account the technical limitations of ocamlformat;
  • a new value before-except-val has been added, placing doc-comments before the corresponding code, but placing doc-comments of val and external declarations after the corresponding declaration;
  • before is unchanged.

Here is the full list of changes made by the 0.14.0 release: https://discuss.ocaml.org/t/ann-ocamlformat-0-14-0/5435

How to migrate from 0.14.0

The 0.14.0 release lead to some regression of the doc-comments behavior that (although intended for us) lead to some surprise from a lot of users. The behavior of doc-comments has thus been reverted to it's 0.13.0 state with the following changes:

The doc-comments-val option has been removed and merged with doc-comments. The placement of documentation comments on val and external items is now controlled by doc-comments .

  • doc-comments=after becomes doc-comments=after-when-possible to take into account the technical limitations of ocamlformat;
  • doc-comments=before is unchanged;
  • doc-comments-val is now replaced with doc-comments

To reproduce the former behaviors

  • doc-comments=before + doc-comments-val=before : now use doc-comments=before ;
  • doc-comments=before + doc-comments-val=after : now use doc-comments=before-except-val ;
  • doc-comments=after + doc-comments-val=before : this behavior did not make much sense and is not available anymore;
  • doc-comments=after + doc-comments-val=after : now use doc-comments=after-when-possible.

How to migrate from 0.14.1

The 0.14.1 release was preserving the behavior of 0.13.0 regarding doc-comments, it added a unset value to the doc-comments-val option. This option has been removed with the following changes:

The doc-comments-val option has been removed and merged with doc-comments. The placement of documentation comments on val and external items is now controlled by doc-comments .

  • doc-comments=after becomes doc-comments=after-when-possible to take into account the technical limitations of ocamlformat;
  • doc-comments=before is unchanged;
  • doc-comments-val is now replaced with doc-comments

To reproduce the former behaviors

  • doc-comments=before + doc-comments-val=before : now use doc-comments=before ;
  • doc-comments=before + doc-comments-val=after : now use doc-comments=before-except-val ;
  • doc-comments=after + doc-comments-val=before : this behavior did not make much sense and is not available anymore;
  • doc-comments=after + doc-comments-val=after : now use doc-comments=after-when-possible.

Thank you

We would like to thank our early users to help us on the road of a stable 1.0.0 release of ocamlformat.

ML Family Workshop 2020: Call for presentations

Leo White announced

ICFP, and by extension the ML workshop, will be now officially be held online with a significantly reduced fee. Due to the change in official status we decided to extend the submission deadline to the end of May.

Important Dates (updated)

  • Friday 29th May (any time zone): Abstract submission deadline
  • Friday 17th July: Author notification
  • Thursday 27th August: ML Family Workshop

memprof-limits preview (and a guide to handle asynchronous exceptions)

Guillaume Munch-Maccagnoni announced

Dear OCamlers, I am happy to pre-announce memprof-limits, an implementation of per-thread global memory limits, and per-thread allocation limits à la Haskell, compatible with systhreads.

Memprof-limits interrupts the execution by raising an asynchronous exception, an exception that can arise at almost any location in the code. I also announce a guide on how to recover from asynchronous exceptions and other unexpected exceptions that you find in the documentation. It summarises knowledge acquired in OCaml by the Coq proof assistant as well as in other programming languages. To my knowledge, this has never been told in OCaml textbooks, so I thought it might be of general interest to you. This research is part of a wider work aiming to regulate the use of asynchronous exceptions in OCaml in coordination with multicore language designers.

Global memory limits let you bound the memory consumption inside specific parts of your program, in terms of memory used by the whole program. It is inspired by this other post, but in a form readily available for use with systhreads.

Allocation limits let you bound the execution of parts of the program measured in number of allocations, analogous to the same feature in Haskell advocated in a nice post by Simon Marlow. Allocation limits count allocations but not deallocations, and is therefore a measure of the work done, which can be more suitable than execution time.

Memprof-limits, as the name tells, uses the upcoming Memprof engine from OCaml 4.11, with a low sampling rate that does not affect performance. A reimplementation of the Memprof interface compatible with memprof-limits running at the same time is provided for profiling needs.

Memprof-limits is available on the public opam repository, but depends on OCaml 4.11 which at the moment is available from the beta opam repository only. It is experimental for reasons explained in the manual.

FAQ

  • “Is it wise to rely on the statistical nature of Memprof? If I set an allocation limit of 100 KB, and run a function that allocates exactly 50 KB, then the function might fail, due to the random nature of Memprof.”

    Memprof-limits is provided with a statistical analysis meant to help you chose appropriate values for the limit depending on a target safe allocation value. (Nice pictures omitted because this discuss does not support svg.)

    Long story short, memprof-limits starts being accurate-enough starting around a safe allocation value of 100 KB with the default sampling rate (meaning a limit of 1 to 3 MB depending on chosen precision), with the ratio between the maximal safe allocation and the limit dropping very quickly for higher values. Correctly, the analysis shows that limits under 500 KB are unreliable.

    I have found that the statistical nature of Memprof makes it very easy to reason about its application and not have to factor in runtime implementation details. In addition, Memprof is nevertheless deterministic, which is (essential and) useful for reproducing runs in test scenarios.

  • “But can we really program with memprof-limits, that is, not only write programs but also reason about them, given the probabilistic nature of the guarantees?”

    Yes, if we make two additional hypotheses:

    1. Allocation limits (as used in Haskell) are used by determining peak reasonable allocation usage empirically and picking a limit at a comfortable margin over it, rather than computing a precise memory bound to be used as a limit. In very controlled environments where the latter would be possible, there probably would be better solutions, and the language this is inspired from makes it very hard to make predictions on memory use.
    2. The programmer is fine with a very unlikely possibility of a false positive; indeed the program is already designed to let true positives fail without bringing down mission-critical parts of the program. For instance they can prefer to see a legitimate client having a connexion closed once every 10ⁿ year for n of their choosing, if that is the price to pay for avoiding being subject to DOS on maliciously-crafted requests.

    Under these hypotheses, the statistical limit is just as reliable as the precise limits à la Haskell.

  • “Is it possible to also implement local memory limits, to bound the memory consumption of a particular function?”

    Yes but read on.

    Yang & Mazières (2014) advocates in favour of an allocator-pays model of cost attribution, and note its similarity with memory profiling. In this model, it is possible for instance to process untrusted user input under some memory limit, before the result is distributed to the rest of the program.

    Implementing memory limits based on the allocator-pays model, by adapting allocation limits to take into account deallocations, would be very easy thanks to the facilities provided by Memprof. Moreover, the statistical analysis of allocation limits can be transposed, and guarantees similarly accuracy at a low runtime cost for limits greater than 100KB.

    There is one surprising difficulty, though, which has to do with the way the GC works. The GC has a space overhead: memory that is wasted because unreachable values are not collected immediately. This overhead has to be taken into account when choosing the limit. However, this overhead is non-local and dependent on the total major heap size: one cannot just say “take the double of the desired limit”. Indeed, active threads will pay for memory that has been allocated in the past and kept alive. More experimentation is needed to provide guidance on how to take the space overhead into account.

  • “Can this be used to bound the consumption of lightweight threads in Lwt and Async?”

    It is straightforward to make memprof-limits parametric in the notion of thread id used to track per-thread limits. However, to the best of my knowledge, Lwt and Async are not meant to play well when the computation is interrupted by asynchronous exceptions. If you have more information about this limitation or are interested in experimenting, please get in touch.

Thanks

Thank you to Jacques-Henri Jourdan for his explanations about Memprof and Stephen Dolan for his feedback.

Tezos 7.0 is now available on opam

Pierre Boutillier announced

Tezos executables and libraries have just been released on opam. You can thus build them from source with a simple opam install tezos and build your own projects upon them.

What is Tezos

Tezos is a distributed consensus platform with meta-consensus capability. Tezos not only comes to consensus about the state of its ledger, like Bitcoin or Ethereum. It also comes to consensus about how the protocol and the nodes should adapt and upgrade. For more information about the project, see https://tezos.com.

Our implementation of Tezos is written in OCaml. It is split into several libraries (command-line interface tezos-clic, peer-to-peer library tezos-p2p, cryptographic primitives tezos-crypto~…) and executables (node ~tezos-node, client ~tezos-client~…).

Useful Links

Source code for this particular implementation can be found at https://gitlab.com/tezos/tezos/. Developer documentation is available at https://tezos.gitlab.io/. In particular, documentation for this specific release (version 7.0) is available at http://tezos.gitlab.io/releases/version-7.html.

Installation Instructions

Tezos (internal compiler in order to self amend itself) requires a specific version of the compiler (OCaml 4.09.1):

opam switch 4.09.1

Tezos also requires some external libraries:

opam depext tezos

Finally, to install all binaries:

opam install tezos

Replying to Nick Betteridge, Raphaël Proust said

Tezos has a soft-updating mechanism that works (roughly) as follows:

The network starts with a genesis protocol (“protocol” here means “economic protocol”: the rules according to which smart contracts are initiated and acted upon, transactions take place, etc.) in which a single public key is specified.

The genesis protocol has no notion of coin, currency, smart-contract, etc. Instead, the genesis protocol knows a single operation: a protocol injection.

The protocol injection for genesis requires the operation to be signed by the private key that matches the public key of the genesis block. And the protocol injection changes, irreversibly, the genesis protocol to a new protocol. This new protocol specifies what constitutes a valid block to add to the chain.

In the Tezos blockchain, the protocol injected on top of genesis included a notion of coins and an in-protocol voting system to inject new protocols based on consensus amongst coin-holders. There is even a system to obtain the protocol sources over the blockchain network so they can be compiled by each node and dynlinked directly in: you don't need to update/restart your node to get the protocol updates. However, this is arbitrary: you can start a new block-chain with a different protocol.

For example, you could re-implement Bitcoin (proof-of-work, coins+transfer, etc.) as a protocol that you inject on top of genesis. Your block chain would have a tezos genesis block, then a block that activate your own version of bitcoin, and then the blocks would be similar to what you would find on the bitcoin block-chain.

Of particular interest to you, the protocol you inject can have entirely different on-chain notions (e.g., a TCG/CCG with no coins at all but a notion of ownership over cards) and different soft-updating mechanism (e.g., the new protocol can accept genesis-style updates (a “dictatorship” where a single person controls the protocol) or even no soft-updating mechanism at all (a “stale” protocol where you need to hard-fork if you want to make significant changes)).

For this use case (of starting your own chain with a different protocol), you might be better off cloning the git repository, doing some minimal clean up, etc. This is because the tezos binaries include the sources for all protocols that have been used on the chain (so you don't need to get them over the network even if you can).

You might be interested in the following blog post about how to write your own protocol: https://blog.nomadic-labs.com/how-to-write-a-tezos-protocol.html

Official OCaml bindings for verified Everest cryptography

Jonathan Protzenko announced

The Everest team is pleased to announce the release of official OCaml bindings for all of our verified cryptographic algorithms, now available through OPAM as packages hacl-star and hacl-star-raw.

We provide bindings for the following:

  • HACL*, a library of pure C algorithms
  • Vale, a collection of optimized core assembly routines for maximum performance
  • EverCrypt, an agile, multiplexing API with CPU auto-detection that brings together HACL* and Vale.

Our code is compiled from the F* programming language to C via the KReMLin compiler ("K&R meets ML"). We offer two OPAM packages:

  • hacl-star-raw consists of low-level ocaml-ctypes bindings generated by KReMLin
  • hacl-star is a hand-written OCaml idiomatic API that uses much more pleasant signatures, types and abstractions and is also safer, as it checks all static preconditions at run-time

We support AES{128,256}-GCM, Chacha20-Poly1305, Curve25519 / Ed25519, P256, MD5, SHA-{1,2,3} (all variants), Blake2 (s&b), HMAC/HKDF, and the HPKE and SecretBox high-level APIs. Some algorithms are optimized for Intel chips, notably AES-GCM – see https://hacl-star.github.io/Supported.html for full details.

General documentation about the project is available at https://hacl-star.github.io/index.html – sample code for the OCaml API is provided as part of the test suite https://github.com/project-everest/hacl-star/tree/master/bindings/ocaml/tests

This work was performed by Victor Dumitrescu from Nomadic Labs, one of the teams responsible for the core development of the Tezos blockchain.

nmea and sail-gadgets

Davide Gessa announced

Ahoy developers, few days ago I published a new ocaml library called nmea, which is essentially a parser for NMEA0183 sentences, a format for encoding instruments data in boats. There are many sentences, regarding GPS, compass data, wind, air pressure, water temperature, waypoints handling, ais, autopilot and more; at the moment the library is able to decode GPS sentences and compass data, but I'll implement more sentences in the spare time. I tested it with my boat GPS and with a gps usb dongle.

After that, I started a new tiny experiment called sail-gadgets, which is a Gtk program that elaborates and displays NMEA data received from various boat instruments (wind vane, autopilot, gps, radar, ais, etc). Sail-gadgets can be extended with "gadgets" modules, each one providing new functionalities and new tabs to the main interface.

Data from sensors are handled using React signals, so in every gadget we can compose data from various sensor to obtain new reactive values.

The gadgets I'm planning to write:

  • dashboard: shows current position, speed, heading, tripdist, compass
  • satview: shows current connected gps satellites (partially done)
  • wind: shows wind indicator with true / apparent speed and direction
  • radar: shows AIS and Radar targets in range
  • mob: allows to drop a marker in the current position, and drive you to that point
  • startline: helper for regatta start
  • track: shows current track in a vector map

The hard thing in my opinion is writing new custom widget with cairo (compass, radar, and things like that).

Finally, the project is intended to run over gtk-broadway, so every html5 enabled device can access the application.

broadway.jpg

Hope there are some sailor here that want to join writing some gadgets :) Repos are:

Is there specialized math library for statistics?

hss asked

I searched to find math library which is written in OCaml, but there are only few repositories.

I'd like to use some function like coefficient correlation, covariance, etc.

I found Lacaml but it seems not to support them.

Could you give some link if you know?

bnguyenvanyen replied

Hi, you can take a look at Owl : https://ocaml.xyz/

There are stat functions and also a lot more

UnixJunkie also replied

There is also this one:
https://github.com/superbobry/pareto
GSL powered OCaml statistics library http://superbobry.github.io/pareto/0.2

And probably even some more:

opam search statistic
# Packages matching: match(*statistic*)
# Name            # Installed # Synopsis
[...]
gsl               --          GSL - Bindings to the GNU Scientific Library
oml               --          Math Library
owl               --          OCaml Scientific and Engineering Computing
owl-plplot        --          OCaml Scientific and Engineering Computing
pareto            --          GSL powered OCaml statistics library.
statsd-client     --          StatsD client library
[...]

New OCaml books?

Axel Wintermann asked

I wonder, why there are no new OCaml books since 2014 year? Many books are published on Haskell, Scala, F# themes, but no OCaml. I think we need new books for learning and for rising interest in our beautiful language.

Takuma Ishikawa replied

Weng Shiwei also replied

Other OCaml News

From the ocamlcore planet blog

Here are links from many OCaml blogs aggregated at OCaml Planet.

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.