Hello

Here is the latest OCaml Weekly News, for the week of June 30 to July
07, 2020.

Table of Contents
─────────────────

Releases of ringo
Multicore OCaml: June 2020
Time expression demo
Interactive OCaml development with utop in Emacs
Old CWN


Releases of ringo
═════════════════

  Archive: [https://discuss.ocaml.org/t/ann-releases-of-ringo/5605/5]


Continuing this thread, Raphaël Proust said
───────────────────────────────────────────

  Ringo provides bounded-size key-value stores. More specifically, it
  provides a functor similar to `Hastbl.Make' except that the number of
  bindings held by the tables is limited: inserting additional bindings
  when the limit has been reached causes some previously inserted
  binding to be removed.

  More more specifically, Ringo provides a function `map_maker' that
  takes parameters to customise the policies that determine the
  behaviour of the cache when supernumerary bindings are inserted, and
  returns the functor described above. Once a module `Cache' is
  instantiated using this functor, it can be used as follows:

  ┌────
  │ let cache = Cache.create size
  │ let fetch_data uri =
  │   match Cache.find_opt cache uri with
  │   | Some data -> data
  │   | None ->
  │     let data = really_fetch_data uri in
  │     Cache.replace cache uri data;
  │     data
  └────

  The cache will only hold up to [size] bindings, which avoids leaking
  memory. Additionally, the parameters for `map_maker' allow you to
  customise:

  • The replacement policy: which binding is removed when a
    supernumerary is inserted (currently supports least-recently used
    and first-in first-out).
  • The overflow policy: whether the cache can weakly hold some
    supernumerary elements (if so, the cache may hold more but the GC
    can always collect them if space is lacking).
  • The accounting precision: whether to keep precise track of
    removed/replaced elements.

  In addition, Ringo also provide set-caches: i.e., sets (rather than
  maps) with bounded size and all the same properties as above.

  Also note Ringo-Lwt (`ringo-lwt') provides Lwt wrappers around Ringo
  caches.

  If you have suggestions for a different concise synopsis for `opam',
  feel free to send them this way.

  Use cases are, I guess, caches. In particular those that might receive
  many elements not all of which you can hold in memory. We use it in a
  few places in the Tezos project to hold resources (blocks, operations,
  etc.) that are fetched from the P2p layer: it avoids having to fetch
  them again from the network.

  I think `anycache', `lru', and `lru-cache' are all alternatives
  available on opam.


Raphaël Proust later added
──────────────────────────

  The documentation is now available online at
  [https://nomadic-labs.gitlab.io/ringo/index.html]

  Of particular interest:
  • [The signature for a `ringo' key-value cache]
  • [The entry point for the `ringo' library] (allowing you to
    instantiate modules with the above signature as well as simple value
    caches)
  • [The signature for `ringo-lwt' cache]


[The signature for a `ringo' key-value cache]
https://nomadic-labs.gitlab.io/ringo/ringo/Ringo/module-type-CACHE_MAP/index.html

[The entry point for the `ringo' library]
https://nomadic-labs.gitlab.io/ringo/ringo/Ringo/index.html

[The signature for `ringo-lwt' cache]
https://nomadic-labs.gitlab.io/ringo/ringo-lwt/Ringo_lwt/Sigs/module-type-CACHE_MAP/index.html


Multicore OCaml: June 2020
══════════════════════════

  Archive:
  [https://discuss.ocaml.org/t/multicore-ocaml-june-2020/6047/1]


Anil Madhavapeddy announced
───────────────────────────

  Welcome to the June 2020 [Multicore OCaml] report! As with [previous
  updates], many thanks to @shakthimaan and @kayceesrk for collating the
  updates for the month of June 2020. /This is an incremental update;
  new readers may find it helpful to flick through the previous posts
  first./

  This month has seen a tremendous surge of activity on the upstream
  OCaml project to prepare for multicore integration, as @xavierleroy
  and the core team have driven a number of initiatives to prepare the
  OCaml project for the full multicore featureset.  To reflect this,
  from next month we will have a status page on the ocaml-multicore wiki
  with the current status of both our multicore branch and the upstream
  OCaml project itself.

  Why not from this month? Well, there's good news and bad news.  [Last
  month], I observed that we are a PR away from most of the opam
  ecosystem working with the multicore branch.  The good news is that we
  are still a single PR away from it working, but it's a different one
  :-) The retrofitting of the `Threads' library has brought up [some
  design complexities], and so rather than putting in a "bandaid" fix,
  we are integrating a comprehensive solution that will work with system
  threads, domains and (eventually) fibres.  That work has taken some
  time to get right, and I hope to be able to update you all on an
  opam-friendly OCaml 4.10.0+multicore in a few weeks.

  Aside from this, there have been a number of other improvements going
  into the multicore branches: [mingw Windows support], [callstack
  improvements], [fixing the Unix module] and so on. The full list is in
  the detailed report later in this update.


[Multicore OCaml] https://github.com/ocaml-multicore/ocaml-multicore

[previous updates] https://discuss.ocaml.org/tag/multicore-monthly

[Last month]
https://discuss.ocaml.org/t/multicore-ocaml-may-2020-update/5898

[some design complexities]
https://github.com/ocaml-multicore/ocaml-multicore/pull/342

[mingw Windows support]
https://github.com/ocaml-multicore/ocaml-multicore/pull/351

[callstack improvements]
https://github.com/ocaml-multicore/ocaml-multicore/pull/363

[fixing the Unix module]
https://github.com/ocaml-multicore/ocaml-multicore/pull/346

Sandmark benchmarks
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  A major milestone in this month has been the upgrade to the latest
  dune.2.6.0 to build Multicore OCaml 4.10.0 for the Sandmark
  benchmarking project. A number of new OPAM packages have been added,
  and the existing packages have been upgraded to their latest
  versions. The Multicore OCaml code base has seen continuous
  performance improvements and enhancements which can be observed from
  the various PRs mentioned in the report.

  We would like to thank:

  • @xavierleroy for working on a number of multicore-prequisite PRs to
    make stock OCaml ready for Multicore OCaml.
  • @camlspotter has reviewed and accepted the camlimages changes and
    made a release of camlimages.5.0.3 required for Sandmark.
  • @dinosaure for updating the decompress test benchmarks for Sandmark
    to build and run with dune.2.6.0 for Multicore OCaml 4.10.0.

  A chapter on Parallel Programming in Multicore OCaml with topics on
  task pool, channels section, profiling with code examples is being
  written. We shall provide an early draft version of the document to
  the community for your valuable feedback.


Papers
╌╌╌╌╌╌

  Our "Retrofitting Parallism onto OCaml" paper has been officially
  accepted at [ICFP 2020] which will be held virtually between August
  23-28, 2020. A [preprint] of the paper was made available earlier, and
  will be updated in a few days with the camera-ready version for ICFP.
  Please do feel free to send on comments and queries even after the
  paper is published, of course.

  Excitingly, another multicore-related paper on [Cosmo: A Concurrent
  Separation Logic for Multicore OCaml] will also be presented at the
  same conference.

  The Multicore OCaml updates are first listed in our report, which are
  followed by improvements to the Sandmark benchmarking
  project. Finally, the changes made to upstream OCaml which include
  both the ongoing and completed tasks are mentioned for your reference.


[ICFP 2020]
https://icfp20.sigplan.org/track/icfp-2020-papers#event-overview

[preprint] https://arxiv.org/abs/2004.11663

[Cosmo: A Concurrent Separation Logic for Multicore OCaml]
http://gallium.inria.fr/~fpottier/publis/mevel-jourdan-pottier-cosmo-2020.pdf


Multicore OCaml
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

◊ Ongoing

  • [ocaml-multicore/ocaml-multicore#339] Proposal for domain-local
    storage

    An RFC proposal to implement a domain-local storage in Multicore
    OCaml. Kindly review the idea and share your feedback!

  • [ocaml-multicore/ocaml-multicore#342] Implementing the threads
    library with Domains

    An effort to rebase @jhwoodyatt's implementation of the Thread
    library for Domains.

  • [ocaml-multicore/ocaml-multicore#357] Implementation of systhreads
    with pthreads

    Exploring the possibilty of implementing systhreads with pthreads,
    while still maintaining compatibility with the existing solution.

  • [ocaml/dune#3548] Dune fails to pick up secondary compiler

    The `ocaml-secondary-compiler' fails to install with
    dune.2.6.0. This is required as Multicore OCaml cannot build the
    latest dune without systhreads support.


  [ocaml-multicore/ocaml-multicore#339]
  https://github.com/ocaml-multicore/ocaml-multicore/issues/339

  [ocaml-multicore/ocaml-multicore#342]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/342

  [ocaml-multicore/ocaml-multicore#357]
  https://github.com/ocaml-multicore/ocaml-multicore/issues/357

  [ocaml/dune#3548] https://github.com/ocaml/dune/issues/3548


◊ Completed

  • [ocaml-multicore/multicore-opam#22] Update dune to 2.6.0

    The dune version in the Multicore OPAM repository is now updated to
    use the latest 2.6.0.

  • [ocaml-multicore/ocaml-multicore#338] Introduce Lazy.try_force and
    Lazy.try_force_val

    An implementation of `Lazy.try_force' and `Lazy.try_force_val'
    functions to implement concurrent lazy abstractions.

  • [ocaml-multicore/ocaml-multicore#340] Fix Atomic.exchange in
    concurrent_minor_gc

    A patch that introduces `Atomic.exchange' through `Atomic.get' that
    provides the appropriate read barrier for correct exchange semantics
    for `caml_atomic_exchange' in `memory.c'.

  • [ocaml-multicore/ocaml-multicore#343] Fix extcall noalloc DWARF

    The DWARF information emitted for `extcall noalloc' had broken
    backtraces and this PR fixes the same.

  • [ocaml-multicore/ocaml-multicore#345] Absolute exception stack

    The representation of the exception stack is changed from relative
    addressing to absolute addressing and the results are promising. The
    Sandmark serial benchmark results after the change is illustrated in
    the following graph:

    [https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/b/b385409b3f9e44cbfef98de668b0b4d0ed403472_2_1380x436.png]

  • [ocaml-multicore/ocaml-multicore#347] Turn on -Werror by default

    Adds a `--enable-warn-error' option to `configure' to treat C
    compiler warnings as errors.

  • [ocaml-multicore/ocaml-multicore#353] Poll for interrupts in
    cpu_relax without locking

    Use `Caml_check_gc_interrupt' first to poll for interrupts without
    locking, and then proceeding to handle the interrupt with the lock.

  • [ocaml-multicore/ocaml-multicore#354] Add Caml_state_field to
    domain_state.h

    The `Caml_state_field' macro definition in domain_state.h is
    required for base-v0.14.0 to build for Multicore OCaml 4.10.0 with
    dune.2.6.0.

  • [ocaml-multicore/ocaml-multicore#355] One more location to poll for
    interrupts without lock

    Another use of `Caml_check_gc_interrupt' first to poll for
    interrupts without lock, similar to
    [ocaml-multicore/ocaml-multicore#353].

  • [ocaml-multicore/ocaml-multicore#356] Backup threads for domain

    Introduces `backup threads' to perform GC and handle service
    interrupts when the domain is blocked in the kernel.

  • [ocaml-multicore/ocaml-multicore#358] Fix up bad CFI information in
    amd64.S

    Add missing `CFI_ADJUST' directives in `runtime/amd64.S' for
    `caml_call_poll' and `caml_allocN'.

  • [ocaml-multicore/ocaml-multicore#359] Inline caml_domain_alone

    The PR makes `caml_domain_alone' an inline function to improve
    performance for `caml_atomic_cas_field' and other atomics in
    `memory.c'.

  • [ocaml-multicore/ocaml-multicore#360] Parallel minor GC inline mask
    rework

    The inline mask rework for the promotion path to the
    `parallel_minor_gc' branch gives a 3-5% performance improvement for
    `test_decompress' sandmark benchmark, and a decrease in the executed
    instructions for all other benchmarks.

  • [ocaml-multicore/ocaml-multicore#361] Mark stack push work credit

    The PR improves the Multicore mark work accounting to be in line
    with stock OCaml.

  • [ocaml-multicore/ocaml-multicore#362] Iloadmut does not clobber rax
    and rdx when we do not have a read barrier

    A code clean-up to free the registers `rax' and `rdx' for OCaml code
    when `Iloadmut' is used.


  [ocaml-multicore/multicore-opam#22]
  https://github.com/ocaml-multicore/multicore-opam/pull/22

  [ocaml-multicore/ocaml-multicore#338]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/338

  [ocaml-multicore/ocaml-multicore#340]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/340

  [ocaml-multicore/ocaml-multicore#343]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/343

  [ocaml-multicore/ocaml-multicore#345]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/345

  [ocaml-multicore/ocaml-multicore#347]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/347

  [ocaml-multicore/ocaml-multicore#353]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/353

  [ocaml-multicore/ocaml-multicore#354]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/354

  [ocaml-multicore/ocaml-multicore#355]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/355

  [ocaml-multicore/ocaml-multicore#356]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/356

  [ocaml-multicore/ocaml-multicore#358]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/358

  [ocaml-multicore/ocaml-multicore#359]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/359

  [ocaml-multicore/ocaml-multicore#360]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/360

  [ocaml-multicore/ocaml-multicore#361]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/361

  [ocaml-multicore/ocaml-multicore#362]
  https://github.com/ocaml-multicore/ocaml-multicore/pull/362


Benchmarking
╌╌╌╌╌╌╌╌╌╌╌╌

◊ Ongoing

  • [ocaml-bench/sandmark#8] Ability to run compiler variants in
    Sandmark

    A feature to specify configure options when building compiler
    variants such as `flambda' is useful for development and
    testing. This feature is being worked upon.

  • [ocaml-bench/sandmark#107] Add Coq benchmarks

    We are continuing to add more benchmarks to Sandmark for Multicore
    OCaml and investigating adding the [Coq] benchmarks to our
    repertoire!

  • [ocaml-bench/sandmark#124] User configurable paramwrapper added to
    Makefile

    A `PARAMWRAPPER' environment variable can be passed as an argument
    by specifying the `--cpu-list' to be used for parallel benchmark
    runs.

  • [ocaml-bench/sandmark#131] Update decompress benchmarks

    Thanks to @dinosaure for updating the decompress benchmarks in order
    to run them with dune.2.6.0 for Multicore OCaml 4.10.0.

  • [ocaml-bench/sandmark#132] Update dependency packages to use
    dune.2.6.0 and Multicore OCaml 4.10.0

    Sandmark has been running with dune.1.11.4, and we need to move to
    the latest dune.2.6.0 for using Multicore OCaml 4.10.0 and beyond,
    as mentioned in [Promote dune to > 2.0]. The PR updates over 30
    dependency packages and successfully builds both serial and parallel
    benchmarks!


  [ocaml-bench/sandmark#8]
  https://github.com/ocaml-bench/sandmark/issues/8

  [ocaml-bench/sandmark#107]
  https://github.com/ocaml-bench/sandmark/issues/107

  [Coq] https://coq.inria.fr/

  [ocaml-bench/sandmark#124]
  https://github.com/ocaml-bench/sandmark/pull/124

  [ocaml-bench/sandmark#131]
  https://github.com/ocaml-bench/sandmark/pull/131

  [ocaml-bench/sandmark#132]
  https://github.com/ocaml-bench/sandmark/pull/132

  [Promote dune to > 2.0]
  https://github.com/ocaml-bench/sandmark/issues/106


◊ Completed

  • [camlspotter/camlimages#1] Use dune-configurator instead of
    configurator for camlimages

    A new release of `camlimages.5.0.3' was made by @camlspotter after
    accepting the changes to camlimages.opam in order to build with
    dune.2.6.0.

  • [ocaml-bench/sandmark#115] Task API Port: LU-Decomposition, Floyd
    Warshall, Mandelbrot, Nbody

    The changes to use the `Domainslib.Task' API for the listed
    benchmarks have been merged.

  • [ocaml-bench/sandmark#121] Mention sudo access for
    run_all_parallel.sh script

    The README.md file has been updated with the necessary `sudo'
    configuration steps to execute the `run_all_parallel.sh' script for
    nightly builds.

  • [ocaml-bench/sandmark#125] Add cubicle benchmarks

    The `German PFS' and `Szymanski's mutual exclusion algorithm'
    cubicle benchmarks have been included in Sandmark.

  • [ocaml-bench/sandmark#126] Update ocaml-versions README to reflect
    4.10.0+multicore

    The README has now been updated to reflect the latest 4.10.0
    Multicore OCaml compiler and its variants.

  • [ocaml-bench/sandmark#129] Add target to run parallel benchmarks in
    the CI

    The .drone.yml file used by the CI has been updated to run both the
    serial and parallel benchmarks.

  • [ocaml-bench/sandmark#130] Add missing dependencies in
    multicore-numerical

    The `domainslib' library has been added to the dune file for the
    multicore-numerical benchmark.


  [camlspotter/camlimages#1]
  https://gitlab.com/camlspotter/camlimages/-/merge_requests/1

  [ocaml-bench/sandmark#115]
  https://github.com/ocaml-bench/sandmark/pull/115

  [ocaml-bench/sandmark#121]
  https://github.com/ocaml-bench/sandmark/pull/121

  [ocaml-bench/sandmark#125]
  https://github.com/ocaml-bench/sandmark/pull/125

  [ocaml-bench/sandmark#126]
  https://github.com/ocaml-bench/sandmark/pull/126

  [ocaml-bench/sandmark#129]
  https://github.com/ocaml-bench/sandmark/pull/129

  [ocaml-bench/sandmark#130]
  https://github.com/ocaml-bench/sandmark/pull/130


OCaml
╌╌╌╌╌

◊ Ongoing

  • [ocaml/ocaml#9541] Add manual page for the instrumented runtime

    The [instrumented runtime] has been merged to OCaml 4.11.0. A manual
    for the same has been created and is under review.

  • [sadigqj/ocaml#1] GC colours change

    This PR removes the grey colour used in stock OCaml to match the
    scheme used by the Multicore major collector. The performance and
    considerations are included for review.


  [ocaml/ocaml#9541] https://github.com/ocaml/ocaml/pull/9541

  [instrumented runtime] https://github.com/ocaml/ocaml/pull/9082

  [sadigqj/ocaml#1] https://github.com/sadiqj/ocaml/pull/1


◊ Completed

  • [ocaml/ocaml#9619] A self-describing representation for function
    closures

    The PR provides a way to record the position of the environment for
    each entry point for function closures.

  • [ocaml/ocaml#9649] Marshaling for the new closure representation

    The `output_value' marshaler has been updated to use the new closure
    representation. There is no change required for the `input_value'
    unmarshaler.

  • [ocaml/ocaml#9655] Introduce type Obj.raw_data and functions
    Obj.raw_field, Obj.set_raw_field to manipulate out-of-heap pointers

    The PR introduces a type `Obj.bits', and functions `Obj.field_bits'
    and `Obj.set_field_bits' to read and write bit representation of
    block fields to support the no-naked-pointer operation.

  • [ocaml/ocaml#9678] Reimplement Obj.reachable_word using a hash table
    to detect sharing

    The `caml_obj_reachable_words' now uses a hash table instead of
    modifying the mark bits of block headers to detect sharing. This is
    required for compatibility with Multicore OCaml.

  • [ocaml/ocaml#9680] Naked pointers and the bytecode interpreter

    The bytecode interpreter implementation is updated to support the
    no-naked-pointers mode operation as required by Multicore OCaml.

  • [ocaml/ocaml#9682] Signal handling in native code without the page
    table

    The patch uses the code fragment table instead of a page table
    lookup for signal handlers to know whether the signal came from
    ocamlopt-generated code.

  • [ocaml/ocaml#9683] globroots.c: adapt to no-naked-pointers mode

    The patch considers out-of-heap pointers as major-heap pointers in
    no-naked-pointers mode for global roots management.

  • [ocaml/ocaml#9689] Generic hashing for the new closure
    representation

    The hashing functions have been updated to use the latest closure
    representation from [ocaml/ocaml#9619] for the no-naked-pointers
    mode.

  • [ocaml/ocaml#9698] The end of the page table is near

    The PR eliminates some of the use of the page tables in the runtime
    system when built with no-naked-pointers mode.

  Our thanks to all the OCaml developers and users in the community for
  their continued support and contribution to the project. Stay safe!


  [ocaml/ocaml#9619] https://github.com/ocaml/ocaml/pull/9619

  [ocaml/ocaml#9649] https://github.com/ocaml/ocaml/pull/9649

  [ocaml/ocaml#9655] https://github.com/ocaml/ocaml/pull/9655

  [ocaml/ocaml#9678] https://github.com/ocaml/ocaml/pull/9678

  [ocaml/ocaml#9680] https://github.com/ocaml/ocaml/pull/9680

  [ocaml/ocaml#9682] https://github.com/ocaml/ocaml/pull/9682

  [ocaml/ocaml#9683] https://github.com/ocaml/ocaml/pull/9683

  [ocaml/ocaml#9689] https://github.com/ocaml/ocaml/pull/9689

  [ocaml/ocaml#9698] https://github.com/ocaml/ocaml/pull/9698


Acronyms
╌╌╌╌╌╌╌╌

  • API: Application Programming Interface
  • CFI: Call Frame Information
  • CI: Continuous Integration
  • DWARF: Debugging With Attributed Record Formats
  • GC: Garbage Collector
  • ICFP: International Conference on Functional Programming
  • OPAM: OCaml Package Manager
  • PR: Pull Request
  • RFC: Request for Comments


Time expression demo
════════════════════

  Archive: [https://discuss.ocaml.org/t/time-expression-demo/6052/1]


Darren announced
────────────────

  An interactive demo for a small part of our time stuff and schedule
  handling library is available here:
  [https://daypack-dev.github.io/time-expr-demo/]

  Time expression is in essence a language for specifying time points or
  time slots precisely and concisely, while trying to mimic natural
  language.

  The implementation of the demo core itself can be seen here:
  [https://github.com/daypack-dev/time-expr-demo/blob/master/src/demo.ml]
  , where the usage of Daypack-lib is shown.

  Lastly, the library is still a prototype, so expect some faults in the
  outputs of the demo here and there.


Interactive OCaml development with utop in Emacs
════════════════════════════════════════════════

  Archive:
  [https://discuss.ocaml.org/t/interactive-ocaml-development-with-utop-in-emacs/6058/1]


Samarth Kishor announced
────────────────────────

  I made a [blog post] about REPL driven development with utop in Emacs
  a few months ago. Please let me know if you found it useful or have
  anything to add!  I'm a bit new to OCaml so any feedback helps.

  There was a [similar post about REPL driven development] last year and
  my post addresses a lot of those points. I wish I'd seen that post
  before I wrote this since there's a ton of useful information in the
  comments.


[blog post]
https://samarthkishor.github.io/posts/interactive_ocaml_development/

[similar post about REPL driven development]
https://discuss.ocaml.org/t/ocaml-repl-driven-development/4068


Old CWN
═══════

  If you happen to miss a CWN, you can [send me a message] and I'll mail
  it to you, or go take a look at [the archive] or the [RSS feed of the
  archives].

  If you also wish to receive it every week by mail, you may subscribe
  [online].

  [Alan Schmitt]


[send me a message] mailto:alan.schmitt@polytechnique.org

[the archive] http://alan.petitepomme.net/cwn/

[RSS feed of the archives] http://alan.petitepomme.net/cwn/cwn.rss

[online] http://lists.idyll.org/listinfo/caml-news-weekly/

[Alan Schmitt] http://alan.petitepomme.net/