[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

[Alan Schmitt <alan.schmitt@polytechnique.org>]

[-- Attachment #1.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[-- Attachment #2.1: Type: text/plain, Size: 23258 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of July 21 to 28,
2020.

As I will be away with no internet next week, the next CWN will be on
August 11.

Table of Contents
─────────────────

Embedded ocaml templates
Proposal: Another way to debug memory leaks
Camlp5 (8.00~alpha01) and pa_ppx (0.01)
OCaml 4.11.0, third (and last?) beta release
Other OCaml News
Old CWN


Embedded ocaml templates
════════════════════════

  Archive: [https://discuss.ocaml.org/t/embedded-ocaml-templates/6124/1]


Emile Trotignon announced
─────────────────────────

  I am very happy to announce the release of ocaml-embedded-templates.

  This is a tool similar to camlmix, but camlmix was not updated for 7
  years, and there is no easy way to handle a lot of templates (my
  command takes a directory as an argument and generate an ocaml module
  by going through the directory recursively) I also choosed to use a
  syntax similar to EJS, and there is a ppx for inline EML.

  You can check it out here :
  [https://github.com/EmileTrotignon/embedded_ocaml_templates]

  Here is a more extensive exemple of what can be done with this :
  [https://github.com/EmileTrotignon/resume_of_ocaml] (This project
  generate my resume/website in both latex and html).

  This is my first opam package : feedback is very much welcome.


Proposal: Another way to debug memory leaks
═══════════════════════════════════════════

  Archive:
  [https://discuss.ocaml.org/t/proposal-another-way-to-debug-memory-leaks/6134/1]


Jim Fehrle said
───────────────

  `memprof' helps you discover where memory was allocated, which is
  certainly useful.  However, that may not be enough information to
  isolate a leak.  Sometimes you'd like to know what variables refer to
  excessive amounts of memory.

  For this, you'd want to examine all the garbage collection roots and
  report how much memory is used by each.  This is useful information if
  you can map a GC root back to a source file and variable.

  I prototyped code to do that to help with Coq bug
  [https://github.com/coq/coq/issues/12487].  It localized several leaks
  enough across over 500 source files so that we could find and fix
  them.  But my prototype code is a bit crude.  I'd like to clean it up
  and submit it as a PR.  Since this could be done in various ways, I
  wanted to get some design/API feedback up front rather than maybe
  doing some of it twice.  Also I'd like to confident that such a PR
  would be accepted and merged in a reasonable amount of time–otherwise
  why bother.

  [caml_do_roots] shows how to access the GC roots.  There are several
  types of roots:
  • global roots, corresponding to top-level variables in source files
  • dynamic global roots
  • stack and local roots
  • global C roots
  • finalized values
  • memprof
  • hook

  *API (in Gc):*

  ┌────
  │ val print_global_reachable : out_channel -> int -> unit
  └────

  Prints a list to `out_channel' of the global roots that reach more
  than the specified number of words.  Each item shows the number of
  reachable words, the associated index of the root in the `*glob' for
  that file and the name of the source file.

  Something like this (but with only filenames rather than pathnames):

  ┌────
  │   102678 field  17 plugins/ltac/pltac.ml
  │   102730 field  18 plugins/ltac/pltac.ml
  │   164824 field  20 plugins/ltac/tacenv.ml
  │  1542857 field  26 plugins/ltac/tacenv.ml
  │ 35253743 field  65 stm/stm.ml
  │ 35201913 field   8 vernac/vernacstate.ml
  │  8991864 field  24 vernac/library.ml
  │   112035 field   8 vernac/egramml.ml
  │  6145454 field  84 vernac/declaremods.ml
  │  6435878 field  89 vernac/declaremods.ml
  └────

  I would use ELF information in the binary file to map from `*glob'
  back to a filename.  For example, the address symbol of the entry
  `camlTest' corresponds to `test.ml'.  This would only work for binary
  executables compiled with the `-g' option.  It wouldn't work for
  byte-compiled code.  It would print an error message if it's not ELF
  or not `-g'.  Also, being a little lazy, how essential is it to
  support 32-bit binaries?  (Q: What happens if you have 2 source files
  with the same name though in different directories?  Would the symbol
  table distinguish them?)

  ┌────
  │ val get_field_index : Obj.t -> int
  └────

  Returns the `*glob' index number for the top-level variable (passed as
  `Obj.repr var').  I expect there's no way to recover variable names
  from the `*glob' index.  In my experiments, it appeared that the
  entries in `*glob' were in the same order as as the variable and
  function declarations.  This would let a developer do a binary search
  in the code to locate the variable which it probably a necessity for
  large, complex files such as Coq's `stm.ml'–3300 lines, 10+ modules
  defined within the file.  (I noticed that variables defined in modules
  defined within the source file were not in `*glob'.  I expect there is
  a root for the module as a whole and that those variables can be
  readily found within that root.)

  This would need an extended explanation in `gc.mli'.

  ┌────
  │ val print_stack_reachable : out_channel -> int -> unit
  └────

  Prints a backtrace to `out_channel' that also shows which roots for
  each frame reach more than the specified number of words.  (I'd keep
  the "item numbers" since there's no way to translate them to variables
  and they might give some clues.)

  ┌────
  │ Called from file "tactics/redexpr.ml" (inlined), line 207, characters 29-40
  │  356758154 item    0 (stack)
  │ Called from file "plugins/ltac/tacinterp.ml", line 752, characters 6-51
  │   17646719 item    0 (stack)
  │     119041 item    1 (stack)
  │ Called from file "engine/logic_monad.ml", line 195, characters 38-43
  │     119130 item    0 (stack)
  │  373378237 item    1 (stack)
  └────

  As it turns out, 90% of the memory in Coq issue mentioned above is
  reachable only from the stack.

  I didn't consider the other types of roots yet, which I don't fully
  understand, such as local roots.  Just covering global and stack roots
  seems like a good contribution.  Dynamic global roots may be easy to
  add if they are otherwise similar to global roots.  For the others I
  could print the reachable words, but I don't know how to direct the
  developer to look at the relevant part of the code, especially if it's
  in C code.  I suppose `print_global_reachable' and
  `print_stack_reachable' could be a single routine as well.  That's
  probably better.

  Let me know your thoughts.


[caml_do_roots]
https://github.com/ocaml/ocaml/blob/80326033cbedfe59c0664e3912f21017e968a1e5/runtime/roots_nat.c#L399


Camlp5 (8.00~alpha01) and pa_ppx (0.01)
═══════════════════════════════════════

  Archive:
  [https://discuss.ocaml.org/t/ann-camlp5-8-00-alpha01-and-pa-ppx-0-01/6144/1]


Chet Murthy announced
─────────────────────

`Camlp5 (8.00~alpha01)' and `pa_ppx (0.01)'
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  I'm pleased to announce the release of two related projects:

  1. [Camlp5]: version 8.00~alpha01 is an alpha release of Camlp5, with
     full support for OCaml syntax up to version 4.10.0, as well as
     minimal compatibility with version 4.11.0. In particular there is
     full support for PPX attributes and extensions.

  2. [pa_ppx]: version 0.01 is a re-implementation of a large number of
     PPX rewriters (e.g. ppx_deriving (std (show, eq, map, etc), yojson,
     sexp, etc), ppx_import, ppx_assert, others) on top of Camlp5, along
     with an infrastructure for developing new ones.

  This allows projects to combine the existing style of Camlp5 syntax
  extension, with PPX rewriting, without having to jump thru hoops to
  invoke camlp5 on some files, and PPX processors on others.

  Camlp5 alone is not compatible with existing PPX rewriters: Camlp5
  syntax-extensions (e.g. "stream parsers") would be rejected by the
  OCaml parser, and PPX extensions/attributes are ignored by Camlp5
  (again, without `pa_ppx').  `pa_ppx' provides Camlp5-compatible
  versions of many existing PPX rewriters, as well as new ones, so that
  one can use Camlp5 syntax extensions as well as PPX rewriters.  In
  addition, some of the re-implemented rewriters are more-powerful than
  their original namesakes, and there are new ones that add interesting
  functionality.


[Camlp5] https://github.com/camlp5/camlp5

[pa_ppx] https://github.com/chetmurthy/pa_ppx


For democratizing macro-extension-authoring in OCaml
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  TL;DR Writing OCaml PPX rewriters is *hard work*.  There is a
  complicated infrastructure that is hard to explain, there are multiple
  such incompatible infrastructures (maybe these are merging?) and it is
  hard enough that most Ocaml programmers do not write macro-extensions
  as a part of their projects.  I believe that using Camlp5 and pa_ppx
  can make it easier to write macro-extensions, via:

  1. providing a simple way of thinking about adding your extension to
     the parsing process.

  2. providing transparent tools (e.g. quotations) for
     pattern-matching/constructing AST fragments

  Explained below in [Macro Extensions with
  Pa_ppx](#macro-extensions-with-pa_ppx).


◊ The original arguments against Camlp4

  The original argument against using Camlp4 as a basis for
  macro-preprocessing in Ocaml, had several points (I can't find the
  original document, but from memory):

  1. *syntax-extension* as the basis of macro-extension leads to brittle
      syntax: multiple syntax extensions often do not combine well.

  2. a different AST type than the Ocaml AST

  3. a different parsing/pretty-printing infrastructure, which must be
     maintained alongside of Ocaml's own parser/pretty-printer.

  4. A new and complicated set of APIs are required to write syntax
     extensions.

  To this, I'll add

  1. Camlp4 was *forked* from Camlp5, things were changed, and hence,
     Camlp4 lost the contribution of its original author.  Hence,
     maintaining Camlp4 was always labor that fell on the Ocaml
     team. [Maybe this doesn't matter, but it counts for something.]


◊ Assessing the arguments, with some hindsight

  1. *syntax-extension* as the basis of macro-extension leads to brittle
      syntax: multiple syntax extensions often do not combine well.

     In retrospect, this is quite valid: even if one prefers and enjoys
     LL(1) grammars and parsing, when multiple authors write
     grammar-extensions which are only combined by third-party projects,
     the conditions are perfect for chaos, and of a sort that
     project-authors simply shouldn't have to sort out.  And this chaos
     is of a different form, than merely having two PPX rewriters use
     the same attribute/extension-names (which is, arguably, easily
     detectable with some straightforward predeclaration).

  2. Camlp4/5 has a different AST type than the Ocaml AST

     Over time, the PPX authors themselves have slowly started to
     conclude that the current reliance on the Ocaml AST is fraught with
     problems.  The "Future of PPX" discussion thread talks about using
     something like s-expressions, and more generally about a
     more-flexible AST type.

  3. a different parsing/pretty-printing infrastructure, which must be
     maintained alongside of Ocaml's own parser/pretty-printer.

     A different AST type necessarily means a different
     parser/pretty-printer.  Of course, one could modify Ocaml's YACC
     parser to produce Camlp5 ASTs, but this is a minor point.

  4. A new and complicated set of APIs are required to write syntax
     extensions.

     With time, it's clear that PPX has produced the same thing.

  5. Maintaining Camlp4 was always labor that fell on the Ocaml team.

     The same argument (that each change to the Ocaml AST requires work
     to update Camlp5) can be made for PPX (specifically, this is the
     raison d'etre of ocaml-migrate-parsetree).  Amusingly, one could
     imagine using ocaml-migrate-parsetree as the basis for making
     Camlp5 OCaml-version-independent, too.  That is, the "backend" of
     Camlp5 could use ocaml-migrate-parsetree to produce ASTs for a
     version of OCaml different from the one on which it was compiled.


Arguments against the current API(s) of PPX rewriting
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  The overall argument is that it's too complicated for most OCaml
  programmers to write their own extensions; what we see instead of a
  healthy ecosystem of many authors writing and helping-improve PPX
  rewriters, is a small number of rewriters, mostly written by Jane
  Street and perhaps one or two other shops.  There are a few big
  reasons why this is the case (which correspond to the responses
  above), but one that isn't mentioned is:

  1. When the "extra data" of a PPX extension or attribute is
     easily-expressed with the fixed syntax of PPX payloads, all is
     `~well~' ok, but certainly not in great shape.  Here's an example:

  ┌────
  │ type package_type =
  │ [%import: Parsetree.package_type
  │ 	  [@with core_type    := Parsetree.core_type [@printer Pprintast.core_type];
  │ 		 Asttypes.loc := Asttypes.loc [@polyprinter fun pp fmt x -> pp fmt x.Asttypes.txt];
  │ 		 Longident.t  := Longident.t [@printer pp_longident]]]
  │ [@@deriving show]
  └────

  The expression-syntax of assignment is used to express type-expression
  rewrites.  And this is necesarily limited, because we cannot (for
  example) specify left-hand-sizes that are type-expressions with
  variables.  It's a perversion of the syntax, when what we really want
  to have is something that is precise: "map this type-expression to
  that type-expression".

  Now, with the new Ocaml 4.11.0 syntax, there's a (partial) solution:
  use "raw-string-extensions" like `{%foo|argle|}'.  This is the same as
  `[%foo {|argle|}]'.  This relies on the PPX extension to parse the
  payload.  But there are problems:

  1. Of course, there's no equivalent `{@foo|argle|}' (and "@@", "@@@"
     of course) for attributes.

  2. If the payload in that string doesn't *itself* correspond to some
     parseable Ocaml AST type, then again, we're stuck: we have to
     cobble together a parser instead of being able to merely extend the
     parser of Ocaml to deal with the case.

  Note well that I'm not saying that we should extend the parsing rules
  of the Ocaml language.  Rather, that with an *extensible parser*
  (hence, LL(1)) we can add new nonterminals, add rules that reference
  existing nonterminals, and thereby get an exact syntax (e.g.) for the
  `ppx_import' example above.  That new nonterminal is used *only* in
  parsing the payload – nowhere else – so we haven't introduced examples
  of objection #1 above.

  And it's not even very hard.


Macro Extensions with Pa_ppx
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  The basic thesis of `pa_ppx' is "let's not throw the baby out with the
  bathwater".  Camlp5 has a lot of very valuable infrastructure that can
  be used to make writing macro-preprocessors much easier.  `pa_ppx'
  adds a few more.

  1. Quotations for patterns and expressions over all important OCaml
     AST types.

  2. "extensible functions" to make the process of recursing down the
     AST transparent, and the meaning of adding code to that process
     equally transparent.

  3. `pa_ppx' introduces "passes" and allows each extension to register
     which other extensions it must follow, and which may follow it;
     then `pa_ppx' topologically sorts them, so there's no need for
     project-authors to figure out how to order their PPX extension
     invocations.

  As an example of a PPX rewriter based on `pa_ppx', here's
  [pa_ppx.here] from the `pa_ppx' tutorial.  In that example, you'll see
  that Camlp5 infrastructure is used to make things easy:

  1. quotations are used to both build the output AST fragment, and to
     pattern-match on inputs.

  2. the "extensible functions" are used to add our little bit of
     rewriter to the top-down recursion.

  3. and we declare our rewriter to the infrastructure (we don't specify
     what passes it must come before or after, since `pa_ppx.here' is so
     simple).


[pa_ppx.here]
https://pa-ppx.readthedocs.io/en/latest/tutorial.html#an-example-ppx-rewriter-based-on-pa-ppx


Conclusion
╌╌╌╌╌╌╌╌╌╌

  I'm not trying to convince you to switch away from PPX to Camlp5.
  Perhaps, I'm not even merely arguing that you should use `pa_ppx' and
  author new macro-extensions on it.  But I *am* arguing that the
  features of

  1. quotations, with antiquotations in as many places as possible, and
     hence, *in places where Ocaml identifiers are not permitted*.

  2. facilities like "extensible functions", with syntax support for
     them

  3. a new AST type, that is suitable for macro-preprocessing, but isn't
     merely "s-expressions" (after all, there's a reason we all use
     strongly-typed languages)

  4. an extensible parser for the Ocaml language, usable in PPX
     attribute/extension payloads

  are important and valuable, and a PPX rewriter infrastructure that
  makes it possible for the masses to write their own macro-extensions,
  is going to incorporate these things.


OCaml 4.11.0, third (and last?) beta release
════════════════════════════════════════════

  Archive:
  [https://discuss.ocaml.org/t/ocaml-4-11-0-third-and-last-beta-release/6149/1]


octachron announced
───────────────────

  The release of OCaml 4.11.0 is near.  As one step further in this
  direction, we have published a third and potentially last beta
  release.

  This new release fixes an infrequent best-fit allocator bug and an
  issue with floating-point software emulation in the ARM EABI port.  On
  the ecosystem side, merlin is now available for this new version of
  OCaml.  The compatibility of the opam ecosystem with OCaml 4.11.0 is
  currently good, and it should be possible to test this beta without
  too much trouble.

  The source code is available at these addresses:

  [https://github.com/ocaml/ocaml/archive/4.11.0+beta3.tar.gz]
  [https://caml.inria.fr/pub/distrib/ocaml-4.11/ocaml-4.11.0+beta3.tar.gz]

  The compiler can also be installed as an OPAM switch with one of the
  following commands:
  ┌────
  │ opam update
  │ opam switch create ocaml-variants.4.11.0+beta3 --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git
  └────
  or
  ┌────
  │ opam update
  │ opam switch create ocaml-variants.4.11.0+beta3+VARIANT --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git
  └────
  where you replace VARIANT with one of these: afl, flambda, fp,
  fp+flambda

  We would love to hear about any bugs. Please report them here:
   [https://github.com/ocaml/ocaml/issues]

  Compared to the previous beta release, the exhaustive list of changes
  is as follows:


Runtime:
╌╌╌╌╌╌╌╌

  • [#9736], [#9749]: Compaction must start in a heap where all free
    blocks are blue, which was not the case with the best-fit
    allocator. (Damien Doligez, report and review by Leo White)

  • + [*new bug fixes*] [#9316], [#9443], [#9463], [#9782]: Use typing
    information from Clambda or mutable Cmm variables. (Stephen Dolan,
    review by Vincent Laviron, Guillaume Bury, Xavier Leroy, and Gabriel
    Scherer; temporary bug report by Richard Jones)


[#9736] https://github.com/ocaml/ocaml/issues/9736

[#9749] https://github.com/ocaml/ocaml/issues/9749

[#9316] https://github.com/ocaml/ocaml/issues/9316

[#9443] https://github.com/ocaml/ocaml/issues/9443

[#9463] https://github.com/ocaml/ocaml/issues/9463

[#9782] https://github.com/ocaml/ocaml/issues/9782


Manual and documentation:
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • [#9541]: Add a documentation page for the instrumented runtime;
    additional changes to option names in the instrumented
    runtime. (Enguerrand Decorne, review by Anil Madhavapeddy, Gabriel
    Scherer, Daniel Bünzli, David Allsopp, Florian Angeletti, and
    Sébastien Hinderer)

  Entries marked with "+" were already present in previous alphas, but
  they have been complemented by new bug fixes.

  If you are interested by the list of new features, and the nearly
  final list of bug fixes the updated change log for OCaml 4.11.0 is
  available at:

  [https://github.com/ocaml/ocaml/blob/4.11/Changes]


[#9541] https://github.com/ocaml/ocaml/issues/9541


Other OCaml News
════════════════

From the ocamlcore planet blog
──────────────────────────────

  Here are links from many OCaml blogs aggregated at [OCaml Planet].

  • [Frama-Clang 0.0.9 is out. Download it here.]


[OCaml Planet] http://ocaml.org/community/planet/

[Frama-Clang 0.0.9 is out. Download it here.]
http://frama-c.com/index.html


Old CWN
═══════

  If you happen to miss a CWN, you can [send me a message] and I'll mail
  it to you, or go take a look at [the archive] or the [RSS feed of the
  archives].

  If you also wish to receive it every week by mail, you may subscribe
  [online].

  [Alan Schmitt]


[send me a message] mailto:alan.schmitt@polytechnique.org

[the archive] http://alan.petitepomme.net/cwn/

[RSS feed of the archives] http://alan.petitepomme.net/cwn/cwn.rss

[online] http://lists.idyll.org/listinfo/caml-news-weekly/

[Alan Schmitt] http://alan.petitepomme.net/


[-- Attachment #2.2: Type: text/html, Size: 37178 bytes --]