[Caml-list] [ANN] Core Suite 111.17.00

[Caml-list] [ANN] Core Suite 111.17.00

[Ben Millwood]

[-- Attachment #1: Type: text/plain, Size: 6936 bytes --]

I am pleased to announce the (somewhat delayed) 111.17.00 release of the
Core suite.

The following packages were upgraded:

- async
- async_extra
- async_inotify
- async_kernel
- async_parallel
- async_unix
- bignum
- core
- core_extended
- core_kernel
- faillib
- jenga
- ocaml_plugin
- patdiff
- patience_diff
- sexplib
- typerep

We're also releasing for the first time a new library called
async_extended, which contains more experimental and less well-scrutinised
code than async, in much the same role that core_extended plays for core.

Note that async_extra, despite the name, does not play that role: the code
in async_extra is subject to the same level of scrutiny as async or
async_kernel. We've had discussions internally about merging async and
async_extra into a single library, since they serve essentially the same
purpose, but it's no-one's top priority.

Files and documentation for this release are available on our website and
all packages are in opam:

  https://ocaml.janestreet.com/ocaml-core/111.17.00/individual
  https://ocaml.janestreet.com/ocaml-core/111.17.00/doc

Here is the list of changes for this version:

## async_extra

- Added module `Persistent_rpc_client`, an RPC client that attempts to
  reconnect when the connection is lost, until a new connection is
  established.
- Significantly sped up the `Rpc` module by removing `Bigstring`
  serialization.

  Performance of the two implementations was tested by building a
  simple client/server executable that would count major cycles.
  Sending 100 byte messages at a rate of 50k/second shows (on both
  sides of the RPC):

  original:
  * ~160 major cycles in 30s
  * CPU usage around 60%

  new:
  * ~10 major cycles in 30s
  * CPU usage <= 2%
- Enabled a version of `Pipe_rpc` and `State_rpc` where the consumer
  can pushback on the producer if it can't consume the contents of the
  pipe fast enough.
- Added `Log.Level.arg : Log.Level.t Command.Spec.Arg_type.t` for
  defining command lines that accept (and autocomplete) log levels.
- Added `Command.async_or_error` and renamed `Command.async_basic` to
  `Command.async`, leaving `async_basic` a deprecated alias for the
  new name.

  `Command.async_or_error` is similar to `Command.basic` and
  `Command.async`, but accepts a `unit Or_error.t Deferred.t` type.
- Added `Persistent_rpc_connection.current_connection`, so that one
  can detect whether one is currently connected.

  ```ocaml
  val current_connection : t -> Rpc.Connection.t option
  ```

## async_inotify

- Upgraded library to use inotify 2.0

## async_kernel

- Renamed `Monitor.errors` to `Monitor.detach_and_get_error_stream`
  and `Monitor.error` as `Monitor.get_next_error`.

  The use of `detach` in the name is intended to make clear that
  errors do not propagate to the parent.

  Added several other non-stream =Monitor= functions to capture common
  use cases of `Monitor.detach_and_get_error_stream`:

  ```ocaml
  detach_and_get_next_error
  detach_and_iter_errors
  detach
  ```

## bignum

- Improved the performance of binprot deserialization by removing the
  allocation of an intermediate type.

## core

- Fixed a bug in `Bigstring.really_recv` if `recv` doesn't receive all
  the data it wants.

  This bug has been around forever; it may not have caused trouble
  because `Bigstring.really_recv` (1) is barely used (the only use is
  in `Bigstring.unmarshal_from_sock`) and (2) passes `recv` the
  `MSG_WAITALL` flag, so it will read the full amount unless it gets
  interrupted by a signal.
- Fixed `Bigstring.read`'s handling of `EINTR` so that it retries
  rather than returning zero.

  This fixes a bug introduced in 111.09 in the interaction between
  `Bigstring.read` and `Async.Reader`.  Prior to 111.09,
  `Bigstring.read` would raise on `EINTR`, and `Async.Reader` would
  propagate the exception.  From 111.09 to 111.16, `Bigstring.read`
  would return zero, which would confuse `Async.Reader` into thinking
  it reached EOF when it hadn't.  From 111.17, `Bigstring.read` will
  retry and not return zero when not at EOF.

  We believe the bug was rare, because otherwise we would have
  frequently seen `EINTR` exceptions prior to 111.09.
- Added `Command.Spec.apply` and `pair`, which allow one to program
  more with `Spec.param` rather than `Spec.t`.

  ```ocaml
  val apply : ('a -> 'b) param -> 'a param -> 'b param
  val pair : 'a param -> 'b param -> ('a * 'b) param
  ```
- Added `Command.Spec.file`, which builds an `Arg_type` value with the
  same autocompletion as `Spec.file`.

  ```ocaml
  (** [file] defines an [Arg_type.t] that completes in the same way as
      [Command.Spec.file], but perhaps with a different type than [string]
or with an
      autocompletion key. *)
  val file
    :  ?key:'a Univ_map.Multi.Key.t
    -> (string -> 'a)
    -> 'a t
  ```

## core_extended

- Added some functions to `Splay_tree`:
  * `length`
  * `keys`
  * `data`
  * `to_alist`
  * `delete_{after,before}`
  * `map`
  * `map_range`
  * `split`.

## core_kernel

- In `Bigstring`, made many operations use compiler primitives new in
  OCaml 4.01.

  Exposed `Bigstring.get` and `set` as compiler primitives in the
  interface.

  Added `Bigstring.unsafe_get_int64_{le,be}_trunc`.
- Made `Error` round trip `exn`, i.e. `Error.to_exn (Error.of_exn exn)
  = exn`.
- Added to `failwiths` an optional `?here:Lexing.position` argument.
- Added `with typerep` to `Flags.S`.
- Optimized `List.dedup []` to return immediately.
- Added `data` argument to polymorphic type
`Hashtbl_intf.Creators.create_options`.

  This allows implementations of `Hashtbl_intf.Creators` to have
  constructor arguments that depend on the type of both key and data
  values.  For example:

  ```ocaml
  module type Hashtbl_creators_with_typerep =
    Hashtbl_intf.Creators
    with type ('key, 'data, 'z) create_options
      =  typerep_of_key:'key Typerep.t
      -> typerep_of_data:'data Typerep.t
      -> 'z
  ```
- Improved the interface for getting `Monad.Make` to define `map` in
  terms of `bind`.

  Instead of passing a `map` function and requiring everyone who wants
  to define `map` using `bind` to call a special function, we use a
  variant type to allow the user to say what they want:

  ```ocaml
  val map : [ `Define_using_bind
            | `Custom of ('a t -> f:('a -> 'b) -> 'b t)
            ]
  ```
- Improved the performance of many `Dequeue` functions.

  Previously, many `Dequeue.dequeue`-type functions worked by raising
  and then catching an exception when the dequeue is empty.  This is
  much slower than just testing for emptiness, which is what the code
  now does.

  This improves the performance of `Async.Writer`, which uses
  `Dequeue.dequeue_front`.

## patdiff

- Removed latex output.

## patience_diff

- Exposed `Patience_diff.matches`.

## sexplib

- Make the camlp4 dependency optional

---

We hope you find it useful!

-- Ben Millwood, on behalf of the Core team

[-- Attachment #2: Type: text/html, Size: 8443 bytes --]

Re: [Caml-list] [ANN] Core Suite 111.17.00

[François Bobot]

Hi,

Thank you for this new release of all these great libraries.

On 17/06/2014 18:43, Ben Millwood wrote:
> ## sexplib
> 
> - Make the camlp4 dependency optional

Is it linked to a path toward extension points (4.02)? It is mentionned[1] that the syntax will
changed, from:

type t = int * string with sexp

to:

type t = int * string [@@sexp] .

But is there some plan for a version of type-conv and sexp/bin-prot/... that will accept the new
syntax with camlp4 for old compiler and without camlp4 but with extension point for new one?

Thanks,


[1] https://blogs.janestreet.com/extension-points-or-how-ocaml-is-becoming-more-like-lisp/

-- 
François Bobot

Re: [Caml-list] [ANN] Core Suite 111.17.00

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

This change isn't really part of that transition, but that transition will
likely be necessary.  My sense is that the right thing is to rewrite the
syntax extensions using extension points, and then write a shim camlp4
extension that converts "with sexp" to the correct annotation.  That said,
this is a long and tricky process that won't happen immediately on 4.02's
release.
On Jun 18, 2014 3:35 AM, "François Bobot" <francois.bobot@cea.fr> wrote:

> Hi,
>
> Thank you for this new release of all these great libraries.
>
> On 17/06/2014 18:43, Ben Millwood wrote:
> > ## sexplib
> >
> > - Make the camlp4 dependency optional
>
> Is it linked to a path toward extension points (4.02)? It is mentionned[1]
> that the syntax will
> changed, from:
>
> type t = int * string with sexp
>
> to:
>
> type t = int * string [@@sexp] .
>
> But is there some plan for a version of type-conv and sexp/bin-prot/...
> that will accept the new
> syntax with camlp4 for old compiler and without camlp4 but with extension
> point for new one?
>
> Thanks,
>
>
> [1]
> https://blogs.janestreet.com/extension-points-or-how-ocaml-is-becoming-more-like-lisp/
>
> --
> François Bobot
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2122 bytes --]

Re: [Caml-list] [ANN] Core Suite 111.17.00

[François Bobot]

On 18/06/2014 11:28, Yaron Minsky wrote:
> This change isn't really part of that transition, but that transition will likely be necessary.  My
> sense is that the right thing is to rewrite the syntax extensions using extension points, and then
> write a shim camlp4 extension that converts "with sexp" to the correct annotation.  That said, this
> is a long and tricky process that won't happen immediately on 4.02's release.

Thank you for the quick answer. That seems the right transition path for the code of a project, but
I wonder what happen at a bigger scale.

After the transition projects will need to support older versions of ocaml that perhaps don't have
extension points. Moreover every projects will not do the transition at the same time.

It seems, IMHO, that type-conv will have to work for the following cases transparently:

- camlp4 (with ocaml < 4.02 or >= 4.02) old syntax
- camlp4 (with ocaml < 4.02 or >= 4.02) new syntax

- extension point (with ocaml >= 4.02) new syntax


Do you agree with that? Do you think it is doable/maintainable until ocaml < 4.02 are not used anymore?


Best,

-- 
François

Re: [Caml-list] [ANN] Core Suite 111.17.00

[Yaron Minsky]

We actually had a discussion about this internally, and I was mostly
convinced that my transition plan is the wrong one --- that it's
probably better to leave the old camlp4 based syntax extensions as
they are, and just mothball them; leave them around, but stop
improving them.  And then people who want to use the new and
better-supported extensions will need to flip to using ppx.  And you
can use the camlp4-based solution to straddle the gap if you need to
support both new and old compilers.

y

On Wed, Jun 18, 2014 at 8:57 AM, François Bobot <francois.bobot@cea.fr> wrote:
> On 18/06/2014 11:28, Yaron Minsky wrote:
>> This change isn't really part of that transition, but that transition will likely be necessary.  My
>> sense is that the right thing is to rewrite the syntax extensions using extension points, and then
>> write a shim camlp4 extension that converts "with sexp" to the correct annotation.  That said, this
>> is a long and tricky process that won't happen immediately on 4.02's release.
>
> Thank you for the quick answer. That seems the right transition path for the code of a project, but
> I wonder what happen at a bigger scale.
>
> After the transition projects will need to support older versions of ocaml that perhaps don't have
> extension points. Moreover every projects will not do the transition at the same time.
>
> It seems, IMHO, that type-conv will have to work for the following cases transparently:
>
> - camlp4 (with ocaml < 4.02 or >= 4.02) old syntax
> - camlp4 (with ocaml < 4.02 or >= 4.02) new syntax
>
> - extension point (with ocaml >= 4.02) new syntax
>
>
> Do you agree with that? Do you think it is doable/maintainable until ocaml < 4.02 are not used anymore?
>
>
> Best,
>
> --
> François

[Caml-list] mysterious cgi problem

[Eliot Handelman]

Hi list,

We have a demo coming up on Monday of an ocaml based music analysis and 
generation system with a
web component for music notation that, irritatingly, has stopped 
working. The interface is a CGI
with an ocaml 4.01 executable. After several harrowing days I narrowed 
the problem down to the
following situation:

X, a basic system, runs fine in firefox/chrome/various linuxen

X + module Y  generates "Not_found" exceptions in the CGI merely by 
linking Y. Y is
otherwise not invoked.  Moreover, I'm unable to trap the exception in 
the CGI itself
(ie, a try/with around the main function is somehow bypassed as the 
apache error long reports).  So
it would seem the that the code fails before the main function is executed.

We were using ocamlnet, but I rolled my own cgi parser and reproduced 
the problem.

Any suggestions? My next step would be to start rolling back the system, 
which is
not pleasant.


thanks,

-- eliot

Re: [Caml-list] mysterious cgi problem

[Siraaj Khandkar]

Not_found must be coming from some initialization procedure in module Y,
so try to trap it from there.


On 06/19/2014 11:41 PM, Eliot Handelman wrote:
> Hi list,
> 
> We have a demo coming up on Monday of an ocaml based music analysis and
> generation system with a
> web component for music notation that, irritatingly, has stopped
> working. The interface is a CGI
> with an ocaml 4.01 executable. After several harrowing days I narrowed
> the problem down to the
> following situation:
> 
> X, a basic system, runs fine in firefox/chrome/various linuxen
> 
> X + module Y  generates "Not_found" exceptions in the CGI merely by
> linking Y. Y is
> otherwise not invoked.  Moreover, I'm unable to trap the exception in
> the CGI itself
> (ie, a try/with around the main function is somehow bypassed as the
> apache error long reports).  So
> it would seem the that the code fails before the main function is executed.
> 
> We were using ocamlnet, but I rolled my own cgi parser and reproduced
> the problem.
> 
> Any suggestions? My next step would be to start rolling back the system,
> which is
> not pleasant.
> 
> 
> thanks,
> 
> -- eliot
>

Re: [Caml-list] mysterious cgi problem

[Mark Shinwell]

Is the OCaml executable compiled to native code?

If so, you may be able to trap this by having the HTTP server execute
gdb with commands to execute your OCaml program, and then give you a
backtrace immediately before it exits.  Ensure your program was
compiled with the -g option to ocamlopt.  Then for example if you
currently run:

/my/executable --foo-bar

you would instead run:

gdb -ex "b exit" -ex "r" -ex "bt" -ex "set confirm 0" -ex "q" --args
/my/executable --foo-bar

Mark

On 20 June 2014 06:48, Siraaj Khandkar <siraaj@khandkar.net> wrote:
> Not_found must be coming from some initialization procedure in module Y,
> so try to trap it from there.
>
>
> On 06/19/2014 11:41 PM, Eliot Handelman wrote:
>> Hi list,
>>
>> We have a demo coming up on Monday of an ocaml based music analysis and
>> generation system with a
>> web component for music notation that, irritatingly, has stopped
>> working. The interface is a CGI
>> with an ocaml 4.01 executable. After several harrowing days I narrowed
>> the problem down to the
>> following situation:
>>
>> X, a basic system, runs fine in firefox/chrome/various linuxen
>>
>> X + module Y  generates "Not_found" exceptions in the CGI merely by
>> linking Y. Y is
>> otherwise not invoked.  Moreover, I'm unable to trap the exception in
>> the CGI itself
>> (ie, a try/with around the main function is somehow bypassed as the
>> apache error long reports).  So
>> it would seem the that the code fails before the main function is executed.
>>
>> We were using ocamlnet, but I rolled my own cgi parser and reproduced
>> the problem.
>>
>> Any suggestions? My next step would be to start rolling back the system,
>> which is
>> not pleasant.
>>
>>
>> thanks,
>>
>> -- eliot
>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] mysterious cgi problem

[Eliot Handelman]

Thanks to those with suggestions. The problem was simply a deeply buried 
Sys.getenv with
the new exception-raising behavior, not expecting apache's environment.

- eliot