caml-list - the Caml user's mailing list
 help / color / mirror / Atom feed
* [Caml-list] The road to OCaml 5.0
@ 2021-10-07  9:48 Florian Angeletti
  2021-10-08  1:42 ` Francois Berenger
  2021-10-08 20:34 ` Christophe Raffalli
  0 siblings, 2 replies; 11+ messages in thread
From: Florian Angeletti @ 2021-10-07  9:48 UTC (permalink / raw)
  To: caml-list

[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]

With the convergence between the multicore and standard runtime across 
OCaml 4.10.0 to 4.13.0, the development of OCaml multicore has reached a point 
where further integration into OCaml's main branch requires fully committing to 
a switch to OCaml multicore. 

The OCaml team has decided that the time has come for such a commitment. 
The new major version, OCaml 5, will be a multicore version of OCaml. 
Moreover, OCaml 4.14 will be the last minor release of the 4.x series of OCaml. 


Multicore Minimum Viable Product (MVP) 
------------------------------------------------------ 

The first version of OCaml multicore, code-named OCaml 5.0, will be 
a Minimum Viable Product focused on: 

- x86-64 
- Linux, MacOS, Windows mingw-w64 
- Parallelism through Domains [1] 
- Concurrency through Effect Handlers [2] (without syntactic support and exposed as functions from the standard library) 

Our plan is to integrate the multicore branch into the main branch during the 
next 6 months. Hopefully, OCaml 5.0 will then be released between March and 
April 2022. 

Note that OCaml 5.0 focuses on minimal (solid) support for the multicore 
runtime system, and will not provide stable user-facing concurrency and 
parallelism libraries. There has been a lot of experimentation [3,4] in the last 
few years, and some work remains to offer long-term, user-facing concurrent and 
parallel programming abstractions. OCaml 5.0 will be a great time to start 
adding concurrency and parallelism to your OCaml programs, but the libraries 
will still be in flux. 


Long term support for OCaml 4.14 
---------------------------------------------- 

While OCaml 5 is stabilising, we plan to extend the support period for 
OCaml 4.14 by publishing minor bugfix releases whenever needed. In particular, 
OCaml 4.14 will be supported until all tier-1 architectures and operating systems 
are available in OCaml 5, and OCaml 5 sequential performance is close enough to 
that of OCaml 4. 


The sequential glaciation 
--------------------------------- 

To ensure that maintainers can concentrate on Multicore integration, and avoid 
any rebase work for the Multicore developers, the trunk branch will be 
feature-frozen starting from November 2021. All non-bugfix non-multicore 
contributions will be delayed to after the Multicore integration. 
We are calling this period the "sequential glaciation". 

We understand that this may be frustrating for our contributors, and apologize 
for the delay in getting your nice work reviewed and merged into the codebase. 
We hope that the sequential glaciation will be a good opportunity to help with 
the Multicore integration, review and testing, and/or focus on non-core-compiler 
efforts and the rest of the OCaml ecosystem. 

With this early feature-freeze, we also plan to release OCaml 4.14.0 in advance, 
between January-February 2022, reducing the concurrency between the OCaml 5.0 
and OCaml 4.14.0 releases. 


References 
--------------- 
[1] "Retrofitting Parallelism onto OCaml", ICFP 2020, https://arxiv.org/abs/2004.11663 
[2] "Retrofitting Concurrency onto OCaml", PLDI 2021, https://arxiv.org/abs/2104.00250 
[3] Domainslib -- Parallel Programming over Multicore OCaml, https://github.com/ocaml-multicore/domainslib 
[4] eio -- Effects-based Parallel IO for OCaml, https://github.com/ocaml-multicore/eio 


Happy hacking, 
Florian Angeletti, for the OCaml team. 


[-- Attachment #2: Type: text/html, Size: 3863 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] The road to OCaml 5.0
  2021-10-07  9:48 [Caml-list] The road to OCaml 5.0 Florian Angeletti
@ 2021-10-08  1:42 ` Francois Berenger
  2021-10-08 20:34 ` Christophe Raffalli
  1 sibling, 0 replies; 11+ messages in thread
From: Francois Berenger @ 2021-10-08  1:42 UTC (permalink / raw)
  To: Florian Angeletti; +Cc: caml-list

On 07/10/2021 18:48, Florian Angeletti wrote:
> With the convergence between the multicore and standard runtime across
> OCaml 4.10.0 to 4.13.0, the development of OCaml multicore has reached
> a point
> where further integration into OCaml's main branch requires fully
> committing to
> a switch to OCaml multicore.
> 
> The OCaml team has decided that the time has come for such a
> commitment.
> The new major version, OCaml 5, will be a multicore version of OCaml.
> Moreover, OCaml 4.14 will be the last minor release of the 4.x series
> of OCaml.
> 
> Multicore Minimum Viable Product (MVP)
> ------------------------------------------------------
> 
> The first version of OCaml multicore, code-named OCaml 5.0, will be
> a Minimum Viable Product focused on:
> 
> - x86-64
> - Linux, MacOS, Windows mingw-w64
> - Parallelism through Domains [1]
> - Concurrency through Effect Handlers [2] (without syntactic support
> and exposed as functions from the standard library)
> 
> Our plan is to integrate the multicore branch into the main branch
> during the
> next 6 months. Hopefully, OCaml 5.0 will then be released between
> March and
> April 2022.
> 
> Note that OCaml 5.0 focuses on minimal (solid) support for the
> multicore
> runtime system, and will not provide stable user-facing concurrency
> and
> parallelism libraries. There has been a lot of experimentation [3,4]
> in the last
> few years, and some work remains to offer long-term, user-facing
> concurrent and
> parallel programming abstractions. OCaml 5.0 will be a great time to
> start
> adding concurrency and parallelism to your OCaml programs, but the
> libraries
> will still be in flux.

The Parany library:
---
Generalized map reduce for multicore computers (unfold, map in parallel, 
fold).
Parany can process in parallel an "infinite" stream of elements (too big 
to fit in memory).
Any Parmap functionality can be reimplemented using parany.
---

already has a git branch which is relying on multicore-OCaml:

https://github.com/UnixJunkie/parany/tree/domains

When OCaml-5.0 ships, I will ship a parany version compatible with it.

I am quite sure Parmap could do the same, by the way (but I'm not the 
Parmap maintainer).

Regards,
F.

> Long term support for OCaml 4.14
> ----------------------------------------------
> 
> While OCaml 5 is stabilising, we plan to extend the support period for
> OCaml 4.14 by publishing minor bugfix releases whenever needed. In
> particular,
> OCaml 4.14 will be supported until all tier-1 architectures and
> operating systems
> are available in OCaml 5, and OCaml 5 sequential performance is close
> enough to
> that of OCaml 4.
> 
> The sequential glaciation
> ---------------------------------
> 
> To ensure that maintainers can concentrate on Multicore integration,
> and avoid
> any rebase work for the Multicore developers, the trunk branch will be
> feature-frozen starting from November 2021. All non-bugfix
> non-multicore
> contributions will be delayed to after the Multicore integration.
> We are calling this period the "sequential glaciation".
> 
> We understand that this may be frustrating for our contributors, and
> apologize
> for the delay in getting your nice work reviewed and merged into the
> codebase.
> We hope that the sequential glaciation will be a good opportunity to
> help with
> the Multicore integration, review and testing, and/or focus on
> non-core-compiler
> efforts and the rest of the OCaml ecosystem.
> 
> With this early feature-freeze, we also plan to release OCaml 4.14.0
> in advance,
> between January-February 2022, reducing the concurrency between the
> OCaml 5.0
> and OCaml 4.14.0 releases.
> 
> References
> ---------------
> [1] "Retrofitting Parallelism onto OCaml", ICFP 2020,
> https://arxiv.org/abs/2004.11663
> [2] "Retrofitting Concurrency onto OCaml", PLDI 2021,
> https://arxiv.org/abs/2104.00250
> [3] Domainslib -- Parallel Programming over Multicore OCaml,
> https://github.com/ocaml-multicore/domainslib
> [4] eio -- Effects-based Parallel IO for OCaml,
> https://github.com/ocaml-multicore/eio
> 
> Happy hacking,
> Florian Angeletti, for the OCaml team.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] The road to OCaml 5.0
  2021-10-07  9:48 [Caml-list] The road to OCaml 5.0 Florian Angeletti
  2021-10-08  1:42 ` Francois Berenger
@ 2021-10-08 20:34 ` Christophe Raffalli
  2021-10-09  0:07   ` [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0) Christophe Raffalli
  1 sibling, 1 reply; 11+ messages in thread
From: Christophe Raffalli @ 2021-10-08 20:34 UTC (permalink / raw)
  To: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 125 bytes --]


Hello dear camelers,

What is the recommanded way to test multicore, passibly via opam, using
flambda ?

Chears,
Christophe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-08 20:34 ` Christophe Raffalli
@ 2021-10-09  0:07   ` Christophe Raffalli
  2021-10-09  0:58     ` Michael Bacarella
  2021-10-09 17:14     ` ygrek
  0 siblings, 2 replies; 11+ messages in thread
From: Christophe Raffalli @ 2021-10-09  0:07 UTC (permalink / raw)
  To: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]


Hello,

I managed to install ocaml 4.12.0 with multicore. I could not parallelise my
code in 5mn ;-) but I check just the sequential speed and got a bit
surprised. On the same examples, same options (flambda everywhere), etc ...

                      Ex 1	Ex 2  	      Ex 3
4.13.1 normal	      45s	12s	      49s
4.12.0 normal	      36s	11s	      45s
4.12.0 multicore      31s	10s	      40s

These are not small differences and it is rather surprising that

4.13.1 is significantly slower than 4.12.0 (20 to 25%)

4.12.0 + multicore is faster on sequential code.

Other people observe the same ?
Any idea ? Should I report an issue for the speed degradation of 4.13.1 ?

Christophe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  0:07   ` [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0) Christophe Raffalli
@ 2021-10-09  0:58     ` Michael Bacarella
  2021-10-09  1:11       ` Michael Bacarella
  2021-10-09  2:10       ` Christophe Raffalli
  2021-10-09 17:14     ` ygrek
  1 sibling, 2 replies; 11+ messages in thread
From: Michael Bacarella @ 2021-10-09  0:58 UTC (permalink / raw)
  To: Christophe Raffalli; +Cc: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

My gut here says you're unwittingly comparing regular ocaml to flambda
ocaml.

https://ocaml.org/manual/flambda.html

Perhaps multicore only comes in flambda flavor now (I notice it's not
available as a switch).

On Fri, Oct 8, 2021 at 5:07 PM Christophe Raffalli <christophe@raffalli.eu>
wrote:

>
> Hello,
>
> I managed to install ocaml 4.12.0 with multicore. I could not parallelise
> my
> code in 5mn ;-) but I check just the sequential speed and got a bit
> surprised. On the same examples, same options (flambda everywhere), etc ...
>
>                       Ex 1      Ex 2          Ex 3
> 4.13.1 normal         45s       12s           49s
> 4.12.0 normal         36s       11s           45s
> 4.12.0 multicore      31s       10s           40s
>
> These are not small differences and it is rather surprising that
>
> 4.13.1 is significantly slower than 4.12.0 (20 to 25%)
>
> 4.12.0 + multicore is faster on sequential code.
>
> Other people observe the same ?
> Any idea ? Should I report an issue for the speed degradation of 4.13.1 ?
>
> Christophe
>

[-- Attachment #2: Type: text/html, Size: 1586 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  0:58     ` Michael Bacarella
@ 2021-10-09  1:11       ` Michael Bacarella
  2021-10-09  1:20         ` Michael Bacarella
  2021-10-09  2:10       ` Christophe Raffalli
  1 sibling, 1 reply; 11+ messages in thread
From: Michael Bacarella @ 2021-10-09  1:11 UTC (permalink / raw)
  To: Christophe Raffalli; +Cc: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

Sorry, you actually said as much "On the same examples, same options *(flambda
everywhere)*,"

Though, the expectation that was thwarted for me is that there's no
specific multicore flambda switch. Just multicore. And that 20-25% speedup
felt like a familiar we-switched-to-flambda speedup to me.

On Fri, Oct 8, 2021 at 5:58 PM Michael Bacarella <
michael.bacarella@gmail.com> wrote:

> My gut here says you're unwittingly comparing regular ocaml to flambda
> ocaml.
>
> https://ocaml.org/manual/flambda.html
>
> Perhaps multicore only comes in flambda flavor now (I notice it's not
> available as a switch).
>
> On Fri, Oct 8, 2021 at 5:07 PM Christophe Raffalli <christophe@raffalli.eu>
> wrote:
>
>>
>> Hello,
>>
>> I managed to install ocaml 4.12.0 with multicore. I could not parallelise
>> my
>> code in 5mn ;-) but I check just the sequential speed and got a bit
>> surprised. On the same examples, same options (flambda everywhere), etc
>> ...
>>
>>                       Ex 1      Ex 2          Ex 3
>> 4.13.1 normal         45s       12s           49s
>> 4.12.0 normal         36s       11s           45s
>> 4.12.0 multicore      31s       10s           40s
>>
>> These are not small differences and it is rather surprising that
>>
>> 4.13.1 is significantly slower than 4.12.0 (20 to 25%)
>>
>> 4.12.0 + multicore is faster on sequential code.
>>
>> Other people observe the same ?
>> Any idea ? Should I report an issue for the speed degradation of 4.13.1 ?
>>
>> Christophe
>>
>

[-- Attachment #2: Type: text/html, Size: 2349 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  1:11       ` Michael Bacarella
@ 2021-10-09  1:20         ` Michael Bacarella
  0 siblings, 0 replies; 11+ messages in thread
From: Michael Bacarella @ 2021-10-09  1:20 UTC (permalink / raw)
  To: Christophe Raffalli; +Cc: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 1894 bytes --]

Oh. Or I'm just out of date. Apparently the way you choose compilers
options (like flambda) in a switch changed awhile ago:

https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779

Sorry about the noise!

On Fri, Oct 8, 2021 at 6:11 PM Michael Bacarella <
michael.bacarella@gmail.com> wrote:

> Sorry, you actually said as much "On the same examples, same options *(flambda
> everywhere)*,"
>
> Though, the expectation that was thwarted for me is that there's no
> specific multicore flambda switch. Just multicore. And that 20-25% speedup
> felt like a familiar we-switched-to-flambda speedup to me.
>
> On Fri, Oct 8, 2021 at 5:58 PM Michael Bacarella <
> michael.bacarella@gmail.com> wrote:
>
>> My gut here says you're unwittingly comparing regular ocaml to flambda
>> ocaml.
>>
>> https://ocaml.org/manual/flambda.html
>>
>> Perhaps multicore only comes in flambda flavor now (I notice it's not
>> available as a switch).
>>
>> On Fri, Oct 8, 2021 at 5:07 PM Christophe Raffalli <
>> christophe@raffalli.eu> wrote:
>>
>>>
>>> Hello,
>>>
>>> I managed to install ocaml 4.12.0 with multicore. I could not
>>> parallelise my
>>> code in 5mn ;-) but I check just the sequential speed and got a bit
>>> surprised. On the same examples, same options (flambda everywhere), etc
>>> ...
>>>
>>>                       Ex 1      Ex 2          Ex 3
>>> 4.13.1 normal         45s       12s           49s
>>> 4.12.0 normal         36s       11s           45s
>>> 4.12.0 multicore      31s       10s           40s
>>>
>>> These are not small differences and it is rather surprising that
>>>
>>> 4.13.1 is significantly slower than 4.12.0 (20 to 25%)
>>>
>>> 4.12.0 + multicore is faster on sequential code.
>>>
>>> Other people observe the same ?
>>> Any idea ? Should I report an issue for the speed degradation of 4.13.1 ?
>>>
>>> Christophe
>>>
>>

[-- Attachment #2: Type: text/html, Size: 3181 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  0:58     ` Michael Bacarella
  2021-10-09  1:11       ` Michael Bacarella
@ 2021-10-09  2:10       ` Christophe Raffalli
  2021-10-09 17:20         ` Xavier Leroy
  1 sibling, 1 reply; 11+ messages in thread
From: Christophe Raffalli @ 2021-10-09  2:10 UTC (permalink / raw)
  To: Michael Bacarella; +Cc: Florian Angeletti, caml-list

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

On 21-10-08 17:58:00, Michael Bacarella wrote:
> My gut here says you're unwittingly comparing regular ocaml to flambda ocaml.
>
> https://ocaml.org/manual/flambda.html
>
> Perhaps multicore only comes in flambda flavor now (I notice it's not available
> as a switch).

No, I am using flambda on all versions.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  0:07   ` [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0) Christophe Raffalli
  2021-10-09  0:58     ` Michael Bacarella
@ 2021-10-09 17:14     ` ygrek
  1 sibling, 0 replies; 11+ messages in thread
From: ygrek @ 2021-10-09 17:14 UTC (permalink / raw)
  To: caml-list


 4.13 default GC changed (to best-fit), that might explain the difference.

-- 

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09  2:10       ` Christophe Raffalli
@ 2021-10-09 17:20         ` Xavier Leroy
  2021-10-10  3:37           ` Christophe Raffalli
  0 siblings, 1 reply; 11+ messages in thread
From: Xavier Leroy @ 2021-10-09 17:20 UTC (permalink / raw)
  To: Christophe Raffalli; +Cc: caml users

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

On Sat, Oct 9, 2021 at 4:11 AM Christophe Raffalli <christophe@raffalli.eu>
wrote:

> On 21-10-08 17:58:00, Michael Bacarella wrote:
> > My gut here says you're unwittingly comparing regular ocaml to flambda
> ocaml.
> >
> > https://ocaml.org/manual/flambda.html
> >
> > Perhaps multicore only comes in flambda flavor now (I notice it's not
> available
> > as a switch).
>
> No, I am using flambda on all versions.
>

Christophe: please put your benchmark online somewhere so that we can make
our measurements.  Without reproducibility, your initial message is
useless.  Thanks.

- Xavier Leroy

[-- Attachment #2: Type: text/html, Size: 1075 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0)
  2021-10-09 17:20         ` Xavier Leroy
@ 2021-10-10  3:37           ` Christophe Raffalli
  0 siblings, 0 replies; 11+ messages in thread
From: Christophe Raffalli @ 2021-10-10  3:37 UTC (permalink / raw)
  To: Xavier Leroy; +Cc: caml users

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

Hello,

First I have cleaned the inlining instruction and get better result leting
flambda deciding alone. This leads to

36s with 4.13.1 sequential
36s with 4.12.0 sequential
32s with 4.12.0 multicore (but not using any parralelism)

So being more reasonable with inlining annotation yields no more
changes between 4.13.1 and 4.12.0.

Still I am surprised that multicore is better (should expect a bit worth ?)

>
> Christophe: please put your benchmark online somewhere so that we can make our
> measurements.  Without reproducibility, your initial message is useless. 
> Thanks.

The code is there (it uses functors a lot, could be a good bench for flambda):

https://github.com/craff/hypersurfaces.git

and depends from (opam version might not be sufficient)

https://github.com/craff/gles3.git
https://github.com/craff/pacomb.git

and other opam packages.

If you "opam pin ." the three git repos above everything should work.

The command to test:

dune exec --release Hyper -- tests/hard/hilbert-sextic.txt -b

It prints its own timing as below among other messages:
   total:   36.329s (   0.012s self) for      1 call(s)

You can use other files in tests or tests/hard.

> - Xavier Leroy
>
H

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2021-10-10  3:38 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-07  9:48 [Caml-list] The road to OCaml 5.0 Florian Angeletti
2021-10-08  1:42 ` Francois Berenger
2021-10-08 20:34 ` Christophe Raffalli
2021-10-09  0:07   ` [Caml-list] Sequential speed 4.12.0 vs 4.13.1 vs multicore (Was: The road to OCaml 5.0) Christophe Raffalli
2021-10-09  0:58     ` Michael Bacarella
2021-10-09  1:11       ` Michael Bacarella
2021-10-09  1:20         ` Michael Bacarella
2021-10-09  2:10       ` Christophe Raffalli
2021-10-09 17:20         ` Xavier Leroy
2021-10-10  3:37           ` Christophe Raffalli
2021-10-09 17:14     ` ygrek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).