[Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Xavier Leroy]

Dear OCaml users,

It was recently made public that Intel processors of the Skylake and Kaby Lake families have a nasty hardware bug concerning hyperthreading.  See, for example
https://arstechnica.com/information-technology/2017/06/skylake-kaby-lake-chips-have-a-crash-bug-with-hyperthreading-enabled/

I wanted to draw the attention of all OCaml users to this issue, because this hardware bug is triggered by the OCaml runtime under fairly common conditions.  Random crashes caused by this bug were observed at Ahrefs and another industrial user of OCaml, and also by users of Coccinelle and of Coq.  (*)

Your OCaml installation may be affected if the following three conditions are met:

1- You're using an Intel processor of the Skylake or Kaby Lake family.
   (AMD processors and older Intel processors are not affected.)
   
2- The processor support hyperthreading (HT) and hyperthreading is enabled in the BIOS.

3- The OCaml runtime system was compiled with certain versions of GCC.
   (We don't know exactly which versions of GCC triggers the issue, but
    it seems that it is never triggered by clang-generated code.
    In particular, this means that Mac OS X users should be safe.)

See https://lists.debian.org/debian-devel/2017/06/msg00308.html for help in determining if your processor is vulnerable, and for instructions on how to address the issue.  Possible fixes include:

- Apply the microcode update recently released by Intel for Skylake processors.  (No such update exists yet for Kaby Lake processors.)
- Disable hyperthreading in the BIOS.
- Reinstall OCaml using clang instead of GCC.

Kind regards,

- Xavier Leroy

(*) It is likely that we OCaml folks were the first to pinpoint this Intel bug.  For more information, see the narratives at
https://caml.inria.fr/mantis/view.php?id=7452
https://medium.com/ahrefs/skylake-bug-a-detective-story-ab1ad2beddcd

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Xavier Leroy]

Clarification:

> Possible fixes include:
> 
> - Apply the microcode update recently released by Intel [...]

Contradicting what I wrote before, microcode updates are available both for Skylake and for Kaby Lake processors.  It's only how to apply those updates that differs slightly between Skylake and Kaby Lake.  Again, see the excellent instructions at
https://lists.debian.org/debian-devel/2017/06/msg00308.html

Sorry for the confusion.

- Xavier Leroy

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Gabriel Scherer]

Here is my list of affected OCaml projects known so far (it seems
surprisingly less rare than I thought at first):

- an (unnamed) industrial user noticed the issue in Spring 2016, as
reported by Xavier on the bugtracker:
  https://caml.inria.fr/mantis/view.php?id=7452#c17109

- ahrefs noticed issues in late 2016, as reported by Joris
Giovannangeli on the bugtracker issue or his longer piece
  https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd

- Lwt users were also affected (first report from Sebastien Mondet in
February 2017):
  https://github.com/ocsigen/lwt/issues/315

- Julia Lawall reported segfaults from a downstream user on the caml-list,
  that seem extreley likely to be related to this issue (they
disappear with hyperthreading disabled)
  https://sympa.inria.fr/sympa/arc/caml-list/2017-06/msg00115.html

At this point it seems reasonable to assume that any
computation-intensive OCaml program that is run in parallel may be
affected.

On Fri, Jun 30, 2017 at 10:15 AM, Xavier Leroy <xavier.leroy@inria.fr> wrote:
> Clarification:
>
>> Possible fixes include:
>>
>> - Apply the microcode update recently released by Intel [...]
>
> Contradicting what I wrote before, microcode updates are available both for Skylake and for Kaby Lake processors.  It's only how to apply those updates that differs slightly between Skylake and Kaby Lake.  Again, see the excellent instructions at
> https://lists.debian.org/debian-devel/2017/06/msg00308.html
>
> Sorry for the confusion.
>
> - Xavier Leroy
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Emilio Jesús Gallego Arias]

Gabriel Scherer <gabriel.scherer@gmail.com> writes:

> Here is my list of affected OCaml projects known so far (it seems
> surprisingly less rare than I thought at first):
>
> - an (unnamed) industrial user noticed the issue in Spring 2016, as
> reported by Xavier on the bugtracker:
>   https://caml.inria.fr/mantis/view.php?id=7452#c17109
>
> - ahrefs noticed issues in late 2016, as reported by Joris
> Giovannangeli on the bugtracker issue or his longer piece
>   https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd
>
> - Lwt users were also affected (first report from Sebastien Mondet in
> February 2017):
>   https://github.com/ocsigen/lwt/issues/315
>
> - Julia Lawall reported segfaults from a downstream user on the caml-list,
>   that seem extreley likely to be related to this issue (they
> disappear with hyperthreading disabled)
>   https://sympa.inria.fr/sympa/arc/caml-list/2017-06/msg00115.html

Here is the only Coq bug that I know of:

https://coq.inria.fr/bugs/show_bug.cgi?id=5517

but as Maxime pointed out, the problem was observed by other people
internally.

E.

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Julia Lawall]

On Fri, 30 Jun 2017, Gabriel Scherer wrote:

> Here is my list of affected OCaml projects known so far (it seems
> surprisingly less rare than I thought at first):
>
> - an (unnamed) industrial user noticed the issue in Spring 2016, as
> reported by Xavier on the bugtracker:
>   https://caml.inria.fr/mantis/view.php?id=7452#c17109
>
> - ahrefs noticed issues in late 2016, as reported by Joris
> Giovannangeli on the bugtracker issue or his longer piece
>   https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd
>
> - Lwt users were also affected (first report from Sebastien Mondet in
> February 2017):
>   https://github.com/ocsigen/lwt/issues/315
>
> - Julia Lawall reported segfaults from a downstream user on the caml-list,
>   that seem extreley likely to be related to this issue (they
> disappear with hyperthreading disabled)
>   https://sympa.inria.fr/sympa/arc/caml-list/2017-06/msg00115.html
>
> At this point it seems reasonable to assume that any
> computation-intensive OCaml program that is run in parallel may be
> affected.

The Coccinelle code is compiled with Parmap, but I don't think that Parmap
is invoked in the failing execution.  I could check on this.

julia


>
> On Fri, Jun 30, 2017 at 10:15 AM, Xavier Leroy <xavier.leroy@inria.fr> wrote:
> > Clarification:
> >
> >> Possible fixes include:
> >>
> >> - Apply the microcode update recently released by Intel [...]
> >
> > Contradicting what I wrote before, microcode updates are available both for Skylake and for Kaby Lake processors.  It's only how to apply those updates that differs slightly between Skylake and Kaby Lake.  Again, see the excellent instructions at
> > https://lists.debian.org/debian-devel/2017/06/msg00308.html
> >
> > Sorry for the confusion.
> >
> > - Xavier Leroy
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Anton Bachin]

[-- Attachment #1: Type: text/plain, Size: 3591 bytes --]

In what I’ve read about the bug, there is no requirement that two virtual cores be running OCaml code (or rather, the malfunctioning instruction sequence). Is that accurate? If so, it shouldn’t matter whether the OCaml code is parallelized or not.

As far as I know, the users reporting in Lwt were not running Lwt or OCaml in parallel.

> El jun. 30, 2017, a las 11:30, Julia Lawall <julia.lawall@lip6.fr <mailto:julia.lawall@lip6.fr>> escribió:
> 
> 
> 
> On Fri, 30 Jun 2017, Gabriel Scherer wrote:
> 
>> Here is my list of affected OCaml projects known so far (it seems
>> surprisingly less rare than I thought at first):
>> 
>> - an (unnamed) industrial user noticed the issue in Spring 2016, as
>> reported by Xavier on the bugtracker:
>>  https://caml.inria.fr/mantis/view.php?id=7452#c17109 <https://caml.inria.fr/mantis/view.php?id=7452#c17109>
>> 
>> - ahrefs noticed issues in late 2016, as reported by Joris
>> Giovannangeli on the bugtracker issue or his longer piece
>>  https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd <https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd>
>> 
>> - Lwt users were also affected (first report from Sebastien Mondet in
>> February 2017):
>>  https://github.com/ocsigen/lwt/issues/315 <https://github.com/ocsigen/lwt/issues/315>
>> 
>> - Julia Lawall reported segfaults from a downstream user on the caml-list,
>>  that seem extreley likely to be related to this issue (they
>> disappear with hyperthreading disabled)
>>  https://sympa.inria.fr/sympa/arc/caml-list/2017-06/msg00115.html <https://sympa.inria.fr/sympa/arc/caml-list/2017-06/msg00115.html>
>> 
>> At this point it seems reasonable to assume that any
>> computation-intensive OCaml program that is run in parallel may be
>> affected.
> 
> The Coccinelle code is compiled with Parmap, but I don't think that Parmap
> is invoked in the failing execution.  I could check on this.
> 
> julia
> 
> 
>> 
>> On Fri, Jun 30, 2017 at 10:15 AM, Xavier Leroy <xavier.leroy@inria.fr <mailto:xavier.leroy@inria.fr>> wrote:
>>> Clarification:
>>> 
>>>> Possible fixes include:
>>>> 
>>>> - Apply the microcode update recently released by Intel [...]
>>> 
>>> Contradicting what I wrote before, microcode updates are available both for Skylake and for Kaby Lake processors.  It's only how to apply those updates that differs slightly between Skylake and Kaby Lake. Again, see the excellent instructions at
>>> https://lists.debian.org/debian-devel/2017/06/msg00308.html <https://lists.debian.org/debian-devel/2017/06/msg00308.html>
>>> 
>>> Sorry for the confusion.
>>> 
>>> - Xavier Leroy
>>> 
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa.inria.fr/sympa/arc/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>> 
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list <https://sympa.inria.fr/sympa/arc/caml-list>
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>> 
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list <https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners <http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-bugs <http://caml.inria.fr/bin/caml-bugs>

[-- Attachment #2: Type: text/html, Size: 14231 bytes --]

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Xavier Leroy]

On 06/30/2017 06:46 PM, Anton Bachin wrote:
> In what I’ve read about the bug, there is no requirement that two virtual cores be running OCaml code (or rather, the malfunctioning instruction sequence). Is that accurate?

That's my guess, too, based on Intel's cryptic description of the issue.  ("Complex set of micro-architectural conditions", you don't say!)  However, the two repro cases I worked with (the one from Ahrefs and the earlier one) were both of the "running several copies of an OCaml program in parallel" kind.

- Xavier Leroy

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Johannes Kanig]

On Sat, 2017-07-01, 0:17:26, Gabriel Scherer <gabriel.scherer@gmail.com>  
wrote:

> Here is my list of affected OCaml projects known so far (it seems
> surprisingly less rare than I thought at first):

To add to the list, we at AdaCore have seen random crashes of long-running  
Why3 processes, that only happen on the Skylake machine of one of our  
developers.

-- 
Johannes Kanig, PhD
Senior Software Engineer
<kanig@adacore.com>

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Mark Shinwell]

The problem with Kaby Lake systems is that the microcode update is
currently only available via a BIOS update rather than, for example, a
package in a typical Linux distribution (e.g. intel-microcode on
Debian).  As such you may find at present that an update is not
available; in these cases you should disable hyperthreading.

Mark

On 30 June 2017 at 15:15, Xavier Leroy <xavier.leroy@inria.fr> wrote:
> Clarification:
>
>> Possible fixes include:
>>
>> - Apply the microcode update recently released by Intel [...]
>
> Contradicting what I wrote before, microcode updates are available both for Skylake and for Kaby Lake processors.  It's only how to apply those updates that differs slightly between Skylake and Kaby Lake.  Again, see the excellent instructions at
> https://lists.debian.org/debian-devel/2017/06/msg00308.html
>
> Sorry for the confusion.
>
> - Xavier Leroy
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Maxime Dénès]

Hello,

On 06/30/2017 03:51 PM, Xavier Leroy wrote:
> Random crashes caused by this bug were observed at Ahrefs and another industrial user of OCaml, and also by users of Coccinelle and of Coq.

Just to emphasize the importance of this problem, we observed that the 
compilation of Coq on impacted machines would typically fail (segfault) 
with probably close to 1.

Kudos for tracking it down.

Maxime.

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Anton Bachin]

> On 06/30/2017 03:51 PM, Xavier Leroy wrote
>> Random crashes caused by this bug were observed at Ahrefs and another industrial user of OCaml, and also by users of Coccinelle and of Coq.
> 
> Just to emphasize the importance of this problem, we observed that the compilation of Coq on impacted machines would typically fail (segfault) with probably close to 1.


To further emphasize the importance of this problem, putting an Lwt program under full load in a stress test triggered segfaults with probability 1, usually within a few minutes, and within at most 30, for the users observing it.

And thanks from me also for tracking it down.

Anton

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[David MENTRÉ]

Hello,

Le 2017-06-30 à 15:51, Xavier Leroy a écrit :
> - Reinstall OCaml using clang instead of GCC.

Is there an easy way to that with opam, assuming opam is already
installed with a set of packages?

I looked in the OPAM's FAQ but found nothing up to now.

Best regards,
david

Re: [Caml-list] [IMPORTANT] Intel Skylake / Kaby Lake hardware bug affects OCaml programs

[Gabriel Scherer]

You can install a new switch from an arbitrary OCaml distribution
source (with arbitrary configure options) using my convenience script:
  https://github.com/gasche/opam-compiler-conf

(You could also set up a local opam repository, and create a new
switch there with extra configure options. This could be nicer if you
plan to apply the changes to several different machines.)

Then you can replicate the installed packages of an other switch by
using "opam switch {export,import}":
  https://opam.ocaml.org/doc/FAQ.html#CanIgetanewswitchwiththesamepackagesinstalled

On Fri, Jul 21, 2017 at 3:00 PM, David MENTRÉ <dmentre@linux-france.org> wrote:
> Hello,
>
> Le 2017-06-30 à 15:51, Xavier Leroy a écrit :
>> - Reinstall OCaml using clang instead of GCC.
>
> Is there an easy way to that with opam, assuming opam is already
> installed with a set of packages?
>
> I looked in the OPAM's FAQ but found nothing up to now.
>
> Best regards,
> david
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs