From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HTML_MESSAGE,MAILING_LIST_MULTI,T_KAM_HTML_FONT_INVALID autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 20829 invoked from network); 3 Sep 2021 19:02:40 -0000 Received: from minnie.tuhs.org (45.79.103.53) by inbox.vuxu.org with ESMTPUTF8; 3 Sep 2021 19:02:40 -0000 Received: by minnie.tuhs.org (Postfix, from userid 112) id CC24E9C8B6; Sat, 4 Sep 2021 05:02:37 +1000 (AEST) Received: from minnie.tuhs.org (localhost [127.0.0.1]) by minnie.tuhs.org (Postfix) with ESMTP id DDC5D9C870; Sat, 4 Sep 2021 05:02:06 +1000 (AEST) Authentication-Results: minnie.tuhs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=serissa.com header.i=@serissa.com header.b="aRil88d0"; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.b="b0xoLSDK"; dkim-atps=neutral Received: by minnie.tuhs.org (Postfix, from userid 112) id AD0C99C870; Sat, 4 Sep 2021 05:02:04 +1000 (AEST) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by minnie.tuhs.org (Postfix) with ESMTPS id 120239BA1E for ; Sat, 4 Sep 2021 05:02:03 +1000 (AEST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id D503C5C0170; Fri, 3 Sep 2021 15:02:01 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Fri, 03 Sep 2021 15:02:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=serissa.com; h= from:message-id:content-type:mime-version:subject:date :in-reply-to:cc:to:references; s=fm3; bh=9FWJFIsyu1CQPyoEkeeFnmB TasX0S7ynI9JHJhivi1I=; b=aRil88d0iNuz6cHpQUzusgGteMq6Gmuio29/H9t U8qBrjX7YWczcEkpxYHyjBNysJi7jtDo/hNUg+u5KHwlWs4uUlem5YtNQtzr7b2N Lmyw4oTHtfjDCQpaKA3sF7LpFWobVNyehMUtBZ51iNvjqOsim4QdxLc/kbgryP4l TSL11QuSpfm9fbHchnFNLxnW48HE6eYhYAUQUE/bsmSvCApBszKySClDdor9Rql9 AHlns7UCZ5lT6lPXIV2b6ecmd15sFZmy1yHjICKCCaGkb9imMap1ws9GVdk7n79g 6EOx5k1ioqIyYIO+ihtjQrq1bfn1HihebIIvcv6bk3seXdg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=9FWJFI syu1CQPyoEkeeFnmBTasX0S7ynI9JHJhivi1I=; b=b0xoLSDK+gWBTfTHKWxpLZ yjmR+cYZWYkjNWIjfRwDRLU5ticVmQhJzekQNNxOJ75/gpGyy6H0X7GagmM72XuP LnaG2aeVPDpqIaRxpc1oGq8/5O4SpLysMAlRoFD/nQ0HyJiGEWmIvCNEQE7HD8qP Gom8yNw8BinGeyHupdvsiDwB2SbxemNrSBFUWQtRTnlMtd0DmVHxDtAqsFTCDZ6K pdx9LKhujnUViJPU4xVT7JyFUlrhI3eFCK/yMSoMJMxzqjRsJKPAd6X2ihvuGmp9 BBHtPgDpUNcFKzE1a9WR+xW8vheHLTDJkGA9vAFHebSHD577KNJSe4mzAJOWfCIw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvjedgudeftdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffktgggufffjgfvfhfosegrtd hmrehhtdejnecuhfhrohhmpefnrgifrhgvnhgtvgcuufhtvgifrghrthcuoehsthgvfigr rhhtsehsvghrihhsshgrrdgtohhmqeenucggtffrrghtthgvrhhnpefgtddtudejgfeiff dthffhgeejfeekkeffffetvefgvdfhgffghfdtjeeludehhfenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvfigrrhhtsehsvghrihhssh grrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 3 Sep 2021 15:02:01 -0400 (EDT) From: Lawrence Stewart Message-Id: <7CE6F620-FABD-4C2C-905A-02919FB0BCD5@serissa.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_FAA2CFFE-3F2D-41D2-8B2C-CB1E94745AC3" Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Date: Fri, 3 Sep 2021 15:02:00 -0400 In-Reply-To: <20210903172848.GF13471@mcvoy.com> To: The Eunuchs Hysterical Society References: <20210903172848.GF13471@mcvoy.com> X-Mailer: Apple Mail (2.3654.120.0.1.13) Subject: Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) X-BeenThere: tuhs@minnie.tuhs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: The Unix Heritage Society mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: tuhs-bounces@minnie.tuhs.org Sender: "TUHS" --Apple-Mail=_FAA2CFFE-3F2D-41D2-8B2C-CB1E94745AC3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I don=E2=80=99t really think out-of-order in hardware causes trouble for = programmers that wasn=E2=80=99t already there when you use -O3. = Compilers will already promote memory to registers and do = interprocedural optimization and reorder memory references. You have to = sprinkle=20 asm volatile("" ::: "memory"); around like pixie dust to make sure the compiler does things in the = order you want, nevermind the hardware. x86 has wildly complex microarchitecture, but the model for a single = thread is perfectly sensible. It seems to work like it was in-order. = OOO isn=E2=80=99t changing that model of execution at all. I mean you = care about it when performance tuning, but not for correctness. Other architectures, ARM, IBM, Alpha, are far worse in this respect. The real problems are when you have multicore in a shared memory space = and you have to read all the fine print in the memory ordering chapter = and understand the subtle differences between LFENCE and SFENCE and = MFENCE. I get that, but I also think shared memory is a failed = experiment and we should have gone with distributed memory and clusters = and message passing. It is possible for mortals to program those. = Regular folks can make 1000 rank MPI programs work, where almost noone = other than Maurice Herlihy or like that can reason about locks and = threads and shared data structures. My wish for a better world is to integrate messaging into the = architecture rather than have an I/O device model for communications. = It is criminal that machine to machine comms are still stuck at 800 = nanoseconds or so latency. It takes 200 instructions or so to send a = message under the best circumstances and a similar number to receive it, = plus bus, adapter, wire, and switch time.=20 -L --Apple-Mail=_FAA2CFFE-3F2D-41D2-8B2C-CB1E94745AC3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 I don=E2=80=99t = really think out-of-order in hardware causes trouble for programmers = that wasn=E2=80=99t already there when you use -O3.  Compilers will = already promote memory to registers and do interprocedural optimization = and reorder memory references.  You have to sprinkle 
asm volatile("" ::: "memory");
around like pixie dust to make sure the compiler does things = in the order you want, nevermind the hardware.

x86 has wildly complex = microarchitecture, but the model for a single thread is perfectly = sensible.  It seems to work like it was in-order. OOO isn=E2=80=99t = changing that model of execution at all.  I mean you care about it = when performance tuning, but not for correctness.

Other architectures, ARM, IBM, Alpha, = are far worse in this respect.

The real problems are when you have multicore in a shared = memory space and you have to read all the fine print in the memory = ordering chapter and understand the subtle differences between LFENCE = and SFENCE and MFENCE.  I get that, but I also think shared memory = is a failed experiment and we should have gone with distributed memory = and clusters and message passing.  It is possible for mortals to = program those.    Regular folks can make 1000 rank MPI = programs work, where almost noone other than Maurice Herlihy or like = that can reason about locks and threads and shared data = structures.

My = wish for a better world is to integrate messaging into the architecture = rather than have an I/O device model for communications.  It is = criminal that machine to machine comms are still stuck at 800 = nanoseconds or so latency.  It takes 200 instructions or so to send = a message under the best circumstances and a similar number to receive = it, plus bus, adapter, wire, and switch time. 

-L



= --Apple-Mail=_FAA2CFFE-3F2D-41D2-8B2C-CB1E94745AC3--