From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: from minnie.tuhs.org (minnie.tuhs.org [50.116.15.146]) by inbox.vuxu.org (Postfix) with ESMTP id F0FF121148 for ; Tue, 14 May 2024 09:11:05 +0200 (CEST) Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id 00692432FD; Tue, 14 May 2024 17:10:59 +1000 (AEST) Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by minnie.tuhs.org (Postfix) with ESMTPS id 41605432F9 for ; Tue, 14 May 2024 17:10:51 +1000 (AEST) Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1ecd9a81966so43561845ad.0 for ; Tue, 14 May 2024 00:10:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715670650; x=1716275450; darn=tuhs.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ThuRgQQb8aw5ZE6oh505IQlQXtE4KLC0XuJ5kOr3Go4=; b=bWBPuDOJ8ikv1BkV2hQXbj4PST6HlHCBb9DCNC2T35MLY9XZDY4g68WYU/fkuM+Jj8 86vb1uKYXyj4ZQ9noX8hIquV1RFHttjisIG+vhgKkDcdo9goxhWi/m2TSrP2L9wO4y0z ggOqEm5wPZyXRZJ54PFzuq8m6OBtl/aqhByS2A30cB8t0WZNtCWa8fh0dPijN8runAoy 8nxZnE9uEAcAcvugHmHWB+aDT6Wbvi4G/xCBylT5ENnMxbZPOD/uPoJMN4Eh4Cxdz3Jx iYjRXODyHsNK3P2AtTKeJJlpHOJ8/LXwnFUyq4weg36w9Sg/ga431TEb/9hKGnajV8Iy IHkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715670650; x=1716275450; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ThuRgQQb8aw5ZE6oh505IQlQXtE4KLC0XuJ5kOr3Go4=; b=bLulQtTbzWv+WB2knUH+3VZu2yFLSYlp/+cJbMvQoHwoAy8x0028PZhHqLyZ/DSY/W sSUzFJdasuIfxGeLWXwQcaM/iSwPNyGo3Q5FtJC5DqkORaLH7yZ9WgvbsOtS4WPt4te1 QUPWhG8/7G/Jc4l7d/hfP2tadgO7RJZNMPsMNmGAvQcFZSF36fvK4GZLEeRbF5+dM8xc iL8J4xzgQ9leaJ9MnbXxqsw1HO2KMYDTgO39QqvbqIcLHBmX9PY/4iU2UwZp9/NtV03L OcGqfK4spKuxfcL03GwUSZaK0rj6xHZdf2AE4+9xQchS1Z60scuTjNiALhBf8MMhu6n1 VOww== X-Gm-Message-State: AOJu0YzZIoWxLeMMThLGF/RzDqKmkgpcwtw335pcCOFKzeWbOQX2n20T TokR46hvL2XG1wXMaXgTxP6ivLOWiBAmEbiWTEMm7yVGKUa/Gx+2sl1tLU7NVTC02q+m41Qzl3e dCsocFKQ/4xINEXHw45O8Xt83PLg= X-Google-Smtp-Source: AGHT+IHaZCPu2eoC8dQwPq6x9UnaJ7cnaDi+BKHnOjFjpWL0k/+rW7Ir3b5gb/+zhLTNWTBkIF27a8ggvPt7q+Kkahg= X-Received: by 2002:a17:90b:2351:b0:2b2:9783:d0ca with SMTP id 98e67ed59e1d1-2b6c71100b2mr17410514a91.12.1715670650082; Tue, 14 May 2024 00:10:50 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Rob Pike Date: Tue, 14 May 2024 17:10:38 +1000 Message-ID: To: Douglas McIlroy Content-Type: multipart/alternative; boundary="000000000000a65fe0061864b3d4" Message-ID-Hash: ESOVO2IDY744PR67XD2LROUCL543KKVZ X-Message-ID-Hash: ESOVO2IDY744PR67XD2LROUCL543KKVZ X-MailFrom: robpike@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: TUHS main list X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: If forking is bad, how about buffering? List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --000000000000a65fe0061864b3d4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I agree with your (as usual) perceptive analysis. Only stopping by to point out that I took the buffering out of cat. I didn't have your perspicacity on why it should happen, just a desire to remove all the damn flags. When I was done, cat.c was 35 lines long. Do a read, do a write, continue until EOF. Guess what? That's all you need if you want to cat files. Sad to say Bell Labs's cat door was hard to open and most of the world still has a cat with flags. And buffers. -rob On Mon, May 13, 2024 at 11:35=E2=80=AFPM Douglas McIlroy < douglas.mcilroy@dartmouth.edu> wrote: > So fork() is a significant nuisance. How about the far more ubiquitous > problem of IO buffering? > > On Sun, May 12, 2024 at 12:34:20PM -0700, Adam Thornton wrote: > > But it does come down to the same argument as > > > https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos1= 9.pdf > > The Microsoft manifesto says that fork() is an evil hack. One of the cite= d > evils is that one must remember to flush output buffers before forking, f= or > fear it will be emitted twice. But buffering is the culprit, not the > victim. Output buffers must be flushed for many other reasons: to avoid > deadlock; to force prompt delivery of urgent output; to keep output from > being lost in case of a subsequent failure. Input buffers can also steal > data by reading ahead into stuff that should go to another consumer. In a= ll > these cases buffering can break compositionality. Yet the manifesto blame= s > an instance of the hazard on fork()! > > To assure compositionality, one must flush output buffers at every > possible point where an unknown downstream consumer might correctly act o= n > the received data with observable results. And input buffering must never > ingest data that the program will not eventually use. These are tough > criteria to meet in general without sacrificing buffering. > > The advent of pipes vividly exposed the non-compositionality of output > buffering. Interactive pipelines froze when users could not provide input > that would force stuff to be flushed until the input was informed by that > very stuff. This phenomenon motivated cat -u, and stdio's convention of > line buffering for stdout. The premier example of input buffering eating > other programs' data was mitigated by "here documents" in the Bourne shel= l. > > These precautions are mere fig leaves that conceal important special > cases. The underlying evil of buffered IO still lurks. The justification = is > that it's necessary to match the characteristics of IO devices and to > minimize system-call overhead. The former necessity requires the attenti= on > of hardware designers, but the latter is in the hands of programmers. Wha= t > can be done to mitigate the pain of border-crossing into the kernel? L4 a= nd > its ilk have taken a whack. An even more radical approach might flow from > the "whitepaper" at www.codevalley.com. > > In any even the abolition of buffering is a grand challenge. > > Doug > --000000000000a65fe0061864b3d4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I agree with your (as usual) perceptive analysis. Only stopping b= y to point out that I took the buffering out of cat. I didn't have your= perspicacity on why it should happen, just a desire to remove all the damn= flags. When I was done, cat.c was 35 lines long. Do a read, do a write, co= ntinue until EOF. Guess what? That's all you need if you want to cat fi= les.

Sad to say Bell Labs's cat door was hard to open and most of the = world still has a cat with flags. And buffers.

-rob


On Mon, May 13= , 2024 at 11:35=E2=80=AFPM Douglas McIlroy <douglas.mcilroy@dartmouth.edu> wrote:
=
So fork() is a significant nuisance. How about the far= more ubiquitous problem of IO buffering?

On Sun, May 12, 2024 at 12:34:20PM -0700, Adam Tho= rnton wrote:
> But it does come down to the same argument as
>=C2=A0https://www.micros= oft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf
<= div dir=3D"ltr">
The Microsoft manifesto says t= hat fork() is an evil hack. One of the cited evils is that one must remembe= r to flush output buffers before forking, for fear it will be emitted twice= . But buffering is the culprit, not the victim. Output buffers must be flus= hed for many other reasons: to avoid deadlock; to force prompt delivery of = urgent output; to keep output from being lost in case of a subsequent failu= re. Input buffers can also steal data by reading ahead into stuff that shou= ld go to another consumer. In all these cases buffering can break compositi= onality. Yet the manifesto blames an instance of the hazard on fork()!=C2= =A0

To assure compositionality, one must= flush output buffers at every possible point where an unknown downstream c= onsumer might correctly act on the received data with observable results. A= nd input buffering must never ingest data that the program will not eventua= lly use. These are tough criteria to meet in general without sacrificing bu= ffering.
The adve= nt of pipes vividly exposed the non-compositionality of output buffering. I= nteractive pipelines froze when users could not provide input that would fo= rce stuff to be flushed until the input was informed by that very stuff. Th= is phenomenon motivated cat -u, and stdio's convention of line bufferin= g for stdout. The premier example of input buffering eating other programs&= #39; data was mitigated by "here documents" in the Bourne shell.<= /span>

=
These precautions= are mere fig leaves that conceal important special cases. The underlying e= vil of buffered IO still lurks. The justification is that it's necessar= y to match the characteristics of IO devices and to minimize system-call ov= erhead.=C2=A0 The former necessity requires the attention of hardware desig= ners, but the latter is in the hands of programmers. What can be done to mi= tigate the pain of border-crossing into the kernel? L4 and its ilk have tak= en a whack. An even more radical approach might flow from the "whitepa= per" at www.co= devalley.com.

In any eve= n the abolition of buffering is a grand challenge.

Doug
--000000000000a65fe0061864b3d4--