From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: from minnie.tuhs.org (minnie.tuhs.org [IPv6:2600:3c01:e000:146::1]) by inbox.vuxu.org (Postfix) with ESMTP id C1B9530872 for ; Fri, 6 Dec 2024 09:16:53 +0100 (CET) Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id 10C7C42761; Fri, 6 Dec 2024 18:16:47 +1000 (AEST) Received: from vmail-relay.servers.aueb.gr (relay3.servers.aueb.gr [195.251.255.93]) by minnie.tuhs.org (Postfix) with ESMTP id 2C4FA42760 for ; Fri, 6 Dec 2024 18:16:41 +1000 (AEST) Received: from vmail-mta.servers.aueb.gr (localhost [127.0.0.1]) by vmail-relay.servers.aueb.gr (Postfix) with ESMTP id 4A509D80AD7; Fri, 6 Dec 2024 10:16:39 +0200 (EET) ARC-Filter: OpenARC Filter v0.1.0 vmail-relay.servers.aueb.gr 4A509D80AD7 ARC-Seal: i=1; a=rsa-sha256; d=aueb.gr; s=arc201901; t=1733472999; cv=none; b=BLmn6VAe29uSMwIMpCKb2mRAbTbVYv+o5LKlJQKfBf3OEAug4P1j14nU4A5pKDY0c31oDoq95Ui+XIGZoalgn3t/mAY1a6QA3xM9YkPj5qtsbTTFcU8Xdjp9iWdkrcn/tDtBoYmQrxPYu+83NDxWqhKSdbobqKd0bLJeyDWnX2uEu4j1pfjwlc+ltaFdx7bPeHtyr3t4y05pnoybfdqEP1Dtvtech4tx/BMk4rgDTA+L4Azd+q4nvMeRj4OjRlHlQuuqWWlyQXmJUmiefS82qmtNeqWoBd1PYyP9Jah4V0D5zmelf+XNsw57aLKfhfq0baer7BZ7hTZ0ADxLBB+1Aw== ARC-Message-Signature: i=1; a=rsa-sha256; d=aueb.gr; s=arc201901; t=1733472999; c=relaxed/simple; bh=sX5vT0Gfa/IEXBXAeTbtpn14rOp30G/yi+OMQ18Fsbc=; h=DKIM-Signature:Received:Message-ID:Date:MIME-Version:User-Agent: Subject:To:References:From:Content-Language:Organization:Phone: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=Uhaxlyrki7M4ev+czjiWaTIOTueuseXFAAT8JrE7TY+zzoTq27OSAqUfhrmC7/dikmdGGKnWSrvzU3XsngbrbDvW5xjCJhI55rA2VWh+083nJ6lH2KUYChYBfJXb7PRs7zf6ILD5LH9R859fMuGBrG9FeKn68BCid2ft6VRK49TOa8lnR8XhZHjDb2wBz6OzfWSHmsbaX6FkrJFuQkQD5XrwX5RDiWluJV9czo5ZfCQ//0pR54FB6NCw6LUIMkz55hwP9Udacvc0SPJIooW2bQaJMAmxrZmufdpEskThuLC4bpBfRm/D2G9JZZiS7V1t3d8EoXWXgdvAtMvgx8nc2w== ARC-Authentication-Results: i=1; aueb.gr; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=aueb.gr; s=201901; t=1733472999; bh=sX5vT0Gfa/IEXBXAeTbtpn14rOp30G/yi+OMQ18Fsbc=; h=Date:Subject:To:References:From:In-Reply-To; b=k/LYpCob1mQKTc5u1HHeD39SUV/vxDKe5YOFaAWiFOOufFYfsXdJ0+idHvs14j6GS 3+O/bjlUKqdHDuUJj4ORW9Ii1EgVJ73n3unJgHDoVN64K8BFLpXYpwhv2XB9rafKqo HaJSsiinkLs2EfAv8YTHpr+kcLlk5GY73p9BxREwXoti2+jJZQIMkjEv97bYw5XSNq tM6Be3/uEQm/GGiOctJn/xDgN5EoaRQMPLfE7xqN6dXxw3cfub07C0bUb34lVuXn91 q95oTBVL3soCCn+AaWxRtsGba1dp5RMOHWD+wiDZGWU39ZSn+ZjaDZj3YlWu1qSvMt HT4GUS36/8irw== Received: from [192.168.136.3] (athedsl-4477484.home.otenet.gr [94.71.30.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dds) by vmail-mta.servers.aueb.gr (Postfix) with ESMTPSA id 26B02D80AA4; Fri, 6 Dec 2024 10:16:39 +0200 (EET) Message-ID: <4c107309-75b7-42b3-ada3-161ec1421bd5@aueb.gr> Date: Fri, 6 Dec 2024 10:16:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Marc Rochkind , tuhs@tuhs.org References: <568FD44F-01FB-441B-846B-7D42C3A8E1FB@canb.auug.org.au> <20241205030843.8552FAB1EDA5@ary.qy> <2fd6a361-b26f-4ca4-9c65-b6ba0559644e@case.edu> <202412052055.4B5KtYf1781856@freefriends.org> <002cf6d3-0ed2-46f2-9fae-19a3e10d7b9c@case.edu> From: Diomidis Spinellis Content-Language: el-en Organization: Athens University of Economics and Business Phone: +30 210 8203621 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: AN6D5HWCDEJXC6LXUHZUQ5L5W5OEYSLG X-Message-ID-Hash: AN6D5HWCDEJXC6LXUHZUQ5L5W5OEYSLG X-MailFrom: dds@aueb.gr X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-tuhs.tuhs.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?) List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 06-Dec-24 01:07, Marc Rochkind wrote: > I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open > access: > > https://ieeexplore.ieee.org/document/7903579 ieeexplore.ieee.org/document/7903579> > > The open source code itself is here: https://github.com/dspinellis/dgsh > > > Maybe an ambitious TUHS contributor can get the code running and give us > a report. I wrote the dgsh code with my co-author Marios Fragkoulis, so I still have it running. Doug McIlroy, who also mentioned dgsh in another message, is too modest to say that its design owes much to his input. I asked him for feedback when I was working on it, and over several iterations he proposed important (and quite demanding as I recall) improvements to its design. The system allows the concise and readable expression of several graph topologies I had in mind when I started working on it, and more [1]. However, it hasn't caught on. I think the main reason is that it is based on modified versions of several existing tools (bash, cmp, comm, cut, diff, diff3, grep, join, paste, perm, sort) [2]. The modifications allow the tools to coordinate between them the setup of pipes when placed in a dgsh graph according to available inputs and required outputs. The changes (especially for bash) aren't small, which meant that I didn't think it was realistic to push them upstream, which now means that the modified tools are out of date and difficult to build. Not sure what can be done to address this problem. It seems that a widely adopted system, such as modern Unix/Linux, has too much inertia for it to adopt potentially disrupting innovations. In retrospect, the way we designed the pipe graph setup could also be improved. The current design involves an initial phase where IPC messages are circulating around the graph to communicate the I/O requirements of each tool, for example that comm(1) should expect input from two processes and output to three processes. The design is brittle and difficult to troubleshoot, because coordination happens dynamically behind the scenes. A better design (and one I think Doug was advocating) would statically analyze the graph's topology and invoke each tool with appropriate parameters or environment variables. However, this design would require significantly more extensive modifications to bash, or the implementation of a new shell. Both approaches required work for which we didn't have the time and energy at the time, and also had their own downsides regarding adoption potential. [1] https://www.spinellis.gr/sw/dgsh/#examples [2] https://www.spinellis.gr/sw/dgsh/#tools Diomidis