From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 10850 invoked from network); 10 Mar 2023 01:32:26 -0000 Received: from minnie.tuhs.org (2600:3c01:e000:146::1) by inbox.vuxu.org with ESMTPUTF8; 10 Mar 2023 01:32:26 -0000 Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id 8F448413B2; Fri, 10 Mar 2023 11:32:23 +1000 (AEST) Received: from mcvoy.com (mcvoy.com [192.169.23.250]) by minnie.tuhs.org (Postfix) with ESMTPS id 28FC941396 for ; Fri, 10 Mar 2023 11:32:17 +1000 (AEST) Received: by mcvoy.com (Postfix, from userid 3546) id 8BBF335E603; Thu, 9 Mar 2023 17:32:16 -0800 (PST) Date: Thu, 9 Mar 2023 17:32:16 -0800 From: Larry McVoy To: Tom Lyon Message-ID: <20230310013216.GP9225@mcvoy.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Message-ID-Hash: HC56ADTU26TTJP4NFXDS5BVQF6KMHYYD X-Message-ID-Hash: HC56ADTU26TTJP4NFXDS5BVQF6KMHYYD X-MailFrom: lm@mcvoy.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: The Eunuchs Hysterical Society X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: scaling on TCP socket connections List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: SGI made TCP go very fast on 200Mhz MIPS processors. The tricks were to mark the page copy on write on output so the driver could use the page without copying and to page flip on input. Only worked with multiples of page sized requests. SGI had very fast sequential I/O through XFS (the trick there was big I/O requests on files opened with O_DIRECT, the volume manager split the request into chunks sending each chunk to a disk controller; when you get the stupid 200Mhz processors out of the way and spin up a boat load of DMA engines, yeah, you can scale up until the bus is full). And they had fast TCP over Hippi. I showed up and asked why hasn't anyone plugged XFS into TCP? In a few weeks I wrote a user level server that demonstrated that it could work and that turned into this talk: http://mcvoy.com/lm/papers/bds.pdf If you don't go look it made O_DIRECT work on NFS and gave much faster NFS sequential I/O performance. Real NFS was 18MB/sec, my stuff was 67MB/sec for a single file, scaled to 100s of MB/sec. Everyone thought I was smart for doing that but the reality was the XFS/XLV folks and the TCP folks had done all the hard work, I just did some plumbing to connect the two fat pipes. I did eventually push it into the kernel because of stuff like the page cache coherency but even that was easy. And I found the SGI writeup of it, it has more perf numbers: http://mcvoy.com/lm/papers/bdspro.pdf If anyone cares I can post a link to the user level server code. On Thu, Mar 09, 2023 at 04:59:54PM -0800, Tom Lyon wrote: > Sun chose UDP for NFS at a point when few if any people believed TCP could > go fast. > I remember (early 80s) being told that one couldn't use TCP/IP in LANs > because they were WAN protocols. In the late 80s, WAN people were saying > you couldn't use TCP/IP because they were LAN protocols. > > But UDP for NFS was more attractive because it was not byte stream > oriented, and didn't require copying to save for retransmissions. And > there was hope we'd be able to do zero copy transmissions from the servers > - also the reason for inventing Jumbo packets to match the 8K page size of > Sun3 systems. > > I did get zero copy serving working with ND (network disk block protocol) - > but it was terribly specific to particular hardware components. > > On Thu, Mar 9, 2023 at 4:24???PM ron minnich wrote: > > > Ca. 1981, if memory serves, having even small numbers of TCP connections > > was not common. > > > > I was told at some point that Sun used UDP for NFS for that reason. It was > > a reasonably big deal when we started to move to TCP for NFS ca 1990 (my > > memory of the date -- I know I did it on my own for SunOS as an experiment > > when I worked at the SRC -- it seemed to come into general use about that > > time). > > > > What kind of numbers for TCP connections would be reasonable in 1980, 90, > > 2000, 2010? > > > > I sort of think I know, but I sort of think I'm probably wrong. > > -- --- Larry McVoy Retired to fishing http://www.mcvoy.com/lm/boat