From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=0.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,RDNS_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.4 Received: (qmail 14461 invoked from network); 1 Jun 2022 04:27:04 -0000 Received: from unknown (HELO tb-ob0.topicbox.com) (64.147.108.117) by inbox.vuxu.org with ESMTPUTF8; 1 Jun 2022 04:27:04 -0000 Received: from tb-mx0.topicbox.com (tb-mx0.nyi.icgroup.com [10.90.30.73]) by tb-ob0.topicbox.com (Postfix) with ESMTP id 6A53724E4B for ; Wed, 1 Jun 2022 00:26:46 -0400 (EDT) (envelope-from bounce.mMf37a0689afc5c54c9aba65d7.r522be890-2105-11eb-b15e-8d699134e1fa@9fans.bounce.topicbox.com) Received: by tb-mx0.topicbox.com (Postfix, from userid 1132) id 65F4225EA3C3; Wed, 1 Jun 2022 00:26:46 -0400 (EDT) ARC-Authentication-Results: i=2; topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=iitbombay-org.20210112.gappssmtp.com header.i=@iitbombay-org.20210112.gappssmtp.com header.b=IKTxOauX header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=iitbombay.org; spf=pass smtp.mailfrom=bakul@iitbombay.org smtp.helo=mail-qt1-f174.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=from:content-type:content-transfer-encoding :mime-version:subject:date:references:to:in-reply-to:message-id :list-help:list-id:list-post:list-subscribe:reply-to :list-unsubscribe; s=sysmsg-1; t=1654057606; bh=Xtt9SMJAIj55p0QK qs4dy/+pGdAUQt1cuXvvM7QnQHQ=; b=QeJM2aa2bVlPLjhBk2RF8/oF7T7K65m1 ZlU70QYP2LaijO/JxHWxwk0wQD833g3SYIi61IblPi1a6usI48acfIkqACZNEz/c wUNnxW+P9+uHTMZMeNq/lmPJ7ajDJSNGVSnqSrdq4RMZMj+UhAtotTmOqTo/LNSa YUlts51pNM0= ARC-Seal: i=2; a=rsa-sha256; cv=pass; d=topicbox.com; s=sysmsg-1; t= 1654057606; b=eB2LvGz3i65WZO/KOFtxfw9pu4qZzjSfEhUtMHwzVdF90Zv011 uyZa1hhmfLuXDeUAEWxj4bqeyuv1oTHhz1eFNT4VmJ9aPRZepfAnnlZx2FQsvToV 6Wz5NDJ/msHVtr9COf2l8i94oI+R0qOHMU9ho6L/TYZLUVM6FTn95YCBA= Authentication-Results: topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=iitbombay-org.20210112.gappssmtp.com header.i=@iitbombay-org.20210112.gappssmtp.com header.b=IKTxOauX header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=iitbombay.org; spf=pass smtp.mailfrom=bakul@iitbombay.org smtp.helo=mail-qt1-f174.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) X-Received-Authentication-Results: tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=iitbombay-org.20210112.gappssmtp.com header.i=@iitbombay-org.20210112.gappssmtp.com header.b=IKTxOauX header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=iitbombay.org; iprev=pass smtp.remote-ip=209.85.160.174 (mail-qt1-f174.google.com); spf=pass smtp.mailfrom=bakul@iitbombay.org smtp.helo=mail-qt1-f174.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=2gvpbHB0; x-me-sender=none; x-ptr=pass smtp.helo=mail-qt1-f174.google.com policy.ptr=mail-qt1-f174.google.com; x-return-mx=pass header.domain=iitbombay.org policy.is_org=yes (MX Records found: alt4.aspmx.l.google.com,alt2.aspmx.l.google.com,alt3.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com); x-return-mx=pass smtp.domain=iitbombay.org policy.is_org=yes (MX Records found: alt4.aspmx.l.google.com,alt2.aspmx.l.google.com,alt3.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=9fans.net; h=from :content-type:content-transfer-encoding:mime-version:subject :date:references:to:in-reply-to:message-id:list-help:list-id :list-post:list-subscribe:reply-to:list-unsubscribe; s=dkim-1; t=1654057606; x=1654144006; bh=Xtt9SMJAIj55p0QKqs4dy/+pGdAUQt1c uXvvM7QnQHQ=; b=kIj2Jamx15o0FUIl24TnkksWRKT/HJFlx3HXhwBq8bVEyune fAgisDRYfKqK5Yx3BFd7PhhCsBgr+rTo+ZtS3nHIp2rxAPsXzV/uWYvkC8cV1Dxz GtpYPgFqROs0oLs98TcJk/+LNFlGyGJPadAJ41QHMN4nWxYEIhIO6cCYqCg= Received: from tb-mx0.topicbox.com (localhost.local [127.0.0.1]) by tb-mx0.topicbox.com (Postfix) with ESMTP id 8323325E9FA7 for <9fans@9fans.net>; Wed, 1 Jun 2022 00:26:33 -0400 (EDT) (envelope-from bakul@iitbombay.org) Received: from tb-mx0.topicbox.com (localhost [127.0.0.1]) by tb-mx0.topicbox.com (Authentication Milter) with ESMTP id 49810C7F74E; Wed, 1 Jun 2022 00:26:33 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1654057593; b=QpSMzZ4qeiprjtPfasiI2cvXdTlvDoZWztrFyrb+ZDiKjsvDHT jiXSMUoWV3EXSWLwP9eNfPprPRZ3rpOGnaONYrrkEldC8DA2obCP7kAlxukasWLq /VwPc7iCBWjkEBoWXOkUeHxZuQIor0aDGgimDgz06R5LZFbnfUvNUJACFvXVmJcT YTic9yNTpqedTXJkRprYguoGmv85S5xbkpVAsAzNl0gREqh0blTV2sWF8zyTH9FT I4YUGH+wHeMuEOznNsMA5vpv3aSsQa9QAgiLCa8qVcPovru63FLmrKkoG3pq88pZ chAtve+6DP8IYv5avaZsQhupBV2RjbcsxaNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=from:content-type:content-transfer-encoding :mime-version:subject:date:references:to:in-reply-to:message-id; s=arcseal; t=1654057593; bh=5oEy4a3e0H+kn2S4iZ8hTb9zNuV9yKps76Q kboFLMGw=; b=Cy07M58nlj0LVyt08GURFUEH70bMtX0FgTCgshAXtL8gRvsapQd 3yLxxpjyld7io8oxFeUkd34W7QhGPLhNFWPMejZCfvZBfkJZ/OPp/ptm7ttNCggQ 0twE5+gJmu0sa4iwTKsc19wwN7PFnOa3QK+hUtWiekoVDmzM5BomKVThX+7htcic j4WDQMRksKZAYSqgo/DMEFF/q2zTnfnobfXNW/V9H303yEHLBXWoJQs9Mq/LaJZI ne8SFdU8ZWPeY/PVnpHsQV/bVwbBp3VyytI+X7aQB0k5jTvX6B48wtGifm8M1/vU vitlvv0gALcslxbIk+vX8soF8tjyV3Gl6Yg== ARC-Authentication-Results: i=1; tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=iitbombay-org.20210112.gappssmtp.com header.i=@iitbombay-org.20210112.gappssmtp.com header.b=IKTxOauX header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=iitbombay.org; iprev=pass smtp.remote-ip=209.85.160.174 (mail-qt1-f174.google.com); spf=pass smtp.mailfrom=bakul@iitbombay.org smtp.helo=mail-qt1-f174.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=2gvpbHB0; x-me-sender=none; x-ptr=pass smtp.helo=mail-qt1-f174.google.com policy.ptr=mail-qt1-f174.google.com; x-return-mx=pass header.domain=iitbombay.org policy.is_org=yes (MX Records found: alt4.aspmx.l.google.com,alt2.aspmx.l.google.com,alt3.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com); x-return-mx=pass smtp.domain=iitbombay.org policy.is_org=yes (MX Records found: alt4.aspmx.l.google.com,alt2.aspmx.l.google.com,alt3.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgedvfedrkeelgdekudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhephfgtgfgguf fffhfvjgfkofesthhqmhdthhdtvdenucfhrhhomhepuegrkhhulhcuufhhrghhuceosggr khhulhesihhithgsohhmsggrhidrohhrgheqnecuggftrfgrthhtvghrnhepvdfgffevie ekgfffveelgeekieevfffhgfdujeetteeludeghfeftedtueefudeinecuffhomhgrihhn pehgihhthhhusgdrtghomhenucfkphepvddtledrkeehrdduiedtrddujeegpddutdejrd dvudehrddvvdefrddvvdelnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehi nhgvthepvddtledrkeehrdduiedtrddujeegpdhhvghlohepmhgrihhlqdhqthduqdhfud ejgedrghhoohhglhgvrdgtohhmpdhmrghilhhfrhhomhepoegsrghkuhhlsehiihhtsgho mhgsrgihrdhorhhgqe X-ME-VSScore: 0 X-ME-VSCategory: clean Received-SPF: pass (iitbombay.org: Sender is authorized to use 'bakul@iitbombay.org' in 'mfrom' identity (mechanism 'include:_spf.google.com' matched)) receiver=tb-mx0.topicbox.com; identity=mailfrom; envelope-from="bakul@iitbombay.org"; helo=mail-qt1-f174.google.com; client-ip=209.85.160.174 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx0.topicbox.com (Postfix) with ESMTPS for <9fans@9fans.net>; Wed, 1 Jun 2022 00:26:32 -0400 (EDT) (envelope-from bakul@iitbombay.org) Received: by mail-qt1-f174.google.com with SMTP id c8so425460qtj.1 for <9fans@9fans.net>; Tue, 31 May 2022 21:26:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=5oEy4a3e0H+kn2S4iZ8hTb9zNuV9yKps76QkboFLMGw=; b=2gvpbHB02cQeFXzJbi5mNlOqALla/LxfGjPaPkR6fhqECKg11rZMF7iPNwEGEGTjVD pRnsZxlyJgXuCUtG1oMgMOSXrcTUX29KVtkDoqYyQ/AAf5ap5TDQBdT9Lz5GA6M1KDM4 EF0mFsowr42KGFpHhrmcG8lwRV+yEtlzW8wwrexnwxrpW8nW6tttAnkmomCLskcpxuAZ OQ7RaBSTXsRsBSrCgDLFfDA6PJdNm4nftLArrad2LhdQvaR+Vmu+5DTcwnVDxoA7RImv X+pYVmDt9u1UcFyHpV+wDYZ/39vHu3Gu3cIgWJmwkHy2/v8k4LwjcwW+AZC8+mKWTm+L Z9Aw== X-Gm-Message-State: AOAM531NJ4sz4bOT9a2dm4GrT4oErDrWjFIVYDY/11gyeAOgeh5rTCuZ MLfjAcZOB8ZQRQ1CeOX5lUTC5JbXosNPPTtG X-Google-Smtp-Source: ABdhPJyheX5XlnD3G6070xYn6pgBTJwh4mRvEO2nEdY1ZM4R0kYrrFnK4Oo9ifA60xbnTpz1M7Mogw== X-Received: by 2002:ac8:5b89:0:b0:304:c69a:d2dd with SMTP id a9-20020ac85b89000000b00304c69ad2ddmr535075qta.570.1654057591878; Tue, 31 May 2022 21:26:31 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id e185-20020a3769c2000000b006a5d4f32e5dsm566392qkc.128.2022.05.31.21.26.31 for <9fans@9fans.net> (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 May 2022 21:26:31 -0700 (PDT) From: Bakul Shah Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.100.31\)) Subject: Re: [9fans] 9p server to multiply 9p messages? Date: Tue, 31 May 2022 21:26:29 -0700 References: <548D3C1F-DFCC-4DCE-AB37-3CA48997AE69@iitbombay.org> To: 9fans <9fans@9fans.net> In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3696.100.31) Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 076c7dea-e163-11ec-aeda-86b1eac6e987 Archived-At: =?UTF-8?B?PGh0dHBzOi8vOWZhbnMudG9waWNib3guY29tL2dyb3Vwcy85?= =?UTF-8?B?ZmFucy9UNzY5ODU0ZmFmZDJiN2QzNS1NZjM3YTA2ODlhZmM1YzU0YzlhYmE2?= =?UTF-8?B?NWQ3Pg==?= List-Help: List-Id: "9fans" <9fans.9fans.net> List-Post: List-Software: Topicbox v0 List-Subscribe: Precedence: list Reply-To: 9fans <9fans@9fans.net> List-Unsubscribe: , Topicbox-Delivery-ID: 2:9fans:437d30aa-c441-11e9-8a57-d036212d11b0:522be890-2105-11eb-b15e-8d699134e1fa:Mf37a0689afc5c54c9aba65d7:1:A8boBW-d3VmIC9KqFLjM7se15vSD4rONatiWUzVm-yk On May 31, 2022, at 9:14 AM, ron minnich wrote: >=20 > On Mon, May 30, 2022 at 12:21 AM Bakul Shah wrote: >> 9p itself is low performance but that is a separate issue. >=20 > Bakul, what are the units? It might be helpful to quantify this > statement. Are you possibly conflating Plan 9 file systems being slow > and 9p being slow? I did a quick test: >From a 9front VM to another machine I get about 11.7 MBps caching. The first time around it was close to 7.3 MBps). >From an Ubuntu VM to another machine I get about 111 MBps (cached. The first time around it was close to 62 MBps). Both VMs run on the same host. Test copies to the same target machine. I used 9p read for 9front, scp for Linux, copy to /dev/null. The target machine is freebsd. The VMs talk to the target over a 1Gbps ethernet (so 111 MBps is the wirespeed limit). 9front uses hjfs. Ubuntu uses ext4. On the host I give a file as the guest "disk", using 'nvme' type device on bhyve to each VM. Both 9front and ubuntu are 64 bit kernels. This is a very rough measurement as there are many differences between the systems. The filesystem overhead is clearly an issue but 10 times worse? ----- Looking at the protocol: For read/write 9p uses 4 byte for size so in theory you can send very large packets but then you have to buffer up a lot of data. Ideally you want streaming (some sort of sliding window). May be you can use the tag field to do something more intelligent. Not sure any implementations do so. You also have head of line blocking if you can have only one TCP connection to a server. > As Rob pointed out in 2013, "If go install is slow on Plan 9, it's > because Plan 9's file system is > slow (which it is and always has been)", so slowness in Plan 9 file > systems is to be expected. >=20 > 9p itself does have its limits, which is why Bell Labs Antwerp started > an effort in 2011 to replace it, but the new work never went very far. >=20 > I also know of a number of efforts in the virtualization world where > 9p was discarded for performance reasons. It's hard to argue with the > 100x performance improvement that comes with virtiofs, for example. Why is virtiofs 100x faster? Just lot of hardwork and tuning? May be that is good place to look to learn what needs to change (in case someone wants to replace 9p with something else)? =20 > Gvisor is replacing 9p: https://github.com/google/gvisor/milestone/6. > Although, in the latter case, I would argue the problem is more with > Linux limitations than 9p limitations -- linux can't seem to walk more > than one pathname component at a time, for example, since it has the > old school namei loop. >=20 > But I'm wondering if you have a measurement with numbers. >=20 > For rough order of magnitude, HPC file systems can deliver 10 Gbytes/ > second for file reads nowadays, but getting there took 20 years of > work. When we ran Plan 9 on Blue Gene, with the 6 Gbyte/second > toroidal mesh connect for each node, we never came remotely close to > that figure. Given that experience, why do you need "numbers"? :-) Running 10Gbps links even @ home is quite doable now. With TCP you can achieve decent performance if not quite wirespeed. NVMe "disks" are pretty damn fast - you can easily get 2-4 GBps. But I think at remote filesystem protocol level you'd have to optimize multiple things in order to get close to wirespeed performance. Minimize copying, increase concurrency, reduce overhead in frequently used common path code, reduce user/kernel crossings etc. I think rdma and mmap will probably get used a lot too (obviously on non-plan9 OSes!). May be if you pushed 9p knowledge down to a smart NIC, it can map a tag value to compute location where the data needs to go. But all this is just handwaving. Without a real project and funding it is hard to get sufficiently motivated to do more. ------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/T769854fafd2b7d35-Mf37a0= 689afc5c54c9aba65d7 Delivery options: https://9fans.topicbox.com/groups/9fans/subscription