From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 3561 invoked from network); 1 Jun 2022 07:25:28 -0000 Received: from tb-ob0.topicbox.com (64.147.108.117) by inbox.vuxu.org with ESMTPUTF8; 1 Jun 2022 07:25:28 -0000 Received: from tb-mx0.topicbox.com (tb-mx0.nyi.icgroup.com [10.90.30.73]) by tb-ob0.topicbox.com (Postfix) with ESMTP id E847325E73 for ; Wed, 1 Jun 2022 03:25:26 -0400 (EDT) (envelope-from bounce.mM9900ebe3ebf76b5d4d4426bd.r522be890-2105-11eb-b15e-8d699134e1fa@9fans.bounce.topicbox.com) Received: by tb-mx0.topicbox.com (Postfix, from userid 1132) id E461325EDAC8; Wed, 1 Jun 2022 03:25:26 -0400 (EDT) ARC-Authentication-Results: i=2; topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=kXwhgDJz header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; spf=pass smtp.mailfrom=23hiro@gmail.com smtp.helo=mail-pl1-f177.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type:list-help:list-id:list-post :list-subscribe:reply-to:content-transfer-encoding :list-unsubscribe; s=sysmsg-1; t=1654068326; bh=0/0lsQlpSlQAF3dJ RfHeH+1FdgpgACRGuz3QQjcQJ6Q=; b=ZeaPfKx0ljnCuFE9L9b3nOoERCpDQuuv usKZahzeTrX76AU/aCeFzCPQUcstax0DHSItEp294zOwssGMuDIq6P07SSUQ5Syi pByIU5/6mTeQqZGOffOlwJwfj4nVHvlVoLhnUVcVxv0rG8ZUyec7Z5MsZJR2PCf0 gjKivxtIq1Q= ARC-Seal: i=2; a=rsa-sha256; cv=pass; d=topicbox.com; s=sysmsg-1; t= 1654068326; b=nNILsIslubhrwQ2O+IqS6VJIy0xEPc6TgXqfk3kH0K2cdZ5lpN vB2QKCKuoo6YCYhrdoayoWbvGywz0ZhKYdvU1a+P8SPPvqlmv7TxxS2YzkcFJiHB dKknQTtP/cCNlpWvVpQrxafRHVFkFLza3wtUaBRkoLfvr+kPShbHhZ0Vk= Authentication-Results: topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=kXwhgDJz header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; spf=pass smtp.mailfrom=23hiro@gmail.com smtp.helo=mail-pl1-f177.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) X-Received-Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=kXwhgDJz header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.214.177 (mail-pl1-f177.google.com); spf=pass smtp.mailfrom=23hiro@gmail.com smtp.helo=mail-pl1-f177.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=zJ1xRbVS; x-me-sender=none; x-ptr=pass smtp.helo=mail-pl1-f177.google.com policy.ptr=mail-pl1-f177.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=9fans.net; h= mime-version:in-reply-to:references:from:date:message-id:subject :to:content-type:list-help:list-id:list-post:list-subscribe :reply-to:content-transfer-encoding:list-unsubscribe; s=dkim-1; t=1654068326; x=1654154726; bh=0/0lsQlpSlQAF3dJRfHeH+1FdgpgACRG uz3QQjcQJ6Q=; b=NDRCFj9CiMjM1oRFPChs3iCQoGsuwCzQCl2qEJYUv2PAt9rP Bk1nOpKvnh15ZdbKNvR2eY/9hdLkFbObZTIVtENi/8z6CK+r8Vdyg1x5pZ2UfzW+ GuyGyUL85afjy3ZytbpPxeYYJiAT8SJxjoEyjBzZdXhmdpUd3+yc3hDx2yk= Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id 2CA8E150E758 for <9fans@9fans.net>; Wed, 1 Jun 2022 03:25:06 -0400 (EDT) (envelope-from 23hiro@gmail.com) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id 2D9A82C802E; Wed, 1 Jun 2022 03:25:06 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1654068305; b=TRATDuonbsQwjztlt3MvMuIaI8wVrw3F3KTZZpYTjrXSZznZqV pgwi2rPwWyBOGNCKMo7wRQBeiByFUvXXHJKwoGnKsROt00Dif4Mlj/D5aKamt7w6 NPE+pNeryP2SZAmg3zuRMdUt573tBhI9nc9g+2bodGuV3Wi9TUuJ2/eDeurlfA4M VLM4/Dk48o4lyAwfUiOU/k8VpM/NLMjqg6clxcXWEtCUVRPOiBdoyDOZc1dNevl3 Ldwxq4PVz0IWH/0U7BTp0c4u4QULN5n3tzwZvxQ3R0TKWIxRTEw2VewCt7wF5Krd APON+QhsZP3Azjzh2ECPLhpUObRGmYtK2Gqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; s=arcseal; t=1654068305; bh=jRrDNKSzQZB0LLbbg6ezZ6D2YiMLD+5F6qiUYhMFzQk=; b=Zv2jiILrlCKx ptiOiSHFN9/dX0RddRAJ3h8jY7Wta4dNm7yJ1D78JQul8bNkXpJ/9JvlSVzBG4yS pvAJV+4migTgdGZ/ZatPDggdmcoJ2V6Bu3O6nF90cn5JKPAbnY165xaMRfPzmAOf 11pb/OB8Gzxo6b3Kjygmhqy3IvluBGaPgUTycJRDyaPb6EYnkvh80lqiycnJgHdZ wdzH4W2iwHHEqdVf1Gv6WYAPPsE2VuZvny653AM7kwef6M2hf2etXD7+32K6figZ QrM/ug3WMAjjqAhZqrXrpDv7rLnSi0XJRo8lqTuB9kFtHr3A8rICAL5Vh3BQwc+H Unh6GPwClw== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=kXwhgDJz header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.214.177 (mail-pl1-f177.google.com); spf=pass smtp.mailfrom=23hiro@gmail.com smtp.helo=mail-pl1-f177.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=zJ1xRbVS; x-me-sender=none; x-ptr=pass smtp.helo=mail-pl1-f177.google.com policy.ptr=mail-pl1-f177.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgedvfedrkeelgdduudeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpegjfhfhff fkuffvtgesthdtredttddtjeenucfhrhhomhephhhirhhouceovdefhhhirhhosehgmhgr ihhlrdgtohhmqeenucggtffrrghtthgvrhhnpedvveegfefgveeigfdthfegveehvddvle eiteeuleejheekjeekjeekgeevgfefgfenucffohhmrghinhepghhithhhuhgsrdgtohhm pdhtohhpihgtsghogidrtghomhenucfkphepvddtledrkeehrddvudegrddujeejnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepvddtledrkeehrddvudeg rddujeejpdhhvghlohepmhgrihhlqdhplhduqdhfudejjedrghhoohhglhgvrdgtohhmpd hmrghilhhfrhhomhepoedvfehhihhrohesghhmrghilhdrtghomheq X-ME-VSScore: 0 X-ME-VSCategory: clean Received-SPF: pass (gmail.com ... _spf.google.com: Sender is authorized to use '23hiro@gmail.com' in 'mfrom' identity (mechanism 'include:_netblocks.google.com' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="23hiro@gmail.com"; helo=mail-pl1-f177.google.com; client-ip=209.85.214.177 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for <9fans@9fans.net>; Wed, 1 Jun 2022 03:25:05 -0400 (EDT) (envelope-from 23hiro@gmail.com) Received: by mail-pl1-f177.google.com with SMTP id w3so956622plp.13 for <9fans@9fans.net>; Wed, 01 Jun 2022 00:25:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=jRrDNKSzQZB0LLbbg6ezZ6D2YiMLD+5F6qiUYhMFzQk=; b=zJ1xRbVSzoxJZBvfS+RmrX4PCYa7nEdFEDYE/WcFnOeoTnc38u2EBVP7IJFjhuVQGi pnUnnBi86dRpvKd0WXtr4qtCpUni0nDU4vHJQXlseqxdKKbpowpNwvp/dW8uOpe7eqq1 3beNuCV3LfLr5FpgNWfxyRIQ7hXYD7/8j04TsUeZAwW4dQEfyeatrS6soIgpSSE/Ha6a tU2ckOoRkq9usgrnr35mFEzpz3ebm7ZxbdQOt3lC1iMluxMjZpDKC2q/Br5NT1sEMwuk Y0uZxkohUsV6vvU/dIfhMgSp/m7niNRjeII5IqSr0I5lBSxswdnG7iKd1kHalkGpNLvK Z4sg== X-Gm-Message-State: AOAM533z/yes/KUEX7AvduFAmorTSG8KO3rdkIbgRfu5+kgbCJOkILZ9 fdPtW6MCAttckbDkM8tDsCDNiyShHW6L8bdJO1WsJy8yseg= X-Google-Smtp-Source: ABdhPJz3mgRs5j0MBiiUuX5GlnPsis56OX5ObdIDopi22I3CMw2pWKB0hc9UgtPoZ8cOZN/+6557kiIoddyhFWy32Bc= X-Received: by 2002:a17:903:120e:b0:15e:84d2:4bbb with SMTP id l14-20020a170903120e00b0015e84d24bbbmr62713454plh.165.1654068303855; Wed, 01 Jun 2022 00:25:03 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a05:7300:64cc:b0:60:be3a:16c2 with HTTP; Wed, 1 Jun 2022 00:25:03 -0700 (PDT) In-Reply-To: References: <548D3C1F-DFCC-4DCE-AB37-3CA48997AE69@iitbombay.org> From: hiro <23hiro@gmail.com> Date: Wed, 1 Jun 2022 10:25:03 +0300 Message-ID: Subject: Re: [9fans] 9p server to multiply 9p messages? To: 9fans <9fans@9fans.net> Content-Type: text/plain; charset=UTF-8 Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: f93d3da4-e17b-11ec-a63e-b3e4c870b231 Archived-At: =?UTF-8?B?PGh0dHBzOi8vOWZhbnMudG9waWNib3guY29tL2dyb3Vwcy85?= =?UTF-8?B?ZmFucy9UNzY5ODU0ZmFmZDJiN2QzNS1NOTkwMGViZTNlYmY3NmI1ZDRkNDQy?= =?UTF-8?B?NmJkPg==?= List-Help: List-Id: "9fans" <9fans.9fans.net> List-Post: List-Software: Topicbox v0 List-Subscribe: Precedence: list Reply-To: 9fans <9fans@9fans.net> Content-Transfer-Encoding: quoted-printable List-Unsubscribe: , Topicbox-Delivery-ID: 2:9fans:437d30aa-c441-11e9-8a57-d036212d11b0:522be890-2105-11eb-b15e-8d699134e1fa:M9900ebe3ebf76b5d4d4426bd:1:bYEyxtkEA41JinUzyGcshdtC7JDyxzk7Zw7Qfg8jD14 And fcp? On 6/1/22, Bakul Shah wrote: > On May 31, 2022, at 9:14 AM, ron minnich wrote: >> >> On Mon, May 30, 2022 at 12:21 AM Bakul Shah wrote: >>> 9p itself is low performance but that is a separate issue. >> >> Bakul, what are the units? It might be helpful to quantify this >> statement. Are you possibly conflating Plan 9 file systems being slow >> and 9p being slow? > > I did a quick test: > > From a 9front VM to another machine I get about 11.7 MBps > caching. The first time around it was close to 7.3 MBps). > > From an Ubuntu VM to another machine I get about 111 MBps > (cached. The first time around it was close to 62 MBps). > > Both VMs run on the same host. Test copies to the same target > machine. I used 9p read for 9front, scp for Linux, copy to > /dev/null. The target machine is freebsd. The VMs talk to > the target over a 1Gbps ethernet (so 111 MBps is the wirespeed > limit). > > 9front uses hjfs. Ubuntu uses ext4. On the host I give a file > as the guest "disk", using 'nvme' type device on bhyve to each > VM. Both 9front and ubuntu are 64 bit kernels. > > This is a very rough measurement as there are many differences > between the systems. The filesystem overhead is clearly an issue > but 10 times worse? > ----- > Looking at the protocol: > > For read/write 9p uses 4 byte for size so in theory you can send > very large packets but then you have to buffer up a lot of data. > Ideally you want streaming (some sort of sliding window). May be > you can use the tag field to do something more intelligent. Not > sure any implementations do so. You also have head of line blocking > if you can have only one TCP connection to a server. > >> As Rob pointed out in 2013, "If go install is slow on Plan 9, it's >> because Plan 9's file system is >> slow (which it is and always has been)", so slowness in Plan 9 file >> systems is to be expected. >> >> 9p itself does have its limits, which is why Bell Labs Antwerp started >> an effort in 2011 to replace it, but the new work never went very far. >> >> I also know of a number of efforts in the virtualization world where >> 9p was discarded for performance reasons. It's hard to argue with the >> 100x performance improvement that comes with virtiofs, for example. > > > Why is virtiofs 100x faster? Just lot of hardwork and tuning? > May be that is good place to look to learn what needs to change > (in case someone wants to replace 9p with something else)? > >> Gvisor is replacing 9p: https://github.com/google/gvisor/milestone/6. >> Although, in the latter case, I would argue the problem is more with >> Linux limitations than 9p limitations -- linux can't seem to walk more >> than one pathname component at a time, for example, since it has the >> old school namei loop. >> >> But I'm wondering if you have a measurement with numbers. >> >> For rough order of magnitude, HPC file systems can deliver 10 Gbytes/ >> second for file reads nowadays, but getting there took 20 years of >> work. When we ran Plan 9 on Blue Gene, with the 6 Gbyte/second >> toroidal mesh connect for each node, we never came remotely close to >> that figure. >=20 > Given that experience, why do you need "numbers"? :-) >=20 > Running 10Gbps links even @ home is quite doable now. With TCP you > can achieve decent performance if not quite wirespeed. NVMe "disks" > are pretty damn fast - you can easily get 2-4 GBps. But I think at > remote filesystem protocol level you'd have to optimize multiple > things in order to get close to wirespeed performance. Minimize > copying, increase concurrency, reduce overhead in frequently used > common path code, reduce user/kernel crossings etc. I think rdma and > mmap will probably get used a lot too (obviously on non-plan9 OSes!). > May be if you pushed 9p knowledge down to a smart NIC, it can map a > tag value to compute location where the data needs to go. >=20 > But all this is just handwaving. Without a real project and funding > it is hard to get sufficiently motivated to do more. >=20 ------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/T769854fafd2b7d35-M9900e= be3ebf76b5d4d4426bd Delivery options: https://9fans.topicbox.com/groups/9fans/subscription