From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=0.2 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED autolearn=no autolearn_force=no version=3.4.4 Received: (qmail 26616 invoked from network); 18 Oct 2022 04:58:47 -0000 Received: from 9front.inri.net (168.235.81.73) by inbox.vuxu.org with ESMTPUTF8; 18 Oct 2022 04:58:47 -0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by 9front; Tue Oct 18 00:57:09 -0400 2022 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id E57115C01B4; Tue, 18 Oct 2022 00:57:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Tue, 18 Oct 2022 00:57:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aqwari.net; h=cc :content-transfer-encoding:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to; s=fm3; t=1666069027; x=1666155427; bh=rdQUhpEuL3 kqJj59AlhjR5oD7VjcEw/s67FRC7HYJAI=; b=ehoa+ZByVXF92Mu/mVx9WAKI4i juft01rSmaLrgF+aClwoGcXuyJ3fB2V8Kjca4rzrrly6szi3gtqWCORrFQ4dLczF 8rEXSnHaLB84njkham4AdDCwjkfNUcW4dDRZ2bQ/2jmwsBk5l55cIos9YQyIg3Wj mzapjIksfdUWF/WLRSElbjm9YWOBPhBuMcnSs8yDDFP5KtKnaBclQT1QSFEEph2O YWxR4+M9nl/W54fQelQ/ZzSB/GVEtTiiWdUhA9g+iN16g/cGLAlCOrgaVdiXQqSC NDfZQIhD2HpC8waIj6qiFnbcsaAabaM8CYDH74FdaLbggBoxBx1cIe/zIs8g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:date:feedback-id:feedback-id:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1666069027; x=1666155427; bh=rdQUhpEuL3kqJj59AlhjR5oD7Vjc Ew/s67FRC7HYJAI=; b=rKaNz0ygVnf2quBssOnGmlPzFrLUV0WX1WLYT0o/EkFF GE4LAB1LvtJugUVyn/Hs1Kg3Grm4hEcpFfeTRGOfAezbIiKDLZ/Dpbk2FIHJ+o3u utgA1RMeQMrfOUDGjw5OY1B+UNX+6Q+myLHIptDKi6VbqD4n6MprjB26CUmMRbuE unlKC0H69foRl4LcqSZQkA+sOUbgFa3OM4GnG4Pb4eRJT733OFnShiNg/tY014Mi ZX5y22NkxLxxjB+FuOCGVwjcWOM46+MW6FaQGPKPqRcxWxJcdBY2mIbPabZMQV2H 8iV76qYzcwdSfcFTZneVvY/pN73fVufDA+/hv66kaQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfeeltddgledtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefkvffufffhgggtgfesthejjedttd dtvdenucfhrhhomhepughrohihohesrghqfigrrhhirdhnvghtnecuggftrfgrthhtvghr nheplefgjefgfedvheetjeelkedvieehveduhffftdeutdeihffgffeukedvledvgfdvne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughrohih ohesrghqfigrrhhirdhnvght X-ME-Proxy: Feedback-ID: i624b4305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for <9front@9front.org>; Tue, 18 Oct 2022 00:57:07 -0400 (EDT) Message-ID: To: 9front@9front.org Date: Tue, 18 Oct 2022 04:57:05 +0000 From: droyo@aqwari.net MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit List-ID: <9front.9front.org> List-Help: X-Glyph: ➈ X-Bullshit: advanced object-oriented general-purpose realtime-java locator Subject: [9front] Speeding up snoopy(8) Reply-To: 9front@9front.org Precedence: bulk I am a plan 9 novice, especially when it comes to performance, so thank you in advance for your patience. I have been trying to diagnose some interesting behavior in 9front's TCP where it appears to be sending a handful of TCP segments out-of-order every 1 second or so at gigabit rates, and this is a of a detour from that. I setup a 9front VM on qemu with 2 cores and 2GB of memory. I have not tried to tune the VM parameters at all, and it's very possible that I'd get different results on real hardware. I did a trivial tcp benchmark from the 9front VM: aux/listen1 tcp!*!9999 cat /dev/zero And connected to the VM from its host. This easily achieves more than gigabit speeds. The client and server are using sub-interfaces on the same physical NIC, so latency is very low. However, when I attempted to capture traffic on the 9front side with ramfs; snoopy -D -f 'tcp(sd=9999)' /net/ether0 >/tmp/pcap I found that about 50% of the packets were not recorded (I knew this because I compared it with a capture I took at the client). With help from IRC, I could see soft overflows incrementing in the /net/ether0/$index/stats file. I patched the stats command so I could watch it during my benchmarks. While cinap offered some alternative filtering options like ipmux, I would like to try to improve snoopy's performance. I tried profiling snoopy with pid=`{ psu | awk '/snoopy/ { print $2 }' } echo profile > /proc/$pid/ctl while() { tprof $pid ; sleep 5 } Here is a sample during the benchmark, with snoopy's output redirected to /dev/null to rule out disk or ramfs bottlenecks: total: 1150 TEXT 00200000 ms % sym 360 31.3 pread 230 20.0 pwrite 120 10.4 nsec 110 9.5 _filterpkt 70 6.0 tracepkt 60 5.2 p_filter 60 5.2 defaultframer 50 4.3 p_filter 40 3.4 be2vlong 20 1.7 read 10 0.8 write 10 0.8 newfilter 10 0.8 main You can see a full 50% of time is spent in pread/pwrite. I patched snoopy to dial /net/ether0!-2 instead of /net/ether0!-1, which truncates the frames to the first 64 bytes, and the overflows went away. The first 64 bytes are generally the most useful bytes, so an easy win would be to use the "-2" dial option when snoopy's "-M" flag is <=64. More generally, though, I think snoopy could be much faster if it read multiple frames per pread() and wrote multiple records per pwrite(). I am a bit surprised to see pwrite() so high even when I am redirecting to /dev/null. It's trivial enough to buffer snoopy's output with bio(2). But I don't know of an existing interface that would let me read more than one frame at a time. Is there one? The only other option I can think of to reduce the effect of pread() would be to move the pread to a separate thread. But I'm curious to hear your thoughts. Thanks! David