From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/51096 Path: main.gmane.org!not-for-mail From: Simon Josefsson Newsgroups: gmane.emacs.gnus.general Subject: Re: nnimap slowness Date: Sun, 30 Mar 2003 21:20:20 +0200 Sender: owner-ding@hpc.uh.edu Message-ID: References: NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: main.gmane.org 1049052113 32023 80.91.224.249 (30 Mar 2003 19:21:53 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Sun, 30 Mar 2003 19:21:53 +0000 (UTC) Cc: ding@hpc.uh.edu Original-X-From: owner-ding@hpc.uh.edu Sun Mar 30 21:21:49 2003 Return-path: Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by main.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 18ziNI-0008Jl-00 for ; Sun, 30 Mar 2003 21:21:49 +0200 Original-Received: from sina.hpc.uh.edu ([129.7.128.10] ident=lists) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 18ziMK-0002W4-00; Sun, 30 Mar 2003 13:20:48 -0600 Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Sun, 30 Mar 2003 13:21:53 -0600 (CST) Original-Received: from yxa.extundo.com (178.230.13.217.in-addr.dgcsystems.net [217.13.230.178]) by sina.hpc.uh.edu (8.9.3/8.9.3) with ESMTP id NAA10122 for ; Sun, 30 Mar 2003 13:21:38 -0600 (CST) Original-Received: from latte.josefsson.org (yxa.extundo.com [217.13.230.178]) (authenticated bits=0) by yxa.extundo.com (8.12.9/8.12.9) with ESMTP id h2UJKKXK023067 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Sun, 30 Mar 2003 21:20:20 +0200 Original-To: David Abrahams Mail-Copies-To: nobody X-Payment: hashcash 1.2 0:030330:dave@boost-consulting.com:e8d42bc22b420f4c X-Hashcash: 0:030330:dave@boost-consulting.com:e8d42bc22b420f4c X-Payment: hashcash 1.2 0:030330:ding@hpc.uh.edu:a7d44f801f45988b X-Hashcash: 0:030330:ding@hpc.uh.edu:a7d44f801f45988b In-Reply-To: (David Abrahams's message of "Sat, 22 Mar 2003 13:48:00 -0500") User-Agent: Gnus/5.090017 (Oort Gnus v0.17) Emacs/21.3.50 (gnu/linux) Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:51096 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:51096 David Abrahams writes: >> For small data, the cause is much more likely to be server slowness or >> network latency, so please send try sending a 5MB attachment or >> something. > > OK, here: Function Name Call Count Elapsed Time Average Time ================================== ========== ============ ============ gnus-summary-reselect-current-group 1 8.132 8.132 gnus-summary-rescan-group 1 8.132 8.132 nnimap-request-group 2 6.43 3.215 imap-send-command-wait 14 6.339 0.4527857142 imap-wait-for-tag 14 6.339 0.4527857142 nnimap-request-update-info-internal 2 6.2989999999 3.1494999999 imap-search 11 5.927 0.5388181818 gnus-group-get-new-news-this-group 1 4.055 4.055 gnus-summary-read-group 1 3.586 3.586 gnus-group-read-group 1 3.586 3.586 gnus-summary-read-group-1 1 3.586 3.586 gnus-select-newsgroup 1 3.526 3.526 gnus-activate-group 1 3.465 3.465 gnus-request-group 1 2.965 2.965 gnus-retrieve-headers 2 0.58 0.29 ... It seems the IMAP SEARCH command is causing the delay (most of the nnimap related delay is in imap-search), suggesting that the server opens large messages when the SEARCH command is invoked. This is unnecessary, since Gnus only searches for flags (unseen, seen, ticked, etc). Perhaps you could forward this observation (after verifying it in the source) to the IMAP server administrator/developers to have them improve searches for flags. > But I still think the fact that I get the "imap read: ..." dance > counting up to 27K *twice* when I do `M-G' and receive no messages is > highly suspicious. I don't think so -- that is displayed when imap.el waits for feedback from the server. M-g causes many commands to be sent to the server, several of which are SEARCH commands (see *imap-log*), so if more than one command takes time, that message is displayed. >> To find out what causes slowness for small data, you must tcpdump or >> ethereal the stream to see which side is delaying the transfer, or if >> it is the network that has high latency. ELP data isn't that useful >> in this case. > > I got myself a copy of Ethereal and captured packets from a `M-G', but > I have no clue yet as to how to get any useful information from what > I'm seeing... > > ...I hate to keep harping on this, but it really has the look/feel of > some inefficiency in what GNUs is doing. > > Looking at the packets one more time it sure looks very much as though > GNUs is receiving the same sequence of packets (about 27K worth) twice > from the IMAP server, though it's pretty hard for me to tell > anything, especially peering through SSL encoding. Right, you need to get a protocol dump with timestamps of the IMAP connection. Gnus do send several SEARCH commands, and this is quite inefficient, but I believe fixing this would require changing the Gnus backend interface, and that means work. Several servers implement flag searching without opening all articles though, so many people don't have this problem.