From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/88562 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: George Clemmer Newsgroups: gmane.emacs.gnus.general Subject: Re: is nnmaildir very slow? Date: Thu, 16 May 2019 11:04:50 -0400 Message-ID: References: <87v9yb6k1z.fsf@ucl.ac.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="132452"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: mu4e 1.0; emacs 26.1 Cc: ding@gnus.org To: Eric S Fraga Original-X-From: ding-owner+M36765@lists.math.uh.edu Thu May 16 17:05:47 2019 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from lists1.math.uh.edu ([129.7.128.208]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hRHx3-000Y6X-Pn for ding-account@gmane.org; Thu, 16 May 2019 17:05:45 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.92) (envelope-from ) id 1hRHwQ-0005aA-2x; Thu, 16 May 2019 10:05:06 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by lists1.math.uh.edu with esmtps (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1hRHwJ-0005XM-Af for ding@lists.math.uh.edu; Thu, 16 May 2019 10:04:59 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92) (envelope-from ) id 1hRHwH-0006ch-NZ for ding@lists.math.uh.edu; Thu, 16 May 2019 10:04:59 -0500 Original-Received: from mail-qt1-f176.google.com ([209.85.160.176]) by quimby.gnus.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1hRHwE-0007ij-3F for ding@gnus.org; Thu, 16 May 2019 17:04:56 +0200 Original-Received: by mail-qt1-f176.google.com with SMTP id t1so4225431qtc.12 for ; Thu, 16 May 2019 08:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:in-reply-to:date :message-id:mime-version:content-transfer-encoding; bh=9vE76qg9tLJ5WgaqcI0Z91fSvwB+SIR1hg/J62D2OmI=; b=Y0uAA1OyJ1qqzYz3qNvMBAAgaPJGQAi5+O4Lk5sVjjVcFaSC8+gRobwlwTjDb7M2Lm Iimh9noPHz1KgAG3cfqOFHOgdAlcFCR3VQR/c9z+BQhfb2xXD5SPbyqStwPSbmBkzyjc P/G92WjwIfMF5mqir3TFPBAhXg9mc+KR/oDdxPyYqsrDL2YAUrrH3AmHuUJfIbAASzrp 2UwWtuT2OXTp8Ss7lRZg+6e4+iIJW4CWrUCmfu3Y6U3ljIoBiMgpQrN523YOsP9tqQfY UtLd/rPmQUxsMe2Do7WfkSJXEDR1t+yu1SdObwaQbDQFt1C6kh8GhWvOw5lvukWOnLEJ gwYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version:content-transfer-encoding; bh=9vE76qg9tLJ5WgaqcI0Z91fSvwB+SIR1hg/J62D2OmI=; b=T8rkJ+kjzPz/MLYqmoDF6pIEDLD+jPViwHi/tccCxG63HhXw7F5m3WrDfxdnsq2a24 LUhEzXTQVixZyCaYNyCu9P0hYtsXw9I6tHMWbPDjZa2H78/ZYaqZVCY8XF8WiXreu5/x fisN4O1jic+Wz45tmZpLTfGw4cPV9/CsqzC4ZWYXGB87J730r+WMERMfCBdrLVxCbQJD mBBnVD/fchwIafiWukJ2UHpi7ys+lVeZ0omUQUJm8PvTb3F908sx20ZMsBDKYt/nUCDb pd/B/cc8yNDKpDdxi+qZOlY6+vqNnFhMOkJdLjbO5nagUCjr48Rn/KMeoewq3tM7IJXr mSpA== X-Gm-Message-State: APjAAAVJWQcDSBFmD/RKAC4H1rWRwWbYu8qiOd92zqFjuWzlc4pc06j8 D/hTrAg+WBk9atocXyUeVPmkRS60 X-Google-Smtp-Source: APXvYqwfCZTqKVCpKiBoakADHUZlgLfTpkYVzXHOWXG2CWinU2YQ4eFHKXbIznlt+aDxnJFQYN5FUQ== X-Received: by 2002:ac8:3701:: with SMTP id o1mr38918596qtb.98.1558019092316; Thu, 16 May 2019 08:04:52 -0700 (PDT) Original-Received: from g1 (c-73-167-118-254.hsd1.ma.comcast.net. [73.167.118.254]) by smtp.gmail.com with ESMTPSA id v22sm3948545qtj.29.2019.05.16.08.04.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 16 May 2019 08:04:51 -0700 (PDT) In-reply-to: <87v9yb6k1z.fsf@ucl.ac.uk> List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:88562 Archived-At: Eric S Fraga writes: > Hello all, > > For a number of reasons, I have settled on nnmaildir for my main group > server. Unfortunately, however, it seems to be very slow at the initial > scan, taking up to several minutes within the gnus-request-scan function > (I added some gnus-message lines in gnus-int.el). I haven't figured out > yet why it is so slow. > > Does anybody else use maildir and see this? > > I am using emacs from git, updated yesterday. The group that takes > several minutes to scan has 8000 emails in it. > > Thanks. Hi Eric, In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against Maildir I found nnmaildir painfully slow to opening large Maildirs. AIUI, it is a known issue with nnmaildir. You may be interested in my post to gmane.emacs.gnus.general at the time re relative performance. I can't find it online, so I am attaching in below. FWIW, due to this issue, I have since abandoned emacs/gnus/nnmaildir and now read all my Maildir mail using emacs/mu4e. - George *** From: myglc2 Newsgroups: gmane.emacs.gnus.general Subject: Re: searching local mail stores Date: Tue, 06 Sep 2016 18:56:56 -0400 I received many helpful comments on my earlier post under this subject. I have since spent a few months off-and-on trying various mail search setups. Along the way I performed a couple benchmarks that might be of interest to list readers, so I am reporting them below. First, to get a handle on comments to the effect that nnmaildir is slow when there are many messages I put ~132,000 messages into 9 Maildir sub directories occupying 2.1G. I configured gnus to treat the top level maildir as a single store. On a 3.4 Ghz machine with a SSD, gnus took ~25 seconds to open the maildir. Initial notmuch indexing took a few minutes. Search performance depended on the number of search hits generated (see discussion below). For comparison, I deleted messages until the maildir had ~15,000 messages in 5 sub-directories occupying 1.1G, at which point gnus opened the maildir in ~5 seconds. Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e both use the xapian search/index engine and have emacs mail search UIs that are independent of gnus. I expected these to perform similarly and in casual comparison this seemed to be the case, so I did not compare the mu4e and notmuch UIs any further. However notmuch also supports the gnus nnir search interface, which allows a gnus 'G G' search to deliver notmuch search results to gnus summary buffers. To me, the appeal of notmuch was the possibility of fast search while otherwise continuing to read messages in gnus. So I compared gnus/notmuch with the mu4e UI, as shown in Table 1, below. Table 1: seconds to operate on a maildir directory containing 15,000 messag= es | operation | gnus/notmuch | mu4e | gnus/notmuch | mu4e (1) | | | first line | first line | All lines | All lines | +------------------+--------------+------------+--------------+-----------| | open | 5 | 1 | 5 | 10 | | (re)sort date | 1 | 1 | 1 | 10 | | (re)sort subject | 1 | 1 | 1 | 10 | Note: mu4e normally limits display to, at most, the first 500 search hits. In the mu4e (1) All lines results above, mu4e was forced to display all search hits by setting 'M-x mu4e-headers-toggle-full-search=E2= =80=99. So, to generalize, mu4e is snappier than gnus/notmuch, unless we force mu4e to display a lot of lines. The biggest difference is the "open" time, which, as demonstrated above, becomes significant when the maildir contains a lot of messages. SEARCH PERFORMANCE: When searching, if the number of search hits is modest (<100), the search/display time is similar for gnus/notmuch and mu4e. However, when a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to display all of the results and mu4e takes 23 (once again setting M-x mu4e-headers-toggle-full-search). So the most noticeable oveall difference between gnus/notmuch and mu4e is the time taken to initially "open" the maildir. Regarding setup, I found mu4e easier with its single point of configuration. In comparison, gnus/notmuch requires a compatible "parallel" configuration of notmuch and gnus. Bottom line: 1) gnus/notmuch/maildir works pretty well (at least on my computer) for 15,000 or so messages. 2) At 100,000 or messages, the gnus maildir startup delay is so tiresome that mu4e becomes truly compelling. FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to search and read mailing list archives that I have mirrored locally into maildir. - George