From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/66223 Path: news.gmane.org!not-for-mail From: Daniel Pittman Newsgroups: gmane.emacs.gnus.general Subject: Re: Slowdowns with big nnmaildir groups Date: Sat, 02 Feb 2008 10:42:56 +1100 Organization: Cybersource: Australia's Leading Linux and Open Source Solutions Company Message-ID: <87k5loqm4f.fsf@rimspace.net> References: <87d4rhm5vv.fsf@member.fsf.org> <87zluk3mws.fsf@member.fsf.org> <87wsporh0k.fsf@rimspace.net> <8763x83hl9.fsf@member.fsf.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1201909956 2724 80.91.229.12 (1 Feb 2008 23:52:36 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 1 Feb 2008 23:52:36 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M14714@lists.math.uh.edu Sat Feb 02 00:52:58 2008 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1JL5gm-00086S-V1 for ding-account@gmane.org; Sat, 02 Feb 2008 00:52:57 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1JL5fr-0003xR-Oj; Fri, 01 Feb 2008 17:51:59 -0600 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1JL5fp-0003wz-Kc for ding@lists.math.uh.edu; Fri, 01 Feb 2008 17:51:57 -0600 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1JL5fi-00082I-UD for ding@lists.math.uh.edu; Fri, 01 Feb 2008 17:51:57 -0600 Original-Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1JL5fk-0007Ry-00 for ; Sat, 02 Feb 2008 00:51:52 +0100 Original-Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JL5fd-0004Ft-84 for ding@gnus.org; Fri, 01 Feb 2008 23:51:45 +0000 Original-Received: from ppp59-167-189-244.static.internode.on.net ([59.167.189.244]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 01 Feb 2008 23:51:45 +0000 Original-Received: from daniel by ppp59-167-189-244.static.internode.on.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 01 Feb 2008 23:51:45 +0000 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 60 Original-X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: ppp59-167-189-244.static.internode.on.net User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/23.0.60 (gnu/linux) Cancel-Lock: sha1:fuJrk4inGmPmzFTnEgaFgu/URDc= X-Spam-Score: -2.6 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:66223 Archived-At: Tassilo Horn writes: > Daniel Pittman writes: > >> You probably want to instrument nnmaildir, or possibly nn, as well as >> gnus. > > I did that (with nnmaildir) and the results are in > <87abmk3k1f.fsf@member.fsf.org>. > >> IIRC the nnmaildir code tries to play nice with other Maildir clients, >> which means reading all the (giant) filenames from disk, which >> probably works the poor kernel quite hard, especially with such a >> large directory. > > Indeed, the filenames are quite big, like: > > 1201848492_1.4393.baldur,U=10600,FMD5=72a62f382d4bdeb4a771c2d3ea493c5d:2,:2, > >> Ensuring your Maildir is on a filesystem that handles large >> directories well is probably the best bet for improving performance. > > Hm, it's on a ext3 filesystem. Maybe I can improve speed with the > dir_index option... It certainly won't hurt. Renewing the directory on a semi-regular basis, by creating a new one, moving all content to it, then replacing the original, will also help. > OfflineIMAP has the possibility to sync an IMAP group to a mbox > instead of a maildir, but I guess that wouldn't improve the situation. > Currently the emacs-devel folder is about 140M big, which would exceed > the buffer size limit on my 30Bit machine. Yes, it wouldn't do much good if it was bigger than your Emacs can read. For performance I would suggest one of two paths: 1. Use OfflineIMAP to sync IMAP to IMAP, and run a local IMAP server. This doesn't fix any performance problem with nnimap proper, but does insulate you from the performance hit of Maildir. (I recommend Dovecot for this purpose, if you must.) 2. Write an nnml (better) or nnmh (easier) back-end for OfflineIMAP, contribute it upstream, and use that. nnml with NOV support would be the best option, but there may be significant complexity in getting this to work coupled with renaming and moving messages between folders. It looks like their code is reasonably well structured and the basics of adding a new back-end will be easy. I don't think you will find the improvements you are looking for with the nnmaildir back-end, though I can't say that with absolute certainty. Regards, Daniel -- Daniel Pittman Phone: 03 9428 6922 1/130-132 Stawell St, Richmond Web: http://www.cyber.com.au Cybersource: Australia's Leading Linux and Open Source Solutions Company