From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/66906 Path: news.gmane.org!not-for-mail From: Dan Christensen Newsgroups: gmane.emacs.gnus.general Subject: nnfolder slow to append to large groups Date: Sat, 03 May 2008 20:55:08 -0400 Message-ID: <87ve1ukhw3.fsf@uwo.ca> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1209862620 30195 80.91.229.12 (4 May 2008 00:57:00 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 4 May 2008 00:57:00 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M15386@lists.math.uh.edu Sun May 04 02:57:33 2008 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1JsSXk-0005LJ-P3 for ding-account@gmane.org; Sun, 04 May 2008 02:57:33 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1JsSVr-0008DK-JT; Sat, 03 May 2008 19:55:35 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1JsSVq-0008D7-7F for ding@lists.math.uh.edu; Sat, 03 May 2008 19:55:34 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1JsSVj-0006A9-PJ for ding@lists.math.uh.edu; Sat, 03 May 2008 19:55:34 -0500 Original-Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1JsSVr-000656-00 for ; Sun, 04 May 2008 02:55:35 +0200 Original-Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JsSVc-0000cd-Al for ding@gnus.org; Sun, 04 May 2008 00:55:20 +0000 Original-Received: from bas3-london14-1096790663.dsl.bell.ca ([65.95.178.135]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 04 May 2008 00:55:20 +0000 Original-Received: from jdc by bas3-london14-1096790663.dsl.bell.ca with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 04 May 2008 00:55:20 +0000 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 56 Original-X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: bas3-london14-1096790663.dsl.bell.ca User-Agent: Gnus/5.110009 (No Gnus v0.9) Emacs/23.0.60 (gnu/linux) Cancel-Lock: sha1:1ZioKD5h8u8KtUYTiJtX/eN0sFw= X-Spam-Score: -2.6 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:66906 Archived-At: I use nnfolder to archive messages, and I have one group in particular that I regularly move messages to whose file is almost 13MB in size. Each time I move a message there, it takes 2 or 3 seconds until I can use emacs again, so I'm wondering whether this can be sped up at all. To be precise, if I move a message from an otherwise empty group, it takes about 1.5 to 2 seconds. If I move from a group which is 1MB in size, it take 2.5 to 3 seconds. This is with everything in cache. Just editing the big file directly in emacs and saving a changed version takes about 1 second. Copying the file with "cp bigfile bigfile2; sync" takes 1.04s elapsed time, roughly in line with saving the file in emacs. Without the sync, it is 0.07s. As another data point, appending to a file this large with, e.g., "echo foo >> bigfile; sync" takes 0.018s elapsed time. All of these are faster than the move using Gnus. First, my question: why should the move take any longer than about 1.04s when I am moving from an otherwise empty group? Resaving the file for the empty group should be instantaneous, so it seems like Gnus is adding some overhead somewhere. Any ideas on how to fix this? ... Hmm, it looks to me like Gnus kills the buffer for the file when not using it, and so the file needs to be reread between operations. Is this true?? If so, is it intentional? Other ideas for improvement: 1) Telling emacs not to call fsync by setting write-region-inhibit-fsync to t helps a lot, but it still takes about 1 second. And while I would be comfortable having this set to t for mail files, I'm not sure about setting it generally. Would an option like nnfolder-inhibit-fsync which tells nnfolder-save-buffer to temporarily set write-region-inhibit-fsync be a horrible idea? 2) Even better would be if emacs could signal to the OS to sync the file, but to do so asynchronously. Is this possible? 3) Another minor factor is that since gnus seems to kill the buffer between uses, a backup file is created each time. But I don't think this causes much overhead, since in my tests emacs saves a file in about the same amount of time whether or not a backup is being updated. 4) The operation that is happening here is an append for the destination group (which is the large one). In my tests, append-to-file is extremely fast. Could nnfolder be taught to use this for the destination group?? Backups wouldn't be made in this case, but maybe that's ok, or maybe nnfolder could do a backup if it's the first change to the file during this session. This last idea would be a huge speed-up. Thoughts? Which function is it that is doing the insertion of the new article? Dan