From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/68517 Path: news.gmane.org!not-for-mail From: David Engster Newsgroups: gmane.emacs.gnus.general Subject: Re: Abandoning the concept of groups as a storage medium? Date: Mon, 04 May 2009 13:07:20 +0200 Message-ID: References: <86zldyhrgy.fsf@lifelogs.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1241435536 18336 80.91.229.12 (4 May 2009 11:12:16 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 4 May 2009 11:12:16 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M16950@lists.math.uh.edu Mon May 04 13:12:05 2009 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1M0w1w-00023D-SS for ding-account@gmane.org; Mon, 04 May 2009 13:08:17 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1M0w1F-0007w2-3U; Mon, 04 May 2009 06:07:33 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1M0w1C-0007vY-VQ for ding@lists.math.uh.edu; Mon, 04 May 2009 06:07:30 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.69) (envelope-from ) id 1M0w15-0000FT-KO for ding@lists.math.uh.edu; Mon, 04 May 2009 06:07:30 -0500 Original-Received: from m61s02.vlinux.de ([83.151.21.164]) by quimby.gnus.org with esmtp (Exim 3.36 #1 (Debian)) id 1M0w1W-0006yQ-00 for ; Mon, 04 May 2009 13:07:50 +0200 Original-Received: from kafka.physik3.gwdg.de ([134.76.92.48] helo=kafka) by m61s02.vlinux.de with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.69) (envelope-from ) id 1M0w13-0002tG-LK for ding@gnus.org; Mon, 04 May 2009 13:07:21 +0200 In-Reply-To: <86zldyhrgy.fsf@lifelogs.com> (Ted Zlatanov's message of "Thu, 30 Apr 2009 14:27:41 -0500") User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.0.90 (gnu/linux) Mail-Copies-To: never Mail-Followup-To: ding@gnus.org X-Spam-Score: -2.6 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:68517 Archived-At: Ted Zlatanov writes: > On Tue, 28 Apr 2009 10:56:39 +0200 David Engster wrote: > > DE> Jan Rychter writes: >>> I want to be able to set flags and tags on *any* E-mail message >>> *anywhere*, not just in the "real group it belongs to". I don't want >>> my "groups" to have anything to do with the way my messages are >>> stored. E-mail should be stored in a key/value store with metadata >>> copied and indexed separately. >>> >>> The problem I see with Gnus is that it is designed around a central >>> concept of a mail backend which exposes groups. > > DE> Yes, but I don't see this as a restriction for what you would like to > DE> have. Gnus can handle "dynamic" groups, where the contents changes all > DE> the time, although it requires a lot of work to get right. As you say, > DE> nnmairix comes pretty close, and it strictly works with the Gnus group > DE> back end API. My main problem with maintaining nnmairix is that the back > DE> ends behave differently, especially when it comes to marks and unread > DE> count. > > The big problem I see is that Gnus can't build groups asynchronously. > Emacs Lisp itself is the impediment here. That's why it better has to build those groups fast. ;-) But you're right, of course. There has been some work on introducing parallelism into Emacs Lisp on emacs-devel. Maybe Emacs 24 will have something like that... > DE> I think the registry should save all important headers of a message, and > DE> maybe also some MIME information, like attachment names. Of course, as > DE> Ted also said, this information can't be saved anymore in plain text > DE> files like gnus.registry.eld, but needs some kind of external database > DE> back end. > > DE> We could then add a back end which can create virtual groups based on > DE> registry information. One could extend nnir to do that, but I'd vote for > DE> creating a completely new one. > > nnregistry? Makes sense. :-) > group list: dynamic based on tags (labels) defined by user > article list in group: generated on entry with a tag search > article retrieve: uses the original article backend Exactly. What makes this stuff difficult is that article numbers in Gnus have to be unique, just like in IMAP. nnmairix "solves" this problem by adding an offset to the article numbers each time a group is re-created. > I'm definitely not going to get to it anytime soon, but if anyone else > feels adventurous, I'll help out any way I can. My guess is that most of the needed code already exists in nnir and nnmairix. Unfortunately, I won't have time to do it either, at least not in the coming months. > DE> However, full text search is another matter entirely. This simply cannot > DE> be done in Emacs Lisp. > > The index necessary for good search performance would be huge, but easy > to store in a database (on a server, on IMAP, whatever). It's easy to > parallelize these searches (especially with IMAP as the backend). So > there's some hope. Maybe. But just feeding such a database through Emacs Lisp would already be a very time consuming task, considering the huge amount of mails many people have (I have about 1GB, and I think this is pretty average). -David