From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/68525 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.gnus.general Subject: Re: Abandoning the concept of groups as a storage medium? Date: Fri, 08 May 2009 13:18:33 -0500 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <86bpq3h30m.fsf@lifelogs.com> References: <86zldyhrgy.fsf@lifelogs.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1241806436 2808 80.91.229.12 (8 May 2009 18:13:56 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 8 May 2009 18:13:56 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M16958@lists.math.uh.edu Fri May 08 20:13:46 2009 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1M2UZr-000699-HA for ding-account@gmane.org; Fri, 08 May 2009 20:13:43 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1M2UZg-0002fd-3L; Fri, 08 May 2009 13:13:32 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1M2UZe-0002fB-H3 for ding@lists.math.uh.edu; Fri, 08 May 2009 13:13:30 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.69) (envelope-from ) id 1M2UZY-0002Da-Nt for ding@lists.math.uh.edu; Fri, 08 May 2009 13:13:30 -0500 Original-Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by quimby.gnus.org with esmtp (Exim 3.36 #1 (Debian)) id 1M2Ua0-0000mE-00 for ; Fri, 08 May 2009 20:13:52 +0200 Original-Received: from list by ciao.gmane.org with local (Exim 4.43) id 1M2UZX-0002mQ-3L for ding@gnus.org; Fri, 08 May 2009 18:13:23 +0000 Original-Received: from 38.98.147.130 ([38.98.147.130]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 May 2009 18:13:23 +0000 Original-Received: from tzz by 38.98.147.130 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 May 2009 18:13:23 +0000 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 56 Original-X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 38.98.147.130 X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6;d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.0.93 (gnu/linux) Cancel-Lock: sha1:W3UFYf7yWDe19PLSgMBwg8AcuQs= X-Spam-Score: -0.7 (/) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:68525 Archived-At: On Mon, 04 May 2009 13:07:20 +0200 David Engster wrote: DE> I think the registry should save all important headers of a message, and DE> maybe also some MIME information, like attachment names. Of course, as DE> Ted also said, this information can't be saved anymore in plain text DE> files like gnus.registry.eld, but needs some kind of external database DE> back end. You're probably thinking of saving the keywords too, yes? Generally any key-val map should be storeable; this is ridiculously easy with proper hashtable serialization, which is coming in Emacs CVS once the pretest is over. I have a patch ready to go. With that serialization, we can finally write out the registry whole or piece by piece in one simple command, and read it back just as easily. With an IMAP storage backend for the registry, this could result in registry entries stored in messages like this: Subject: [entry subject] Keywords: [entry keywords] Message-ID: [entry ID].gnus.registry (so it doesn't collide with the original) ... other headers ... [whole serialized entry as the body] so then we could do IMAP SEARCH by keyword, for example. >> nnregistry? >> group list: dynamic based on tags (labels) defined by user >> article list in group: generated on entry with a tag search >> article retrieve: uses the original article backend DE> Exactly. What makes this stuff difficult is that article numbers in Gnus DE> have to be unique, just like in IMAP. nnmairix "solves" this problem by DE> adding an offset to the article numbers each time a group is re-created. I wonder if it would be possible to use message IDs as article numbers. DE> However, full text search is another matter entirely. This simply cannot DE> be done in Emacs Lisp. >> >> The index necessary for good search performance would be huge, but easy >> to store in a database (on a server, on IMAP, whatever). It's easy to >> parallelize these searches (especially with IMAP as the backend). So >> there's some hope. DE> Maybe. But just feeding such a database through Emacs Lisp would already DE> be a very time consuming task, considering the huge amount of mails many DE> people have (I have about 1GB, and I think this is pretty average). I think it's possible, as long as we keep the indexing and searching on the server, and Emacs Lisp is only the query and display agent. Displaying large amounts of data is not so bad if you can get it 50 results at a time... Ted