From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/42522 Path: main.gmane.org!not-for-mail From: Zlatko Calusic Newsgroups: gmane.emacs.gnus.general Subject: Re: db-backed mail back end Date: Thu, 24 Jan 2002 14:50:03 +0100 Sender: owner-ding@hpc.uh.edu Message-ID: References: Reply-To: zlatko.calusic@iskon.hr NNTP-Posting-Host: coloc-standby.netfonds.no Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: main.gmane.org 1035177752 12273 80.91.224.250 (21 Oct 2002 05:22:32 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Mon, 21 Oct 2002 05:22:32 +0000 (UTC) Cc: ding@gnus.org Return-Path: Original-Received: (qmail 11161 invoked from network); 24 Jan 2002 13:53:24 -0000 Original-Received: from malifon.math.uh.edu (mail@129.7.128.13) by mastaler.com with SMTP; 24 Jan 2002 13:53:24 -0000 Original-Received: from sina.hpc.uh.edu ([129.7.128.10] ident=lists) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 16TkJM-0005U0-00; Thu, 24 Jan 2002 07:53:04 -0600 Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Thu, 24 Jan 2002 07:52:59 -0600 (CST) Original-Received: from sclp3.sclp.com (qmailr@sclp3.sclp.com [209.196.61.66]) by sina.hpc.uh.edu (8.9.3/8.9.3) with SMTP id HAA15903 for ; Thu, 24 Jan 2002 07:52:46 -0600 (CST) Original-Received: (qmail 11150 invoked by alias); 24 Jan 2002 13:52:45 -0000 Original-Received: (qmail 11145 invoked from network); 24 Jan 2002 13:52:45 -0000 Original-Received: from inje.iskon.hr (213.191.128.16) by gnus.org with SMTP; 24 Jan 2002 13:52:45 -0000 Original-Received: from magla.zg.iskon.hr (oganj.iskon.hr [213.191.128.21]) by mail.iskon.hr (8.11.4/8.11.4/Iskon 8.11.3-1) with ESMTP id g0ODo4M11262; Thu, 24 Jan 2002 14:50:16 +0100 (MET) Original-Received: from magla.zg.iskon.hr (zcalusic@localhost [127.0.0.1]) by magla.zg.iskon.hr (8.12.1/8.12.1/Debian -5) with ESMTP id g0ODo4KY005606; Thu, 24 Jan 2002 14:50:04 +0100 Original-Received: (from zcalusic@localhost) by magla.zg.iskon.hr (8.12.1/8.12.1/Debian -5) id g0ODo3Te005603; Thu, 24 Jan 2002 14:50:03 +0100 Original-To: Karl Kleinpaste X-Face: s71Vs\G4I3mB$X2=P4h[aszUL\%"`1!YRYl[JGlC57kU-`kxADX}T/Bq)Q9.$fGh7lFNb.s i&L3xVb:q_Pr}>Eo(@kU,c:3:64cR]m@27>1tGl1):#(bs*Ip0c}N{:JGcgOXd9H'Nwm:}jLr\FZtZ pri/C@\,4lW<|jrq^<):Nk%Hp@G&F"r+n1@BoH In-Reply-To: (Karl Kleinpaste's message of "Thu, 24 Jan 2002 06:58:48 -0500") Original-Lines: 30 User-Agent: Gnus/5.090005 (Oort Gnus v0.05) XEmacs/21.4 (Common Lisp, i386-debian-linux) Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:42522 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:42522 Karl Kleinpaste writes: > Simon Josefsson writes: >> NNML scales linearly with the number of messages in the group, does >> it not? Gnus definitely doesn't scale linearly, so redesigning >> something to get support for large mail/news backends should go into >> Gnus, I think. > > I have been wondering about this since this discussion started. The > slowness of entering a large group of 10K messages, or 100K, has very > little to do with getting at either the overview data or the message > files. It has to do with threading, scoring, and sorting. I don't > know what Gnus' threading algorithm is, but I suspect its performance > is at least as poor as O(n log n) and may even be as bad as O(n^2). It is definitely O(n^2), I was measuring it recently. I have an nnml group (ok, linux-kernel list, why keep it secret :)) which hovers around the number of 15000 unread mails (12k - 17k typicaly, last few months). It takes minutes! to enter that group on a quite fast machine with a gobs of memory. It's no mystery I'm not eager to start reading that list, and tomorrow things get only worse. :) My observings are that threading and sorting take a lot of time (not scoring!). And I found it is exactly O(n^2). If I read, kill, whatever... 30% of articles in the group, elapsed time during entering the group cuts in half. I was delibaretely measuring it to see what is the upper limit of usability. Quite lowish, if I may add. -- Zlatko