Huge groups

Huge groups

[Matt Simmons]

What are people doing with huge groups?  Case in point: my xemacs-beta
(nnml)group, which is currently approaching 10,000 messages (it's at
9995 now), and has a ~2.8M .overview file.  My poor P6-180 takes over
a minute just to open it[1], and about 10-15 seconds to append the
first message when inc'ing it's spool.

Matt

Footnotes: 
[1]  Solaris 2.5.1, XEmacs 20.3 MULE

-- 
     Matt Simmons  -  simmonmt@acm.org  -  http://www.netcom.com/~simmonmt
     People say that everyone has a few skeletons in their closet.  Not me.
    Well, not yet anyway.  I mean, the bodies are still decomposing.  --R.M.
				     Weiner

Re: Huge groups

[SL Baur]

Matt Simmons <simmonmt@acm.org> writes:

> What are people doing with huge groups?

I split them up periodically when the pain gets too bad.  If one renames
the original group to something else (for xemacs-beta I name them by
developmental version) and create a new group with the original name,
one's email filters continue to work.

There's some pain as threading gets lost for a time, but the renewed
speed of entry outweighs it for me.

Re: Huge groups

[Jason L Tibbitts III]

>>>>> "MS" == Matt Simmons <simmonmt@acm.org> writes:

MS> What are people doing with huge groups? 

Note that the list archive has a 3.5MB overview file and has 15787 messages
in it (not including this one).  I don't think anyone's foolish enough to
try and open it (C-u G A, I think) but if they did they'd be watching the
FTP clicker counting up for quite a while.

A long time ago we talked about a backend like nnml but with subdirectories
holding chunks of messages.  It never got off the ground.

 - J<

Re: Huge groups

[Steinar Bang]

>>>>> Jason L Tibbitts III <tibbs@hpc.uh.edu>:

>>>>> "MS" == Matt Simmons <simmonmt@acm.org> writes:
MS> What are people doing with huge groups? 

> Note that the list archive has a 3.5MB overview file and has 15787 messages
> in it (not including this one).  I don't think anyone's foolish enough to
> try and open it (C-u G A, I think) but if they did they'd be watching the
> FTP clicker counting up for quite a while.

> A long time ago we talked about a backend like nnml but with subdirectories
> holding chunks of messages.  It never got off the ground.

A faster backend? Are there some formats that could be used? Isn't the
qmail native format supposed to be fast? (at least according to Dan
Bernstein...) 

How about whatever the Cyrus IMAP server uses?

Re: Huge groups

[Jason L Tibbitts III]

>>>>> "SB" == Steinar Bang <sb@metis.no> writes:

SB> A faster backend? Are there some formats that could be used? Isn't the
SB> qmail native format supposed to be fast? (at least according to Dan
SB> Bernstein...)

Well, I think it would be hard to get much faster than nnml.  You could
have something like nnfolder but keep an overview file that stores byte
offsets within the folder for each message, so it's one open and one seek
to grab a message.  (I use this format in an archiver I'm working with.)

But the question here is how to handle tens of thousands of messages, when
even the overhead of loading the overview file is too much.  The solution
is, of course, a database, but a simpler solution would be to just split
the messages up into smaller collections and optimize the common operations
to only deal with one piece of the collection.

I suppose this is easy enough to do manually in the cases where it is
really needed.  The only place where it can't be done is with the list
archive (which is why I brought the idea up ages ago).  The solution to
that was the ding-list-recent archive, accomplished through procmail
trickery.

 - J<

Re: Huge groups

[Stefan Waldherr]

>>>>> "Jason" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:

>>>>> "SB" == Steinar Bang <sb@metis.no> writes:
  SB> A faster backend? Are there some formats that could be used? Isn't the
  SB> qmail native format supposed to be fast? (at least according to Dan
  SB> Bernstein...)

  Jason> Well, I think it would be hard to get much faster than nnml.  You
  Jason> could have something like nnfolder but keep an overview file that

I think that nnml is pretty darn slow wrt speed. Selecting a group with, say,
800 articles takes 11s on my machines. This is XEmacs, not Emacs (it seems to
be a little faster under Emacs). Exmh on the other hand, which I used
previously, takes for the same group far less than 5s. I would really like to
find out if other people have equivalent times ...

cu
Stefan.
-- 
Stefan Waldherr                office +1 (412) 268-3837
                                  fax +1 (412) 268-5576
                               e-Mail swa@cs.cmu.edu
                                  www http://www.waldherr.org/

Re: Huge groups

[Hrvoje Niksic]

Stefan Waldherr <swa@cs.cmu.edu> writes:

> >>>>> "Jason" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:
> 
> >>>>> "SB" == Steinar Bang <sb@metis.no> writes:
>   SB> A faster backend? Are there some formats that could be used? Isn't the
>   SB> qmail native format supposed to be fast? (at least according to Dan
>   SB> Bernstein...)
> 
>   Jason> Well, I think it would be hard to get much faster than nnml.  You
>   Jason> could have something like nnfolder but keep an overview file that
> 
> I think that nnml is pretty darn slow wrt speed. Selecting a group
> with, say, 800 articles takes 11s on my machines. This is XEmacs,
> not Emacs (it seems to be a little faster under Emacs). Exmh on the
> other hand, which I used previously, takes for the same group far
> less than 5s. I would really like to find out if other people have
> equivalent times ...

This has nothing with the merits of nnml compared to speed of other
formats.  The overhead you see is due to the speed (slowness) of Emacs 
Lisp, and various display bottlenecks.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
* Q: What is an experienced Emacs user?
* A: A person who wishes that the terminal had pedals.

Re:

[jari.aalto]

 98-01-21 Stefan Waldherr <swa@cs.cmu.edu> list.ding
| >>>>> "Jason" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:
| I think that nnml is pretty darn slow wrt speed. Selecting a group with, say,
| 800 articles takes 11s on my machines. This is XEmacs

I was also wondering if gnus could keep the most read mail group summaries 
in the emacs and not build them from scratch when I change mail groups.
I use non-compiled version and it takes 10-25 seconds to view mail group 
of 600+ messages.

I'm not complain about the speed, because I use uncompiled lisp on purpose.
I was just wondering if there would be ways to "cache" summary threads.
I don't know how they are implemented, so it could be very well 
impossible to change the existing summary creation code.

jari

Re:

[Lars Magne Ingebrigtsen]

<jari.aalto@poboxes.com> writes:

> I was also wondering if gnus could keep the most read mail group summaries 
> in the emacs and not build them from scratch when I change mail groups.

No.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Huge groups

[Jan Vroonhof]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> A long time ago we talked about a backend like nnml but with subdirectories
> holding chunks of messages.  It never got off the ground.

XEmacs at least can talk natively to a DBM type database library.
Wouldn't that be ideal?

Jan

Re: Huge groups

[Jason L Tibbitts III]

>>>>> "JV" == Jan Vroonhof <vroonhof@math.ethz.ch> writes:

JV> XEmacs at least can talk natively to a DBM type database
JV> library. Wouldn't that be ideal?

Depends on whether you ever want to see your mail in some place other than
Gnus.  I for one like to use glimpse and grep over my stored mail.  Not
really possible with a binary database.

 - J<

Re: Huge groups

[Steinar Bang]

>>>>> Jason L Tibbitts III <tibbs@hpc.uh.edu>:

>>>>> "JV" == Jan Vroonhof <vroonhof@math.ethz.ch> writes:
JV> XEmacs at least can talk natively to a DBM type database
JV> library. Wouldn't that be ideal?

> Depends on whether you ever want to see your mail in some place other than
> Gnus.  I for one like to use glimpse and grep over my stored mail.  Not
> really possible with a binary database.

But you could use DBM files instead of an .overview file, and maybe
get a speedup from that?  If that's where Gnus spends it time, at
least.

(I suspect that with the current backend interface this won't give any
speedups, since Gnus needs to hold the entire Summary buffer
information in memory to do the threading and scoring and stuff. And
even without doing any sort of profiling I'm gueesing that there is a
performance bottle neck here)

Re: Huge groups

[Lars Syrstad]

* Steinar Bang
| 
| But you could use DBM files instead of an .overview file, and maybe
| get a speedup from that?  If that's where Gnus spends it time, at
| least.
| 
| (I suspect that with the current backend interface this won't give any
| speedups, since Gnus needs to hold the entire Summary buffer
| information in memory to do the threading and scoring and stuff. And
| even without doing any sort of profiling I'm gueesing that there is a
| performance bottle neck here)

Any reason why the necessary information to do threading and scoring
could not go into the Overview-file, which in turns goes into a DBM
file?

nnml is much more static than nntp, and Gnus _knows_ when a nnml-group
receives new articles or when old articles are expired, since it's
Gnus itself which performs these actions.  If some effort was made to
thread the group once-and-for-all when articles arrive and disappear,
building the summary buffer ought to be faster.

(Not that I know much about the internals of nnml, mind you.)

  - Lars.
-- 
Fly like a rock from the roof to the basement
The last thing to go through my mind is the pavement
(Falling out of love with life)
    -- Skyclad: Helium

Re: Huge groups

[Steinar Bang]

>>>>> Lars Syrstad <lars@no482rs1.aventura.no>:

>> (I suspect that with the current backend interface this won't give any
>> speedups, since Gnus needs to hold the entire Summary buffer
>> information in memory to do the threading and scoring and stuff. And
>> even without doing any sort of profiling I'm gueesing that there is a
>> performance bottle neck here)

> Any reason why the necessary information to do threading and scoring
> could not go into the Overview-file, which in turns goes into a DBM
> file?

The necessary information is there already.  I just think that the
bottleneck is after this information has been turned into lisp lists
so that Gnus can process it, not in the file format itself, and not in
the reading and the parsing of the .overview file format.

But I haven't profiled anything, so I'm just guessing at this.

Re: Huge groups

[Lars Syrstad]

* Steinar Bang
| >>>>> Lars Syrstad <lars@no482rs1.aventura.no>:
|
| > Any reason why the necessary information to do threading and scoring
| > could not go into the Overview-file, which in turns goes into a DBM
| > file?
| 
| The necessary information is there already.  I just think that the
| bottleneck is after this information has been turned into lisp lists
| so that Gnus can process it, not in the file format itself, and not in
| the reading and the parsing of the .overview file format.

Yes, but couldn't the lisp lists be stored?  Why build them from
scratch each time the group is being visited, when Gnus knows whenever
an article enters or leaves the group and could update the list then?

  - Lars.
-- 
Fly like a rock from the roof to the basement
The last thing to go through my mind is the pavement
(Falling out of love with life)
    -- Skyclad: Helium

Re: Huge groups

[Steinar Bang]

>>>>> Lars Syrstad <lars@no482rs1.aventura.no>:

> Yes, but couldn't the lisp lists be stored?  Why build them from
> scratch each time the group is being visited, when Gnus knows whenever
> an article enters or leaves the group and could update the list then?

You would have to build the actual data structures in memory,
whichever way you go.  I'm *guessing* (note! I do *not* know) that
this would be the bottleneck.

Re: Huge groups

[Lars Magne Ingebrigtsen]

Lars Syrstad <lars@no482rs1.aventura.no> writes:

> | The necessary information is there already.  I just think that the
> | bottleneck is after this information has been turned into lisp lists
> | so that Gnus can process it, not in the file format itself, and not in
> | the reading and the parsing of the .overview file format.
> 
> Yes, but couldn't the lisp lists be stored?

The threading is done via obarrays, which do not have a read syntax.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Huge groups

[David Aspinwall]

>>"sb" == Steinar Bang <sb@metis.no> writes:

> But you could use DBM files instead of an .overview file, and maybe
> get a speedup from that?  If that's where Gnus spends it time, at
> least.

I don't think DBM files work on microsoft OSes, since they don't
support hole-y files.

-- 
David Aspinwall		aspin@EnsembleSolutions.com