many articles

many articles

[Phil Humpherys]

I have a group with 11000 email messages in it.  I'm running on a
machine with 32M of ram and 128M of memory...  I invoked the
summary buffer of the group and it took an hour....  thrashing my
machine horribly.

Is it not a good idea to have so many articles in a group?  It's
just killed my machine...

--
Phil Humpherys <phumpherys@utah-inter.net>   DriverSoft
Unix Systems Administrator                   Mobile: +1.801.725.3257 
WWW/PGPkeys: http://www.spire.com/~humphery

Re: many articles

[Matt Simmons]

>>>>> "Phil" == Phil Humpherys <phumpherys@utah-inter.net> writes:

    Phil> I have a group with 11000 email messages in it.  I'm running on a
    Phil> machine with 32M of ram and 128M of memory...  I invoked the
    Phil> summary buffer of the group and it took an hour....  thrashing my
    Phil> machine horribly.

    Phil> Is it not a good idea to have so many articles in a group?  It's
    Phil> just killed my machine...

Only 11000 messages?  Feh..

[simmonmt@auckland]:1:39pm:~/Mail> wc -l xemacs/.overview
   21849 xemacs/.overview

... over NFS ... from an IPX.

Think of it as your computer's way of telling you to go do something
else for a while. =)

Of course, 32M (I assume the 32M above is the chip RAM, and the 128M
is swap) is basically a joke for running XEmacs (I assume the same is
true for GNU Emacs) under X, to say nothing of groups with tens of
thousands of messages.  I have 224M, and it still kills me.  Of
course, it's not swapping, but it still has a mountain of data to chew
through[1].  According to top, the footprint of my current XEmacs is 43M
(though only 27M is swapped in at the moment).  You really should go
buy more RAM if you'd like to keep your sanity.

Matt

Footnotes: 
[1]  Look at the size of your .overview file.  Mine is 6.2M, so yours
     is probably around 3.1M.  I don't know exactly how Gnus figures
     out threads, but I'd assume it's reading the entire thing in (and 
     converting it to an internal data structure) at least once.

-- 
     Matt Simmons  -  simmonmt@acm.org  -  http://www.netcom.com/~simmonmt
    You know when people see a cat's litter box, they always say, "Oh, have
      you got a cat?"  Just once I wanted to say, "No, it's for company!"

Re: many articles

[Phil Humpherys]

Matt Simmons <simmonmt@acm.org> writes:

> Of course, 32M (I assume the 32M above is the chip RAM, and the
> 128M is swap) is basically a joke for running XEmacs (I assume
> the same is true for GNU Emacs) under X, to say nothing of
> groups with tens of thousands of messages.  I have 224M, and it
> still kills me.  Of course, it's not swapping, but it still has
> a mountain of data to chew through[1].

Matt, thanks for your response.  Yes, I recognize that a "mere"
32M of ram isn't very much anymore.  It is sufficient for all my
other tasks running emacs.  It's only on these very large groups
that it kinda dies.

Now, if I were to run mh (or even mh-e), I'd have absolutely no
problems with these large groups (or folders as the case may be).
Hence, my complaint.

One soolution I can think of is to run gnus with an mh backend,
and when I needto do something with these really large groups, I
bail on [x]emacs and go straight mh.  Then, I can always come
back to xemacs later whenever I want.

I think I'd like to explore this further, but there's very little
documentation I can find on gnus specifics with the mh backend.
If anyone one there has some experience here I'd appreciate some
feedback.  The examples in the gnus manual are very sparse.

> According to top, the footprint of my current XEmacs is 43M
> (though only 27M is swapped in at the moment).  You really should go
> buy more RAM if you'd like to keep your sanity.

Well, this may be an option down the road, but for now it simply
isn't.  Instead, I'm going to have to modify how I interact with
my email somewhat.  I'm dedicated to staying with gnus, but I'm
going to have make it work with mh as I've described above.

Thanks again for your response.  I'd appreciate any further help
or suggestions.

--
Phil Humpherys <phumpherys@utah-inter.net>   DriverSoft
Unix Systems Administrator                   Mobile: +1.801.725.3257 
WWW/PGPkeys: http://www.spire.com/~humphery

Re: many articles

[Harry Putnam]

Phil Humpherys <phumpherys@utah-inter.net> writes:

> Well, this may be an option down the road, but for now it simply
> isn't.  Instead, I'm going to have to modify how I interact with
> my email somewhat.  I'm dedicated to staying with gnus, but I'm
> going to have make it work with mh as I've described above.
> 
> Thanks again for your response.  I'd appreciate any further help
> or suggestions.
Another course would be to break the monster groups down by mnth or
whatever into several nnml groups.

Or you could setup a tempory split to split them into nnfolder groups
quite quickly.  This *should* generate the groups named in the split.  

Something like this should do it.  Evaluate the temp split, then enter
the group and process mark every thing you want split out (From the
top of summary buffer do C-u 11,000 `#' or whatever number) Then press 
`B r' (respool) and choose `nnfolder' as the backend to do the
respooling.  Then step back and watch it fly...

(beware untested)
;;temp split
(setq nnmail-split-methods
      '(("group-mnth1" "^Date:.*Jan.*1997")
          ("group-mnth2" "^Date:.*Feb.*1997")
          ("group-mnth3" "Date.*Mar.*1997")
            ("group-mnth4" "^Date:.*Apr.*1997")))

etc etc

-- 
Harry Putnam reader@newsguy.com  
Running Redhat Linux-5.1

Re: many articles

[Kai Grossjohann]

>>>>> On 19 Sep 1998, Phil Humpherys said:

  Phil> One soolution I can think of is to run gnus with an mh backend,

Don't do that.

  Phil> and when I needto do something with these really large groups, I
  Phil> bail on [x]emacs and go straight mh.  Then, I can always come
  Phil> back to xemacs later whenever I want.

nnml is just like nnmh, only with .overview files ==> *much* faster!
nnmh will need to read every file in the directory to assemble summary
information, nnml just reads the .overview file.

You could switch just that one group to nnmh.  Or you could use MH on
nnml folders, IF YOU TAKE CARE TO DO IT READ-ONLY.

kai
-- 
OOP: object oriented programming;  OOPS: object oriented mistakes

Re: many articles

[Matt Simmons]

>>>>> "Kai" == Kai Grossjohann <grossjohann@amaunet.cs.uni-dortmund.de> writes:

>>>>> On 19 Sep 1998, Phil Humpherys said:

    Phil> and when I needto do something with these really large groups, I
    Phil> bail on [x]emacs and go straight mh.  Then, I can always come
    Phil> back to xemacs later whenever I want.

    Kai> nnml is just like nnmh, only with .overview files ==> *much* faster!
    Kai> nnmh will need to read every file in the directory to assemble summary
    Kai> information, nnml just reads the .overview file.

This could be his problem.  Phil - what was the backend for the group
that caused your machine to eat itself?

Back in the (thankfully past) mists of time, when my XEmacs spool was
just over 10000 messages, I was using a 32M Intel machine to read my
mail.[1]  It never took more than 2 El stops (5-10 minutes) for it to
pop up the summary buffer.  This was with nnml.  Judging from your
other posts, and from the symptoms you described, it sounds like
you're still using nnmh.  You really really should consider
switching to nnml.  This is a reversible step if you decide that you
hate nnml.

Matt

Footnotes: 
[1]  Of course, I also used the same machine, but with 16M, but I've
     tried to suppress that particularly painful memory.

-- 
     Matt Simmons  -  simmonmt@acm.org  -  http://www.netcom.com/~simmonmt
    You know when people see a cat's litter box, they always say, "Oh, have
      you got a cat?"  Just once I wanted to say, "No, it's for company!"

Re: many articles

[Phil Humpherys]

Matt Simmons <simmonmt@acm.org> writes:

> This could be his problem.  Phil - what was the backend for the
> group that caused your machine to eat itself?

nnml.  ;)

> Back in the (thankfully past) mists of time, when my XEmacs
> spool was just over 10000 messages, I was using a 32M Intel
> machine to read my mail.[1] It never took more than 2 El stops
> (5-10 minutes) for it to pop up the summary buffer.  This was
> with nnml.  Judging from your other posts, and from the
> symptoms you described, it sounds like you're still using nnmh.

Nope.  It took 30 minutes....  running nnml

--
Phil Humpherys <phumpherys@utah-inter.net>   DriverSoft
Unix Systems Administrator                   Mobile: +1.801.725.3257 
WWW/PGPkeys: http://www.spire.com/~humphery

Re: many articles

[Kai Grossjohann]

>>>>> On 20 Sep 1998, Phil Humpherys said:

  Phil> Nope.  It took 30 minutes....  running nnml

Try setting gnus-verbose to something high then tell us what Gnus is
doing all the time.  If you have something in some hook that sorts the
summary by date, for example, that could account for the time -- Gnus
needs a long time to parse all these dates.  Sorting by number is much
faster and pretty much equivalent to sorting by arrival date.

kai
-- 
OOP: object oriented programming;  OOPS: object oriented mistakes