9fans - fans of the OS Plan 9 from Bell Labs
 help / color / mirror / Atom feed
From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Subject: Re: [9fans] Setting up Mail in Acme on the Raspberry Pi.
Date: Mon, 13 Oct 2014 12:15:31 -0400	[thread overview]
Message-ID: <3f0b54378d4d9670e206e0b6428ce886@ladd.quanstro.net> (raw)
In-Reply-To: <aa10988b58cce70b061bd10396209d28@quintile.net>

On Sun Oct 12 14:37:47 EDT 2014, steve@quintile.net wrote:
> I am fairly sure the problem is to do with RAM size rather than the raspberry pi per-se.
> 4000 messages takes up a lot of space - and upas stores messages in RAM.

it's a little worse than this, actually.

since upas stores messages in mbox format, the whole file needs to be read or written on
update.  certainly one could optimize the read bit, but that would be difficult  this means
that the the whole mbox gets written to the dump every day, and you need about 2x the
mailbox size ram for each upas/fs that is run.  this does not work out well for large mm
messages, or small ram boxes like the pi.

the solutions to this are straightforward
(1) store one message per file,
(2) cache important data in an index to avoid opening all files,
(3) avoid O(n²) startup time due to small hash table sizes and high load factor,
(4) load message data on demand so ram required is MAX(largest mm hunk, 10mb).

i currently have 1000 messages in my inbox, but i have used nupas with 45000 messages.
some more work is necessary to handle a quarter million messages comfortably, as that's
too many for one directory.

nupas is just in /sys/src/cmd/upas on 9atom; the original is no more.

- erik



  parent reply	other threads:[~2014-10-13 16:15 UTC|newest]

Thread overview: 123+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-10-11 12:27 Mats Olsson
2014-10-11 13:31 ` Quintile
2014-10-12  7:28   ` Mats Olsson
2014-10-12  7:37     ` Quintile
2014-10-12 13:04       ` Mats Olsson
2014-10-12 14:18         ` Quintile
2014-10-12 14:23         ` Richard Miller
2014-10-12 18:18     ` Eduardo Alvarez
2014-10-12 18:36       ` Steve Simon
2014-10-12 18:58         ` Kurt H Maier
2014-10-13  0:53           ` kokamoto
2014-10-13 11:40             ` Mats Olsson
2014-10-13 14:28               ` p.d.finn
2014-10-13 16:07             ` erik quanstrom
2014-10-13 16:15         ` erik quanstrom [this message]
2014-10-13 16:46           ` Eduardo Alvarez
2014-10-13 23:55             ` Anthony Sorace
2014-10-14  2:41               ` Winston Kodogo
2014-10-14  2:51                 ` Kurt H Maier
2014-10-14  3:00                   ` Winston Kodogo
2014-10-14  3:08                     ` Skip Tavakkolian
2014-10-14  3:25                       ` Winston Kodogo
2014-10-14 19:09                       ` Wes Kussmaul
2014-10-14 20:03                         ` Skip Tavakkolian
2014-10-14 20:29                           ` Wes Kussmaul
2014-10-14 22:32                             ` Winston Kodogo
2014-10-14  9:09                 ` Steve Simon
2014-10-14 11:14                   ` Rudolf Sykora
2014-10-14 12:04                     ` Steve Simon
2014-10-14 12:23                     ` Anthony Sorace
2014-10-14 12:46                       ` Richard Miller
2014-10-14 15:20                       ` erik quanstrom
2014-10-14 15:22                     ` erik quanstrom
2014-10-14 16:08                       ` Rudolf Sykora
2014-10-14 17:29                         ` erik quanstrom
2014-10-14 18:40                         ` Anthony Sorace
2014-10-15  6:37                           ` Rudolf Sykora
2014-10-14 15:57                 ` erik quanstrom
2014-10-13 17:10           ` Bakul Shah
2014-10-13 19:01             ` Mats Olsson
2014-10-13 19:37             ` Steffen Nurpmeso
2014-10-15 12:21             ` trebol
2014-10-15 12:46               ` Steffen Nurpmeso
2014-10-15 13:00                 ` trebol
2014-10-19 19:43                   ` Mats Olsson
2014-10-19 22:26                     ` P. D. Finn
2014-10-20  7:04                       ` Mats Olsson
2014-10-20  7:28                         ` P. D. Finn
2014-10-20 11:11                           ` Mats Olsson
2014-10-20 11:34                             ` Steve Simon
2014-10-20 17:25                               ` Mats Olsson
2014-10-20 17:28                                 ` Mats Olsson
2014-10-20 17:32                                   ` Mats Olsson
2014-10-20 17:49                                     ` Quintile
2014-10-20 18:44                                       ` Mats Olsson
2014-10-21 12:21                                         ` Mats Olsson
2014-10-21 12:57                                           ` k0ga
2014-10-27 15:58                               ` erik quanstrom
2014-10-27 16:34                                 ` lucio
2014-10-27 19:10                                   ` erik quanstrom
2014-10-29 21:43                                     ` Mats Olsson
2014-10-29 21:48                                       ` Mats Olsson
2014-10-29 21:58                                       ` Steve Simon
2014-10-30 11:13                                         ` Mats Olsson
2014-10-30 14:08                                           ` Kurt H Maier
2014-10-30 14:18                                             ` Mats Olsson
2014-10-30 14:34                                               ` Kurt H Maier
2014-10-30 16:00                                           ` erik quanstrom
2014-10-30 16:11                                             ` Mats Olsson
2014-10-30 16:14                                               ` Mats Olsson
2014-10-31 10:26                                                 ` Mats Olsson
2014-10-31 10:51                                                   ` trebol
2014-10-31 10:59                                                   ` trebol
2014-10-31 12:04                                                     ` Mats Olsson
2014-10-31 13:09                                                   ` Steffen Nurpmeso
2014-10-31 13:30                                                     ` erik quanstrom
2014-10-31 14:29                                                       ` Mats Olsson
2014-10-31 18:09                                                       ` Steffen Nurpmeso
2014-11-04  4:24                                                         ` erik quanstrom
2014-11-04 14:27                                                           ` Steffen Nurpmeso
2014-11-05 18:21                                                             ` Mats Olsson
2014-11-05 18:26                                                               ` Mats Olsson
2014-11-05 18:38                                                                 ` Steffen Nurpmeso
2014-11-05 18:55                                                                   ` Mats Olsson
2014-11-05 18:31                                                               ` Steffen Nurpmeso
2014-11-05 17:30                                                           ` Mats Olsson
2014-11-05 17:40                                                             ` Anthony Sorace
2014-11-05 18:19                                                               ` Steffen Nurpmeso
2014-11-06 13:41                                                                 ` erik quanstrom
2014-11-06 19:12                                                                   ` Steffen Nurpmeso
2014-11-07  9:27                                                                     ` Mats Olsson
2014-10-31 19:35                                           ` Quintile
2014-10-31 20:19                                             ` Mats Olsson
2014-10-31 21:14                                               ` Richard Miller
2014-10-31 21:53                                                 ` Mats Olsson
2014-11-03  8:54                                                   ` Mats Olsson
2014-11-03 23:49                                                     ` trebol
2014-11-03  8:57                                                   ` Mats Olsson
2014-11-04  4:06                                                     ` erik quanstrom
2014-11-01  0:14                                                 ` Anthony Martin
2014-11-01  8:16                                                   ` Mats Olsson
2014-11-01  8:34                                                   ` Skip Tavakkolian
2014-11-01 10:32                                                   ` Charles Forsyth
2014-11-01 12:24                                     ` Charles Forsyth
2014-11-04  4:12                                       ` erik quanstrom
2014-10-21 12:58 Steve Simon
2014-10-22  9:06 ` Mats Olsson
2014-10-22 10:32   ` Charles Forsyth
2014-10-22 10:35     ` Charles Forsyth
2014-10-22 12:24       ` Mats Olsson
2014-10-22 13:29         ` Charles Forsyth
2014-10-22 13:59           ` Mats Olsson
2014-10-22 12:38   ` Steve Simon
2014-10-22 14:34   ` Kurt H Maier
2014-10-22 14:46     ` Charles Forsyth
2014-10-22 15:12       ` Kurt H Maier
2014-10-22 15:19         ` Mats Olsson
2014-10-22 18:14           ` Skip Tavakkolian
2014-10-22 20:04             ` Quintile
2014-10-23 20:45               ` Winston Kodogo
2014-10-23 20:57                 ` Charles Forsyth
2014-10-23 21:03                 ` Winston Kodogo
2014-10-24  6:50                   ` Mats Olsson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3f0b54378d4d9670e206e0b6428ce886@ladd.quanstro.net \
    --to=quanstro@quanstro.net \
    --cc=9fans@9fans.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).