Re: [9front] upas: lies about how it works

Re: [9front] upas: lies about how it works

[Steve Simon]

having used plan9 as my primary email service for 15 years, there are still subtleties i didn't realise in there.

excellent description ori.

-Steve


> On 25 Oct 2020, at 1:22 am, ori@eigenstate.org wrote:
> 
> Finally, I want to have some docs that give someone like
> me an idea of how things are supposed to fit together.
> I found figuring out upas configs unnecessarily painful,
> partly because upas itself has way too many layers of
> wrappers, but partly because we have documentation of
> the parts but not how they're intended to slot together.
> 
> I reverse engineered all of this from reading the upas
> example files and code, so it's certain to be full of
> lies -- here's a draft for review:
> 
> 
> Upas: Theory of Operation
> -------------------------
> 
> Upas is the Plan 9 mail system. It's used for
> viewing mail, sending mail, and receiving mail.
> It comes with clients and servers for SMTP,
> IMAP, etc. It also provides a powerful toolkit
> for spam filtering and mail processing.
> 
> Upas is configured through a scattering of
> methods. There are a few config files, and
> a number of scripts which users are intended
> to customize.
> 
> Here's a list of some important files:
> 
>   remotemail
>       The script that you customize
>       for delivering mail to remote
>       systems.
> 
>   qmail
>       Enqueues mail for later delivery,
>       applying filters along the way.
> 
>   rewrite
>       Rewrites and matches the destination
>       of the email, deciding which mail
>       box or smtp server to put the message
>       into.
> 
>   smtpd.conf
>       Configures the SMTP server
> 
>   validateaddress:
>       Checks if we should deliver to an
>       address on this system.
> 
> There are a number of additional files not mentioned
> in this summary.
> 
> 
> Viewing
> -------
> 
> Viewing mail with upas involves very few moving
> parts. Upas/fs connects to most mail protocols,
> and provides a consistent file system interface
> for all of them, abstracting the storage system
> away from the mail clients.
> 
> Upas/fs knows how to render a file system for
> local mailboxes, maildirs, pop, and imap,
> serving them up in a multi-level heirarchy
> in /mail/fs, with one subdirectory for each
> mailbox mounted:
> 
>   /mail/fs/$mbox/$mail/$subfiles
> 
> For example, to see who sent the first email in
> the default mailbox, you could run:
> 
>   cat /mail/fs/mbox/1/from
> 
> Typically, you'd access mail/fs through a client
> such as nedmail or acme Mail.
> 
> Upas/fs only has one config file in /mail/lib,
> for configuring which headers are shown.
> 
> 
> Sending And Receiving
> ---------------------
> 
> Sending and receving email via SMTP in upas is
> a similar operation: A mail is entered into the
> pipeline, is routed, and is delivered to the
> appropriate destination.
> 
> Upas/send is the heart of the delivery pipeline.
> Sending invokes upas/marshal to drop an email
> into upas/send, while receiving does this via
> upas/smtpd.
> 
> Sending and receiving in upas both roughly follow
> the same path. Both of them take an email, and
> dump it into upas/send, which applies the rewrite
> rules and sends it on to further routing depending
> on the destination of the email.
> 
> The major difference between sending and receiving
> is in the starting point: When composing an email
> on plan 9, it gets sent to upas/marshal to drop it
> into the delivery pipeline. When plan 9 is set up
> to recieve mail directly, mail comes in through
> upas/smtpd.
> 
> Send accepts a well formed email, and applies
> the rewrite rules in /mail/lib/rewrite. The
> rewrite rules are expected to match an email
> address and take an appropriate action.
> 
> With a typical rewrite configuration, if the
> mail matches a local user, then the email will
> get deposited into their mailbox. Otherwise,
> the email is punted to /mail/lib/remotemail.
> 
> With the default gateway setup, the pipeline
> looks something like this, where the rewrite
> rules that upas/send uses to interpret email
> enqueues it using qmail:
> 
>   upas/marshal => upas/send =>
>   /mail/lib/qmail => qer =>
>   /mail/lib/remotemail => upas/smtp
> 
> With the example smtp setup, rewrite also
> handles delivering emails locally.
> 
> However, because of the flexibility of the
> rewrite rules, everything after upas/send
> can be swapped out and replaced.
> 
> Marshal
> -------
> 
> Marshal is the simpler of the two entry
> points into the mail system. All it does
> is take a message that you may type by
> hand, and formats it into an rfc822 envelope,
> and (depending on flags) passes it on to send.
> 
> It's also used in the receiving pipeline,
> but only as an address validator, using the
> 'x' flag to examine whether an address is
> deliverable.
> 
> Smtpd
> -----
> 
> Filtering
> ---------
> 
> In addition to the config files in /mail/lib,
> each user can configure mail filtering by
> editing /mail/box/$user/pipeto. This is where.
> for example, spam filtering would be done.
> 
> An example of spam filtering is in:
> 
>   /mail/lib/smtpd.example/pipeto.bayes
> 
> There are some more complicated examples in
> 
>   /sys/src/cmd/upas/filterkit/pipeto.sample
>   /sys/src/cmd/upas/filterkit/pipefrom.sample
> 
> The scripts are run as user 'none', to protect
> you from any funny business.
> 
> Upas ships with a number of programs designed
> to work with the pipefrom or pipeto setup.
> 
> These include:
> 
>   upas/filter
>   upas/list
>   upas/deliver
>   upas/token
>   upas/vf
>   upas/bayes
> 
> There's also a utility library used by rc
> to make pipe scripts easier. It can be loaded
> like this:
> 
>   . /mail/lib/pipeto.lib $*
> 
> The pipeto script is invoked as:
> 
>   rfc822-email | pipeto destaddr destmbox
> 
> and isw expected to eventually invoke
> 
>   upas/deliver
> 
> to deliver their filtered emails.
> 
> Storage Formats
> ---------------
> 
> If /mail/box/$user/$mbox is a file, then it's assumed
> to be in mbox format. If it's a directory, then it's
> assumed to be in mdir format. If the mailbox does not
> exist, then a new maildir is created.
> 
> The Binaries
> ------------
> 
> Spam filtering:
>   upas/addhash:    Merges bayes token hash tables togheter
>   upas/bayes:    Evaluates bayes tokens
>   upas/msgtok:    Tokenizes spam for bayesian filter
>   upas/isspam:    Checks if a message is spam.
>   upas/token:    Creates a message hash
>   upas/spf:    Verifies SPF records
>   upas/ratfs:    Spam blocklist FS
>   upas/vf:    Virus filtering.
>   upas/list:    Maintains and checks lists of users
>   upas/scanmail:    Fixed-pattern spam filtering
> 
> Mail filtering
>   upas/aliasmail:    Manages translating mail aliases
>   upas/deliver:    Drops a message into a specific mailbox
>   upas/filter:    Reroutes messages to different mailboxes
> 
> Mail serving:
>   upas/imap4d:    Serves imap
>   upas/pop3:    Serves pop3
>   upas/smtpd:    Serves smtp
> 
> Sending:
>   upas/marshal:    Submits a message for delivery
>   upas/smtp:    Sends a message to another mail server
> 
> Mailing lists:
>   upas/ml:    Receives and bounces mailing list messages.
>   upas/mlmgr:    Manages mailing lists
>   upas/mlowner:    Manages mailing list owner/control messages
> 
> Internal Plumbing:
>   upas/qer:    Enqueues commands
>   upas/runq:    Runs and retries enqueued commands
>   upas/mbappend:    Appends messages to mbox or mdir mailboxes
>   upas/send:    Starts the email delivery process
> 
> Utilities:
>   upas/msgcat:    Shows message contents
>   upas/testscan:    Dry run of scanmail
>   upas/spam:    Marks an email as spam
>   upas/unspam:    Reduces spam weight of message tokens
>   upas/tfmt:    Prevents topposting
>   upas/unesc:    Interpret =?foo?bar?=char?= escapes
> 
> Clients:
>   upas/fs:    Renders a mailbox as a file system