From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from balrog.mythic-beasts.com ([46.235.227.24]) by ewsd; Sat Oct 31 19:54:58 -0400 2020
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
	d=quintile.net; s=mythic-beasts-k1; h=To:Date:Subject:From;
	bh=0jsEmihDWL5FP/ju0dVf4XKgKd8l3TkOGJrnyBgEXfo=; b=k0tfwac+sE7u3iu9eVYaPf3f6x
	O12VVDqa51sGMTK2tFMa2dyK87yXxXyFbzXuLXZJFAXlHzcKkNnXQjryhAtf9IQEgG9Oqr0aZLIW0
	fc7vDvlbVUUBKGhx2ZCtnT7vKhT1+vrKI9YKUOe1JuAVv8w+3ROXuhGdPuXFA4hQUfJ9EffVJTD+A
	FuqD6ifrI/jO/biWdfIO+BKUBfd6A27oj0ICW8kXQJ0M54seOArdb5oy11vsL4qDVnqbHm377nX8G
	BxrMPsBwQvSyFuCRd5kFqIaP5FAjZAyVDRCQ9665nrAGJwUkrsZ9b0mgPV6awj71lLUGL2oxn3AIJ
	1ouZdGaA==;
Received: from 132.198.187.81.in-addr.arpa ([81.187.198.132]:57070 helo=[192.168.1.179])
	by balrog.mythic-beasts.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92.3)
	(envelope-from <steve@quintile.net>)
	id 1kZ0hu-0003BI-QK
	for 9front@9front.org; Sat, 31 Oct 2020 23:54:55 +0000
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Steve Simon <steve@quintile.net>
Mime-Version: 1.0 (1.0)
Subject: Re: [9front] upas: lies about how it works
Message-Id: <18161A24-D133-4034-82F2-0E341981C71A@quintile.net>
Date: Sat, 31 Oct 2020 23:54:49 +0000
To: 9front@9front.org
X-Mailer: iPhone Mail (18A8395)
X-BlackCat-Spam-Score: 14
X-Spam-Status: No, score=1.4
List-ID: <9front.9front.org>
List-Help: <http://lists.9front.org>
X-Glyph: ➈
X-Bullshit: immutable interface WEB2.0 over TOR lifecycle scripting database 

=EF=BB=BFhaving used plan9 as my primary email service for 15 years, there a=
re still subtleties i didn't realise in there.

excellent description ori.

-Steve


> On 25 Oct 2020, at 1:22 am, ori@eigenstate.org wrote:
>=20
> =EF=BB=BFFinally, I want to have some docs that give someone like
> me an idea of how things are supposed to fit together.
> I found figuring out upas configs unnecessarily painful,
> partly because upas itself has way too many layers of
> wrappers, but partly because we have documentation of
> the parts but not how they're intended to slot together.
>=20
> I reverse engineered all of this from reading the upas
> example files and code, so it's certain to be full of
> lies -- here's a draft for review:
>=20
>=20
> Upas: Theory of Operation
> -------------------------
>=20
> Upas is the Plan 9 mail system. It's used for
> viewing mail, sending mail, and receiving mail.
> It comes with clients and servers for SMTP,
> IMAP, etc. It also provides a powerful toolkit
> for spam filtering and mail processing.
>=20
> Upas is configured through a scattering of
> methods. There are a few config files, and
> a number of scripts which users are intended
> to customize.
>=20
> Here's a list of some important files:
>=20
>   remotemail
>       The script that you customize
>       for delivering mail to remote
>       systems.
>=20
>   qmail
>       Enqueues mail for later delivery,
>       applying filters along the way.
>=20
>   rewrite
>       Rewrites and matches the destination
>       of the email, deciding which mail
>       box or smtp server to put the message
>       into.
>=20
>   smtpd.conf
>       Configures the SMTP server
>=20
>   validateaddress:
>       Checks if we should deliver to an
>       address on this system.
>=20
> There are a number of additional files not mentioned
> in this summary.
>=20
>=20
> Viewing
> -------
>=20
> Viewing mail with upas involves very few moving
> parts. Upas/fs connects to most mail protocols,
> and provides a consistent file system interface
> for all of them, abstracting the storage system
> away from the mail clients.
>=20
> Upas/fs knows how to render a file system for
> local mailboxes, maildirs, pop, and imap,
> serving them up in a multi-level heirarchy
> in /mail/fs, with one subdirectory for each
> mailbox mounted:
>=20
>   /mail/fs/$mbox/$mail/$subfiles
>=20
> For example, to see who sent the first email in
> the default mailbox, you could run:
>=20
>   cat /mail/fs/mbox/1/from
>=20
> Typically, you'd access mail/fs through a client
> such as nedmail or acme Mail.
>=20
> Upas/fs only has one config file in /mail/lib,
> for configuring which headers are shown.
>=20
>=20
> Sending And Receiving
> ---------------------
>=20
> Sending and receving email via SMTP in upas is
> a similar operation: A mail is entered into the
> pipeline, is routed, and is delivered to the
> appropriate destination.
>=20
> Upas/send is the heart of the delivery pipeline.
> Sending invokes upas/marshal to drop an email
> into upas/send, while receiving does this via
> upas/smtpd.
>=20
> Sending and receiving in upas both roughly follow
> the same path. Both of them take an email, and
> dump it into upas/send, which applies the rewrite
> rules and sends it on to further routing depending
> on the destination of the email.
>=20
> The major difference between sending and receiving
> is in the starting point: When composing an email
> on plan 9, it gets sent to upas/marshal to drop it
> into the delivery pipeline. When plan 9 is set up
> to recieve mail directly, mail comes in through
> upas/smtpd.
>=20
> Send accepts a well formed email, and applies
> the rewrite rules in /mail/lib/rewrite. The
> rewrite rules are expected to match an email
> address and take an appropriate action.
>=20
> With a typical rewrite configuration, if the
> mail matches a local user, then the email will
> get deposited into their mailbox. Otherwise,
> the email is punted to /mail/lib/remotemail.
>=20
> With the default gateway setup, the pipeline
> looks something like this, where the rewrite
> rules that upas/send uses to interpret email
> enqueues it using qmail:
>=20
>   upas/marshal =3D> upas/send =3D>
>   /mail/lib/qmail =3D> qer =3D>
>   /mail/lib/remotemail =3D> upas/smtp
>=20
> With the example smtp setup, rewrite also
> handles delivering emails locally.
>=20
> However, because of the flexibility of the
> rewrite rules, everything after upas/send
> can be swapped out and replaced.
>=20
> Marshal
> -------
>=20
> Marshal is the simpler of the two entry
> points into the mail system. All it does
> is take a message that you may type by
> hand, and formats it into an rfc822 envelope,
> and (depending on flags) passes it on to send.
>=20
> It's also used in the receiving pipeline,
> but only as an address validator, using the
> 'x' flag to examine whether an address is
> deliverable.
>=20
> Smtpd
> -----
>=20
> Filtering
> ---------
>=20
> In addition to the config files in /mail/lib,
> each user can configure mail filtering by
> editing /mail/box/$user/pipeto. This is where.
> for example, spam filtering would be done.
>=20
> An example of spam filtering is in:
>=20
>   /mail/lib/smtpd.example/pipeto.bayes
>=20
> There are some more complicated examples in
>=20
>   /sys/src/cmd/upas/filterkit/pipeto.sample
>   /sys/src/cmd/upas/filterkit/pipefrom.sample
>=20
> The scripts are run as user 'none', to protect
> you from any funny business.
>=20
> Upas ships with a number of programs designed
> to work with the pipefrom or pipeto setup.
>=20
> These include:
>=20
>   upas/filter
>   upas/list
>   upas/deliver
>   upas/token
>   upas/vf
>   upas/bayes
>=20
> There's also a utility library used by rc
> to make pipe scripts easier. It can be loaded
> like this:
>=20
>   . /mail/lib/pipeto.lib $*
>=20
> The pipeto script is invoked as:
>=20
>   rfc822-email | pipeto destaddr destmbox
>=20
> and isw expected to eventually invoke
>=20
>   upas/deliver
>=20
> to deliver their filtered emails.
>=20
> Storage Formats
> ---------------
>=20
> If /mail/box/$user/$mbox is a file, then it's assumed
> to be in mbox format. If it's a directory, then it's
> assumed to be in mdir format. If the mailbox does not
> exist, then a new maildir is created.
>=20
> The Binaries
> ------------
>=20
> Spam filtering:
>   upas/addhash:    Merges bayes token hash tables togheter
>   upas/bayes:    Evaluates bayes tokens
>   upas/msgtok:    Tokenizes spam for bayesian filter
>   upas/isspam:    Checks if a message is spam.
>   upas/token:    Creates a message hash
>   upas/spf:    Verifies SPF records
>   upas/ratfs:    Spam blocklist FS
>   upas/vf:    Virus filtering.
>   upas/list:    Maintains and checks lists of users
>   upas/scanmail:    Fixed-pattern spam filtering
>=20
> Mail filtering
>   upas/aliasmail:    Manages translating mail aliases
>   upas/deliver:    Drops a message into a specific mailbox
>   upas/filter:    Reroutes messages to different mailboxes
>=20
> Mail serving:
>   upas/imap4d:    Serves imap
>   upas/pop3:    Serves pop3
>   upas/smtpd:    Serves smtp
>=20
> Sending:
>   upas/marshal:    Submits a message for delivery
>   upas/smtp:    Sends a message to another mail server
>=20
> Mailing lists:
>   upas/ml:    Receives and bounces mailing list messages.
>   upas/mlmgr:    Manages mailing lists
>   upas/mlowner:    Manages mailing list owner/control messages
>=20
> Internal Plumbing:
>   upas/qer:    Enqueues commands
>   upas/runq:    Runs and retries enqueued commands
>   upas/mbappend:    Appends messages to mbox or mdir mailboxes
>   upas/send:    Starts the email delivery process
>=20
> Utilities:
>   upas/msgcat:    Shows message contents
>   upas/testscan:    Dry run of scanmail
>   upas/spam:    Marks an email as spam
>   upas/unspam:    Reduces spam weight of message tokens
>   upas/tfmt:    Prevents topposting
>   upas/unesc:    Interpret =3D?foo?bar?=3Dchar?=3D escapes
>=20
> Clients:
>   upas/fs:    Renders a mailbox as a file system