Re: [9front] /mail/lib: cleanup proposal

[Stanley Lieber <sl@stanleylieber.com>]

On September 20, 2020 2:05:01 PM EDT, ori@eigenstate.org wrote:
>While it's fresh in my mind, I'm looking at /mail/lib. There
>are a lot of example files there, and it's a bit confusing
>what needs to be configured, when the scripts are invoked,
>what's intended to be configuration, and what's probably
>best to leave alone.
>
>What I'd like is to ship 9ront up with a minimal config
>that works out of the box for sending email through a
>remote SMTP server. Ideally, all we'd need would be
>setting a from address, and adding the keys in factotum.
>
>In an ideal world, I'd just have to do:
>
>	echo 'ori@eigenstate.org' >> $home/lib/mail/sender
>
>I'd also like to take a minimal config for receiving email
>on @EXAMPLE.COM and put that into a separate directory, like:
>
>	/mail/lib/example.direct
>
>For that, I need to understand what exactly belongs in a
>config, and what it does.
>
>So, to that end, here's my (likely wrong) understanding
>of what the files in /lib/mail do, and whether we actually
>need them.
>
>It seems like the shit in /mail/lib falls into five
>categories:
>	- Config (and scripts that act as config)
>	- Example config (which isn't labelled as such)
>	- Bodies for /bin/upas/foo stubs
>	- State for upas programs
>	- Admin utilities not invoked by the mail system
>
>So, my current understanding of the files:
>
>	blocked:
>		optional; a list of ip addresses that are probably
>		spammy. Horribly out of date, should probably not
>		be in our default config.
>	
>	classify.re:
>		useful for mail servers, but not human config, can
>		we move to a place that makes it clear? /mail/state,
>		or something?
>	
>	gone.fishing:
>		optional: away message handling. would like to
>		document, but not put into the example config.
>	
>	gone.msg:
>		see above.
>	
>	ignore:
>		needed, but why? a list of headers for upas/fs
>		to filter out. I don't think any of our mail
>		clients show the headers raw, seems like we
>		can drop this file...?
>	
>	isspam.rc:
>	spam.rc:
>	unspam.rc:
>	msgcat.rc:
>		mandatory and not config. Exec'ed by /bin/upas/isspam;
>		doesn't seem to	be intended for edits by user. would
>		like to the contents to /bin/upas/isspam, and maybe
>		add an 'if(test -f /mail/lib/isspam.rc) exec...' for
>		allowing customization.
>	
>		At the very least, would like to move out of /mail/lib
>		to a place that makes it clear it's probably not
>		for users to touch in most cases. /mail/rc? /mail/lib/rc?
>	
>	justqmail:
>		optional? admin script? used by lazyqmail
>	
>	kickqueue:
>		optional? admin util; do we need to have this
>		in /lib/mail?
>	
>	mailnews:
>		obsolete: seems to be a util for sending newsletters
>		about plan 9 to a list of subscribers.
>	
>	namefiles:
>		mandatory for smtpd: a list of alias lists; needed for
>		smtpd config, not for local mailing.
>	
>	names.local:
>		mandatory for smtpd: the alias list included by namefiles.
>	
>	patterns:
>		optional. hardcoded spam regexes -- horribly out
>		of date, probably not useful any more for fighting
>		spam. If you want to get the same effect, you can
>		probably do better with a pipeto; I'd like to
>		delete support entirely.
>	
>	pipeto.bayes:
>		optional: an example of how to send email to our
>		bayesian spam filter. upas/bayes needs documentation.
>	
>	pipeto.lib:
>		not config: a set of utility scripts that the .rc
>		files include. Move to /mail/rc?
>	
>	prof.mbox:
>	prof.spam:
>		optional? not entirely sure what these are for.
>		Seems to be some sort of state tracking for spam
>		filtering? Anyone?
>	
>		If it's state tracking, see notes on classify.re
>	
>	qmail:
>		optional? another admin tool? can't find anything
>		that calls this directly in upas or /mail/lib.
>	
>	remotemail:
>		mandatory: used for sending email.
>	
>	rewrite:
>		mandatory: used for rerouting email.
>	
>	rewrite.direct:
>	rewrite.gateway:
>		example: copy to rewrite and edit
>	
>	setup.bayes:
>		optional, stupid: sets up files that the bayesian
>		filter uses. The bayesian filter should just do
>		that itself if the files don't exist. Should
>		fix bayes and delete this.
>	
>	smtpd.conf:
>		mandatory (for smtp servers): exactly what it says
>		on the tin, the smtp config.
>	
>	validateaddress:
>		mandatory for smtpd: checks if the destination
>		of a received email is valid for this system.
>		maybe should turn it into an optional script,
>		and instead teach smtpd to check if a user
>		or alias exists directly.
>	
>	validateattachment:
>		optional? mostly seems silly to have on by
>		default.
>	
>	white.starter:
>		example? an example of what whitelists look like,
>		probably not needed.
>
>With that, it seems like we could clean things up so that
>/mail/lib defaults to containing:
>
>	/mail/lib/rewrite.example
>	/mail/lib/remotemail
>	/mail/lib/ignore (todo, remove?)
>
>We can rewrite remotemail to default to routing based on
>factotum and the from address, rather than needing a bunch
>of config. Ideally, it'd be something like:
>
>	# allow per-user overrides
>	if(test -f $home/lib/mail/remotemail)
>		exec $home/lib/mail/remotemail
>	if(test -f $home/lib//mail/sender)
>		from=`$nl{cat $home/lib/mail/sender
>	# scrape server, username from factotum, route based
>	# on that.
>	user=`$nl{awk '/...server=.../{print user}' /mnt/factotum/ctl}
>	user=`$nl{awk '/...server=.../{print server}' /mnt/factotum/ctl}
>
>I'm not sure how to get our domain into the
>rewrite rules, so until we figure that out it'll have to
>remain an example.
>
>For smtpd, it seems like we can have:
>
>	/mail/lib/directmail.example/smtpd.conf
>	/mail/lib/directmail.example/validateaddress
>	/mail/lib/directmail.example/rewrite
>	/mail/lib/directmail.example/namefiles
>	/mail/lib/directmail.example/alias.local
>
>The rest can probably be put into other examples.
>
>	/mail/lib/spam.example/...
>	/mail/lib/fishing.example/...
>	...etc...
>
>The script stubs could either be inlined into /bin/upas/foo,
>or put into:
>
>	/mail/rc/*.rc
>
>and the admin scripts into
>
>	/mail/util/$script
>
>I think this would make it easier to figure out what's going
>on with upas configuration, and get a working system.
>
>Thoughts? Complaints? "Don't touch the artwork!"?

please keep in mind all our shit (read: this mailing list, and my personal mail) runs on upas before committing breaking changes.

strongly in favor of simplifying setup. whatever we're currently shipping is the remnants of what actually ran at the labs. all that configuration (anti-spam, etc.) was theirs. i got all my own stuff working by reading the man pages and trial and error.

i thought incoming messages were piped through qmail somewhere along the line.

sl