/mail/lib: cleanup proposal

/mail/lib: cleanup proposal

[ori]

While it's fresh in my mind, I'm looking at /mail/lib. There
are a lot of example files there, and it's a bit confusing
what needs to be configured, when the scripts are invoked,
what's intended to be configuration, and what's probably
best to leave alone.

What I'd like is to ship 9ront up with a minimal config
that works out of the box for sending email through a
remote SMTP server. Ideally, all we'd need would be
setting a from address, and adding the keys in factotum.

In an ideal world, I'd just have to do:

	echo 'ori@eigenstate.org' >> $home/lib/mail/sender

I'd also like to take a minimal config for receiving email
on @EXAMPLE.COM and put that into a separate directory, like:

	/mail/lib/example.direct

For that, I need to understand what exactly belongs in a
config, and what it does.

So, to that end, here's my (likely wrong) understanding
of what the files in /lib/mail do, and whether we actually
need them.

It seems like the shit in /mail/lib falls into five
categories:
	- Config (and scripts that act as config)
	- Example config (which isn't labelled as such)
	- Bodies for /bin/upas/foo stubs
	- State for upas programs
	- Admin utilities not invoked by the mail system

So, my current understanding of the files:

	blocked:
		optional; a list of ip addresses that are probably
		spammy. Horribly out of date, should probably not
		be in our default config.
	
	classify.re:
		useful for mail servers, but not human config, can
		we move to a place that makes it clear? /mail/state,
		or something?
	
	gone.fishing:
		optional: away message handling. would like to
		document, but not put into the example config.
	
	gone.msg:
		see above.
	
	ignore:
		needed, but why? a list of headers for upas/fs
		to filter out. I don't think any of our mail
		clients show the headers raw, seems like we
		can drop this file...?
	
	isspam.rc:
	spam.rc:
	unspam.rc:
	msgcat.rc:
		mandatory and not config. Exec'ed by /bin/upas/isspam;
		doesn't seem to	be intended for edits by user. would
		like to the contents to /bin/upas/isspam, and maybe
		add an 'if(test -f /mail/lib/isspam.rc) exec...' for
		allowing customization.
	
		At the very least, would like to move out of /mail/lib
		to a place that makes it clear it's probably not
		for users to touch in most cases. /mail/rc? /mail/lib/rc?
	
	justqmail:
		optional? admin script? used by lazyqmail
	
	kickqueue:
		optional? admin util; do we need to have this
		in /lib/mail?
	
	mailnews:
		obsolete: seems to be a util for sending newsletters
		about plan 9 to a list of subscribers.
	
	namefiles:
		mandatory for smtpd: a list of alias lists; needed for
		smtpd config, not for local mailing.
	
	names.local:
		mandatory for smtpd: the alias list included by namefiles.
	
	patterns:
		optional. hardcoded spam regexes -- horribly out
		of date, probably not useful any more for fighting
		spam. If you want to get the same effect, you can
		probably do better with a pipeto; I'd like to
		delete support entirely.
	
	pipeto.bayes:
		optional: an example of how to send email to our
		bayesian spam filter. upas/bayes needs documentation.
	
	pipeto.lib:
		not config: a set of utility scripts that the .rc
		files include. Move to /mail/rc?
	
	prof.mbox:
	prof.spam:
		optional? not entirely sure what these are for.
		Seems to be some sort of state tracking for spam
		filtering? Anyone?
	
		If it's state tracking, see notes on classify.re
	
	qmail:
		optional? another admin tool? can't find anything
		that calls this directly in upas or /mail/lib.
	
	remotemail:
		mandatory: used for sending email.
	
	rewrite:
		mandatory: used for rerouting email.
	
	rewrite.direct:
	rewrite.gateway:
		example: copy to rewrite and edit
	
	setup.bayes:
		optional, stupid: sets up files that the bayesian
		filter uses. The bayesian filter should just do
		that itself if the files don't exist. Should
		fix bayes and delete this.
	
	smtpd.conf:
		mandatory (for smtp servers): exactly what it says
		on the tin, the smtp config.
	
	validateaddress:
		mandatory for smtpd: checks if the destination
		of a received email is valid for this system.
		maybe should turn it into an optional script,
		and instead teach smtpd to check if a user
		or alias exists directly.
	
	validateattachment:
		optional? mostly seems silly to have on by
		default.
	
	white.starter:
		example? an example of what whitelists look like,
		probably not needed.

With that, it seems like we could clean things up so that
/mail/lib defaults to containing:

	/mail/lib/rewrite.example
	/mail/lib/remotemail
	/mail/lib/ignore (todo, remove?)

We can rewrite remotemail to default to routing based on
factotum and the from address, rather than needing a bunch
of config. Ideally, it'd be something like:

	# allow per-user overrides
	if(test -f $home/lib/mail/remotemail)
		exec $home/lib/mail/remotemail
	if(test -f $home/lib//mail/sender)
		from=`$nl{cat $home/lib/mail/sender
	# scrape server, username from factotum, route based
	# on that.
	user=`$nl{awk '/...server=.../{print user}' /mnt/factotum/ctl}
	user=`$nl{awk '/...server=.../{print server}' /mnt/factotum/ctl}

I'm not sure how to get our domain into the
rewrite rules, so until we figure that out it'll have to
remain an example.

For smtpd, it seems like we can have:

	/mail/lib/directmail.example/smtpd.conf
	/mail/lib/directmail.example/validateaddress
	/mail/lib/directmail.example/rewrite
	/mail/lib/directmail.example/namefiles
	/mail/lib/directmail.example/alias.local

The rest can probably be put into other examples.

	/mail/lib/spam.example/...
	/mail/lib/fishing.example/...
	...etc...

The script stubs could either be inlined into /bin/upas/foo,
or put into:

	/mail/rc/*.rc

and the admin scripts into

	/mail/util/$script

I think this would make it easier to figure out what's going
on with upas configuration, and get a working system.

Thoughts? Complaints? "Don't touch the artwork!"?

Re: [9front] /mail/lib: cleanup proposal

[Kurt H Maier]

On Sun, Sep 20, 2020 at 11:05:01AM -0700, ori@eigenstate.org wrote:
> 
> Thoughts? Complaints? "Don't touch the artwork!"?
> 

Touch the hell out of this artwork.  Your proposal would also simplify
a part of the fqa that people frequently complain about.

khm

Re: [9front] /mail/lib: cleanup proposal

[hiro]

agreed, this is not art

Re: [9front] /mail/lib: cleanup proposal

[Stanley Lieber]

On September 20, 2020 2:05:01 PM EDT, ori@eigenstate.org wrote:
>While it's fresh in my mind, I'm looking at /mail/lib. There
>are a lot of example files there, and it's a bit confusing
>what needs to be configured, when the scripts are invoked,
>what's intended to be configuration, and what's probably
>best to leave alone.
>
>What I'd like is to ship 9ront up with a minimal config
>that works out of the box for sending email through a
>remote SMTP server. Ideally, all we'd need would be
>setting a from address, and adding the keys in factotum.
>
>In an ideal world, I'd just have to do:
>
>	echo 'ori@eigenstate.org' >> $home/lib/mail/sender
>
>I'd also like to take a minimal config for receiving email
>on @EXAMPLE.COM and put that into a separate directory, like:
>
>	/mail/lib/example.direct
>
>For that, I need to understand what exactly belongs in a
>config, and what it does.
>
>So, to that end, here's my (likely wrong) understanding
>of what the files in /lib/mail do, and whether we actually
>need them.
>
>It seems like the shit in /mail/lib falls into five
>categories:
>	- Config (and scripts that act as config)
>	- Example config (which isn't labelled as such)
>	- Bodies for /bin/upas/foo stubs
>	- State for upas programs
>	- Admin utilities not invoked by the mail system
>
>So, my current understanding of the files:
>
>	blocked:
>		optional; a list of ip addresses that are probably
>		spammy. Horribly out of date, should probably not
>		be in our default config.
>	
>	classify.re:
>		useful for mail servers, but not human config, can
>		we move to a place that makes it clear? /mail/state,
>		or something?
>	
>	gone.fishing:
>		optional: away message handling. would like to
>		document, but not put into the example config.
>	
>	gone.msg:
>		see above.
>	
>	ignore:
>		needed, but why? a list of headers for upas/fs
>		to filter out. I don't think any of our mail
>		clients show the headers raw, seems like we
>		can drop this file...?
>	
>	isspam.rc:
>	spam.rc:
>	unspam.rc:
>	msgcat.rc:
>		mandatory and not config. Exec'ed by /bin/upas/isspam;
>		doesn't seem to	be intended for edits by user. would
>		like to the contents to /bin/upas/isspam, and maybe
>		add an 'if(test -f /mail/lib/isspam.rc) exec...' for
>		allowing customization.
>	
>		At the very least, would like to move out of /mail/lib
>		to a place that makes it clear it's probably not
>		for users to touch in most cases. /mail/rc? /mail/lib/rc?
>	
>	justqmail:
>		optional? admin script? used by lazyqmail
>	
>	kickqueue:
>		optional? admin util; do we need to have this
>		in /lib/mail?
>	
>	mailnews:
>		obsolete: seems to be a util for sending newsletters
>		about plan 9 to a list of subscribers.
>	
>	namefiles:
>		mandatory for smtpd: a list of alias lists; needed for
>		smtpd config, not for local mailing.
>	
>	names.local:
>		mandatory for smtpd: the alias list included by namefiles.
>	
>	patterns:
>		optional. hardcoded spam regexes -- horribly out
>		of date, probably not useful any more for fighting
>		spam. If you want to get the same effect, you can
>		probably do better with a pipeto; I'd like to
>		delete support entirely.
>	
>	pipeto.bayes:
>		optional: an example of how to send email to our
>		bayesian spam filter. upas/bayes needs documentation.
>	
>	pipeto.lib:
>		not config: a set of utility scripts that the .rc
>		files include. Move to /mail/rc?
>	
>	prof.mbox:
>	prof.spam:
>		optional? not entirely sure what these are for.
>		Seems to be some sort of state tracking for spam
>		filtering? Anyone?
>	
>		If it's state tracking, see notes on classify.re
>	
>	qmail:
>		optional? another admin tool? can't find anything
>		that calls this directly in upas or /mail/lib.
>	
>	remotemail:
>		mandatory: used for sending email.
>	
>	rewrite:
>		mandatory: used for rerouting email.
>	
>	rewrite.direct:
>	rewrite.gateway:
>		example: copy to rewrite and edit
>	
>	setup.bayes:
>		optional, stupid: sets up files that the bayesian
>		filter uses. The bayesian filter should just do
>		that itself if the files don't exist. Should
>		fix bayes and delete this.
>	
>	smtpd.conf:
>		mandatory (for smtp servers): exactly what it says
>		on the tin, the smtp config.
>	
>	validateaddress:
>		mandatory for smtpd: checks if the destination
>		of a received email is valid for this system.
>		maybe should turn it into an optional script,
>		and instead teach smtpd to check if a user
>		or alias exists directly.
>	
>	validateattachment:
>		optional? mostly seems silly to have on by
>		default.
>	
>	white.starter:
>		example? an example of what whitelists look like,
>		probably not needed.
>
>With that, it seems like we could clean things up so that
>/mail/lib defaults to containing:
>
>	/mail/lib/rewrite.example
>	/mail/lib/remotemail
>	/mail/lib/ignore (todo, remove?)
>
>We can rewrite remotemail to default to routing based on
>factotum and the from address, rather than needing a bunch
>of config. Ideally, it'd be something like:
>
>	# allow per-user overrides
>	if(test -f $home/lib/mail/remotemail)
>		exec $home/lib/mail/remotemail
>	if(test -f $home/lib//mail/sender)
>		from=`$nl{cat $home/lib/mail/sender
>	# scrape server, username from factotum, route based
>	# on that.
>	user=`$nl{awk '/...server=.../{print user}' /mnt/factotum/ctl}
>	user=`$nl{awk '/...server=.../{print server}' /mnt/factotum/ctl}
>
>I'm not sure how to get our domain into the
>rewrite rules, so until we figure that out it'll have to
>remain an example.
>
>For smtpd, it seems like we can have:
>
>	/mail/lib/directmail.example/smtpd.conf
>	/mail/lib/directmail.example/validateaddress
>	/mail/lib/directmail.example/rewrite
>	/mail/lib/directmail.example/namefiles
>	/mail/lib/directmail.example/alias.local
>
>The rest can probably be put into other examples.
>
>	/mail/lib/spam.example/...
>	/mail/lib/fishing.example/...
>	...etc...
>
>The script stubs could either be inlined into /bin/upas/foo,
>or put into:
>
>	/mail/rc/*.rc
>
>and the admin scripts into
>
>	/mail/util/$script
>
>I think this would make it easier to figure out what's going
>on with upas configuration, and get a working system.
>
>Thoughts? Complaints? "Don't touch the artwork!"?

please keep in mind all our shit (read: this mailing list, and my personal mail) runs on upas before committing breaking changes.

strongly in favor of simplifying setup. whatever we're currently shipping is the remnants of what actually ran at the labs. all that configuration (anti-spam, etc.) was theirs. i got all my own stuff working by reading the man pages and trial and error.

i thought incoming messages were piped through qmail somewhere along the line.

sl

Re: [9front] /mail/lib: cleanup proposal

[ori]

> > I think this would make it easier to figure out what's going
> > on with upas configuration, and get a working system.
> > 
> > Thoughts? Complaints? "Don't touch the artwork!"?
>
> please keep in mind all our shit (read: this mailing list, and my
> personal mail) runs on upas before committing breaking changes.

Yep, definitely. That's why I'm trying to validate my assumptions
on what's actually considered configuration, and what's dumped
in there.

Note, /mail/lib isn't in hg, so sysupdate doesn't touch it.
The changes I'm proposing would be in the defaults for new
installs.

Any breakage would be from changes in /sys/src/cmd/upas, and
I don't see any problem with keeping things compatible. At
most it'd mean checking an extra path to decide what command
to run.

> strongly in favor of simplifying setup.  whatever we're currently
> shipping is the remnants of what actually ran at the labs.  all that
> configuration (anti-spam, etc.) was theirs.  i got all my own stuff
> working by reading the man pages and trial and error.

Yeah. I'm doing that now as well for testing the smtpd changes
for date and time. I'm realizing that this stuff is simpler than
it looks, but the example config really overcomplicates things.

> i thought incoming messages were piped through qmail somewhere along
> the line.

Ah, I missed it in the rewrite file. Thanks for the cross check.

Re: [9front] /mail/lib: cleanup proposal

[Lyndon Nerenberg]

ori@eigenstate.org writes:

> 	mailnews:
> 		obsolete: seems to be a util for sending newsletters
> 		about plan 9 to a list of subscribers.

Please keep this.  It's used to mail new entries in /lib/news
to a list of subscribers.  It's very handy for people who don't
log in regularly to keep up to date with system changes.

--lyndon

Re: [9front] /mail/lib: cleanup proposal

[ori]

>> > I think this would make it easier to figure out what's going
>> > on with upas configuration, and get a working system.
>> > 
>> > Thoughts? Complaints? "Don't touch the artwork!"?
>>
>> please keep in mind all our shit (read: this mailing list, and my
>> personal mail) runs on upas before committing breaking changes.
> 
> Yep, definitely. That's why I'm trying to validate my assumptions
> on what's actually considered configuration, and what's dumped
> in there.
> 
> Note, /mail/lib isn't in hg, so sysupdate doesn't touch it.
> The changes I'm proposing would be in the defaults for new
> installs.
> 
> Any breakage would be from changes in /sys/src/cmd/upas, and
> I don't see any problem with keeping things compatible. At
> most it'd mean checking an extra path to decide what command
> to run.
> 
>> strongly in favor of simplifying setup.  whatever we're currently
>> shipping is the remnants of what actually ran at the labs.  all that
>> configuration (anti-spam, etc.) was theirs.  i got all my own stuff
>> working by reading the man pages and trial and error.
> 
> Yeah. I'm doing that now as well for testing the smtpd changes
> for date and time. I'm realizing that this stuff is simpler than
> it looks, but the example config really overcomplicates things.
> 
>> i thought incoming messages were piped through qmail somewhere along
>> the line.
> 
> Ah, I missed it in the rewrite file. Thanks for the cross check.

So, after messing around with this over the weekend, I think I
have a reasonable few defaults for /mail/lib.

The files that we need for reasonable defaults to send mail:

	/mail/lib/ignore
	/mail/lib/qmail
	/mail/lib/remotemail
	/mail/lib/rewrite

I'd like to make upas/fs use reasonable defaults for the ignore
file so we can remove it (and maybe flip it to a keephdr file).

I've also rewritten remotemail, so that it uses factotum to figure
out where to send to. That means that configuring mail is adding

	upasname=user@server.com

to your environment, and adding an 'email=user@server.com' to your
smtp factotum key so we can decide which smtp server to use:

	echo 'key proto=pass server=server.com ' \
		'service=smtp user=user >>email=user@server.com<< !password=12345'

Here's the rewritten remotemail:

	#!/bin/rc
	
	# allow users to do their own mail setup
	if(test -x $home/lib/mail/remotemail)
		exec $home/mail/lib/remotemail
	
	sender=$2
	svcpat='[ 	]service=smtp[ 	]'
	addrpat='[ 	]email='$2'[ 	]'
	config=`{grep -e $svcpat -e $addrpat /mnt/factotum/ctl}
	if(~ $#config 0)
		exit 'no server'
	for(kv in $config){
		parts=`'='{echo -n $kv}
		switch($parts(1)){
		case server;	server=$parts(2)
		case user;	login=$parts(2)
		}
	}
	
	exec /bin/upas/smtp -as -u $login $server $addr $sender $*(4-)

The ignore and qmail files are the same as we ship today.
The rewrite rules are the same as our cureent rewrite.gateway,
though I want to slim it down a bit, or at least comment it
better. Separate mail incoming for that.

The other files seem like they can become examples -- I'll look at
putting together a proposal for some example configurations that
we can ship separately.

Is there anything that this setup is missing?

Is there some edge case that this will break on?

Re: [9front] /mail/lib: cleanup proposal

[sirjofri+ml-9front]

Hey ori,

> I've also rewritten remotemail, so that it uses factotum to figure
> out where to send to. That means that configuring mail is adding
>
> upasname=user@server.com
>
> to your environment, and adding an 'email=user@server.com' to your
> smtp factotum key so we can decide which smtp server to use:
>
> echo 'key proto=pass server=server.com ' \
> 'service=smtp user=user >>email=user@server.com<< !password=12345'
>
> Here's the rewritten remotemail:
>
> #!/bin/rc
>
> # allow users to do their own mail setup
> if(test -x $home/lib/mail/remotemail)
> exec $home/mail/lib/remotemail
>
> sender=$2
> svcpat='[   ]service=smtp[  ]'
> addrpat='[  ]email='$2'[  ]'
> config=`{grep -e $svcpat -e $addrpat /mnt/factotum/ctl}
> if(~ $#config 0)
> exit 'no server'
> for(kv in $config){
> parts=`'='{echo -n $kv}
> switch($parts(1)){
> case server;  server=$parts(2)
> case user;  login=$parts(2)
> }
> }
>
> exec /bin/upas/smtp -as -u $login $server $addr $sender $*(4-)

I like the changes. Also may I suggest that you add my patch for smtp? 
Then unrecognized certificates during smtp can be ignored using a flag 
resulting in encrypted mails. Modern providers like this, although it 
makes mails somewhat unsecure (mails are encrypted which is more secure, 
but no one can ensure you're talking to the right server). Otherwise we 
need to handle certificates better (maybe trust on first use?)

Also since my patch provides a flag server admins can choose which 
security feature they want.

sirjofri