* /mail/lib: cleanup proposal
@ 2020-09-20 18:05 ori
2020-09-20 19:01 ` [9front] " Kurt H Maier
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: ori @ 2020-09-20 18:05 UTC (permalink / raw)
To: 9front
While it's fresh in my mind, I'm looking at /mail/lib. There
are a lot of example files there, and it's a bit confusing
what needs to be configured, when the scripts are invoked,
what's intended to be configuration, and what's probably
best to leave alone.
What I'd like is to ship 9ront up with a minimal config
that works out of the box for sending email through a
remote SMTP server. Ideally, all we'd need would be
setting a from address, and adding the keys in factotum.
In an ideal world, I'd just have to do:
echo 'ori@eigenstate.org' >> $home/lib/mail/sender
I'd also like to take a minimal config for receiving email
on @EXAMPLE.COM and put that into a separate directory, like:
/mail/lib/example.direct
For that, I need to understand what exactly belongs in a
config, and what it does.
So, to that end, here's my (likely wrong) understanding
of what the files in /lib/mail do, and whether we actually
need them.
It seems like the shit in /mail/lib falls into five
categories:
- Config (and scripts that act as config)
- Example config (which isn't labelled as such)
- Bodies for /bin/upas/foo stubs
- State for upas programs
- Admin utilities not invoked by the mail system
So, my current understanding of the files:
blocked:
optional; a list of ip addresses that are probably
spammy. Horribly out of date, should probably not
be in our default config.
classify.re:
useful for mail servers, but not human config, can
we move to a place that makes it clear? /mail/state,
or something?
gone.fishing:
optional: away message handling. would like to
document, but not put into the example config.
gone.msg:
see above.
ignore:
needed, but why? a list of headers for upas/fs
to filter out. I don't think any of our mail
clients show the headers raw, seems like we
can drop this file...?
isspam.rc:
spam.rc:
unspam.rc:
msgcat.rc:
mandatory and not config. Exec'ed by /bin/upas/isspam;
doesn't seem to be intended for edits by user. would
like to the contents to /bin/upas/isspam, and maybe
add an 'if(test -f /mail/lib/isspam.rc) exec...' for
allowing customization.
At the very least, would like to move out of /mail/lib
to a place that makes it clear it's probably not
for users to touch in most cases. /mail/rc? /mail/lib/rc?
justqmail:
optional? admin script? used by lazyqmail
kickqueue:
optional? admin util; do we need to have this
in /lib/mail?
mailnews:
obsolete: seems to be a util for sending newsletters
about plan 9 to a list of subscribers.
namefiles:
mandatory for smtpd: a list of alias lists; needed for
smtpd config, not for local mailing.
names.local:
mandatory for smtpd: the alias list included by namefiles.
patterns:
optional. hardcoded spam regexes -- horribly out
of date, probably not useful any more for fighting
spam. If you want to get the same effect, you can
probably do better with a pipeto; I'd like to
delete support entirely.
pipeto.bayes:
optional: an example of how to send email to our
bayesian spam filter. upas/bayes needs documentation.
pipeto.lib:
not config: a set of utility scripts that the .rc
files include. Move to /mail/rc?
prof.mbox:
prof.spam:
optional? not entirely sure what these are for.
Seems to be some sort of state tracking for spam
filtering? Anyone?
If it's state tracking, see notes on classify.re
qmail:
optional? another admin tool? can't find anything
that calls this directly in upas or /mail/lib.
remotemail:
mandatory: used for sending email.
rewrite:
mandatory: used for rerouting email.
rewrite.direct:
rewrite.gateway:
example: copy to rewrite and edit
setup.bayes:
optional, stupid: sets up files that the bayesian
filter uses. The bayesian filter should just do
that itself if the files don't exist. Should
fix bayes and delete this.
smtpd.conf:
mandatory (for smtp servers): exactly what it says
on the tin, the smtp config.
validateaddress:
mandatory for smtpd: checks if the destination
of a received email is valid for this system.
maybe should turn it into an optional script,
and instead teach smtpd to check if a user
or alias exists directly.
validateattachment:
optional? mostly seems silly to have on by
default.
white.starter:
example? an example of what whitelists look like,
probably not needed.
With that, it seems like we could clean things up so that
/mail/lib defaults to containing:
/mail/lib/rewrite.example
/mail/lib/remotemail
/mail/lib/ignore (todo, remove?)
We can rewrite remotemail to default to routing based on
factotum and the from address, rather than needing a bunch
of config. Ideally, it'd be something like:
# allow per-user overrides
if(test -f $home/lib/mail/remotemail)
exec $home/lib/mail/remotemail
if(test -f $home/lib//mail/sender)
from=`$nl{cat $home/lib/mail/sender
# scrape server, username from factotum, route based
# on that.
user=`$nl{awk '/...server=.../{print user}' /mnt/factotum/ctl}
user=`$nl{awk '/...server=.../{print server}' /mnt/factotum/ctl}
I'm not sure how to get our domain into the
rewrite rules, so until we figure that out it'll have to
remain an example.
For smtpd, it seems like we can have:
/mail/lib/directmail.example/smtpd.conf
/mail/lib/directmail.example/validateaddress
/mail/lib/directmail.example/rewrite
/mail/lib/directmail.example/namefiles
/mail/lib/directmail.example/alias.local
The rest can probably be put into other examples.
/mail/lib/spam.example/...
/mail/lib/fishing.example/...
...etc...
The script stubs could either be inlined into /bin/upas/foo,
or put into:
/mail/rc/*.rc
and the admin scripts into
/mail/util/$script
I think this would make it easier to figure out what's going
on with upas configuration, and get a working system.
Thoughts? Complaints? "Don't touch the artwork!"?
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [9front] /mail/lib: cleanup proposal
2020-09-20 18:05 /mail/lib: cleanup proposal ori
@ 2020-09-20 19:01 ` Kurt H Maier
2020-09-20 20:19 ` hiro
2020-09-20 21:27 ` Stanley Lieber
2020-09-21 17:53 ` Lyndon Nerenberg
2 siblings, 1 reply; 8+ messages in thread
From: Kurt H Maier @ 2020-09-20 19:01 UTC (permalink / raw)
To: 9front
On Sun, Sep 20, 2020 at 11:05:01AM -0700, ori@eigenstate.org wrote:
>
> Thoughts? Complaints? "Don't touch the artwork!"?
>
Touch the hell out of this artwork. Your proposal would also simplify
a part of the fqa that people frequently complain about.
khm
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9front] /mail/lib: cleanup proposal
2020-09-20 18:05 /mail/lib: cleanup proposal ori
2020-09-20 19:01 ` [9front] " Kurt H Maier
@ 2020-09-20 21:27 ` Stanley Lieber
2020-09-21 4:48 ` ori
2020-09-21 17:53 ` Lyndon Nerenberg
2 siblings, 1 reply; 8+ messages in thread
From: Stanley Lieber @ 2020-09-20 21:27 UTC (permalink / raw)
To: 9front
On September 20, 2020 2:05:01 PM EDT, ori@eigenstate.org wrote:
>While it's fresh in my mind, I'm looking at /mail/lib. There
>are a lot of example files there, and it's a bit confusing
>what needs to be configured, when the scripts are invoked,
>what's intended to be configuration, and what's probably
>best to leave alone.
>
>What I'd like is to ship 9ront up with a minimal config
>that works out of the box for sending email through a
>remote SMTP server. Ideally, all we'd need would be
>setting a from address, and adding the keys in factotum.
>
>In an ideal world, I'd just have to do:
>
> echo 'ori@eigenstate.org' >> $home/lib/mail/sender
>
>I'd also like to take a minimal config for receiving email
>on @EXAMPLE.COM and put that into a separate directory, like:
>
> /mail/lib/example.direct
>
>For that, I need to understand what exactly belongs in a
>config, and what it does.
>
>So, to that end, here's my (likely wrong) understanding
>of what the files in /lib/mail do, and whether we actually
>need them.
>
>It seems like the shit in /mail/lib falls into five
>categories:
> - Config (and scripts that act as config)
> - Example config (which isn't labelled as such)
> - Bodies for /bin/upas/foo stubs
> - State for upas programs
> - Admin utilities not invoked by the mail system
>
>So, my current understanding of the files:
>
> blocked:
> optional; a list of ip addresses that are probably
> spammy. Horribly out of date, should probably not
> be in our default config.
>
> classify.re:
> useful for mail servers, but not human config, can
> we move to a place that makes it clear? /mail/state,
> or something?
>
> gone.fishing:
> optional: away message handling. would like to
> document, but not put into the example config.
>
> gone.msg:
> see above.
>
> ignore:
> needed, but why? a list of headers for upas/fs
> to filter out. I don't think any of our mail
> clients show the headers raw, seems like we
> can drop this file...?
>
> isspam.rc:
> spam.rc:
> unspam.rc:
> msgcat.rc:
> mandatory and not config. Exec'ed by /bin/upas/isspam;
> doesn't seem to be intended for edits by user. would
> like to the contents to /bin/upas/isspam, and maybe
> add an 'if(test -f /mail/lib/isspam.rc) exec...' for
> allowing customization.
>
> At the very least, would like to move out of /mail/lib
> to a place that makes it clear it's probably not
> for users to touch in most cases. /mail/rc? /mail/lib/rc?
>
> justqmail:
> optional? admin script? used by lazyqmail
>
> kickqueue:
> optional? admin util; do we need to have this
> in /lib/mail?
>
> mailnews:
> obsolete: seems to be a util for sending newsletters
> about plan 9 to a list of subscribers.
>
> namefiles:
> mandatory for smtpd: a list of alias lists; needed for
> smtpd config, not for local mailing.
>
> names.local:
> mandatory for smtpd: the alias list included by namefiles.
>
> patterns:
> optional. hardcoded spam regexes -- horribly out
> of date, probably not useful any more for fighting
> spam. If you want to get the same effect, you can
> probably do better with a pipeto; I'd like to
> delete support entirely.
>
> pipeto.bayes:
> optional: an example of how to send email to our
> bayesian spam filter. upas/bayes needs documentation.
>
> pipeto.lib:
> not config: a set of utility scripts that the .rc
> files include. Move to /mail/rc?
>
> prof.mbox:
> prof.spam:
> optional? not entirely sure what these are for.
> Seems to be some sort of state tracking for spam
> filtering? Anyone?
>
> If it's state tracking, see notes on classify.re
>
> qmail:
> optional? another admin tool? can't find anything
> that calls this directly in upas or /mail/lib.
>
> remotemail:
> mandatory: used for sending email.
>
> rewrite:
> mandatory: used for rerouting email.
>
> rewrite.direct:
> rewrite.gateway:
> example: copy to rewrite and edit
>
> setup.bayes:
> optional, stupid: sets up files that the bayesian
> filter uses. The bayesian filter should just do
> that itself if the files don't exist. Should
> fix bayes and delete this.
>
> smtpd.conf:
> mandatory (for smtp servers): exactly what it says
> on the tin, the smtp config.
>
> validateaddress:
> mandatory for smtpd: checks if the destination
> of a received email is valid for this system.
> maybe should turn it into an optional script,
> and instead teach smtpd to check if a user
> or alias exists directly.
>
> validateattachment:
> optional? mostly seems silly to have on by
> default.
>
> white.starter:
> example? an example of what whitelists look like,
> probably not needed.
>
>With that, it seems like we could clean things up so that
>/mail/lib defaults to containing:
>
> /mail/lib/rewrite.example
> /mail/lib/remotemail
> /mail/lib/ignore (todo, remove?)
>
>We can rewrite remotemail to default to routing based on
>factotum and the from address, rather than needing a bunch
>of config. Ideally, it'd be something like:
>
> # allow per-user overrides
> if(test -f $home/lib/mail/remotemail)
> exec $home/lib/mail/remotemail
> if(test -f $home/lib//mail/sender)
> from=`$nl{cat $home/lib/mail/sender
> # scrape server, username from factotum, route based
> # on that.
> user=`$nl{awk '/...server=.../{print user}' /mnt/factotum/ctl}
> user=`$nl{awk '/...server=.../{print server}' /mnt/factotum/ctl}
>
>I'm not sure how to get our domain into the
>rewrite rules, so until we figure that out it'll have to
>remain an example.
>
>For smtpd, it seems like we can have:
>
> /mail/lib/directmail.example/smtpd.conf
> /mail/lib/directmail.example/validateaddress
> /mail/lib/directmail.example/rewrite
> /mail/lib/directmail.example/namefiles
> /mail/lib/directmail.example/alias.local
>
>The rest can probably be put into other examples.
>
> /mail/lib/spam.example/...
> /mail/lib/fishing.example/...
> ...etc...
>
>The script stubs could either be inlined into /bin/upas/foo,
>or put into:
>
> /mail/rc/*.rc
>
>and the admin scripts into
>
> /mail/util/$script
>
>I think this would make it easier to figure out what's going
>on with upas configuration, and get a working system.
>
>Thoughts? Complaints? "Don't touch the artwork!"?
please keep in mind all our shit (read: this mailing list, and my personal mail) runs on upas before committing breaking changes.
strongly in favor of simplifying setup. whatever we're currently shipping is the remnants of what actually ran at the labs. all that configuration (anti-spam, etc.) was theirs. i got all my own stuff working by reading the man pages and trial and error.
i thought incoming messages were piped through qmail somewhere along the line.
sl
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [9front] /mail/lib: cleanup proposal
2020-09-20 21:27 ` Stanley Lieber
@ 2020-09-21 4:48 ` ori
2020-10-04 23:35 ` ori
0 siblings, 1 reply; 8+ messages in thread
From: ori @ 2020-09-21 4:48 UTC (permalink / raw)
To: sl, 9front
> > I think this would make it easier to figure out what's going
> > on with upas configuration, and get a working system.
> >
> > Thoughts? Complaints? "Don't touch the artwork!"?
>
> please keep in mind all our shit (read: this mailing list, and my
> personal mail) runs on upas before committing breaking changes.
Yep, definitely. That's why I'm trying to validate my assumptions
on what's actually considered configuration, and what's dumped
in there.
Note, /mail/lib isn't in hg, so sysupdate doesn't touch it.
The changes I'm proposing would be in the defaults for new
installs.
Any breakage would be from changes in /sys/src/cmd/upas, and
I don't see any problem with keeping things compatible. At
most it'd mean checking an extra path to decide what command
to run.
> strongly in favor of simplifying setup. whatever we're currently
> shipping is the remnants of what actually ran at the labs. all that
> configuration (anti-spam, etc.) was theirs. i got all my own stuff
> working by reading the man pages and trial and error.
Yeah. I'm doing that now as well for testing the smtpd changes
for date and time. I'm realizing that this stuff is simpler than
it looks, but the example config really overcomplicates things.
> i thought incoming messages were piped through qmail somewhere along
> the line.
Ah, I missed it in the rewrite file. Thanks for the cross check.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9front] /mail/lib: cleanup proposal
2020-09-21 4:48 ` ori
@ 2020-10-04 23:35 ` ori
2020-10-05 5:15 ` sirjofri+ml-9front
0 siblings, 1 reply; 8+ messages in thread
From: ori @ 2020-10-04 23:35 UTC (permalink / raw)
To: ori, sl, 9front
>> > I think this would make it easier to figure out what's going
>> > on with upas configuration, and get a working system.
>> >
>> > Thoughts? Complaints? "Don't touch the artwork!"?
>>
>> please keep in mind all our shit (read: this mailing list, and my
>> personal mail) runs on upas before committing breaking changes.
>
> Yep, definitely. That's why I'm trying to validate my assumptions
> on what's actually considered configuration, and what's dumped
> in there.
>
> Note, /mail/lib isn't in hg, so sysupdate doesn't touch it.
> The changes I'm proposing would be in the defaults for new
> installs.
>
> Any breakage would be from changes in /sys/src/cmd/upas, and
> I don't see any problem with keeping things compatible. At
> most it'd mean checking an extra path to decide what command
> to run.
>
>> strongly in favor of simplifying setup. whatever we're currently
>> shipping is the remnants of what actually ran at the labs. all that
>> configuration (anti-spam, etc.) was theirs. i got all my own stuff
>> working by reading the man pages and trial and error.
>
> Yeah. I'm doing that now as well for testing the smtpd changes
> for date and time. I'm realizing that this stuff is simpler than
> it looks, but the example config really overcomplicates things.
>
>> i thought incoming messages were piped through qmail somewhere along
>> the line.
>
> Ah, I missed it in the rewrite file. Thanks for the cross check.
So, after messing around with this over the weekend, I think I
have a reasonable few defaults for /mail/lib.
The files that we need for reasonable defaults to send mail:
/mail/lib/ignore
/mail/lib/qmail
/mail/lib/remotemail
/mail/lib/rewrite
I'd like to make upas/fs use reasonable defaults for the ignore
file so we can remove it (and maybe flip it to a keephdr file).
I've also rewritten remotemail, so that it uses factotum to figure
out where to send to. That means that configuring mail is adding
upasname=user@server.com
to your environment, and adding an 'email=user@server.com' to your
smtp factotum key so we can decide which smtp server to use:
echo 'key proto=pass server=server.com ' \
'service=smtp user=user >>email=user@server.com<< !password=12345'
Here's the rewritten remotemail:
#!/bin/rc
# allow users to do their own mail setup
if(test -x $home/lib/mail/remotemail)
exec $home/mail/lib/remotemail
sender=$2
svcpat='[ ]service=smtp[ ]'
addrpat='[ ]email='$2'[ ]'
config=`{grep -e $svcpat -e $addrpat /mnt/factotum/ctl}
if(~ $#config 0)
exit 'no server'
for(kv in $config){
parts=`'='{echo -n $kv}
switch($parts(1)){
case server; server=$parts(2)
case user; login=$parts(2)
}
}
exec /bin/upas/smtp -as -u $login $server $addr $sender $*(4-)
The ignore and qmail files are the same as we ship today.
The rewrite rules are the same as our cureent rewrite.gateway,
though I want to slim it down a bit, or at least comment it
better. Separate mail incoming for that.
The other files seem like they can become examples -- I'll look at
putting together a proposal for some example configurations that
we can ship separately.
Is there anything that this setup is missing?
Is there some edge case that this will break on?
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [9front] /mail/lib: cleanup proposal
2020-10-04 23:35 ` ori
@ 2020-10-05 5:15 ` sirjofri+ml-9front
0 siblings, 0 replies; 8+ messages in thread
From: sirjofri+ml-9front @ 2020-10-05 5:15 UTC (permalink / raw)
To: 9front
Hey ori,
> I've also rewritten remotemail, so that it uses factotum to figure
> out where to send to. That means that configuring mail is adding
>
> upasname=user@server.com
>
> to your environment, and adding an 'email=user@server.com' to your
> smtp factotum key so we can decide which smtp server to use:
>
> echo 'key proto=pass server=server.com ' \
> 'service=smtp user=user >>email=user@server.com<< !password=12345'
>
> Here's the rewritten remotemail:
>
> #!/bin/rc
>
> # allow users to do their own mail setup
> if(test -x $home/lib/mail/remotemail)
> exec $home/mail/lib/remotemail
>
> sender=$2
> svcpat='[ ]service=smtp[ ]'
> addrpat='[ ]email='$2'[ ]'
> config=`{grep -e $svcpat -e $addrpat /mnt/factotum/ctl}
> if(~ $#config 0)
> exit 'no server'
> for(kv in $config){
> parts=`'='{echo -n $kv}
> switch($parts(1)){
> case server; server=$parts(2)
> case user; login=$parts(2)
> }
> }
>
> exec /bin/upas/smtp -as -u $login $server $addr $sender $*(4-)
I like the changes. Also may I suggest that you add my patch for smtp?
Then unrecognized certificates during smtp can be ignored using a flag
resulting in encrypted mails. Modern providers like this, although it
makes mails somewhat unsecure (mails are encrypted which is more secure,
but no one can ensure you're talking to the right server). Otherwise we
need to handle certificates better (maybe trust on first use?)
Also since my patch provides a flag server admins can choose which
security feature they want.
sirjofri
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9front] /mail/lib: cleanup proposal
2020-09-20 18:05 /mail/lib: cleanup proposal ori
2020-09-20 19:01 ` [9front] " Kurt H Maier
2020-09-20 21:27 ` Stanley Lieber
@ 2020-09-21 17:53 ` Lyndon Nerenberg
2 siblings, 0 replies; 8+ messages in thread
From: Lyndon Nerenberg @ 2020-09-21 17:53 UTC (permalink / raw)
To: 9front
ori@eigenstate.org writes:
> mailnews:
> obsolete: seems to be a util for sending newsletters
> about plan 9 to a list of subscribers.
Please keep this. It's used to mail new entries in /lib/news
to a list of subscribers. It's very handy for people who don't
log in regularly to keep up to date with system changes.
--lyndon
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-10-05 5:15 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-20 18:05 /mail/lib: cleanup proposal ori
2020-09-20 19:01 ` [9front] " Kurt H Maier
2020-09-20 20:19 ` hiro
2020-09-20 21:27 ` Stanley Lieber
2020-09-21 4:48 ` ori
2020-10-04 23:35 ` ori
2020-10-05 5:15 ` sirjofri+ml-9front
2020-09-21 17:53 ` Lyndon Nerenberg
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).