making sendmail call to msmtp asynchronous?

making sendmail call to msmtp asynchronous?

[Eric Abrahamsen]

I'm trying to make mail sending/receiving as non-blocking as possible. I
was pleased with msmtp and having started using the sister program mpop
to feed gnus mail. I read four different POP accounts and it was slow.
Now I'm wondering how hard it would be to make the call to msmtp for
outgoing mail asynchronous. Would it be possible just to dump the mail
on msmtp, and have a callback that either pops up a buffer (for errors)
or a message in the minibuffer (for success)? How difficult would that
be?

Thanks!
Eric

Re: making sendmail call to msmtp asynchronous?

[Erik Colson]

Ehlo,

msmtp does not support queueing. Therefor it 'blocks' every mail client
which sends mail through it. My opinion is that the fix should be msmtp
to support queues.
Of course asynchronous connections in gnus would be welcome, but in this
case I think the problem is at msmtp's side.

So the obvious solution is to move to postfix which added support for
smtp-server selection based on the sender's address envelope in version
2.7 I think...

PS: Look at "sender_dependent_default_transport_maps" in postfix
documentation for this feature

Best
-- 
erik colson
ecocode.net

Re: making sendmail call to msmtp asynchronous?

[Eric Abrahamsen]

Erik Colson <eco@ecocode.net> writes:

> Ehlo,
>
> msmtp does not support queueing. Therefor it 'blocks' every mail client
> which sends mail through it. My opinion is that the fix should be msmtp
> to support queues.
> Of course asynchronous connections in gnus would be welcome, but in this
> case I think the problem is at msmtp's side.
>
> So the obvious solution is to move to postfix which added support for
> smtp-server selection based on the sender's address envelope in version
> 2.7 I think...
>
> PS: Look at "sender_dependent_default_transport_maps" in postfix
> documentation for this feature
>
> Best

Thanks! Msmtp does come with a "queuing" script, but I don't know if
it's queuing in the proper sense, that would allow it not to block, I'll
take a look. Thanks for the hints!

E

Re: making sendmail call to msmtp asynchronous?

[Lars Magne Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Would it be possible just to dump the mail on msmtp, and have a
> callback that either pops up a buffer (for errors) or a message in the
> minibuffer (for success)? How difficult would that be?

It sounds kinda complicated to me, and probably with not a lot of gain
to be had.  Unless you're sending massive amounts of massive emails.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: making sendmail call to msmtp asynchronous?

[Eric Abrahamsen]

On Wed, Mar 30 2011, Lars Magne Ingebrigtsen wrote:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Would it be possible just to dump the mail on msmtp, and have a
>> callback that either pops up a buffer (for errors) or a message in the
>> minibuffer (for success)? How difficult would that be?
>
> It sounds kinda complicated to me, and probably with not a lot of gain
> to be had.  Unless you're sending massive amounts of massive emails.  :-)

Fair enough!

Re: making sendmail call to msmtp asynchronous?

[Richard Riley]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On Wed, Mar 30 2011, Lars Magne Ingebrigtsen wrote:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Would it be possible just to dump the mail on msmtp, and have a
>>> callback that either pops up a buffer (for errors) or a message in the
>>> minibuffer (for success)? How difficult would that be?
>>
>> It sounds kinda complicated to me, and probably with not a lot of gain
>> to be had.  Unless you're sending massive amounts of massive emails.  :-)
>
> Fair enough!
>

I used to use msmtp to talk to gmail - amongst others.

The problem was the time it sometimes took - hanging gnus for upto 15
seconds or so.

(Linux solution)

So I went back to using exim4 : gnus dumps the send off to exim4 and
exim does its stuff in the bg.

Not without pain. Of course. Exim4 is up there with the worst when it
comes to "simple tweaks" and enabling exim4 for multiple smarthosts
proved impossible (for me) give or take despite some old configs out
there in google land for exim1-3 but not 4. Why not use exim4s smtp send
facility? Its a pain if you travel : here I sit on holiday and smtp
ports for outgoing are blocked by the ISP and/or the hotel router. Gah!

Using now only gmail servers I found a nice solution that might help
some others. Yes, use exim4 smarthost. Use one of your gmail smtp
servers for outgoing and so configure that in your exim4 smarthost
setup. Normally it rewrites your "from" address to that of the owner of
the smtp account. You an stop this by granting permission to other gmail
accounts to use this account on its behalf. So if on account A I allow
account B access then account B's "from address" may be listed in the
outgoing email sent by A's SMTP server. This option is in
account/additional email users via the gmail web interface. 

http://www.besttechie.net/2010/03/27/how-to-use-gmail-for-multiple-email-accounts/

This is not without risk if you're being spammy/trolling etc as the
header details will still reveal the account name/owner of the SMTP
server. I guess the solution in that case if you dont want that is to
simply create an "anonymous" gmail account in order just to use that
smtp service for that account from your other gmail accounts.

Still, its another option and "works for me".

Re: making sendmail call to msmtp asynchronous?

[Matthias Andree]

On Wed, Mar 30, 2011 at 08:38:46AM +0200, Richard Riley wrote:
> Not without pain. Of course. Exim4 is up there with the worst when it
> comes to "simple tweaks" and enabling exim4 for multiple smarthosts
> proved impossible (for me) give or take despite some old configs out
> there in google land for exim1-3 but not 4. Why not use exim4s smtp send
> facility? Its a pain if you travel : here I sit on holiday and smtp
> ports for outgoing are blocked by the ISP and/or the hotel router. Gah!

While I could offer you sample configs from my Cygwin installation, I've
found out the hard way (this time on FreeBSD) that Exim4 has a very
awkward lock-destination-sites behaviour that requires major manual
interventions to purge the retry/site database to get mails unstuck,
thus I decided I'm not going to install any more of that.

While Postfix is a bit more of an effort to configure (enable
sender-based authentication, enable sender-based relay, enable smtp (not
smtpd)-side tls, enable smtp-side sasl, permit plaintext authentication
on secure tls channels, set up all the maps, I find it's more
transparent and has less magic special casing underneath that confuses
the heck out of myself.

Re: making sendmail call to msmtp asynchronous?

[Richard Riley]

Matthias Andree <ma+gnus@dt.e-technik.uni-dortmund.de> writes:

> On Wed, Mar 30, 2011 at 08:38:46AM +0200, Richard Riley wrote:
>> Not without pain. Of course. Exim4 is up there with the worst when it
>> comes to "simple tweaks" and enabling exim4 for multiple smarthosts
>> proved impossible (for me) give or take despite some old configs out
>> there in google land for exim1-3 but not 4. Why not use exim4s smtp send
>> facility? Its a pain if you travel : here I sit on holiday and smtp
>> ports for outgoing are blocked by the ISP and/or the hotel router. Gah!
>
> While I could offer you sample configs from my Cygwin installation, I've
> found out the hard way (this time on FreeBSD) that Exim4 has a very
> awkward lock-destination-sites behaviour that requires major manual
> interventions to purge the retry/site database to get mails unstuck,
> thus I decided I'm not going to install any more of that.
>
> While Postfix is a bit more of an effort to configure (enable
> sender-based authentication, enable sender-based relay, enable smtp (not
> smtpd)-side tls, enable smtp-side sasl, permit plaintext authentication
> on secure tls channels, set up all the maps, I find it's more
> transparent and has less magic special casing underneath that confuses
> the heck out of myself.
>

Much as I dislike Exim4 docs, I have to stick up for it here. Using a
single smarthost as I described is pretty straightforward.

Clearing frozen mails caused by a destination refusing your mail because
your IP is blacklisted or smtp ports are blocked is a google away.

I did originally use the exim4 sendmail interpretation (having first had
my business/home IP cleared out of spamhaus! It took about a day for all
major email providers to get the updated spam lists and see my ip
cleared) and that was great but of course my recent roaming adventures
put paid to that and its back to gmail smtp smart hosts.

-- 

☘ http://www.shamrockirishbar.com, http://splash-of-open-sauce.blogspot.com/ http://www.richardriley.net

using Exim 4 w/ multiple smarthosts (was: making sendmail call to msmtp asynchronous?)

[Matthias Andree]

Am 30.03.2011 14:01, schrieb Richard Riley:
> Matthias Andree <ma+gnus@dt.e-technik.uni-dortmund.de> writes:
> 
>> On Wed, Mar 30, 2011 at 08:38:46AM +0200, Richard Riley wrote:
>>> Not without pain. Of course. Exim4 is up there with the worst when it
>>> comes to "simple tweaks" and enabling exim4 for multiple smarthosts
>>> proved impossible (for me) give or take despite some old configs out
>>> there in google land for exim1-3 but not 4. Why not use exim4s smtp send
>>> facility? Its a pain if you travel : here I sit on holiday and smtp
>>> ports for outgoing are blocked by the ISP and/or the hotel router. Gah!
>>
>> While I could offer you sample configs from my Cygwin installation, I've
>> found out the hard way (this time on FreeBSD) that Exim4 has a very
>> awkward lock-destination-sites behaviour that requires major manual
>> interventions to purge the retry/site database to get mails unstuck,
>> thus I decided I'm not going to install any more of that.
>>
>> While Postfix is a bit more of an effort to configure (enable
>> sender-based authentication, enable sender-based relay, enable smtp (not
>> smtpd)-side tls, enable smtp-side sasl, permit plaintext authentication
>> on secure tls channels, set up all the maps, I find it's more
>> transparent and has less magic special casing underneath that confuses
>> the heck out of myself.
>>
> 
> Much as I dislike Exim4 docs, I have to stick up for it here. Using a
> single smarthost as I described is pretty straightforward.

The docs are quite detailed, however the hard-wired defaults and to a
lesser extent the default configuration take you in for a few surprises.

> Clearing frozen mails caused by a destination refusing your mail because
> your IP is blacklisted or smtp ports are blocked is a google away.

...or just because the Exim4 host has been down for a while.  BTST, and
that takes users by surprise, so you need to figure out how to kick and
purge /var/spool/exim/db/* so that exim actually tries again.  It's not
sufficient to use exim -qff in such situations. :-(

The delay_after_cutoff=true default that can cause routers to get jammed
and bounce all mail is quite unobvious...


It's not Exim's TLS/SASL configuration or sender-dependent smarthost
configuration, but exactly the retry-, wait- and delay_after_cutoff
stuff.  I haven't had such nasty effects as I've had with Exim's
remote_smtp in a dozen years with Postfix.  Anyways, here we go, a few
comments inlined.  Works for me with the default Exim configuration
around it with Exim 4.70 on Cygwin 1.5 and 1.7 and Exim 4.75 on FreeBSD
8.2 (the original intent was to log everything while I had to use
Outlook 2003 and didn't trust it and make sure that I know where the
credentials are so that I could exclude them from or encrypt them for
the networked backup).

Note that Postfix supports per-sender authentication, too, not just
per-smarthost authentication.  In that case, Postfix figures out by
itself that it can only reuse TLS connections for mail from the same
sender. I wouldn't know how to tell Exim4 that.

This snippet, however, probably would not exist had there been a Postfix
port to Windows or Cygwin in 2007 :-)

########################################################################
# Exim 4 sender-dependent smarthosts:

begin routers

smarthost1:
   driver = manualroute
   domains = ! +local_domains
   senders = address1@example.org : address2@example.org
   transport = remote_smtp_ssl
   route_data = "mailhost1.example.net::587"

# smarthost2 also for bounces (nothing between = and :)
smarthost2:
   driver = manualroute
   domains = ! +local_domains
   senders = : domain2.example
   transport = remote_smtp_ssl
   route_data = "hermes2.example.org::587"

# smarthost3 uses an autossh tunnel:
smarthost_freebsd:
   driver = manualroute
   domains = ! +local_domains
   senders = FreeBSD.org
   transport = remote_smtp
   route_data = "localhost::1234"
   self = send

# other stuff (forward, local delivery etc.) goes here.
# the example configure file is quite sound.

# ...

begin transports

# ...

# note this is the simple way, all remote_smtp_ssl
# routed mail requires TLS and AUTH and assumes
# trusted certs in /etc/ssl/certs
#
remote_smtp_ssl:
  driver = smtp
  hosts_require_tls  = *
  hosts_require_auth = *
  tls_verify_certificates = /etc/ssl/certs
  delay_after_cutoff = false

# ...

begin authenticators

PLAIN:
  driver                     = plaintext
  server_set_id              = $auth2
  server_prompts             = :
  server_condition           = Authentication is not yet configured
  server_advertise_condition = ${if def:tls_cipher }
  client_send =
"${extract{auth_plain}{${lookup{$host}lsearch{/usr/local/etc/exim/smtp_auth}{$value}fail}}}"

#############################################################

... the credentials are in the file /usr/local/etc/exim/smtp_auth
(buried deep inside client_send) in the form

mailhost1.example.net:                   auth_plain=^USERNAME^PASSWORD

########


HTH
Matthias

Re: using Exim 4 w/ multiple smarthosts

[Richard Riley]

Matthias Andree <ma+gnus@dt.e-technik.uni-dortmund.de> writes:

> Am 30.03.2011 14:01, schrieb Richard Riley:
>> Matthias Andree <ma+gnus@dt.e-technik.uni-dortmund.de> writes:
>> 
>>> On Wed, Mar 30, 2011 at 08:38:46AM +0200, Richard Riley wrote:
>>>> Not without pain. Of course. Exim4 is up there with the worst when it
>>>> comes to "simple tweaks" and enabling exim4 for multiple smarthosts
>>>> proved impossible (for me) give or take despite some old configs out
>>>> there in google land for exim1-3 but not 4. Why not use exim4s smtp send
>>>> facility? Its a pain if you travel : here I sit on holiday and smtp
>>>> ports for outgoing are blocked by the ISP and/or the hotel router. Gah!
>>>
>>> While I could offer you sample configs from my Cygwin installation, I've
>>> found out the hard way (this time on FreeBSD) that Exim4 has a very
>>> awkward lock-destination-sites behaviour that requires major manual
>>> interventions to purge the retry/site database to get mails unstuck,
>>> thus I decided I'm not going to install any more of that.
>>>
>>> While Postfix is a bit more of an effort to configure (enable
>>> sender-based authentication, enable sender-based relay, enable smtp (not
>>> smtpd)-side tls, enable smtp-side sasl, permit plaintext authentication
>>> on secure tls channels, set up all the maps, I find it's more
>>> transparent and has less magic special casing underneath that confuses
>>> the heck out of myself.
>>>
>> 
>> Much as I dislike Exim4 docs, I have to stick up for it here. Using a
>> single smarthost as I described is pretty straightforward.
>
> The docs are quite detailed, however the hard-wired defaults and to a
> lesser extent the default configuration take you in for a few surprises.
>
>> Clearing frozen mails caused by a destination refusing your mail because
>> your IP is blacklisted or smtp ports are blocked is a google away.
>
> ...or just because the Exim4 host has been down for a while.  BTST, and
> that takes users by surprise, so you need to figure out how to kick and
> purge /var/spool/exim/db/* so that exim actually tries again.  It's not
> sufficient to use exim -qff in such situations. :-(
>
> The delay_after_cutoff=true default that can cause routers to get jammed
> and bounce all mail is quite unobvious...
>
> It's not Exim's TLS/SASL configuration or sender-dependent smarthost
> configuration, but exactly the retry-, wait- and delay_after_cutoff
> stuff.  I haven't had such nasty effects as I've had with Exim's
> remote_smtp in a dozen years with Postfix.  Anyways, here we go, a few
> comments inlined.  Works for me with the default Exim configuration
> around it with Exim 4.70 on Cygwin 1.5 and 1.7 and Exim 4.75 on FreeBSD
> 8.2 (the original intent was to log everything while I had to use
> Outlook 2003 and didn't trust it and make sure that I know where the
> credentials are so that I could exclude them from or encrypt them for
> the networked backup).
>
> Note that Postfix supports per-sender authentication, too, not just
> per-smarthost authentication.  In that case, Postfix figures out by
> itself that it can only reuse TLS connections for mail from the same
> sender. I wouldn't know how to tell Exim4 that.
>
> This snippet, however, probably would not exist had there been a Postfix
> port to Windows or Cygwin in 2007 :-)
>
> ########################################################################
> # Exim 4 sender-dependent smarthosts:
>
> begin routers
>
> smarthost1:
>    driver = manualroute
>    domains = ! +local_domains
>    senders = address1@example.org : address2@example.org
>    transport = remote_smtp_ssl
>    route_data = "mailhost1.example.net::587"
>
> # smarthost2 also for bounces (nothing between = and :)
> smarthost2:
>    driver = manualroute
>    domains = ! +local_domains
>    senders = : domain2.example
>    transport = remote_smtp_ssl
>    route_data = "hermes2.example.org::587"
>
> # smarthost3 uses an autossh tunnel:
> smarthost_freebsd:
>    driver = manualroute
>    domains = ! +local_domains
>    senders = FreeBSD.org
>    transport = remote_smtp
>    route_data = "localhost::1234"
>    self = send
>
> # other stuff (forward, local delivery etc.) goes here.
> # the example configure file is quite sound.
>
> # ...
>
> begin transports
>
> # ...
>
> # note this is the simple way, all remote_smtp_ssl
> # routed mail requires TLS and AUTH and assumes
> # trusted certs in /etc/ssl/certs
> #
> remote_smtp_ssl:
>   driver = smtp
>   hosts_require_tls  = *
>   hosts_require_auth = *
>   tls_verify_certificates = /etc/ssl/certs
>   delay_after_cutoff = false
>
> # ...
>
> begin authenticators
>
> PLAIN:
>   driver                     = plaintext
>   server_set_id              = $auth2
>   server_prompts             = :
>   server_condition           = Authentication is not yet configured
>   server_advertise_condition = ${if def:tls_cipher }
>   client_send =
> "${extract{auth_plain}{${lookup{$host}lsearch{/usr/local/etc/exim/smtp_auth}{$value}fail}}}"
>
> #############################################################
>
> ... the credentials are in the file /usr/local/etc/exim/smtp_auth
> (buried deep inside client_send) in the form
>
> mailhost1.example.net:                   auth_plain=^USERNAME^PASSWORD
>
> ########
>
> HTH
> Matthias

Interesting post, but I think this doesnt work with multiple gmail smtp
servers since they are all keyed by the same smtp server name. It needs
to be keyed by the from address. Hairy stuff all in all. I am surprised
its not a default config example to be honest. I do have a set up in
emacs for dynamically channging he smtp server and credentials based on
from/posting style in emacs but of course this isnt asynchronous and
causes an "intolerable" (aren't we spoiled these days? ;)) delay.



-- 
☘ http://www.shamrockirishbar.com, http://splash-of-open-sauce.blogspot.com/ http://www.richardriley.net

Re: making sendmail call to msmtp asynchronous?

[Leonidas Tsampros]

Matthias Andree <ma+gnus@dt.e-technik.uni-dortmund.de> writes:
> On Wed, Mar 30, 2011 at 08:38:46AM +0200, Richard Riley wrote:
>> Not without pain. Of course. Exim4 is up there with the worst when it
>> comes to "simple tweaks" and enabling exim4 for multiple smarthosts
>> proved impossible (for me) give or take despite some old configs out
>> there in google land for exim1-3 but not 4. Why not use exim4s smtp send
>> facility? Its a pain if you travel : here I sit on holiday and smtp
>> ports for outgoing are blocked by the ISP and/or the hotel router. Gah!
>
> While I could offer you sample configs from my Cygwin installation, I've
> found out the hard way (this time on FreeBSD) that Exim4 has a very
> awkward lock-destination-sites behaviour that requires major manual
> interventions to purge the retry/site database to get mails unstuck,
> thus I decided I'm not going to install any more of that.
>
> While Postfix is a bit more of an effort to configure (enable
> sender-based authentication, enable sender-based relay, enable smtp (not
> smtpd)-side tls, enable smtp-side sasl, permit plaintext authentication
> on secure tls channels, set up all the maps, I find it's more
> transparent and has less magic special casing underneath that confuses
> the heck out of myself.

Hi guys,

having tried to do the above, I eventually ended up using smtp.gmail.com
as my only postfix smarthost with smtp-side sasl, relaying from there
all of my sender addresses. (There are some buttons for this procedure
on the gmail settings).

Also, I ended up using this setup too

        http://www.emacswiki.org/emacs/MultipleSMTPAccounts

since I need to use other smtp servers occassionally too.

Ping me offlist if you want sample postfix config, but I think it's dead
easy to do.

The only + of my suggestions is that it's the simplest outgoing email
setup I could think of. I'm using this setup for the last 6 months and
I'm pretty happy.

Best Regards,
Leonidas Tsampros

Re: making sendmail call to msmtp asynchronous?

[Sivaram Neelakantan]

On Thu, Mar 31 2011,Leonidas Tsampros wrote:


[snipped 22 lines]

> Hi guys,
>
> having tried to do the above, I eventually ended up using smtp.gmail.com
> as my only postfix smarthost with smtp-side sasl, relaying from there
> all of my sender addresses. (There are some buttons for this procedure
> on the gmail settings).
>
> Also, I ended up using this setup too
>
>         http://www.emacswiki.org/emacs/MultipleSMTPAccounts
>
> since I need to use other smtp servers occassionally too.
>
> Ping me offlist if you want sample postfix config, but I think it's dead
> easy to do.
>
> The only + of my suggestions is that it's the simplest outgoing email
> setup I could think of. I'm using this setup for the last 6 months and
> I'm pretty happy.
>
> Best Regards,
> Leonidas Tsampros
>
>

Why don't you update the emacswiki with your setup, if you have time?
 Possibly link it from the msmtp/multiplesmtpaccounts page?

 sivaram
 --