New imap Implementation and Keepalive

New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]


Firstly I want to thank everyone involved for the speedup in the new
imap implementation. However, I am encountering one annoying problem- it
seems like the new implementation does not keep the connection alive
like the old one does. In the old implementation I just need to enter my
imap password once interactively per Gnus session (I wasn't using
auth-sources at that time). With the new implementation my imaps server
(dovecot) drops connection due to inactivity and I need to enter my long
gpg passphrase again if my pass phrase cache have expired.

Thanks.

Charles

-- 
Why use Windows, since there is a door?
(By fachat@galileo.rhein-neckar.de, Andre Fachat)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Charles Philip Chan <cpchan@sympatico.ca> writes:

> However, I am encountering one annoying problem- it seems like the new
> implementation does not keep the connection alive like the old one
> does.

What's the recommended keepalive method?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Charles Philip Chan <cpchan@sympatico.ca> writes:
>
>> However, I am encountering one annoying problem- it seems like the new
>> implementation does not keep the connection alive like the old one
>> does.
>
> What's the recommended keepalive method?

I think you send the NOOP command.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> I think you send the NOOP command.

Right.  How often is it natural to do so?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Frank Schmitt <ich@frank-schmitt.net> writes:
>
>> I think you send the NOOP command.
>
> Right.  How often is it natural to do so?

Mutt sends it every 15 minutes and their FAQ says servers are required
to keep the connection alive for 30. See
http://wiki.mutt.org/?MuttFaq/RemoteFolder

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> Mutt sends it every 15 minutes and their FAQ says servers are required
> to keep the connection alive for 30. See
> http://wiki.mutt.org/?MuttFaq/RemoteFolder

Right.  Hm...  I wonder what's the simplest way to implement it....
Just a run-at-time timer, I think...  and I'll have to keep track of
when the last command was so that I don't issue a NOOP while something
else is doing...  stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Uday S Reddy]

On 9/23/2010 3:42 AM, Charles Philip Chan wrote:
>
> Firstly I want to thank everyone involved for the speedup in the new
> imap implementation. However, I am encountering one annoying problem- it
> seems like the new implementation does not keep the connection alive
> like the old one does. In the old implementation I just need to enter my
> imap password once interactively per Gnus session (I wasn't using
> auth-sources at that time). With the new implementation my imaps server
> (dovecot) drops connection due to inactivity and I need to enter my long
> gpg passphrase again if my pass phrase cache have expired.

I think it is a bit unkind to keep IMAP sessions alive just for the sake of it. 
  The mail server administrators set their time-out policies to make the best 
possible use of resources.  If everybody forcibly keeps their sessions alive, 
the mail servers are likely to run out of connections and refuse new connections.

If the problem is really that the password cache is expiring, it is far better 
to block that instead of trying to tax the mail server.

My two cents.

Cheers,
Uday

Re: New imap Implementation and Keepalive

[Frank Schmitt]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> On 9/23/2010 3:42 AM, Charles Philip Chan wrote:
>>
>> Firstly I want to thank everyone involved for the speedup in the new
>> imap implementation. However, I am encountering one annoying problem- it
>> seems like the new implementation does not keep the connection alive
>> like the old one does. In the old implementation I just need to enter my
>> imap password once interactively per Gnus session (I wasn't using
>> auth-sources at that time). With the new implementation my imaps server
>> (dovecot) drops connection due to inactivity and I need to enter my long
>> gpg passphrase again if my pass phrase cache have expired.
>
> I think it is a bit unkind to keep IMAP sessions alive just for the
> sake of it. The mail server administrators set their time-out policies
> to make the best possible use of resources.  If everybody forcibly
> keeps their sessions alive, the mail servers are likely to run out of
> connections and refuse new connections.
>
> If the problem is really that the password cache is expiring, it is
> far better to block that instead of trying to tax the mail server.

No. For IMAP it is perfectly normal that the client keeps the connection
open as long as he is active. It's even in the RFC. Otherwise e.g. IMAP
idle wouldn't be possible. The NOOP command is there for exactly this purpose.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> No. For IMAP it is perfectly normal that the client keeps the
> connection open as long as he is active. It's even in the
> RFC. Otherwise e.g. IMAP idle wouldn't be possible. The NOOP command
> is there for exactly this purpose.

I've now implemented this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've now implemented this.

Thanks Lar. I can see that the connection is keepalive by this:

,----[ Output of "ps aux | grep imap" ]
| dovecot  13679  0.0  0.0  15244  2756 ?        Ss   Sep24   0:00 imap-login --ssl
| hoor     13682  0.0  0.1  21272  5564 ?        S    Sep24   0:02 imap
| hoor     16713  0.0  0.0   6532   808 pts/4    S+   00:38   0:00 grep imap
`----

However, there is still one annoying problem- pinentry-gtk-2 is still
popping up (I need to hit "cancel" every time).

Thanks.

Charles

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

Charles Philip Chan <cpchan@sympatico.ca> writes:

> However, there is still one annoying problem- pinentry-gtk-2 is still
> popping up (I need to hit "cancel" every time).

I spoke too soon. If I hit "cancel", I have no problems using my dovecot
server. However, it cannot enter my local leafnode server (for nntp)
which requires no auth.

Thanks.

Charles

-- 
"It's God.  No, not Richard Stallman, or Linus Torvalds, but God."
(By Matt Welsh)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Charles Philip Chan <cpchan@sympatico.ca> writes:

>> However, there is still one annoying problem- pinentry-gtk-2 is still
>> popping up (I need to hit "cancel" every time).
>
> I spoke too soon. If I hit "cancel", I have no problems using my dovecot
> server. However, it cannot enter my local leafnode server (for nntp)
> which requires no auth.

I'm not sure I understand.  After the latest nnimap change, you're
prompted for passwords for your nntp server, which requires no password? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm not sure I understand.  After the latest nnimap change, you're
> prompted for passwords for your nntp server, which requires no
> password?

No, what I mean is after the latest change the server is kept alive, but
pinentry-gtk-2 is still popping up. If I click "cancel" on the dialog, I
can enter my imap groups with no problem, but gnus cannot connect to my
leafnode server. However, if I enter in my passphrase, everything is
fine.

Thanks.
Charles 

-- 
"I'd crawl over an acre of 'Visual This++' and 'Integrated Development
That' to get to gcc, Emacs, and gdb.  Thank you."
(By Vance Petree, Virginia Power)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Charles Philip Chan <cpchan@sympatico.ca> writes:

> No, what I mean is after the latest change the server is kept alive, but
> pinentry-gtk-2 is still popping up. If I click "cancel" on the dialog, I
> can enter my imap groups with no problem, but gnus cannot connect to my
> leafnode server. However, if I enter in my passphrase, everything is
> fine.

pinentry-gtk-2 pops up randomly?

(setq debug-on-quit t) and then `C-g' when it asks you, and post the
resulting backtrace.  That should tell us what's really asking for the
password.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> pinentry-gtk-2 pops up randomly?
>
> (setq debug-on-quit t) and then `C-g' when it asks you, and post the
> resulting backtrace.  That should tell us what's really asking for the
> password.

Sorry, I can't do that. The calling program does not respond to key
presses when pinentry-gtk-2 is up. There is no way for me to do C-g.

Charles

-- 
"Besides, I think [Slackware] sounds better than 'Microsoft,' don't you?"
(By Patrick Volkerding)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Lars Magne Ingebrigtsen]

Charles Philip Chan <cpchan@sympatico.ca> writes:

>> pinentry-gtk-2 pops up randomly?
>>
>> (setq debug-on-quit t) and then `C-g' when it asks you, and post the
>> resulting backtrace.  That should tell us what's really asking for the
>> password.
>
> Sorry, I can't do that. The calling program does not respond to key
> presses when pinentry-gtk-2 is up. There is no way for me to do C-g.

I have no idea what pinentry-gtk-2 is.  Could you disable it, or
whatever it is that uses it?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: New imap Implementation and Keepalive

[Dave Goldberg]

> Charles Philip Chan <cpchan@sympatico.ca> writes:
>>> pinentry-gtk-2 pops up randomly?
>>> 
>>> (setq debug-on-quit t) and then `C-g' when it asks you, and post the
>>> resulting backtrace.  That should tell us what's really asking for the
>>> password.
>> 
>> Sorry, I can't do that. The calling program does not respond to key
>> presses when pinentry-gtk-2 is up. There is no way for me to do C-g.

> I have no idea what pinentry-gtk-2 is.  Could you disable it, or
> whatever it is that uses it?

It sounds to me like Charles only has gpg2 installed but interaction
with Emacs really requires gpg1.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> It sounds to me like Charles only has gpg2 installed but interaction
> with Emacs really requires gpg1.

Yes, this is true. My distro (OpenSUSE 11.3) doesn't even offer gpg1. I
wasn't aware that people are still using gpg1. Why does Emacs not work
with gpg2?

Charles

-- 
"Open Standards, Open Documents, and Open Source"

  -- Scott Bradner (Open Sources, 1999 O'Reilly and Associates)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[David Engster]

Charles Philip Chan writes:
> Dave Goldberg <david.goldberg6@verizon.net> writes:
>
>> It sounds to me like Charles only has gpg2 installed but interaction
>> with Emacs really requires gpg1.
>
> Yes, this is true. My distro (OpenSUSE 11.3) doesn't even offer gpg1. I
> wasn't aware that people are still using gpg1. Why does Emacs not work
> with gpg2?

gpg2's gpg-agent always uses the program you specified through the
option 'pinentry-program' to securely query you for passphrases. There
are several you can use, including pinentry-gtk2, pinentry-qt4, or
pinentry-ncurses. This is actually to relief the applications from the
task of asking for a passphrase, because many do this the wrong way,
possibly leaving the passphrase in RAM (and hence possibly swap).

My wild guess would be that you are using an encrypted authinfo, and
Emacs has to read that file to see if there are possible credentials
stored there it might need to connect to the server. gpg-agent is
caching the passphrases you enter for a certain time, which you can
specify through options 'default-cache-ttl' and 'max-cache-ttl'.

-David

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I have no idea what pinentry-gtk-2 is.  Could you disable it, or
> whatever it is that uses it?

,----
| pinentry is a small collection of dialog programs that allow GnuPG to
| read passphrases and PIN numbers in a secure manner. There are versions
| for the common GTK and Qt toolkits as well as for the text terminal
| (Curses).
`----

The pinentry suite is part of gpg and cannot be turned off in my gpg
version.

Charles

-- 
"...Unix, MS-DOS, and Windows NT (also known as the Good, the Bad, and
the Ugly)."
(By Matt Welsh)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[CHENG Gao]

*On Sat, 25 Sep 2010 10:07:17 -0400
* Also sprach Charles Philip Chan <cpchan@sympatico.ca>:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> I'm not sure I understand.  After the latest nnimap change, you're
>> prompted for passwords for your nntp server, which requires no
>> password?
>
> No, what I mean is after the latest change the server is kept alive, but
> pinentry-gtk-2 is still popping up. If I click "cancel" on the dialog, I
> can enter my imap groups with no problem, but gnus cannot connect to my
> leafnode server. However, if I enter in my passphrase, everything is
> fine.
>
> Thanks.
> Charles 

I use also gpg2+gpg-agent+pinentry though under MacOSX. And I use
leafnode like you. I dont have any problem so I guess problem may lie in
your settings.

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

CHENG Gao <chenggao@cyberhut.org> writes:

> I use also gpg2+gpg-agent+pinentry though under MacOSX. And I use
> leafnode like you. I dont have any problem so I guess problem may lie
> in your settings.

So the pinentry dialog doesn't pop up for you after the passphrase
cache expires? 

Charles

-- 
There are no threads in a.b.p.erotica,  so there's no  gain in using a
threaded news reader.
(Unknown source)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]


Charles Philip Chan <cpchan@sympatico.ca> writes:

> CHENG Gao <chenggao@cyberhut.org> writes:
>
>> I use also gpg2+gpg-agent+pinentry though under MacOSX. And I use
>> leafnode like you. I dont have any problem so I guess problem may lie
>> in your settings.
>
> So the pinentry dialog doesn't pop up for you after the passphrase
> cache expires? 

I use gpg with pinentry-gtk2.  When trying to sign or decrypt, if the
passphrase is not cached, the dialog pops up, and then when it's done,
the operation proceeds.  Then future operations succeed until the cache
expires.

It's true that when the window pops up interrupting emacs doesn't make
the window go away.  What happens is that emacs ran gpg which sent a
query to the gpg-agent process, which popped up the window.  So you
could argue that gpg should withdraw the request when killed, and that
this is a gpg bug.  But I don't think there's anything wrong with gnus.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

Greg Troxel <gdt@work.lexort.com> writes:

> It's true that when the window pops up interrupting emacs doesn't make
> the window go away.  What happens is that emacs ran gpg which sent a
> query to the gpg-agent process, which popped up the window.  So you
> could argue that gpg should withdraw the request when killed, and that
> this is a gpg bug.

It is not suppose to run gpg again because gnus is still connected to
the imap server. This is what imap keepalive is for and why I requested
it.

> But I don't think there's anything wrong with gnus.

The dialog does not pop up again with any other program that supports
keepalive. Like I said, if you kill the dialog, you will find out that
indeed one is still logged in to the imap server, but then gnus will not
connect to the leafnode server which does not require a user name and
password. Like I said, everything is fine if I enter in my passphrase
again.

Charles

-- 
"sic transit discus mundi"
(From the System Administrator's Guide, by Lars Wirzenius)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Uday S Reddy]

On 9/24/2010 1:34 PM, Frank Schmitt wrote:

> No. For IMAP it is perfectly normal that the client keeps the connection
> open as long as he is active. It's even in the RFC.

Yes, I see that.  I think this part of the spec was most likely 
counter-productive.  There seems to have been an arms race with clients trying 
to keep the connections alive and the servers or server managers trying to 
close them before the clients get around to doing that.  In the end, several 
ISP's now close IMAP connections within minutes.  Even timeouts as short as 30 
seconds have been reported.

Perhaps this game has now been lost irretrievably.

Note that if the client keeps the session alive, then it should take 
responsibility to eventually close the connection.

Cheers,
Uday

Re: New imap Implementation and Keepalive

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> I think it is a bit unkind to keep IMAP sessions alive just for the
> sake of it. The mail server administrators set their time-out policies
> to make the best possible use of resources.

This should be implemented as an option like other email programs such
as mutt, Pine, Thunderbird, etc. My imap server is actually local on my
own machine.

> If the problem is really that the password cache is expiring, it is
> far better to block that instead of trying to tax the mail server.

Security?

Charles

-- 
"Never make any mistaeks."
(Anonymous, in a mail discussion about to a kernel bug report.)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: New imap Implementation and Keepalive

[Uday S Reddy]

On 9/24/2010 2:00 PM, Charles Philip Chan wrote:

>
>> If the problem is really that the password cache is expiring, it is
>> far better to block that instead of trying to tax the mail server.
>
> Security?

My memory is that auth-source.el caches passwords securely.

If your password cache is vulnerable, then your open IMAP session is equally 
vulnerable.  So, I don't see any any particular security advantage to expiring 
the password cache while keeping the sessions alive.

Cheers,
Uday

Re: New imap Implementation and Keepalive

[Rupert Swarbrick]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> On 9/24/2010 2:00 PM, Charles Philip Chan wrote:
>
>>
>>> If the problem is really that the password cache is expiring, it is
>>> far better to block that instead of trying to tax the mail server.
>>
>> Security?
>
> My memory is that auth-source.el caches passwords securely.
>
> If your password cache is vulnerable, then your open IMAP session is
> equally vulnerable.  So, I don't see any any particular security
> advantage to expiring the password cache while keeping the sessions
> alive.
>
> Cheers,
> Uday

Could the reason be that many people use the same password for multiple
services?

Rupert

[-- Attachment #2: Type: application/pgp-signature, Size: 315 bytes --]

Re: New imap Implementation and Keepalive

[Ted Zlatanov]

On Sat, 25 Sep 2010 13:59:03 +0100 Uday S Reddy <u.s.reddy@cs.bham.ac.uk> wrote: 

USR> My memory is that auth-source.el caches passwords securely.

Sorry, it doesn't.  ELisp is just not able to do that to any reasonable
degree of security.  Unless I misunderstand what you mean by "securely."

Ted

Re: New imap Implementation and Keepalive

[Uday S Reddy]

On 9/25/2010 4:21 PM, Ted Zlatanov wrote:
> On Sat, 25 Sep 2010 13:59:03 +0100 Uday S Reddy<u.s.reddy@cs.bham.ac.uk>  wrote:
>
> USR>  My memory is that auth-source.el caches passwords securely.
>
> Sorry, it doesn't.

You are right.  It turns out that I have been using EasyPG's passphrase 
caching, which is probably not all that secure but it serves my needs fine.

Cheers,
Uday

Re: New imap Implementation and Keepalive

[Austin F. Frank]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

On Sat, Sep 25 2010, Uday S Reddy wrote:

> On 9/24/2010 2:00 PM, Charles Philip Chan wrote:
>>
>>> If the problem is really that the password cache is expiring, it is
>>> far better to block that instead of trying to tax the mail server.
>>
>> Security?
>
> My memory is that auth-source.el caches passwords securely.

What does your auth-source-cache contain?  Mine has plaintext
passwords.  Not sure if this is a problem or not, but from my naive
perspective it does seem like secure caching.

/au

-- 
Austin Frank
http://aufrank.net
GPG Public Key (D7398C2F): http://aufrank.net/personal.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]