*reproducible* lost marks with nnimap

*reproducible* lost marks with nnimap

[David Abrahams]

I figured out how to reproduce the "nnimap lost marks" problem:

  1. visit the *Summary INBOX* buffer
  2. read a few mails so they are marked `expired'
  3. *wait* for the IMAP server to get bored and close the connection.
  4. Hit `M-g' in the summary buffer to get new mail

Note that the mails you previously read will be marked unread again.
    
FWIW, I'm reading my mail agentized, using openssl to communicate with
my IMAP server.

I hope this information helps someone to fix this persistent problem!

Thanks,
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> I figured out how to reproduce the "nnimap lost marks" problem:
>
>   1. visit the *Summary INBOX* buffer
>   2. read a few mails so they are marked `expired'
>   3. *wait* for the IMAP server to get bored and close the connection.
>   4. Hit `M-g' in the summary buffer to get new mail
>
> Note that the mails you previously read will be marked unread again.

Did you type 'q' in the *Summary* buffer?  Remember, flags are not
synchronized with the server until you do, and the 'q' has
successfully exited the group.  You can press 'Q', or simply C-k a
summary buffer, and any flag changes you made (except explicit delete)
will be lost.  So prematurely sending the flag updates is not right.

Re: *reproducible* lost marks with nnimap

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> I figured out how to reproduce the "nnimap lost marks" problem:
>>
>>   1. visit the *Summary INBOX* buffer
>>   2. read a few mails so they are marked `expired'
>>   3. *wait* for the IMAP server to get bored and close the connection.
>>   4. Hit `M-g' in the summary buffer to get new mail
>>
>> Note that the mails you previously read will be marked unread again.
>
> Did you type 'q' in the *Summary* buffer?  

No.

> Remember, flags are not synchronized with the server until you do,
> and the 'q' has successfully exited the group.  

I understand you're saying that's how it works, but:

  1. From a usability standpoint, it's just crazy that messages I've
     read become unread when I ask the summary to update with `M-g'.
     What possible purpose could that serve?

  2. After the server times out and says goodbye, nnimap often fails
     to re-establish the connection until after *two* tries.  The
     result, apparently, is that even pressing `q' sometimes drops
     marks because flag synchronization fails and there's no retry.

> You can press 'Q', or simply C-k a summary buffer, and any flag
> changes you made (except explicit delete) will be lost.  So
> prematurely sending the flag updates is not right.

Then you should consider sending them when `M-g' is done.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> Remember, flags are not synchronized with the server until you do,
>> and the 'q' has successfully exited the group.  
>
> I understand you're saying that's how it works, but:
>
>   1. From a usability standpoint, it's just crazy that messages I've
>      read become unread when I ask the summary to update with `M-g'.
>      What possible purpose could that serve?

You can still press 'q' in the summary buffer to save your flag
changes (including which articles you read).  Pressing M-g shouldn't
change this.  This isn't nnimap specific, so changing this probably
should be discussed by itself first.

>   2. After the server times out and says goodbye, nnimap often fails
>      to re-establish the connection until after *two* tries.  The
>      result, apparently, is that even pressing `q' sometimes drops
>      marks because flag synchronization fails and there's no retry.

This sounds like a nnimap bug, and I see it too.  I think it only
applies to non-native network connections (e.g., via tls, ssh etc).
Fixing this problem might make you not perceive the first problem as a
problem, so it is important.  It hasn't annoyed me enough to debug
yet, if anyone else like to debug it, that would be good.

Re: *reproducible* lost marks with nnimap

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> Remember, flags are not synchronized with the server until you do,
>>> and the 'q' has successfully exited the group.  
>>
>> I understand you're saying that's how it works, but:
>>
>>   1. From a usability standpoint, it's just crazy that messages I've
>>      read become unread when I ask the summary to update with `M-g'.
>>      What possible purpose could that serve?
>
> You can still press 'q' in the summary buffer to save your flag
> changes (including which articles you read).  

But it takes a long time to do that (possibly because I have a great
many messages in my INBOX?), and closes my summary buffer.  That's a
big interruption.

> Pressing M-g shouldn't change this.  This isn't nnimap specific, so
> changing this probably should be discussed by itself first.

Maybe so.  Be my guest, everyone! ;-)

>>   2. After the server times out and says goodbye, nnimap often fails
>>      to re-establish the connection until after *two* tries.  The
>>      result, apparently, is that even pressing `q' sometimes drops
>>      marks because flag synchronization fails and there's no retry.
>
> This sounds like a nnimap bug, and I see it too.  I think it only
> applies to non-native network connections (e.g., via tls, ssh etc).
> Fixing this problem might make you not perceive the first problem as a
> problem, so it is important.  It hasn't annoyed me enough to debug
> yet, if anyone else like to debug it, that would be good.

It has annoyed me enough, for a very long time, but I got lost in the
tangle of Gnus and ran out of time.

It didn't seem to happen when I used Gnus under XEmacs, FWIW.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: *reproducible* lost marks with nnimap

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> Remember, flags are not synchronized with the server until you do,
>>> and the 'q' has successfully exited the group.  
>>
>> I understand you're saying that's how it works, but:
>>
>>   1. From a usability standpoint, it's just crazy that messages I've
>>      read become unread when I ask the summary to update with `M-g'.
>>      What possible purpose could that serve?
>
> You can still press 'q' in the summary buffer to save your flag
> changes (including which articles you read).  

But it takes a long time to do that (possibly because I have a great
many messages in my INBOX?), and closes my summary buffer.  That's a
big interruption.

> Pressing M-g shouldn't change this.  This isn't nnimap specific, so
> changing this probably should be discussed by itself first.

Maybe so.  Be my guest, everyone! ;-)

>>   2. After the server times out and says goodbye, nnimap often fails
>>      to re-establish the connection until after *two* tries.  The
>>      result, apparently, is that even pressing `q' sometimes drops
>>      marks because flag synchronization fails and there's no retry.
>
> This sounds like a nnimap bug, and I see it too.  I think it only
> applies to non-native network connections (e.g., via tls, ssh etc).
> Fixing this problem might make you not perceive the first problem as a
> problem, so it is important.  It hasn't annoyed me enough to debug
> yet, if anyone else like to debug it, that would be good.

It has annoyed me enough, for a very long time, but I got lost in the
tangle of Gnus and ran out of time.

It didn't seem to happen when I used Gnus under XEmacs, FWIW.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: *reproducible* lost marks with nnimap

[Kai Grossjohann]

Simon Josefsson <jas@extundo.com> writes:

> You can still press 'q' in the summary buffer to save your flag
> changes (including which articles you read).  Pressing M-g shouldn't
> change this.  This isn't nnimap specific, so changing this probably
> should be discussed by itself first.

I thought that M-g did an implicit q, so to speak?  So it would be
good, imvho, to have M-g from the summary buffer also save the marks
to the server.

Kai

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

Kai Grossjohann <kai@emptydomain.de> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> You can still press 'q' in the summary buffer to save your flag
>> changes (including which articles you read).  Pressing M-g shouldn't
>> change this.  This isn't nnimap specific, so changing this probably
>> should be discussed by itself first.
>
> I thought that M-g did an implicit q, so to speak?  So it would be
> good, imvho, to have M-g from the summary buffer also save the marks
> to the server.

Ah.  I was talking about M-g in the group buffer, but I now see David
said summary buffer.  Hm.  Yes, then it sounds like a bug.  But I
can't reproduce it.  M-g in a summary buffer set the flags correctly
here.  (M-g in a group buffer does not set flags, as expected.)

Re: *reproducible* lost marks with nnimap

[Kai Grossjohann]

Simon Josefsson <jas@extundo.com> writes:

> Ah.  I was talking about M-g in the group buffer, but I now see David
> said summary buffer.  Hm.  Yes, then it sounds like a bug.  But I
> can't reproduce it.  M-g in a summary buffer set the flags correctly
> here.

Even after the server times out the connection?

Kai

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

Kai Grossjohann <kai@emptydomain.de> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Ah.  I was talking about M-g in the group buffer, but I now see David
>> said summary buffer.  Hm.  Yes, then it sounds like a bug.  But I
>> can't reproduce it.  M-g in a summary buffer set the flags correctly
>> here.
>
> Even after the server times out the connection?

Doesn't M-g abort with an error in that case?  The summary buffer
should still be alive though, so if you fix the connection, pressing q
or M-g in the summary buffer should save the flags.

I suspect that if the bug that nnimap doesn't reopen connections
automatically in some situations (starttls) was fixed, this would take
care of this other problem too.

Re: *reproducible* lost marks with nnimap

[Chris Shenton]

Simon Josefsson <jas@extundo.com> writes:

> I suspect that if the bug that nnimap doesn't reopen connections
> automatically in some situations (starttls) was fixed, this would take
> care of this other problem too.

Aye, there's the rub!  I sent in a little more info on my problems
with this a few days ago, Subject "nnimap:$server unavailable... Go offline?"

I also tried using an always SSLified SMTPS server but found that it
had bugs moving articles from non-nnimap to nnimap groups. Someone
else has experienced similar problems regarding openssl problems:

  http://www.lysator.liu.se/~forsberg/docs/Gnus_and_Courier.xml

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

> I also tried using an always SSLified SMTPS server but found that it
> had bugs moving articles from non-nnimap to nnimap groups. Someone
> else has experienced similar problems regarding openssl problems:
>
>   http://www.lysator.liu.se/~forsberg/docs/Gnus_and_Courier.xml

Adding -quiet to OpenSSL is the default these days, I think.  I couldn't
find any other problem discussed, but I didn't read all of it.

Re: *reproducible* lost marks with nnimap

[Chris Shenton]

"Simon Josefsson" <jas@extundo.com> writes:

> Adding -quiet to OpenSSL is the default these days, I think.  I couldn't
> find any other problem discussed, but I didn't read all of it.

It's in there, by default, so there still seems to be a problem:

imap-ssl-program's value is
("openssl s_client -quiet -ssl3 -connect %s:%p" "openssl s_client -quiet -ssl2 -connect %s:%p" "s_client -quiet -ssl3 -connect %s:%p" "s_client -quiet -ssl2 -connect %s:%p")

Re: *reproducible* lost marks with nnimap

[Simon Josefsson]

Chris Shenton <chris@shenton.org> writes:

> "Simon Josefsson" <jas@extundo.com> writes:
>
>> Adding -quiet to OpenSSL is the default these days, I think.  I couldn't
>> find any other problem discussed, but I didn't read all of it.
>
> It's in there, by default, so there still seems to be a problem:
>
> imap-ssl-program's value is
> ("openssl s_client -quiet -ssl3 -connect %s:%p" "openssl s_client -quiet -ssl2 -connect %s:%p" "s_client -quiet -ssl3 -connect %s:%p" "s_client -quiet -ssl2 -connect %s:%p")

What is the problem?  If I read the web page correctly, whatever
problem mentioned there went away when -quiet was added.

Re: *reproducible* lost marks with nnimap

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> Kai Grossjohann <kai@emptydomain.de> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> Ah.  I was talking about M-g in the group buffer, but I now see David
>>> said summary buffer.  Hm.  Yes, then it sounds like a bug.  But I
>>> can't reproduce it.  M-g in a summary buffer set the flags correctly
>>> here.
>>
>> Even after the server times out the connection?
>
> Doesn't M-g abort with an error in that case?  

Why not just have it try twice when it fails?

> The summary buffer should still be alive though, so if you fix the
> connection, pressing q or M-g in the summary buffer should save the
> flags.

Doesn't seem to work reliably.

> I suspect that if the bug that nnimap doesn't reopen connections
> automatically in some situations (starttls) was fixed, this would take
> care of this other problem too.

Sounds like a good idea.  Who's qualified to fix that bug? ;-)

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: *reproducible* lost marks with nnimap

[Kevin Greiner]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Kai Grossjohann <kai@emptydomain.de> writes:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>>> Ah.  I was talking about M-g in the group buffer, but I now see David
>>>> said summary buffer.  Hm.  Yes, then it sounds like a bug.  But I
>>>> can't reproduce it.  M-g in a summary buffer set the flags correctly
>>>> here.
>>>
>>> Even after the server times out the connection?
>>
>> Doesn't M-g abort with an error in that case?  
>
> Why not just have it try twice when it fails?
>
>> The summary buffer should still be alive though, so if you fix the
>> connection, pressing q or M-g in the summary buffer should save the
>> flags.
>
> Doesn't seem to work reliably.
>
>> I suspect that if the bug that nnimap doesn't reopen connections
>> automatically in some situations (starttls) was fixed, this would take
>> care of this other problem too.
>
> Sounds like a good idea.  Who's qualified to fix that bug? ;-)

Whoever works on this might want to first look at nntp-with-open-group.  It
was written to solve similar problems with nntp servers.

Kevin