Returning to ticks on read-only imap servers

Returning to ticks on read-only imap servers

[Michael Welsh Duggan]

I'd like to revisit the issue of ticks on read-only nnimap servers.  On
a particular cyrus server, the select-result is: 

((((t
 ("OK"
  ("READ-WRITE")
  "Completed")
 ("FLAGS"
  ("\\Answered" "\\Flagged" "\\Draft" "\\Deleted" "\\Seen"
 "$Forwarded"))
 ("OK"
  ("PERMANENTFLAGS" "(\\Seen)"))
 ("2659" "EXISTS")
 ("0" "RECENT")
 ("OK"
  ("UIDVALIDITY" "1022333920"))
 ("OK"
  ("UIDNEXT" "15114"))
 ("OK"
  ("NOMODSEQ")
  "Sorry," "modsequences" "have" "not" "been" "enabled" "on" "this"
  "mailbox"))

Setting anything by \Seen will cause the response: 

 NO Permission denied

I need non-seen ticks to be handled by the client on this server.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Returning to ticks on read-only imap servers

[Steinar Bang]

>>>>> Michael Welsh Duggan <md5i@md5i.com>:

> I'd like to revisit the issue of ticks on read-only nnimap servers. 

My personal opinion is that read-only nnimap servers isn't such a hot
idea.  NNTP access to whatever would have been better.

However...

> I need non-seen ticks to be handled by the client on this server.

...a fallback to storing ticks in the .newsrc.eld is probably the
expected behaviour (ie. if things are to work as transparently across
backends as possible)...?

Re: Returning to ticks on read-only imap servers

[Michael Welsh Duggan]

Steinar Bang <sb@dod.no> writes:

>>>>>> Michael Welsh Duggan <md5i@md5i.com>:
>
>> I'd like to revisit the issue of ticks on read-only nnimap servers. 
>
> My personal opinion is that read-only nnimap servers isn't such a hot
> idea.  NNTP access to whatever would have been better.
>
> However...

Yeah.  This used to be a server with read/write access to personal
groups, and read-only access to public groups.  Then for some reason
*cough* government clients *cough* our personal mail stuff got switched
to the-never-to-be-damned-enough Exchange.

>> I need non-seen ticks to be handled by the client on this server.
>
> ...a fallback to storing ticks in the .newsrc.eld is probably the
> expected behaviour (ie. if things are to work as transparently across
> backends as possible)...?

I agree that this is probably the correct fallback.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Returning to ticks on read-only imap servers

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> I'd like to revisit the issue of ticks on read-only nnimap servers.

I think the last time I looked at this, I stopped when I found out that
EXAMINE doesn't return enough data to say whether a group is read-only
or not.  So nnimap has to issue a SELECT (which is slow) and then
possibly store the information somewhere for later.  Perhaps the first
time it sees the group?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Returning to ticks on read-only imap servers

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
>> I'd like to revisit the issue of ticks on read-only nnimap servers.
>
> I think the last time I looked at this, I stopped when I found out that
> EXAMINE doesn't return enough data to say whether a group is read-only
> or not.  So nnimap has to issue a SELECT (which is slow) and then
> possibly store the information somewhere for later.  Perhaps the first
> time it sees the group?

I'm going to brainstorm here, so apologies if this seems a little
rambling.

Hmm...  When does the information about marks need to be known?  In
order to set a mark, you need to first enter the group, which
necessitates a select.  So, when it comes to setting marks, I think that
the information is on hand is available.  (EXAMINE probably doesn't give
you the right information because when EXAMINE is in play, the group is
read-only by definition.)  As for knowing about existing marks
beforehand...  If the group is not read-only, this already works.  If it
is read-only, the marks are stored on the client (in the .newsrc.eld).
Based on the fact that ticks that were set before nnimap was rewritten
still exist and work, it would seem that client-side marks are heeded
despite the fact that they do not exist on the server.  Unless this is
only the case because these marked articles are not in the last 100...

(BTW, that 100 should be in a variable, instead of being a constant
value.  I can see wanting to possibly change that number, possibly even
on a per-group basis.)

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Returning to ticks on read-only imap servers

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> Hmm...  When does the information about marks need to be known?

Immediately.  :-)

When you do a `g', you get lists of marks and stuff from the server.
(nnimap issues a series of EXAMINE + UID FETCH FLAGS commands.)  If it's
a normal read/write mailbox, then the absence of a \Seen, for instance,
means that Gnus should consider the message to be unread.  If it's a
read-only mailbox, then the absence of \Seen doesn't mean anything, and
Gnus shouldn't alter any of the data it has on the article in question.

So Gnus needs to now the status before doing anything, really, which
seems to imply that nnimap has to do a SELECT on any mailboxes it hasn't
seen before.  Hm.  That's actually not very difficult, since it already
knows this stuff.

So it just means that it has to stash this flag somewhere, and the group
info is as good as place as any, I guess.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Returning to ticks on read-only imap servers

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> So Gnus needs to now the status before doing anything, really,
LMI> which seems to imply that nnimap has to do a SELECT on any
LMI> mailboxes it hasn't seen before.  Hm.  That's actually not
LMI> very difficult, since it already knows this stuff.

A SELECT on any group it hasn't seen before is a good idea anyway.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Returning to ticks on read-only imap servers

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> LMI> So Gnus needs to now the status before doing anything, really,
> LMI> which seems to imply that nnimap has to do a SELECT on any
> LMI> mailboxes it hasn't seen before.  Hm.  That's actually not
> LMI> very difficult, since it already knows this stuff.
>
> A SELECT on any group it hasn't seen before is a good idea anyway.

Yeah.  I think I have a clear idea how to proceed here now.  For groups
it knows already, it'll output either EXAMINE+FETCH (or EXAMINE QRESYNC)
for the servers that support that.  For groups it doesn't know, it'll
output a SELECT+FETCH 1:*.  That SELECT will tell nnimap whether it
supports flags or not, and nnimap can then stash that info.

In the same sweep, it'll examine the EXAMINE results for mismatches in
UIDVALIDITY, and do an extra SELECT+FETCH 1:* for those groups.

So for normal `g' work, it'll be no slower than today (and much, much
faster for servers with QRESYNC).  Only the appearance of new groups, or
UIDVALIDITY mismatches, will trigger more chatter between Gnus and the
IMAP server.

I'll work on implementing this this weekend.  Once it's implemented, the
very first time you use Gnus after the push, Gnus will issue a
SELECT+FETCH 1:* for all your groups to get a complete data set again.

And I'll change where nnimap stashes the data.  I was confused, so I
stashed the active data (and stuff) in the info marks, while they should
be in the info parameters section.  This will also fix itself the first
time you run this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Returning to ticks on read-only imap servers

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Fri, Oct 08 2010, Lars Magne Ingebrigtsen wrote:

> Yeah.  I think I have a clear idea how to proceed here now.  For groups
> it knows already, it'll output either EXAMINE+FETCH (or EXAMINE QRESYNC)
> for the servers that support that.  For groups it doesn't know, it'll
> output a SELECT+FETCH 1:*.  That SELECT will tell nnimap whether it
> supports flags or not, and nnimap can then stash that info.
>
> In the same sweep, it'll examine the EXAMINE results for mismatches in
> UIDVALIDITY, and do an extra SELECT+FETCH 1:* for those groups.
>
> So for normal `g' work, it'll be no slower than today (and much, much
> faster for servers with QRESYNC).  Only the appearance of new groups, or
> UIDVALIDITY mismatches, will trigger more chatter between Gnus and the
> IMAP server.
>
> I'll work on implementing this this weekend.  Once it's implemented, the
> very first time you use Gnus after the push, Gnus will issue a
> SELECT+FETCH 1:* for all your groups to get a complete data set again.

That overlaps was I was saying in an earlier mail today, using
UIDVALIDITY to resync everything rather than fetch the 100 last mails.

The only question left is, are you forced to to a FETCH 1:*, isn't a
STATUS enough? That would be very faster. You'd do the FETCH on group
entering.

My 2¢,

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Returning to ticks on read-only imap servers

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> The only question left is, are you forced to to a FETCH 1:*, isn't a
> STATUS enough? That would be very faster. You'd do the FETCH on group
> entering.

No, a STATUS only gives you the high watermark in the group.  If the
high watermark increases by 10000, there may be only a single new
article in the mailbox.  Or there may be 10000 new messages, but 9999 of
them have already been read.

So you need to do a FETCH MARKS to find out what's really happened.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Returning to ticks on read-only imap servers

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Fri, Oct 08 2010, Lars Magne Ingebrigtsen wrote:

> No, a STATUS only gives you the high watermark in the group.  If the
> high watermark increases by 10000, there may be only a single new
> article in the mailbox.  Or there may be 10000 new messages, but 9999 of
> them have already been read.

No, it gives you if the box changed, and how many messages/unread it
has. That looks like enough info to print the group buffer. Or not?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Returning to ticks on read-only imap servers

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> No, it gives you if the box changed, and how many messages/unread it
> has. That looks like enough info to print the group buffer. Or not?

Not.  :-)  You only know the number, not which ones, so you can't really
say what happened.

If you had high 50, unseen 25, and it increase to high 55, unseen 23,
how do you know whether that's 5 new articles, and two old that were
read, or 1 new article, and three that were read?

Just work through the scenarios, and keep in mind that the UIDNEXT isn't
necessarily consecutive. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Returning to ticks on read-only imap servers

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

On Fri, Oct 08 2010, Lars Magne Ingebrigtsen wrote:

> Not.  :-)  You only know the number, not which ones, so you can't really
> say what happened.

But you do not care about which ones in the group buffer, I think.

> If you had high 50, unseen 25, and it increase to high 55, unseen 23,
> how do you know whether that's 5 new articles, and two old that were
> read, or 1 new article, and three that were read?

What's "high"? The last article number (UIDNEXT?) ?

> Just work through the scenarios, and keep in mind that the UIDNEXT isn't
> necessarily consecutive. 

I did not talk about UIDNEXT, but about UIDVALIDITY, which if it
changes, indicate a change in the mailbox.

My point is that in the *Group* buffer we (mostly) need how many
messages and how many unread message there's in the group, and STATUS
give that for free, with an indicator (UIDVALIDITY) indicating *if* we
need to do a FETCH 1:* on group entering. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Returning to ticks on read-only imap servers

[Dan Christensen]

Julien Danjou <julien@danjou.info> writes:

> My point is that in the *Group* buffer we (mostly) need how many
> messages and how many unread message there's in the group, and STATUS
> give that for free, with an indicator (UIDVALIDITY) indicating *if* we
> need to do a FETCH 1:* on group entering. :-)

I'm no expert, but my understanding is that in normal operation,
UIDVALIDITY never changes, even when messages arrive, are deleted, or
flags change.  So it can't help solve the problem at hand.

Dan

Re: Returning to ticks on read-only imap servers

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On Fri, Oct 08 2010, Dan Christensen wrote:

> I'm no expert, but my understanding is that in normal operation,
> UIDVALIDITY never changes, even when messages arrive, are deleted, or
> flags change.  So it can't help solve the problem at hand.

That's totally the opposite. A UIDVALIDITY number design a mailbox in a
state. If any change occurs, the UIDVALIDITY number changes. That's how
you know you need to do a FETCH 1:* to resynchronize your (cached)
version of the mailbox.

Read section 2.3.1.1 of RFC3501.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Returning to ticks on read-only imap servers

[Michael Welsh Duggan]

Julien Danjou <julien@danjou.info> writes:

> On Fri, Oct 08 2010, Dan Christensen wrote:
>
>> I'm no expert, but my understanding is that in normal operation,
>> UIDVALIDITY never changes, even when messages arrive, are deleted, or
>> flags change.  So it can't help solve the problem at hand.
>
> That's totally the opposite. A UIDVALIDITY number design a mailbox in a
> state. If any change occurs, the UIDVALIDITY number changes. That's how
> you know you need to do a FETCH 1:* to resynchronize your (cached)
> version of the mailbox.
>
> Read section 2.3.1.1 of RFC3501.

After re-reading that section, I think you are mistaken.  UIDVALIDITY is
guaranteed not to change as long as the messages in the mailbox have not
changed their UIDs (or rather, as long as the mailbox maintains its
identity).  It does not change with the addition, or deletion, of
messages to that mailbox.  On the other hand, the UIDNEXT does change
with the addition of new messages to the mailbox.  It will not, however,
change with the deletion of messages.  The triple of mailbox name,
UIDVALIDITY, and message UID will always refer to the same message no
matter what, as long as that message still exists.

"First, the next unique identifier value MUST NOT change unless new
 messages are added to the mailbox; and second, the next unique
 identifier value MUST change whenever new messages are added to the
 mailbox, even if those new messages are subsequently expunged."

In the above extract, the "next unique identifier value" refers to
UIDNEXT, not UIDVALIDITY.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Returning to ticks on read-only imap servers

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Sat, Oct 09 2010, Michael Welsh Duggan wrote:

> After re-reading that section, I think you are mistaken.  UIDVALIDITY is
> guaranteed not to change as long as the messages in the mailbox have not
> changed their UIDs (or rather, as long as the mailbox maintains its
> identity).  It does not change with the addition, or deletion, of
> messages to that mailbox.  On the other hand, the UIDNEXT does change
> with the addition of new messages to the mailbox.  It will not, however,
> change with the deletion of messages.  The triple of mailbox name,
> UIDVALIDITY, and message UID will always refer to the same message no
> matter what, as long as that message still exists.

Yeah, re-re-reading I think you're right indeed.. UIDVALIDITY is not the
thing I wanted it to be.

That sucks.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]