QRESYNC, nnimap and vanished

QRESYNC, nnimap and vanished

[Julien Danjou]

Hi,

I imagine this question is more directed to Lars, but maybe others Gnus
hackers can enlighten me. I noticed that when a message is reported as
VANISHED by QRESYNC on a nnimap server, Gnus does not update the number
of total message in the group.
Actually, it seems the current code mark the vanished messages as
`read', which is not really what's happening. I've tried to debug this,
but I'm still trying to understand all the data structure of Gnus nn*
things right now.

So my questions are
1. is that fixable?
2. What's the info structure means. I understand it's 
 (groupname (method+groupname a-number (ranges) ?? method (somestuff)))
in somestuff I recognize imap keywords like modseq, uidvalidity. What's
the `active' range?

If there's some documentation, please point me at it. Otherwise, if
someone can explain it to me I'd be glad to enhance the code source to
add such a documentation. :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: QRESYNC, nnimap and vanished

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]


Maybe related, maybe not, I found that the "implement QRESYNC" commit
causes dovecot 1.2.14 to exit abruptly - I sent the details in
<smur5fddegh.fsf@linuxpal.mit.edu>.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Actually, it seems the current code mark the vanished messages as
> `read', which is not really what's happening. I've tried to debug this,
> but I'm still trying to understand all the data structure of Gnus nn*
> things right now.

Do you mean that Gnus shouldn't mark vanished messages as read, or that
it doesn't successfully mark them as read?

> So my questions are
> 1. is that fixable?
> 2. What's the info structure means. I understand it's 
>  (groupname (method+groupname a-number (ranges) ?? method (somestuff)))
> in somestuff I recognize imap keywords like modseq, uidvalidity. What's
> the `active' range?

It's just a stashed version of the previous active range, so that it
knows how to do partial flag updates, when needed.  (Which it isn't for
the QRESYNC case.)

> If there's some documentation, please point me at it. Otherwise, if
> someone can explain it to me I'd be glad to enhance the code source to
> add such a documentation. :)

The `gnus-info-*' accessors are self-documenting.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Julien Danjou]

On Thu, Oct 28 2010, Lars Magne Ingebrigtsen wrote:

> Do you mean that Gnus shouldn't mark vanished messages as read, or that
> it doesn't successfully mark them as read?

I mean that it should not mark them as read. It does that successfully
for now.

I rather think it should be removed from the active range (if I
understand what the active range is correctly).

> The `gnus-info-*' accessors are self-documenting.  :-)

Ok, I spotted them then. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I mean that it should not mark them as read. It does that successfully
> for now.
>
> I rather think it should be removed from the active range (if I
> understand what the active range is correctly).

The active, er, range isn't a range.  It's just a min-max pair.  So
there's nothing to remove from.

The correct thing (that is, the only thing) to do is to mark them as
read.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> The correct thing (that is, the only thing) to do is to mark
LMI> [VANISHED mails] as read.

Read or canceled?  When articles are moved to another group they
show up with the G flag in the old group's summary; nntp articles
which were deleted on the server after the summary buffer was made
and before that article was selected aquire the G flag.  It seems
that imap VANISHED articles should do so, too.  Yes?

Or perhaps as both?

I don't have a QRESYNC server (yet), but my presumption from reading the
rfc is that VANISHED is essentially the same as expired articles in nntp.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> Read or canceled?  When articles are moved to another group they
> show up with the G flag in the old group's summary; nntp articles
> which were deleted on the server after the summary buffer was made
> and before that article was selected aquire the G flag.

Well, not really.  The summary buffer just displays stuff with "G" if
Gnus thought that the message existed upon group entry, but then it
turned out that it didn't exist anyway.  It's not a persistent mark --
the next time you enter the group the message is just marked as read.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Well, not really.  The summary buffer just displays stuff with "G" if
> Gnus thought that the message existed upon group entry, but then it
> turned out that it didn't exist anyway.  It's not a persistent mark --
> the next time you enter the group the message is just marked as read.

Right!

Interpretation, not mark setting!

Re: QRESYNC, nnimap and vanished

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> The correct thing (that is, the only thing) to do is to mark them as
> read.

Um... if they actually are vanished from the IMAP mailbox, then how can
you mark them as anything?  Or rather: where do you set the marks?

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Um... if they actually are vanished from the IMAP mailbox, then how can
> you mark them as anything?  Or rather: where do you set the marks?

Anything that's not unread is read.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Um... if they actually are vanished from the IMAP mailbox, then how can
>> you mark them as anything?  Or rather: where do you set the marks?

> Anything that's not unread is read.

You're saying that you're not trying to set marks on non-existing
articles, but it's how you interpret the responses that determine what
they are shown as?

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> You're saying that you're not trying to set marks on non-existing
> articles, but it's how you interpret the responses that determine what
> they are shown as?

Uhm.  Yes.  I think.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Julien Danjou]

On Sat, Oct 30 2010, Lars Magne Ingebrigtsen wrote:

>> You're saying that you're not trying to set marks on non-existing
>> articles, but it's how you interpret the responses that determine what
>> they are shown as?
>
> Uhm.  Yes.  I think.  :-)

But then, is there any chance/way to update the number of total messages
shown for the group in the group buffer?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> But then, is there any chance/way to update the number of total messages
> shown for the group in the group buffer?

Uhm...  I don't u...  oh!  You want something that says how many
articles there are in total in the group?

Gnus doesn't keep track of that at all.

This is a number that could actually be kept track of.  Most of the
backends (the major exception being nntp) has this info available, but
there's no mechanism for transporting this info over to the Gnus side.
We could add a new "mark" or group parameter -- like `exists' or
something -- and make that available for the group formatting
functions.

nnimap would probably be a good place to implement this as a test case,
but it should be even easier to output from backends like nnml, where
the data is also readily available.  I'm not sure what the interface for
this should be, though...  oh, wait, I know -- nnml could just implement
*-retrieve-group-data-early (as a noop) and
*-finish-retrieve-group-infos (which would do the actual work).  For
instance. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Dave Goldberg]

> Julien Danjou <julien@danjou.info> writes:
>> But then, is there any chance/way to update the number of total messages
>> shown for the group in the group buffer?

> Uhm...  I don't u...  oh!  You want something that says how many
> articles there are in total in the group?

> Gnus doesn't keep track of that at all.

If I understand what you're talking about it tries.  See %t in gnus-group-line-format (which is documented to be an estimate).  It seems to be accurate for all my nnml groups at home.  I'll have to wait til I'm in the office on Monday to see how it's doing for my nnimap groups.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> If I understand what you're talking about it tries.  See %t in
> gnus-group-line-format (which is documented to be an estimate).

It's just (- MAX MIN) of the article numbers.  It's usually accurate for
... some backends, unless you've cancelled/moved articles.

With some IMAP implementations, it's totally inaccurate.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: QRESYNC, nnimap and vanished

[Julien Danjou]

On Sat, Oct 30 2010, Lars Magne Ingebrigtsen wrote:

> Uhm...  I don't u...  oh!  You want something that says how many
> articles there are in total in the group?

Yeah! :-)

> Gnus doesn't keep track of that at all.

You confirm what I though to have understood.

> We could add a new "mark" or group parameter -- like `exists' or
> something -- and make that available for the group formatting
> functions.

[…]


If you say so. That's still out of my scope unfortunately. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: QRESYNC, nnimap and vanished

[Julien Danjou]

On Sat, Oct 30 2010, Lars Magne Ingebrigtsen wrote:

> nnimap would probably be a good place to implement this as a test case,
> but it should be even easier to output from backends like nnml, where
> the data is also readily available.  I'm not sure what the interface for
> this should be, though...  oh, wait, I know -- nnml could just implement
> *-retrieve-group-data-early (as a noop) and
> *-finish-retrieve-group-infos (which would do the actual work).  For
> instance. 

Any update on this? :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: QRESYNC, nnimap and vanished

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Any update on this? :)

Nope.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen