'g' transfers a lot of data

'g' transfers a lot of data

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

Hi,

Since today, pressing 'g' shows:

      nnimap read 133k

Sometimes it even goes to +1000K. Reading the nnimap buffer shows that
it fetch a lot of flags for a lot of UIDs.

* OK [CLOSED] Previous mailbox closed.
* FLAGS (%Answered %Flagged %Deleted %Seen %Draft Old gnus-expire gnus-save $mine store-dormant gnus-dormant gnus-killed gnus-forward)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 11761 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1247083159] UIDs valid
* OK [UIDNEXT 26253] Predicted next UID
* OK [HIGHESTMODSEQ 393] Highest
13774 OK [READ-ONLY] Select completed.
* 11672 FETCH (UID 26152 FLAGS (%Seen))
* 11673 FETCH (UID 26153 FLAGS (%Seen))
* 11674 FETCH (UID 26154 FLAGS (%Seen))
* 11675 FETCH (UID 26155 FLAGS (%Seen))
* 11676 FETCH (UID 26156 FLAGS (%Seen))
* 11677 FETCH (UID 26157 FLAGS (%Answered %Seen))
* 11678 FETCH (UID 26158 FLAGS (%Seen))
[...]
* 11760 FETCH (UID 26249 FLAGS (%Seen))
* 11761 FETCH (UID 26250 FLAGS (%Seen))
13775 OK Fetch completed.
* OK [CLOSED] Previous mailbox closed.
* FLAGS (%Answered %Flagged %Deleted %Seen %Draft Old gnus-expire gnus-forward)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 11605 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1246989894] UIDs valid
* OK [UIDNEXT 12555] Predicted next UID
* OK [HIGHESTMODSEQ 197] Highest
13776 OK [READ-ONLY] Select completed.
* 11543 FETCH (UID 12461 FLAGS (%Seen))
[...]

etc, etc.

Lars, is that normal?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: 'g' transfers a lot of data

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Since today, pressing 'g' shows:
>
>       nnimap read 133k

Nothing has changed there -- I just added the messaging.  :-)

> Sometimes it even goes to +1000K. Reading the nnimap buffer shows that
> it fetch a lot of flags for a lot of UIDs.

How many nnimap groups do you have?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: 'g' transfers a lot of data

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Tue, Oct 05 2010, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
> Nothing has changed there -- I just added the messaging.  :-)

Ok, that's good to know. I missed that commit, so I was wondering what
was happening! :-)

>> Sometimes it even goes to +1000K. Reading the nnimap buffer shows that
>> it fetch a lot of flags for a lot of UIDs.
>
> How many nnimap groups do you have?

Around 97.

And when the fetch is ~300 K, 'g' is quite fast (~2s) but when it's 1M,
it's more like 5s. But I'm on a fast LAN. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: 'g' transfers a lot of data

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

>>> Sometimes it even goes to +1000K. Reading the nnimap buffer shows that
>>> it fetch a lot of flags for a lot of UIDs.
>>
>> How many nnimap groups do you have?
>
> Around 97.
>
> And when the fetch is ~300 K, 'g' is quite fast (~2s) but when it's 1M,
> it's more like 5s. But I'm on a fast LAN. :-)

Let's see...  yeah, it syncs 100 articles "too much", and 100 lines is
about 3K, så 300K sounds right.  The point of doing 100 articles "too
much" is to get any flag changes done by other MUAs, and the idea is
that it's most likely to happen to quite recent messages.

This is 100 flags on articles we've seen before -- not new articles.

But perhaps 100 is too much?  25 might be more reasonable?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: 'g' transfers a lot of data

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On Tue, Oct 05 2010, Lars Magne Ingebrigtsen wrote:

> Let's see...  yeah, it syncs 100 articles "too much", and 100 lines is
> about 3K, så 300K sounds right.  The point of doing 100 articles "too
> much" is to get any flag changes done by other MUAs, and the idea is
> that it's most likely to happen to quite recent messages.
>
> This is 100 flags on articles we've seen before -- not new articles.
>
> But perhaps 100 is too much?  25 might be more reasonable?

If I understand correctly, that seems a very dangerous behaviour. You
mean that if I put/remove the \Seen flags on a very old mail from
another Gnus instance, the other one will not see it?

Maybe 100 is not enough actually?
How others MUA plays with that?

(Sorry if I misunderstand something :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: 'g' transfers a lot of data

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> If I understand correctly, that seems a very dangerous behaviour. You
> mean that if I put/remove the \Seen flags on a very old mail from
> another Gnus instance, the other one will not see it?

Not until you visit the group,... I think...?

Ie. the group will show up with no unread articles until you've visited
it. 

Re: 'g' transfers a lot of data

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> If I understand correctly, that seems a very dangerous behaviour. You
> mean that if I put/remove the \Seen flags on a very old mail from
> another Gnus instance, the other one will not see it?

Yes.

> Maybe 100 is not enough actually?
> How others MUA plays with that?

The only way to get the flags data (without QRESYNC) is to say "UID
FETCH FLAGS 1:*".  If you have 200K messages in that group, it'll take a
while.

When all IMAP servers get QRESYNC support this will be a moot issue, but
at present you have to trade off accuracy for speed in this area.  I
think other MUAs do opportunistic FLAGS sync in the background.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: 'g' transfers a lot of data

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Thu, Oct 07 2010, Lars Magne Ingebrigtsen wrote:

>> Maybe 100 is not enough actually?
>> How others MUA plays with that?
>
> The only way to get the flags data (without QRESYNC) is to say "UID
> FETCH FLAGS 1:*".  If you have 200K messages in that group, it'll take a
> while.
>
> When all IMAP servers get QRESYNC support this will be a moot issue, but
> at present you have to trade off accuracy for speed in this area.  I
> think other MUAs do opportunistic FLAGS sync in the background.

I took a look a mutt, and from what I understand it does 2 things:
- to know if there's new mail, it uses the STATUS command.

  . STATUS INBOX (UIDNEXT UIDVALIDITY UNSEEN RECENT MESSAGES)
  * STATUS "INBOX" (MESSAGES 11790 RECENT 0 UIDNEXT 26293 UIDVALIDITY 1247083159 UNSEEN 1)

  That seems enough to use for 'g'.

- Store the UIDVALIDITY value permanently for each mailbox, and whenever
  it changes refetch all information for all mails in that mailbox.

That sounds safer than the current behaviour, and probably not
complicated to implement. WDYT Lars? :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]