updating backend marks immediately

updating backend marks immediately

[Dan Christensen]

I'd like to update the marks on my imap server whenever they change in
an nnimap summary buffer.  This seems to do the trick:

  (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)

It updates the imap server, but doesn't save the .newsrc or the dribble
file, so it's quick.

I'm posting this in case it is useful to others, but also to check that
it is safe.  Could Gnus get confused by this?  Would it be better to call

  (gnus-summary-save-newsrc)

which also saves the information to the dribble buffer and saves the
dribble file? 

Dan

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I'm posting this in case it is useful to others, but also to check that
> it is safe.  Could Gnus get confused by this? 

It should be fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Ted Zlatanov]

On Thu, 04 Nov 2010 16:32:45 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> I'd like to update the marks on my imap server whenever they change in
DC> an nnimap summary buffer.  This seems to do the trick:

DC>   (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)

DC> It updates the imap server, but doesn't save the .newsrc or the dribble
DC> file, so it's quick.

DC> I'm posting this in case it is useful to others, but also to check that
DC> it is safe.  Could Gnus get confused by this?  Would it be better to call

DC>   (gnus-summary-save-newsrc)

DC> which also saves the information to the dribble buffer and saves the
DC> dribble file? 

I would find it surprising if `Q' didn't work.  I think your change
would break that.

Ted

Re: updating backend marks immediately

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 04 Nov 2010 16:32:45 -0400 Dan Christensen <jdc@uwo.ca> wrote: 
>
> DC> I'd like to update the marks on my imap server whenever they change in
> DC> an nnimap summary buffer.  This seems to do the trick:
>
> DC>   (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)
>
> DC> It updates the imap server, but doesn't save the .newsrc or the dribble
> DC> file, so it's quick.
>
> I would find it surprising if `Q' didn't work.  I think your change
> would break that.

I think you're right, but for me, it's a worthwhile trade-off.  I
essentially never use Q, but I like my imap server to be kept
up-to-date.  For example, without the above, I might walk away from my
laptop while my INBOX is open, and when I get to work, none of the
triage I've done is visible.

(Except that moves and deletes are propagated immediately.  They can't
be undone with Q either, so in a sense my adjustment above makes things
more consistent.)

In any case, I wasn't proposing that Gnus be changed to behave this way
by default.  Maybe it could be enabled with a server or group parameter.
Or maybe anyone who wants to do it can google and find my solution
above.

Dan

Re: updating backend marks immediately

[Ted Zlatanov]

On Tue, 09 Nov 2010 20:13:58 -0500 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Thu, 04 Nov 2010 16:32:45 -0400 Dan Christensen <jdc@uwo.ca> wrote: 
>> 
DC> I'd like to update the marks on my imap server whenever they change in
DC> an nnimap summary buffer.  This seems to do the trick:
>> 
DC> (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)
>> 
DC> It updates the imap server, but doesn't save the .newsrc or the dribble
DC> file, so it's quick.
>> 
>> I would find it surprising if `Q' didn't work.  I think your change
>> would break that.

DC> I think you're right, but for me, it's a worthwhile trade-off.  I
DC> essentially never use Q, but I like my imap server to be kept
DC> up-to-date.  For example, without the above, I might walk away from my
DC> laptop while my INBOX is open, and when I get to work, none of the
DC> triage I've done is visible.

DC> (Except that moves and deletes are propagated immediately.  They can't
DC> be undone with Q either, so in a sense my adjustment above makes things
DC> more consistent.)

DC> In any case, I wasn't proposing that Gnus be changed to behave this way
DC> by default.  Maybe it could be enabled with a server or group parameter.
DC> Or maybe anyone who wants to do it can google and find my solution
DC> above.

I think it's worth a global variable.  gnus-update-marks-immediately or
something.  I can't imagine why someone would want it just for a few
servers or groups.  But it should be documented with lots of warnings.

Anyone else with a comment?

Ted

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DC> (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)

[...]

> I think it's worth a global variable.  gnus-update-marks-immediately or
> something.

I think the hook thing is a perfectly good solution here.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I can't imagine why someone would want it just for a few
> servers or groups. 

Well, I don't see much point with local mail or nntp, since what I
did only propagates the mark changes to the remote server, but doesn't
update the .newsrc.  What does gnus-summary-update-info do for backends
besides nnimap?

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> DC> (add-hook 'gnus-summary-update-hook 'gnus-summary-update-info)
>
>> I think it's worth a global variable.  gnus-update-marks-immediately or
>> something.
>
> I think the hook thing is a perfectly good solution here.

I don't mind either way.  But I should say that I've been noticing some
strange behavior with the above hook installed.  

First, I think there's a bug that was introduced when nnimap moves
within the same server were sped up a few weeks ago: if I have changed
the marks on an article, but not saved the summary, then after the move,
the article has its old marks.  This used to happen a long time ago,
then it was fixed a few months ago, and then it was broken again
recently.  (The above hook mostly hides this problem, but such moves
should be made to work for people not using the hook.)

The other bug/problem is harder to reproduce, and has occurred only
since adding the above hook.  Sometimes in an auto-expire group, I 
mark an article as unread (`u') or as read (`d') to get rid of the
expirable mark.  If I move the article, it sometimes retains the
expirable mark, even though marks should be propagated to the server.
Or, if I do `M-g' to regenerate the summary, sometimes the article
retains the expirable mark.  But I can't consistently reproduce this.

Finally, there's another problem, but I don't know if it's related to
the above hook.  I'm sometimes finding that gnus loses all expirable
marks on all articles, in both nnimap and nnfolder groups.  Again, I
can't figure out when this happens.  It might be related to using
`M-g' or `/ N' or such things.  Sorry I can't pin this down further.

I'm using git gnus from Nov 6:  commit 9f19eacda...

Dan

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> First, I think there's a bug that was introduced when nnimap moves
> within the same server were sped up a few weeks ago: if I have changed
> the marks on an article, but not saved the summary, then after the move,
> the article has its old marks.

Hm.  I don't seem to be able to reproduce this...

> The other bug/problem is harder to reproduce, and has occurred only
> since adding the above hook.  Sometimes in an auto-expire group, I 
> mark an article as unread (`u') or as read (`d') to get rid of the
> expirable mark.  If I move the article, it sometimes retains the
> expirable mark, even though marks should be propagated to the server.

Have a look in the *imap log* buffer and see what commands are actually
sent to the server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> First, I think there's a bug that was introduced when nnimap moves
>> within the same server were sped up a few weeks ago: if I have changed
>> the marks on an article, but not saved the summary, then after the move,
>> the article has its old marks.
>
> Hm.  I don't seem to be able to reproduce this...

I just tried it in one group, and it didn't happen.  But when I
tried it in another group, I had a problem.  Here are the steps
I did:

Enter group "generic", which is auto-expired.  Read message 2816,
which was already marked as expirable.  Decide I want to keep it,
so mark it unread with `u' and then move it to INBOX.  Exit
generic, enter INBOX, and the article appears with an expirable
mark.  (INBOX is not auto-expirable.)

The log is below.  You'll see that the unmark operation is never sent to
the server.  (I disabled my hook that does this whenever the marks change.)

16:36:13 10269 SELECT "generic"
16:36:13 10270 UID FETCH 2811:2819 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
16:36:20 10271 UID FETCH 2816 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
16:36:20 10272 UID COPY 2816 "INBOX"
16:36:20 10273 UID STORE 2816 +FLAGS.SILENT (\Deleted)
16:36:20 10274 UID EXPUNGE 2816
16:36:20 10275 EXAMINE "INBOX"
16:36:20 10276 UID SEARCH HEADER Message-Id "<7B6B9B16F3304C8F9C0E26E33E9B0074@terry>"
16:36:20 10277 SELECT "INBOX"
16:36:20 10278 SELECT "INBOX"
16:36:20 10279 UID FETCH 1:* FLAGS
16:36:21 10280 SELECT "generic"
16:36:21 10281 UID STORE 2816 -FLAGS.SILENT (gnus-expire)
16:36:21 10282 EXAMINE "INBOX"
16:36:21 10283 UID FETCH 6545:* FLAGS
16:36:21 10284 EXAMINE "Sent"
16:36:21 10285 UID FETCH 5038:* FLAGS
16:36:22 10286 SELECT "INBOX"
16:36:22 10287 UID FETCH 6637:6645 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])

>> The other bug/problem is harder to reproduce, and has occurred only
>> since adding the above hook.  Sometimes in an auto-expire group, I 
>> mark an article as unread (`u') or as read (`d') to get rid of the
>> expirable mark.  If I move the article, it sometimes retains the
>> expirable mark, even though marks should be propagated to the server.
>
> Have a look in the *imap log* buffer and see what commands are actually
> sent to the server.

The problem that I don't usually notice the problem until I enter the
group later.  But if it happens again, I'll try to get a useful log.
It'll be easier once the above problem is fixed.

(Still using git from Nov 6, but I'm happy to upgrade if you think this
was fixed in the last week.)

Dan

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Enter group "generic", which is auto-expired.  Read message 2816,
> which was already marked as expirable.  Decide I want to keep it,
> so mark it unread with `u' and then move it to INBOX.  Exit
> generic, enter INBOX, and the article appears with an expirable
> mark.  (INBOX is not auto-expirable.)

The auto-expirable etc stuff was too complicated to reproduce, but if I
enter a group, tick an article, and then `B c' it so another group, then
I get this:

06:23:30 199 SELECT "4ad".
06:23:30 200 UID FETCH 34 BODY.PEEK[].
06:23:30 201 EXAMINE DOES.NOT.EXIST.
06:23:30 202 APPEND "ebay" {13799}.
06:23:30 203 EXAMINE "ebay".
06:23:30 204 UID SEARCH HEADER Message-Id "<i829u4+gqf7@eGroups.com>".
06:23:30 205 SELECT "ebay".
06:23:30 206 UID STORE 5 +FLAGS.SILENT (\Seen \Flagged).
06:23:30 207 SELECT "ebay".
06:23:30 208 UID FETCH 1:* FLAGS.

So the \Flagged mark is copied over to the article in the new group,
even though the article in the original group doesn't have it set yet on
the IMAP server.

> The log is below.  You'll see that the unmark operation is never sent to
> the server.  (I disabled my hook that does this whenever the marks change.)

Hm...  Oh!  I see.  D'oh.

The COPY command, of course, copies over the flags to the new group.
But Gnus doesn't know anything about that, because it assumes that it
controls all the flags.  But in this case, it kinda really needs to
clear all the marks, and then re-set the ones it thinks should be
there.

Does IMAP have a clear-all-flags-on-this-article command?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Does IMAP have a clear-all-flags-on-this-article command?

Yes.  I've now made it clear the marks before moving.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Does IMAP have a clear-all-flags-on-this-article command?
>
> Yes.  I've now made it clear the marks before moving.

This seems to have fixed the problem.  Thanks!

Dan

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

>>> Does IMAP have a clear-all-flags-on-this-article command?
>>
>> Yes.  I've now made it clear the marks before moving.
>
> This seems to have fixed the problem.  Thanks!

It occurs to me that I should probably have do it the other way around.
That is, set the marks on the article before moving it, and then not
bother to copy over the marks afterwards.  The result should be much the
same, but slightly more secure if something fails in the middle.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>>> Yes.  I've now made it clear the marks before moving.
>
> It occurs to me that I should probably have do it the other way around.
> That is, set the marks on the article before moving it, and then not
> bother to copy over the marks afterwards.  The result should be much the
> same, but slightly more secure if something fails in the middle.

I think what you propose might be better.  What I see now is that
when I move a message from my INBOX to another folder (a common
operation), my mail-notification daemon pops up a notification saying
that there's a new message in my INBOX.  That's because Gnus briefly
removes the marks.  Instead, Gnus should just upload any pending changes
to the marks (and only if there are any changes to save), and then move
the article.

This should also be faster, since now there are several round-trips, 
even if the marks don't need to be updated:  1) clear the marks,
2) do the move, 3) fix the marks in the target group (and some other
actions that I'm omitting, like looking up the article number).
In many cases, 1 and 3 can be skipped, and when the marks do need
to be adjusted, it should still be doable in two steps, not three.

Dan

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

>> It occurs to me that I should probably have do it the other way around.
>> That is, set the marks on the article before moving it, and then not
>> bother to copy over the marks afterwards.  The result should be much the
>> same, but slightly more secure if something fails in the middle.
>
> I think what you propose might be better.

Yeah, I'm attempting to do that now, but it isn't totally trivial.
There's apparently no single function to just sync all the (current)
flags from a single article to the backend?  So I think I'll have to
write that first.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yeah, I'm attempting to do that now, but it isn't totally trivial.
> There's apparently no single function to just sync all the (current)
> flags from a single article to the backend?  So I think I'll have to
> write that first.

Done.  It was kinda trivial, really.  So it now ensures that all marks
are set before issuing the COPY command.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Done.  It was kinda trivial, really.  So it now ensures that all marks
> are set before issuing the COPY command.

Wasn't the marks set on the article in the first place?

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Wasn't the marks set on the article in the first place?

Nope.  The marks are updated on group exit (normally), so if you remove
a tick mark from an article and then immediately move it, the article in
the other group would regain the tick mark.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Nope.  The marks are updated on group exit (normally), so if you
> remove a tick mark from an article and then immediately move it, the
> article in the other group would regain the tick mark.

I see.  Good to know.

Re: updating backend marks immediately

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Yeah, I'm attempting to do that now, but it isn't totally trivial.
>> There's apparently no single function to just sync all the (current)
>> flags from a single article to the backend?  So I think I'll have to
>> write that first.
>
> Done.  It was kinda trivial, really.  So it now ensures that all marks
> are set before issuing the COPY command.

Thanks!  It looks like an extra STORE command is being done, though.
Below is an example where I didn't change the marks on an article before
moving it (from "test" to the same group), and yet the first line shows
a bunch of flags being removed.  And I don't know why \Seen would be
removed, since the article was marked as read.  Then in command 308, the
\Seen flag is restored, which seems inefficient.

And are commands 309 and 310 needed?

Dan

10:14:38 302 UID STORE 10719 -FLAGS.SILENT (\Seen \Flagged \Answered
gnus-expire gnus-dormant gnus-save gnus-download gnus-forward)

10:14:38 303 UID FETCH 10719 (UID RFC822.SIZE BODYSTRUCTURE
BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References
In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])

10:14:38 304 UID COPY 10719 "test"

10:14:38 305 UID STORE 10719 +FLAGS.SILENT (\Deleted)

10:14:38 306 UID EXPUNGE 10719

10:14:38 307 UID SEARCH HEADER Message-Id
"<2A24BF61-650C-468E-B14C-D9351659FACD@birs.ca>"

10:14:38 308 UID STORE 10728 +FLAGS.SILENT (\Seen)

10:14:38 309 SELECT "test"

10:14:38 310 UID FETCH 1:* FLAGS

Re: updating backend marks immediately

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> And are commands 309 and 310 needed?

[...]

> 10:14:38 309 SELECT "test"
> 10:14:38 310 UID FETCH 1:* FLAGS

It's this code that's being run:

    ;; Re-activate all groups that have been moved to.
    (with-current-buffer gnus-group-buffer
      (let ((gnus-group-marked to-groups))
	(gnus-group-get-new-news-this-group nil t)))

I think the idea here is that if you move something to a group, that
group's group line should be updated immediately, so you don't have to
wait until the next `g' to find the article.  And
`gnus-group-get-new-news-this-group' does the equivalent of `M-g' on the
group, so it's slow.

I'd be fine with just removing that code snippet altogether.  Does `B m'
have to update the resulting groups immediately?  
        
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: updating backend marks immediately

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> And are commands 309 and 310 needed?
>>
>> 10:14:38 309 SELECT "test"
>> 10:14:38 310 UID FETCH 1:* FLAGS
>
> I think the idea here is that if you move something to a group, that
> group's group line should be updated immediately, so you don't have to
> wait until the next `g' to find the article.  And
> `gnus-group-get-new-news-this-group' does the equivalent of `M-g' on the
> group, so it's slow.
>
> I'd be fine with just removing that code snippet altogether.  Does `B m'
> have to update the resulting groups immediately?  

I don't mind keeping it, as my imap server is local.  Ideally, since we
know which articles we added to the group, we could update the
information without contacting the server.  But unless someone else
finds that the current way is too slow, maybe just leaving things as
they are is fine.

Dan