Bad interaction between nnimap + agent + expiry (bug?)

Bad interaction between nnimap + agent + expiry (bug?)

[Gregory Novak]

I am using Gnus to read e-mail via the nnimap backend.  When I mark an
email as expirable in my INBOX and then quit the group, it gets filed
into a folder with the name recd-mail-2005-12 (for example; code
attached below).  This works as I expect it when I'm talking to the
IMAP server directly (gnus-agent is "plugged").  That is, the expired
messages end up in the group recd-mail-2005-12 marked 'O'.

However, I would like to be able to read e-mail when not connected to
the internet, and I've found that reading mail via the agent is more
pleasant in that there are fewer annoying pauses while Gnus talks to
the imap server.  Therefore I would like to use the agent.

The problem is that when I:
1) 'J j' to "unplug" the agent
2) Enter a group, hit 'E' to expire some messages, then quit the group
3) 'J j' to "plug" the agent
4) 'J Y' to synchronize agent flags with the server
5) Enter the group in question and then quit it (so that expiry runs
its course)

I would like for the expired messages to end up in recd-mail-2005-12
with the mark 'O'.  What actually happens is that they end up in
recd-mail-2005-12 marked 'E'.

I would call this a bug but I'm willing to entertain the notion that
there's some sublime logic to what Gnus is doing.  I've spent most of
today stepping through various gnus functions, trying to figure out
why on Earth messages that were marked as expiriable while the agent
was unplugged should be treated differently from messages that were
marked as expirable when talking to the imap server.  I've had no
success.  Two reasonable candidates that failed were:

1) the value of gnus-preserve-flags makes no difference
2) the (when (and (gnus-agent ... line at the end of
gnus-request-accept-article in gnus-int.el seemed like a likely place
for the funny business, but commenting it out made no difference.

So, a few questions:
1) Why is this happening?
2) How can I make it stop?
3) Is the expiry information stored within Gnus or on the imap server?
4) What is the 'cannonical' way to find out what flags are attached to
a given message?  (This was a major sticking point when trying to step
through the various functions... I would like to be able to see
exactly where the 'expired' flag was set, but couldn't seem to get
reasonable answers.  That is, gnus-article-mark returned some value
even if the message number I gave didn't exist, and
gnus-summary-article-mark only returned non-nil for the message in
question after the whole expiry process was done and I had quit and
re-entered the group recd-mail-2005-12).

Ok, that's it.  Thanks for any guidance/solutions.

Greg

Re: Bad interaction between nnimap + agent + expiry (bug?)

[Simon Josefsson]

Gregory Novak <novak@ucolick.org> writes:

> I am using Gnus to read e-mail via the nnimap backend.  When I mark an
> email as expirable in my INBOX and then quit the group, it gets filed
> into a folder with the name recd-mail-2005-12 (for example; code
> attached below).

There were no code attached.  How do you expire it into
recd-mail-2005-12?

> This works as I expect it when I'm talking to the IMAP server
> directly (gnus-agent is "plugged").  That is, the expired messages
> end up in the group recd-mail-2005-12 marked 'O'.
>
> However, I would like to be able to read e-mail when not connected to
> the internet, and I've found that reading mail via the agent is more
> pleasant in that there are fewer annoying pauses while Gnus talks to
> the imap server.  Therefore I would like to use the agent.
>
> The problem is that when I:
> 1) 'J j' to "unplug" the agent
> 2) Enter a group, hit 'E' to expire some messages, then quit the group
> 3) 'J j' to "plug" the agent
> 4) 'J Y' to synchronize agent flags with the server
> 5) Enter the group in question and then quit it (so that expiry runs
> its course)
>
> I would like for the expired messages to end up in recd-mail-2005-12
> with the mark 'O'.  What actually happens is that they end up in
> recd-mail-2005-12 marked 'E'.
>
> I would call this a bug but I'm willing to entertain the notion that
> there's some sublime logic to what Gnus is doing.

It seem buggy, yes.

Perhaps (setq imap-log t) will help.  Then you should be able to see
if Gnus is asking the server to remove the "gnus-expire" flag on the
article in question, and whether it later somehow end up adding that
flag again to the article when it has moved to recd-mail-2005-12.

> 3) Is the expiry information stored within Gnus or on the imap server?

It depends.  If the server support client flags, Gnus will store the
expiry flag as a "gnus-expire" flag on the article.  If the server
doesn't support this, you shouldn't see expire marks at all, I think,
or something other weird could happen.  Client-specific flags is also
a poorly implemented area in IMAP, so it is not impossible that you
are experiencing a server bug.

> 4) What is the 'cannonical' way to find out what flags are attached to
> a given message?  (This was a major sticking point when trying to step
> through the various functions... I would like to be able to see
> exactly where the 'expired' flag was set, but couldn't seem to get
> reasonable answers.  That is, gnus-article-mark returned some value
> even if the message number I gave didn't exist, and
> gnus-summary-article-mark only returned non-nil for the message in
> question after the whole expiry process was done and I had quit and
> re-entered the group recd-mail-2005-12).

There are several answers here.

The IMAP server keep the real flags on the message.  A 'UID FETCH 4711
(FLAGS)' query should get you the flags.

Gnus keep a copy of the flags, updated at irregular intervals, in
~/.newsrc.eld (which contain setq's).

Inside a summary buffer, there are several variables that store future
flags, which are not set directly on the server, but will be set if
you quit the group.

Hope this helps...  my time to work on this is limited, so I hope
others will help or you'll be able to debug it yourself.

Re: Bad interaction between nnimap + agent + expiry (bug?)

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> If the server support client flags, Gnus will store the expiry flag
> as a "gnus-expire" flag on the article.

A related question: If I'm running Gnus on one computer and set an
IMAP message to expire ("E" is visible in the Summary), then later run
Gnus on a different computer connected to the same IMAP server,
assuming the server supports client flags, should the second Gnus
instance detect the server-stored expiration flag and mark its copy of
that same message for expiration?

-- 
Steven E. Harris

Re: Bad interaction between nnimap + agent + expiry (bug?)

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> If the server support client flags, Gnus will store the expiry flag
>> as a "gnus-expire" flag on the article.
>
> A related question: If I'm running Gnus on one computer and set an
> IMAP message to expire ("E" is visible in the Summary), then later run
> Gnus on a different computer connected to the same IMAP server,
> assuming the server supports client flags, should the second Gnus
> instance detect the server-stored expiration flag and mark its copy of
> that same message for expiration?

Yes.  The E mark should be visible in the Summary buffer on the second
Gnus if the first Gnus had set the mark.

Do (setq imap-log t) and search for "gnus-expire".  You can tell if
the server support client flags by looking at the PERMANENTFLAGS:

* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  

The \* means that client flags are supported.

Re: Bad interaction between nnimap + agent + expiry (bug?)

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> Yes.  The E mark should be visible in the Summary buffer on the
> second Gnus if the first Gnus had set the mark.

I've noticed messages marked as expirable in one instance not be
recognized as expirable in another. (Mark message while at work, go
home, connect, and find the same message unmarked.) The server does
appear to support client flags:

,----
| 252 SELECT "INBOX"
| * FLAGS (gnus-expire gnus-forward gnus-dormant \Draft \Answered \Flagged \Deleted \Seen \Recent)
| * OK [PERMANENTFLAGS (gnus-expire gnus-forward gnus-dormant \* \Draft \Answered \Flagged \Deleted \Seen)] Limited
| * 148 EXISTS
| * 0 RECENT
| * OK [UIDVALIDITY 1067275254] Ok
| * OK [MYRIGHTS "acdilrsw"] ACL
| 252 OK [READ-WRITE] Ok
`----

I'll have to try some debugging and see if I can reproduce the
problem.

-- 
Steven E. Harris

Re: Bad interaction between nnimap + agent + expiry (bug?)

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Yes.  The E mark should be visible in the Summary buffer on the
>> second Gnus if the first Gnus had set the mark.
>
> I've noticed messages marked as expirable in one instance not be
> recognized as expirable in another. (Mark message while at work, go
> home, connect, and find the same message unmarked.) The server does
> appear to support client flags:
>
> ,----
> | 252 SELECT "INBOX"
> | * FLAGS (gnus-expire gnus-forward gnus-dormant \Draft \Answered \Flagged \Deleted \Seen \Recent)
> | * OK [PERMANENTFLAGS (gnus-expire gnus-forward gnus-dormant \* \Draft \Answered \Flagged \Deleted \Seen)] Limited
> | * 148 EXISTS
> | * 0 RECENT
> | * OK [UIDVALIDITY 1067275254] Ok
> | * OK [MYRIGHTS "acdilrsw"] ACL
> | 252 OK [READ-WRITE] Ok
> `----
>
> I'll have to try some debugging and see if I can reproduce the
> problem.

Following how flags are set on a particular message in *imap-log* in
both Gnus may help.