nnimap problem solved by removing .agentview and .overview

nnimap problem solved by removing .agentview and .overview

[Mats Lidell]

Hi,

I read my mail daily with squirrelmail as well as with gnus. I have
noticed that mail I read with squirrelmail isn't always possible to
see in gnus. (I haven't check if the problem always is there. Some
mails might show up in both.) I have however found a workaround. If I
delete the .agentview and .overview files the summary is initialized
correctly and the mail read by squirrelmail is seen in gnus as well.

There is however no way that this situation can be known from gnus. If
it wasn't that I remember having seen those mails I wouldn't have
missed them. So I feel that the situation is bad. nnimap shouldn't
behave like this. Why isn't the overview-file updated when the
contents in the imap folder is changed from outside of gnus?

I have reported this before but haven't gotten any good response saying
either that my setup is broken or that nnimap is broken. Am I the only
one seeing this?

Yours
-- 
%% Mats

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Mats Lidell <matsl@contactor.se> writes:

> Am I the only one seeing this?

No, I see it too. My remedy is usually to enter the group -- usually
my INBOX -- and run `gnus-agent-expire-group', exit the group, and
reenter. It's a pain and it doesn't seem like an acceptable
solution. Somehow other mail clients figure this mismatch out and
adapt accordingly.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Kevin Greiner]

Mats Lidell <matsl@contactor.se> writes:

> I read my mail daily with squirrelmail as well as with gnus. I have
> noticed that mail I read with squirrelmail isn't always possible to
> see in gnus. (I haven't check if the problem always is there. Some
> mails might show up in both.) I have however found a workaround. If I
> delete the .agentview and .overview files the summary is initialized
> correctly and the mail read by squirrelmail is seen in gnus as well.
>
> There is however no way that this situation can be known from gnus. If
> it wasn't that I remember having seen those mails I wouldn't have
> missed them. So I feel that the situation is bad. nnimap shouldn't
> behave like this. Why isn't the overview-file updated when the
> contents in the imap folder is changed from outside of gnus?
>
> I have reported this before but haven't gotten any good response saying
> either that my setup is broken or that nnimap is broken. Am I the only
> one seeing this?

The problem is that, when the agent is turned on, your backend
requests don't go directly to the backend.  They instead go to the
agent which tries to respond using what it has locally cached.  If,
and only if, you request something that is not in the cache, will the
agent request the MISSING content from the real backend.

Frankly, I don't see how to make the agent work better with nnimap
without introducing nnimap specific code into the agent or changing
the backend interface.  Neither is something to take lightly so I
don't believe that this situation will be changing anytime soon.

One possibility, have you set gnus-agent-consider-all-articles to t?
If you do so, the agent will let gnus download read messages.  That
may fix part of your problem.

If that doesn't help, then I'd suggest that you either 1) turn off the
agent, or 2) periodically delete the .agentview file.  You shouldn't
need to delete the .overview file.

Kevin

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> The problem is that, when the agent is turned on, your backend
> requests don't go directly to the backend.  They instead go to the
> agent which tries to respond using what it has locally cached.  If,
> and only if, you request something that is not in the cache, will the
> agent request the MISSING content from the real backend.

How is one ever able to see something that something exists that is
not locally cached? How does Gnus ever know that there is something
new on the IMAP server that it has not seen?

Also, I thought this behind-the-back-change is what the IMAP
UIDVALIDITY response is supposed to help discern. I thought that with
UIDVALIDITY Gnus could detect that it wasn't the last client to have
altered the mailbox. Reading RFC 3501, I see that's not the case;
change in UIDVALIDITY only warns that the old message IDs may have
changed.

What if, upon contacting the IMAP server and noting agreeable
UIDVALIDITY, Gnus does a SEARCH or UID query from its last known
message ID, checking to see if any new IDs are present that would not
have appeared in a check for what is UNSEEN? Isn't there some way like
this to figure out that the server has messages that the client has
never heard of?

> Frankly, I don't see how to make the agent work better with nnimap
> without introducing nnimap specific code into the agent or changing
> the backend interface.  Neither is something to take lightly so I
> don't believe that this situation will be changing anytime soon.

Are these interfaces documented somewhere? It's hard to comment on
potential solutions without knowing the current constraints. Again,
it's surprising that even after four or five years of Gnus supporting
IMAP, we still haven't done better than other IMAP clients.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> The problem is that, when the agent is turned on, your backend
>> requests don't go directly to the backend.  They instead go to the
>> agent which tries to respond using what it has locally cached.  If,
>> and only if, you request something that is not in the cache, will the
>> agent request the MISSING content from the real backend.
>
> How is one ever able to see something that something exists that is
> not locally cached? How does Gnus ever know that there is something
> new on the IMAP server that it has not seen?
>
> Also, I thought this behind-the-back-change is what the IMAP
> UIDVALIDITY response is supposed to help discern. I thought that with
> UIDVALIDITY Gnus could detect that it wasn't the last client to have
> altered the mailbox. Reading RFC 3501, I see that's not the case;
> change in UIDVALIDITY only warns that the old message IDs may have
> changed.
>
> What if, upon contacting the IMAP server and noting agreeable
> UIDVALIDITY, Gnus does a SEARCH or UID query from its last known
> message ID, checking to see if any new IDs are present that would not
> have appeared in a check for what is UNSEEN? Isn't there some way like
> this to figure out that the server has messages that the client has
> never heard of?

Gnus is supposed the invoke these SEARCH requests every time, upon
entering the group, to sync all flags, and its list of which articles
exists.  However, if you only press 'g' and then ask the agent to
fetch articles, the group might not have been entered, so nnimap
haven't had any chance of updating the information.

However, if entering and exiting a group when plugged in doesn't
notice all articles, there is a bug.  I suspect the problem lies in
the agent rather than in nnimap, or (perhaps even more likely) in
interaction between nnimap and the agent.

Is there a simple recipe for reproducing this?

>> Frankly, I don't see how to make the agent work better with nnimap
>> without introducing nnimap specific code into the agent or changing
>> the backend interface.  Neither is something to take lightly so I
>> don't believe that this situation will be changing anytime soon.
>
> Are these interfaces documented somewhere? It's hard to comment on
> potential solutions without knowing the current constraints.

I'd even take another step back: Exactly what problem is in the design
of agent make it incompatible with nnimap?  I believe the designs are
pretty compatible.

> Again, it's surprising that even after four or five years of Gnus
> supporting IMAP, we still haven't done better than other IMAP
> clients.

Not many people are working on nnimap.el, nor improving the generic
backend interface, these days, it seems...

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> Gnus is supposed the invoke these SEARCH requests every time, upon
> entering the group, to sync all flags, and its list of which articles
> exists.  However, if you only press 'g' and then ask the agent to
> fetch articles, the group might not have been entered, so nnimap
> haven't had any chance of updating the information.

I never ask the agent to download articles explicitly other than by
way of actually selecting an article, which triggers
gnus-select-article-hook to call gnus-agent-fetch-selected-article.

> However, if entering and exiting a group when plugged in doesn't
> notice all articles, there is a bug.

This happens to me pretty much every day.

> I suspect the problem lies in the agent rather than in nnimap, or
> (perhaps even more likely) in interaction between nnimap and the
> agent.

Yes, I never saw this problem with nnimap itself, only with the agent
in the mix.

> Is there a simple recipe for reproducing this?

Here's my workflow:

o At work, start Gnus in plugged mode. Read e-mail all day.

o At the end of the day, by way of an rsync-dependent script, send all
  my changed score files, ~/.newsrc, and ~/.newsrc.eld files to my
  ISP's disk.
  Note that I don't sync the agent directories. My understanding is
  that those are for local cache management. As I want to have a
  different cache at home, I leave it to Gnus to figure out the
  appropriate caching operations.

o At home, again using this rsync script, download updates to my score
  files, ~/.newsrc, and ~/.newsrc.eld.

o Start Gnus in plugged mode. New IMAP messages are visible, but the
  summary buffer in nnimap groups does not manifest messages that
  arrived while I was reading at work.

o Reversing the process, send changed files to my ISP's disk.

o Get to work again the next day, download changed files from my ISP's
  disk.

o Start Gnus in plugged mode. New IMAP messages are visible, but no
  messages that arrived while I was reading at home are visible.


Running gnus-agent-expire-group coaxes Gnus into re-syncing the
message list, though sometimes my marks on newly-discovered messages
get lost in the process.

> I'd even take another step back: Exactly what problem is in the
> design of agent make it incompatible with nnimap?  I believe the
> designs are pretty compatible.

Yes, that's a good question.

> Not many people are working on nnimap.el, nor improving the generic
> backend interface, these days, it seems...

Understood. If the problem lies with the agent, then calling nnimap
"done enough" would be just fine.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

>> Is there a simple recipe for reproducing this?
>
> Here's my workflow:
>
> o At work, start Gnus in plugged mode. Read e-mail all day.
>
> o At the end of the day, by way of an rsync-dependent script, send all
>   my changed score files, ~/.newsrc, and ~/.newsrc.eld files to my
>   ISP's disk.
>   Note that I don't sync the agent directories. My understanding is
>   that those are for local cache management. As I want to have a
>   different cache at home, I leave it to Gnus to figure out the
>   appropriate caching operations.

I'm not completely sure, but there might be a problem with your
workflow.  The .newsrc.eld contain active info, and the agent may be
using the active info to decide which headers to download.  So what is
happening is pretty much this:

1. (work machine) active range in .newsrc is 1-4711, which match agent
   overview's.

2. (home machine) active range in .newsrc is 1-4711 (by way of rsync),
   which might not match agent overview's, because they were created
   when the active range was 1-4000 (yesterday).

3. (home machine) as you read new e-mail, the active range increase to
   1-5711, and the agent overview include 4711-5711 data.

4. (work machine) active range in .newsrc is 1-5711 (by way of rsync)
   but agent overview only has 1-4711.  the agent trusts 1-5711 and
   assumes 4711-5711 doesn't exist, and doesn't try to download them
   again.

In other words, you need to sync .overview too.

You may need to sync .agentview as well, but then again you might need
to _not_ sync .agentview.

In the worst case, there is some information in .agentview that need
to be rsync'ed and some information that need to not be rsync'ed.  If
so, we should probably try to separate the data that is local state,
i.e. cache directory, from the .newsrc state data.

If someone recalls exactly how the data in .agentview is used by the
agent, that would answer whether you need to sync it or not.  I don't
recall.

Perhaps you need to take a step back to solve this:

Exactly why are you syncing .newsrc* at all?

Messages flags are supposed to be synchronized via the IMAP server, so
there's no need to synchronize the files for them.  Most other data
that is stored in .newsrc.eld can be stored in .emacs or .gnus.  The
only major thing, that I recall, is that Topics structures can't be
defined except by via .newsrc.eld.

Getting rid of .newsrc.eld would be nice.  Move all setup related
things to .gnus, and all state data into ~/News or wherever they
belong.

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> I'm not completely sure, but there might be a problem with your
> workflow.

[...]

> In other words, you need to sync .overview too.

That would be easy, if we know that it's the correct solution.

> You may need to sync .agentview as well, but then again you might
> need to _not_ sync .agentview.
>
> In the worst case, there is some information in .agentview that need
> to be rsync'ed and some information that need to not be rsync'ed.  If
> so, we should probably try to separate the data that is local state,
> i.e. cache directory, from the .newsrc state data.

Yes, that would be a good way to go, as we often hear of people trying
to figure out how to use Gnus from different locations and keep a
fairly consistent view of their message and server data.

> Perhaps you need to take a step back to solve this:
>
> Exactly why are you syncing .newsrc* at all?

It's not just IMAP. My NNTP ranges are in there, so I need those to be
consistent between work and home. Perhaps those should be split out
into per-server .newsrc files. I can't remember how to get Gnus to
split that stuff out, and I don't know if it works properly.

Can you recommend an arrangement that would let me maintain state for
two NNTP servers (at present, Panix and Gmane) but would not require
shipping .newsrc and .newsrc.eld around?

> Messages flags are supposed to be synchronized via the IMAP server,
> so there's no need to synchronize the files for them.

Yes, I agree.

> Most other data that is stored in .newsrc.eld can be stored in
> .emacs or .gnus.

I do have all my settings in .gnus or other configuration files that
it loads in turn.

> The only major thing, that I recall, is that Topics structures can't
> be defined except by via .newsrc.eld.

Right. I had forgotten about that one, but now I recall occasionally
having to go into .newsrc.eld to fix topic tree problems manually.

> Getting rid of .newsrc.eld would be nice.  Move all setup related
> things to .gnus, and all state data into ~/News or wherever they
> belong.

Let's say that my topics were stable for now. Is there some way I
could rearrange my .newsrc* files with current functionality in Gnus
that would let me synchronize ranges for my NNTP servers, but not for
my IMAP servers?

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

>> Perhaps you need to take a step back to solve this:
>>
>> Exactly why are you syncing .newsrc* at all?
>
> It's not just IMAP. My NNTP ranges are in there, so I need those to be
> consistent between work and home. Perhaps those should be split out
> into per-server .newsrc files. I can't remember how to get Gnus to
> split that stuff out, and I don't know if it works properly.
>
> Can you recommend an arrangement that would let me maintain state for
> two NNTP servers (at present, Panix and Gmane) but would not require
> shipping .newsrc and .newsrc.eld around?

NNTP doesn't need .newsrc.eld, does it?  IIRC, Gnus should pick up new
information from .newsrc, if it is newer than .newsrc.eld.  For flags,
syncing ~/News/marks/ should suffice, if the marks stuff is working
for nntp.

>> Getting rid of .newsrc.eld would be nice.  Move all setup related
>> things to .gnus, and all state data into ~/News or wherever they
>> belong.
>
> Let's say that my topics were stable for now. Is there some way I
> could rearrange my .newsrc* files with current functionality in Gnus
> that would let me synchronize ranges for my NNTP servers, but not for
> my IMAP servers?

The above might work.  Naturally, I haven't tested it.

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> NNTP doesn't need .newsrc.eld, does it?

Can anyone else help answer this question? I don't want to let this
topic drop off, as the same goals and problems will persist.

> IIRC, Gnus should pick up new information from .newsrc, if it is
> newer than .newsrc.eld.

I noticed that .newsrc only stores ranges for my primary NNTP
server. Gmane is one of my secondary servers, and I need those ranges
synced as well.

> For flags, syncing ~/News/marks/ should suffice, if the marks stuff
> is working for nntp.

Looking in some of the .marks files under my ~/[News]/marks directory,
I see that there is a "read" sexp with ranges that match .newsrc. So
perhaps .marks is /all/ I need to sync NNTP.

Can anyone comment on how Gnus reconciles differences between the
.marks files and .newsrc[.eld]? Which ones will take precedence?

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

"Steven E. Harris" <seh@panix.com> writes:

> Can anyone comment on how Gnus reconciles differences between the
> .marks files and .newsrc[.eld]? Which ones will take precedence?

Following up to myself, I just tried an experiment. I exited Gnus and
manually edited one of my .marks files to back off the read range by a
single message, leaving the corresponding entries in .newsrc and
.newsrc.eld untouched. Upon restarting Gnus, the Group and Summary
buffer reflected that I had not read the last article in the edited
group.

So it does seem that if NNTP can be synced with the .marks files, and
IMAP groups don't need .newsrc.eld, then I can stop syncing .newsrc
and .newsrc.eld -- topic changes notwithstanding.

Are there any files that nnimap groups do need synced, or can they
recover /everything/ Gnus knows about them from the server?

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

>> For flags, syncing ~/News/marks/ should suffice, if the marks stuff
>> is working for nntp.
>
> Looking in some of the .marks files under my ~/[News]/marks directory,
> I see that there is a "read" sexp with ranges that match .newsrc. So
> perhaps .marks is /all/ I need to sync NNTP.
>
> Can anyone comment on how Gnus reconciles differences between the
> .marks files and .newsrc[.eld]? Which ones will take precedence?

.marks will take precedence, that was the reason for introducing the
"marks" backend interface.  I.e., to allow backends to override what
Gnus knows about groups (from newsrc.eld).  Nnimap uses this to tell
Gnus about flag changes on the server.  Later it was realized that
nnml and nnfolder could benefit from the same interface, to allow
synchronization of nnml/nnfolder's to carry flags with them.

>> Can anyone comment on how Gnus reconciles differences between the
>> .marks files and .newsrc[.eld]? Which ones will take precedence?
>
> Following up to myself, I just tried an experiment. I exited Gnus and
> manually edited one of my .marks files to back off the read range by a
> single message, leaving the corresponding entries in .newsrc and
> .newsrc.eld untouched. Upon restarting Gnus, the Group and Summary
> buffer reflected that I had not read the last article in the edited
> group.
>
> So it does seem that if NNTP can be synced with the .marks files, and
> IMAP groups don't need .newsrc.eld, then I can stop syncing .newsrc
> and .newsrc.eld -- topic changes notwithstanding.

Yup.

> Are there any files that nnimap groups do need synced, or can they
> recover /everything/ Gnus knows about them from the server?

Some esoteric things aren't synced via the server.  For example:
bookmarks, group parameters (but use `gnus-parameters' instead),
topics.  I think we've eliminated all major values.

To see what is stored in .newsrc.eld for a group, type `G E' on it.
Anything there you don't see in .marks won't be synced.

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> Some esoteric things aren't synced via the server.  For example:
> bookmarks, group parameters (but use `gnus-parameters' instead),
> topics.  I think we've eliminated all major values.

Does gnus-parameters respect topic containment? That is, can one
emulate topic "group" parameters using gnus-parameters? It turns out
that after many years of use, most of my "group" parameters are
actually in topics.

Trying to craft a regexp to match all the groups under each topic that
holds parameters can be difficult. Perhaps I can set the variable
gnus-topic-topology somewhere else besides .newsrc.eld.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Some esoteric things aren't synced via the server.  For example:
>> bookmarks, group parameters (but use `gnus-parameters' instead),
>> topics.  I think we've eliminated all major values.
>
> Does gnus-parameters respect topic containment?

No.  It can probably be fixed though.

> Trying to craft a regexp to match all the groups under each topic that
> holds parameters can be difficult. Perhaps I can set the variable
> gnus-topic-topology somewhere else besides .newsrc.eld.

You can set it anywhere, but it might get overridden by any topics
commands you invoke in group buffer, and the new value will be saved,
and may even be read later on.

Hm.  Perhaps we could add a new, customizable, variable to specify
topics structure.  If set, its value would override
`gnus-topic-toplogy'.  The Topics commands would have to be changed to
update the new variable, though, which might be complicated.  For
customizable variables, those commands can use customize-save, but
what if the user set the variable manually somewhere?

gnus-parameters remodeled for Topics (was: nnimap problem solved by removing .agentview and .overview)

[Ted Zlatanov]

On Mon, 16 Aug 2004, jas@extundo.com wrote:

> "Steven E. Harris" <seh@panix.com> writes:
> 
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> Some esoteric things aren't synced via the server.  For example:
>>> bookmarks, group parameters (but use `gnus-parameters' instead),
>>> topics.  I think we've eliminated all major values.
>>
>> Does gnus-parameters respect topic containment?
> 
> No.  It can probably be fixed though.

I think this would be a great feature.  The customization interface
currently looks like this:

INS DEL Cons-cell:
            Regexp: 
            Repeat:
            INS DEL Lisp expression: nil

I think topics could simply be an alternate choice under Regexp; you
should be able to say "Topic Regexp," "Topic Exact," "Group Regexp,"
and "Group Exact."

Also, the Lisp Expression should be an alternate, but not the main
choice.  The user should be able to pick from a predefined list of
variables.  That would improve usability.

Ted

Re: gnus-parameters remodeled for Topics

[Steven E. Harris]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> I think topics could simply be an alternate choice under Regexp; you
> should be able to say "Topic Regexp," "Topic Exact," "Group Regexp,"
> and "Group Exact."

Yes, that would be perfect. I'd use "Topic Exact" and just move the
parameter sexps out .newsrc.eld.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

>> So it does seem that if NNTP can be synced with the .marks files, and
>> IMAP groups don't need .newsrc.eld, then I can stop syncing .newsrc
>> and .newsrc.eld -- topic changes notwithstanding.
>
> Yup.

Nope, it turns out IMAP does not work properly without syncing
.newsrc. Today I sent all my marks files home, but did not send
.newsrc.eld along with them. When I connected to my nnimap servers at
home, the message counts in the Group buffer are all wrong. Well, sort
of; they count how many messages Gnus has not seen at home, not how
many of my IMAP messages are unread. My INBOX, for example, noted 8
new messages, but upon entering the group there were no new ones
there. I had read them all at work already.

Also, it looks like the expire mark is not synchronized. All of the
IMAP messages that arrived in my INBOX today at work were marked as
expirable (manually with 'E'), but none of those messages are marked
as expirable now at home. Is the expirable mark supposedly stored on
the server? If not, is .newsrc.eld the only place where expire marks
are stored?

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>>> So it does seem that if NNTP can be synced with the .marks files, and
>>> IMAP groups don't need .newsrc.eld, then I can stop syncing .newsrc
>>> and .newsrc.eld -- topic changes notwithstanding.
>>
>> Yup.
>
> Nope, it turns out IMAP does not work properly without syncing
> .newsrc. Today I sent all my marks files home, but did not send
> .newsrc.eld along with them. When I connected to my nnimap servers at
> home, the message counts in the Group buffer are all wrong. Well, sort
> of; they count how many messages Gnus has not seen at home, not how
> many of my IMAP messages are unread. My INBOX, for example, noted 8
> new messages, but upon entering the group there were no new ones
> there. I had read them all at work already.

The unread article count in group buffer is only an estimate.  In
situations like this, it will be wrong.  But I thought there was some
workaround for this.  What's the value of
`gnus-after-getting-new-news-hook' for you?  It should include
`gnus-fixup-nnimap-unread-after-getting-new-news'.

> Also, it looks like the expire mark is not synchronized. All of the
> IMAP messages that arrived in my INBOX today at work were marked as
> expirable (manually with 'E'), but none of those messages are marked
> as expirable now at home. Is the expirable mark supposedly stored on
> the server? If not, is .newsrc.eld the only place where expire marks
> are stored?

They are supposed to be stored on the server, if the server support
client-specific flags.  Do (setq imap-log t) and look in *imap-log*
buffer for the PERMANENTFLAGS value for the mailbox, e.g.:

* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk \*)]  

The \* is the critical value.  If it is absent, Gnus cannot store
non-standard flags on the server.

Re: nnimap problem solved by removing .agentview and .overview

[Jochen Küpper]

On Tue, 17 Aug 2004 10:23:28 +0200 Simon Josefsson wrote:

Simon> The unread article count in group buffer is only an estimate.
Simon> In situations like this, it will be wrong. But I thought there
Simon> was some workaround for this. What's the value of
Simon> `gnus-after-getting-new-news-hook' for you? It should include
Simon> `gnus-fixup-nnimap-unread-after-getting-new-news'.

I have a similar problem. The hook in question has the following value
,----
| gnus-after-getting-new-news-hook's value is 
| (gnus-display-time-event-handler gnus-fixup-nnimap-unread-after-getting-new-news)
`----

I do not sync any files between work and home computer and let
synchronization completely up to IMAP. The I routinely get wrong
unread message counts in the Group buffer. Generally these are fixed
by doing M-g on the groups line in the Group buffer (not always,
though).

Moreover I often have stale article headers in the Summary buffer
which cannot be selected since there is "No such article". (This is
probably related to the agent?)

I haven't bother to work around it sing M-g is simple enough (for the
first case at least it is a "fix"). Nevertheless I could do some
testing.

Greetings,
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Liberté, Égalité, Fraternité                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Simon Josefsson <jas@extundo.com> writes:

> The unread article count in group buffer is only an estimate.  In
> situations like this, it will be wrong.  But I thought there was
> some workaround for this.  What's the value of
> `gnus-after-getting-new-news-hook' for you?  It should include
> `gnus-fixup-nnimap-unread-after-getting-new-news'.

,----[ gnus-after-getting-new-news-hook ]
| `gnus-after-getting-new-news-hook' is a variable declared in Lisp.
|   -- loaded from "gnus-start"
| 
| Value: (gnus-display-time-event-handler gnus-fixup-nnimap-unread-after-getting-new-news)
| 
| Documentation:
| *A hook run after Gnus checks for new news when Gnus is already running.
`----

The fixup hook is in there, but it's not doing its job. I did note
that hitting M-g on the group before entering it fixes the
count. Perhaps this hook is not getting run upon initial connection.

> They are supposed to be stored on the server, if the server support
> client-specific flags.  Do (setq imap-log t) and look in *imap-log*
> buffer for the PERMANENTFLAGS value for the mailbox, e.g.:
>
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk \*)]  
>
> The \* is the critical value.  If it is absent, Gnus cannot store
> non-standard flags on the server.

,----
| 187 EXAMINE "INBOX.lists.gnus-ding"
| * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
| * OK [PERMANENTFLAGS ()] No permanent flags permitted
| * 29 EXISTS
| * 0 RECENT
| * OK [UIDVALIDITY 1067647543] Ok
| * OK [MYRIGHTS "acdilrsw"] ACL
| 187 OK [READ-ONLY] Ok
`----

Lovely. Is this something that I can appeal to my ISP's administrators
to enable, or is it an inherent feature lack in a given IMAP
implementation (in this case, Courier)?

Just to reiterate, though, .newsrc.eld is in fact quite necessary for
maintaining proper IMAP state if the server isn't as complicit as Gnus
would prefer. Perhaps we can move these nnimap marks out of
.newsrc.eld, or at least echo them into marks files like the NNTP
backends do.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> The unread article count in group buffer is only an estimate.  In
>> situations like this, it will be wrong.  But I thought there was
>> some workaround for this.  What's the value of
>> `gnus-after-getting-new-news-hook' for you?  It should include
>> `gnus-fixup-nnimap-unread-after-getting-new-news'.
>
> ,----[ gnus-after-getting-new-news-hook ]
> | `gnus-after-getting-new-news-hook' is a variable declared in Lisp.
> |   -- loaded from "gnus-start"
> | 
> | Value: (gnus-display-time-event-handler gnus-fixup-nnimap-unread-after-getting-new-news)
> | 
> | Documentation:
> | *A hook run after Gnus checks for new news when Gnus is already running.
> `----
>
> The fixup hook is in there, but it's not doing its job. I did note
> that hitting M-g on the group before entering it fixes the
> count. Perhaps this hook is not getting run upon initial connection.

If someone could debug this, that would be good.  One would have to
verify that the IMAP data returned from the server is actually
accurate, first, and then check that the hook does the right thing.
I'm not sure I can assist with this via email...

>> They are supposed to be stored on the server, if the server support
>> client-specific flags.  Do (setq imap-log t) and look in *imap-log*
>> buffer for the PERMANENTFLAGS value for the mailbox, e.g.:
>>
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk \*)]  
>>
>> The \* is the critical value.  If it is absent, Gnus cannot store
>> non-standard flags on the server.
>
> ,----
> | 187 EXAMINE "INBOX.lists.gnus-ding"
> | * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
> | * OK [PERMANENTFLAGS ()] No permanent flags permitted
> | * 29 EXISTS
> | * 0 RECENT
> | * OK [UIDVALIDITY 1067647543] Ok
> | * OK [MYRIGHTS "acdilrsw"] ACL
> | 187 OK [READ-ONLY] Ok
> `----
>
> Lovely. Is this something that I can appeal to my ISP's administrators
> to enable, or is it an inherent feature lack in a given IMAP
> implementation (in this case, Courier)?

Most likely the latter, alas.

> Just to reiterate, though, .newsrc.eld is in fact quite necessary for
> maintaining proper IMAP state if the server isn't as complicit as Gnus
> would prefer. Perhaps we can move these nnimap marks out of
> .newsrc.eld, or at least echo them into marks files like the NNTP
> backends do.

Yes, that would be possible.  Nnimap could decide which marks the
server doesn't support, and write those in ~/News/marks/nnimap:foo/ or
somewhere, which then could be synced.

Re: nnimap problem solved by removing .agentview and .overview

[Mats Lidell]

>>>>> Simon wrote:

Simon> Is there a simple recipe for reproducing this?

My recipe is like this.

1. Send myself a letter. (I have splitting configured so gnus will
   automatically place mail form INBOX in the folder 'incoming')
2. Get new mail by pressing 'g'. See that there is one new letter in
   'incoming'.
3. Enter the summary for 'incoming' and see that it is the letter I
   sent myself. (Don't read it.)
4. Send myself another letter.
5. Save and exit gnus and exit (X)Emacs.
6. Read mail with squirrelmail using the same source (imap and same
   server).
7. See the new mail (the second sent above) in INBOX. (There is no
   splitting activated in squirrelmail so the mail stays in INBOX.)
8. Use squirrelmail to move the new mail to the folder 'incoming'
   (simulating the fancy splitting in gnus ;-)
9. Go to 'incoming' in squirrelmail and look at the two new letters
   there. (The two I just sent myself.) (Don't read them.)
10. Exit squirrelmail.
11. Start (X)Emacs and go back into gnus.
12. See that the *Group* says there are two new mails in incoming.
13. Enter the summary and find only one new letter. The first.

Voila

Yours
-- 
%% Mats

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

Mats Lidell <matsl@contactor.se> writes:

>>>>>> Simon wrote:
>
> Simon> Is there a simple recipe for reproducing this?
>
> My recipe is like this.
>
> 1. Send myself a letter. (I have splitting configured so gnus will
>    automatically place mail form INBOX in the folder 'incoming')
> 2. Get new mail by pressing 'g'. See that there is one new letter in
>    'incoming'.
> 3. Enter the summary for 'incoming' and see that it is the letter I
>    sent myself. (Don't read it.)
> 4. Send myself another letter.
> 5. Save and exit gnus and exit (X)Emacs.
> 6. Read mail with squirrelmail using the same source (imap and same
>    server).
> 7. See the new mail (the second sent above) in INBOX. (There is no
>    splitting activated in squirrelmail so the mail stays in INBOX.)
> 8. Use squirrelmail to move the new mail to the folder 'incoming'
>    (simulating the fancy splitting in gnus ;-)
> 9. Go to 'incoming' in squirrelmail and look at the two new letters
>    there. (The two I just sent myself.) (Don't read them.)
> 10. Exit squirrelmail.
> 11. Start (X)Emacs and go back into gnus.
> 12. See that the *Group* says there are two new mails in incoming.
> 13. Enter the summary and find only one new letter. The first.

This suggest a bug, somewhere; the above should work fine.

Could you get *imap-log* dumps from the first and second part of
running Gnus?  (setq imap-log t)

Could you also collect *imap-log* for:

14. Quit summary buffer.
15. rm -rf ~/News/agent/nnimap/SERVER/GROUP
16. C-u RET the summary buffer.

Can you see the article in the summary buffer?

Re: nnimap problem solved by removing .agentview and .overview

[Mats Lidell]

>>>>> Simon wrote:

Simon> Could you get *imap-log* dumps from the first and second part
Simon> of running Gnus?  (setq imap-log t)

I'm sorry. I just tried to recreate the problem with logging active
but unfortunately I failed. I can't get the problem now after
following my own recipe.

I have been working without the agent for a while so for creating the
requested logs I turned on the agent again, turned on imap logging and
tried to follow my own instructions. No articles were lost
though. Sigh.

So I guess this item is closed until someone comes back with logs that
can explain what happens.

Yours
-- 
%% Mats

Re: nnimap problem solved by removing .agentview and .overview

[Kevin Greiner]

Simon Josefsson <jas@extundo.com> writes:

> "Steven E. Harris" <seh@panix.com> writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> The problem is that, when the agent is turned on, your backend
>>> requests don't go directly to the backend.  They instead go to the
>>> agent which tries to respond using what it has locally cached.  If,
>>> and only if, you request something that is not in the cache, will the
>>> agent request the MISSING content from the real backend.
>>
>> How is one ever able to see something that something exists that is
>> not locally cached? How does Gnus ever know that there is something
>> new on the IMAP server that it has not seen?

Some requests are only intercepted with running unplugged.  For
example the request to get the active range is passed through to the
backend with running plugged.  Let's work an example for those reading
along.

  Yesterday, we'll asume that your active range consisted of eight
  messages (numbered 1 through 8) and that you used gnus to view the
  folder's summary so all eight are known to the agent.

  Today, we'll assume that two new messages have arrived so we know
  that your active range is now 1-10.  When gnus starts it queries the
  server for the current active range so now gnus knows that your
  active range is 1-10.  Now, you open the folder to display the
  summary buffer.  The summary functions make a request for all
  article headers in the range 1-10.  That request is redirected to
  the agent by gnus internals.  The agent checks its .overview file,
  sees that it has headers for articles 1-8 (from yesterday) and
  forwards a modified request for the range 9-10 to the server.  The
  agent gets the headers for articles 9-10 from the server, merges
  them into its .overview so that it never needs to ask the server for
  them, then constructs a response that satisfies the summary buffer's
  original request for article headers in the range 1-10.

>> Also, I thought this behind-the-back-change is what the IMAP
>> UIDVALIDITY response is supposed to help discern. I thought that with
>> UIDVALIDITY Gnus could detect that it wasn't the last client to have
>> altered the mailbox. Reading RFC 3501, I see that's not the case;
>> change in UIDVALIDITY only warns that the old message IDs may have
>> changed.

>> What if, upon contacting the IMAP server and noting agreeable
>> UIDVALIDITY, Gnus does a SEARCH or UID query from its last known
>> message ID, checking to see if any new IDs are present that would not
>> have appeared in a check for what is UNSEEN? Isn't there some way like
>> this to figure out that the server has messages that the client has
>> never heard of?
>
> Gnus is supposed the invoke these SEARCH requests every time, upon
> entering the group, to sync all flags, and its list of which articles
> exists.  However, if you only press 'g' and then ask the agent to
> fetch articles, the group might not have been entered, so nnimap
> haven't had any chance of updating the information.
>
> However, if entering and exiting a group when plugged in doesn't
> notice all articles, there is a bug.  I suspect the problem lies in
> the agent rather than in nnimap, or (perhaps even more likely) in
> interaction between nnimap and the agent.
>
> Is there a simple recipe for reproducing this?
>
>>> Frankly, I don't see how to make the agent work better with nnimap
>>> without introducing nnimap specific code into the agent or changing
>>> the backend interface.  Neither is something to take lightly so I
>>> don't believe that this situation will be changing anytime soon.
>>
>> Are these interfaces documented somewhere? It's hard to comment on
>> potential solutions without knowing the current constraints.
>
> I'd even take another step back: Exactly what problem is in the design
> of agent make it incompatible with nnimap?  I believe the designs are
> pretty compatible.

The agent assumes that the backend will never change the state of
anything already known to the agent.  The fact that UIDVALIDITY exists
is proof that that isn't the case with imap.

We could propagate the uid concept into the agent then have the agent
verify its uid against the backend's uid before trusting its own
cache.  It would probably work but I'm worried about the performance
impact (would nnimap have to query the server each time the uid is
checked?) and having to dummy up uid values for all of the other
backends.

>> Again, it's surprising that even after four or five years of Gnus
>> supporting IMAP, we still haven't done better than other IMAP
>> clients.
>
> Not many people are working on nnimap.el, nor improving the generic
> backend interface, these days, it seems...

I agree.

Kevin

Re: nnimap problem solved by removing .agentview and .overview

[Simon Josefsson]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

>> I'd even take another step back: Exactly what problem is in the design
>> of agent make it incompatible with nnimap?  I believe the designs are
>> pretty compatible.
>
> The agent assumes that the backend will never change the state of
> anything already known to the agent.  The fact that UIDVALIDITY exists
> is proof that that isn't the case with imap.

Right.  Even without the Agent, nnimap doesn't handle UIDVALIDITY
changes anyway, though...

> We could propagate the uid concept into the agent then have the agent
> verify its uid against the backend's uid before trusting its own
> cache.  It would probably work but I'm worried about the performance
> impact (would nnimap have to query the server each time the uid is
> checked?) and having to dummy up uid values for all of the other
> backends.

UIDVALIDITY is sent automatically when you enter a mailbox in IMAP, or
when you do a STATUS on a group (like Gnus does for each mailbox when
you press 'g'), so it doesn't cost much.  Nnimap store the current
uidvalidity as a group parameter too.  Nnimap.el used to compare the
value against the server, but there is no sane action nnimap.el can
take when the numbers doesn't match, so the comparison and user query
was disabled.

I share your feelings, it might hurt more than it pays to introduce
the uidvalidity concept in Gnus.  Perhaps still not that many IMAP
servers implement UIDVALIDITY in the right way too.  Some used to
increment UIDVALIDITY all the time (some used to set it to current
unix time...).  Some never increment it, even when UIDs are no longer
valid.  Ignoring UIDVALIDITY used to work better in practice, than
implement the proper handling of UIDVALIDITY changes.  That might have
changed, but then there is the problem of deciding how to actually
implement the proper handling, too..  it looks ugly.

Re: nnimap problem solved by removing .agentview and .overview

[Ted Zlatanov]

On Thu, 12 Aug 2004, jas@extundo.com wrote:

> Not many people are working on nnimap.el, nor improving the generic
> backend interface, these days, it seems...

Is there a specific wishlist?

Ted

Re: nnimap problem solved by removing .agentview and .overview

[Chris Green]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> Is there a specific wishlist?

My wishlist item for nnimap is for it to handle a situation that I've
had many difficulties figuring out.  
(setq gnus-secondary-select-methods
      '((nnimap "csoft"
		(nnimap-address "mail102.csoft.net")
		(nnimap-nov-is-evil t)
		(nnimap-stream ssl)
		(nnimap-list-pattern ("INBOX" "INBOX*"))
		)))
                
(setq gnus-message-archive-group "nnimap+csoft:INBOX.Sent")

When the session times out, I have to M-g to get nnimap to reconnect
to the server.

In the Gcc realm to INBOX.Sent, I often get an (error) "IMAP process
not running" connected and my current *mail* buffer is sent out via
SMTP but not archived on the imap server.

I'd like nnimap to quietly attempt to reconnect once before raising an
error in both of these situations.

Merely killing the openssl process isn't good enough to replicate the
situation because nnimap seems to handle the termination of the app
just fine.  The server in question is a courier imap server if it
happens to make a difference.
-- 
Chris Green <cmg@dok.org>
Don't use a big word where a diminutive one will suffice.

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

Chris Green <cmg@dok.org> writes:

> When the session times out, I have to M-g to get nnimap to reconnect
> to the server.

I have similar trouble using nnimap with openssl and starttls on
Cygwin. There seems to be some problem with how Gnus or XEmacs detects
that a connection dropped.

Sometimes when I go to check for new messages in an nnimap group, I'll
try, say, "1-g" to check groups at level one. That will hang, so I'll
hit C-g to cancel (sometimes it takes many tries), then place my
cursor on the group in question and hit M-g. That /sometimes/ checks
the messages again with the same dropped or corrupted connection, so I
have to stop Gnus again, go to the Server buffer, and try to close the
supposedly open connection there. Of course, trying to close the
connection makes Gnus hang again, so several more M-g attempts may
finally yield Gnus recognizing that it has no open connection to this
server. Only at that point can I tell Gnus to open a fresh connection
to check if I have any new messages. This whole process usually takes
close to a minute and around 50 keystrokes.

I don't notice the same problems, at least to the same degree, with
direct connections to the IMAP servers. It's those programs in between
Gnus and the IMAP servers such as openssl and starttls that aggravate
these flaky connection problems.

Even worse is trying to quit Gnus. I notice that for around the last
year of keeping up to date with CVS, Gnus does not quit cleanly if any
of its open connections had dropped. I have to manually kill some or
all of the openssl and starttls processes, as well as hit C-g many
times, to get Gnus to shut down. I can't tell if this problem is
directly related to nnimap or not.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Steven E. Harris]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> Is there a specific wishlist?

Make it faster to check for new messages and then enter a group with
new messages to see the headers. At present, when I hit 1-g to check
for new mail in one nnimap group, it sometimes takes Gnus about 10
seconds to report whether a message arrived or not (with
`nnimap-need-unselect-to-notice-new-mail' set to t). Then, if I notice
the '0' message count change to something positive, I hit enter to
select the group. Gnus then takes about ten more seconds to present a
one- or two-line Summary buffer showing the newly arrived
messages.

These nnimap groups are agentized, but I'm guessing that this slow
behavior described here involves Gnus asking the server for message
ranges and parsing the headers on new messages.

The IMAP server I use most frequently is Courier:

,----
| * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE
|   THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA
|   IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready.
|   Copyright 1998-2004 Double Precision, Inc.
|   See COPYING for distribution information.
`----

I can understand the '1-g' operation taking a while, or selecting a
group to take a while, but it's painful that both take a while and
both are necessary just to see a new header in the Summary.

-- 
Steven E. Harris

Re: nnimap problem solved by removing .agentview and .overview

[Steinar Bang]

>>>>> "Ted Zlatanov" <tzz@lifelogs.com>:

> On Thu, 12 Aug 2004, jas@extundo.com wrote:

>> Not many people are working on nnimap.el, nor improving the generic
>> backend interface, these days, it seems...

> Is there a specific wishlist?

For nnimap, partial downloading, comes to mind.

Ie. make it possible to leave big attachments on the server, while
viewing the text parts of the message, and only download the
attachments when you actually wish to view the attachment.