Agent & disconnected IMAP: move messages between folders?

Agent & disconnected IMAP: move messages between folders?

[John Owens]

Greetings, a few years ago I looked at Gnus as a mail client.
I didn't adopt it for one main reason: though it supports
disconnected IMAP (I spend a lot of time on my computer out
of Internet range), it doesn't allow me to move messages
between IMAP folders during disconnected operation. I am
presuming the way this would be done is to queue up move
commands and execute them once IMAP is reconnected (this is
how it is done in Wanderlust, an emacs-IMAP mailer).

(I read email in the following way: most stuff comes into my
inbox, I read it, I answer it, I file it into a folder. Thus
such a capability is useful to me. I am totally willing to hear
about a more Gnus-y way to handle this, but I don't think 
"auto-filter every message that comes in into its destination"
is ideal for me, since I get lots of mail that would be hard
to categorize like that.)

Has there been any progress in the last few years toward
supporting this capability? The FAQ still says no, but perhaps
it is reflecting a released version and newer versions have
added this capability.

Thanks!

JDO

Re: Agent & disconnected IMAP: move messages between folders?

[Steinar Bang]

>>>>> John Owens <john_owens@yahoo.com>:

> Greetings, a few years ago I looked at Gnus as a mail client.  I
> didn't adopt it for one main reason: though it supports disconnected
> IMAP (I spend a lot of time on my computer out of Internet range), it
> doesn't allow me to move messages between IMAP folders during
> disconnected operation.

That's still the case I think.

At least it was the last time I tried (which admittedly was a while
back).

I think you would have to do quite a few changes to the Agent
architecture to support it.

The way it is now, is that if you have an article cached in the agent,
and then move the original article to a new folder, the moved article
won't be cached until you download it in the new folder (or read it, if
you have added gnus-agent-fetch-selected-article to
gnus-select-article-hook).

On a side note: another thing that's missing from the agentized imap
support, is partial download of messages (ie. download the text and
leave the attachments on the server, until you actually want to read
them).

This isn't as critical as it was ten years ago, but it's still is a
capability of the IMAP protocol that isn't utilized by nnimap.

Re: Agent & disconnected IMAP: move messages between folders?

[John Owens]

Steinar Bang <sb <at> dod.no> writes:

> The way it is now, is that if you have an article cached in the agent,
> and then move the original article to a new folder, the moved article
> won't be cached until you download it in the new folder (or read it, if
> you have added gnus-agent-fetch-selected-article to
> gnus-select-article-hook).

That's not a deal-breaker. It's not ideal, but it's not a deal-breaker.
Are you suggesting that it would work properly, but just not cache in
the expected way?

JDO

Re: Agent & disconnected IMAP: move messages between folders?

[Steinar Bang]

>>>>> John Owens <john_owens@yahoo.com>:

> Are you suggesting that it would work properly, but just not cache in
> the expected way?

No.  I didn't intend to imply that.  As I said, it didn't work the last
time I tried it, and I don't _really_ know the current state (but I
haven't seen any changes fly past that would support it).

I just mentioned the missing cache of a moved article, to illustrate the
not-quite-ideal fit between the Agent and the nnimap backend (and the
capabilities of IMAP).

Re: Agent & disconnected IMAP: move messages between folders?

[John Owens]

John Owens <john_owens <at> yahoo.com> writes:

> (I read email in the following way: most stuff comes into my
> inbox, I read it, I answer it, I file it into a folder. Thus
> such a capability is useful to me. I am totally willing to hear
> about a more Gnus-y way to handle this, but I don't think 
> "auto-filter every message that comes in into its destination"
> is ideal for me, since I get lots of mail that would be hard
> to categorize like that.)

So I'd like to hear about a more Gnus-y way to do this. The good
thing about Gnus (compared to Wanderlust) is it has a much larger
user community and a much more active developer community. 
Disconnected IMAP would seem to be something that would be useful
for lots of people, so I'd suspect that the reason it's not 
currently supported (at least this feature) is less "this is a
feature we haven't got to yet" and more "well, people just use Gnus
in this different way that obviates the need for such a feature".
Thoughts? What is the common usage pattern for disconnected IMAP?

JDO

Re: Agent & disconnected IMAP: move messages between folders?

[Greg Troxel]

John Owens <john_owens@yahoo.com> writes:

> John Owens <john_owens <at> yahoo.com> writes:
>
>> (I read email in the following way: most stuff comes into my
>> inbox, I read it, I answer it, I file it into a folder. Thus
>> such a capability is useful to me. I am totally willing to hear
>> about a more Gnus-y way to handle this, but I don't think 
>> "auto-filter every message that comes in into its destination"
>> is ideal for me, since I get lots of mail that would be hard
>> to categorize like that.)
>
> So I'd like to hear about a more Gnus-y way to do this. The good
> thing about Gnus (compared to Wanderlust) is it has a much larger
> user community and a much more active developer community. 
> Disconnected IMAP would seem to be something that would be useful
> for lots of people, so I'd suspect that the reason it's not 
> currently supported (at least this feature) is less "this is a
> feature we haven't got to yet" and more "well, people just use Gnus
> in this different way that obviates the need for such a feature".
> Thoughts? What is the common usage pattern for disconnected IMAP?

I've somewhat struggled with the same issue.

The gnus-y way is to have mail arrive and leave it there, treating it as
read news, and never deleting it.  I don't read mail like that either.

What I do is filter most of my mail to separate IMAP folders (outside of
gnus, at delivery time).  Examples are mailing lists, commit messsages
per project, plus anything else that I can identify and don't need to
know about in a timely manner.  Spam scoring N spamassassin points for N
>=1 goes to spam.N.  Many of these folders I read only with gnus.  On
some I use total-expire, and most of the rest auto-expire.

Then my INBOX gets unfiltered mail, and some of that I file manually
(with B m).  On other messages I type 'd' which is bound to:

(setq gdt-trash-newsgroup "nnimap+[machine-name]:Trash")
(defun gdt-gnus-move-to-trash ()
  (interactive)
  (gnus-summary-move-article nil gdt-trash-newsgroup))

via
      (add-hook
       'gnus-summary-mode-hook
       (lambda ()
	 (local-set-key "d" 'gdt-gnus-move-to-trash)))

so I generally keep my INBOX small.  This is not gnus-y, but how I want
to read mail, and how I can use other IMAP clients against the same
mailbox - a lot of the point of sorting is to make looking at INBOX on
my phone reasonable.

I don't think there is philosophical opposition to disconnected IMAP -
it's just hard.  Right now the agent seems to be a read-only cache of
the server.  With disconnected IMAP you keep log of operations to play
back later.  I don't know whether implementations typically modify the
cache or play back the log locally when reading the cache, but I suspect
the cache reflects the state with the log operations applied.  All of
this seems doable, and seems like a natural extension to the agent.  You
might want to look at the Coda filesystem for inspiration of how to deal
with things, but Thunderbird might be more useful.  It seems there
should be an open-source disconnected-mode IMAP library.  I wonder if
it's just easier to use an offline IMAP proxy, and if those really
support disconnected mode.

Before getting to offline IMAP, there are some cache consistency bugs to
fix.  When I have the agent enabled, I find ghost articles.  I suspect
that these are from moving the article from a group (message from a
folder :-) with another client.  So there needs to be some sort of
"resynchronize" operation where messages in a folder are listed by UID
or something and then removed from the agent cache if gone on the
server.  Plus any changes made to the server need to be made in the
cache.  Then a change in architecture so that changes are made to the
cache, which makes them locally and then puts them in the playback log,
doing them immediately if possible.

Full disconnected operation might be really hard, but just having the "B
m" operation be supported would be a huge win, since most of what I want
to do disconnected is clean up my INBOX: file messages away and delete
them (which for me is refile to Trash, with Trash having total-expire).

Re: Agent & disconnected IMAP: move messages between folders?

[John Sullivan]

Greg Troxel <gdt@work.lexort.com> writes:

> I don't think there is philosophical opposition to disconnected IMAP -
> it's just hard.  Right now the agent seems to be a read-only cache of
> the server.  With disconnected IMAP you keep log of operations to play
> back later.  I don't know whether implementations typically modify the
> cache or play back the log locally when reading the cache, but I suspect
> the cache reflects the state with the log operations applied.  All of
> this seems doable, and seems like a natural extension to the agent.  You
> might want to look at the Coda filesystem for inspiration of how to deal
> with things, but Thunderbird might be more useful.  It seems there
> should be an open-source disconnected-mode IMAP library.  I wonder if
> it's just easier to use an offline IMAP proxy, and if those really
> support disconnected mode.

I've been getting around this problem happily for a couple years now by
having gnus talk to a local IMAP server running right on my laptop
(courier in my case), and syncing to my upstream IMAP server with the
program offlineimap. I do that for mail and leafnode for nntp, and life
is pretty good offline.


-- 
John Sullivan
Emacs Planner Maintainer
http://wjsullivan.net/PlannerMode.html
GPG Key: AE8600B6

Re: Agent & disconnected IMAP: move messages between folders?

[Xavier Maillard]

John Sullivan <john@wjsullivan.net> writes:

> Greg Troxel <gdt@work.lexort.com> writes:
>
>> I don't think there is philosophical opposition to disconnected IMAP -
>> it's just hard.  Right now the agent seems to be a read-only cache of
>> the server.  With disconnected IMAP you keep log of operations to play
>> back later.  I don't know whether implementations typically modify the
>> cache or play back the log locally when reading the cache, but I suspect
>> the cache reflects the state with the log operations applied.  All of
>> this seems doable, and seems like a natural extension to the agent.  You
>> might want to look at the Coda filesystem for inspiration of how to deal
>> with things, but Thunderbird might be more useful.  It seems there
>> should be an open-source disconnected-mode IMAP library.  I wonder if
>> it's just easier to use an offline IMAP proxy, and if those really
>> support disconnected mode.
>
> I've been getting around this problem happily for a couple years now by
> having gnus talk to a local IMAP server running right on my laptop
> (courier in my case), and syncing to my upstream IMAP server with the
> program offlineimap. I do that for mail and leafnode for nntp, and life
> is pretty good offline.

Just doing the same for a few days now and it really changed my
life :) Until now, I used nnmaildir backend but I find it, erm,
buggy thus I changed my mind and I am now logging to a local
dovecot IMAP server. Revolutionary !

Xavier

Re: Agent & disconnected IMAP: move messages between folders?

[Steven E. Harris]

Greg Troxel <gdt@work.lexort.com> writes:

> When I have the agent enabled, I find ghost articles.  I suspect that
> these are from moving the article from a group (message from a folder
> :-)) with another client.  So there needs to be some sort of
> "resynchronize" operation where messages in a folder are listed by UID
> or something and then removed from the agent cache if gone on the
> server.

Yes, I've been suffering this problem with the agent for years and
years. The only remedy I've found is to occasionally run
`gnus-agent-expire-group' manually when looking at the Summary Buffer
within the afflicted mailbox group. I don't know why this function
doesn't get run automatically, or more frequently. Sometimes it's the
only way not only to get rid of these "ghost articles", but also to make
the articles that arrived and were noticed outside of Gnus available
within Gnus.

-- 
Steven E. Harris