nnmaildir vs offlineimap: patch to use flags properly

nnmaildir vs offlineimap: patch to use flags properly

[Magnus Henoch]

Hi all,

Last week I posted a patch for nnmaildir to make it use maildir
filename-based flags, in order for it to play nicely with offlineimap
WRT marking messages as read/unread.

Here it is:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8055#41

But it seems like nobody saw it, so I'm making some noise here.  What do
you think about the patch?

Regards,
Magnus

Re: nnmaildir vs offlineimap: patch to use flags properly

[poirier]

On Thu. 2012-08-23 at 07:04 PM EDT, Magnus Henoch <magnus.henoch@gmail.com> wrote:

> Last week I posted a patch for nnmaildir to make it use maildir
> filename-based flags, in order for it to play nicely with offlineimap
> WRT marking messages as read/unread.
>
> Here it is:
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8055#41
>
> But it seems like nobody saw it, so I'm making some noise here.  What do
> you think about the patch?

I can't speak to the implementation, but I'd be very happy to see gnus
interoperate compatibly with other maildir clients that flags.

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Magnus Henoch <magnus.henoch@gmail.com> writes:

> But it seems like nobody saw it, so I'm making some noise here.  What do
> you think about the patch?

I've tried it out, and from my initial testing it appears to work quite
well. I've been using a script to sync the flags back and forth on every
run of offlineimap, which is sort of hit and miss. As in, there have
been a few mails that are not updated. I think this has to do with the
renaming of filenames that offlineimap does to keep track of files. So I
had a bunch of mails that were not marked with the S flag, but which
gnus considered to be seen. When enabling your patch, those turned up
again, and I did not seem to be able to mark them as read. Marking them
externally (in this case using Mutt) did seem to clear everything up
though, and new mails are marked correctly. Moving mails around between
folders also seems to work. I'll keep using this for a few days and see
if anything odd turns up.

Other than that, many thanks for writing this patch; I've been bothered
by this for a while, but not enough to do something about it. I don't
suppose there's any way to sync/store expired flags in the same way? I
seem to be able to add an E flag to mails in my maildir, and offlineimap
syncs it to the server, but then it seems to be discarded; at least it's
removed by offlineimap on the next sync. I'm afraid I don't know enough
about the imap protocol to know whether this is just because my imap
server is rubbish, or whether it's expected protocol behaviour...

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> I'll keep using this for a few days and see if anything odd turns up.

Following up on this, it seems gnus seemed to delete all expired
messages in my inbox, instead of only those older than seven days as
it's supposed to. I don't know whether this is related to your patch,
but thought I'd mention it...

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Magnus Henoch]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Toke Høiland-Jørgensen <toke@toke.dk> writes:
>
>> I'll keep using this for a few days and see if anything odd turns up.
>
> Following up on this, it seems gnus seemed to delete all expired
> messages in my inbox, instead of only those older than seven days as
> it's supposed to. I don't know whether this is related to your patch,
> but thought I'd mention it...

Right, nnmaildir uses the modtime of the article file to decide its age,
just like nnml does, but unlike nnml, the function
`nnmaildir-request-expire-articles' contains its own logic to decide
whether the article is expirable instead of calling
`nnmail-expired-article-p'.  Not sure if that would make a difference,
but that's definitely duplication worth getting rid of.

I can't reproduce the problem myself; I marked an article a few days old
as expirable and exited the group, but it wasn't expired immediately.

Regards,
Magnus

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Magnus Henoch <magnus.henoch@gmail.com> writes:

> I can't reproduce the problem myself; I marked an article a few days
> old as expirable and exited the group, but it wasn't expired
> immediately.

Well, it has only happened once, and I did several exits from the same
folder before then without issue. I am not entirely sure that I did not
do something wrong, but I don't *think* so... :)

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Well, it has only happened once, and I did several exits from the same
> folder before then without issue. I am not entirely sure that I did not
> do something wrong, but I don't *think* so... :)

Okay, early deletion of expired mails is definitely happening more
often. Not the first time when entering a folder, but afterwards. I.e.
expire messages in folder, exit folder, reenter it (possibly with other
new mails in it), exit again, expired messages are gone, even with a
seven-day expiry timer...

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Okay, early deletion of expired mails is definitely happening more
> often. Not the first time when entering a folder, but afterwards. I.e.
> expire messages in folder, exit folder, reenter it (possibly with other
> new mails in it), exit again, expired messages are gone, even with a
> seven-day expiry timer...

*ahem* - Turns out that it was a misconfigured offlineimap that was
deleting the mails. Sorry for the confusion (and the spam). The patch
seems to work quite well now that I fixed that issue... :)

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Magnus Henoch]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Magnus Henoch <magnus.henoch@gmail.com> writes:
>
>> But it seems like nobody saw it, so I'm making some noise here.  What do
>> you think about the patch?
>
> I've tried it out, and from my initial testing it appears to work quite
> well. I've been using a script to sync the flags back and forth on every
> run of offlineimap, which is sort of hit and miss. As in, there have
> been a few mails that are not updated. I think this has to do with the
> renaming of filenames that offlineimap does to keep track of files. So I
> had a bunch of mails that were not marked with the S flag, but which
> gnus considered to be seen. When enabling your patch, those turned up
> again, and I did not seem to be able to mark them as read. Marking them
> externally (in this case using Mutt) did seem to clear everything up
> though, and new mails are marked correctly. Moving mails around between
> folders also seems to work. I'll keep using this for a few days and see
> if anything odd turns up.

Right, if the file name is not formatted as nnmaildir expects it to
be... I'll have a look at that and try to make it expect the
unexpected.  You don't happen to be able to find what the file names of
those messages were before they were fixed?

> Other than that, many thanks for writing this patch; I've been bothered
> by this for a while, but not enough to do something about it. I don't
> suppose there's any way to sync/store expired flags in the same way? I
> seem to be able to add an E flag to mails in my maildir, and offlineimap
> syncs it to the server, but then it seems to be discarded; at least it's
> removed by offlineimap on the next sync. I'm afraid I don't know enough
> about the imap protocol to know whether this is just because my imap
> server is rubbish, or whether it's expected protocol behaviour...

As I understand it, an IMAP client is allowed to set its own flags on
the server (and nnimap does this to save e.g. expired and dormant
flags), but there is no standard way to store non-standard flags with
Maildir, and offlineimap seems not to be able to handle it.  I'd be
happy to be proven wrong, though.

Regards,
Magnus

Re: nnmaildir vs offlineimap: patch to use flags properly

[Toke Høiland-Jørgensen]

Magnus Henoch <magnus.henoch@gmail.com> writes:

> Right, if the file name is not formatted as nnmaildir expects it to
> be... I'll have a look at that and try to make it expect the
> unexpected.  You don't happen to be able to find what the file names of
> those messages were before they were fixed?

Sorry, no; I fixed them all already, and it did not occur to me to look
at the file names. They showed up correctly in Mutt, though, so I'm not
sure they were that garbled. And gnus could read them, marking them as
read just didn't work...

> As I understand it, an IMAP client is allowed to set its own flags on
> the server (and nnimap does this to save e.g. expired and dormant
> flags), but there is no standard way to store non-standard flags with
> Maildir, and offlineimap seems not to be able to handle it. I'd be
> happy to be proven wrong, though.

I've experimented with setting an E flag on the mails (by adding it to
the file name), and I've seen offlineimap assert that it is syncing the
flag to the server, but on the next sync they are removed in the
maildir... Dunno if this is just because my server is wonky...

-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

Re: nnmaildir vs offlineimap: patch to use flags properly

[Lars Ingebrigtsen]

Magnus Henoch <magnus.henoch@gmail.com> writes:

> Last week I posted a patch for nnmaildir to make it use maildir
> filename-based flags, in order for it to play nicely with offlineimap
> WRT marking messages as read/unread.
>
> Here it is:
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8055#41

It seems like a good patch.  Do you have Emacs copyright assignment
papers on file?

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: nnmaildir vs offlineimap: patch to use flags properly

[Magnus Henoch]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Magnus Henoch <magnus.henoch@gmail.com> writes:
>
>> Last week I posted a patch for nnmaildir to make it use maildir
>> filename-based flags, in order for it to play nicely with offlineimap
>> WRT marking messages as read/unread.
>>
>> Here it is:
>>
>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8055#41
>
> It seems like a good patch.  Do you have Emacs copyright assignment
> papers on file?

Yes, I do.

Regards,
Magnus