concurrent synchronization of nnmaildir directories

concurrent synchronization of nnmaildir directories

[James Leifer]

Hi,

I'm switching some of my groups from the trusty-rusty nnmh to the more
robust nnmaildir 8-).  My main motivation is to be able to easily
synchronize maildirs on my desktop and laptop.  I'll do the sync with
unison or some scripts I cook up myself.

To do the synchronization, I have to be able to add, modify, and
delete already delivered messages.  In particular, is it safe to do
the following operations while nnmaildir is *running*?

* Add a new message with its marks, i.e. add a file in foo/cur and
  update foo/.nnmaildir/{marks/{read,reply,...},nov} appropriately.

* Modify the marks of an existing message, i.e. add and delete entries
  in foo/.nnmaildir/marks/{read,reply,...}

* Delete an existing message, i.e. delete a file in foo/cur and the
  corresponding entries in foo/.nnmaildir/{marks/{read,reply,...},nov}.

Since the filesystem doesn't allow me to add a message and its marks
in a single atomic step, does the order of the operations and the
modification times of the files matter?

Is it dangerous to do these operations while nnmaildir is regenerating
nov data, reading or saving marks, etc?

Do I need to do anything to foo/.nnmaildir/num?

Comments would be gratefully received.

-James

Re: concurrent synchronization of nnmaildir directories

[Paul Jarc]

James Leifer <James.Leifer@inria.fr> wrote:
> In particular, is it safe to do the following operations while
> nnmaildir is *running*?
>
> * Add a new message with its marks, i.e. add a file in foo/cur and
>   update foo/.nnmaildir/{marks/{read,reply,...},nov} appropriately.

Should be safe, I think.  I'd add the message before the marks.
Otherwise, if you happen to be exiting the group in Gnus at the same
time, nnmaildir would delete the (seemingly useless) marks if the
message doesn't exist yet.

> * Modify the marks of an existing message, i.e. add and delete entries
>   in foo/.nnmaildir/marks/{read,reply,...}

Safe.

> * Delete an existing message, i.e. delete a file in foo/cur and the
>   corresponding entries in foo/.nnmaildir/{marks/{read,reply,...},nov}.

Safe.  nnmaildir will clean up the cruft in .nnmaildir on its own
later on, at *Summary* exit.

> Is it dangerous to do these operations while nnmaildir is regenerating
> nov data, reading or saving marks, etc?

nnmaildir will use whatever data is there when it happens to look.
So, e.g., a message might have been added, but its marks may not be
there yet, so in Gnus, the message will seem to lack some marks it
should have.  If it's a problem for you for that to happen, then don't
do that.  nnmaildir will pick up the rest of the data the next time it
checks the filesystem, so the situation should straighten itself out
eventually, but there could be problematic cases: e.g., removing the
'expire mark on an old message that's about to be deleted will not
reliably save the message if you do it at the same time as expiry;
expiry might hit the message first.

> Do I need to do anything to foo/.nnmaildir/num?

No.  It's best to let nnmaildir manage that.


paul

Re: concurrent synchronization of nnmaildir directories

[James Leifer]

prj@po.cwru.edu (Paul Jarc) writes:

(snipping liberally)

>> * Modify the marks of an existing message, i.e. add and delete entries
>>   in foo/.nnmaildir/marks/{read,reply,...}
>
> Safe.

Paul, thanks for your reply.

Let me clarify this last point...

Suppose a message uniq has the set of flags F when the group is
opened.

Suppose in gnus I add the set of the flags X0 and remove the set Y0.
Now, when nnmaildir flushes its state, it will change the on-disc flag
set to

   F \ Y0 U X0

(using \ for set difference and U for set union).  Suppose instead
that before it flushes its state, an external program adds the set of
flags X1 and removes the set Y1, leaving the state on disc

   F \ Y1 U X1

Now what will nnmaildir do when it flushes its changes?  The reason I
ask is as follows: some of the flags it would normally delete (in Y0)
are no longer there on disc; others it would add (in X0) are already
there.

Warm regards,
-James

P.S. If you get a chance, could you have a look at my
performance/design question concerning constant messages
(http://article.gmane.org/gmane.emacs.gnus.general/55114/)?

Re: concurrent synchronization of nnmaildir directories

[Paul Jarc]

James Leifer <James.Leifer@inria.fr> wrote:
> some of the flags it would normally delete (in Y0) are no longer
> there on disc; others it would add (in X0) are already there.

That shouldn't cause any trouble.  It ought to work fine.

> P.S. If you get a chance, could you have a look at my
> performance/design question concerning constant messages

Will do.


paul