Protect against servers stepping on each other's toes

Protect against servers stepping on each other's toes

[Kai Großjohann]

If I have two servers using the same directory, there is potential for
the two stepping on each other's toes.

Here's a suggestion: when a server is first activated, it writes
something special to its directory which says what server `owns' that
directory.

But even before that, each server looks to see if some other server
has already written that the directory is `owned' by it.  When this
happens, a warning is issued or something like that.

Of course, given a directory d, all backends should use the same
method for `claiming' the directory; else they might trample on each
other's territory after all.

Also, maybe we want to put the claiming information inthe active
file.  Maybe in the directory _and_ in the active file?

Thoughts?
kai
-- 
GNU/Linux provides a nice `poweroff' command, but where is `poweron'?

Re: Protect against servers stepping on each other's toes

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
> If I have two servers using the same directory, there is potential for
> the two stepping on each other's toes.

I think much of this could be prevented by using different defaults
for different backends, or (like nnmaildir does) by requiring the user
to explicitly specify a directory in the server parameters.

> Here's a suggestion: when a server is first activated, it writes
> something special to its directory which says what server `owns' that
> directory.

Are we trying to protect against two servers in two different Emacsen?
If not, we don't need to touch the directory itself.  Something
internal to Gnus would do.

Is this indicator supposed to stay there all the time, or will it be
removed when the server is closed?

> But even before that, each server looks to see if some other server
> has already written that the directory is `owned' by it.  When this
> happens, a warning is issued or something like that.

There's a race condition there.  But that's not a problem for two
servers in the same Emacs.

> Also, maybe we want to put the claiming information inthe active
> file.  Maybe in the directory _and_ in the active file?

Putting it in both wouldn't provide any additional usefullness.
Putting it in the active file would mean we aren't protected against
other Emacsen, but in that case a nonpersistent variable will do; it
doesn't need to be saved anywhere.


paul

Re: Protect against servers stepping on each other's toes

[Kai Großjohann]

prj@po.cwru.edu (Paul Jarc) writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
>> If I have two servers using the same directory, there is potential for
>> the two stepping on each other's toes.
> 
> I think much of this could be prevented by using different defaults
> for different backends, or (like nnmaildir does) by requiring the user
> to explicitly specify a directory in the server parameters.

It's too late now.  Imagine the huge number of people out there with
(nnml "") in their gnus-secondary-select-methods.  So clearly the
default for nnml must be ~/Mail.  And then imagine the other huge
number of people out there with (nnfolder "") in
gnus-secondary-select-methods.  So the default for nnfolder must be
~/Mail.

Or are you trying to say that all these Gnusae should suddenly stop
working unless reconfigured?  I think this is not a viable alternative :-|

>> Here's a suggestion: when a server is first activated, it writes
>> something special to its directory which says what server `owns' that
>> directory.
> 
> Are we trying to protect against two servers in two different
> Emacsen?  If not, we don't need to touch the directory itself.
> Something internal to Gnus would do.

I mean the same Emacs.

The trouble is with people who do this:

(add-to-list 'gnus-secondary-select-methods '(nnml ""))
(add-to-list 'gnus-secondary-select-methods '(nnfolder ""))

See?  This is the situation I was thinking about.

> Is this indicator supposed to stay there all the time, or will it be
> removed when the server is closed?

I guess it should be left there -- after all, my "nnml:" server has
been owning the ~/Mail directory for many years now, and it's not
going to stop tomorrow.

>> But even before that, each server looks to see if some other server
>> has already written that the directory is `owned' by it.  When this
>> happens, a warning is issued or something like that.
> 
> There's a race condition there.  But that's not a problem for two
> servers in the same Emacs.

Right.

>> Also, maybe we want to put the claiming information inthe active
>> file.  Maybe in the directory _and_ in the active file?
> 
> Putting it in both wouldn't provide any additional usefullness.
> Putting it in the active file would mean we aren't protected against
> other Emacsen, but in that case a nonpersistent variable will do; it
> doesn't need to be saved anywhere.

Maybe some crooked mind does something like this:

(add-to-list 'gnus-secondary-select-methods 
             '(nnfolder "a"
                (nnfolder-directory "/tmp/a")
                (nnfolder-active-file "/tmp/active")))
(add-to-list 'gnus-secondary-select-methods 
             '(nnfolder "b"
                (nnfolder-directory "/tmp/b")
                (nnfolder-active-file "/tmp/active")))

Note how they use different directories but the same active file.

But you're right, it seems that it is sufficient to set a variable
value, such as gnus-occupied-directories or something like this.

kai
-- 
GNU/Linux provides a nice `poweroff' command, but where is `poweron'?

Re: Protect against servers stepping on each other's toes

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I think much of this could be prevented by using different defaults
>> for different backends, or (like nnmaildir does) by requiring the user
>> to explicitly specify a directory in the server parameters.
> 
> It's too late now.  Imagine the huge number of people out there with
> (nnml "") in their gnus-secondary-select-methods.

Yeah.  Just wishful thinking, I guess.

>> Putting it in the active file would mean we aren't protected against
>> other Emacsen, but in that case a nonpersistent variable will do; it
>> doesn't need to be saved anywhere.
> 
> Maybe some crooked mind does something like this:
> 
> (add-to-list 'gnus-secondary-select-methods 
>              '(nnfolder "a"
>                 (nnfolder-directory "/tmp/a")
>                 (nnfolder-active-file "/tmp/active")))
> (add-to-list 'gnus-secondary-select-methods 
>              '(nnfolder "b"
>                 (nnfolder-directory "/tmp/b")
>                 (nnfolder-active-file "/tmp/active")))

Well, marking the directory (by whatever means) won't help there.
Should there be a gnus-occupied-active-files as well?


paul

Re: Protect against servers stepping on each other's toes

[Kai Großjohann]

prj@po.cwru.edu (Paul Jarc) writes:

> Well, marking the directory (by whatever means) won't help there.
> Should there be a gnus-occupied-active-files as well?

That's what I was trying to say.  I think I didn't express myself
clearly enough.

kai
-- 
Lisp is kinda like tpircstsoP

Re: Protect against servers stepping on each other's toes

[Matt Armstrong]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> The trouble is with people who do this:
> 
> (add-to-list 'gnus-secondary-select-methods '(nnml ""))
> (add-to-list 'gnus-secondary-select-methods '(nnfolder ""))

And I just spent hours discovering that this happens by default if you
have:

    (setq gnus-select-method '(nnml ""))

And then "B c" a message into an nnfolder:FOO directory.  They both do
stuff in ~/Mail.  I.e. FOO gets added to ~/Mail/active.

This would ordinarily not be a problem, but it seems that something
goes through ~/Mail/active and creates nnml groups for everything in
it.  So I end up with both a "FOO" and a "nnfolder:FOO" in my
.newsrc.eld.  Insanity follows.

I've not yet verified if this happens only because I have '(nnml "")
as my primary select method.

This is particularly bad because Gnus magically creates the
"nnfolder:" method for me, using what I assume are the defaults.



> Maybe some crooked mind does something like this:
> 
> (add-to-list 'gnus-secondary-select-methods 
>              '(nnfolder "a"
>                 (nnfolder-directory "/tmp/a")
>                 (nnfolder-active-file "/tmp/active")))
> (add-to-list 'gnus-secondary-select-methods 
>              '(nnfolder "b"
>                 (nnfolder-directory "/tmp/b")
>                 (nnfolder-active-file "/tmp/active")))
> 
> Note how they use different directories but the same active file.
> 
> But you're right, it seems that it is sufficient to set a variable
> value, such as gnus-occupied-directories or something like this.

Yes, gnus-occupied-directories would catch unintended problems like I
describe above.  Were it implemented now, I'd be asleep this very
moment.  :-)


-- 
matt

Re: Protect against servers stepping on each other's toes

[Kai Großjohann]

Matt Armstrong <matt@lickey.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
>> The trouble is with people who do this:
>> 
>> (add-to-list 'gnus-secondary-select-methods '(nnml ""))
>> (add-to-list 'gnus-secondary-select-methods '(nnfolder ""))
> 
> And I just spent hours discovering that this happens by default if you
> have:
> 
>     (setq gnus-select-method '(nnml ""))
> 
> And then "B c" a message into an nnfolder:FOO directory.  They both do
> stuff in ~/Mail.  I.e. FOO gets added to ~/Mail/active.

Yes, this is quite bad, for nnml, nnfolder and nnmh all want to use
the same files and directories.  You were right that `B m' to a
non-existing server magically creates it, using the default settings.

But it seems indeed that a variable gnus-occupied-files would be just
right.  It could contain directory names and file names.  And when
Gnus tries to create another server using one of these, a warning or
error message can be issued.

kai
-- 
Lisp is kinda like tpircstsoP

Re: Protect against servers stepping on each other's toes

[Matt Armstrong]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Yes, this is quite bad, for nnml, nnfolder and nnmh all want to use
> the same files and directories.  You were right that `B m' to a
> non-existing server magically creates it, using the default
> settings.
> 
> But it seems indeed that a variable gnus-occupied-files would be
> just right.  It could contain directory names and file names.  And
> when Gnus tries to create another server using one of these, a
> warning or error message can be issued.

I agree completely.  Even a fairly cryptic error message would be
better than the current behavior.  :-)


-- 
matt

Re: Protect against servers stepping on each other's toes

[Kai Großjohann]

Any takers?
kai
-- 
Lisp is kinda like tpircstsoP

Re: Protect against servers stepping on each other's toes

[Paul Jarc]

Matt Armstrong <matt@lickey.com> wrote:
> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>> The trouble is with people who do this:
>> 
>> (add-to-list 'gnus-secondary-select-methods '(nnml ""))
>> (add-to-list 'gnus-secondary-select-methods '(nnfolder ""))
...
> I've not yet verified if this happens only because I have '(nnml "")
> as my primary select method.

It happens because you have '(nnml "") as a select method.  It doesn't
matter whether it's primary, secondary, or foreign.


paul

Re: Protect against servers stepping on each other's toes

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> Matt Armstrong <matt@lickey.com> wrote:
>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>>> The trouble is with people who do this:
>>> 
>>> (add-to-list 'gnus-secondary-select-methods '(nnml ""))
>>> (add-to-list 'gnus-secondary-select-methods '(nnfolder ""))
> ...
>> I've not yet verified if this happens only because I have '(nnml "")
>> as my primary select method.
> 
> It happens because you have '(nnml "") as a select method.  It
> doesn't matter whether it's primary, secondary, or foreign.

This is particularly dangerous because operations such as "B c" can
cause select methods to magically appear.  I wonder if this is wise.

I also notice that if I gnus-server-regenerate-server my "nnml:"
server, it gets confused by the Mail/archive directory (which is an
nnfolder: thing by default, and is placed in Mail/ by default).

Have all these issues always been present?  I think not.  Hasn't Gnus
historically only relied on .newsrc.eld for group info, and only
recently looked at files on disk to try to find groups for the
backends?


-- 
matt

Re: Protect against servers stepping on each other's toes

[Kai Großjohann]

Matt Armstrong <matt@lickey.com> writes:

> This is particularly dangerous because operations such as "B c" can
> cause select methods to magically appear.  I wonder if this is wise.

Same for me.  This has been discussed before.

> I also notice that if I gnus-server-regenerate-server my "nnml:"
> server, it gets confused by the Mail/archive directory (which is an
> nnfolder: thing by default, and is placed in Mail/ by default).

Right.  I wish somebody found the time to implement the warnings that
we have discussed a few days ago.

> Have all these issues always been present?  I think not.  Hasn't
> Gnus historically only relied on .newsrc.eld for group info, and
> only recently looked at files on disk to try to find groups for the
> backends?

Yes, these issues have been present since day one.  Astonishing that
there is now such a rush of bug reports about just this, right?  But
it seems that's just the way it is.  It was like that for me with
Tramp, too: for weeks, nothing, and then the same bug report N times.

kai
-- 
Lisp is kinda like tpircstsoP

Re: Protect against servers stepping on each other's toes

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
>> If I have two servers using the same directory, there is potential for
>> the two stepping on each other's toes.
> 
> I think much of this could be prevented by using different defaults
> for different backends, or (like nnmaildir does) by requiring the user
> to explicitly specify a directory in the server parameters.

I agree, this would be a simpler solution.  I'm using ~/Mail/nnml/,
~/Mail/nnml+foo/, ~/Mail/nnfolder+bar/ etc personally, but I guess it
would be difficult to make something like that the default now as it
would force people to reorganize their mail.