[9fans] upas locking

[9fans] upas locking

[Francisco J Ballesteros]

Hi,

we found recently that we had a problem with /mail/tmp/L.mbox
used by upasfs to lock.

Looking into, we found that in upas/common/libsys.c the
function generating the lock file name returns always "L.mbox"
but not .../L.<mbox file name> as it seems it should.

Also, the function doing the actual lock hides the error string.

I copied our changed file to /n/sources/contrib/nemo/libsys.c

Didn't create a patch yet because I'm not sure it's the right
change for this problem.

Re: [9fans] upas locking

[erik quanstrom]

> Looking into, we found that in upas/common/libsys.c the
> function generating the lock file name returns always "L.mbox"
> but not .../L.<mbox file name> as it seems it should.

i think the comment above the function is important
"we use one lock per directory".  it's been a long time since
i worked on nupas, so i don't recall all the details, but
iirc, imap4d relies on having one lock per directory.

of course, nupas takes pressure off the directory lock
by using mdir format, which doesn't require a lock.  the
indicies are lock files themselves.

under what circumstaces did you find a problem?

- erik

Re: [9fans] upas locking

[Francisco J Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

Yes, I noticed the comment, but I thought it was wrong.

The problem was that we had quite a number of upas/fs's and scanmails
running and it was because one of them had a problem and kept a
/mail/tmp/L.mbox file locked. All other ones kept waiting for that
(although all of them were using different "mboxes" created by the
piped script at /mail/tmp/$pid.blah.blah

Making this change made problems go away, and I didn't notice any
problem in imap so far. I guess I'll have to reread the imap source to see
if it really requires one lock per dir instead of one per mbox, but even so,
I think imap should use a different lock for that, it's upas/fs the one I'm
talking about.

thanks for the hint

On Apr 18, 2013, at 4:04 PM, erik quanstrom <quanstro@quanstro.net> wrote:

> i think the comment above the function is important
> "we use one lock per directory".  it's been a long time since
> i worked on nupas, so i don't recall all the details, but
> iirc, imap4d relies on having one lock per directory.


[-- Attachment #2: Type: text/html, Size: 4581 bytes --]

Re: [9fans] upas locking

[erik quanstrom]

> The problem was that we had quite a number of upas/fs's and scanmails
> running and it was because one of them had a problem and kept a
> /mail/tmp/L.mbox file locked. All other ones kept waiting for that
> (although all of them were using different "mboxes" created by the
> piped script at /mail/tmp/$pid.blah.blah
>

i'm confused.  doesn't this just make the problem smaller rather
than fixing it?  i would think if scanmail has the same issue, then
mail will still go undelivered, right?

we don't run scanmail.  since scanmail relies on building rules, it
needs constant maintence.  and i didn't want to be on the hook
for that.

instead, i use mechanical defensive measures in smtpd and
validatesender.  the most effective measure is enforcing the
rfc2821 rule that the ehlo string must look up in dns.
to appease imap4 clients, there's a flag to force auth
for clients ehlo'ing incorrectly.

spamhaus does a good job cleaning up what remains.

- erik

http://www.quanstro.net/magic/man2html/8/smtp

(yes, that's ugly!)

Re: [9fans] upas locking

[Fco. J. Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

Yes, perhaps we should stop using that.
Nevertheless, the problem I was trying to address is that
everyone is sharing /mail/tmp (see the scripts in /mail/lib,
although I'm sure you know them from memory :) ), and thus
if one has one problem, it can lock everyone else using /mail/tmp.

I'll take a look now to the countermeasures you suggest.
I think we are applying a few other measures, including
validatesender, but I'll take a look.

[-- Attachment #2: Type: message/rfc822, Size: 5896 bytes --]

From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Subject: Re: [9fans] upas locking
Date: Thu, 18 Apr 2013 10:42:56 -0400
Message-ID: <2e14c2fbb566df6d129d01cc4d835243@kw.quanstro.net>

> The problem was that we had quite a number of upas/fs's and scanmails
> running and it was because one of them had a problem and kept a
> /mail/tmp/L.mbox file locked. All other ones kept waiting for that
> (although all of them were using different "mboxes" created by the
> piped script at /mail/tmp/$pid.blah.blah
>

i'm confused.  doesn't this just make the problem smaller rather
than fixing it?  i would think if scanmail has the same issue, then
mail will still go undelivered, right?

we don't run scanmail.  since scanmail relies on building rules, it
needs constant maintence.  and i didn't want to be on the hook
for that.

instead, i use mechanical defensive measures in smtpd and
validatesender.  the most effective measure is enforcing the
rfc2821 rule that the ehlo string must look up in dns.
to appease imap4 clients, there's a flag to force auth
for clients ehlo'ing incorrectly.

spamhaus does a good job cleaning up what remains.

- erik

http://www.quanstro.net/magic/man2html/8/smtp

(yes, that's ugly!)

Re: [9fans] upas locking

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 190 bytes --]

On 18 April 2013 15:51, Fco. J. Ballesteros <nemo@lsub.org> wrote:

> if one has one problem, it can lock everyone else using /mail/tmp.


I use a private ramfs, which is also faster.

[-- Attachment #2: Type: text/html, Size: 507 bytes --]

Re: [9fans] upas locking

[lucio]

> On 18 April 2013 15:51, Fco. J. Ballesteros <nemo@lsub.org> wrote:
>
>> if one has one problem, it can lock everyone else using /mail/tmp.
>
>
> I use a private ramfs, which is also faster.

Obvious, when one points it out!

++L

Re: [9fans] upas locking

[Francisco J Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 431 bytes --]

yes, except that the whole thing
got worse because one of our servers
had a lot of memory used due to
those and we put back the bind for 
tmp. 

On Apr 18, 2013, at 5:33 PM, Charles Forsyth <charles.forsyth@gmail.com> wrote:

> 
> On 18 April 2013 15:51, Fco. J. Ballesteros <nemo@lsub.org> wrote:
>> if one has one problem, it can lock everyone else using /mail/tmp.
> 
> I use a private ramfs, which is also faster.

[-- Attachment #2: Type: text/html, Size: 1034 bytes --]

Re: [9fans] upas locking

[erik quanstrom]

On Thu Apr 18 12:10:28 EDT 2013, nemo@lsub.org wrote:

> yes, except that the whole thing
> got worse because one of our servers
> had a lot of memory used due to
> those and we put back the bind for
> tmp.

ramfs -l 10m

(the 9atom version of ramfs does proper accounting to
make this work.)

also, if you're using anything like mdir, you can deliver
the mail and delete it automaticly later.

- erik