[9front] issues with the mailing list

[9front] issues with the mailing list

[kemal]

(ccing it to sl in case mailing list shits itself again)
i am experiencing issues with the mailing list, but it's now worse.
i have took one message after more than 10 hours, that outrageous.
is this the nature of mlmgr or is there a problem with something?
does anybody experience issues with the mails coming very late?

Re: [9front] issues with the mailing list

[hiro]

if i sample the last 20 mails i can see that approximately every few
mails it reaches a delay in the 1-hour range

On 3/25/21, kemal <kemali13@protonmail.com> wrote:
> (ccing it to sl in case mailing list shits itself again)
> i am experiencing issues with the mailing list, but it's now worse.
> i have took one message after more than 10 hours, that outrageous.
> is this the nature of mlmgr or is there a problem with something?
> does anybody experience issues with the mails coming very late?
>

Re: [9front] issues with the mailing list

[Stanley Lieber]

1ess; walk /mail/queue|wc -l                              6906

divide this number roughly in three to get the number of presently undelivered messages (to all lists; this machine serves all/only 9front/cat-v/werc lists). of those, the vast majority are transient dns lookup failures for good domains. we alleviated this somewhat already by extending the lookup timeout, but clearly something else is going on.

all of this started when I moved the server back to 9cloud from ramnode. the first thing we noticed was the default setup hammered the now non-ssd disk by launching runq every time it processed a message. this brought the whole machine to a crawl (this machine also serves the websites -- people complained). that was solved by parallelizing a limited number of runqs and launching them periodically from cron. the side effect of this is exposing all kinds of weirdness that was previously masked by the never-ending blast of runqs, which, in spite of transient failures, always eventually clears out the queue.

I'm not yet sure why the transient dns failures are happening.

sl

Re: [9front] issues with the mailing list

[Stanley Lieber]

On March 25, 2021 9:48:40 AM EDT, Stanley Lieber <sl@stanleylieber.com> wrote:
>1ess; walk /mail/queue|wc -l                              6906
>
>divide this number roughly in three to get the number of presently undelivered messages (to all lists; this machine serves all/only 9front/cat-v/werc lists). of those, the vast majority are transient dns lookup failures for good domains. we alleviated this somewhat already by extending the lookup timeout, but clearly something else is going on.
>
>all of this started when I moved the server back to 9cloud from ramnode. the first thing we noticed was the default setup hammered the now non-ssd disk by launching runq every time it processed a message. this brought the whole machine to a crawl (this machine also serves the websites -- people complained). that was solved by parallelizing a limited number of runqs and launching them periodically from cron. the side effect of this is exposing all kinds of weirdness that was previously masked by the never-ending blast of runqs, which, in spite of transient failures, always eventually clears out the queue.
>
>I'm not yet sure why the transient dns failures are happening.
>
>sl
>

the server has now been configured to run its own resolver. we'll see if this results in fewer transient lookup failures.

sl

Re: [9front] issues with the mailing list

[kemal]

On Thursday, March 25, 2021 10:42 AM, hiro <23hiro@gmail.com> wrote:

> if i sample the last 20 mails i can see that approximately every few
> mails it reaches a delay in the 1-hour range

i just received your mail in **7 hours**.

Re: [9front] issues with the mailing list

[Philip Silva]

Actually yes, in the Thread about Plan 9 Foundation some messages in between arrived with some delay. (I saw replies but not the original message) Looks to me like all messages of that thread are there now


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Am Donnerstag, 25. März 2021 11:42 schrieb hiro <23hiro@gmail.com>:

> if i sample the last 20 mails i can see that approximately every few
> mails it reaches a delay in the 1-hour range
>
> On 3/25/21, kemal kemali13@protonmail.com wrote:
>
> > (ccing it to sl in case mailing list shits itself again)
> > i am experiencing issues with the mailing list, but it's now worse.
> > i have took one message after more than 10 hours, that outrageous.
> > is this the nature of mlmgr or is there a problem with something?
> > does anybody experience issues with the mails coming very late?

Re: [9front] issues with the mailing list

[kemal]

your message got sent late but latest messages are coming much faster compared
to older messages.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, March 25, 2021 2:17 PM, Stanley Lieber <sl@stanleylieber.com> wrote:

> On March 25, 2021 9:48:40 AM EDT, Stanley Lieber sl@stanleylieber.com wrote:
>
> > 1ess; walk /mail/queue|wc -l 6906
> > divide this number roughly in three to get the number of presently undelivered messages (to all lists; this machine serves all/only 9front/cat-v/werc lists). of those, the vast majority are transient dns lookup failures for good domains. we alleviated this somewhat already by extending the lookup timeout, but clearly something else is going on.
> > all of this started when I moved the server back to 9cloud from ramnode. the first thing we noticed was the default setup hammered the now non-ssd disk by launching runq every time it processed a message. this brought the whole machine to a crawl (this machine also serves the websites -- people complained). that was solved by parallelizing a limited number of runqs and launching them periodically from cron. the side effect of this is exposing all kinds of weirdness that was previously masked by the never-ending blast of runqs, which, in spite of transient failures, always eventually clears out the queue.
> > I'm not yet sure why the transient dns failures are happening.
> > sl
>
> the server has now been configured to run its own resolver. we'll see if this results in fewer transient lookup failures.
>
> sl

Re: [9front] issues with the mailing list

[toby]

On 25/03/2021 10:34, kemal wrote:
> does anybody experience issues with the mails coming very late?

I often experience such delays, but I am not sure exactly where my 
problem lies. Some of those that I experience may be spam-filter related 
at my end; it seems that some aspect of the way that mlmgr works (I 
haven't yet looked very far into it) causes unusually high spamminesses 
in my rspamd, and particularly for some senders (some messages are even 
rejected). I keep training it, but I think I have yet to reach the ~200 
training events it requires to actually take any notice.

Toby

Re: [9front] issues with the mailing list

[Stanley Lieber]

On March 25, 2021 2:07:35 PM EDT, kemal <kemali13@protonmail.com> wrote:
>your message got sent late but latest messages are coming much faster compared
>to older messages.
>
>‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>On Thursday, March 25, 2021 2:17 PM, Stanley Lieber <sl@stanleylieber.com> wrote:
>
>> On March 25, 2021 9:48:40 AM EDT, Stanley Lieber sl@stanleylieber.com wrote:
>>
>> > 1ess; walk /mail/queue|wc -l 6906
>> > divide this number roughly in three to get the number of presently undelivered messages (to all lists; this machine serves all/only 9front/cat-v/werc lists). of those, the vast majority are transient dns lookup failures for good domains. we alleviated this somewhat already by extending the lookup timeout, but clearly something else is going on.
>> > all of this started when I moved the server back to 9cloud from ramnode. the first thing we noticed was the default setup hammered the now non-ssd disk by launching runq every time it processed a message. this brought the whole machine to a crawl (this machine also serves the websites -- people complained). that was solved by parallelizing a limited number of runqs and launching them periodically from cron. the side effect of this is exposing all kinds of weirdness that was previously masked by the never-ending blast of runqs, which, in spite of transient failures, always eventually clears out the queue.
>> > I'm not yet sure why the transient dns failures are happening.
>> > sl
>>
>> the server has now been configured to run its own resolver. we'll see if this results in fewer transient lookup failures.
>>
>> sl
>
>
>

1ess; date                                             
Thu Mar 25 15:47:40 EDT 2021                           1ess; walk /mail/queue|wc -l                              
5575