From mboxrd@z Thu Jan 1 00:00:00 1970 From: bakul@bitblocks.com (Bakul Shah) Date: Fri, 01 Dec 2017 15:42:15 -0800 Subject: [TUHS] signals and blocked in I/O In-Reply-To: Your message of "Fri, 01 Dec 2017 15:09:34 -0800." <20171201230934.GA24335@mcvoy.com> References: <20171201154448.GL3924@mcvoy.com> <20171201161810.GM3924@mcvoy.com> <20171201172603.GO3924@mcvoy.com> <20171201223859.GX3924@mcvoy.com> <20171201230302.0DC351FA41@orac.inputplus.co.uk> <20171201230934.GA24335@mcvoy.com> Message-ID: <20171201234230.F33D4156E523@mail.bitblocks.com> On Fri, 01 Dec 2017 15:09:34 -0800 Larry McVoy wrote: Larry McVoy writes: > On Fri, Dec 01, 2017 at 11:03:02PM +0000, Ralph Corderoy wrote: > > Hi Larry, > > > > > > So OOM code kills a (random) process in hopes of freeing up some > > > > pages but if this process is stuck in diskIO, nothing can be freed > > > > and everything grinds to a halt. > > > > > > Yep, exactly. > > > > Is that because the pages have been dirty for so long they've reached > > the VM-writeback timeout even though there's no pressure to use them for > > something else? Or has that been lengthened because you don't fear > > power loss wiping volatile RAM? > > I'm tinkering with the pageout daemon so I'm trying to apply memory > pressure. I have 10 25GB processes (25GB malloced) and the processes just > walk the memory over and over. This is on a 256GB main memory machine > (2 socket haswell, 28 cpus, 28 1TB SSDs, on loan from Netflix). How many times do processes walk their memory before this condition occurs? So what may be happening is that a process references a page, it page faults, the kernel finds its phys page has been paged out, so it looks for a free page and once a free page is found, the process will block on page in. Or if there is no free page, it has to wait until some other dirty page is paged out (but this would be a different wait queue). As more and more processes do this, the system runs out of all free pages. Can you find out how many processes are waiting under what conditions, how long they wait and how these queue lengths are changing over time? You can use a ring buffer to capture last 2^N measurements and dump them in the debugger when everything grinds to a halt. > It's the old "10 pounds of shit in a 5 pound bag" problem, same old stuff, > just a bigger bag. > > The problem is that OOM can't kill the processes that are the problem, > they are stuck in disk wait. That's why I started asking why can't you > kill a process that's in the middle of I/O. The OS equivalent of RED (random early drop) would be if a process kills itself. e.g. when some critical metric crosses a highwater mark. Another option would be to return with an EFAULT and the process can either kill itself or free up the page or something. [I have used EFAULT to dyanmically allocate *more* pages but no reason why the same can be used to free up memory!]