From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qa0-f53.google.com (mail-qa0-f53.google.com [209.85.216.53]) by kanga.kvack.org (Postfix) with ESMTP id ECF3F6B0036 for ; Tue, 17 Dec 2013 11:54:34 -0500 (EST) Received: by mail-qa0-f53.google.com with SMTP id j5so2801915qaq.19 for ; Tue, 17 Dec 2013 08:54:34 -0800 (PST) Received: from mail-qe0-f47.google.com (mail-qe0-f47.google.com [209.85.128.47]) by mx.google.com with ESMTPS id f1si15000774qar.20.2013.12.17.08.54.31 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 17 Dec 2013 08:54:32 -0800 (PST) Received: by mail-qe0-f47.google.com with SMTP id t7so5389449qeb.20 for ; Tue, 17 Dec 2013 08:54:31 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20131217160455.GG18680@sgi.com> References: <20131212180037.GA134240@sgi.com> <20131213214437.6fdbf7f2.akpm@linux-foundation.org> <20131216171214.GA15663@sgi.com> <20131217160455.GG18680@sgi.com> From: Andy Lutomirski Date: Tue, 17 Dec 2013 08:54:10 -0800 Message-ID: Subject: Re: [RFC PATCH 0/3] Change how we determine when to hand out THPs Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Alex Thorlton Cc: Andrew Morton , "linux-mm@kvack.org" , "Kirill A. Shutemov" , Benjamin Herrenschmidt , Rik van Riel , Wanpeng Li , Mel Gorman , Michel Lespinasse , Benjamin LaHaise , Oleg Nesterov , "Eric W. Biederman" , Al Viro , David Rientjes , Zhang Yanfei , Peter Zijlstra , Johannes Weiner , Michal Hocko , Jiang Liu , Cody P Schafer , Glauber Costa , Kamezawa Hiroyuki , Naoya Horiguchi , "linux-kernel@vger.kernel.org" , Andrea Arcangeli On Tue, Dec 17, 2013 at 8:04 AM, Alex Thorlton wrote: > On Mon, Dec 16, 2013 at 05:43:40PM -0800, Andy Lutomirski wrote: >> On Mon, Dec 16, 2013 at 9:12 AM, Alex Thorlton wrote: >> >> Please cc Andrea on this. >> > >> > I'm going to clean up a few small things for a v2 pretty soon, I'll be >> > sure to cc Andrea there. >> > >> >> > My proposed solution to the problem is to allow users to set a >> >> > threshold at which THPs will be handed out. The idea here is that, when >> >> > a user faults in a page in an area where they would usually be handed a >> >> > THP, we pull 512 pages off the free list, as we would with a regular >> >> > THP, but we only fault in single pages from that chunk, until the user >> >> > has faulted in enough pages to pass the threshold we've set. Once they >> >> > pass the threshold, we do the necessary work to turn our 512 page chunk >> >> > into a proper THP. As it stands now, if the user tries to fault in >> >> > pages from different nodes, we completely give up on ever turning a >> >> > particular chunk into a THP, and just fault in the 4K pages as they're >> >> > requested. We may want to make this tunable in the future (i.e. allow >> >> > them to fault in from only 2 different nodes). >> >> >> >> OK. But all 512 pages reside on the same node, yes? Whereas with thp >> >> disabled those 512 pages would have resided closer to the CPUs which >> >> instantiated them. >> > >> > As it stands right now, yes, since we're pulling a 512 page contiguous >> > chunk off the free list, everything from that chunk will reside on the >> > same node, but as I (stupidly) forgot to mention in my original e-mail, >> > one piece I have yet to add is the functionality to put the remaining >> > unfaulted pages from our chunk *back* on the free list after we give up >> > on handing out a THP. Once this is in there, things will behave more >> > like they do when THP is turned completely off, i.e. pages will get >> > faulted in closer to the CPU that first referenced them once we give up >> > on handing out the THP. >> >> This sounds like it's almost the worst possible behavior wrt avoiding >> memory fragmentation. If userspace mmaps a very large region and then >> starts accessing it randomly, it will allocate a bunch of contiguous >> 512-page regions, claim one page from each, and return the other 511 >> pages to the free list. Memory is now maximally fragmented from the >> point of view of future THP allocations. > > Maybe I'm missing the point here to some degree, but the way I think > about this is that if we trigger the behavior to return the pages to the > free list, we don't *want* future THP allocations in that range of > memory for the current process anyways. So, having the memory be > fragmented from the point of view of future THP allocations isn't an > issue. > Except that you're causing a problem for the whole system because one process is triggering the "hugepages aren't helpful" heuristic. --Andy > - Alex -- Andy Lutomirski AMA Capital Management, LLC -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org