From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48F9DC48BD1 for ; Fri, 11 Jun 2021 23:23:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF0EE613EE for ; Fri, 11 Jun 2021 23:23:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF0EE613EE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4AF7C6B006C; Fri, 11 Jun 2021 19:23:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4603C6B006E; Fri, 11 Jun 2021 19:23:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 328386B0070; Fri, 11 Jun 2021 19:23:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 030C96B006C for ; Fri, 11 Jun 2021 19:23:33 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8B07018010BE8 for ; Fri, 11 Jun 2021 23:23:33 +0000 (UTC) X-FDA: 78243021906.09.9F0885B Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP id AE1519001E40 for ; Fri, 11 Jun 2021 23:23:25 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id B560A611B0; Fri, 11 Jun 2021 23:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1623453812; bh=XOEMUfV+WcQDf35lku9MLHKhHAejd6xTD1D2y888KPc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=lFLO/9Gsx859HAXaCCzSLD8Gw3QzamYRBaEPq/TiuZhWk9BD5Dzso8RXEz6k4LZoM QihxTbK4XrEAxlqeQc9I+uC62BiuMNYcEX8/POQ9svXYqKYqJQVxuB7O5W9aGGFepR 6R9igfkDOjyisiN9kHsF0+h+4Lc9AsUsj4MspD/I= Date: Fri, 11 Jun 2021 16:23:31 -0700 From: Andrew Morton To: Mel Gorman Cc: Zi Yan , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM Subject: Re: [PATCH v2] mm/page_alloc: Allow high-order pages to be stored on the per-cpu lists Message-Id: <20210611162331.272f67eabffa491fc83798b4@linux-foundation.org> In-Reply-To: <20210611135753.GC30378@techsingularity.net> References: <20210611135753.GC30378@techsingularity.net> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.31; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b="lFLO/9Gs"; dmarc=none; spf=pass (imf19.hostedemail.com: domain of akpm@linux-foundation.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org X-Rspamd-Server: rspam02 X-Stat-Signature: io15c9hzdk96g1zqw31q4yxezuze4p6j X-Rspamd-Queue-Id: AE1519001E40 X-HE-Tag: 1623453805-300275 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 11 Jun 2021 14:57:53 +0100 Mel Gorman wrote: > Changelog since v1 > o Fix boot problem on KVM with hotplug memory nodes (ziy) > o Correct PCP list lookup in bulk page allocator > > The per-cpu page allocator (PCP) only stores order-0 pages. This means > that all THP and "cheap" high-order allocations including SLUB contends > on the zone->lock. This patch extends the PCP allocator to store THP and > "cheap" high-order pages. Note that struct per_cpu_pages increases in > size to 256 bytes (4 cache lines) on x86-64. > > Note that this is not necessarily a universal performance win because of > how it is implemented. High-order pages can cause pcp->high to be exceeded > prematurely for lower-orders so for example, a large number of THP pages > being freed could release order-0 pages from the PCP lists. Hence, much > depends on the allocation/free pattern as observed by a single CPU to > determine if caching helps or hurts a particular workload. > > That said, basic performance testing passed. The following is a netperf > UDP_STREAM test which hits the relevant patches as some of the network > allocations are high-order. > > netperf-udp > 5.13.0-rc2 5.13.0-rc2 > mm-pcpburst-v3r4 mm-pcphighorder-v1r7 > Hmean send-64 261.46 ( 0.00%) 266.30 * 1.85%* > Hmean send-128 516.35 ( 0.00%) 536.78 * 3.96%* > Hmean send-256 1014.13 ( 0.00%) 1034.63 * 2.02%* > Hmean send-1024 3907.65 ( 0.00%) 4046.11 * 3.54%* > Hmean send-2048 7492.93 ( 0.00%) 7754.85 * 3.50%* > Hmean send-3312 11410.04 ( 0.00%) 11772.32 * 3.18%* > Hmean send-4096 13521.95 ( 0.00%) 13912.34 * 2.89%* > Hmean send-8192 21660.50 ( 0.00%) 22730.72 * 4.94%* > Hmean send-16384 31902.32 ( 0.00%) 32637.50 * 2.30%* > > >From a functional point of view, a patch like this is necessary to > make bulk allocation of high-order pages work with similar performance > to order-0 bulk allocations. The bulk allocator is not updated in this > series as it would have to be determined by bulk allocation users how > they want to track the order of pages allocated with the bulk allocator. > > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -198,7 +198,7 @@ extern void post_alloc_hook(struct page *page, unsigned int order, > gfp_t gfp_flags); > extern int user_min_free_kbytes; > > -extern void free_unref_page(struct page *page); > +extern void free_unref_page(struct page *page, unsigned int order); > extern void free_unref_page_list(struct list_head *list); > > extern void zone_pcp_update(struct zone *zone, int cpu_online); > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index f24f509c3ee3..8472bae567f0 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -676,10 +676,53 @@ static void bad_page(struct page *page, const char *reason) > add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); > } > > +static inline unsigned int order_to_pindex(int migratetype, int order) > +{ > + int base = order; > + > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + if (order > PAGE_ALLOC_COSTLY_ORDER) { > + VM_BUG_ON(order != pageblock_order); > + base = PAGE_ALLOC_COSTLY_ORDER + 1; > + } > +#else > + VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); > +#endif > + > + return (MIGRATE_PCPTYPES * base) + migratetype; > +} > + > +static inline int pindex_to_order(unsigned int pindex) > +{ > + int order = pindex / MIGRATE_PCPTYPES; > + > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + if (order > PAGE_ALLOC_COSTLY_ORDER) { > + order = pageblock_order; > + VM_BUG_ON(order != pageblock_order); Somebody has trust issues? > + } > +#else > + VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); > +#endif > + > + return order; > +} Do we really need all these assertions, long-term?