From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA1AFC433F5 for ; Fri, 3 Sep 2021 04:19:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2510D60724 for ; Fri, 3 Sep 2021 04:19:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2510D60724 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B402D8D0002; Fri, 3 Sep 2021 00:19:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF01C8D0001; Fri, 3 Sep 2021 00:19:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DEB68D0002; Fri, 3 Sep 2021 00:19:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 8EEBE8D0001 for ; Fri, 3 Sep 2021 00:19:13 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 41C8E2CBCB for ; Fri, 3 Sep 2021 04:19:13 +0000 (UTC) X-FDA: 78544957386.07.714C101 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id B96EF504F7EE for ; Fri, 3 Sep 2021 04:19:12 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2576ED6E; Thu, 2 Sep 2021 21:19:12 -0700 (PDT) Received: from [10.163.72.65] (unknown [10.163.72.65]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0D9943F5A1; Thu, 2 Sep 2021 21:19:08 -0700 (PDT) Subject: Re: [FIX PATCH 2/2] mm/page_alloc: Use accumulated load when building node fallback list To: Bharata B Rao , linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, kamezawa.hiroyu@jp.fujitsu.com, lee.schermerhorn@hp.com, mgorman@suse.de, Krupa.Ramakrishnan@amd.com, Sadagopan.Srinivasan@amd.com References: <20210830121603.1081-1-bharata@amd.com> <20210830121603.1081-3-bharata@amd.com> From: Anshuman Khandual Message-ID: Date: Fri, 3 Sep 2021 09:50:09 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20210830121603.1081-3-bharata@amd.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B96EF504F7EE X-Stat-Signature: 8s17hpryqgrctmikmm6yxuyits8zjt1h X-HE-Tag: 1630642752-460422 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 8/30/21 5:46 PM, Bharata B Rao wrote: > From: Krupa Ramakrishnan > > In build_zonelists(), when the fallback list is built for the nodes, > the node load gets reinitialized during each iteration. This results > in nodes with same distances occupying the same slot in different > node fallback lists rather than appearing in the intended round- > robin manner. This results in one node getting picked for allocation > more compared to other nodes with the same distance. > > As an example, consider a 4 node system with the following distance > matrix. > > Node 0 1 2 3 > ---------------- > 0 10 12 32 32 > 1 12 10 32 32 > 2 32 32 10 12 > 3 32 32 12 10 > > For this case, the node fallback list gets built like this: > > Node Fallback list > --------------------- > 0 0 1 2 3 > 1 1 0 3 2 > 2 2 3 0 1 > 3 3 2 0 1 <-- Unexpected fallback order > > In the fallback list for nodes 2 and 3, the nodes 0 and 1 > appear in the same order which results in more allocations > getting satisfied from node 0 compared to node 1. > > The effect of this on remote memory bandwidth as seen by stream > benchmark is shown below: > > Case 1: Bandwidth from cores on nodes 2 & 3 to memory on nodes 0 & 1 > (numactl -m 0,1 ./stream_lowOverhead ... --cores ) > Case 2: Bandwidth from cores on nodes 0 & 1 to memory on nodes 2 & 3 > (numactl -m 2,3 ./stream_lowOverhead ... --cores ) > > ---------------------------------------- > BANDWIDTH (MB/s) > TEST Case 1 Case 2 > ---------------------------------------- > COPY 57479.6 110791.8 > SCALE 55372.9 105685.9 > ADD 50460.6 96734.2 > TRIADD 50397.6 97119.1 > ---------------------------------------- > > The bandwidth drop in Case 1 occurs because most of the allocations > get satisfied by node 0 as it appears first in the fallback order > for both nodes 2 and 3. > > This can be fixed by accumulating the node load in build_zonelists() > rather than reinitializing it during each iteration. With this the > nodes with the same distance rightly get assigned in the round robin > manner. In fact this was how it was originally until the > commit f0c0b2b808f2 ("change zonelist order: zonelist order selection > logic") dropped the load accumulation and resorted to initializing > the load during each iteration. While zonelist ordering was removed by > commit c9bff3eebc09 ("mm, page_alloc: rip out ZONELIST_ORDER_ZONE"), > the change to the node load accumulation in build_zonelists() remained. > So essentially this patch reverts back to the accumulated node load > logic. > > After this fix, the fallback order gets built like this: > > Node Fallback list > ------------------ > 0 0 1 2 3 > 1 1 0 3 2 > 2 2 3 0 1 > 3 3 2 1 0 <-- Note the change here > > The bandwidth in Case 1 improves and matches Case 2 as shown below. > > ---------------------------------------- > BANDWIDTH (MB/s) > TEST Case 1 Case 2 > ---------------------------------------- > COPY 110438.9 110107.2 > SCALE 105930.5 105817.5 > ADD 97005.1 96159.8 > TRIADD 97441.5 96757.1 > ---------------------------------------- > > The correctness of the fallback list generation has been verified > for the above node configuration where the node 3 starts as > memory-less node and comes up online only during memory hotplug. > > [bharata@amd.com: Added changelog, review, test validation] > > Fixes: f0c0b2b808f2 ("change zonelist order: zonelist order selection > logic") > Signed-off-by: Krupa Ramakrishnan > Co-developed-by: Sadagopan Srinivasan > Signed-off-by: Sadagopan Srinivasan > Signed-off-by: Bharata B Rao > --- > mm/page_alloc.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 22f7ad6ec11c..47f4d160971e 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -6268,7 +6268,7 @@ static void build_zonelists(pg_data_t *pgdat) > */ > if (node_distance(local_node, node) != > node_distance(local_node, prev_node)) > - node_load[node] = load; > + node_load[node] += load; > > node_order[nr_nodes++] = node; > prev_node = node; > Reviewed-by: Anshuman Khandual