From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 061D7C433EF for ; Thu, 13 Jan 2022 14:24:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41A2F6B0072; Thu, 13 Jan 2022 09:24:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A24D6B0073; Thu, 13 Jan 2022 09:24:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21B236B0074; Thu, 13 Jan 2022 09:24:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0208.hostedemail.com [216.40.44.208]) by kanga.kvack.org (Postfix) with ESMTP id 0C7246B0072 for ; Thu, 13 Jan 2022 09:24:53 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A101E181CA080 for ; Thu, 13 Jan 2022 14:24:52 +0000 (UTC) X-FDA: 79025485224.18.6B92742 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf02.hostedemail.com (Postfix) with ESMTP id 7ACD280004 for ; Thu, 13 Jan 2022 14:24:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1642083891; x=1673619891; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=2lwgn5Ak8FrAWeuWC+qm8/A59fs8zZe6onPZCttDNOQ=; b=BhvtrM9oRLI7JcaIn4ovDsWofTWj6i5d6KggsIiOT7WDY0G8SHQLxFNH 7bltwFEx1p3Ev1OEB2Z0Zq613JRmmB6iUcLEh8/XJaLsUmHIp5MXhKn9+ K5uBp9B6HAWPSR/+/c8yFhY/hGDJTqF04VAZRFKXDVU4Mp+4YhpdA43ND rxI/eO69kZ40Ulb5PovjxSgWpGPyHP7YJb2t9ecfTCq/PEgLgKyl1Dqv9 Cx2usMIY22ljjg5XbRNUX7HDGUQFwm5uTioRb4hMvpErwCksdB5tyorSN 1XcpeLfn08OHTmhlkdMSvD+gvVknU0Yij6AKNpbch4AvNvLcPcKQk36LQ Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10225"; a="241576793" X-IronPort-AV: E=Sophos;i="5.88,286,1635231600"; d="scan'208";a="241576793" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jan 2022 06:24:49 -0800 X-IronPort-AV: E=Sophos;i="5.88,286,1635231600"; d="scan'208";a="529674005" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.239.13.11]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jan 2022 06:24:45 -0800 From: "Huang, Ying" To: Peter Zijlstra Cc: Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Feng Tang , Andrew Morton , Michal Hocko , Rik van Riel , Mel Gorman , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt , Hasan Al Maruf Subject: Re: [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system References: <20211207022757.2523359-1-ying.huang@intel.com> <87sftsumqd.fsf@yhuang6-desk2.ccr.corp.intel.com> <87o84fu9f3.fsf@yhuang6-desk2.ccr.corp.intel.com> Date: Thu, 13 Jan 2022 22:24:43 +0800 In-Reply-To: (Peter Zijlstra's message of "Thu, 13 Jan 2022 14:00:39 +0100") Message-ID: <871r1bu310.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7ACD280004 X-Stat-Signature: 9zgud8o3tro1rdjg3keiqumox9gkuqqo Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=BhvtrM9o; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf02.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=ying.huang@intel.com X-HE-Tag: 1642083891-506830 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Peter Zijlstra writes: > On Thu, Jan 13, 2022 at 08:06:40PM +0800, Huang, Ying wrote: >> Peter Zijlstra writes: >> > On Thu, Jan 13, 2022 at 03:19:06PM +0800, Huang, Ying wrote: >> >> Peter Zijlstra writes: >> >> > On Tue, Dec 07, 2021 at 10:27:51AM +0800, Huang Ying wrote: > >> >> >> After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory >> >> >> for use like normal RAM"), the PMEM could be used as the >> >> >> cost-effective volatile memory in separate NUMA nodes. In a typical >> >> >> memory tiering system, there are CPUs, DRAM and PMEM in each physical >> >> >> NUMA node. The CPUs and the DRAM will be put in one logical node, >> >> >> while the PMEM will be put in another (faked) logical node. >> >> > >> >> > So what does a system like that actually look like, SLIT table wise, and >> >> > how does that affect init_numa_topology_type() ? >> >> >> >> The SLIT table is as follows, > > > >> >> node distances: >> >> node 0 1 2 3 >> >> 0: 10 21 17 28 >> >> 1: 21 10 28 17 >> >> 2: 17 28 10 28 >> >> 3: 28 17 28 10 >> >> >> >> init_numa_topology_type() set sched_numa_topology_type to NUMA_DIRECT. >> >> >> >> The node 0 and node 1 are onlined during boot. While the PMEM node, >> >> that is, node 2 and node 3 are onlined later. As in the following dmesg >> >> snippet. >> > >> > But how? sched_init_numa() scans the *whole* SLIT table to determine >> > nr_levels / sched_domains_numa_levels, even offline nodes. Therefore it >> > should find 4 distinct distance values and end up not selecting >> > NUMA_DIRECT. >> > >> > Similarly for the other types it uses for_each_online_node(), which >> > would include the pmem nodes once they've been onlined, but I'm thinking >> > we explicitly want to skip CPU-less nodes in that iteration. >> >> I used the debug patch as below, and get the log in dmesg as follows, >> >> [ 5.394577][ T1] sched_numa_topology_type: 0, levels: 4, max_distance: 28 >> >> I found that I forget another caller of init_numa_topology_type() run >> during hotplug. I will add another printk() to show it. Sorry about >> that. > > Can you try with this on? > > I'm suspecting there's a problem with init_numa_topology_type(); it will > never find the max distance due to the _online_ clause in the iteration, > since you said the pmem nodes are not online yet. > > --- > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > index d201a7052a29..53ab9c63c185 100644 > --- a/kernel/sched/topology.c > +++ b/kernel/sched/topology.c > @@ -1756,6 +1756,8 @@ static void init_numa_topology_type(void) > return; > } > } > + > + WARN(1, "no NUMA type determined"); > } Hi, Peter, I have run the test, the warning is triggered in the dmesg as follows. I will continue to debug hotplug tomorrow. [ 5.400923][ T1] ------------[ cut here ]------------ [ 5.401917][ T1] no NUMA type determined [ 5.401921][ T1] WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:1760 init_numa_topology_type+0x199/0x1c0 [ 5.403918][ T1] Modules linked in: [ 5.404917][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.16.0-rc8-00053-gbe30433a13c0 #1 [ 5.405917][ T1] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5C620.86B.0D.01.0286.011120190816 01/11/2019 [ 5.406917][ T1] RIP: 0010:init_numa_topology_type+0x199/0x1c0 [ 5.407917][ T1] Code: de 82 41 89 dc e8 07 4f 4e 00 3d 00 04 00 00 44 0f 4e e0 3d ff 03 00 00 0f 8e ca fe ff ff 48 c7 c7 a7 88 55 82 e8 0c e5 b3 00 <0f> 0b e9 74 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f [ 5.408917][ T1] RSP: 0000:ffffc900000b7e00 EFLAGS: 00010286 [ 5.409917][ T1] RAX: 0000000000000000 RBX: 0000000000000400 RCX: c0000000ffff7fff [ 5.410917][ T1] RDX: ffffc900000b7c28 RSI: 00000000ffff7fff RDI: 0000000000000000 [ 5.411917][ T1] RBP: 000000000000001c R08: 0000000000000000 R09: ffffc900000b7c20 [ 5.412917][ T1] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000400 [ 5.413917][ T1] R13: 0000000000000400 R14: 0000000000000400 R15: 000000000000000c [ 5.414917][ T1] FS: 0000000000000000(0000) GS:ffff88903f600000(0000) knlGS:0000000000000000 [ 5.415917][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.416917][ T1] CR2: ffff88df7fc01000 CR3: 0000005f7ec0a001 CR4: 00000000007706f0 [ 5.417917][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5.418917][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5.419917][ T1] PKRU: 55555554 [ 5.420917][ T1] Call Trace: [ 5.421919][ T1] [ 5.422919][ T1] sched_init_numa+0x4a7/0x5c0 [ 5.423918][ T1] sched_init_smp+0x18/0x79 [ 5.424918][ T1] kernel_init_freeable+0x136/0x276 [ 5.425918][ T1] ? rest_init+0x100/0x100 [ 5.426917][ T1] kernel_init+0x16/0x140 [ 5.427917][ T1] ret_from_fork+0x1f/0x30 [ 5.428918][ T1] [ 5.429919][ T1] ---[ end trace aa5563c4363f1ba3 ]--- [ 5.430917][ T1] sched_numa_topology_type: 0, levels: 4, max_distance: 28 Best Regards, Huang, Ying