From: "Huang, Ying" <ying.huang@intel.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Mel Gorman <mgorman@suse.de>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Feng Tang <feng.tang@intel.com>,
Andrew Morton <akpm@linux-foundation.org>,
Michal Hocko <mhocko@suse.com>, Rik van Riel <riel@surriel.com>,
Mel Gorman <mgorman@techsingularity.net>,
Dave Hansen <dave.hansen@linux.intel.com>,
Yang Shi <shy828301@gmail.com>, Zi Yan <ziy@nvidia.com>,
Wei Xu <weixugc@google.com>, osalvador <osalvador@suse.de>,
Shakeel Butt <shakeelb@google.com>,
Hasan Al Maruf <hasanalmaruf@fb.com>
Subject: Re: [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system
Date: Thu, 13 Jan 2022 15:19:06 +0800 [thread overview]
Message-ID: <87sftsumqd.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <Yd79b6PptQMNzDRw@hirez.programming.kicks-ass.net> (Peter Zijlstra's message of "Wed, 12 Jan 2022 17:10:23 +0100")
[-- Attachment #1: Type: text/plain, Size: 3326 bytes --]
Hi, Peter,
Peter Zijlstra <peterz@infradead.org> writes:
> On Tue, Dec 07, 2021 at 10:27:51AM +0800, Huang Ying wrote:
>> After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory
>> for use like normal RAM"), the PMEM could be used as the
>> cost-effective volatile memory in separate NUMA nodes. In a typical
>> memory tiering system, there are CPUs, DRAM and PMEM in each physical
>> NUMA node. The CPUs and the DRAM will be put in one logical node,
>> while the PMEM will be put in another (faked) logical node.
>
> So what does a system like that actually look like, SLIT table wise, and
> how does that affect init_numa_topology_type() ?
The SLIT table is as follows,
[000h 0000 4] Signature : "SLIT" [System Locality Information Table]
[004h 0004 4] Table Length : 0000042C
[008h 0008 1] Revision : 01
[009h 0009 1] Checksum : 59
[00Ah 0010 6] Oem ID : "INTEL "
[010h 0016 8] Oem Table ID : "S2600WF "
[018h 0024 4] Oem Revision : 00000001
[01Ch 0028 4] Asl Compiler ID : "INTL"
[020h 0032 4] Asl Compiler Revision : 20091013
[024h 0036 8] Localities : 0000000000000004
[02Ch 0044 4] Locality 0 : 0A 15 11 1C
[030h 0048 4] Locality 1 : 15 0A 1C 11
[034h 0052 4] Locality 2 : 11 1C 0A 1C
[038h 0056 4] Locality 3 : 1C 11 1C 0A
The `numactl -H` output is as follows,
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
node 0 size: 64136 MB
node 0 free: 5981 MB
node 1 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 64466 MB
node 1 free: 10415 MB
node 2 cpus:
node 2 size: 253952 MB
node 2 free: 253920 MB
node 3 cpus:
node 3 size: 253952 MB
node 3 free: 253951 MB
node distances:
node 0 1 2 3
0: 10 21 17 28
1: 21 10 28 17
2: 17 28 10 28
3: 28 17 28 10
init_numa_topology_type() set sched_numa_topology_type to NUMA_DIRECT.
The node 0 and node 1 are onlined during boot. While the PMEM node,
that is, node 2 and node 3 are onlined later. As in the following dmesg
snippet.
[ 2.252573][ T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[ 2.259224][ T0] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff]
[ 2.266139][ T0] ACPI: SRAT: Node 2 PXM 2 [mem 0x1080000000-0x4f7fffffff] non-volatile
[ 2.274267][ T0] ACPI: SRAT: Node 1 PXM 1 [mem 0x4f80000000-0x5f7fffffff]
[ 2.281271][ T0] ACPI: SRAT: Node 3 PXM 3 [mem 0x5f80000000-0x9e7fffffff] non-volatile
[ 2.289403][ T0] NUMA: Initialized distance table, cnt=4
[ 2.294934][ T0] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff]
[ 2.306266][ T0] NODE_DATA(0) allocated [mem 0x107ffd5000-0x107fffffff]
[ 2.313115][ T0] NODE_DATA(1) allocated [mem 0x5f7ffd0000-0x5f7fffafff]
[ 5.391151][ T1] smp: Brought up 2 nodes, 96 CPUs
Full dmesg is attached.
Best Regards,
Huang, Ying
[-- Attachment #2: dmesg.xz --]
[-- Type: application/x-xz, Size: 27120 bytes --]
next prev parent reply other threads:[~2022-01-13 7:19 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-07 2:27 Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 1/6] NUMA Balancing: add page promotion counter Huang Ying
2021-12-07 6:05 ` Hasan Al Maruf
2021-12-08 2:16 ` Huang, Ying
2021-12-17 7:25 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 2/6] NUMA balancing: optimize page placement for memory tiering system Huang Ying
2021-12-07 6:36 ` Hasan Al Maruf
2021-12-08 3:16 ` Huang, Ying
2021-12-17 7:35 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 3/6] memory tiering: skip to scan fast memory Huang Ying
2021-12-17 7:41 ` Baolin Wang
2021-12-07 2:27 ` [PATCH -V10 RESEND 4/6] memory tiering: hot page selection with hint page fault latency Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 5/6] memory tiering: rate limit NUMA migration throughput Huang Ying
2021-12-07 2:27 ` [PATCH -V10 RESEND 6/6] memory tiering: adjust hot threshold automatically Huang Ying
2022-01-12 16:10 ` [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system Peter Zijlstra
2022-01-13 7:19 ` Huang, Ying [this message]
2022-01-13 9:49 ` Peter Zijlstra
[not found] ` <87o84fu9f3.fsf@yhuang6-desk2.ccr.corp.intel.com>
2022-01-13 13:00 ` Peter Zijlstra
2022-01-13 13:13 ` Huang, Ying
2022-01-13 14:24 ` Huang, Ying
2022-01-14 5:24 ` Huang, Ying
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sftsumqd.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@linux.intel.com \
--cc=feng.tang@intel.com \
--cc=hasanalmaruf@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=osalvador@suse.de \
--cc=peterz@infradead.org \
--cc=riel@surriel.com \
--cc=shakeelb@google.com \
--cc=shy828301@gmail.com \
--cc=weixugc@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox