On 1/18/2025 4:09 AM, Zi Yan wrote: > On 17 Jan 2025, at 0:00, Hyeonggon Yoo wrote: > >> On 1/6/2025 9:06 PM, Bruno Faccini wrote: >>> Current fake-numa implementation prevents new Numa >>> nodes to be later hot-plugged by drivers. >>> A common symptom of this limitation is the >>> "node was absent from the node_possible_map" >>> message by associated warning in mm/memory_hotplug.c: >>> add_memory_resource(). >>> This comes from the lack of remapping in both >>> pxm_to_node_map[] and node_to_pxm_map[] tables >>> to take fake-numa nodes into account and thus >>> triggers collisions with original and physical nodes >>> only-mapping that had been determined from BIOS tables. >>> This patch fixes this by doing the necessary node-ids >>> translation in both pxm_to_node_map[]/node_to_pxm_map[] >>> tables. >>> node_distance[] table has also been fixed accordingly. >>> >>> Signed-off-by: Bruno Faccini >> >> Hi Bruno, >> >> This commit causes WARN() and disables fakenuma >> when I pass "numa=fake=4U" on my QEMU setup. >> >> Attaching WARN() log and git bisect log to help debugging. > > Is your VM getting 4 NUMA nodes at the end? No, it only gets node 0 at the end. > Can you also share your QEMU command line and your kernel config? The config is attached (fakenuma.config) QEMU command line: $ qemu-system-x86_64 \ -cpu host \ -enable-kvm \ -smp $(nproc) \ -m 32G \ -nographic \ -kernel arch/x86/boot/bzImage \ -initrd ../../rootfs.cpio.gz \ -nic user,model=virtio-net-pci \ -append "console=ttyS0 numa=fake=4U" > This can help Bruno reproduce the issue locally. At least, I > cannot reproduce it on mm-everything-2025-01-16-23-18. Still reproduced on mm-everything-2025-01-18-04-55 with my setup. > I also > notice that my VM does not have “No NUMA configuration found” > (in your log) in its kernel log and that might explain > why I could not reproduce it. Did you do NUMA configuration (-numa in QEMU) for your VM? I didn't, but it's still expected to work 'cause it's fake numa. Best, Hyeonggon >> >> [WARN() splat] >> >> [ 0.009676] No NUMA configuration found >> [ 0.009677] Faking a node at [mem 0x0000000000000000-0x000000083fffffff] >> [ 0.009705] Fake node size 16895MB too small, increasing to 16896MB >> [ 0.009708] Faking node 0 at [mem 0x0000000000001000-0x0000000420000fff] (16896MB) >> [ 0.009711] Faking node 1 at [mem 0x0000000420001000-0x000000083fffffff] (16895MB) >> [ 0.009738] ------------[ cut here ]------------ >> [ 0.009739] -1 max nid when expected 1 >> [ 0.009760] WARNING: CPU: 0 PID: 0 at drivers/acpi/numa/srat.c:120 fix_pxm_node_maps+0x577/0x810 >> [ 0.009766] Modules linked in: >> [ 0.009769] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.13.0-rc6+ #31 >> [ 0.009771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 >> [ 0.009772] RIP: 0010:fix_pxm_node_maps+0x577/0x810 >> [ 0.009774] Code: 8b 85 40 ff ff ff 44 8b 8d 44 ff ff ff 8b 85 48 ff ff ff e9 83 fb ff ff 44 89 f2 44 89 ce 48 c7 c7 aa 2f 84 8d e8 b9 71 01 fe <0f> 0b 41 b8 ff ff ff ff e9 ba fb ff ff 48 c7 c7 a0 d8 d1 8d 44 89 >> [ 0.009776] RSP: 0000:ffffffff8da03be0 EFLAGS: 00010046 ORIG_RAX: 0000000000000000 >> [ 0.009778] RAX: 0000000000000000 RBX: 0000000000000400 RCX: 0000000000000000 >> [ 0.009779] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 >> [ 0.009780] RBP: ffffffff8da03ca8 R08: 0000000000000000 R09: 0000000000000000 >> [ 0.009781] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 >> [ 0.009782] R13: 00000000000007ff R14: 0000000000000001 R15: 0000000000000000 >> [ 0.009783] FS: 0000000000000000(0000) GS:ffffffff8de8c000(0000) knlGS:0000000000000000 >> [ 0.009784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 0.009785] CR2: ffff90e4bde01000 CR3: 00000001bd03e000 CR4: 00000000000200b0 >> [ 0.009788] Call Trace: >> [ 0.009790] >> [ 0.009792] ? show_regs+0x71/0x90 >> [ 0.009797] ? __warn+0x8d/0x150 >> [ 0.009799] ? fix_pxm_node_maps+0x577/0x810 >> [ 0.009802] ? report_bug+0x1ab/0x1c0 >> [ 0.009805] ? fixup_exception+0x27/0x380 >> [ 0.009809] ? early_fixup_exception+0xa2/0xf0 >> [ 0.009813] ? do_early_exception+0x28/0x90 >> [ 0.009816] ? early_idt_handler_common+0x2f/0x3a >> [ 0.009819] ? fix_pxm_node_maps+0x577/0x810 >> [ 0.009822] ? numa_cleanup_meminfo+0x8c/0x5b0 >> [ 0.009828] numa_emulation+0x4e3/0xad0 >> [ 0.009830] ? __pfx_dummy_numa_init+0x10/0x10 >> [ 0.009833] ? __pfx_dummy_numa_init+0x10/0x10 >> [ 0.009836] numa_memblks_init+0x10c/0x2c0 >> [ 0.009838] ? __pfx_dummy_numa_init+0x10/0x10 >> [ 0.009840] numa_init+0x61/0x3e0 >> [ 0.009842] ? topology_register_boot_apic+0x2a/0x40 >> [ 0.009846] x86_numa_init+0x65/0x80 >> [ 0.009848] initmem_init+0xe/0x20 >> [ 0.009850] setup_arch+0x9d8/0xfa0 >> [ 0.009852] ? _printk+0x58/0x90 >> [ 0.009855] start_kernel+0x5f/0xb50 >> [ 0.009860] x86_64_start_reservations+0x18/0x30 >> [ 0.009863] x86_64_start_kernel+0xc0/0x110 >> [ 0.009864] ? setup_ghcb+0xe/0x140 >> [ 0.009867] common_startup_64+0x13e/0x141 >> [ 0.009871] >> [ 0.009871] ---[ end trace 0000000000000000 ]--- >> >> [Git bisect log] >> >> # bad: [f378252a2168c2fbf8fc08b635061e5f6748c1f2] kasan: sw_tags: use str_on_off() helper in kasan_init_sw_tags() >> git bisect bad f378252a2168c2fbf8fc08b635061e5f6748c1f2 >> # good: [cbc5dde0a461240046e8a41c43d7c3b76d5db952] fs/proc: fix softlockup in __read_vmcore (part 2) >> git bisect good cbc5dde0a461240046e8a41c43d7c3b76d5db952 >> # good: [fb73203263021f2ee9dd54d280b1c543d10acd76] mm: move common part of pagetable_*_ctor to helper >> git bisect good fb73203263021f2ee9dd54d280b1c543d10acd76 >> # bad: [8fd966c9310f14d5bbe653cf087853582170aad6] mm, swap: remove old allocation path for HDD >> git bisect bad 8fd966c9310f14d5bbe653cf087853582170aad6 >> # bad: [8646971f02651146554233d63fdd721f5f060973] mm: shmem: skip swapcache for swapin of synchronous swap device >> git bisect bad 8646971f02651146554233d63fdd721f5f060973 >> # good: [b8ba614eaa8dd31702c61e7df7231b1b18b99259] mm/damon/paddr: report filter-passed bytes back for DAMOS_STAT action >> git bisect good b8ba614eaa8dd31702c61e7df7231b1b18b99259 >> # good: [fd0935b8e9e8c2edf05a3d0ffa09d0aa3e9cf2dd] Docs/ABI/damon: document per-region DAMOS filter-passed bytes stat file >> git bisect good fd0935b8e9e8c2edf05a3d0ffa09d0aa3e9cf2dd >> # good: [3ffd3cd7e2e8793d3b5fbb83c03668c47ab6d599] selftests/damon: remove tests for DAMON debugfs interface >> git bisect good 3ffd3cd7e2e8793d3b5fbb83c03668c47ab6d599 >> # good: [bf9f93f0b7bff8af780330f5094775a39caf3612] mm/damon: remove DAMON debugfs interface >> git bisect good bf9f93f0b7bff8af780330f5094775a39caf3612 >> # bad: [1746744be6ff271e29345a5be4cc144aa33b10ab] mm/memmap: prevent double scanning of memmap by kmemleak >> git bisect bad 1746744be6ff271e29345a5be4cc144aa33b10ab >> # bad: [ca19a5ab8756ef78c34d02d2b8c266a9cc7bf57d] mm/fake-numa: allow later numa node hotplug >> git bisect bad ca19a5ab8756ef78c34d02d2b8c266a9cc7bf57d >> # first bad commit: [ca19a5ab8756ef78c34d02d2b8c266a9cc7bf57d] mm/fake-numa: allow later numa node hotplug >> >>> --- >>> drivers/acpi/numa/srat.c | 86 ++++++++++++++++++++++++++++++++++++ >>> include/acpi/acpi_numa.h | 5 +++ >>> include/linux/numa_memblks.h | 3 ++ >>> mm/numa_emulation.c | 45 ++++++++++++++++--- >>> mm/numa_memblks.c | 2 +- >>> 5 files changed, 133 insertions(+), 8 deletions(-) >>> >>> diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c >>> index bec0dcd1f9c3..59fffe34c9d0 100644 >>> --- a/drivers/acpi/numa/srat.c >>> +++ b/drivers/acpi/numa/srat.c >>> @@ -81,6 +81,92 @@ int acpi_map_pxm_to_node(int pxm) >>> } >>> EXPORT_SYMBOL(acpi_map_pxm_to_node); >>> +#ifdef CONFIG_NUMA_EMU >>> +/* >>> + * Take max_nid - 1 fake-numa nodes into account in both >>> + * pxm_to_node_map()/node_to_pxm_map[] tables. >>> + */ >>> +int __init fix_pxm_node_maps(int max_nid) >>> +{ >>> + static int pxm_to_node_map_copy[MAX_PXM_DOMAINS] __initdata >>> + = { [0 ... MAX_PXM_DOMAINS - 1] = NUMA_NO_NODE }; >>> + static int node_to_pxm_map_copy[MAX_NUMNODES] __initdata >>> + = { [0 ... MAX_NUMNODES - 1] = PXM_INVAL }; >>> + int i, j, index = -1, count = 0; >>> + nodemask_t nodes_to_enable; >>> + >>> + if (numa_off || srat_disabled()) >>> + return -1; >>> + >>> + /* find fake nodes PXM mapping */ >>> + for (i = 0; i < MAX_NUMNODES; i++) { >>> + if (node_to_pxm_map[i] != PXM_INVAL) { >>> + for (j = 0; j <= max_nid; j++) { >>> + if ((emu_nid_to_phys[j] == i) && >>> + WARN(node_to_pxm_map_copy[j] != PXM_INVAL, >>> + "Node %d is already binded to PXM %d\n", >>> + j, node_to_pxm_map_copy[j])) >>> + return -1; >>> + if (emu_nid_to_phys[j] == i) { >>> + node_to_pxm_map_copy[j] = >>> + node_to_pxm_map[i]; >>> + if (j > index) >>> + index = j; >>> + count++; >>> + } >>> + } >>> + } >>> + } >>> + if (WARN(index != max_nid, "%d max nid when expected %d\n", >>> + index, max_nid)) >>> + return -1; >>> + >>> + nodes_clear(nodes_to_enable); >>> + >>> + /* map phys nodes not used for fake nodes */ >>> + for (i = 0; i < MAX_NUMNODES; i++) { >>> + if (node_to_pxm_map[i] != PXM_INVAL) { >>> + for (j = 0; j <= max_nid; j++) >>> + if (emu_nid_to_phys[j] == i) >>> + break; >>> + /* fake nodes PXM mapping has been done */ >>> + if (j <= max_nid) >>> + continue; >>> + /* find first hole */ >>> + for (j = 0; >>> + j < MAX_NUMNODES && >>> + node_to_pxm_map_copy[j] != PXM_INVAL; >>> + j++) >>> + ; >>> + if (WARN(j == MAX_NUMNODES, >>> + "Number of nodes exceeds MAX_NUMNODES\n")) >>> + return -1; >>> + node_to_pxm_map_copy[j] = node_to_pxm_map[i]; >>> + node_set(j, nodes_to_enable); >>> + count++; >>> + } >>> + } >>> + >>> + /* creating reverse mapping in pxm_to_node_map[] */ >>> + for (i = 0; i < MAX_NUMNODES; i++) >>> + if (node_to_pxm_map_copy[i] != PXM_INVAL && >>> + pxm_to_node_map_copy[node_to_pxm_map_copy[i]] == NUMA_NO_NODE) >>> + pxm_to_node_map_copy[node_to_pxm_map_copy[i]] = i; >>> + >>> + /* overwrite with new mapping */ >>> + for (i = 0; i < MAX_NUMNODES; i++) { >>> + node_to_pxm_map[i] = node_to_pxm_map_copy[i]; >>> + pxm_to_node_map[i] = pxm_to_node_map_copy[i]; >>> + } >>> + >>> + /* enable other nodes found in PXM for hotplug */ >>> + nodes_or(numa_nodes_parsed, nodes_to_enable, numa_nodes_parsed); >>> + >>> + pr_debug("found %d total number of nodes\n", count); >>> + return 0; >>> +} >>> +#endif >>> + >>> static void __init >>> acpi_table_print_srat_entry(struct acpi_subtable_header *header) >>> { >>> diff --git a/include/acpi/acpi_numa.h b/include/acpi/acpi_numa.h >>> index b5f594754a9e..99b960bd473c 100644 >>> --- a/include/acpi/acpi_numa.h >>> +++ b/include/acpi/acpi_numa.h >>> @@ -17,11 +17,16 @@ extern int node_to_pxm(int); >>> extern int acpi_map_pxm_to_node(int); >>> extern unsigned char acpi_srat_revision; >>> extern void disable_srat(void); >>> +extern int fix_pxm_node_maps(int max_nid); >>> extern void bad_srat(void); >>> extern int srat_disabled(void); >>> #else /* CONFIG_ACPI_NUMA */ >>> +static inline int fix_pxm_node_maps(int max_nid) >>> +{ >>> + return 0; >>> +} >>> static inline void disable_srat(void) >>> { >>> } >>> diff --git a/include/linux/numa_memblks.h b/include/linux/numa_memblks.h >>> index cfad6ce7e1bd..dd85613cdd86 100644 >>> --- a/include/linux/numa_memblks.h >>> +++ b/include/linux/numa_memblks.h >>> @@ -29,7 +29,10 @@ int __init numa_cleanup_meminfo(struct numa_meminfo *mi); >>> int __init numa_memblks_init(int (*init_func)(void), >>> bool memblock_force_top_down); >>> +extern int numa_distance_cnt; >>> + >>> #ifdef CONFIG_NUMA_EMU >>> +extern int emu_nid_to_phys[MAX_NUMNODES]; >>> int numa_emu_cmdline(char *str); >>> void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys, >>> unsigned int nr_emu_nids); >>> diff --git a/mm/numa_emulation.c b/mm/numa_emulation.c >>> index 031fb9961bf7..9d55679d99ce 100644 >>> --- a/mm/numa_emulation.c >>> +++ b/mm/numa_emulation.c >>> @@ -8,11 +8,12 @@ >>> #include >>> #include >>> #include >>> +#include >>> #define FAKE_NODE_MIN_SIZE ((u64)32 << 20) >>> #define FAKE_NODE_MIN_HASH_MASK (~(FAKE_NODE_MIN_SIZE - 1UL)) >>> -static int emu_nid_to_phys[MAX_NUMNODES]; >>> +int emu_nid_to_phys[MAX_NUMNODES]; >>> static char *emu_cmdline __initdata; >>> int __init numa_emu_cmdline(char *str) >>> @@ -379,6 +380,7 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) >>> size_t phys_size = numa_dist_cnt * numa_dist_cnt * sizeof(phys_dist[0]); >>> int max_emu_nid, dfl_phys_nid; >>> int i, j, ret; >>> + nodemask_t physnode_mask = numa_nodes_parsed; >>> if (!emu_cmdline) >>> goto no_emu; >>> @@ -395,7 +397,6 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) >>> * split the system RAM into N fake nodes. >>> */ >>> if (strchr(emu_cmdline, 'U')) { >>> - nodemask_t physnode_mask = numa_nodes_parsed; >>> unsigned long n; >>> int nid = 0; >>> @@ -465,9 +466,6 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) >>> */ >>> max_emu_nid = setup_emu2phys_nid(&dfl_phys_nid); >>> - /* commit */ >>> - *numa_meminfo = ei; >>> - >>> /* Make sure numa_nodes_parsed only contains emulated nodes */ >>> nodes_clear(numa_nodes_parsed); >>> for (i = 0; i < ARRAY_SIZE(ei.blk); i++) >>> @@ -475,10 +473,21 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) >>> ei.blk[i].nid != NUMA_NO_NODE) >>> node_set(ei.blk[i].nid, numa_nodes_parsed); >>> - numa_emu_update_cpu_to_node(emu_nid_to_phys, ARRAY_SIZE(emu_nid_to_phys)); >>> + /* fix pxm_to_node_map[] and node_to_pxm_map[] to avoid collision >>> + * with faked numa nodes, particularly during later memory hotplug >>> + * handling, and also update numa_nodes_parsed accordingly. >>> + */ >>> + ret = fix_pxm_node_maps(max_emu_nid); >>> + if (ret < 0) >>> + goto no_emu; >>> + >>> + /* commit */ >>> + *numa_meminfo = ei; >>> + >>> + numa_emu_update_cpu_to_node(emu_nid_to_phys, max_emu_nid + 1); >>> /* make sure all emulated nodes are mapped to a physical node */ >>> - for (i = 0; i < ARRAY_SIZE(emu_nid_to_phys); i++) >>> + for (i = 0; i < max_emu_nid + 1; i++) >>> if (emu_nid_to_phys[i] == NUMA_NO_NODE) >>> emu_nid_to_phys[i] = dfl_phys_nid; >>> @@ -501,12 +510,34 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) >>> numa_set_distance(i, j, dist); >>> } >>> } >>> + for (i = 0; i < numa_distance_cnt; i++) { >>> + for (j = 0; j < numa_distance_cnt; j++) { >>> + int physi, physj; >>> + u8 dist; >>> + >>> + /* distance between fake nodes is already ok */ >>> + if (emu_nid_to_phys[i] != NUMA_NO_NODE && >>> + emu_nid_to_phys[j] != NUMA_NO_NODE) >>> + continue; >>> + if (emu_nid_to_phys[i] != NUMA_NO_NODE) >>> + physi = emu_nid_to_phys[i]; >>> + else >>> + physi = i - max_emu_nid; >>> + if (emu_nid_to_phys[j] != NUMA_NO_NODE) >>> + physj = emu_nid_to_phys[j]; >>> + else >>> + physj = j - max_emu_nid; >>> + dist = phys_dist[physi * numa_dist_cnt + physj]; >>> + numa_set_distance(i, j, dist); >>> + } >>> + } >>> /* free the copied physical distance table */ >>> memblock_free(phys_dist, phys_size); >>> return; >>> no_emu: >>> + numa_nodes_parsed = physnode_mask; >>> /* No emulation. Build identity emu_nid_to_phys[] for numa_add_cpu() */ >>> for (i = 0; i < ARRAY_SIZE(emu_nid_to_phys); i++) >>> emu_nid_to_phys[i] = i; >>> diff --git a/mm/numa_memblks.c b/mm/numa_memblks.c >>> index a3877e9bc878..ff4054f4334d 100644 >>> --- a/mm/numa_memblks.c >>> +++ b/mm/numa_memblks.c >>> @@ -7,7 +7,7 @@ >>> #include >>> #include >>> -static int numa_distance_cnt; >>> +int numa_distance_cnt; >>> static u8 *numa_distance; >>> nodemask_t numa_nodes_parsed __initdata; > > > Best Regards, > Yan, Zi >