Re: [PATCH v4 00/14] kexec: introduce Kexec HandOver (KHO)

[Mike Rapoport <rppt@kernel.org>]

Hi Pratyush,

On Wed, Feb 26, 2025 at 08:08:27PM +0000, Pratyush Yadav wrote:
> Hi Mike,
> 
> On Thu, Feb 06 2025, Mike Rapoport wrote:
> 
> > From: "Mike Rapoport (Microsoft)" <rppt@kernel.org>
> >
> > Hi,
> >
> > This a next version of Alex's "kexec: Allow preservation of ftrace buffers"
> > series (https://lore.kernel.org/all/20240117144704.602-1-graf@amazon.com),
> > just to make things simpler instead of ftrace we decided to preserve
> > "reserve_mem" regions.
> [...]
> 
> I applied the patches on top of v6.14-rc1 and tried them out on an x86
> qemu machine . When I do a plain KHO activate and kexec, I get the below
> errors on boot. This causes networking to fail on the VM. The errors are
> consistent and happen every kexec-reboot, though fairly late in boot
> after systemd tries to bring up network. The same setup has worked fine
> with Alex's v3 of KHO patches.
> 
> Do you see anything obvious that might cause this? I can try to debug
> this tomorrow, but if it rings any loud bells it would be nice to know.

Thanks for the report!
It didn't ring any bells, but after I've found the issue and a
fast-and-dirty fix.

The scratch areas are allocated from high addresses and there is no scratch
memory to satisfy memblock_alloc_low() in swiotb, so second kernel produces
a couple of 

software IO TLB: swiotlb_memblock_alloc: Failed to allocate 67108864 bytes for tlb structure

and without those buffers e1000 can't dma :(

A quick fix would be to add another scratch area in the lower memory
(below). I'll work on a better fix.

diff --git a/kernel/kexec_handover.c b/kernel/kexec_handover.c
index c26753d613cb..37bb54cdb130 100644
--- a/kernel/kexec_handover.c
+++ b/kernel/kexec_handover.c
@@ -623,13 +623,13 @@ static phys_addr_t __init scratch_size(int nid)
 static void kho_reserve_scratch(void)
 {
 	phys_addr_t addr, size;
-	int nid, i = 1;
+	int nid, i = 2;
 
 	if (!kho_enable)
 		return;
 
 	/* FIXME: deal with node hot-plug/remove */
-	kho_scratch_cnt = num_online_nodes() + 1;
+	kho_scratch_cnt = num_online_nodes() + 2;
 	size = kho_scratch_cnt * sizeof(*kho_scratch);
 	kho_scratch = memblock_alloc(size, PAGE_SIZE);
 	if (!kho_scratch)
@@ -644,6 +644,15 @@ static void kho_reserve_scratch(void)
 	kho_scratch[0].addr = addr;
 	kho_scratch[0].size = size;
 
+	addr = 	memblock_phys_alloc_range(size, CMA_MIN_ALIGNMENT_BYTES,
+					  MEMBLOCK_LOW_LIMIT,
+					  ARCH_LOW_ADDRESS_LIMIT);
+	if (!addr)
+		goto err_free_scratch_areas;
+
+	kho_scratch[1].addr = addr;
+	kho_scratch[1].size = size;
+
 	for_each_online_node(nid) {
 		size = scratch_size(nid);
 		addr = memblock_alloc_range_nid(size, CMA_MIN_ALIGNMENT_BYTES,

> -- 
> Regards,
> Pratyush Yadav
> 

-- 
Sincerely yours,
Mike.