Re: [BUG ?] Offline Memory gets stuck in offline_pages()

["Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com>]

All,

Some progress updates

When issue occurs, calling __drain_all_pages() can make offline_pages() escape from the loop.

> 
> Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
> Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
> Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
> Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
> Jun 28 15:29:26 linux kernel: page dumped because: trouble page...
> 

With this problematic page structure contents, it seems that the
list_head = {ffffdfbd9e603788, ffffd4f0ffd97ef0} is valid.

I guess it was linking to the pcp_list, so I dumped the per_cpu_pages[cpu].count
in every in critical timings.

An example is as below,
offline_pages()
{
	// per_cpu_pages[1].count = 0
	zone_pcp_disable() // will call __drain_all_pages()
	// per_cpu_pages[1].count = 188
	do {
		do {
			scan_movable_pages()
			ret = do_migrate_range()
		} while (!ret)

		ret = test_pages_isolated()

		if(is the 1st iteration)
			// per_cpu_pages[1].count = 182

		if (issue occurs) { /* if the loop take beyond 10 seconds */
			// per_cpu_pages[1].count = 61
			__drain_all_pages()
			// per_cpu_pages[1].count = 0
			/* will escape from the outer loop in later iterations */
		}
	} while (ret)
}

Some interesting points:
  - After the 1st __drain_all_pages(), per_cpu_pages[1].count increased to 188 from 0,
    does it mean it's racing with something...?
  - per_cpu_pages[1].count will decrease but not decrease to 0 during iterations
  - when issue occurs, calling __drain_all_pages() will decrease per_cpu_pages[1].count to 0.

So I wonder if it's fine to call __drain_all_pages() in the loop?

Looking forward to your insights.


Thanks
Zhijian


On 01/07/2024 09:25, Zhijian Li (Fujitsu) wrote:
> Hi all
> 
> 
> Overview:
> During testing the CXL memory hotremove, we noticed that `daxctl offline-memory dax0.0`
> would get stuck forever sometimes. daxctl offline-memory dax0.0 will write "offline" to
> /sys/devices/system/memory/memoryNNN/state.
> 
> Workaround:
> When it happens, we can type Ctrl-C to abort it and then retry again.
> Then the CXL memory is able to offline successfully.
> 
> Where the kernel gets stuck:
> After digging into the kernel, we found that when the issue occurs, the kernel
> is stuck in the outer loop of offline_pages(). Below is a piece of the
> highlighted offline_pages():
> 
> ```
> int __ref offline_pages()
> {
>     do { // outer loop
>       pfn = start_pfn;
>       do {
>         ret = scan_movable_pages(pfn, end_pfn, &pfn);  // It returns -ENOENT
>         if (!ret)
>            do_migrate_range(pfn, end_pfn);             // Not reach here
>       } while (!ret);
>       ret = test_pages_isolated(start_pfn, end_pfn, MEMORY_OFFLINE);
>       } while (ret);                                   // ret is -EBUSY
> }
> ```
> 
> In this case, we dumped the first page that cannot be isolated (see dump_page below), it's
> content does not change in each iteration.:
> ```
> Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
> Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
> Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
> Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
> Jun 28 15:29:26 linux kernel: page dumped because: trouble page...
> ```
> 
> Every time the issue occurs, the content of the page structure is similar.
> 
> Questions:
> Q1. Is this behavior expected? At least for an OS administrator, it should return
>       promptly (success or failure) instead of hanging indefinitely.
> Q2. Regarding the offline_pages() function, encountering such a page indeed causes
>       an endless loop. Shouldn't another part of the kernel timely changed the state
>       of this page?
> 
>       When I use the workaround mentioned above (Ctrl-C and try offline again), I find
>       that the page state changes (see dump_page below):
> ```
> Jun 28 15:33:12 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
> Jun 28 15:33:12 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
> Jun 28 15:33:12 linux kernel: raw: 009fffffc0000000 dead000000000100 dead000000000122 0000000000000000
> Jun 28 15:33:12 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
> Jun 28 15:33:12 linux kernel: page dumped because: previous trouble page
> ```
> 
> What our test does:
> We have a CXL memory device, which is configured as kmem and online into the MOVABLE
> zone as NUMA node2. We run two processes, consume-memory and offline-memory, in parallel,
> see the pseudo code below:
> 
> ```
> main()
> {
>       if (fork() == 0)
>           numactl -m 2 ./consume-memory
>       else {
>           daxctl offline-memory dax0.0
>           wait()
>       }
> }
> ```
> 
> Attached is the process information (when it gets stuck):
> ```
> root 25716 0.0 0.0 2460 1408 pts/0 S+ 15:28 0:00 ./main
> root 25719 0.0 0.0 0 0 pts/0 Z+ 15:28 0:00 [consume-memory] <defunct>
> root 25720 98.6 0.0 9476 3740 pts/0 R+ 15:28 0:26 daxctl offline-memory /dev/dax0.0
> ```
> 
> Feel free to let me know if you need more details.
> Thank you for your attention to this issue. Looking forward to your insights.
> 
> Thanks
> Zhijian