Re: BUG_ON() in pfn_swap_entry_to_page()

[David Hildenbrand <david@redhat.com>]

On 25.04.24 16:33, Felix Kuehling wrote:
> 
> 
> On 2024-04-25 5:32, David Hildenbrand wrote:
>> On 24.04.24 21:45, Felix Kuehling wrote:
>>> Sorry for top-posting. I'm resurrecting an old thread here because I
>>> think I ran into the same problem with this assertion failing on Linux
>>> 6.7:
>>>
>>> static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
>>> {
>>>           struct page *p = pfn_to_page(swp_offset_pfn(entry));
>>>
>>>           /*
>>>            * Any use of migration entries may only occur while the
>>>            * corresponding page is locked
>>>            */
>>> -->     BUG_ON(is_migration_entry(entry) && !PageLocked(p));
>>>
>>>           return p;
>>> }
>>>
>>> It looks like this thread just fizzled two years ago. Did anything
>>> ever come of this?
>>>
>>> Maybe I should add that I saw this in a pre-silicon test environment.
>>> I've never seen this on real hardware. Maybe something timing-sensitive.
>>
>> In the past, it indicated a swp pte corruption, that would e.g., mess up
>> the stored PFN ot the swap entry type.
>>
>> On which call chain do you see that?
>>
> 
> This is the backtrace, it's coming from hmm_range_fault. Looks like the
> swap entries are from migrated DEVICE_PRIVATE pages.

Thanks, on which kernel version can you reproduce this?

> 
> [Apr 3 20:11] ------------[ cut here ]------------
> [  +0.000041] kernel BUG at include/linux/swapops.h:466!
> [  +0.000691] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> [  +0.000342] CPU: 2 PID: 49 Comm: kworker/2:1 Not tainted
> 6.7.0-kfd-compute-rocm-npi-186 #1
> [  +0.000556] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> [  +0.000703] Workqueue: events amdgpu_irq_handle_ih_soft [amdgpu]
> [  +0.000501] RIP: 0010:migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000389] Code: fe ff ff 48 8d 7c 24 07 e8 02 7e f0 ff e9 58 fe ff
> ff 48 8b 43 08 a8 01 75 3f 66 90 48 89 d8 48 8b 00 a8 01 0f 85 f1 fd ff
> ff <0f> 0b 48 8d 58 ff e9 f7 fd ff ff 48 89 d8 f7 c3 ff 0f 00 00 75 df
> [  +0.001161] RSP: 0018:ffffb211c01bb788 EFLAGS: 00010246
> [  +0.000339] RAX: 017fff8000080018 RBX: fffff682c40ce8c0 RCX:
> 0000000000000001
> [  +0.000463] RDX: 0000000000000000 RSI: ffff977a45034840 RDI:
> 000000000000001a
> [  +0.000454] RBP: ffff977a45034840 R08: 68000000001033a3 R09:
> 0000000000000030
> [  +0.000451] R10: ffffb211c01bb6a8 R11: 0000000000000001 R12:
> ffff977a46bd1318
> [  +0.000461] R13: 0000000000000003 R14: 4000000000000000 R15:
> ffffb211c01bb9b8
> [  +0.000454] FS:  0000000000000000(0000) GS:ffff977dafd00000(0000)
> knlGS:0000000000000000
> [  +0.000518] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  +0.000372] CR2: 00007fa2d1cba000 CR3: 00000001030d2004 CR4:
> 0000000000770ef0
> [  +0.000453] PKRU: 55555554
> [  +0.000182] Call Trace:
> [  +0.000171]  <TASK>
> [  +0.000147]  ? die+0x37/0x90
> [  +0.000211]  ? do_trap+0xe0/0x110
> [  +0.000221]  ? migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000351]  ? do_error_trap+0x98/0x120
> [  +0.000252]  ? migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000346]  ? migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000355]  ? exc_invalid_op+0x52/0x70
> [  +0.000254]  ? migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000345]  ? asm_exc_invalid_op+0x1a/0x20
> [  +0.000274]  ? migration_entry_wait_on_locked+0x26b/0x2b0
> [  +0.000361]  ? migration_entry_wait+0x4e/0x160
> [  +0.000293]  ? lock_release+0x119/0x260
> [  +0.000255]  migration_entry_wait+0x105/0x160
> [  +0.000290]  hmm_vma_walk_pmd+0x822/0x8a0
> [  +0.000263]  walk_pgd_range+0x40b/0x900
> [  +0.000268]  __walk_page_range+0x205/0x220

I wonder if that is coming from pmd_migration_entry_wait() or 
migration_entry_wait() --  the "?" above adds uncertainty :)

Likely it's from migration_entry_wait().

I was first concerned about the lack of PTL in this function, but 
migration_entry_wait() will take the PTL and re-read the PTE.

So when we call into migration_entry_wait_on_locked(), we are holding 
the PTL and we verified that we indeed have a migration entry.

So if we fail in 
migration_entry_wait_on_locked()->pfn_swap_entry_folio(), we verified 
under PTL and still have a migration entry.

The referenced folio is indeed not locked then.

-- 
Cheers,

David / dhildenb