Re: [RFC Patch 0/5] Make anon_vma operations testable

[Wei Yang <richard.weiyang@gmail.com>]

On Tue, May 27, 2025 at 01:31:47PM +0200, David Hildenbrand wrote:
>On 27.05.25 08:34, Wei Yang wrote:
>> On Wed, May 14, 2025 at 01:23:18AM +0000, Wei Yang wrote:
>> > On Wed, Apr 30, 2025 at 09:47:16AM +0200, David Hildenbrand wrote:
>> > [...]
>> > > > > > Agreed, skimming over the tests there are some nice diagrams and cases.
>> > > > > > 
>> > > > > > But I would hope that for most of these cases we could test on a higher
>> > > > > > level: test our expectations when running real programs that we want to
>> > > > > > check, especially when performing internal changes on how we handle anon
>> > > > > > memory + rmap.
>> > > > > > 
>> > > > > > E.g., do fork(), then test if we can successfully perform rmap
>> > > > > > lookups/updates (e.g., migrate folio to a different numa node etc).
>> > > > > > 
>> > > > > 
>> > > > > That's a great point! Wei - if you could look at making some self-tests
>> > > > > (i.e. that live in tools/testing/selftests/mm) that try to recreate _real_
>> > > > > scenarios that use the rmap like this and assert correct behaviour there,
>> > > > > that could be a positive way of moving forward with this.
>> > > > > 
>> > > > 
>> 
>> Ping
>
>Thanks for reminding me and sorry for the late reply.
>
>> 
>> > > > I am trying to understand what scenarios you want.
>> > > 
>> > 
>> > Sorry for the late reply, I handled other things a while.
>> > 
>> > > That is exactly the task to figure out: how can we actually test our rmap
>> > > implementation from a higher level. The example regarding fork and migration
>> > > is possibly a low-hanging fruit.
>> > 
>> > If my understanding is correct, you suggested two high level way:
>> > 
>> > 1. fork + migrate (move_pages)
>
>Yes, that should be one way of testing it. We could even get multiple child
>processes into place.
>
>> > 
>> > > 
>> > > We might already have the functionality to achieve it, *maybe* we'd even want
>> > > some extensions to make it all even easier to test.
>> > > 
>> > > For example, MADV_PAGEOUT is refused on folios that are mapped into multiple
>> > > processes. Maybe we'd want the option to *still* page it out, just like
>> > > MPOL_MF_MOVE_ALL allows with CAP_SYS_NICE to *still* migrate a folio that is
>> > > mapped into multiple processes.
>> > > 
>> > 
>> > 2. madvise(MADV_PAGEOUT)
>> > 
>> > Not fully get it here. You mean fork + madvise(MADV_PAGEOUT) + migrate ?
>
>fork + madvise(MADV_PAGEOUT) only.
>
>Then verify, that it was actually paged out.
>
>For example, we could have 10 processes mapping the page, then call
>madvise(MADV_PAGEOUT) from one process. We can verify in all processes if the
>page is actually no longer mapped using /proc/self/pagemap

Got it, thanks.

Will prepare the test cases.

>
>We could likely do it with anon pages (swap), shmem pages (swap) and
>pagecache pages (file backed).
>
>... and possibly even with KSM pages!
>
>> > 
>> > But we need to enable pageout in this way first.
>
>Yes, madvise(MADV_PAGEOUT) needs to be extended to allow forcing a pageout
>(e.g., a new MADVISE_PAGEOUT_SHARED that only works with with CAP_SYY_ADMIN).
>
>> > 
>> > I am not sure why this one is easier way to test. Would you mind sharing more
>> > idea on this?
>
>It might be easier than the migration case, because for migration we
>currently need 2 NUMA nodes ...
>
>> > 
>> > > Some rmap tests could make sense for both, anon and pagecache folios.
>> > > 
>> > > > 
>> > > > Something like below?
>> > > > 
>> > > >     * fork and migrate a range in child
>> > > >     * fork/unmap in parent and migrate a range in child
>> > > > 
>> > > > If the operation is successful, then we are good, right?
>> > > 
>> > > Yes. And one can come up with a bunch of similar rmap test cases, like doing
>> > > a partial mremap() of a THP, then testing if the rmap walk still works as
>> > > expected, pairing the whole thing with for etc.
>> > > 
>> > 
>> > For both way, we could arrange all those scenarios and also do partial
>> > mremap() during it.
>
>Exactly.
>
>> > 
>> > > One "problem" here is that even with MPOL_MF_MOVE_ALL,
>> > > move_pages() will not move a folio if it already resides on the target node.
>> > > So one always needs two NUMA nodes, which is a bit suboptimal for testing
>> > > purposes.
>> > > 
>> > > For testing purposes, it could have been helpful a couple of times already to
>> > > just have a way of migrating a folio even if it already resides on the
>> > > expected node.
>> > > 
>> > 
>> > This looks we need a new flag for it?
>> > 
>> > Here is my plan if my understanding is correct.
>> > 
>> > 1. Add test cases for fork + migrate. We may limit it only works on machine
>> >    with 2 NUMA nodes.
>> > 2. Enable move_pages() on local node, then remove the test limitation
>> > 3. Enable madvise(MADV_PAGEOUT) with multiple mapping, then add related cases
>> > 4. Add mremap() or other cases
>> > 
>> > In general, to verify rmap dose the work correctly, my idea is to
>> > 
>> >   * mmap(MAP_SHARED)
>>>   * write some initial data before fork>>   * after fork and migrate, we
>write some different data to it
>> >   * if each process do see the new data, rmap is good.
>
>With pageout (see above) test, we can just verify using /proc/self/pagemap in
>all processes whether the page was paged out (IOW: the rmap was able to
>identify all page locations). And that should work with anon/shmem/file/ksm
>pages IIUC.
>
>> > 
>> > Does it sound good to you?
>
>Yes, that absolutely goes into the right direction. Having also rmap tests
>for KSM as raised above might be a real benefit.
>
>
>-- 
>Cheers,
>
>David / dhildenb

-- 
Wei Yang
Help you, Help me