Re: [PATCH RFC] mm: implement granular soft-dirty vma support

[David Hildenbrand <david@redhat.com>]

On 20.12.22 17:26, Muhammad Usama Anjum wrote:
> The VM_SOFTDIRTY is used to mark a whole VMA soft-dirty. Sometimes
> soft-dirty and non-soft-dirty VMAs are merged making the non-soft-dirty
> region soft-dirty. This creates problems as the whole VMA region comes
> out to be soft-dirty while in-reality no there is no soft-dirty page.
> This can be solved by not merging the VMAs with different soft-dirty
> flags. But it has potential to cause regression as the number of VMAs
> can increase drastically.

I'm not going to look into the details of this RFC, but (1) it looks 
over-engineered and (2) is increases the size of each and every VMA in 
the system.

Let's talk about what happens when we stop merging VMAs where 
VM_SOFTDIRTY differs:

(a) Not merging VMAs when VM_SOFTDIRTY differs will only affect
     processes with active softdirty tracking (i.e., after
     CLEAR_REFS_SOFT_DIRTY). All other VMAs have VM_SOFTDIRTY set and
     will get merged. Processes without CLEAR_REFS_SOFT_DIRTY behave the
     same.

(b) After CLEAR_REFS_SOFT_DIRTY, old mappings will have VM_SOFTDIRTY set
     but new ones won't. We can't merge them. Consequently, we might not
     merge these VMAs and create more.

(c) The problem about (b) is that it will get worse every time we
     CLEAR_REFS_SOFT_DIRTY, because we're not merging the old ones that
     could get merged.


To tackle (c), we can simply try merging VMAs in clear_refs_write() when 
clearing VM_SOFTDIRTY. We're already properly holding the mmap lock in 
write mode, so it's easy to check if we can merge the modified VMA into 
the previous one or into the next one -- or if we can merge all of them 
into a single VMA.


For (b), the usual thing we care about is most probably

[!VM_SOFTDIRTY][VM_SOFTDIRTY]

No longer getting merged into a single VMA. This would imply that during 
(b), we could have doubled the #VMAs.

Of course, there are cases like

[!VM_SOFTDIRTY][VM_SOFTDIRTY][!VM_SOFTDIRTY]

where we could triple them or even a chain like

[!VM_SOFTDIRTY][VM_SOFTDIRTY][!VM_SOFTDIRTY][VM_SOFTDIRTY]...

where the #VMAs could explode. BUT, that would imply that someone has to 
do fine-grained mmap()'s over an existing mmap()'s (or into holes) and 
heavily relies on VMA merging to happen. I assume that only with 
anonymous memory this really works as expected, so I am not sure how 
likely such a pattern is in applications we care about and if we should 
really care.

My suggestion would be to

(1) Make the new merging behavior (consider VM_SOFTDIRTY or ignore
     VM_SOFTDIRTY) configurable (per process? for the system) to not
     accidentally degrade existing users of soft-dirty tracking with such
     "special" applications.
(2) Implement conditional VMA merging during CLEAR_REFS_SOFT_DIRTY: if
     we consider VM_SOFTDIRTY when making merging decisions, we want to
     try merging VMAs here after clearing VM_SOFTDIRTY.
(2) For your use case, enable the new behavior and eventually slightly
     bump up the #VMA limit in case you're dealing with "special"
     applications.

(1) would be called something like "precise_softdirty_tracking". 
Disabled is old handling, enabled is new handling. Precision here means 
that we're not over-indicating softdirty due to VMA merging.


Anything important I am missing? Are we aware of such applications for 
your use-case?

-- 
Thanks,

David / dhildenb