[Question] performance regression after VM migration due to anon THP split in CoW

[Question] performance regression after VM migration due to anon THP split in CoW

[Jinjiang Tu]

Hi,

We noticed a performance regression in benchmark memtester[1] after
upgrading the kernel. THP is enabled by default 
(/sys/kernel/mm/transparent_hugepage/enabled
is set to "always"). The issue arises when we migrate a virtual machine
that has 125G total memory and 124G free memory to another host. And then,
we run the command `memtester 120G` in the VM. The benchmark takes about
20 seconds to consume 120G memory in v4.18, but takes about 160 seconds in
v5.10. This issue exists in mainline kernel too.

We find commit 3917c80280c9 ("thp: change CoW semantics for anon-THP")
leads to the performance regression. Since this commit, When we trigger a
write fault on a anon THP, we split the PMD and allocate a 4K page, instead
of allocating the full anon THP. When a VM is migrating (based on qemu[2]),
if the page is marked zero page in the source VM, the destination VM will
call mmap and read the region to allocate memory, making the region mapped
by the zero THP. When we run memtester in the destination VM after VM
migration finishes, memtester(in VM) will allocate large amounts of free
memory and write to them, cause CoW of anon THP and THP split, further
cause performance regression. After reverting this commit, performance
regression disappears.

This commit optimises some scenarios such as Redis, but may lead to
performance regression in some other scenarios, such as VM migration.
How could we solve this issue? Maybe we could add a new sysctl to let users
decide whether to CoW the full anon THP or not?

Thanks.

[1] https://github.com/jnavila/memtester/tree/master
[2] https://github.com/qemu/qemu/blob/master/migration/ram.c

Re: [Question] performance regression after VM migration due to anon THP split in CoW

[David Hildenbrand]

[-- Attachment #1: Type: text/plain, Size: 2640 bytes --]

Hi,

Likely the mailing lists won‘t like my mail from this Google Mail client ;)

Jinjiang Tu <tujinjiang@huawei.com> schrieb am Sa. 29. Juni 2024 um 11:18:

> Hi,
>
> We noticed a performance regression in benchmark memtester[1] after
> upgrading the kernel. THP is enabled by default
> (/sys/kernel/mm/transparent_hugepage/enabled
> is set to "always"). The issue arises when we migrate a virtual machine
> that has 125G total memory and 124G free memory to another host. And then,
> we run the command `memtester 120G` in the VM. The benchmark takes about
> 20 seconds to consume 120G memory in v4.18, but takes about 160 seconds in
> v5.10. This issue exists in mainline kernel too.
>

Simple: use preallocation in QEMU. „prealloc=on“ for host memory backends,
for example.


> We find commit 3917c80280c9 ("thp: change CoW semantics for anon-THP")
> leads to the performance regression. Since this commit, When we trigger a
> write fault on a anon THP, we split the PMD and allocate a 4K page, instead
> of allocating the full anon THP. When a VM is migrating (based on qemu[2]),
> if the page is marked zero page in the source VM, the destination VM will
> call mmap and read the region to allocate memory, making the region mapped
> by the zero THP. When we run memtester in the destination VM after VM
> migration finishes, memtester(in VM) will allocate large amounts of free
> memory and write to them, cause CoW of anon THP and THP split, further
> cause performance regression. After reverting this commit, performance
> regression disappears.


You talk about COW of anon THP, whereby your scenario really only relied on
COW of the huge zeropage.

Wouldn’t you would get a similar result when disabling the huge zeropage?


>
> This commit optimises some scenarios such as Redis, but may lead to
> performance regression in some other scenarios, such as VM migration.
> How could we solve this issue? Maybe we could add a new sysctl to let users
> decide whether to CoW the full anon THP or not?
>

I‘m not convinced the use case you present really warrants a toggle for
that. In your case you only want to change semantics on COW fault to the
huge zeropage. But …

Using preallocation in QEMU will give you all anon THP right from the
start, avoiding any cow. Sure, you consume all memory right away, but after
all that‘s what your use case triggers either way. And it might all be even
faster. :)

Cheers!


> Thanks.
>
> [1] https://github.com/jnavila/memtester/tree/master
> [2] https://github.com/qemu/qemu/blob/master/migration/ram.c
>
>
>

[-- Attachment #2: Type: text/html, Size: 3935 bytes --]

Re: [Question] performance regression after VM migration due to anon THP split in CoW

[Jinjiang Tu]

在 2024/6/29 17:45, David Hildenbrand 写道:
> Hi,
>
> Likely the mailing lists won‘t like my mail from this Google Mail 
> client ;)
>
> Jinjiang Tu <tujinjiang@huawei.com> schrieb am Sa. 29. Juni 2024 um 11:18:
>
>     Hi,
>
>     We noticed a performance regression in benchmark memtester[1] after
>     upgrading the kernel. THP is enabled by default
>     (/sys/kernel/mm/transparent_hugepage/enabled
>     is set to "always"). The issue arises when we migrate a virtual
>     machine
>     that has 125G total memory and 124G free memory to another host.
>     And then,
>     we run the command `memtester 120G` in the VM. The benchmark takes
>     about
>     20 seconds to consume 120G memory in v4.18, but takes about 160
>     seconds in
>     v5.10. This issue exists in mainline kernel too.
>
>
> Simple: use preallocation in QEMU. „prealloc=on“ for host memory 
> backends, for example.
>
>
>     We find commit 3917c80280c9 ("thp: change CoW semantics for anon-THP")
>     leads to the performance regression. Since this commit, When we
>     trigger a
>     write fault on a anon THP, we split the PMD and allocate a 4K
>     page, instead
>     of allocating the full anon THP. When a VM is migrating (based on
>     qemu[2]),
>     if the page is marked zero page in the source VM, the destination
>     VM will
>     call mmap and read the region to allocate memory, making the
>     region mapped
>     by the zero THP. When we run memtester in the destination VM after VM
>     migration finishes, memtester(in VM) will allocate large amounts
>     of free
>     memory and write to them, cause CoW of anon THP and THP split, further
>     cause performance regression. After reverting this commit, performance
>     regression disappears.
>
>
> You talk about COW of anon THP, whereby your scenario really only 
> relied on COW of the huge zeropage.
>
> Wouldn’t you would get a similar result when disabling the huge zeropage?
>
>
>
>     This commit optimises some scenarios such as Redis, but may lead to
>     performance regression in some other scenarios, such as VM migration.
>     How could we solve this issue? Maybe we could add a new sysctl to
>     let users
>     decide whether to CoW the full anon THP or not?
>
>
> I‘m not convinced the use case you present really warrants a toggle 
> for that. In your case you only want to change semantics on COW fault 
> to the huge zeropage. But …
>
> Using preallocation in QEMU will give you all anon THP right from the 
> start, avoiding any cow. Sure, you consume all memory right away, but 
> after all that‘s what your use case triggers either way. And it might 
> all be even faster. :)
>
> Cheers!
>
Thanks for reply. The two methods both work. But they both lead to large 
memory consumption even though the VM doesn't need so much memory right now.
>
>
>     Thanks.
>
>     [1] https://github.com/jnavila/memtester/tree/master
>     [2] https://github.com/qemu/qemu/blob/master/migration/ram.c
>
>

Re: [Question] performance regression after VM migration due to anon THP split in CoW

[David Hildenbrand]

On 04.07.24 15:31, Jinjiang Tu wrote:
> 
> 在 2024/6/29 17:45, David Hildenbrand 写道:
>> Hi,
>>
>> Likely the mailing lists won‘t like my mail from this Google Mail
>> client ;)
>>
>> Jinjiang Tu <tujinjiang@huawei.com> schrieb am Sa. 29. Juni 2024 um 11:18:
>>
>>      Hi,
>>
>>      We noticed a performance regression in benchmark memtester[1] after
>>      upgrading the kernel. THP is enabled by default
>>      (/sys/kernel/mm/transparent_hugepage/enabled
>>      is set to "always"). The issue arises when we migrate a virtual
>>      machine
>>      that has 125G total memory and 124G free memory to another host.
>>      And then,
>>      we run the command `memtester 120G` in the VM. The benchmark takes
>>      about
>>      20 seconds to consume 120G memory in v4.18, but takes about 160
>>      seconds in
>>      v5.10. This issue exists in mainline kernel too.
>>
>>
>> Simple: use preallocation in QEMU. „prealloc=on“ for host memory
>> backends, for example.
>>
>>
>>      We find commit 3917c80280c9 ("thp: change CoW semantics for anon-THP")
>>      leads to the performance regression. Since this commit, When we
>>      trigger a
>>      write fault on a anon THP, we split the PMD and allocate a 4K
>>      page, instead
>>      of allocating the full anon THP. When a VM is migrating (based on
>>      qemu[2]),
>>      if the page is marked zero page in the source VM, the destination
>>      VM will
>>      call mmap and read the region to allocate memory, making the
>>      region mapped
>>      by the zero THP. When we run memtester in the destination VM after VM
>>      migration finishes, memtester(in VM) will allocate large amounts
>>      of free
>>      memory and write to them, cause CoW of anon THP and THP split, further
>>      cause performance regression. After reverting this commit, performance
>>      regression disappears.
>>
>>
>> You talk about COW of anon THP, whereby your scenario really only
>> relied on COW of the huge zeropage.
>>
>> Wouldn’t you would get a similar result when disabling the huge zeropage?
>>
>>
>>
>>      This commit optimises some scenarios such as Redis, but may lead to
>>      performance regression in some other scenarios, such as VM migration.
>>      How could we solve this issue? Maybe we could add a new sysctl to
>>      let users
>>      decide whether to CoW the full anon THP or not?
>>
>>
>> I‘m not convinced the use case you present really warrants a toggle
>> for that. In your case you only want to change semantics on COW fault
>> to the huge zeropage. But …
>>
>> Using preallocation in QEMU will give you all anon THP right from the
>> start, avoiding any cow. Sure, you consume all memory right away, but
>> after all that‘s what your use case triggers either way. And it might
>> all be even faster. :)
>>
>> Cheers!
>>
> Thanks for reply. The two methods both work. But they both lead to large
> memory consumption even though the VM doesn't need so much memory right now.

Please see

https://lkml.kernel.org/r/1cfae0c0-96a2-4308-9c62-f7a640520242@arm.com

on a related discussion.

-- 
Cheers,

David / dhildenb