Re: Increase Default vm_max_map_count to Improve Compatibility with Modern Games

Re: Increase Default vm_max_map_count to Improve Compatibility with Modern Games

[Liam R. Howlett]

Adding Pierre-Loup to cc list.


* David Hildenbrand <david@redhat.com> [691231 23:00]:
> On 02.04.24 09:34, Oleksandr Natalenko wrote:
> > Hello.
> > 
> > On středa 20. března 2024 22:05:34, CEST vincentdelor@free.fr wrote:
> > > Hello,
> > > 
> > > I am writing to highlight an issue impacting many Linux users, especially those who enjoy modern gaming. The current default setting of `vm_max_map_count` at 65530 has been linked to crashes or launch failures in several contemporary games.
> > > 
> > > To address this, I have opened a detailed bug report (218616 – Increase Default vm_max_map_count to Improve Gaming Experience on Linux) available at: 218616 – Increase Default vm_max_map_count to Improve Gaming Experience on Linux (kernel.org) .
> > > 

This is change is getting more traction as distributions switch to a
higher number of VMAs.  I feel we need to educate them as to what this
really means and why it is unnecessary and wrong.

> > > 
> > > We have identified that several modern games such as Hogwarts Legacy, Star Citizen, and others experience crashes or fail to start on Linux due to the default `vm_max_map_count` being set to 65530. These issues can be mitigated by increasing the `vm_max_map_count` value to over 1048576, which has been confirmed to resolve the crashes without introducing additional bugs related to map handling.
> > > 
> > > This issue affects a wide range of users and has been noted in distributions like Fedora and Pop!_OS, which have already adjusted this value to accommodate modern gaming requirements.
> > > 
> > > For reference, here is the change for Fedora:
> > > https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount
> > > 
> > > Here is a list of games affected by this low value in vm_max_map_count as reported to Valve:
> > > 
> > > THE FINALS
> > > https://github.com/ValveSoftware/Proton/issues/7317#issuecomment-1974837850
> > > 
> > > Hogwarts Legacy
> > > https://github.com/ValveSoftware/Proton/issues/6510#issuecomment-1422781100
> > > 
> > > DayZ
> > > https://github.com/ValveSoftware/Proton/issues/3899#issuecomment-1304397069
> > > 
> > > Counter-Strike 2
> > > https://github.com/ValveSoftware/Proton/issues/2704#issuecomment-1705199788
> > > 
> > > 

Most of these do not have the vma information available anymore, if it
was there (expired pastebin links, etc).

...

> > > 
> > > **Affected Games:**
> > > - Hogwarts Legacy
> > > - Star Citizen
> > > - THE FINALS
> > > - DayZ
> > > - Counter-Strike 2
> > > - Payday 2
> > > - (and potentially others)
> > > 
> > > **References:**
> > > - Fedora's change documentation: https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount
> > > - Various user reports and confirmations on gaming performance improvement with increased `vm_max_map_count`.

Absolutely not.  This will do nothing for performance.  The game may run
vs not run, but it won't get faster.

...

> 
> Using a high VMA count usually implies that the application is doing
> something suboptimal. Having many VMAs can degrade performance of MM
> operations and result in memory waste.
> 
> Running into VMA limits usually implies that the application is not
> optimized and should handle things differently. Likely, the memory allocator
> is doing something "bad" (e.g., mmap()+munmap() instead of MADV_DONTNEED)
> and should be optimized.
> 

To be clear, what you are doing here is akin to adding more memory to
your system when there is a memory leak.  This is not the solution you
should be pushing.  Ironically, this is using more memory and performing
worse than it should.  At best, the limit increase is a workaround for
buggy programs.

At worst, you are enabling bad things to keep happening and normalising
poor programming choices.  Please put pressure on the applications that
clearly have issues.

To put this into perspective, normal applications on Linux use in the
100s.  Insane applications (chromium) use 1000 to 2000. The heavier user
is Android and that uses in the 1000s regularly (usually topping out at
around 3000). You are allowing one process to use over 65,530 vmas.

Again, this is per process.

Currently, if you run Wolfenstein II: The New Colossus (proton game),
there are 4 proton processes with 115, 104, 99, and 24 vmas.  I wanted a
newer example, but steam (or nvidia) is having A Day on my gaming PC and
won't start.

You are enabling (and normalising across multiple popular distributions)
a change from less than 120 to 1048576.  That is an increas of ~870x
of virtual memory areas - not tiny chunks of memory, areas that can span
large amounts of memory.  Assuming the *best* scenario in the buggy
programs, you'd use 65531 vmas - that is still a multiple of 546x the
number used by wolfenstein's largest thread.

> I don't think we should be raising the limit for everybody out there.

If there is an underlying technical reason for needing this number of
vmas, then it isn't provided.  I'm going to guess it's DRM/anti-cheat
that needs fixing.

I'd like the problem to please be fixed and not hide it.  You are at a
performance disadvantage with the current approach - and enabling others
to do the same.

Thank you,
Liam

Re: Increase Default vm_max_map_count to Improve Compatibility with Modern Games

[Pierre-Loup A. Griffais]

On 4/12/24 10:18 AM, Liam R. Howlett wrote:
> 
> Adding Pierre-Loup to cc list.
> 
> 
> * David Hildenbrand <david@redhat.com> [691231 23:00]:
>> On 02.04.24 09:34, Oleksandr Natalenko wrote:
>>> Hello.
>>>
>>> On středa 20. března 2024 22:05:34, CEST vincentdelor@free.fr wrote:
>>>> Hello,
>>>>
>>>> I am writing to highlight an issue impacting many Linux users, especially those who enjoy modern gaming. The current default setting of `vm_max_map_count` at 65530 has been linked to crashes or launch failures in several contemporary games.
>>>>
>>>> To address this, I have opened a detailed bug report (218616 – Increase Default vm_max_map_count to Improve Gaming Experience on Linux) available at: 218616 – Increase Default vm_max_map_count to Improve Gaming Experience on Linux (kernel.org) .
>>>>
> 
> This is change is getting more traction as distributions switch to a
> higher number of VMAs.  I feel we need to educate them as to what this
> really means and why it is unnecessary and wrong.
> 
>>>>
>>>> We have identified that several modern games such as Hogwarts Legacy, Star Citizen, and others experience crashes or fail to start on Linux due to the default `vm_max_map_count` being set to 65530. These issues can be mitigated by increasing the `vm_max_map_count` value to over 1048576, which has been confirmed to resolve the crashes without introducing additional bugs related to map handling.
>>>>
>>>> This issue affects a wide range of users and has been noted in distributions like Fedora and Pop!_OS, which have already adjusted this value to accommodate modern gaming requirements.
>>>>
>>>> For reference, here is the change for Fedora:
>>>> https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount
>>>>
>>>> Here is a list of games affected by this low value in vm_max_map_count as reported to Valve:
>>>>
>>>> THE FINALS
>>>> https://github.com/ValveSoftware/Proton/issues/7317#issuecomment-1974837850
>>>>
>>>> Hogwarts Legacy
>>>> https://github.com/ValveSoftware/Proton/issues/6510#issuecomment-1422781100
>>>>
>>>> DayZ
>>>> https://github.com/ValveSoftware/Proton/issues/3899#issuecomment-1304397069
>>>>
>>>> Counter-Strike 2
>>>> https://github.com/ValveSoftware/Proton/issues/2704#issuecomment-1705199788
>>>>
>>>>
> 
> Most of these do not have the vma information available anymore, if it
> was there (expired pastebin links, etc).
> 
> ...
> 
>>>>
>>>> **Affected Games:**
>>>> - Hogwarts Legacy
>>>> - Star Citizen
>>>> - THE FINALS
>>>> - DayZ
>>>> - Counter-Strike 2
>>>> - Payday 2
>>>> - (and potentially others)
>>>>
>>>> **References:**
>>>> - Fedora's change documentation: https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount
>>>> - Various user reports and confirmations on gaming performance improvement with increased `vm_max_map_count`.
> 
> Absolutely not.  This will do nothing for performance.  The game may run
> vs not run, but it won't get faster.

It's possible / likely the user was confused about the performance 
impact, here. The reason we make that change is to enable affected games 
to run at all, that would otherwise crash when they reach the limit.

> 
> ...
> 
>>
>> Using a high VMA count usually implies that the application is doing
>> something suboptimal. Having many VMAs can degrade performance of MM
>> operations and result in memory waste.
>>
>> Running into VMA limits usually implies that the application is not
>> optimized and should handle things differently. Likely, the memory allocator
>> is doing something "bad" (e.g., mmap()+munmap() instead of MADV_DONTNEED)
>> and should be optimized.
>>
> 
> To be clear, what you are doing here is akin to adding more memory to
> your system when there is a memory leak.  This is not the solution you
> should be pushing.  Ironically, this is using more memory and performing
> worse than it should.  At best, the limit increase is a workaround for
> buggy programs.
> 
> At worst, you are enabling bad things to keep happening and normalising
> poor programming choices.  Please put pressure on the applications that
> clearly have issues.

We don't get to prescribe what those applications do. The fact of the 
matter is that there are several high-performance memory allocators in 
wide use by game applications that make heavy internal use of mmap(), 
and that using hundreds of thousands of different memory mappings is 
well supported on the platform those applications were written for. (or 
mapping regions with different permissions, which results in different 
regions after platform translation to Linux happens within Wine)

Pointing out that there exists one game that doesn't happen to do that 
is not terribly useful for the purpose of this discussion.

The problem statement seems pretty simple - distributions that want to 
support those usecases out of the box can make that change, like we've 
done for years on SteamOS. On those that don't, users of those 
applications will have to discover and learn to apply the change by hand 
after having a likely sub-par experience trying to get their game up and 
running.

I've yet to hear a specific downside of making the change other than a 
real concern about DoS of kernel memory in another discussion - it seems 
to me like there is much lower hanging fruit for DoSing a Linux system 
you have shell access to, at the moment.

Thanks,
  - Pierre-Loup

> 
> To put this into perspective, normal applications on Linux use in the
> 100s.  Insane applications (chromium) use 1000 to 2000. The heavier user
> is Android and that uses in the 1000s regularly (usually topping out at
> around 3000). You are allowing one process to use over 65,530 vmas.
> 
> Again, this is per process.
> 
> Currently, if you run Wolfenstein II: The New Colossus (proton game),
> there are 4 proton processes with 115, 104, 99, and 24 vmas.  I wanted a
> newer example, but steam (or nvidia) is having A Day on my gaming PC and
> won't start.
> 
> You are enabling (and normalising across multiple popular distributions)
> a change from less than 120 to 1048576.  That is an increas of ~870x
> of virtual memory areas - not tiny chunks of memory, areas that can span
> large amounts of memory.  Assuming the *best* scenario in the buggy
> programs, you'd use 65531 vmas - that is still a multiple of 546x the
> number used by wolfenstein's largest thread.
> 
>> I don't think we should be raising the limit for everybody out there.
> 
> If there is an underlying technical reason for needing this number of
> vmas, then it isn't provided.  I'm going to guess it's DRM/anti-cheat
> that needs fixing.
> 
> I'd like the problem to please be fixed and not hide it.  You are at a
> performance disadvantage with the current approach - and enabling others
> to do the same.
> 
> Thank you,
> Liam
> 

Re: Increase Default vm_max_map_count to Improve Compatibility with Modern Games

[Liam R. Howlett]

* Pierre-Loup A. Griffais <pgriffais@valvesoftware.com> [240414 20:22]:
> 
> 
...

> > 
> > To be clear, what you are doing here is akin to adding more memory to
> > your system when there is a memory leak.  This is not the solution you
> > should be pushing.  Ironically, this is using more memory and performing
> > worse than it should.  At best, the limit increase is a workaround for
> > buggy programs.
> > 
> > At worst, you are enabling bad things to keep happening and normalising
> > poor programming choices.  Please put pressure on the applications that
> > clearly have issues.
> 
> We don't get to prescribe what those applications do. The fact of the matter
> is that there are several high-performance memory allocators in wide use by
> game applications that make heavy internal use of mmap(), and that using
> hundreds of thousands of different memory mappings is well supported on the
> platform those applications were written for. (or mapping regions with
> different permissions, which results in different regions after platform
> translation to Linux happens within Wine)

Thank you for the information on the situation that causes the kernel to
use such a large number of vmas.

The mmap operations will run faster if there are significantly less
vmas.  Having such a large number of objects will cause the faulting of
information into the memory to be slower, and that would hold true for
all platforms.

If this is for high-performance, then it would be unlikely that it was
designed to run with 65,530 objects to search.  It is also odd that
there are several allocators running into the same issue. If I were to
guess, the allocators are trying to bypass the operating systems use of
memory and implement another way of tracking it specific to your usecase
for speed.  It sounds like it is being translated incorrectly and
causing a monster data structure to track it on the kernel side.

If it's a translation layer in wine making a decision on how to
translate a particular set of calls then it could be fixed, or at least
examined for inefficiencies.

Either way, the performance will be sub-optimal on the page fault path
(probably the most common) and any other path that uses such a large
number of vmas.

> 
> Pointing out that there exists one game that doesn't happen to do that is
> not terribly useful for the purpose of this discussion.

I provided the data I could collect reasonably quickly, but the scale of
the difference was the important part of my statement.

> 
> The problem statement seems pretty simple - distributions that want to
> support those usecases out of the box can make that change, like we've done
> for years on SteamOS. On those that don't, users of those applications will
> have to discover and learn to apply the change by hand after having a likely
> sub-par experience trying to get their game up and running.

This number of vmas is indicating an issue with the utilisation of the
virtual memeory areas.  Increasing the limit is allowing the game to
run, but it will not be performant. It is unfortunate that the solution
was to increase the value.

> 
> I've yet to hear a specific downside of making the change other than a real
> concern about DoS of kernel memory in another discussion - it seems to me
> like there is much lower hanging fruit for DoSing a Linux system you have
> shell access to, at the moment.

Poor performance is the downside. The specific downside is the overly
large data structure that the kernel has to navigate on every page fault
or any other vma operation.  This isn't specific to changing the number,
but to the fact that it needed to be changed in the first place.

Is there an upper limit of vmas that you have seen?  Can you provide a
copy of the mappings when you see this for testing?  This works out to a
5 level maple tree.

Thanks,
Liam

Re: Increase Default vm_max_map_count to Improve Compatibility with Modern Games

[Pierre-Loup A. Griffais]

On 4/15/24 12:57 PM, Liam R. Howlett wrote:
> * Pierre-Loup A. Griffais <pgriffais@valvesoftware.com> [240414 20:22]:
>>
>>
> ...
> 
>>>
>>> To be clear, what you are doing here is akin to adding more memory to
>>> your system when there is a memory leak.  This is not the solution you
>>> should be pushing.  Ironically, this is using more memory and performing
>>> worse than it should.  At best, the limit increase is a workaround for
>>> buggy programs.
>>>
>>> At worst, you are enabling bad things to keep happening and normalising
>>> poor programming choices.  Please put pressure on the applications that
>>> clearly have issues.
>>
>> We don't get to prescribe what those applications do. The fact of the matter
>> is that there are several high-performance memory allocators in wide use by
>> game applications that make heavy internal use of mmap(), and that using
>> hundreds of thousands of different memory mappings is well supported on the
>> platform those applications were written for. (or mapping regions with
>> different permissions, which results in different regions after platform
>> translation to Linux happens within Wine)
> 
> Thank you for the information on the situation that causes the kernel to
> use such a large number of vmas.
> 
> The mmap operations will run faster if there are significantly less
> vmas.  Having such a large number of objects will cause the faulting of
> information into the memory to be slower, and that would hold true for
> all platforms.
> 
> If this is for high-performance, then it would be unlikely that it was
> designed to run with 65,530 objects to search.  It is also odd that
> there are several allocators running into the same issue. If I were to
> guess, the allocators are trying to bypass the operating systems use of
> memory and implement another way of tracking it specific to your usecase
> for speed.  It sounds like it is being translated incorrectly and
> causing a monster data structure to track it on the kernel side.
> 
> If it's a translation layer in wine making a decision on how to
> translate a particular set of calls then it could be fixed, or at least
> examined for inefficiencies.

I mentioned translation because it can play a role if the original 
mappings contain regions with different permissions, as it would need to 
translate those into several different mappings on Linux, but I wouldn't 
expect it's really having a meaningful effect. By and large, I think 
those mappings are coming as-is through the app.

> 
> Either way, the performance will be sub-optimal on the page fault path
> (probably the most common) and any other path that uses such a large
> number of vmas.
> 
>>
>> Pointing out that there exists one game that doesn't happen to do that is
>> not terribly useful for the purpose of this discussion.
> 
> I provided the data I could collect reasonably quickly, but the scale of
> the difference was the important part of my statement.
> 
>>
>> The problem statement seems pretty simple - distributions that want to
>> support those usecases out of the box can make that change, like we've done
>> for years on SteamOS. On those that don't, users of those applications will
>> have to discover and learn to apply the change by hand after having a likely
>> sub-par experience trying to get their game up and running.
> 
> This number of vmas is indicating an issue with the utilisation of the
> virtual memeory areas.  Increasing the limit is allowing the game to
> run, but it will not be performant. It is unfortunate that the solution
> was to increase the value.

Games don't necessarily care if mmap() (and ensuing faults) is a bit 
slower than the fastest case. Doing such an operation is already 
considered a relatively slow path and would likely happen on a resource 
loading thread instead of the hot main loop.

> 
>>
>> I've yet to hear a specific downside of making the change other than a real
>> concern about DoS of kernel memory in another discussion - it seems to me
>> like there is much lower hanging fruit for DoSing a Linux system you have
>> shell access to, at the moment.
> 
> Poor performance is the downside. The specific downside is the overly
> large data structure that the kernel has to navigate on every page fault
> or any other vma operation.  This isn't specific to changing the number,
> but to the fact that it needed to be changed in the first place.
> 
> Is there an upper limit of vmas that you have seen?  Can you provide a
> copy of the mappings when you see this for testing?  This works out to a
> 5 level maple tree.

I don't really know of an upper limit. I can provide a contrasting 
anecdote that seems to use a fair amount of mappings - running the title 
`Hogwarts Legacy` after having loaded into interactive gameplay in the 
initial area:

plagman@redcore:~$ cat /proc/2009007/maps | wc -l
27217

Here's a copy of /proc/maps if you're curious:

https://www.dropbox.com/scl/fi/rf970vdxoexsx8u1otufl/hogwarts_maps?rlkey=ws8uwz9ivjo6rh0y9h15nsbna&dl=0

I'm guessing there is a guard page after all of those mmap()ed 
mini-arenas the allocator creates, effectively doubling the mapping count.

Thanks,
  - Pierre-Loup

> 
> Thanks,
> Liam
>