Re: [PATCH v16 0/2] ACPI: Add support for ACPI RAS2 feature table

[Daniel Ferguson <danielf@os.amperecomputing.com>]

> v12 -> v13:
> 1. Fixed some bugs reported and changes wanted by Borislav.
>    https://lore.kernel.org/all/20250910192707.GAaMHRCxWx37XitN3t@fat_crate.local/ 
> 
> 2. Tried modifying the patch header as commented by Borislav.
> 
> 3. Fixed a bug reported by Yazen.
>    https://lore.kernel.org/all/20250909162434.GB11602@yaz-khff2.amd.com/
> 
> 4. Changed setting 'Requested Address Range' for GET_PATROL_PARAMETERS
>    command to meet the requirements from Daniel for Ampere Computing
>    platform. 
>    https://lore.kernel.org/all/7a211c5c-174c-438b-9a98-fd47b057ea4a@os.amperecomputing.com/
> 
> 5. In RAS2 driver, removed support for scrub control attributes 'addr' and
>    'size' for the time being with the expectation that a firmware will do
>    the full node demand scrubbing and may enable these attributes in the
>    future.
>    
> 6. Add 'enable_demand' attribute to the EDAC scrub interface to start/stop
>    the demand scrub, which is used for the RAS2 demand scrub control.
> 

I have tested v16 of the driver, and it works as intended.


However, we have some concerns with the limitation that memory scrubbing can
only be invoked at the NUMA node level without specifying the address range. On
a server system with high core count and multi-terabyte DDR capacity, there are
several workloads that are limited by the total DRAM memory bandwidth available
on the system. Triggering an on-demand scrub while one of these workloads is
running on the system results in a measurable drop in workload performance if
the scrub rate is configured to be high. Triggering an on-demand scrub with low
scrub rate would be a very high latency operation due to the large amount of
memory that needs to be scrubbed.

Triggering a scrub on a NUMA node with large capacity, like multiple terabytes,
could have:

* high latency to completion of the on-demand scrub operation if configured for
low scrub rate

* high performance impact to applications that require memory bandwidth if
on-demand scrub operation is configured for high scrub rate

Additionally, there are multiple use cases we have in mind on data center
systems for being able to specify the address range to scrub from the OS. These
scenarios are primarily related to understanding the health of memory modules by
triggering on-demand memory scrub operations to single page regions.



Scenario 1:

A memory device on the server encounters CE errors; due to some failed DRAM
cells.  Over time enough errors are reported to a user-mode RAS error service,
like RASDaemon, that trigger a policy to inspect the “health” of the memory
associated with that page address. The policy triggers the inspection using an
on-demand scrub of the page to see how many errors are reported. It then
triggers the same scrub to see if the same errors are reported again. This
inspection would indicate to the RASDaemon that there is a failed DRAM cell in
this range. The policy manager decides to offline the page till the module can
be replaced when the system is serviced.

Scenario 2:

The OS has offlined a collection of pages due to UE(s) being reported on this
region due to a transient environmental issue; any reference to these pages is
poisoned by the hardware. Newer ARM ISA extensions have provisions for removal
of poison from a page. The RAS policy manager can test these pages for
functionality by triggering an on-demand memory scrub, potentially multiple
times, before adding them back into the normal memory pool.

While triggering a NUMA node level scrub would allow us to support these use
cases, the cost of doing so multiple times with a high scrub rate would be
prohibitive. At the same time, an on-demand scrub with low scrub rate may not be
fast enough to allow the policy manager to make a timely decision about the
health of the memory module.



Being able to specify the address range would allow us to contain the issue to a
smaller part of the memory with minimal impact to other users on the system.

So basically, if we can't specify the address range with on-demand scrubbing, we
might as well just use the background scrubber.

Side note about our platform: We do not allow the user to turn off both
background scrubber and on-demand scrubber. We automatically turn background
scrubber on when on-demand finishes. So, if on-demand is not running, then
background is.
That bypasses a lot of code state in the driver around whether or not background
scrubber is currently running, because you cannot turn our background scrubber
on through the driver interfaces.

Regards,
~Daniel