Re: [PATCH v2 0/8] Cache coherency management subsystem

[<dan.j.williams@intel.com>]

Peter Zijlstra wrote:
> On Wed, Jul 09, 2025 at 10:32:16PM -0700, dan.j.williams@intel.com wrote:
> 
> > Theoretically there could be a threshold at which a CLFLUSHOPT loop is a
> > better option, but I would rather it be the case* that software CXL
> > cache management is stop-gap for early generation CXL platforms.
> 
> So isn't the problem that CLFLUSH and friends take a linear address
> rather than a physical address? I suppose we can use our 1:1 mapping in
> this case, is all of CXL in the 1:1 map?

Currently CXL on the unplug path does:

arch_remove_memory() /* drop direct map */
cxl_region_invalidate_memregion() /* wbinvd_on_all_cpus() */
cxl_region_decode_reset() /* physically unmap memory */

...and on the plug path:
cxl_region_decode_commit() /* physically map memory */.
cxl_region_invalidate_memregion() /* wbinvd_on_all_cpus() */
arch_add_memory() /* setup direct map */

Moving this to virtual address based flushing would need some callbacks
from the memory_hotplug code to run flushes for memory spaces that are
being physically reconfigured.

...unplug:

arch_remove_memory()
    clwb_on_all_cpus_before_unmap()
cxl_region_decode_reset()

...plug:

cxl_region_decode_commit()
arch_add_memory()
    clflushopt_on_all_cpus_before_use()

However, this raises a question in my mind. Should not all memory
hotplug drivers in the kernel be doing cache management when the
physical contents of a memory range may have changed behind a CPUs back?

Unless I am missing something it looks like the ACPI memory hotplug
driver, for example, has never considered that an unplug/replug event
may leave stale data in the CPU cache.

I note drm_clflush_pages() is existing infrastructure and perhaps CXL
should uplevel/unify on that common helper?