Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)

[Michal Hocko <mhocko@suse.com>]

On Wed 19-04-17 17:52:38, Balbir Singh wrote:
> This is a request for comments on the discussed approaches
> for coherent memory at mm-summit (some of the details are at
> https://lwn.net/Articles/717601/). The latest posted patch
> series is at https://lwn.net/Articles/713035/. I am reposting
> this as RFC, Michal Hocko suggested using HMM for CDM, but
> we believe there are stronger reasons to use the NUMA approach.
> The earlier patches for Coherent Device memory were implemented
> and designed by Anshuman Khandual.
> 
> Jerome posted HMM-CDM at https://lwn.net/Articles/713035/.
> The patches do a great deal to enable CDM with HMM, but we
> still believe that HMM with CDM is not a natural way to
> represent coherent device memory and the mm will need
> to be audited and enhanced for it to even work.
> 
> With HMM we'll see ZONE_DEVICE pages mapped into
> user space and that would mean a thorough audit of all code
> paths to make sure we are ready for such a use case and enabling
> those use cases, like with HMM CDM patch 1, which changes
> move_pages() and migration paths. I've done a quick
> evaluation to check for features and found limitationd around
> features like migration (page cache
> migration), fault handling to the right location
> (direct page cache allocation in the coherent memory), mlock
> handling, RSS accounting, memcg enforcement for pages not on LRU, etc.

Are those problems not viable to solve?

[...]
> Introduction
> 
> CDM device memory is cache coherent with system memory and we would like
> this to show up as a NUMA node, however there are certain algorithms
> that might not be currently suitable for N_COHERENT_MEMORY
> 
> 1. AutoNUMA balancing

OK, I can see a reason for that but theoretically the same applies to
cpu less numa nodes in general, no?

> 2. kswapd reclaim

How is the memory reclaim handled then? How are users expected to handle
OOM situation?

> The reason for exposing this device memory as NUMA is to simplify
> the programming model, where memory allocation via malloc() or
> mmap() for example would seamlessly work across both kinds of
> memory. Since we expect the size of device memory to be smaller
> than system RAM, we would like to control the allocation of such
> memory. The proposed mechanism reuses nodemasks and explicit
> specification of the coherent node in the nodemask for allocation
> from device memory. This implementation also allows for kernel
> level allocation via __GFP_THISNODE and existing techniques
> such as page migration to work.

so it basically resembles isol_cpus except for memory, right. I believe
scheduler people are more than unhappy about this interface...

Anyway, I consider CPUless nodes a dirty hack (especially when I see
them mostly used with poorly configured LPARs where no CPUs are left for
a particular memory).  Now this is trying to extend this concept even
further to a memory which is not reclaimable by the kernel and requires
an explicit and cooperative memory reclaim from userspace. How is this
going to work? The memory also has a different reliability properties
from RAM which user space doesn't have any clue about from the NUMA
properties exported. Or am I misunderstanding it? That all sounds quite
scary to me.

I very much agree with the last email from Mel and I would really like
to see how would a real application benefit from these nodes.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>