Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)

[Reza Arbab <arbab@linux.vnet.ibm.com>]

On Mon, May 01, 2017 at 02:56:34PM -0700, John Hubbard wrote:
>On 05/01/2017 02:04 PM, Reza Arbab wrote:
>>On Mon, May 01, 2017 at 01:41:55PM -0700, John Hubbard wrote:
>>>1. A way to move pages between NUMA nodes, both virtual address 
>>>and physical address-based, from kernel mode.
>>
>>Jerome's migrate_vma() and migrate_dma() should have this covered, 
>>including DMA-accelerated copy.
>
>Yes, that's good. I wasn't sure from this discussion here if either or 
>both of those would be used, but now I see.
>
>Are those APIs ready for moving pages between NUMA nodes? As there is 
>no NUMA node id in the API, are we relying on the pages' membership 
>(using each page and updating which node it is on)?

Yes. Those APIs work by callback. The alloc_and_copy() function you 
provide will be called at the appropriate point in the migration. Yours 
would allocate from a specific destination node, and copy using DMA.

>>>5. Something to handle the story of bringing NUMA nodes online and 
>>>putting them back offline, given that they require a device driver 
>>>that may not yet have been loaded. There are a few minor missing bits 
>>>there.
>>
>>This has been prototyped with the driver doing memory 
>>hotplug/hotremove. Could you elaborate a little on what you feel is 
>>missing?
>>
>
>We just worked through how to deal with this in our driver, and I 
>remember feeling worried about the way NUMA nodes can only be put 
>online via a user space action (through sysfs). It seemed like you'd 
>want to do that from kernel as well, when a device driver gets loaded.

That's true. I don't think we have a way to online/offline from a 
driver. To online, the alternatives are memhp_auto_online (incapable of 
doing online_movable), or udev rules (not ideal in this driver 
controlled memory use case). To offline, nothing that I know of.

>I was also uneasy about user space trying to bring a node online before 
>the associated device driver was loaded, and I think it would be nice 
>to be sure that that whole story is looked at.
>
>The theme here is that driver load/unload is, today, independent from 
>the NUMA node online/offline, and that's a problem. Not a huge one, 
>though, just worth enumerating here.

-- 
Reza Arbab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>