Re: [HMM-v25 00/19] HMM (Heterogeneous Memory Management) v25

["Figo.zhang" <figo1802@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7383 bytes --]

2017-12-14 0:12 GMT+08:00 Jerome Glisse <jglisse@redhat.com>:

> On Wed, Dec 13, 2017 at 08:10:42PM +0800, Figo.zhang wrote:
> > it mention that HMM provided two functions:
> >
> > 1. one is mirror the page table between CPU and a device(like GPU, FPGA),
> > so i am confused that:  duplicating the CPU page table into a device page
> > table means
> >
> > that copy the CPU page table entry into device page table? so the device
> > can access the CPU's virtual address? and that device can access the the
> CPU
> > allocated physical memory which map into this VMA, right?
> >
> > for example:  VMA -> PA (CPU physical address)
> >
> > mirror: fill the the PTE entry of this VMA into GPU's  page table
> >
> > so:
> >
> > For CPU's view: it can access the PA
> >
> > For GPU's view: it can access the CPU's VMA and PA
> >
> > right?
>
> Correct. This is for platform/device without ATS/PASID. Note that
> HMM only provide helpers to snapshot the CPU page table and properly
> synchronize with concurrent CPU page table update. Most of the code
> is really inside the device driver as each device driver has its own
> architecture and its own page table format.
>
>
> > 2. other function is migrate CPU memory to device memory, what is the
> > application scenario ?
>
> Second part of HMM is to allow to register "special" struct page
> (they are not on the LRU and are associated with a device). Having
> struct page allow most of the kernel memory management to be
> oblivous to the underlying memory type (regular DDR memort or device
> memory).
>
> The migrate helper introduced with HMM is to allow to migrate to
> and from device memory using DMA engine and not CPU memcopy. It
> is needed because on some platform CPU can not access the device
> memory and moreover DMA engine reach higher bandwidth more easily
> than CPU memcopy.
>
> Again this is an helper. The policy on what to migrate, when, ...
> is outside HMM for now we assume that the device driver is the
> best place to have this logic. Maybe in few year once we have more
> device driver using that kind of feature we may grow common code
> to expose common API to userspace for migration policy.


>
> > some input data created by GPU and migrate back to CPU memory? use for
> CPU
> > to access GPU's data?
>
> It can be use in any number of way. So yes all the scenario you
> list do apply. On platform where CPU can not access device memory
> you need to migrate back to regular memory for CPU access.
>
> Note that the physical memory use behind a virtual address pointer
> is transparent to the application thus you do not need to modify
> it in anyway. That is the whole point of HMM.
>
>
> > 3. function one is help the GPU to access CPU's VMA and  CPU's physical
> > memory, if CPU want to access GPU's memory, still need to
> > specification device driver API like IOCTL+mmap+DMA?
>
> No you do not need special memory allocator with an HMM capable
> device (and device driver). HMM mirror functionality is to allow
> any pointer to point to same memory on both a device and CPU for
> a given application. This is the Fine-Grained system SVM as
> specified in the OpenCL 2.0 specification.
>
> Basic example is without HMM:
>     mul_mat_on_gpu(float *r, float *a, float *b, unsigned m)
>     {
>         gpu_buffer_t gpu_r, gpu_a, gpu_b;
>
>         gpu_r = gpu_alloc(m*m*sizeof(float));
>         gpu_a = gpu_alloc(m*m*sizeof(float));
>         gpu_b = gpu_alloc(m*m*sizeof(float));
>         gpu_copy_to(gpu_a, a, m*m*sizeof(float));
>         gpu_copy_to(gpu_b, b, m*m*sizeof(float));
>
>         gpu_mul_mat(gpu_r, gpu_a, gpu_b, m);
>
>         gpu_copy_from(gpu_r, r, m*m*sizeof(float));
>     }
>

The traditional workflow is:
1. the pointer a, b and r are total point to the CPU memory
2. create/alloc three GPU buffers: gpu_a, gpu_b, gpu_r
3. copy CPU memory a and b to GPU memory gpu_b and gpu_b
4. let the GPU to do the calculation
5.  copy the result from GPU buffer (gpu_r) to CPU buffer (r)

is it right?


>
> With HMM:
>     mul_mat_on_gpu(float *r, float *a, float *b, unsigned m)
>     {
>         gpu_mul_mat(r, a, b, m);
>     }
>

with HMM workflow:
1. CPU has three buffer: a, b, r, and it is physical addr is : pa, pb, pr
     and GPU has tree physical buffer: gpu_a, gpu_b, gpu_r
2. GPU want to access buffer a and b, cause a GPU page fault
3. GPU report a page fault to CPU
4. CPU got a GPU page fault:
                * unmap the buffer a,b,r (who do it? GPU driver?)
                * copy the buffer a ,b's content to GPU physical buffers:
gpu_a, gpu_b
                * fill the GPU page table entry with these pages (gpu_a,
gpu_b, gpu_r) of the CPU virtual address: a,b,r;

5. GPU do the calculation
6. CPU want to get result from buffer r and will cause a CPU page fault:
7. in CPU page fault:
             * unmap the GPU page table entry for virtual address a,b,r.
(who do the unmap? GPU driver?)
             * copy the GPU's buffer content (gpu_a, gpu_b, gpu_r) to
CPU buffer (abr)
             * fill the CPU page table entry: virtual_addr -> buffer
(pa,pb,pr)
8. so the CPU can get the result form buffer r.

my guess workflow is right?
it seems need two copy, from CPU to GPU, and then GPU to CPU for result.
* is it CPU and GPU have the  page table concurrently, so
no page fault occur?
* how about the performance? it sounds will create lots of page fault.


> So it is going from a world with device specific allocation
> to a model where any regular process memory (outcome of an
> mmap to a regular file or for anonymous private memory). can
> be access by both CPU and GPU using same pointer.
>
>
> Now on platform like PCIE where CPU can not access the device
> memory in cache coherent way (also with no garanty regarding
> CPU atomic operations) you can not have the CPU access the device
> memory without breaking memory model expected by the programmer.
>
> Hence if some range of the virtual address space of a process
> has been migrated to device memory it must be migrated back to
> regular memory.
>
> On platform like CAPI or CCIX you do not need to migrate back
> to regular memory on CPU access. HMM provide helpers to handle
> both cases.
>
>
> > 4. is it any example? i remember it has a dummy driver in older patchset
> > version. i canot find in this version.
>
> Dummy driver and userspace:
> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-next
> https://github.com/glisse/hmm-dummy-test-suite
>
> nouveau prototype:
> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
>
> I was waiting on rework of nouveau memory management before
> working on final implementation of HMM inside nouveau. That
> rework is now mostly ready:
>
> https://github.com/skeggsb/nouveau/tree/devel-fault
>
> I intend to start working on final HMM inside nouveau after
> the end of year celebration and i hope to have it in some
> working state in couple month. At the same time we are working
> on an open source userspace to make use of that (probably
> an OpenCL runtime first but we are looking into other thing
> such as OpenMP, CUDA, ...).
>
> Plans is to upstream all this next year, all the bits are
> slowly cooking.
>
> Cheers,
> Jérôme
>

[-- Attachment #2: Type: text/html, Size: 9642 bytes --]