Re: [External] Re: [RFC 0/4] Introduce VDUSE - vDPA Device in Userspace

[Yongji Xie <xieyongji@bytedance.com>]

[-- Attachment #1: Type: text/plain, Size: 4234 bytes --]

On Tue, Oct 20, 2020 at 5:13 PM Jason Wang <jasowang@redhat.com> wrote:

>
> On 2020/10/20 下午4:35, Yongji Xie wrote:
> >
> >
> > On Tue, Oct 20, 2020 at 4:01 PM Jason Wang <jasowang@redhat.com
> > <mailto:jasowang@redhat.com>> wrote:
> >
> >
> >     On 2020/10/20 下午3:39, Yongji Xie wrote:
> >     >
> >     >
> >     > On Tue, Oct 20, 2020 at 11:20 AM Jason Wang <jasowang@redhat.com
> >     <mailto:jasowang@redhat.com>
> >     > <mailto:jasowang@redhat.com <mailto:jasowang@redhat.com>>> wrote:
> >     >
> >     >
> >     >     On 2020/10/19 下午10:56, Xie Yongji wrote:
> >     >     > This series introduces a framework, which can be used to
> >     implement
> >     >     > vDPA Devices in a userspace program. To implement it, the
> work
> >     >     > consist of two parts: control path emulating and data path
> >     >     offloading.
> >     >     >
> >     >     > In the control path, the VDUSE driver will make use of
> message
> >     >     > mechnism to forward the actions (get/set features, get/st
> >     status,
> >     >     > get/set config space and set virtqueue states) from
> >     virtio-vdpa
> >     >     > driver to userspace. Userspace can use read()/write() to
> >     >     > receive/reply to those control messages.
> >     >     >
> >     >     > In the data path, the VDUSE driver implements a MMU-based
> >     >     > on-chip IOMMU driver which supports both direct mapping and
> >     >     > indirect mapping with bounce buffer. Then userspace can
> access
> >     >     > those iova space via mmap(). Besides, eventfd mechnism is
> >     used to
> >     >     > trigger interrupts and forward virtqueue kicks.
> >     >
> >     >
> >     >     This is pretty interesting!
> >     >
> >     >     For vhost-vdpa, it should work, but for virtio-vdpa, I think we
> >     >     should
> >     >     carefully deal with the IOMMU/DMA ops stuffs.
> >     >
> >     >
> >     >     I notice that neither dma_map nor set_map is implemented in
> >     >     vduse_vdpa_config_ops, this means you want to let vhost-vDPA
> >     to deal
> >     >     with IOMMU domains stuffs.  Any reason for doing that?
> >     >
> >     > Actually, this series only focus on virtio-vdpa case now. To
> >     support
> >     > vhost-vdpa,  as you said, we need to implement
> >     dma_map/dma_unmap. But
> >     > there is a limit that vm's memory can't be anonymous pages which
> >     are
> >     > forbidden in vm_insert_page(). Maybe we need to add some limits on
> >     > vhost-vdpa?
> >
> >
> >     I'm not sure I get this, any reason that you want to use
> >     vm_insert_page() to VM's memory. Or do you mean you want to implement
> >     some kind of zero-copy?
> >
> >
> >
> > If my understanding is right, we will have a QEMU (VM) process and a
> > device emulation process in the vhost-vdpa case, right? When I/O
> > happens, the virtio driver in VM will put the IOVA to vring and device
> > emulation process will get the IOVA from vring. Then the device
> > emulation process will translate the IOVA to its VA to access the dma
> > buffer which resides in VM's memory. That means the device emulation
> > process needs to access VM's memory, so we should use vm_insert_page()
> > to build the page table of the device emulation process.
>
>
> Ok, I get you now. So it looks to me the that the real issue is not the
> limitation to anonymous page but see the comments above vm_insert_page():
>
> "
>
>   * The page has to be a nice clean _individual_ kernel allocation.
> "
>
> So I suspect that using vm_insert_page() to share pages between
> processes is legal. We need inputs from MM experts.
>
>
Yes,  vm_insert_page() can't be used in this case. So could we add the
shmfd into the vhost iotlb msg and pass it to the device emulation process
as a new iova_domain, just like vhost-user does.

Thanks,
Yongji


>

>
> >
> >     I guess from the software device implemention in user space it
> >     only need
> >     to receive IOVA ranges and map them in its own address space.
> >
> >
> > How to map them in its own address space if we don't use
> vm_insert_page()?
>
>

[-- Attachment #2: Type: text/html, Size: 6152 bytes --]