From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 391E2C433DB for ; Mon, 28 Dec 2020 08:14:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A9E82221F9 for ; Mon, 28 Dec 2020 08:14:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A9E82221F9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 021798D0006; Mon, 28 Dec 2020 03:14:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EED0E8D0003; Mon, 28 Dec 2020 03:14:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB4EE8D0006; Mon, 28 Dec 2020 03:14:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id C2B0A8D0003 for ; Mon, 28 Dec 2020 03:14:27 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 89750180AD806 for ; Mon, 28 Dec 2020 08:14:27 +0000 (UTC) X-FDA: 77641978974.27.rod52_570a35127492 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 6F5BD3D668 for ; Mon, 28 Dec 2020 08:14:27 +0000 (UTC) X-HE-Tag: rod52_570a35127492 X-Filterd-Recvd-Size: 10763 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Dec 2020 08:14:26 +0000 (UTC) Received: by mail-ej1-f49.google.com with SMTP id w1so13157480ejf.11 for ; Mon, 28 Dec 2020 00:14:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=pdUVhZZoD73iZZaWy47RptTdXwr6Fs6H6YNsjspMfnc=; b=rGetNqB+aGY/xVlipnoIT0fAIAGr/yBoQGgx2lPEiyYnB8gfivT8wIv8WU2aysCfDO qwlaENDU9RsIErLtxrP7YWlwbOY4L1UCHBptLr+boI3G1i38hyaR664YiyXBnPNFDBpM 0HBs4CzwS3IDlvgtQrbefhLPP7gPAD3WY4Ej6mlmUsCaEzxfAWZgZg9I/0te1+bfgQVr HAZwsSfZ1x7/hVOAEka56mBZVwVyUbrFcxglR82DUiuhWlb2y4Gl152cw5t7EuQF2Aqt 6VhsIYdjs2QV9ZD1Yf0AQG5fv4HZhQdUDJeWzVJ+QKmu+kVdfrKR8ndKH9S3zLz9sDds V/qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=pdUVhZZoD73iZZaWy47RptTdXwr6Fs6H6YNsjspMfnc=; b=UOPqnNn8OuhuxASWrifmawUGFAOVlXPjaaFBOaPoxNaw/isutcwj8xqDUXGjJt9ngZ yfGmU3LEhBJwtVqBdbHUaWOOWcOxlSJer0qCopah5UHV8usRIM877cwZECSk0SujSKcp 6zIV5V4zrxGQ4e5YfhxO/vPrBLpEvRBxDK3+9sY/SwhO7/qKCqIXmu8JXjwf+qGFSnP9 kbZOMJdsVKGENsy3cFI1PhF5pwKolZHD2S2FpjQ0OaCofJ7MwURP6JqcI70GLxA+riS1 Mu+Ys8n1JvdPQE0D4AaZblHW1W+hqSRyX9vHY6lFQd71ytVd8Ao40VCVBM4UeSm/UxeI 0EWQ== X-Gm-Message-State: AOAM533fzasPzhsvycrW++rdOu6g9IVBgSqXJIg9G7y3h6676YYWRwL8 Id0/s5j1QMAM97HrJhpyrLafO7nKMNF22kDAeP77 X-Google-Smtp-Source: ABdhPJwbxfSEbmzvq4wvRiInGgXqJZk2slatkY0WzHI97TYYQyUD8aoWsACzZHQX4SEdFBjcKHVC/Dr4TbPobxkdr1Q= X-Received: by 2002:a17:906:edc8:: with SMTP id sb8mr40936245ejb.247.1609143265352; Mon, 28 Dec 2020 00:14:25 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <20201222145221.711-10-xieyongji@bytedance.com> <6818a214-d587-4f0b-7de6-13c4e7e94ab6@redhat.com> <595fe7d6-7876-26e4-0b7c-1d63ca6d7a97@redhat.com> <0e6faf9c-117a-e23c-8d6d-488d0ec37412@redhat.com> <2b24398c-e6d9-14ec-2c0d-c303d528e377@redhat.com> In-Reply-To: <2b24398c-e6d9-14ec-2c0d-c303d528e377@redhat.com> From: Yongji Xie Date: Mon, 28 Dec 2020 16:14:14 +0800 Message-ID: Subject: Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message To: Jason Wang Cc: "Michael S. Tsirkin" , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 28, 2020 at 3:43 PM Jason Wang wrote: > > > On 2020/12/25 =E4=B8=8B=E5=8D=886:31, Yongji Xie wrote: > > On Fri, Dec 25, 2020 at 2:58 PM Jason Wang wrote: > >> > >> On 2020/12/24 =E4=B8=8B=E5=8D=883:37, Yongji Xie wrote: > >>> On Thu, Dec 24, 2020 at 10:41 AM Jason Wang wro= te: > >>>> On 2020/12/23 =E4=B8=8B=E5=8D=888:14, Yongji Xie wrote: > >>>>> On Wed, Dec 23, 2020 at 5:05 PM Jason Wang wr= ote: > >>>>>> On 2020/12/22 =E4=B8=8B=E5=8D=8810:52, Xie Yongji wrote: > >>>>>>> To support vhost-vdpa bus driver, we need a way to share the > >>>>>>> vhost-vdpa backend process's memory with the userspace VDUSE proc= ess. > >>>>>>> > >>>>>>> This patch tries to make use of the vhost iotlb message to achiev= e > >>>>>>> that. We will get the shm file from the iotlb message and pass it > >>>>>>> to the userspace VDUSE process. > >>>>>>> > >>>>>>> Signed-off-by: Xie Yongji > >>>>>>> --- > >>>>>>> Documentation/driver-api/vduse.rst | 15 +++- > >>>>>>> drivers/vdpa/vdpa_user/vduse_dev.c | 147 +++++++++++++++++++= +++++++++++++++++- > >>>>>>> include/uapi/linux/vduse.h | 11 +++ > >>>>>>> 3 files changed, 171 insertions(+), 2 deletions(-) > >>>>>>> > >>>>>>> diff --git a/Documentation/driver-api/vduse.rst b/Documentation/d= river-api/vduse.rst > >>>>>>> index 623f7b040ccf..48e4b1ba353f 100644 > >>>>>>> --- a/Documentation/driver-api/vduse.rst > >>>>>>> +++ b/Documentation/driver-api/vduse.rst > >>>>>>> @@ -46,13 +46,26 @@ The following types of messages are provided = by the VDUSE framework now: > >>>>>>> > >>>>>>> - VDUSE_GET_CONFIG: Read from device specific configuration = space > >>>>>>> > >>>>>>> +- VDUSE_UPDATE_IOTLB: Update the memory mapping in device IOTLB > >>>>>>> + > >>>>>>> +- VDUSE_INVALIDATE_IOTLB: Invalidate the memory mapping in devic= e IOTLB > >>>>>>> + > >>>>>>> Please see include/linux/vdpa.h for details. > >>>>>>> > >>>>>>> -In the data path, VDUSE framework implements a MMU-based on-chip= IOMMU > >>>>>>> +The data path of userspace vDPA device is implemented in differe= nt ways > >>>>>>> +depending on the vdpa bus to which it is attached. > >>>>>>> + > >>>>>>> +In virtio-vdpa case, VDUSE framework implements a MMU-based on-c= hip IOMMU > >>>>>>> driver which supports mapping the kernel dma buffer to a use= rspace iova > >>>>>>> region dynamically. The userspace iova region can be created= by passing > >>>>>>> the userspace vDPA device fd to mmap(2). > >>>>>>> > >>>>>>> +In vhost-vdpa case, the dma buffer is reside in a userspace memo= ry region > >>>>>>> +which will be shared to the VDUSE userspace processs via the fil= e > >>>>>>> +descriptor in VDUSE_UPDATE_IOTLB message. And the corresponding = address > >>>>>>> +mapping (IOVA of dma buffer <-> VA of the memory region) is also= included > >>>>>>> +in this message. > >>>>>>> + > >>>>>>> Besides, the eventfd mechanism is used to trigger interrupt = callbacks and > >>>>>>> receive virtqueue kicks in userspace. The following ioctls o= n the userspace > >>>>>>> vDPA device fd are provided to support that: > >>>>>>> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vd= pa_user/vduse_dev.c > >>>>>>> index b974333ed4e9..d24aaacb6008 100644 > >>>>>>> --- a/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>>>> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>>>> @@ -34,6 +34,7 @@ > >>>>>>> > >>>>>>> struct vduse_dev_msg { > >>>>>>> struct vduse_dev_request req; > >>>>>>> + struct file *iotlb_file; > >>>>>>> struct vduse_dev_response resp; > >>>>>>> struct list_head list; > >>>>>>> wait_queue_head_t waitq; > >>>>>>> @@ -325,12 +326,80 @@ static int vduse_dev_set_vq_state(struct vd= use_dev *dev, > >>>>>>> return ret; > >>>>>>> } > >>>>>>> > >>>>>>> +static int vduse_dev_update_iotlb(struct vduse_dev *dev, struct = file *file, > >>>>>>> + u64 offset, u64 iova, u64 size, u8 = perm) > >>>>>>> +{ > >>>>>>> + struct vduse_dev_msg *msg; > >>>>>>> + int ret; > >>>>>>> + > >>>>>>> + if (!size) > >>>>>>> + return -EINVAL; > >>>>>>> + > >>>>>>> + msg =3D vduse_dev_new_msg(dev, VDUSE_UPDATE_IOTLB); > >>>>>>> + msg->req.size =3D sizeof(struct vduse_iotlb); > >>>>>>> + msg->req.iotlb.offset =3D offset; > >>>>>>> + msg->req.iotlb.iova =3D iova; > >>>>>>> + msg->req.iotlb.size =3D size; > >>>>>>> + msg->req.iotlb.perm =3D perm; > >>>>>>> + msg->req.iotlb.fd =3D -1; > >>>>>>> + msg->iotlb_file =3D get_file(file); > >>>>>>> + > >>>>>>> + ret =3D vduse_dev_msg_sync(dev, msg); > >>>>>> My feeling is that we should provide consistent API for the usersp= ace > >>>>>> device to use. > >>>>>> > >>>>>> E.g we'd better carry the IOTLB message for both virtio/vhost driv= ers. > >>>>>> > >>>>>> It looks to me for virtio drivers we can still use UPDAT_IOTLB mes= sage > >>>>>> by using VDUSE file as msg->iotlb_file here. > >>>>>> > >>>>> It's OK for me. One problem is when to transfer the UPDATE_IOTLB > >>>>> message in virtio cases. > >>>> Instead of generating IOTLB messages for userspace. > >>>> > >>>> How about record the mappings (which is a common case for device hav= e > >>>> on-chip IOMMU e.g mlx5e and vdpa simlator), then we can introduce io= ctl > >>>> for userspace to query? > >>>> > >>> If so, the IOTLB UPDATE is actually triggered by ioctl, but > >>> IOTLB_INVALIDATE is triggered by the message. Is it a little odd? > >> > >> Good point. > >> > >> Some questions here: > >> > >> 1) Userspace think the IOTLB was flushed after IOTLB_INVALIDATE syscal= l > >> is returned. But this patch doesn't have this guarantee. Can this lead > >> some issues? > > I'm not sure. But should it be guaranteed in VDUSE userspace? Just > > like what vhost-user backend process does. > > > I think so. This is because the userspace device needs a way to > synchronize invalidation with its datapath. Otherwise, guest may thing > the buffer is freed to be used by other parts but the it actually can be > used by the VDUSE device. The may cause security issues. > > > > > >> 2) I think after VDUSE userspace receives IOTLB_INVALIDATE, it needs t= o > >> issue a munmap(). What if it doesn't do that? > >> > > Yes, the munmap() is needed. If it doesn't do that, VDUSE userspace > > could still access the corresponding guest memory region. > > > I see. So all the above two questions are because VHOST_IOTLB_INVALIDATE > is expected to be synchronous. This need to be solved by tweaking the > current VDUSE API or we can re-visit to go with descriptors relaying firs= t. > Actually all vdpa related operations are synchronous in current implementation. The ops.set_map/dma_map/dma_unmap should not return until the VDUSE_UPDATE_IOTLB/VDUSE_INVALIDATE_IOTLB message is replied by userspace. Could it solve this problem? Thanks, Yongji