From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A509C433EF for ; Wed, 30 Mar 2022 18:22:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 425208D0001; Wed, 30 Mar 2022 14:22:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AD836B0073; Wed, 30 Mar 2022 14:22:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24DDE8D0001; Wed, 30 Mar 2022 14:22:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 0E9776B0072 for ; Wed, 30 Mar 2022 14:22:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BA4A94C1 for ; Wed, 30 Mar 2022 18:22:25 +0000 (UTC) X-FDA: 79301872650.11.40032B8 Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227]) by imf29.hostedemail.com (Postfix) with ESMTP id EFCA4120008 for ; Wed, 30 Mar 2022 18:22:24 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: krisman) with ESMTPSA id 5FDC81F45F1F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1648664543; bh=sYm9jFXdrgdVFCDucYlpXP1Kp/s8ldkKOJ7mfqFc47g=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=aIZcFo5g1HqWaoOkPGUjVdIx3TE8m0Li0nQHSAI+ebjOiNJ4Bttags1vgirq7SeEz rOFHGLfBfkeKZaS6uGONwAjzX9p/MrcACJlMlxxcBceoZK2oMf1oq+aiXBlGBQKeUv Zp+GzlarvnxVmLOlYnDgIcZL2B5CD1RyCZtV9V+Wtq1kyPBVxoGJHO6jiY3Sg81KI5 /tomjYsrjzmSzs95G7/aE0Fr2FDB8uq0zShFL5LqHpIDZHJfaD37O8kzh0pGei9sad ZFIcBbu76ONDUE3nQ5JmjBI81oUWjOVaSa/QNBd21I5b4sZlBUUIsUUmVSAlNzhOBd YyHBv/J9UBZhA== From: Gabriel Krisman Bertazi To: Ming Lei Cc: Hannes Reinecke , lsf-pc@lists.linux-foundation.org, linux-block@vger.kernel.org, Xiaoguang Wang , linux-mm@kvack.org Subject: Re: [LSF/MM/BPF TOPIC] block drivers in user space Organization: Collabora References: <87tucsf0sr.fsf@collabora.com> <986caf55-65d1-0755-383b-73834ec04967@suse.de> <87o81prfrg.fsf@collabora.com> <87bkxor7ye.fsf@collabora.com> Date: Wed, 30 Mar 2022 14:22:20 -0400 In-Reply-To: (Ming Lei's message of "Wed, 30 Mar 2022 09:55:56 +0800") Message-ID: <87tubfpag3.fsf@collabora.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Stat-Signature: bfsydoea6g3ief8hc6wm6xq3dy1ycd38 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EFCA4120008 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=aIZcFo5g; dmarc=pass (policy=none) header.from=collabora.com; spf=pass (imf29.hostedemail.com: domain of krisman@collabora.com designates 46.235.227.227 as permitted sender) smtp.mailfrom=krisman@collabora.com X-Rspam-User: X-HE-Tag: 1648664544-958445 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Ming Lei writes: > On Tue, Mar 29, 2022 at 01:20:57PM -0400, Gabriel Krisman Bertazi wrote: >> Ming Lei writes: >> >> >> I was thinking of something like this, or having a way for the server to >> >> only operate on the fds and do splice/sendfile. But, I don't know if it >> >> would be useful for many use cases. We also want to be able to send the >> >> data to userspace, for instance, for userspace networking. >> > >> > I understand the big point is that how to pass the io data to ubd driver's >> > request/bio pages. But splice/sendfile just transfers data between two FDs, >> > then how can the block request/bio's pages get filled with expected data? >> > Can you explain a bit in detail? >> >> Hi Ming, >> >> My idea was to split the control and dataplanes in different file >> descriptors. >> >> A queue has a fd that is mapped to a shared memory area where the >> request descriptors are. Submission/completion are done by read/writing >> the index of the request on the shared memory area. >> >> For the data plane, each request descriptor in the queue has an >> associated file descriptor to be used for data transfer, which is >> preallocated at queue creation time. I'm mapping the bio linearly, from >> offset 0, on these descriptors on .queue_rq(). Userspace operates on >> these data file descriptors with regular RW syscalls, direct splice to >> another fd or pipe, or mmap it to move data around. The data is >> available on that fd until IO is completed through the queue fd. After >> an operation is completed, the fds are reused for the next IO on that >> queue position. >> >> Hannes has pointed out the issues with fd limits. :) > > OK, thanks for the detailed explanation! > > Also you may switch to map each request queue/disk into a FD, and every > request is mapped to one fixed extent of the 'file' via rq->tag since we > have max sectors limit for each request, then fd limits can be avoided. > > But I am wondering if this way is friendly to userspace side implementation, > since there isn't buffer, only FDs visible to userspace. The advantages would be not mapping the request data in userspace if we could avoid it, since it would be possible to just forward the data inside the kernel. But my latest understanding is that most use cases will want to directly manipulate the data anyway, maybe to checksum, or even for sending through userspace networking. It is not clear to me anymore that we'd benefit from not always mapping the requests to userspace. I've been looking at your implementation and I really like how simple it is. I think it's the most promising approach for this feature I've reviewed so far. I'd like to send you a few patches for bugs I found when testing it and keep working on making it upstreamable. How can I send you those patches? Is it fine to just email you or should I also cc linux-block, even though this is yet out-of-tree code? -- Gabriel Krisman Bertazi