From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3946C28B30 for ; Thu, 20 Mar 2025 21:30:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E59A1280003; Thu, 20 Mar 2025 17:30:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE20E280001; Thu, 20 Mar 2025 17:30:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C839F280003; Thu, 20 Mar 2025 17:30:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A235C280001 for ; Thu, 20 Mar 2025 17:30:36 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6D74581839 for ; Thu, 20 Mar 2025 21:30:37 +0000 (UTC) X-FDA: 83243223714.26.4ECB09E Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf22.hostedemail.com (Postfix) with ESMTP id BCC3BC0018 for ; Thu, 20 Mar 2025 21:30:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QzV8meK9; spf=pass (imf22.hostedemail.com: domain of djwong@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=djwong@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742506235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9vZpMizKgAyAMa4UOcZimNI50LLtxd0YQ5tfHL3kcvU=; b=RDcrmT/HWaQydKNh+j6zESic3pM2NzcuMEWownxzP/GSSanR2iLDn9+THIeLb4NIxMHXZA FH/OvpptAtyc4y0tX5PEUGMQzSQVcGl+RhSK0VGAKSMFbP3ZTwcjvmXc186wMIrIS19/Tw UqXOpgGdHPfLekkLSOoXiELGE2J+ZXs= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QzV8meK9; spf=pass (imf22.hostedemail.com: domain of djwong@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=djwong@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742506235; a=rsa-sha256; cv=none; b=g+csPX4tM7TgwFzRm84NvD2fs3qZgIX3L3PIYD/Jk4Sra7K0l2Z5q57bc3BDQYBkGyunWk HaZq9JGJVPpgb3hk/lVIxx4j3j7w/HavEVBn0lgydwkrGY749VTUhylH4jomGQ/MXgVsko slQW+oJAYsbsfxG9uVDU9q2+3l/mklM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 62E9E61133; Thu, 20 Mar 2025 21:30:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BEB52C4CEDD; Thu, 20 Mar 2025 21:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742506234; bh=CgP4k9ANcDRlennigDXlxcwKnu1yZe66xO2hywmqiCU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=QzV8meK9/w+yYwh3ZBWM8Gffl7+cYsZkDptGBNJtxYtbcsK3/i5Dl5EyWG+TzDlIg tJiMr3QwOx03nB7BN41036DgwHAgN+ZzX7x58hu6US40VQkjWuLw3yXoOAW4K8+EOw XUJP8YrxETf3DYBnPR1Ew6s42+VSEG0svtfui2gWYicr/SQ7g+zVfF+BM+KLlI7sTP C2jdAVaoCv4y83XHvlq2+YO9Mmd1IZsfB5Y58iB/iQSu+8006NOXV5VLJUfNyHimyh sej5PtejGRksl0YRmHSUQIuygNw1DsYYDneN35uwXk0leAae0ejqZFddwSKkWTXOsD sqbgLXqiFBfhA== Date: Thu, 20 Mar 2025 14:30:34 -0700 From: "Darrick J. Wong" To: Ritesh Harjani Cc: Luis Chamberlain , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, lsf-pc@lists.linux-foundation.org, david@fromorbit.com, leon@kernel.org, hch@lst.de, kbusch@kernel.org, sagi@grimberg.me, axboe@kernel.dk, joro@8bytes.org, brauner@kernel.org, hare@suse.de, willy@infradead.org, john.g.garry@oracle.com, p.raghav@samsung.com, gost.dev@samsung.com, da.gomez@samsung.com Subject: Re: [LSF/MM/BPF TOPIC] breaking the 512 KiB IO boundary on x86_64 Message-ID: <20250320213034.GG2803730@frogsfrogsfrogs> References: <87o6xvsfp7.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87o6xvsfp7.fsf@gmail.com> X-Rspamd-Queue-Id: BCC3BC0018 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: wesimr1poxxty88tt4nt3ne865oj6dqy X-HE-Tag: 1742506235-983723 X-HE-Meta: U2FsdGVkX19qdQfPyyq/t6BFW3QJYbT7D8yVcQ2h43fSEU4BUBQQbUHzNyby0DCTvuCckANIfokexe74mZynviYcGzvyXNVL9itwlRiEIQ3msczegxUJk3r/dL2CYRAeu4Xlj4OkDmX2XtsfNoa8e/zZNuq3Eos3APbW/LFpBYLFg8Ljv/i7jkUE1oXkSLGsaZTW4AG2KwnEoXC2dPBqPNIVtFfoxkiwJqR7Ezf8zmtpvQ4+isNsL3GtyDuw4tfDyw/zVDwxXBaw2FbpBO0dMRgTEVd8tT42DONE/c7zqSmuG36eIM4T5xdKEOxgWgEXoFLlBjykFBzbU3cZDg/tP+loq8AVXNz4PvM+YfFTDoBoJ+Qx5rYCoh89vPEyMhcw1TO26mUCINhOcJ6RFw/63RP5yKPNTf8GWflY28wqaS0Is99jIpWVq03Vf8oxPedn98ESpMYVtH8wM9xF/kQbr/l/CTPzwoCQShnZFAnm0ZBnd2GHrgIa0mbiDUG6h4VS1kB9YPGxC4/MBAHZZ5jLbcPGr++Zniq9uCYnrfXJQNUQIbLGL4SGFqUZk9blGmNHgN7Tibn3TFrewBEsXWFVgVmpwq4biy+VkTT1op3e7d6dfht9KcLJ8pa11M2UjHmkZJUgzNTRgwhX+C27qCYSVGBg1MQOHG82SR752SD5ZjkR2Qv1p2zU3WReTPPJuO8g5uOR4KRyI0XTv8OzsmisnDhPrPRqhSH6tMXlernYA0knimINtP19V4MqOErc2yCcPudWhwdUwdk5YR5uh52y/e6Dmcvg13iRskax0FU0SW7zCKu95OfN23/qdMQ4uwbviLBFYdTRAAu09FT0paP9Wdsi1o0Ir+sIof3bjWoq7gBh3sOZqZpZ2LZRUIU4u06vBIXsLQ2y9udpaedjBsmanTRrLK1hMLt4zRV7uFZxJ+EM4prviC28oL/5mNUYJmlOvZN9yIlv4mq0gQGbcpq lhqsf5n8 Dzn1Bkuca8iUKemQndqpq4t7Nxhj846CmeFpC0gpXh9wxDGbdBaQEtJXB3vlNyxcwBrX1dBZ5h0/a3FXwL1PvlsM23h/KF3UrGTYd2Y0ZKj5QriXccnT3/uvsZ4X39oo0ZZx6wM+KK/xmOxE4rziMRzwer1OVBmipfuRPu0tvon3cQqwkdVvFHsfa7p+L7E+2s2GJWOFPflWtwNzlL4kaLh37ukZRnSsMJe9pmd9mRYt+x3NGxnq1xqElkjgjI3eKTUBkgMsGEBar0Hrvffn985xw8DOSdZ0YPnziNlZggj5lx9wVXHGDtoAllwFGV5uFFUDQXplJ1yM/rOiwVOAL7UDhsKaR6p2Tmnz88JtJBqqZxTsdaijShgkfuVi2WbJheXMDC7oX9qEZKfxNpA+A6CkO9uo4s7A6cILvUQ+Wd5OxXB3fa3tt9VZPtYo9TV3X+yqhCgDVXODzKWA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Mar 21, 2025 at 12:16:28AM +0530, Ritesh Harjani wrote: > Luis Chamberlain writes: > > > We've been constrained to a max single 512 KiB IO for a while now on x86_64. > > This is due to the number of DMA segments and the segment size. With LBS the > > segments can be much bigger without using huge pages, and so on a 64 KiB > > block size filesystem you can now see 2 MiB IOs when using buffered IO. > > But direct IO is still crippled, because allocations are from anonymous > > memory, and unless you are using mTHP you won't get large folios. mTHP > > is also non-deterministic, and so you end up in a worse situation for > > direct IO if you want to rely on large folios, as you may *sometimes* > > end up with large folios and sometimes you might not. IO patterns can > > therefore be erratic. > > > > As I just posted in a simple RFC [0], I believe the two step DMA API > > helps resolve this. Provided we move the block integrity stuff to the > > new DMA API as well, the only patches really needed to support larger > > IOs for direct IO for NVMe are: > > > > iomap: use BLK_MAX_BLOCK_SIZE for the iomap zero page > > blkdev: lift BLK_MAX_BLOCK_SIZE to page cache limit > > Maybe some naive questions, however I would like some help from people > who could confirm if my understanding here is correct or not. > > Given that we now support large folios in buffered I/O directly on raw > block devices, applications must carefully serialize direct I/O and > buffered I/O operations on these devices, right? > > IIUC. until now, mixing buffered I/O and direct I/O (for doing I/O on > /dev/xxx) on separate boundaries (blocksize == pagesize) worked fine, > since direct I/O would only invalidate its corresponding page in the > page cache. This assumes that both direct I/O and buffered I/O use the > same blocksize and pagesize (e.g. both using 4K or both using 64K). > However with large folios now introduced in the buffered I/O path for > block devices, direct I/O may end up invalidating an entire large folio, > which could span across a region where an ongoing direct I/O operation I don't understand the question. Should this read ^^^ "buffered"? As in, directio submits its write bio, meanwhile another thread initiates a buffered write nearby, the write gets a 2MB folio, and then the post-write invalidation knocks down the entire large folio? Even though the two ranges written are (say) 256k apart? --D > is taking place. That means, with large folio support in block devices, > application developers must now ensure that direct I/O and buffered I/O > operations on block devices are properly serialized, correct? > > I was looking at posix page [1] and I don't think posix standard defines > the semantics for operations on block devices. So it is really upto the > individual OS implementation, correct? > > And IIUC, what Linux recommends is to never mix any kind of direct-io > and buffered-io when doing I/O on raw block devices, but I cannot find > this recommendation in any Documentation? So can someone please point me > one where we recommend this? > > [1]: https://pubs.opengroup.org/onlinepubs/9799919799/ > > > -ritesh > > > > > The other two nvme-pci patches in that series are to just help with > > experimentation now and they can be ignored. > > > > It does beg a few questions: > > > > - How are we computing the new max single IO anyway? Are we really > > bounded only by what devices support? > > - Do we believe this is the step in the right direction? > > - Is 2 MiB a sensible max block sector size limit for the next few years? > > - What other considerations should we have? > > - Do we want something more deterministic for large folios for direct IO? > > > > [0] https://lkml.kernel.org/r/20250320111328.2841690-1-mcgrof@kernel.org > > > > Luis >