From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDFA0C35FFF for ; Fri, 21 Mar 2025 02:26:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08521280002; Thu, 20 Mar 2025 22:26:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 036E9280001; Thu, 20 Mar 2025 22:26:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3EB8280002; Thu, 20 Mar 2025 22:26:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C11FF280001 for ; Thu, 20 Mar 2025 22:26:17 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 618C8C0D10 for ; Fri, 21 Mar 2025 02:26:19 +0000 (UTC) X-FDA: 83243968878.04.44B76B3 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf07.hostedemail.com (Postfix) with ESMTP id CDEB840002 for ; Fri, 21 Mar 2025 02:26:17 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IovbopYe; spf=pass (imf07.hostedemail.com: domain of ritesh.list@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=ritesh.list@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742523977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=GXCXUHTUv85OtHijOMda+VCkM60vFQI9b1pD1y/4h8s=; b=KSypeXEJ83zUKvEi774DX7tfB+ANAMMSpNGnJFf6PY5viiue25COfHeOwROqJE3kMk2LWn x2hk+sUYlBjpKLk6okbBh/Qi0G26OBrI+MOQcfkcklRfZph8Wa9KKTohT1POHg2u9sUBUw SAY8U96p2Wgznj9tKYchyTJc9/JK/e4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IovbopYe; spf=pass (imf07.hostedemail.com: domain of ritesh.list@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=ritesh.list@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742523977; a=rsa-sha256; cv=none; b=1Fz83EIBJ0UZSxQhfzVRj+9FRetwao4sDe8NE1htGlJVKCKZfAETLE+HNd+82on+T415Dl QyMGCS+tKd/ArAgZPeQ0WHeRcfisPFcYFHHQUKbMTUO83lL09r/WMv142/YFAL1La/8x2T PMLZTzNBzUfv/kVGtdaxCr9QPeiWUYU= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-226185948ffso29257835ad.0 for ; Thu, 20 Mar 2025 19:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742523976; x=1743128776; darn=kvack.org; h=references:message-id:date:in-reply-to:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=GXCXUHTUv85OtHijOMda+VCkM60vFQI9b1pD1y/4h8s=; b=IovbopYeD9z7+k95Xf6mdjPtumOK48UJ6SYcw5KfWnP53glkD9ZdS1UOUTuDWLayNt d2OPYYLgT7U1nz/nxPSLJjk2hEnoh8HIoPGR+emuAk1d4MivxuUDoGp9ZAudLA95yU6P KOcnTbaagvVAtwFt4l1cDQWR+2QyWVo7aYjiGeb2v3YmZ0EpOZTudc2rvbsJsQgYVE8o 6iVUep6IKxzgnEWAaedLXnJPPPgfvvpYTQ5KMjRyNaqQu82s/Y74/IsIDSp3vzbIn65c FvsJ4WHH5YLtTki7eWFTvbdFFRITVSuELkMUqzOnxfPfRHPHzKdDlLVvLZE5MZlQDOQv HHHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742523976; x=1743128776; h=references:message-id:date:in-reply-to:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GXCXUHTUv85OtHijOMda+VCkM60vFQI9b1pD1y/4h8s=; b=oyRUlhxeoKG248dZGfZPglq+KXE9Lmu9Vbx5CweD1n2AejXbzkRAijxyDhTmJz06+J BPANqaHmCm8RT5c4iajrTo/M42ISbawh0UKZN4fxfLJ455ad4CD1K4AEHATMwiIrdhvq +pb4ETdrwdVmJKd2sQuF+cNC4IJlgibKuXWLvNlip3xfmNTR3FW3xIO2FMcd9r++8MEQ doIPIDnJya5sSmXUy59ZbsiBar2v6/WFY3Eblq1zpFAVH0vMXN3OSH/5+GVkhlzy4nmC MYj/vxfHGLTeKjPKaMidgdGciqTcj1Evckb6Wq9f+zC9SSq8Q8a26dafvvAZH5HCjrDI 10dg== X-Forwarded-Encrypted: i=1; AJvYcCVHIhLuQfAwHcT5Px1wxapE7y8UN9CDROdxdo3UlZvnQuY6dq3aI0XHZyuHGHfPqJGrM8sEBkTFXQ==@kvack.org X-Gm-Message-State: AOJu0Yx3sJV3qZhMmMTpdya7yIrHZCQWzhPCUuMCqk0QOLGKcwD/kS+l ZebyRUhL2HFAXSeHOxSNDxzP3GKy55H2UUczbS4SEodif85SR8TU X-Gm-Gg: ASbGncu/bSBLrpihQQP90J+CwFRo8yRFGdbGhaV/qFq0H2HPEXXMMC0Lsv5hF+ky/ut 4Qi3X2feyVPujukgzLg7kux4iesEdWyTYl0vArPhSNGRIWblcY1xW890p/qnPrWeKn5y7fu0qW1 tRW/Gw7sN+v8h0RSl1UxG9kpl1Ju11i1a5B91kJLcP4l6VmnYzqvh4MBTLNi68JMlWga+EAqtFK vPuVis9JqVx54zASCJRtRNoJZzqAmgDrznqllS4XTqeV3yimFJ+VwZnfZ4ZwgRGHAQN5XGXuZS2 SGZ9dAzEYJushnSuSgGF3uUStSV4y3SqR7xdeA== X-Google-Smtp-Source: AGHT+IFqmFLsmg9/ahn7SjRWI3Rn/0OLHyEDGSMmYz4To8UsuGaDaTjz2ptscspGRtgdnCPweTQAog== X-Received: by 2002:a17:902:f64b:b0:220:c4e8:3b9f with SMTP id d9443c01a7336-22780bb1312mr26955965ad.0.1742523976363; Thu, 20 Mar 2025 19:26:16 -0700 (PDT) Received: from dw-tp ([171.76.82.198]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-227811da561sm5313945ad.185.2025.03.20.19.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 19:26:15 -0700 (PDT) From: Ritesh Harjani (IBM) To: "Darrick J. Wong" Cc: Luis Chamberlain , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, lsf-pc@lists.linux-foundation.org, david@fromorbit.com, leon@kernel.org, hch@lst.de, kbusch@kernel.org, sagi@grimberg.me, axboe@kernel.dk, joro@8bytes.org, brauner@kernel.org, hare@suse.de, willy@infradead.org, john.g.garry@oracle.com, p.raghav@samsung.com, gost.dev@samsung.com, da.gomez@samsung.com Subject: Re: [LSF/MM/BPF TOPIC] breaking the 512 KiB IO boundary on x86_64 In-Reply-To: <20250320213034.GG2803730@frogsfrogsfrogs> Date: Fri, 21 Mar 2025 07:43:09 +0530 Message-ID: <87jz8jrv0q.fsf@gmail.com> References: <87o6xvsfp7.fsf@gmail.com> <20250320213034.GG2803730@frogsfrogsfrogs> X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: CDEB840002 X-Stat-Signature: j5rojjewnjm7do6ittkfbtp55e3o4may X-HE-Tag: 1742523977-743947 X-HE-Meta: U2FsdGVkX18FHKOd+FfojlrFEJ0uERgKU2WSYbWkUf6tu5hVgzA4XhQGYqLo+a0kSjrpTQv+295ytvw3RP671Z+JJZUCiT1T8tMUPFnKELByk3mxsAryoAJgfQZ73HNXlI+3owRuhgNJ3hpIwvrVR9qeMaPyibre6fPGstKZ/5ko8NkUkwKncxzxwaiI1BhuemBwWgIdde1GsPMNSOO4GgGFjNIlf29dMzSbw11/gmvB4EY+1iNIqROSiCLzlYjv7YHbr1xvVDr91WMtp20Pj6Blr17jSLEqvKMNyCvXzeu62mT7L5r1dd0p923LhcMUO5CdYiuhro6iqcarpjJb5sYVw5j1CLnakJvkdSeK4FFGLuuaJ/2oep4uq+a0y4xuVFZMgpcA5Bg226eBvoBYTor6Pld2kDsF/P/Dq9zMQ/n6buCf3Wdn9GBQkRWiRTGmO50sDAD+s0cdqNNhOKTUJ4R6ZIX0AljeutMTVQvQVGNhLvy7dDa25kkQfVyEGmCBz4ZW82wyLAY39WnixBn78FmiI8E8jU7h08MrY5g1zFqzVuL2IXCWq51igskyrxZmOPfGIZlumo0BZ558c8+s25R1ie/E4T+wn8fumWHRM72M7Rt2xh7gduhrylfh4Eo7gqgjLToPwR9Dp14RBUMIqXliZESs+37Pewhz/YVP0C5jJJ1slzXFn0q4x4VU2Y7IBhlTRoNkuoJpXZnjbj0a6cJ+5c976ibq/BZMpgty8iRq8yt08HIGMji/0vkq1uZ82tL8yTdAdoj4cYv7qfZ77LXo2EVnQNfX4qKaJCsQm1rp8rouextLXNPv2bdBmLUmM1IeTbHXaAHgEd5hVdsTEsqz9cs3yHh+HKRSXkYw3w+vg9sbwokhq/zmza1FBxAawc+QowMh6FNAZIMLUs1N4Q2Ngu553ly9gHYUWT/Vf2u+tEuC1Vq6bfZvNXQH1LuDbdsqdzJTvLoKcCpWzJb SSypfXEr 2NT3IPJeTp3GHY1DRFkBBIFYx+0I7ngGl7ALeczJwdp+3wS9Zo2Lw9/bolT6zNAckfqTBw/ib2SFer8jz4DgTY+/8uvbU2QcTYfLO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: "Darrick J. Wong" writes: > On Fri, Mar 21, 2025 at 12:16:28AM +0530, Ritesh Harjani wrote: >> Luis Chamberlain writes: >> >> > We've been constrained to a max single 512 KiB IO for a while now on x86_64. >> > This is due to the number of DMA segments and the segment size. With LBS the >> > segments can be much bigger without using huge pages, and so on a 64 KiB >> > block size filesystem you can now see 2 MiB IOs when using buffered IO. >> > But direct IO is still crippled, because allocations are from anonymous >> > memory, and unless you are using mTHP you won't get large folios. mTHP >> > is also non-deterministic, and so you end up in a worse situation for >> > direct IO if you want to rely on large folios, as you may *sometimes* >> > end up with large folios and sometimes you might not. IO patterns can >> > therefore be erratic. >> > >> > As I just posted in a simple RFC [0], I believe the two step DMA API >> > helps resolve this. Provided we move the block integrity stuff to the >> > new DMA API as well, the only patches really needed to support larger >> > IOs for direct IO for NVMe are: >> > >> > iomap: use BLK_MAX_BLOCK_SIZE for the iomap zero page >> > blkdev: lift BLK_MAX_BLOCK_SIZE to page cache limit >> >> Maybe some naive questions, however I would like some help from people >> who could confirm if my understanding here is correct or not. >> >> Given that we now support large folios in buffered I/O directly on raw >> block devices, applications must carefully serialize direct I/O and >> buffered I/O operations on these devices, right? >> >> IIUC. until now, mixing buffered I/O and direct I/O (for doing I/O on >> /dev/xxx) on separate boundaries (blocksize == pagesize) worked fine, >> since direct I/O would only invalidate its corresponding page in the >> page cache. This assumes that both direct I/O and buffered I/O use the >> same blocksize and pagesize (e.g. both using 4K or both using 64K). >> However with large folios now introduced in the buffered I/O path for >> block devices, direct I/O may end up invalidating an entire large folio, >> which could span across a region where an ongoing direct I/O operation > > I don't understand the question. Should this read ^^^ "buffered"? oops, yes. > As in, directio submits its write bio, meanwhile another thread > initiates a buffered write nearby, the write gets a 2MB folio, and > then the post-write invalidation knocks down the entire large folio? > Even though the two ranges written are (say) 256k apart? > Yes, Darrick. That is my question. i.e. w/o large folios in block devices one could do direct-io & buffered-io in parallel even just next to each other (assuming 4k pagesize). |4k-direct-io | 4k-buffered-io | However with large folios now supported in buffered-io path for block devices, the application cannot submit such direct-io + buffered-io pattern in parallel. Since direct-io can end up invalidating the folio spanning over it's 4k range, on which buffered-io is in progress. So now applications need to be careful to not submit any direct-io & buffered-io in parallel with such above patterns on a raw block device, correct? That is what I would like to confirm. > --D > >> is taking place. That means, with large folio support in block devices, >> application developers must now ensure that direct I/O and buffered I/O >> operations on block devices are properly serialized, correct? >> >> I was looking at posix page [1] and I don't think posix standard defines >> the semantics for operations on block devices. So it is really upto the >> individual OS implementation, correct? >> >> And IIUC, what Linux recommends is to never mix any kind of direct-io >> and buffered-io when doing I/O on raw block devices, but I cannot find >> this recommendation in any Documentation? So can someone please point me >> one where we recommend this? And this ^^^ -ritesh >> >> [1]: https://pubs.opengroup.org/onlinepubs/9799919799/ >> >> >> -ritesh >> >> > >> > The other two nvme-pci patches in that series are to just help with >> > experimentation now and they can be ignored. >> > >> > It does beg a few questions: >> > >> > - How are we computing the new max single IO anyway? Are we really >> > bounded only by what devices support? >> > - Do we believe this is the step in the right direction? >> > - Is 2 MiB a sensible max block sector size limit for the next few years? >> > - What other considerations should we have? >> > - Do we want something more deterministic for large folios for direct IO? >> > >> > [0] https://lkml.kernel.org/r/20250320111328.2841690-1-mcgrof@kernel.org >> > >> > Luis >>