From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9CDDC36005 for ; Mon, 28 Apr 2025 09:24:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39DAC6B0095; Mon, 28 Apr 2025 05:24:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34FA26B0096; Mon, 28 Apr 2025 05:24:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 155A06B0099; Mon, 28 Apr 2025 05:24:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DFA286B0095 for ; Mon, 28 Apr 2025 05:24:01 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 227E8811B9 for ; Mon, 28 Apr 2025 09:24:03 +0000 (UTC) X-FDA: 83382915966.30.0AF6886 Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf16.hostedemail.com (Postfix) with ESMTP id 8D779180013 for ; Mon, 28 Apr 2025 09:24:01 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oo6f8Lit; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of leon@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745832241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P3z0T+ZgkMIUWqLcWm2tKeR9BajP2OyD9df/TDQ/JBM=; b=3KOLbR2CEyyeh6u9EcIddYda/sztfC4k1tMF2IzRA822SkGWXQrXEH1D6gO908hH/kcEZ9 cPigmKds4Ka1AvQJE83lT1MUgMA1WEr3C3RA4ZeXQ8uUnyjc0N/s688eKEJrXZs7lGZLzt bC5jUFqDREbBiDGTLwC20r3CBelhAUY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745832241; a=rsa-sha256; cv=none; b=B2v0rhPS5QEeQiscvwqe4J5iW+WaGGsCZ94BKMN4UfJtCKfhSu1gxt8DihKo0oSBj3PQ2r 7ohtgI9X5aXtXIiWQNlGMfkMNEFlro1K3/BNsCyEy/YYMX0janPSDhqYslPV+970WVRZ7L 7NG6rmIaiTs/FK3hewl4bhV1uULtji8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oo6f8Lit; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of leon@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=leon@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 57EC761162; Mon, 28 Apr 2025 09:23:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11CAFC4CEEE; Mon, 28 Apr 2025 09:24:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832240; bh=cn4ZAo5vXix5AGqKt4IWuh2vUXjY6lms9OP/AMjOBBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oo6f8LitBAP+zsGFjpfl+0mKIJC5Ky3ldfas2CEBRMfKaB8MBnnjLgNzm9zM6tZmT gFqE6/2vvpVC3VBAQvhHlTGEYrSHDsE1l5GlwZAkFZ61myGlxGalP4JwuRBNe5Xlms 0LasKT819i+2DFexVc3tcm6U24G3gzB7SPTdpUOmZ0/AxBENdqtZKvrWiW/XO5sPNm kg3YYe08XQScGeBLTSqR5fK19dmrijYQZUR1wb+msSqBR2zHb8K53hQh1vE3wec+WB v8fzSVgfYh4rO84M4uuhfM4r7TSXXi2mlVErAce+d2QxOulbFGJ5HrsUWjnM4mRpF7 d145U3IkIKPog== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 19/24] block: don't merge different kinds of P2P transfers in a single bio Date: Mon, 28 Apr 2025 12:22:25 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8D779180013 X-Rspam-User: X-Stat-Signature: 3tbtmigg3ifdxr6maz4ccexa3mtdxuc1 X-HE-Tag: 1745832241-151879 X-HE-Meta: U2FsdGVkX19W7bKrpfmHgm2HPvQGItbvL1j8WtkZrpI4QejfoLbe2oogo4KvV32TUiWJcZAJFMRyqdH1+T+nMAIftM78tKE3slVSW4dQMoE7Vi2/eHWsTcBJpwVUz5Nv3MMseR4+kfKe5+6Y1GLMBWcrRvW623DK4/Ep+hY4feCfPNtWu4INXRtGPb4qr7BC5ix0zdiqwM6HAe2u+eZ0eZHu+ZDUVhEb5QWzNOOazkCfhJJQG3sGfseTEiEFAk9qH6Z9AIukj+MznIQ8ieW+y8VOqgJz5BRF9bdxfWn6Uu+dH58MI0aInnvs4uDp8sh9k4iLeWQkjozmk7xNUrqo8uZtsk+TPCKD276G+D92KahU9iMw+65UPlycAEYxjLTX7zLFOT6i4/3MusxnOXIfcMCIT4M1iU9jKMuEhWJ2wm4mgPf33HFCKks4wn7DKozt2V/F8BeRjSSjpb9p9B29khpW7IjB02hJTM3VDtkaPj15BBbNrEcbkECMWUuUViG5SR6GqpDQJxUrlaGr3OMIDgmiWPCMjK8ryh/raCOZMMTCtl6yn99Drjf4RDjfjpkVpgc+JNBEVjGw682cdQrbuK/m+EMGaDR7WSjUQcctMdkv572UbC9zTiTUgKUXrHKgXXepw8TjtIDjKwrxy1jmAmSY9yv1g9C1G/xuh3/ma2qtMjKueIBkaznS5ensPCj8Rcr1fcb3mZu/a/6eabQlK7mOv57lFENKnPiQ66IF0kWTQ3BFJKMDc6q/cwuRiwIy0Pfpn4UDkBNSOaK683KBoQjh6jTtzyN+jTUPcuXGM4UIbccn3BY92iNq12J7fMXmHPpOM4YWPK/5z8HYn52RhOdofZgElI8H+HYSNJRI5exHTYgyf2ZNLKGp6oqux6f4b6vv2NIJljRAvoFY9N26/y72vSJ5z7i4gZ+sq0kuWIAaL7yDTMxM0NtMaB3dxrGj38aalzJxTThgL7oGchk jUE9GRD4 s/6BRsjh5iUdfJa7GQqjwNNLtupaj8tiAgQmY09U4Vnfga2LmkvVnikiJRbZxjhq5z9jjaJSLUAwRRg/jcHDIWtfm1U/xWC65Fpfso7oUOXmhDH6lkOhiGjOLLc5RT6iMNB50lta5tym+RB+g6A7LWHInBEZbe2z8BNHqvo3RI0vWyuEhUHcT6gCalkIVT3j17+tYkmdsIhM5G8g1j3Y0cbV2nLEEKM5tSpAIZFCbDIEvBOa8VMQOUfdAqHGfXLAKQqgoSqs8QfhANXyhfkQDJmtQK2M2idzWjPZ9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig To get out of the dma mapping helpers having to check every segment for it's P2P status, ensure that bios either contain P2P transfers or non-P2P transfers, and that a P2P bio only contains ranges from a single device. This means we do the page zone access in the bio add path where it should be still page hot, and will only have do the fairly expensive P2P topology lookup once per bio down in the dma mapping path, and only for already marked bios. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- block/bio.c | 17 ++++++++++------- block/blk-merge.c | 17 +++++++++++------ include/linux/blk_types.h | 2 ++ 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3047fa3f4b32..279eac2396bf 100644 --- a/block/bio.c +++ b/block/bio.c @@ -928,8 +928,6 @@ static bool bvec_try_merge_page(struct bio_vec *bv, struct page *page, return false; if (xen_domain() && !xen_biovec_phys_mergeable(bv, page)) return false; - if (!zone_device_pages_have_same_pgmap(bv->bv_page, page)) - return false; *same_page = ((vec_end_addr & PAGE_MASK) == ((page_addr + off) & PAGE_MASK)); @@ -998,11 +996,16 @@ static int bio_add_page_int(struct bio *bio, struct page *page, if (bio->bi_iter.bi_size > UINT_MAX - len) return 0; - if (bio->bi_vcnt > 0 && - bvec_try_merge_page(&bio->bi_io_vec[bio->bi_vcnt - 1], - page, len, offset, same_page)) { - bio->bi_iter.bi_size += len; - return len; + if (bio->bi_vcnt > 0) { + struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1]; + + if (bvec_try_merge_page(bv, page, len, offset, same_page)) { + bio->bi_iter.bi_size += len; + return len; + } + } else { + if (is_pci_p2pdma_page(page)) + bio->bi_opf |= REQ_P2PDMA | REQ_NOMERGE; } if (bio->bi_vcnt >= bio->bi_max_vecs) diff --git a/block/blk-merge.c b/block/blk-merge.c index fdd4efb54c6c..d9691e900cc6 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -320,12 +320,17 @@ int bio_split_rw_at(struct bio *bio, const struct queue_limits *lim, unsigned nsegs = 0, bytes = 0; bio_for_each_bvec(bv, bio, iter) { - /* - * If the queue doesn't support SG gaps and adding this - * offset would create a gap, disallow it. - */ - if (bvprvp && bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) - goto split; + if (bvprvp) { + /* + * If the queue doesn't support SG gaps and adding this + * offset would create a gap, disallow it. + */ + if (bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) + goto split; + } else { + if (is_pci_p2pdma_page(bv.bv_page)) + bio->bi_opf |= REQ_P2PDMA | REQ_NOMERGE; + } if (nsegs < lim->max_segments && bytes + bv.bv_len <= max_bytes && diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index dce7615c35e7..94cf146e8ce6 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -378,6 +378,7 @@ enum req_flag_bits { __REQ_DRV, /* for driver use */ __REQ_FS_PRIVATE, /* for file system (submitter) use */ __REQ_ATOMIC, /* for atomic write operations */ + __REQ_P2PDMA, /* contains P2P DMA pages */ /* * Command specific flags, keep last: */ @@ -410,6 +411,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) #define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_P2PDMA (__force blk_opf_t)(1ULL << __REQ_P2PDMA) #define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP) -- 2.49.0