From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11412C71155 for ; Tue, 17 Jun 2025 04:19:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A25F16B0095; Tue, 17 Jun 2025 00:19:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6ED6B0096; Tue, 17 Jun 2025 00:19:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8ECE26B0098; Tue, 17 Jun 2025 00:19:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 829B66B0095 for ; Tue, 17 Jun 2025 00:19:06 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 487E3101D89 for ; Tue, 17 Jun 2025 04:19:06 +0000 (UTC) X-FDA: 83563587492.19.BBC5660 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf15.hostedemail.com (Postfix) with ESMTP id 665E7A0017 for ; Tue, 17 Jun 2025 04:19:04 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AU5Ibrip; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750133944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tfXoidyywq4prG1qxFfXKWvZLzM8FnofZM+bLZ5UOeQ=; b=XVjLk42mywZEJCZKbYxcIgZ5vzm4UOVe5BGEty6dgURFZa8d0xID1HdV34lbURZM3Me+ng IE0mH9y4Mh2ygz7kzCvPbDFaudzFeLdN82xy9CACkuPNbv/Oc0VciaAIhzzuK1fjSOLjLg 3kCjJ0c6aL8IKGfGg/ike6PZrsvxbDY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750133944; a=rsa-sha256; cv=none; b=tdzT/arxWRFeb9yfA9HhiTHkuHAKBQJrRgs/ICxsGwSoCcu0IdWvPKJLcvAMqp5ktCqhhQ ZIpZ2hjOUrdQVCh7X+5l+3MEC5qD7/03T3/A5j+806V+uvIddSVK2RCgaBzl5Oem4vjSVd 6Ase0t4EOsxThsBWSXb8hvb+oTBB8Ps= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AU5Ibrip; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2352400344aso47838625ad.2 for ; Mon, 16 Jun 2025 21:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1750133943; x=1750738743; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tfXoidyywq4prG1qxFfXKWvZLzM8FnofZM+bLZ5UOeQ=; b=AU5Ibripze/sFmnwG6mwyoyh/AoIMDefrsXud2HAUbEszvIO5q9PxBg+98naWloli/ /PeT2wGPAYiUzma1eENuWGAqvL+4q+WcOPLY33kx+mO8O0RMiXOFw5YDzvxXEU07Tt36 lPq8Jr2mGBmI9lxGvEou8t8EX2T8Wtg6JYMyDDh4qnmNiZ+MFUMyByG3nhZAb/FyFmH5 gaXl3xMhsQ6UphOR+tf/nV6dwAeiPMURbUFtIJQfJMGwbimM7Nft+GQwK+1yD9QAH8PW LBkQMx2m2hzKRXjG0B1BTc7OxshWCl3YfWejl6J7ent2bx1rvCB4P4tzJMRd+zZwOA3S daXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750133943; x=1750738743; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tfXoidyywq4prG1qxFfXKWvZLzM8FnofZM+bLZ5UOeQ=; b=Vrf6aufoY0bag+AJvTgkFOfJlqz/qrJDr710BlwSuijoIjpX9zCQQLF4UdR2/EYyxl E4YCypm0BUFsr3BPrPg3hNednXaDQwlVQnMYFsWZeSp/+KjMFnDlda7GcecQTTnodsG0 GYNkKOGppzN3MJtD4ArEQtjxM0iAg5g4h3QRzkfoZF7UYuFbX/Y4pl5N/BXvlg7rg3pF MUsdq3r8oX1yDDF1/3ItexuLhUZMzjHgl3lIpCq5BYsnHNcu1SUQChll6Xf1+RDps7JE t9u+amqHGP5QLBkpWI6PRezBzgU/M10P+jzL2S1qZnba702EZm7wOAopHvRSyb3oMPFQ Od9Q== X-Forwarded-Encrypted: i=1; AJvYcCVuR8kDwNU3jijaoqTbRHblxLQJ7hpTHF2FfAXE2SVKOPidkKBpMhZCrLT40rX9WJ3RYNn6aIKVOw==@kvack.org X-Gm-Message-State: AOJu0YxzlPwp8wIockTe9hSGYytC8G4WYBt3w07a7p2620Ri0Zqcpys1 zJB+AU+VHMbqwFWBfg45CbwjLHF1r4BtwtFzpbwOsFRCHPx299nc+CiV4qllKSRPE5zxUxeBR0x OgcdR X-Gm-Gg: ASbGncvQh0nQZL/P7DSNXrMb2805b6+WqqLqS4QRmSUFcpX9cVIgntFJ9UzZ6D5Himn EJY2EBhU89FjnGNoQuE1CYaYMGQkbQ3nooTOPsao5sSzux1wSUA31i5M0Tn2xUPo/RjszThf59Z 20YqIknIK53r9FJTi4Of/kDntswnRIylZHrtlcb4++e4kktIaRSWlA+wG+69Y+sw3VGG1UIpBL0 ZgbGH9DHATnHPlb3jZjugVgXZmUKDLfHGquD2CzRA8X7P+TcZdXmZn35G4AGM/lLzudi/vrebMx lQOIk3FzUdjdXstUB7BdWp4ctqvELDBEdSqO0s/G+Bcu0NizjYk19/5aV9opMQ276LxoMfFD/u+ REJbiWjjAGxBfMw== X-Google-Smtp-Source: AGHT+IHqTbrzP939yrSnByQC0rtnaJOgibU9Byk10ulJjvMNZlvHwikXDBV86s1p8bsCh5VpmAqFCw== X-Received: by 2002:a17:902:d481:b0:231:d0a8:5179 with SMTP id d9443c01a7336-2366b003c6cmr173603795ad.23.1750133943270; Mon, 16 Jun 2025 21:19:03 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2365d88c029sm69798345ad.26.2025.06.16.21.18.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Jun 2025 21:19:02 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 3/3] vfio/type1: optimize vfio_unpin_pages_remote() for large folio Date: Tue, 17 Jun 2025 12:18:21 +0800 Message-ID: <20250617041821.85555-4-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250617041821.85555-1-lizhe.67@bytedance.com> References: <20250617041821.85555-1-lizhe.67@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 665E7A0017 X-Stat-Signature: roy77d69k83u8dj159wxg4edp7jxrjet X-Rspam-User: X-HE-Tag: 1750133944-323779 X-HE-Meta: U2FsdGVkX19yn0/N0OwrlTyblAj79GjsMd1DGlZjdtLaGRittFjT7s/2REKAaDtlYW4pFlpXVY7qTL1zuHlwfp/SDgn9MWbK+VyxdPIZnYocn2aZ/ylLO3M+fzUdGu9VBEM9zybQQr52J7KqsVfCZS9K78mivre8Mygv30IGRBBsvRhoDTdRJPHfuaMLDIBi85lFlEbCEw1FxwF1EjGtrcz913Nz0I7E1DzNS/k9EYnynf5r23KuZxQDc9gASaVgoKZflWdFkxEeMQHFGxsR38yHA4YS6pYa5HLzJBpqQvRW4qPZwiwlGHwl7sB6slLD2ghJXjna+YWLp/VN4FsSr+ay+WTTjsbmXfTtLpkcmi10FAiWKrnMbnS4hajHvydcaNlGzkLFmFNhOW95cAeDFZTqDTNfBV/Q5jCWr2yrGdLIQojo1PsGaFQpET1qR4H3BYcyIX5zWpJ9T8FRof7YZNOE1yB6Ze+6qgkxBPcAOF1HpxMFCs1uPMtSV/9dKLQ1NT3DUluNv6muJduwRvYE9TmLN3VQf9lIB6Uv+D396XcLnWO+xJIz6oHOXiDdWlK4hln0KHRMhWKjuplZbAejYRlNOBXzlKMFwrgATkDcIK6M5gtrsKPuP5mJFAxun4Ug3ZFZqB1f+INgh4b6zD7Jgyxa7C9EaKd2UuK306V0Ujtw7IPXxRSpiYMwN56TDz6WkKt2TUn4jAslFGehDf3hU3y64CqHbp/O00qPm0toHttF/obunHeI5qhmOTSjWvfpkpxoLSdyzn/bJtixK4Y28VXxtmTbtteD9ptxUucunsCG2QrszSSU/AaB012rWMsWdUQQsOsjXJphMroOZhZjbf0LzKeMbN+iKKbeNWOp2ox5VL8YBTTYuE1riLXYIN2FcrD9/WI3HTYp6IqSpTBQ11S0gV+dzs8CzqMsb4Y9S5Z6qXzkv7Y96UBfprJzS0CNL/rvI6kkKtQ6qfdo8ci dmC///wf Bfge4Fy/rFYVNnQyoPY6//ekYYZYx/mfNGHptRDT1oWWkECL8moBOfz4o/GMQR/puto8/cVXyncjPf65NAjayNyPUl7zQKZccrmW/u/bTW6A4BxQQDd04+to0hZOp0rRobJUJEIK08cS1H32IoMYlVGdpc2PknCH33RXt4U+UT1bwGV542Z+w0FX2UlqLvp5sM/GboVaSOl/LuncMRKFfOZAgW/zK8PiC69gNKj2Jwm0eLZGWvFWQ9c7RhoWkAfFZ5cS3xR7RIkuLoKOZIl2AhOgWiO9F5t37sD6coJjlt52H9zEfqZKgbQf9StXPvP1CMNi6iNzM/mIcSHmTptIKhZA8bpO36yB4x3/q2+GbCMMEDreH+Sj43gYFu0EL0C75ssWjFiQ5ChUQW1AL2i5aJK0oSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Li Zhe When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. This patch optimize this process by batching the put_pfn() operations. The performance test results, based on v6.15, for completing the 16G VFIO IOMMU DMA unmapping, obtained through unit test[1] with slight modifications[2], are as follows. Base(v6.15): ./vfio-pci-mem-dma-map 0000:03:00.0 16 ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.047 s (338.6 GB/s) VFIO UNMAP DMA in 0.138 s (116.2 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.280 s (57.2 GB/s) VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.052 s (308.3 GB/s) VFIO UNMAP DMA in 0.139 s (115.1 GB/s) Map[3] + This patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.028 s (563.9 GB/s) VFIO UNMAP DMA in 0.049 s (325.1 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.294 s (54.4 GB/s) VFIO UNMAP DMA in 0.296 s (54.1 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.033 s (485.1 GB/s) VFIO UNMAP DMA in 0.049 s (324.4 GB/s) For large folio, we achieve an approximate 64% performance improvement in the VFIO UNMAP DMA item. For small folios, the performance test results appear to show no significant changes. [1]: https://github.com/awilliam/tests/blob/vfio-pci-mem-dma-map/vfio-pci-mem-dma-map.c [2]: https://lore.kernel.org/all/20250610031013.98556-1-lizhe.67@bytedance.com/ [3]: https://lore.kernel.org/all/20250529064947.38433-1-lizhe.67@bytedance.com/ Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 35 +++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index e952bf8bdfab..159ba80082a8 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -806,11 +806,38 @@ static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, bool do_accounting) { long unlocked = 0, locked = vpfn_pages(dma, iova, npage); - long i; - for (i = 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + while (npage) { + long nr_pages = 1; + + if (!is_invalid_reserved_pfn(pfn)) { + struct page *page = pfn_to_page(pfn); + struct folio *folio = page_folio(page); + long folio_pages_num = folio_nr_pages(folio); + + /* + * For a folio, it represents a physically + * contiguous set of bytes, and all of its pages + * share the same invalid/reserved state. + * + * Here, our PFNs are contiguous. Therefore, if we + * detect that the current PFN belongs to a large + * folio, we can batch the operations for the next + * nr_pages PFNs. + */ + if (folio_pages_num > 1) + nr_pages = min_t(long, npage, + folio_pages_num - + folio_page_idx(folio, page)); + + unpin_user_folio_dirty_locked(folio, nr_pages, + dma->prot & IOMMU_WRITE); + unlocked += nr_pages; + } + + pfn += nr_pages; + npage -= nr_pages; + } if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); -- 2.20.1