From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B0B3C83F18 for ; Thu, 10 Jul 2025 08:54:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E2E88D0002; Thu, 10 Jul 2025 04:54:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 293C38D0001; Thu, 10 Jul 2025 04:54:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D0BD8D0002; Thu, 10 Jul 2025 04:54:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0E8298D0001 for ; Thu, 10 Jul 2025 04:54:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D521E801E5 for ; Thu, 10 Jul 2025 08:54:34 +0000 (UTC) X-FDA: 83647744068.13.F419F73 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf08.hostedemail.com (Postfix) with ESMTP id EEEF8160008 for ; Thu, 10 Jul 2025 08:54:32 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FnnrF+wc; spf=pass (imf08.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752137673; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i4mO6QS2sYfv7EymtBvZKW+/VtzFT/ogV5HmARdE954=; b=Skdx6LwEI8UQz7P10xSq+ushstinnvkoB5+yaEPWoDkyTgYGVZxwSqBCeCBjOuDBdDJ8/9 Hks2Dm9fPrfFTmpFRTJdNBJW4DNyQSSGCwtI48VOZVyvlkPcjdK1fFnNjSdqcdVjKgBTq7 gX/SNdBPYBWY7i/O3VGPO5U5Nals1+U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752137673; a=rsa-sha256; cv=none; b=ARYYkzsoTz/zf+iAbIq9WaNZXwLzW1W7e0YIQ4rK5ULl/WNjZ9+BmVxOQGu6TC4fAgOUtr /RCMGPumtU4B9db+qXqF0O3RGw34j64kODRbYQ7Jv3qNsCba+ffUWaxg0x8x1g0s/xNwd4 ak+w5koRm0KHkt5rjWbyu/h0pTBg89o= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FnnrF+wc; spf=pass (imf08.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-3122368d7c4so739518a91.1 for ; Thu, 10 Jul 2025 01:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137672; x=1752742472; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i4mO6QS2sYfv7EymtBvZKW+/VtzFT/ogV5HmARdE954=; b=FnnrF+wc52OGTIw8fLTkiN4UqleTxZ9QLxgpICaPUGAUu5ehwqLgoHyjTLru8TpoKM /Ni/DiBA7iFO7wa9gNC9vZTKor6sfNjXjTz8nfXOSdSTQHuIUwlBe/682bW5PDEs6vnU HyZ4KPD8j/Y0h5y2qlD57ygILwlqg6c48xZDCTRrIkgV+Bx2/TYaQNOT6DqblBnaZ5wS ZsbDJbG7LFHfzozflKLqr91C1JiEPdAjX817rHzgW+z33zrrkKXbUVqALLU/FGPEXXTY is/QN3a02NOg7Bom/tecCaMDgmNC3UMZeMzGrdwg6ugluWiIT+s+XOjwEB1mWFoaXZZ1 H4MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137672; x=1752742472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i4mO6QS2sYfv7EymtBvZKW+/VtzFT/ogV5HmARdE954=; b=OBRSEa7PN1wnABrnHRbXV4uakmU0D9KtjroNm4g/M1iOPOpftGqb9yuJzXS82vJDT5 OVZQFeVJ80YSsul5GBDAlop3U2c2XBWSAFiMsyQvMOSkcXuQUNZcYzdx5amZSG/IGY+j dG13lME0GWO7TOWYZwc5ec50Sb0BiqcmtoOzUfu/o2y6SVZhtx1i4eRKfSu5QgOkAKlb j2A2ivjl2Fg1HdiOdY9wr5830bbv64NzlLLTAP+HbfOKvIL7aaqPeYWTmCRihViIu3W2 zBkKpA6a6SvnaXVXzIRrg2rgwOUPqk2TgUvoH7F4UEgmptMpQEdJi4rAyt14bWVysOFU w+2g== X-Forwarded-Encrypted: i=1; AJvYcCU1VjSLkq1cRBr18I2nE/IvYhSSYU4t7pv2zHaOKFNV8DNpunSaZkvBuykcw547EGA7RDcm36fF3Q==@kvack.org X-Gm-Message-State: AOJu0Yx9n0eD7R+Q8NaiB/SM1ZuNYdTG0kfOmwY7tz8O1F22OvhmaaL+ GG5Ny/I4OaJ+fWDpwPP87zjLvmOvrmX1+xqXZD9ByKmLkclHDAJ4X3zv0EYGmBkFWUc= X-Gm-Gg: ASbGncuCo5GUSrclnaNbxxcXL3n562JKvY1D++CJJqeddOnRIQ74mGgp0nAUx/s8HKx mNZr8LzTlC5TRv/c8OiDxjme/hF/yN4pD1y6PwGyKatbQMb4l5Qu55gk3UwqfnxEVr7kBlfmGBv qeBybHiCqcWCYbG+cY5000CD01Oj0FjMIhkq8RDuMrCTOqu8Gr01eAjf5ETdNhIIfoZNDKPHUjB ZG/O+WKU6o+cp5q+3a5Can0sPMx+4JM7fpTB/vPeGIk3NeHAmq8ZuzzXXU5EEi+GE9Dcr38phPM SvhlGOvp1c3AG0WIO9rk9slbHAUJC+U52hnufP9P8GsJiszjQ+PLji6DZkjVZgc0lPmxQHHyIOx 7GaHNcD6oeilSMw== X-Google-Smtp-Source: AGHT+IHpGnM8/RSgCKrl528Fh5n6c4YlUKSZpBOagB5O4dzJXeVEsMY3FEHcb5l/h+FzLkNngxE9NQ== X-Received: by 2002:a17:90b:4a4a:b0:311:b005:93d4 with SMTP id 98e67ed59e1d1-31c2fdd1479mr9203981a91.25.1752137671529; Thu, 10 Jul 2025 01:54:31 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:31 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 5/5] vfio/type1: optimize vfio_unpin_pages_remote() Date: Thu, 10 Jul 2025 16:53:55 +0800 Message-ID: <20250710085355.54208-6-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: EEEF8160008 X-Stat-Signature: b7oa565ei8ik37zrkqwhfyw5p17zfypk X-HE-Tag: 1752137672-253700 X-HE-Meta: U2FsdGVkX1938A0yZErVKuSMLZtj4dM+Hj4GGKm4Ph1R4l97EwbLO7cS6AViLwThlGjw0yruFzkI7EPKiBWb0VCXdHKnSitXiydf4BuwqZbi+/gtwZXHAfSdV7dXyW656f0wR/fROYL1yyVkFXDOC04KxQB0HS1YPUBRAzHhIH0DTQGzy0guEc5lOhJLHJ8vaU29+PYFIA+u83QSFRLXumtrrZTkr3zDFVH+J+riPRm50QZ3S4QmVOYDtEmBfoFNOS007/LPggtNjSrM9yFBf8MbrM6tJo+89vN4qyVeYw51Ei9s7kDFpQZOj40bcXr8fNB/msoT2241KixVKS3C9+/JccGevqOJTtIUxQPkfVjBhnRnRhsCmWQ06s/2BOvZfz3bto8+mGKmuuugTpoggm6iSkeq5W6ipkPZXUXAWbcxweIOAH8HC8cQm+expyIppcZbk5RIiyiU2FhbhDpgJhVufRIDoD8MHUS9y/5gqha/b0Mi+9XV9Doum5TyzvuKnmddHywyU7baobLU7TW8lVXUAIoMzb+krwgQMN33YbLwA8I742uVlN58XxwdPIbpAwu8s4hb5U9OzeaKmcjVjCzDkbVpe3ISZFCBCC/jbkWRr/BNW2jLcVRUT6b17APFK4beRa3NGdobrXsf34JxGXKcl8AZPovFIgEmKnR7CPw2E6Gt8zHq4dafq82QPFLSP+QfzZCDl5NsKYjiEcF1uoPfKNIT0nD5fIHMAOMdDLCUf+T4S0Dw8pcb92vnO3c0lQQlDGduh9KbjNLESvhAt0Pr8ACloKEyD2AxjZn+3CGaDudfzOhqGk4y8t2cEdwkdIGI0RjppdDpKS+7i56E6d05aIbXTy1ReSquMLP1j+tjU+6ETL6tpRceFUDHUmCgz4RYF7YbaxKmvlNZSrwDbDi+m4/x+5bJiZSYQvs/OyoSD6+UR7/DAo4l5RDEA10NT8xqn6F2+/UgLNoa1oT ltJA18j0 3t2fEFSP29mOhp5X+9tedIsO2g6V8xItoT+n3QE87xVDqMVyT0miUQ5UZfNqtS/NKqFCu8IQfaakW5B5R0qoqvkih+jXjQDHIZHxX49NM1Y8i8/aPdMIZ8y3OEhMZocBR8vY5mWvFvbO2GR2UECRyxPbqngIfgKeF3/md3XymAcIUPwAmn2k7b9NMfcVyjNKxmHoOUxcin/XKa4n2vQscmxiOMU9aLJ0rHmlB5XHAHJfnXYIHRSLb5UlllaMfavgqUugtPEM9mEIehBbT8Pw5VtgHKO+ghcikBGVX70ykd6cqPV6f7RvI3GdgvKa2TWJLbUeuXHOxSOUKQ1XmzbRJpMs9AAMyxQVTZxR4S+vdv9C4wCaHIJAPcJe7uZgWFXS5daOuvGE+akXAp86oic/AZ+ne7owxbFwo/6/2msldTtAldNgzg3QCP6HK0iswQwoccCMKiMWE0B/CEmYEDBNJMxBeVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Li Zhe When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. It would be very rare for reserved PFNs and non reserved will to be mixed within the same range. So this patch utilizes the has_rsvd variable introduced in the previous patch to determine whether batch put_pfn() operations can be performed. Moreover, compared to put_pfn(), unpin_user_page_range_dirty_lock() is capable of handling large folio scenarios more efficiently. The performance test results for completing the 16G VFIO IOMMU DMA unmapping are as follows. Base(v6.16-rc4): ./vfio-pci-mem-dma-map 0000:03:00.0 16 ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.135 s (118.6 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.136 s (117.3 GB/s) With this patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.045 s (357.0 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.288 s (55.6 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.045 s (353.9 GB/s) For large folio, we achieve an over 66% performance improvement in the VFIO UNMAP DMA item. For small folios, the performance test results appear to show a slight improvement. Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe Reviewed-by: David Hildenbrand Acked-by: David Hildenbrand --- drivers/vfio/vfio_iommu_type1.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 61455d725412..133f284ae168 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -792,17 +792,29 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, return pinned; } +static inline void put_valid_unreserved_pfns(unsigned long start_pfn, + unsigned long npage, int prot) +{ + unpin_user_page_range_dirty_lock(pfn_to_page(start_pfn), npage, + prot & IOMMU_WRITE); +} + static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { long unlocked = 0, locked = vpfn_pages(dma, iova, npage); - long i; - for (i = 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + if (dma->has_rsvd) { + unsigned long i; + for (i = 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } else { + put_valid_unreserved_pfns(pfn, npage, dma->prot); + unlocked = npage; + } if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); -- 2.20.1