From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC830C71155 for ; Tue, 17 Jun 2025 04:18:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 543786B007B; Tue, 17 Jun 2025 00:18:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F46D6B0088; Tue, 17 Jun 2025 00:18:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 409F66B0089; Tue, 17 Jun 2025 00:18:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3273B6B007B for ; Tue, 17 Jun 2025 00:18:47 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 75BDF1D838F for ; Tue, 17 Jun 2025 04:18:46 +0000 (UTC) X-FDA: 83563586652.29.2A00A01 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf10.hostedemail.com (Postfix) with ESMTP id E4FF6C000C for ; Tue, 17 Jun 2025 04:18:43 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ck9SMCoz; spf=pass (imf10.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750133924; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=igAZEPeaIEDiheZvfxf+oF+fwfb6UvykItlBM60MOcA=; b=N8c5CnYINtladc5xoOAeN2c1RrSzTD/XV5zYqEvN/XwKj3kbYNkTzIiiJ+3gI1vCfkrf5f C5cdufbKYETkqoNjzxy7WvX3BYwSWCpgxcFpv0hZryBhagKh65B/aDgXNULg+Sd1HyymP9 yIY78C4ZiMgL0BbyWxFmjHlvzNUwITk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ck9SMCoz; spf=pass (imf10.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750133924; a=rsa-sha256; cv=none; b=U4ia9o29mDsq1jS1zHqjB0/+NjY/wtluoA5qlcg+6ormShN2cpRbu0ZP9ZKF62GUc+ZdwP oxDoz04YFjlEVWur906XTYwdJxCJsFU+Hi/VKHZAbfePPOOCg7C+44gVi2ShoARBXZ0tDK nl3vjvKN3o7PHzSHe4O37hxTl5TThR8= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-234fcadde3eso66521795ad.0 for ; Mon, 16 Jun 2025 21:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1750133922; x=1750738722; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=igAZEPeaIEDiheZvfxf+oF+fwfb6UvykItlBM60MOcA=; b=ck9SMCozPOAzvfYfjfKpA+7gMpBV/aEH0jkHUyWIlVMcxNXKEK68g5V2KyYwXwdZdd vIPycYBcZUFMzoLxAvMnNmjgH+0dqW0NxG904K4yGq9MXk79ssIh53pumrHovj/0EdlI AT3SJ2R+cgo0Fw6ANAs3ZImtBSz2vhQrUYR3beOYiYkDy84EOlqWh+EVrDQOEa162f/Z F+RLa76nWf8B0vFB9sM6En2rue0aezEXq9Fw+oeqAQxsRtDNmp1bQbIkbyKyRiqlJdBO XcGV0Yz+e6dKxhfCf4wQ8TJ9u6907dAUFBBnn2mypfTBGrmb+MGTMc1b/OVc3mPfF1xi 8yvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750133922; x=1750738722; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=igAZEPeaIEDiheZvfxf+oF+fwfb6UvykItlBM60MOcA=; b=qUZ8tMaHRbNx0UchAxBSM2i/CwAjiSe4hzHncNSAe5+lLP6XluSCaFKyKe3D9r9ulj PWIHh7jWbyp4dRPl7BoJ7uyv2xZVRfuD9kch/nhwD0Sfd3M/pjKsczYwrDMlBbaIm2Fi GFUo5wNM2JKP0vYUyjXf0bZhq99aYnto2CGPNAtqp4yI7V3ZpeI5aroOUTu5+4Vpq1SX iNHh+N9ZE/CfcGg86olV7K7dOQb1dFcwXySZtwy4lAkqffY53gn2p4w8oiO+IZQC3AEP SJILwtwVrldyELU0PNjSpnZRD6tHLletusYcENqOwIjMfp9cNnymtz7ANytuMZZiM0Sr 3Dwg== X-Forwarded-Encrypted: i=1; AJvYcCVJcn6CY+cMVwuodnEis+IiuKHCtgkoJCJ508fKEVjXnmr7MOMHcfwUQeIfmJPSCMXEaON56CYgQw==@kvack.org X-Gm-Message-State: AOJu0YxULHbvt40d7mg1uH3ufZbext6EvChhVbGrLvRmxUI68fwPoIdx H2PwR+j6huYMWEBFPDqoF35/5yvh0KpSDWRDV4QmBTGF77iAiKVMPmnRFMMOXCbZIGQ= X-Gm-Gg: ASbGncv8DlAYMD1008N4fH+/Gb3WpII8xJ58je4us1sfbuoMpZSSiAekqIeXiQIrNsg T4dEYvkhEgKXU2AQeS+ptlJpJkWE4cq+cTr1+CapHjNSwlTri1oqgsRt8X1LOAXr2PoxNq2nwJ1 g7ELcTCpf3uoNnIbL4HoQLlM4Yvd0MKxS6nvP88/9ByY2n5OUqZH5+WWpk/8dLGzXWu1aQnSM9D /MuU/ZBwrU0X4Do79zmpkyM8qONe+BYjlRqal6FurSfuzukCEXXo5fDs4YoOB9wpu6FwYYvnLFl 9pcSz6tKgxqOSY2ZVL4sR8Lu9TtblnKcP4wawTWxJdby6hKRlFov7qqtI+eJTR461ueZFwESwq4 HNOMvq6pd0MXS2g== X-Google-Smtp-Source: AGHT+IFsY8746XIZ16bXJPVrgLyl93cy8mP7x1VEWMaLf/+bipVG7ov5clC26qJPWsPEVOqwRUn6mg== X-Received: by 2002:a17:902:d4c9:b0:235:f4f7:a633 with SMTP id d9443c01a7336-2366b3ac074mr204669445ad.28.1750133922275; Mon, 16 Jun 2025 21:18:42 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2365d88c029sm69798345ad.26.2025.06.16.21.18.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Jun 2025 21:18:41 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 0/3] optimize vfio_unpin_pages_remote() for large folio Date: Tue, 17 Jun 2025 12:18:18 +0800 Message-ID: <20250617041821.85555-1-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: E4FF6C000C X-Rspamd-Server: rspam07 X-Stat-Signature: bzyff5r6kja3xqfjo3pa9whe1e6e65aw X-Rspam-User: X-HE-Tag: 1750133923-10333 X-HE-Meta: U2FsdGVkX19h/XqyQ6Hp4rsRZ9lMfRw8jAEQeuj5Te7wQWJjeHxUnMxP4LZLNzfwzsC/odWatuE4z9ZGqPPDVDl2H+aqrqK1SXtRhYAB2wukS+4JWXfmU1CUL4xESQ96HnyX/DU+27cMREdAcp2I39ZZ6Y8StKsuqiyUu7DCB4Zin/4t3Xe6kqqiO4qqIy+mmztlVRA8+37blxcv/szU9m0cyoHSOVHc/Tf2n81lzc4yjh38TtdBsSwh49VT+Ka3GEfaMPWmY5+Ns+zUNfTyl07hnEdmHhWVdeB+YH+/O+G4eSqGDZ56FlGca1fQLrJb18FbF4pRDA8YZRpCzsn/kAqlKsradY4EGIrlzD69aGcV/OEOEMCpwJcJ4MdZFsG7cBl+B1ks0UpKGJ/dFcxFuDvoGqt3xwXe1OmJjq6ykkBjoz7geu+N9NrY24rSYVAJYmLhDrjgDmlHctgFaTJeQlOGY8gQ7siKA6ysMYDNjBMdFCGnNwj+DdKi9SaPa9wpm//zTRS/uQF2lHA2NYlKIjklXmQ9AVFfrgnpRxSIrW73X821RKYNYmfaPHQnK69MbREcKbcg/PqbycIEhubIye4YmmCVlygjkQsc2r8HUBZedz7QkRnRev9UCLfGXg4xsZqvBb9kpQF/TDAo5AJF97UtAl5SVJn6s7LFGxVNEOxu36kATOyiws2aaP7pRbQGx/ne6i336yZ38twvOMKvigOnHe70o3ZjV3jNElsxd7FX03HjrcktUEYBY/vUCnSGfI8Sp+flE3l+4o5XIx7zqE3abb7vdUp4cLTOpCRlukTxzh0QdawxUbzgrmtu8WkL531kuSov5sRQsJqjDJK33u9xSzSToaxreuY4UAmu1Pus0LqqTFJCMBCpWyvzviu/Mk9KcmOheYXWbP+f57MqUWmuiijugaqiW5ExA36i5CAUycnWPUcvY9WF20dtqCwbEJcwSLY/zByEVr7M8Pq ZrtopNfR j8y9XgoGdOJp5x2M0PGoVowQfF26NJPivIaqgJ/KkDQXN5GmsxjiLVYUpoG2T5pYcLmTHOAt4WREbW8cLno3qipBaZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Li Zhe This patchset is based on patch 'vfio/type1: optimize vfio_pin_pages_remote() for large folios'[1]. When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. We can optimize this process by batching the put_pfn() operations. The first patch batches the vfio_find_vpfn() calls in function vfio_unpin_pages_remote(). However, performance testing indicates that this patch does not seem to have a significant impact. The primary reason is that the vpfn rb tree is generally empty. Nevertheless, we believe it can still offer performance benefits in certain scenarios and also lays the groundwork for the third patch. The second patch introduces a new interface, unpin_user_folio_dirty_locked(), to conditionally mark a folio as dirty and unpin it. This interface will be used by the third patch. The third patch, using the method described earlier, optimizes the performance of vfio_unpin_pages_remote() for large folio scenarios. The performance test results, based on v6.15, for completing the 16G VFIO IOMMU DMA unmapping, obtained through unit test[2] with slight modifications[3], are as follows. Base(v6.15): ./vfio-pci-mem-dma-map 0000:03:00.0 16 ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.047 s (338.6 GB/s) VFIO UNMAP DMA in 0.138 s (116.2 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.280 s (57.2 GB/s) VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.052 s (308.3 GB/s) VFIO UNMAP DMA in 0.139 s (115.1 GB/s) Map[1] + First patch: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.027 s (596.1 GB/s) VFIO UNMAP DMA in 0.138 s (115.8 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.292 s (54.8 GB/s) VFIO UNMAP DMA in 0.310 s (51.6 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.032 s (506.5 GB/s) VFIO UNMAP DMA in 0.140 s (114.1 GB/s) Map[1] + This patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.028 s (563.9 GB/s) VFIO UNMAP DMA in 0.049 s (325.1 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.294 s (54.4 GB/s) VFIO UNMAP DMA in 0.296 s (54.1 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.033 s (485.1 GB/s) VFIO UNMAP DMA in 0.049 s (324.4 GB/s) The first patch appears to have negligible impact on the performance of VFIO UNMAP DMA. With the second and the third patch, we achieve an approximate 64% performance improvement in the VFIO UNMAP DMA item for large folios. For small folios, the performance test results appear to show no significant changes. [1]: https://lore.kernel.org/all/20250529064947.38433-1-lizhe.67@bytedance.com/ [2]: https://github.com/awilliam/tests/blob/vfio-pci-mem-dma-map/vfio-pci-mem-dma-map.c [3]: https://lore.kernel.org/all/20250610031013.98556-1-lizhe.67@bytedance.com/ Changelogs: v3->v4: - Introduce a new interface unpin_user_folio_dirty_locked(). Its purpose is to conditionally mark a folio as dirty and unpin it. This interface will be called in the VFIO DMA unmap process. - Revert the related changes to put_pfn(). - Update the performance test results. v2->v3: - Split the original patch into two separate patches. - Add several comments specific to large folio scenarios. - Rename two variables. - The update to iova has been removed within the loop in vfio_unpin_pages_remote(). - Update the performance test results. v1->v2: - Refactor the implementation of the optimized code v3: https://lore.kernel.org/all/20250616075251.89067-1-lizhe.67@bytedance.com/ v2: https://lore.kernel.org/all/20250610045753.6405-1-lizhe.67@bytedance.com/ v1: https://lore.kernel.org/all/20250605124923.21896-1-lizhe.67@bytedance.com/ Li Zhe (3): vfio/type1: batch vfio_find_vpfn() in function vfio_unpin_pages_remote() gup: introduce unpin_user_folio_dirty_locked() vfio/type1: optimize vfio_unpin_pages_remote() for large folio drivers/vfio/vfio_iommu_type1.c | 37 ++++++++++++++++++++++++++------- include/linux/mm.h | 2 ++ mm/gup.c | 27 ++++++++++++++++++------ 3 files changed, 53 insertions(+), 13 deletions(-) -- 2.20.1