From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49252C02194 for ; Thu, 6 Feb 2025 19:14:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD2D0280005; Thu, 6 Feb 2025 14:14:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C81AD280004; Thu, 6 Feb 2025 14:14:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B231D280005; Thu, 6 Feb 2025 14:14:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 92F98280004 for ; Thu, 6 Feb 2025 14:14:57 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1DBCFC142C for ; Thu, 6 Feb 2025 19:14:57 +0000 (UTC) X-FDA: 83090472234.18.BEB5CEA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id C3DE61C0016 for ; Thu, 6 Feb 2025 19:14:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OG10MBCE; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738869295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+F8Mn89b0OdrghV9tYd0P21FtLLxAlCu0Zd7fsJXe8g=; b=JJADzklZDWW1U/2ojgpBx9LUfcCFZE8aQsOuijThyH0uXiJPBcYIide7ZzitLvuPTEYhRx /tcat3kcCTg7qZWDv+Zzga+Ub61IDJ8NFTzBYGQ1SBcQVv3k1CDMejrwjHCxhLD5UTHnJ2 f40beYh4Jzs+A+roi92Xy3RZnPnreN8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OG10MBCE; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738869295; a=rsa-sha256; cv=none; b=Ju3OxpW4fC2Qamk4fKgeXi4iv6SkOY9pcWH/oWSp1egk0fpN5ohC782mcFrJ+JM0QtXDdn 5od+113ot5TvnKyolfsmfjqCXzq5le8GlMl7Y7GjTOrDFqxq3s2GAedqNCoCt4wMQXEqzE VmxML5T7XiQIGk9mt2G090q3lY62Zus= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738869294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+F8Mn89b0OdrghV9tYd0P21FtLLxAlCu0Zd7fsJXe8g=; b=OG10MBCEg9TsLX3qbrGQbwtQpwtbpXXpAY1SRjmtggclhWQRNCCAGzo+80MxnvDGiSXQ+K csVDdcQ5nddRQlUphyRfuonENr4Lo8Rz5UeKhzikGYFtSKnEZkcLQu23oLDfTLtymG5MP2 WtPToSmb+irsXlzp1vT/Z59/a/8d4Ps= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615-4rxPX1UPNiWH3hSZDcOT4w-1; Thu, 06 Feb 2025 14:14:52 -0500 X-MC-Unique: 4rxPX1UPNiWH3hSZDcOT4w-1 X-Mimecast-MFC-AGG-ID: 4rxPX1UPNiWH3hSZDcOT4w Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7bf4704255dso211397385a.0 for ; Thu, 06 Feb 2025 11:14:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738869292; x=1739474092; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+F8Mn89b0OdrghV9tYd0P21FtLLxAlCu0Zd7fsJXe8g=; b=kV4JMwMpPaY9WsOo0vqo/lLDLSGX04Rm+hN9pY3BJ8M8wuI3jGRuIgyRzb+cigY4pW y+VROvGB5yGXitKZfzIFhwDtwC9qp7vB/TZ4KufFAp0upDZvQ7K8agtdx6SJTGd+5m5/ ryjFAAXpECOGN6BE/7LW4C237Ef+pIDLhwHVALIhTKfyyQsvuf3aoxhVcJQsy89TfBgA M+kH/DUrqQ/ybo8aaII4ybqM74gpT+Ql97xQZfpVilfCphsZfj6WUq5SYA4Xhu12PhtX rc2QxuswX1gsHjlEf5yNCyGD/QfgwufBwGh/9DL+GvuPdPyf/IRrPP+6hvX4gid/Q9fK jivA== X-Forwarded-Encrypted: i=1; AJvYcCX1J/h+FTX/9a/lCYCvZDBZ46h+oHCj9mXn9CfhUL2vyDtHXBjx/lUMOnipLHHixF2Xy/xFBSp6tw==@kvack.org X-Gm-Message-State: AOJu0Yz8AYjWp6ZmNFX9yAe58q+HbQMlNlqvUliltUC0qbnUtmI6RuVm 4xRPYvuuq0fXQBw/+TLU+gfFcMWmA9LhyNHJvpLMPYmKqzO+be4sfY72oL+gD0l68PjYSMpTj8B 4fD6omXyk0dHksUmzm/vC6YhwxF8SV7FPHHlsbB/lk+nV2V1b X-Gm-Gg: ASbGncuEqVMGMsmMfabBohT1+K9KwTuQ6qcc1gwn4TktMIV/tr/1bcVoANmy+N0nqKh 3GHtCMQlFwDTv1EAIOkLnBodmlb8R4yFuQgRZ0XjROnJRt6YfBC1NycrmEHvA6DjOLuw7jeH8Cx mrfC/FVwhE8BuBBTTkvV4TfwPI6P38q9FSfZJcHc3A6nxO2yAUu6CkuFp5tjveqrg3q8d18DSkx W7M7HYNEkEPfTIEf9WClw7TkMaBha61+Tbe4+xWxAvda36ZRWYrT1lI+iyH4mXxk8mQcoH6VL5g Z5A3sngEaIO0jDM0BmXDduNk4mHq02T/hToIyfF98Zfr7bE4 X-Received: by 2002:a05:620a:25d0:b0:7b6:d4a2:f123 with SMTP id af79cd13be357-7c047ba50e6mr43437185a.2.1738869292202; Thu, 06 Feb 2025 11:14:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IEe9Zvk86HxhKs+COosIxGv3oUlSB9oKcrcLV0AUOxRdp3FalJ6v6z6IavQ7kZESBaVcNxunw== X-Received: by 2002:a05:620a:25d0:b0:7b6:d4a2:f123 with SMTP id af79cd13be357-7c047ba50e6mr43434385a.2.1738869291903; Thu, 06 Feb 2025 11:14:51 -0800 (PST) Received: from x1.local (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c041dfb976sm92254985a.43.2025.02.06.11.14.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Feb 2025 11:14:50 -0800 (PST) Date: Thu, 6 Feb 2025 14:14:49 -0500 From: Peter Xu To: Alex Williamson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, mitchell.augustin@canonical.com, clg@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org Subject: Re: [PATCH 0/5] vfio: Improve DMA mapping performance for huge pfnmaps Message-ID: References: <20250205231728.2527186-1-alex.williamson@redhat.com> MIME-Version: 1.0 In-Reply-To: <20250205231728.2527186-1-alex.williamson@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: IQbn40whT-em6kITBXfLnOgxsIou715jVTIKxGrrQvY_1738869292 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C3DE61C0016 X-Stat-Signature: t7pyuodbza7o1ycx7ma9zj6kyniidtym X-HE-Tag: 1738869294-522591 X-HE-Meta: U2FsdGVkX1+8gESJ0GRpwCnNlsukxgZVQQYsv1H/CUopcVinatKT4spMxY9HUq19oCr4tBXPLYGUJMbSC5fpIFO/3K+adf4Fhy4qgsH8fDgUhdMItxWY+FAI5m0cbSlMeFAacBTTnwqha11cB9Zoabrnr6Q93J0EYkNglOlRkeYXHM9xV/2ZEd715UERZB4Kp14Z2ivkKCdY18d41DMEhA5TRJZ1iGBQspzBS7xhs3IXjWnDts7NzmNCtTBrF0mCTMxsG1F7dazkIZ2z8Yih6LAHtPxI31SAvi+EQiI5szaChQ8Yt2b8/5zDpmhROK+HH1O+xbGxBZsVOCYZqgyQcVRDBL0wfazt5aUjUjfqZ3gHYXUBC7kdPTNTJyP7CJ1uiRH08ZAVU0mWN1R0zcfbwDM8QpNbdbMoL6EBupZBITpjC0Z2/M/htCdn4k6LKlIXxR/whceVqf/DU1fIifUj/nhHRvt/ppgCJjH0wqnhVQLbXJ54b388MXfSZo30n/AATVX8HDeqA9q9e0w/UWQgpq7MoVMQFnrKU+znBkpV71U9+apIMD3g1ChF1sH7svFjY+QACGD7ozvaTN/P4cAUBXPUkaAzaMrb1hHHnX6Sz79uK4V2KDoiBCTiv//es7dwZvFIrCvgOs8it7RQeB15A6ms/YDuquxuNmfXu8KUZX11EKyak38NN466jEpdXu9anTFzlSmXobUeGGlIkSODhmxqhgaCaJDtO+Xu+J76hflPRbj7wKPAdrR5Ir6+N6EGqzeuvrfSA7mTkeDBuUaEy3OHTz2oHTO2TU0gHX/SapfbC06gsrW38AC6GLeVw147nBq6PzUC2+rRDBwdmgeVNbXlWgbjeeA8DcBWZ1r/FwZp5m09k+eaSlmIHbPnScUZiRJfUTOBeCC3C03baXQEgcct4em3WgzKdl88yEqGWZiHQ5tkP7rb5VTp7xWMboPeepPNb9ao4BC5w6KaSnh km0Pech/ N1bugX7yG2UIzEel5WFyVf0sdQWE1AOrM3uI36Xy0/HAfxJR50vSkl8X0nANngGzGhV0OvhvAYAylEK4S7dZDDygxNOkovLNmbDscQgu6kyN8x+xBOOnyIJONtq8f4PJraz5LqYznFdIakPVIUGT0NvE5E2QZaUGWDFkQNKONS9zoA3J8Sdicgp4ZP7nmpR+nHLdJAxBgyu5/MKhy7+fgkMF1pVRpos8ccbzJlyfl5m/j8wmMEbeGb+E9FygHGgCnCFnZpu6DEJXUXfmj3bLUqZsnrVy8kXiTrm5mSby3vCS66X1VGRzQzAs0g4ziYFY1yptVXeCuGW7evHEghjvkkO7J31gNksJoz/6S+KOTUKjqaDMvi7LpFoxTKgFzDIMffoMh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Feb 05, 2025 at 04:17:16PM -0700, Alex Williamson wrote: > As GPU BAR sizes increase, the overhead of DMA mapping pfnmap ranges has > become a significant overhead for VMs making use of device assignment. > Not only does each mapping require upwards of a few seconds, but BARs > are mapped in and out of the VM address space multiple times during > guest boot. Also factor in that multi-GPU configurations are > increasingly commonplace and BAR sizes are continuing to increase. > Configurations today can already be delayed minutes during guest boot. > > We've taken steps to make Linux a better guest by batching PCI BAR > sizing operations[1], but it only provides and incremental improvement. > > This series attempts to fully address the issue by leveraging the huge > pfnmap support added in v6.12. When we insert pfnmaps using pud and pmd > mappings, we can later take advantage of the knowledge of the mapping > level page mask to iterate on the relevant mapping stride. In the > commonly achieved optimal case, this results in a reduction of pfn > lookups by a factor of 256k. For a local test system, an overhead of > ~1s for DMA mapping a 32GB PCI BAR is reduced to sub-millisecond (8M > page sized operations reduced to 32 pud sized operations). > > Please review, test, and provide feedback. I hope that mm folks can > ack the trivial follow_pfnmap_args update to provide the mapping level > page mask. Naming is hard, so any preference other than pgmask is > welcome. Thanks, > > Alex > > [1]https://lore.kernel.org/all/20250120182202.1878581-1-alex.williamson@redhat.com/ > > > Alex Williamson (5): > vfio/type1: Catch zero from pin_user_pages_remote() > vfio/type1: Convert all vaddr_get_pfns() callers to use vfio_batch > vfio/type1: Use vfio_batch for vaddr_get_pfns() > mm: Provide page mask in struct follow_pfnmap_args > vfio/type1: Use mapping page mask for pfnmaps FWIW: Reviewed-by: Peter Xu Thanks, -- Peter Xu