From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2623C02199 for ; Wed, 5 Feb 2025 23:18:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FC556B0099; Wed, 5 Feb 2025 18:18:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4851E6B009A; Wed, 5 Feb 2025 18:18:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FC99280001; Wed, 5 Feb 2025 18:18:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 10B8F6B0099 for ; Wed, 5 Feb 2025 18:18:26 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7360C12090E for ; Wed, 5 Feb 2025 23:18:25 +0000 (UTC) X-FDA: 83087456970.17.B9D093D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id B892A40006 for ; Wed, 5 Feb 2025 23:18:23 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VA41LJHX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738797503; a=rsa-sha256; cv=none; b=vLlqMpulToX2rOu9voIuDzZFWa1nXAS0aiWRf4cykq/vdp2iGinO3FB3XdfVu47vh8+vJf kcyIGZgL4wPXOnHdGoDRQa0E4JTAgtoEfbLeECo4fVeIUdM9kA/Z5P5JSQIr96tvxB94aG c0KbAOL3k5EQMuoul2GIFVey4qmw9VU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VA41LJHX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738797503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FZa80BhZdx9gmyyMrwsj4JyFeb2ASk0SBEuM3S7wkWc=; b=QaaTUrRxsV4m0Fn54iY4K0qsJiUXfmf7bSPmQbr8x8xJSnlytX+vgJiVDC3UVBkT9Gtv/G tK9JEybJg7yUtN+BbVpywc9y0oTXEW2xnBw6ev2C2piZ39WV6pD43J6VaBSvqakecb/uZS jJcxU4iHRFhz+pXZ3eIfNVHu5pybbec= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738797503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZa80BhZdx9gmyyMrwsj4JyFeb2ASk0SBEuM3S7wkWc=; b=VA41LJHXtOLrxc5rqjO+kFP3/UHo6s/Q+9Q/M+PgRSVy0G/lS4MAILOPxAz6XqnQszD1tU U4OqxFSRN/xZEIyWmv0FVIaJ7mM3SV8HlhLQcs4TU1tq0KYzu6Rx4QAiEzeLKKeOKswd/N KWg4d3N45sXwPCgtHZ50hsh8Oo1QW7U= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-TONjRLI-McyooT1dQwr4iQ-1; Wed, 05 Feb 2025 18:18:20 -0500 X-MC-Unique: TONjRLI-McyooT1dQwr4iQ-1 X-Mimecast-MFC-AGG-ID: TONjRLI-McyooT1dQwr4iQ Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B0063180087D; Wed, 5 Feb 2025 23:18:18 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.81.141]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DC67C1800570; Wed, 5 Feb 2025 23:18:16 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org Subject: [PATCH 5/5] vfio/type1: Use mapping page mask for pfnmaps Date: Wed, 5 Feb 2025 16:17:21 -0700 Message-ID: <20250205231728.2527186-6-alex.williamson@redhat.com> In-Reply-To: <20250205231728.2527186-1-alex.williamson@redhat.com> References: <20250205231728.2527186-1-alex.williamson@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspam-User: X-Rspamd-Queue-Id: B892A40006 X-Rspamd-Server: rspam10 X-Stat-Signature: ipzcnkern531wywysmo3yhxgb5ejyqhf X-HE-Tag: 1738797503-359150 X-HE-Meta: U2FsdGVkX1/4wcH/iwUIpZcC117iZ2XNLwwky9dFfXa6ElufvfGpx10HbMwUa0CUhf6sCB1OTeNs1G5Gqj0X0XHSUFRPZ0zgfGTySpVjbeA1JruiGGN/zLQdI3pYr0BbwZNH/+Ln+t2hqR77+wmTH4vVMpQJ8GG+Kd7aHZp7MAz7BzT0rYEyIe+WLwj8emiS5tUG+43401crHx/BPXJmI01ziJS4ExtjjtGjk4Y6HFzQYkVhYFwvZ7UV/tkssQFsKlfkeMk6FuxRRQ56e4DuAeEuaUyNW/oKUbYEUMd3ppIL8jLC9ly5rNbFphGV1198M5k1vjPhRqpTugIWq9Mab6Vopr2mMYWnz1OOnm5EqE//I/McdFm4Yn6K4effN0Ss7tfFhpIoWAiHbbdZuJJxRfL5bS6IFg3/MkorxPRDkqIO5CI/q6Vd0GUcd5tpEZ8iNZNpFJHz6hGLHcRWlPD3mHy0Q8O4jekMSC+5SlHaz+1lgSH5MDQwsue69WzVV/RuQtw3y8ZDXdLRSuCEkb4xKGzZ2VKgKLa2M7iSyzVbcNl7IVaLZIsN56XYuYasAuMBM+rm3q5TyRGp5w5FxAjZbjwt9BxA5RFdGSlD4iI2ygRgS+yo6s3VotDOtO8Fc3U62JcR2mwXNTp2CVeDZA9aWNRQbhXOgn1gWcKFaEi91J2hdgvM3CX+NkKXTPsOL39LYLtYK1mqGfbESKMQ790Vfuwxok1N6bhiC0C76qoYrkmqGtyfJHnnooFaoM2ZsqJbTgBlkWGkbqhKmlpNu8RcY7igqXbTkI/0SMhsjtaHOy2ghTkIeEF1mcefJQBW3XyEdXuNI/dcdIeOaRPZEciVdsjcEP2oLdQmwPJunFOJ9nhP+kd0ut8U/dq4nvDvof0a6iWlyIfJHUM5iXMrgM+o18xAkdjYXtUNl4A+DE1KAxrDmZlvQS2n9L/DFEiVkuxDfRHtN5XQwhVVa5HjeOZ d4ZuESYX F294jjqvmYCfPTtp9xUXQoj03zMMG/xtTzc02Zx6YVaQLWN3qz61k7wKXH82FMgyR9g7xwt/CyU6BBIYkAlSp2Ma8BkmbRq4JZEbpr7AnIAATNRDca9y67Qs9YI08KDyZlgsUN4YxZSKDYMUDC60fPXtjCACJlTYIXOM5Hy4qUfuGXQwF3WA+7S0D5Eyo2hsEBNL2sVak5ZvzYlHz8JJu52aBPYzmnEY4+8OUzyLhEslZthZITODYrs02Qw2Pp8j2OixQK/bx3JWtLGXcOHJ6P968bblaPnuq9pLbWoIEVbeuwxzKy30CAXMpvFd4JC2QXi0h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and pmd mappings for well aligned mappings. follow_pfnmap_start() walks the page table and therefore knows the page mask of the level where the address is found and returns this through follow_pfnmap_args.pgmask. Subsequent pfns from this address until the end of the mapping page are necessarily consecutive. Use this information to retrieve a range of pfnmap pfns in a single pass. With optimal mappings and alignment on systems with 1GB pud and 4KB page size, this reduces iterations for DMA mapping PCI BARs by a factor of 256K. In real world testing, the overhead of iterating pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to sub-millisecond overhead. Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 939920454da7..6f3e8d981311 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -520,7 +520,7 @@ static void vfio_batch_fini(struct vfio_batch *batch) static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, - bool write_fault) + unsigned long *pgmask, bool write_fault) { struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; @@ -544,10 +544,12 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, return ret; } - if (write_fault && !args.writable) + if (write_fault && !args.writable) { ret = -EFAULT; - else + } else { *pfn = args.pfn; + *pgmask = args.pgmask; + } follow_pfnmap_end(&args); return ret; @@ -590,15 +592,23 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, vma = vma_lookup(mm, vaddr); if (vma && vma->vm_flags & VM_PFNMAP) { - ret = follow_fault_pfn(vma, mm, vaddr, pfn, prot & IOMMU_WRITE); + unsigned long pgmask; + + ret = follow_fault_pfn(vma, mm, vaddr, pfn, &pgmask, + prot & IOMMU_WRITE); if (ret == -EAGAIN) goto retry; if (!ret) { - if (is_invalid_reserved_pfn(*pfn)) - ret = 1; - else + if (is_invalid_reserved_pfn(*pfn)) { + unsigned long epfn; + + epfn = (((*pfn << PAGE_SHIFT) + ~pgmask + 1) + & pgmask) >> PAGE_SHIFT; + ret = min_t(int, npages, epfn - *pfn); + } else { ret = -EFAULT; + } } } done: -- 2.47.1