From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F978C3ABBC for ; Tue, 6 May 2025 05:01:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4D226B0088; Tue, 6 May 2025 01:01:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FBEA6B0089; Tue, 6 May 2025 01:01:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4B56B008A; Tue, 6 May 2025 01:01:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6F5956B0088 for ; Tue, 6 May 2025 01:01:40 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D687D81764 for ; Tue, 6 May 2025 05:01:41 +0000 (UTC) X-FDA: 83411285202.25.04D2D41 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id E898C40005 for ; Tue, 6 May 2025 05:01:39 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746507700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PW7j8Xh9Go/MjkEh3jK4l41WEBNWUsO1zFPeyn5nGIY=; b=XSXItLNaU2JVgR+wSsyqe6+q738wxkoXiaVoT3Vd3UEEsOEr9dVtjUFbDbZWEvVhFmkK5W DMyv2dkdUU6s2yOLsIOHq7rcxox/PnPkJKU9S5f6LASLu4FMEKMlpCTLbXMnb6YFO9ZI8M CChvjVxngoVYYIHjD8fNEhV2oAtlpYQ= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746507700; a=rsa-sha256; cv=none; b=xYJW006xni6+fRLZMlgSRUfwneGPQH9xxdbuDfnOImOvzJUJwUKy4YtsYXG80IAtlA3Nj3 vDeY3Uwoc+A30j7AWsbo+GGDQmjnUk2xHGjUmDiv5sMjgqQk0i1HbIirL9Yh2ZacXtDyHL qPXL/KBQTMKTYkn8dx1sFw+/CqUOey4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AA970113E; Mon, 5 May 2025 22:01:29 -0700 (PDT) Received: from K4MQJ0H1H2.emea.arm.com (K4MQJ0H1H2.blr.arm.com [10.162.43.13]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0F0973F5A1; Mon, 5 May 2025 22:01:32 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, peterx@redhat.com, ryan.roberts@arm.com, mingo@kernel.org, libang.li@antgroup.com, maobibo@loongson.cn, zhengqi.arch@bytedance.com, baohua@kernel.org, anshuman.khandual@arm.com, willy@infradead.org, ioworker0@gmail.com, yang@os.amperecomputing.com, Dev Jain Subject: [PATCH 3/3] mm: Optimize mremap() by PTE batching Date: Tue, 6 May 2025 10:30:56 +0530 Message-Id: <20250506050056.59250-4-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250506050056.59250-1-dev.jain@arm.com> References: <20250506050056.59250-1-dev.jain@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 841r4txkakrogxcty6cc8qmq9893g9x1 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E898C40005 X-Rspam-User: X-HE-Tag: 1746507699-845885 X-HE-Meta: U2FsdGVkX199aFVvyCYyjqYnHlWU8q4wnZsgo/rJfgq6GBGO60dIBkhyRuo9ZWt2gQ/atrd5Y14i5VHFnJqEJx/DA9As4KAP3DFaBUylcnZaGaPMYnvVP8YIaN9cAZL80rh6LoTJNdEBtlPqtLNVbw742FghhSmnjS1doy5PXKX5vUeXQRwJHkunc+kzZJHexdz/NotguaMykCc8l+R3nfYc3ZRJuwYJv5DERClY4gXFjzdwZcpqANGTn0voRI7fwTFeJxHvg7wCOQP4zUpzAJajRyMsiIHXMp7gRYVE1gRW8js9nlc+lo/TW7h/ET2pOltj3XLAIPrHLMa7musHQwKrKS5cxq+2WEzqSZPLeDvUpCUbDo8TdnVeOtQ6SfnP+gH0YIbpgcKpJ3UOX1da94TK6S6tGjJYBO3F/YaBWki4jfS/DJ6Sg3vilSuSIdt3oo2QB9XWOEQqo/Nvvbz3mqbzwfGk2xAHwDAqh68Odx2xQFYTluEMTjACV2s1nZm2sxkzVpPPylDpfwVMzEuWH1cX9O+i0rq312Cu+sQAV8B/5hSljbjxBKbitR/Xp8DMUzvHoM/6WRxbXSorRUA5dNXp63oZSM2BoWyYjEd9ipX7LErTNnzI83LcT30lGb4qOJS6iItiX04uRvr3AC9sqInC/IFjgT3CHLumwb6nm7jSevPIShTUBJRKQXEeKBOzT8isynyNrY9tcxWXJo2bppZDT/pkJ3dlwdBIcTjCbt6XrX+di1leCBOnT575N8cVIAAay44UywAOHbGEjZSGFQKyWUgpxoYNpsfLp1IaOUv5bTHU64VfrhsZk+vgqUIG/DD/SPuqNwUsJEB8sFjG97q10MwP9Y7D X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use folio_pte_batch() to optimize move_ptes(). Use get_and_clear_full_ptes() so as to elide TLBIs on each contig block, which was previously done by ptep_get_and_clear(). Signed-off-by: Dev Jain --- mm/mremap.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 1a08a7c3b92f..3621c07d8eea 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -176,7 +176,7 @@ static int move_ptes(struct pagetable_move_control *pmc, struct vm_area_struct *vma = pmc->old; bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma); struct mm_struct *mm = vma->vm_mm; - pte_t *old_ptep, *new_ptep, pte; + pte_t *old_ptep, *new_ptep, old_pte, pte; pmd_t dummy_pmdval; spinlock_t *old_ptl, *new_ptl; bool force_flush = false; @@ -185,6 +185,7 @@ static int move_ptes(struct pagetable_move_control *pmc, unsigned long old_end = old_addr + extent; unsigned long len = old_end - old_addr; int err = 0; + int nr; /* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma @@ -237,10 +238,14 @@ static int move_ptes(struct pagetable_move_control *pmc, for (; old_addr < old_end; old_ptep++, old_addr += PAGE_SIZE, new_ptep++, new_addr += PAGE_SIZE) { - if (pte_none(ptep_get(old_ptep))) + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int max_nr = (old_end - old_addr) >> PAGE_SHIFT; + + nr = 1; + old_pte = ptep_get(old_ptep); + if (pte_none(old_pte)) continue; - pte = ptep_get_and_clear(mm, old_addr, old_ptep); /* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the @@ -252,8 +257,17 @@ static int move_ptes(struct pagetable_move_control *pmc, * the TLB entry for the old mapping has been * flushed. */ - if (pte_present(pte)) + if (pte_present(old_pte)) { + if ((max_nr != 1) && maybe_contiguous_pte_pfns(old_ptep, old_pte)) { + struct folio *folio = vm_normal_folio(vma, old_addr, old_pte); + + if (folio && folio_test_large(folio)) + nr = folio_pte_batch(folio, old_addr, old_ptep, + old_pte, max_nr, fpb_flags, NULL, NULL, NULL); + } force_flush = true; + } + pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, 0); pte = move_pte(pte, old_addr, new_addr); pte = move_soft_dirty_pte(pte); @@ -266,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc, else if (is_swap_pte(pte)) pte = pte_swp_clear_uffd_wp(pte); } - set_pte_at(mm, new_addr, new_ptep, pte); + set_ptes(mm, new_addr, new_ptep, pte, nr); } } -- 2.30.2