From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22B1CC3ABAC for ; Tue, 6 May 2025 10:10:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D604E6B0089; Tue, 6 May 2025 06:10:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0C3F6B008A; Tue, 6 May 2025 06:10:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD7E66B008C; Tue, 6 May 2025 06:10:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9EE126B0089 for ; Tue, 6 May 2025 06:10:54 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 03E761621D9 for ; Tue, 6 May 2025 10:10:56 +0000 (UTC) X-FDA: 83412064554.15.E551C90 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 2E902180004 for ; Tue, 6 May 2025 10:10:55 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746526255; a=rsa-sha256; cv=none; b=vL/LQPG0LAwTfbjQrHmtXqNb710utA9xcgDbVb5+Z8/anP6Rtm3A0SoqK5MsejQ0BbcmkB AYw0NCOrZNIdYF2NnbonCsyHlwpWJN4Q0j1mJLAhR3Oz6szyc9MydRKYbZnYoJqlNjn/4z PQflBHOKam/zw7Neav+VaoEwmxFc4rE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746526255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VZivUpkxio9yoGzxvu/qDqKh8TbJdLOcSgmNZyuzuf8=; b=5XR2nSkQilqPcdBBHl0XZCz9hf7hlwQcxpzLP9fbtqDyUwF1OG4pCLZOd6quHrDmXHlC7K JtboT83oQTDo0PoPwbOY+oBqj2G9oQIJmkZlY/L9epEe3c7cAh+UGCLpVTVm6LER8kiJd5 zcA68nObsJNsiQky1oQAu9rZ5LxLXWM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AC3C8113E; Tue, 6 May 2025 03:10:44 -0700 (PDT) Received: from [10.163.53.181] (unknown [10.163.53.181]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBDCC3F5A1; Tue, 6 May 2025 03:10:44 -0700 (PDT) Message-ID: Date: Tue, 6 May 2025 15:40:32 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 3/3] mm: Optimize mremap() by PTE batching To: Dev Jain , akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, peterx@redhat.com, ryan.roberts@arm.com, mingo@kernel.org, libang.li@antgroup.com, maobibo@loongson.cn, zhengqi.arch@bytedance.com, baohua@kernel.org, willy@infradead.org, ioworker0@gmail.com, yang@os.amperecomputing.com References: <20250506050056.59250-1-dev.jain@arm.com> <20250506050056.59250-4-dev.jain@arm.com> Content-Language: en-US From: Anshuman Khandual In-Reply-To: <20250506050056.59250-4-dev.jain@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 2E902180004 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: oykpmau1zdo7rp39p38h8w4rtkhiccen X-HE-Tag: 1746526255-664208 X-HE-Meta: U2FsdGVkX1+nsdRT472BuY7QUK2uBzr8pwjyKPA1JGt9AxvGVhZcJf7HR+eFZhdBCQLu/8N9duRRu9v96BSBHd5JeRlidbmMGPh93ZSu+RMykL+029u2g3tDvb/1knfiBTQ5G/nxs3V5J3Xok/JEICy1g7UT2Bc9QziRck/bqkySSqDVs3D/RYeDq1P5EsEavht2sSt0UFhDNM6IW3K+4jPGAi6RIe6uKxweOOdmOTWDG8K+SGLnk6Li9y114JSfkM9Mnx8KQ1MtJdsvl8FLrZ1ylEl4lbsJK+b0raDZ/jgMWPdEASQd1qssU+63+g8P7Hku9Inz/NHgskTHImq9pqWrfZ24IzSCmjMNFDsf6HsnYVzRCgs9B8w+zA1g/0Kub3sUCgNi411sV5PEmMhzZJ1qKEJEWsU8txeBgQZLPIEh/JeiAK1d7rCRlXq4iS1cpjaqzj+lca1hMGdSxsD7cfBU27B17Yiy+QjgSBF4UxplTE3ht57YpdDaP9oCqqmYEVfKtGtd9gg4p96BV2kpYs2euhS5KcW88nLqHFW/DpM0zz3zVDgEg+jTdzre4scmuUhRz9iO3PVzoW/4eHH3T+3PxbE6wb7bct4ZB3Xx9/dyVa0InFnL2346k1akCD1LWI7pLqs10A6glj2wPo4urmjPn7Ia9I3FFMc9Do/PwzaiwPSGrYKxHE6hWREWfank6bptpCyY14xFckMR3PXD/rcfjABOQpdDhgqQdgLkdxI/VtMbwI4Kz7JzS7+sB15b3gaurlQlGcyQMLNT9rQkSzNGKlI8shQlMOSxKobkIkFx5yftCvCScDwfNm8cnTWP8jN/i/M2a1sYGpUGS2fL+O2rDEU9MepXlpEcYVkOEsaW5KY3PB5LtbQuS/CPd3JyOCv1w/KOSfy+t0LSElSYhYx9sonUZvTZpPN46pwG06qkV5VUU3/GdD+YZsQ5jfazDtteDjKCGCXpLlUpch5 9slYDoOo Db4bSSXNj466ApMZyAYu6UdKHBqE9ENVL8iM8y/eE6WQpzaePBQ/VtrKr2cIqi282L3iTEpRgtcW8+OptZbGCCUd3pB9qgGna/6A3A9h7esQ+PYtbzggJgNHQ1Lm7pyhfDjhBsk9YG4QyGrZ1/Y4osRuF5SIVH3wLHyG8BgRWKn4RCIY3/RsarrwAfsP/E1MpHnGHlcH3yoNUwmM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 5/6/25 10:30, Dev Jain wrote: > Use folio_pte_batch() to optimize move_ptes(). Use get_and_clear_full_ptes() > so as to elide TLBIs on each contig block, which was previously done by > ptep_get_and_clear(). > > Signed-off-by: Dev Jain > --- > mm/mremap.c | 24 +++++++++++++++++++----- > 1 file changed, 19 insertions(+), 5 deletions(-) > > diff --git a/mm/mremap.c b/mm/mremap.c > index 1a08a7c3b92f..3621c07d8eea 100644 > --- a/mm/mremap.c > +++ b/mm/mremap.c > @@ -176,7 +176,7 @@ static int move_ptes(struct pagetable_move_control *pmc, > struct vm_area_struct *vma = pmc->old; > bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma); > struct mm_struct *mm = vma->vm_mm; > - pte_t *old_ptep, *new_ptep, pte; > + pte_t *old_ptep, *new_ptep, old_pte, pte; > pmd_t dummy_pmdval; > spinlock_t *old_ptl, *new_ptl; > bool force_flush = false; > @@ -185,6 +185,7 @@ static int move_ptes(struct pagetable_move_control *pmc, > unsigned long old_end = old_addr + extent; > unsigned long len = old_end - old_addr; > int err = 0; > + int nr; > > /* > * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma > @@ -237,10 +238,14 @@ static int move_ptes(struct pagetable_move_control *pmc, > > for (; old_addr < old_end; old_ptep++, old_addr += PAGE_SIZE, > new_ptep++, new_addr += PAGE_SIZE) { > - if (pte_none(ptep_get(old_ptep))) > + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; > + int max_nr = (old_end - old_addr) >> PAGE_SHIFT; > + > + nr = 1; > + old_pte = ptep_get(old_ptep); > + if (pte_none(old_pte)) > continue; > > - pte = ptep_get_and_clear(mm, old_addr, old_ptep); > /* > * If we are remapping a valid PTE, make sure > * to flush TLB before we drop the PTL for the > @@ -252,8 +257,17 @@ static int move_ptes(struct pagetable_move_control *pmc, > * the TLB entry for the old mapping has been > * flushed. > */ > - if (pte_present(pte)) > + if (pte_present(old_pte)) { > + if ((max_nr != 1) && maybe_contiguous_pte_pfns(old_ptep, old_pte)) { maybe_contiguous_pte_pfns() cost will be applicable for memory areas greater than a single PAGE_SIZE (i.e max_nr != 1) ? This helper extracts an additional consecutive pte, ensures that it is valid mapped and extracts pfn before comparing for the span. There is some cost associated with the above code sequence which looks justified for sequential access of memory buffers that has consecutive physical memory backing. But what happens when such buffers are less probable, will those buffers take a performance hit for all the comparisons that just turn out to be negative ? > + struct folio *folio = vm_normal_folio(vma, old_addr, old_pte); > + > + if (folio && folio_test_large(folio)) > + nr = folio_pte_batch(folio, old_addr, old_ptep, > + old_pte, max_nr, fpb_flags, NULL, NULL, NULL); > + } > force_flush = true; > + } > + pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, 0); > pte = move_pte(pte, old_addr, new_addr); > pte = move_soft_dirty_pte(pte); > > @@ -266,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc, > else if (is_swap_pte(pte)) > pte = pte_swp_clear_uffd_wp(pte); > } > - set_pte_at(mm, new_addr, new_ptep, pte); > + set_ptes(mm, new_addr, new_ptep, pte, nr); > } > } >