From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB554C36010 for ; Mon, 7 Apr 2025 06:33:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 886996B0005; Mon, 7 Apr 2025 02:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80F686B0007; Mon, 7 Apr 2025 02:33:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AF646B0008; Mon, 7 Apr 2025 02:33:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 494746B0005 for ; Mon, 7 Apr 2025 02:33:45 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A29A6140D97 for ; Mon, 7 Apr 2025 06:33:44 +0000 (UTC) X-FDA: 83306281968.30.75C4650 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf02.hostedemail.com (Postfix) with ESMTP id 657518000F for ; Mon, 7 Apr 2025 06:33:41 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=u7P11cph; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744007623; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KcRhuW9PUbZW9/kAH2D13dhIoDb6tqrn1VA1EdFByS8=; b=S5ny6s/LBjlGrTGv1qmiz2R97j/UFngqD4S5hJe/onn9s0J8hYaEldX4p2pHTp+bwafbM6 MomlW5LsVufOb4dogRhrsImlAJ25dL3yE2+xqb068J8dN9NIT/RuZyG5agUYY7ajIDqplg MjTA2jaGwbUkMscUXg4K0JwFXd+wO6E= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=u7P11cph; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744007623; a=rsa-sha256; cv=none; b=Yd9S94/aec3yxjCs1WjIwHYP3gZzs8LmO0EIbjcJJLVORfumCtliQfBh7IHlPh7cHhHcHD zABgwFXoa3/1kXnFcH6TBlTY+abBAqPGQSM21IaEbJPj9h9dvXOYEwrhvB5tIpcjSpsEsR ItRq5EVhA+VJYRhN5JA3rtePUe9OzkA= DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1744007617; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=KcRhuW9PUbZW9/kAH2D13dhIoDb6tqrn1VA1EdFByS8=; b=u7P11cphJSUKsNVuDNOLIc8wV7VX21MjWsph+Nnh3T/bQVYXYx+TrmhHLEzaoJnWEKle/AsrL23GEAqezhTEFls3WqO02b9Gf2cPK5iYSbWCNjQykNA8DFaGaZhE0+ZFJDBrH+bUaQxh0TjfBKk1NfUS3sMhjPADT0ZS6GqZZ24= Received: from 30.74.144.125(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WVoJvbb_1744007614 cluster:ay36) by smtp.aliyun-inc.com; Mon, 07 Apr 2025 14:33:35 +0800 Message-ID: Date: Mon, 7 Apr 2025 14:33:34 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/2] mm: mincore: use folio_pte_batch() to batch process large folios To: David Hildenbrand , Ryan Roberts , akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, 21cnbao@gmail.com, ziy@nvidia.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <7ad05bc9299de5d954fb21a2da57f46dd6ec59d0.1742960003.git.baolin.wang@linux.alibaba.com> <54886038-3707-4ea0-bd84-00a8f4a19a6a@arm.com> <145ec273-7223-45b8-a7f6-4e593a3cc8ee@arm.com> From: Baolin Wang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 657518000F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: hy73z9armq9pj7aqjk8w77y4cutf7kx3 X-HE-Tag: 1744007621-470159 X-HE-Meta: U2FsdGVkX1/N+2NALERUITFZ4ko/cSC9paR4SbpRMBAyPhe3aGuHHWb6fYeW/54rTZqRkeht7LESgjdEMupK8nVybsSrkyjUd6ikWKJSRkRynHeJ5ZWM1IgFHdmonh4I37/RSzYy1mhqM3gkHjII7IVDEbfjidwMQ+yX2jEuFWiTO2pm/QT/VISUo4kVmeSKtygtso+V3todH3yK3Up3GgemyrDcSQ0D71KnZ8Uee9k+At5cvp8WUOpUC7BwNVUhdDWaAB0OVYnLe/M7TRCCXiGsT7nqUUcDcvAg9bfODBUfpKAP7WzumekOMRDUEBPtVzVDe53iTBREf4l8OfIWqcrx2vl/GQ197nrtipnq5Cmn1lXVLbfhSGYvKVrAy07YO54Qweh7uYx2zr3vQtRZQ8IrrZLNVhWpTEltKp2q7SmU1yNgvvaXNZCkwNP16YD/vRXriwaf/lRoXf27nAz7IFa3mYcI2poz+A6ZZvQcf/GEcEOOIxkCRMH1sQHq+2aSpEYqqLfsdBpeTLLYedB3ZlKp2WJkMScD/YO/VSAAywTDhof98WCI2iWCu9HXB7L19C/Qp0RGiYuFEhqh1mPVYEkh730Tu1cFv2CXacWxQv/HEJ2cGeaeL2kaapozJeQOKBV/EAxqvkYwaqfeqajGa3jlkhAb5C4POvB51r9m/xfEng6vqktreqfKhozrC7JYy09CA+l+rHIFVApZHBgVPz+kb9JvOdRQkKE4mziSflf4nHVrQAdvXldA6mWpx+kOyyPtw9wmfDXGfk16Zw0GeUMpY9cLTv2qkyWXut1fKPdqFdPlbimfr+DE8XWUIIV2nyb7+zKvKPS+uwjvxN/k/88WtnfrEquDnURYb/LVBHMZY7I07FMZdSr1wGrFZJhu0eJXL7Wh63BRPYDyPLTXM6zf5OawMZ4P+onTsMI4NjNEyWE/XRxQUv74tC2aK5mCYaAZ4raEomd2WS9l6WX hIzRGaSi 6aSjEyWsfrXkE6cJad8hmSOP33EH/LQeLrS4Ub0ARxpXzJZc3FDC0Ft/czFzJRVjb1a7Gna4klJ9DNEd+8lYpjNL8tiKiCzS+9BPGPrJ4Prv6s8psAbqwFpIyA3kcEkoT3PuOo7F/4k2plxLK0GZGSHk6tAGxDDH/Hyq+JWrDsXfPblIxtjzFJEF+8PEWDs8zFWOYs8ack4iKnsHYTm2BJcvrbxdsLe5q81UoXVibqKr9jed8XLbRWrve/PDSIFJhaJmHFj2MvyyK5yI8V0SBynVNfT+ayZrCcLBe11mlamZS+MK5RlrJ0rBcDPJj9IFJtDXsNIyos3v/7n/cFEcwLJDz8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2025/4/1 21:04, David Hildenbrand wrote: > On 01.04.25 12:45, Ryan Roberts wrote: >> On 30/03/2025 15:57, Baolin Wang wrote: >>> >>> >>> On 2025/3/27 22:08, Ryan Roberts wrote: >>>> On 25/03/2025 23:38, Baolin Wang wrote: >>>>> When I tested the mincore() syscall, I observed that it takes >>>>> longer with >>>>> 64K mTHP enabled on my Arm64 server. The reason is the >>>>> mincore_pte_range() >>>>> still checks each PTE individually, even when the PTEs are contiguous, >>>>> which is not efficient. >>>>> >>>>> Thus we can use folio_pte_batch() to get the batch number of the >>>>> present >>>>> contiguous PTEs, which can improve the performance. I tested the >>>>> mincore() >>>>> syscall with 1G anonymous memory populated with 64K mTHP, and >>>>> observed an >>>>> obvious performance improvement: >>>>> >>>>> w/o patch        w/ patch        changes >>>>> 6022us            1115us            +81% >>>>> >>>>> Moreover, I also tested mincore() with disabling mTHP/THP, and did not >>>>> see any obvious regression. >>>>> >>>>> Signed-off-by: Baolin Wang >>>>> --- >>>>>    mm/mincore.c | 27 ++++++++++++++++++++++----- >>>>>    1 file changed, 22 insertions(+), 5 deletions(-) >>>>> >>>>> diff --git a/mm/mincore.c b/mm/mincore.c >>>>> index 832f29f46767..88be180b5550 100644 >>>>> --- a/mm/mincore.c >>>>> +++ b/mm/mincore.c >>>>> @@ -21,6 +21,7 @@ >>>>>      #include >>>>>    #include "swap.h" >>>>> +#include "internal.h" >>>>>      static int mincore_hugetlb(pte_t *pte, unsigned long hmask, >>>>> unsigned long >>>>> addr, >>>>>                unsigned long end, struct mm_walk *walk) >>>>> @@ -105,6 +106,7 @@ static int mincore_pte_range(pmd_t *pmd, >>>>> unsigned long >>>>> addr, unsigned long end, >>>>>        pte_t *ptep; >>>>>        unsigned char *vec = walk->private; >>>>>        int nr = (end - addr) >> PAGE_SHIFT; >>>>> +    int step, i; >>>>>          ptl = pmd_trans_huge_lock(pmd, vma); >>>>>        if (ptl) { >>>>> @@ -118,16 +120,31 @@ static int mincore_pte_range(pmd_t *pmd, >>>>> unsigned long >>>>> addr, unsigned long end, >>>>>            walk->action = ACTION_AGAIN; >>>>>            return 0; >>>>>        } >>>>> -    for (; addr != end; ptep++, addr += PAGE_SIZE) { >>>>> +    for (; addr != end; ptep += step, addr += step * PAGE_SIZE) { >>>>>            pte_t pte = ptep_get(ptep); >>>>>    +        step = 1; >>>>>            /* We need to do cache lookup too for pte markers */ >>>>>            if (pte_none_mostly(pte)) >>>>>                __mincore_unmapped_range(addr, addr + PAGE_SIZE, >>>>>                             vma, vec); >>>>> -        else if (pte_present(pte)) >>>>> -            *vec = 1; >>>>> -        else { /* pte is a swap entry */ >>>>> +        else if (pte_present(pte)) { >>>>> +            if (pte_batch_hint(ptep, pte) > 1) { >>>>> +                struct folio *folio = vm_normal_folio(vma, addr, >>>>> pte); >>>>> + >>>>> +                if (folio && folio_test_large(folio)) { >>>>> +                    const fpb_t fpb_flags = FPB_IGNORE_DIRTY | >>>>> +                                FPB_IGNORE_SOFT_DIRTY; >>>>> +                    int max_nr = (end - addr) / PAGE_SIZE; >>>>> + >>>>> +                    step = folio_pte_batch(folio, addr, ptep, pte, >>>>> +                            max_nr, fpb_flags, NULL, NULL, NULL); >>>>> +                } >>>>> +            } >>>> >>>> You could simplify to the following, I think, to avoid needing to >>>> grab the folio >>>> and call folio_pte_batch(): >>>> >>>>              else if (pte_present(pte)) { >>>>                  int max_nr = (end - addr) / PAGE_SIZE; >>>>                  step = min(pte_batch_hint(ptep, pte), max_nr); >>>>              } ... >>>> >>>> I expect the regression you are seeing here is all due to calling >>>> ptep_get() for >>>> every pte in the contpte batch, which will cause 16 memory reads per >>>> pte (to >>>> gather the access/dirty bits). For small folios its just 1 read per >>>> pte. >>> >>> Right. >>> >>>> pte_batch_hint() will skip forward in blocks of 16 so you now end up >>>> with the >>>> same number as for the small folio case. You don't need all the >>>> fancy extras >>>> that folio_pte_batch() gives you here. >>> >>> Sounds reasonable. Your suggestion looks simple, but my method can >>> batch the >>> whole large folio (such as large folios containing more than 16 >>> contiguous PTEs) >>> at once. >> >> Sure but folio_pte_batch() just implements that with another loop that >> calls >> pte_batch_hint(), so it all amounts to the same thing. In fact there >> are some >> extra checks in folio_pte_batch() that you don't need so it might be a >> bit slower. Right. I tested your suggestion, yes, much better. > I don't enjoy open-coding the batching, especially if only cont-pte > users will benefit from it. But I also don't enjoy the open-coded > pte_batch_hint() :) > > But we really don't need the folio here, so I assume the short variant > you (Ryan) suggest is alright to just avoid the ptep_get(). > > As Oscar says, these details might soon be hidden inside a new page > table walker API (even though it will likely end up using > folio_pte_batch() internally, TBD). OK. I can drop this patch if it will be addressed in the following patches.