From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1001BCFA455 for ; Wed, 23 Oct 2024 16:51:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 818E66B0095; Wed, 23 Oct 2024 12:51:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77B586B0098; Wed, 23 Oct 2024 12:51:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A6B46B0099; Wed, 23 Oct 2024 12:51:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 326AF6B0095 for ; Wed, 23 Oct 2024 12:51:04 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 04B0B120AEE for ; Wed, 23 Oct 2024 16:50:47 +0000 (UTC) X-FDA: 82705456638.28.582FC4B Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf27.hostedemail.com (Postfix) with ESMTP id 3C37840003 for ; Wed, 23 Oct 2024 16:50:43 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=flViH6L9; spf=pass (imf27.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729702184; a=rsa-sha256; cv=none; b=W6LKXAERrCHxdkjTQjiEoUasnMSk59ux7h+3NsepcOdL6tbnU2XGOFhMYYC+mlX4NkA9tj BkJBpuWfKzQzbdRFUhtJYAjfb8qC2cFG5wMSlxxcmUtyOjGisEg91/JopOSypKEM9zA4kH mzXj27o/ZGiTeexACYFVq1aJpCQ9mJI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=flViH6L9; spf=pass (imf27.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729702184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zxfn5b79k6me2s/9AvmRP7791ZkTsUdHbmXJ22J7PRo=; b=3sMtGgZ4uqx5FxV7Vh1IMSNQnOTsH+x6F8qIoN9ZmBjTgk3J44959DPAVmTJwCrN8R+d45 kZjeyfUVgnH6bur5eSOQzOVy4nhJFmqprCiaUcG+26Uuxmo+oixQ99icHxvVskJ8g3xKNz r1AJAQo7SLG2VcH2lH59IHK6uloxEKo= Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a9a6b4ca29bso607052066b.3 for ; Wed, 23 Oct 2024 09:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729702260; x=1730307060; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=zxfn5b79k6me2s/9AvmRP7791ZkTsUdHbmXJ22J7PRo=; b=flViH6L9hoeuBvj8qVgo3QNqpKqErXEf7k/062VBzMru7KLs05P+3W0VRINmMZfyoM KQb2LJ2lpy9mH8Lpkr7hM/eRJmQ1PDQUDBShNPGWkDDOqyYkupjRdqQV8pIqPGgh7PAK us0MXvuBtlNPdGO8fEHFUk3Hiy2kNENohb+u5wzmN6qe2rrl0pf2eijl1+Xi89wS5b6c hnyyD+okDusgcvQqfC+0Kz6z6nJPZ3IZ+pKChmwycXAnvbbgRl4DcPod/KirItqPVDCp VWV/qFcZjrBi2yumQ8HrBdjeptpdZMy2qPxI5PmLK3xaFhZuRxNOenj7VOHu1Gg4A+fQ 812g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729702260; x=1730307060; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zxfn5b79k6me2s/9AvmRP7791ZkTsUdHbmXJ22J7PRo=; b=Y2rYV2ELODQ8YLZTPmQppNqDoHH4L8tabubafrShFepTYUKogoQMRGp5PJgpZZHF8J 0sHeVfZDWgBDJlYyQTRmyLMJD91IaXpbCagbOOOkdq+20KrCqi46o5jbLIw5i+E7vkau UW5zMYAZlf6xlBCVLFG1IsbcdLqP7bNycI73l3XgLMEJw9FArqeOI2ypSETS/jA8+5GC IDCZqtqxYLsrzNuBKLM2OUnTZ1flbwhwpx1lf4V6dNb1viS4qaTBCaFeF/PGAJV27kai 0Q3Mcc0XVl8hF+PleNu/4F+a1ULM3+O/tE/qt6b3es0Ux1fCtCz1wTbpqVnJoabi6fWU +aRQ== X-Forwarded-Encrypted: i=1; AJvYcCWlanRMzYWjQjIFQLwolgfd3IaC8xOqhcSzlEs4gGHYx7jtX/Hmg86GRlqQP91t9b8lvjQbPW+i9g==@kvack.org X-Gm-Message-State: AOJu0YyVSLWygR7S3opQO9Dzmi7lI7jIU6CTW6gJwKZUmnVcXw6vmTh2 xtZNM7Ze134mYdVJv1vwFiI7p6+DIaJkhxk9BSNwV8WxN3GTmvJJ X-Google-Smtp-Source: AGHT+IEhrLLD5HvyIKyLI5g3e1IdyncoU1fUcI0auVC3YW7OsZscRMICc8NmT8AQ5o2gGWB19MmZHQ== X-Received: by 2002:a17:906:7310:b0:a9a:6bd:95dc with SMTP id a640c23a62f3a-a9abf92f667mr312077066b.46.1729702259873; Wed, 23 Oct 2024 09:50:59 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:eb:d0d0:c7fd:c82c? ([2620:10d:c092:500::7:ca73]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9a912d6333sm506568266b.10.2024.10.23.09.50.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 23 Oct 2024 09:50:59 -0700 (PDT) Message-ID: <27d04537-ebda-4462-b008-878390d4b211@gmail.com> Date: Wed, 23 Oct 2024 17:50:58 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp To: Zi Yan , yuzhao@google.com Cc: akpm@linux-foundation.org, linux-mm@kvack.org, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, david@redhat.com, npache@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, ryncsn@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Shuang Zhai References: <20240830100438.3623486-1-usamaarif642@gmail.com> <20240830100438.3623486-3-usamaarif642@gmail.com> <4B7449C4-D2B8-4285-973C-97178C09CE7E@nvidia.com> Content-Language: en-US From: Usama Arif In-Reply-To: <4B7449C4-D2B8-4285-973C-97178C09CE7E@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: ns7qqa19zprywxeiro84qdpucz5dbocb X-Rspamd-Queue-Id: 3C37840003 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1729702243-876411 X-HE-Meta: U2FsdGVkX18tsu/78Y0t0MUXeIUu9oSqKxbpU20DjdtkOcO+vgFo18vJcAYnI1m8tt+smzKkJUM/ofrWJNg/ry9rPLAiOz7vBF6SSfcLETXwknJu77QhDrXdyx8+d49iIu3Miak0KJZHrXX2NCrZ95AkkjdLziB27FPDzMakTTbDIule/04JkeSAUZOt3WFlrFKzPmlHQW9WpmuwTm9RFdQPTdTpR2ZpVmmW/OYrXlgFIhWnvPHJfCWZGznUekJJYKn98SIGCmfHslnVDcqwUusu11iFQa5vJt7i4SHem88jWVVY2XhB3kjJ7WFnueS9qG+4sC0Uq161xXwANgNabUf3oxljS029wiMdjELPGis5iHL2xcHb/SIZYH7kIetvT79dX/UQ9Ya3Q/1QYwfUTPzaqwqLaFKjGorZVjqrr7EgkqCMEk1KtoJcM/hld4AZfWBKIisMTpVOnEZB7B1JSMUIih3U8FTPR/6OTJPmibn9bLHLMSohBx0LBjR14jilDQg03cy8Fz+58B2xTs1MDy/h4xHRt/WwYcLIfVhvXYrSzRvKcTDfepYhOtMD74aHdpWXHVD7JC3KFF7ogCIV57C85MmLzza4RPQM0P77YwnVXUKYioiqdMKgAqmSvT/aSJ83nI7ZouwLWY2wunT6fXJ7n4jHvV//syQs/9G5/jJEPi2MbaCHooj6MQRBa9yLJILehHQFQ1FsHXW0ncXxaIXg/AkNhdXj/68GzOyWfa7+JsO/vxucNSrnpvhI0CWAkfGeN0mpBazEZaw7G+wTaaXKhYNumFl6dM/AKbAeD9NSj67JPm+W0ln9/vTY3gNW2CgLWN+HCR0c/SL4J2JtAQf+drhbMG4Vt550WIuUkKIR1Dm8GezSTik+3qMbiLNMl82H9lP8ckcS+nZ3L0ae/Cc5EKvqPdqPXzbdLi545BjvRcSXNFplUcqxYdQNmnpbKAmNkQ1B+o8uHl8ug09 BCYn0yCJ rJU3la/BjxX/Hqr0INJr4TcXgowxvd1YvMuIQVsJC2HIo1EsDT152Ocj8eFvFN/WJv7L3VsIByAh/yRgpKl7DoEZ0yOwnxfu+jR4bXg/rKFmJz/bIoQWKfYaBbMpE8/RIhj7Mmi8hVqZuy+b2W2oQgZLXGFPL6tM2Lntnjr/gGJM4zrAvJjjq3S4WOWlXJtugHOfMimajCcTBRJMpk3WFdcD9FA+zX/zZ2WZyQNOiqyFZZKFjEGCsxsiFyle9PEpdBlov+KTkUJraMX9tSUBysSu8cvIDqLyLZ7MmUWPTYaa1P3HVGGSXgxezzEp36NDBpQT25u62rPITghfjqSC3Dek3Kbm9G/fW+5Ubv7K3060os3qdpdpe+anjhUVOYK2NvrU/0XGxlQ2rN3ZXRWx3N10V6imRbIUBbdWOz4hVEkjZIB/5Vah/q220OWbILyxO51JJf1Q9mPRGkKwm20lFtyc3ejMN3uj3j3SC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 23/10/2024 17:21, Zi Yan wrote: > On 30 Aug 2024, at 6:03, Usama Arif wrote: > >> From: Yu Zhao >> >> Here being unused means containing only zeros and inaccessible to >> userspace. When splitting an isolated thp under reclaim or migration, >> the unused subpages can be mapped to the shared zeropage, hence saving >> memory. This is particularly helpful when the internal >> fragmentation of a thp is high, i.e. it has many untouched subpages. >> >> This is also a prerequisite for THP low utilization shrinker which will >> be introduced in later patches, where underutilized THPs are split, and >> the zero-filled pages are freed saving memory. >> >> Signed-off-by: Yu Zhao >> Tested-by: Shuang Zhai >> Signed-off-by: Usama Arif >> --- >> include/linux/rmap.h | 7 ++++- >> mm/huge_memory.c | 8 ++--- >> mm/migrate.c | 72 ++++++++++++++++++++++++++++++++++++++------ >> mm/migrate_device.c | 4 +-- >> 4 files changed, 75 insertions(+), 16 deletions(-) >> >> diff --git a/include/linux/rmap.h b/include/linux/rmap.h >> index 91b5935e8485..d5e93e44322e 100644 >> --- a/include/linux/rmap.h >> +++ b/include/linux/rmap.h >> @@ -745,7 +745,12 @@ int folio_mkclean(struct folio *); >> int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, >> struct vm_area_struct *vma); >> >> -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked); >> +enum rmp_flags { >> + RMP_LOCKED = 1 << 0, >> + RMP_USE_SHARED_ZEROPAGE = 1 << 1, >> +}; >> + >> +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags); >> >> /* >> * rmap_walk_control: To control rmap traversing for specific needs >> diff --git a/mm/huge_memory.c b/mm/huge_memory.c >> index 0c48806ccb9a..af60684e7c70 100644 >> --- a/mm/huge_memory.c >> +++ b/mm/huge_memory.c >> @@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, >> return false; >> } >> >> -static void remap_page(struct folio *folio, unsigned long nr) >> +static void remap_page(struct folio *folio, unsigned long nr, int flags) >> { >> int i = 0; >> >> @@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr) >> if (!folio_test_anon(folio)) >> return; >> for (;;) { >> - remove_migration_ptes(folio, folio, true); >> + remove_migration_ptes(folio, folio, RMP_LOCKED | flags); >> i += folio_nr_pages(folio); >> if (i >= nr) >> break; >> @@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, >> >> if (nr_dropped) >> shmem_uncharge(folio->mapping->host, nr_dropped); >> - remap_page(folio, nr); >> + remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0); >> >> /* >> * set page to its compound_head when split to non order-0 pages, so >> @@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, >> if (mapping) >> xas_unlock(&xas); >> local_irq_enable(); >> - remap_page(folio, folio_nr_pages(folio)); >> + remap_page(folio, folio_nr_pages(folio), 0); >> ret = -EAGAIN; >> } >> >> diff --git a/mm/migrate.c b/mm/migrate.c >> index 6f9c62c746be..d039863e014b 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list) >> return true; >> } >> >> +static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw, >> + struct folio *folio, >> + unsigned long idx) >> +{ >> + struct page *page = folio_page(folio, idx); >> + bool contains_data; >> + pte_t newpte; >> + void *addr; >> + >> + VM_BUG_ON_PAGE(PageCompound(page), page); > > This should be: > > diff --git a/mm/migrate.c b/mm/migrate.c > index e950fd62607f..7ffdbe078aa7 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -206,7 +206,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw, > pte_t newpte; > void *addr; > > - VM_BUG_ON_PAGE(PageCompound(page), page); > + if (PageCompound(page)) > + return false; > VM_BUG_ON_PAGE(!PageAnon(page), page); > VM_BUG_ON_PAGE(!PageLocked(page), page); > VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page); > > Otherwise, splitting anonymous large folios to non order-0 ones just > triggers this BUG_ON. > That makes sense, would you like to send the fix? Adding Yu Zhao to "To" incase he has any objections. Thanks, Usama >> + VM_BUG_ON_PAGE(!PageAnon(page), page); >> + VM_BUG_ON_PAGE(!PageLocked(page), page); >> + VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page); >> + >> + if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags & VM_LOCKED) || >> + mm_forbids_zeropage(pvmw->vma->vm_mm)) >> + return false; >> + >> + /* >> + * The pmd entry mapping the old thp was flushed and the pte mapping >> + * this subpage has been non present. If the subpage is only zero-filled >> + * then map it to the shared zeropage. >> + */ >> + addr = kmap_local_page(page); >> + contains_data = memchr_inv(addr, 0, PAGE_SIZE); >> + kunmap_local(addr); >> + >> + if (contains_data) >> + return false; >> + >> + newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address), >> + pvmw->vma->vm_page_prot)); >> + set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte); >> + >> + dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio)); >> + return true; >> +} >> + >> +struct rmap_walk_arg { >> + struct folio *folio; >> + bool map_unused_to_zeropage; >> +}; >> + >> /* >> * Restore a potential migration pte to a working pte entry >> */ >> static bool remove_migration_pte(struct folio *folio, >> - struct vm_area_struct *vma, unsigned long addr, void *old) >> + struct vm_area_struct *vma, unsigned long addr, void *arg) >> { >> - DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION); >> + struct rmap_walk_arg *rmap_walk_arg = arg; >> + DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); >> >> while (page_vma_mapped_walk(&pvmw)) { >> rmap_t rmap_flags = RMAP_NONE; >> @@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio *folio, >> continue; >> } >> #endif >> + if (rmap_walk_arg->map_unused_to_zeropage && >> + try_to_map_unused_to_zeropage(&pvmw, folio, idx)) >> + continue; >> >> folio_get(folio); >> pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); >> @@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio *folio, >> * Get rid of all migration entries and replace them by >> * references to the indicated page. >> */ >> -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked) >> +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) >> { >> + struct rmap_walk_arg rmap_walk_arg = { >> + .folio = src, >> + .map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE, >> + }; >> + >> struct rmap_walk_control rwc = { >> .rmap_one = remove_migration_pte, >> - .arg = src, >> + .arg = &rmap_walk_arg, >> }; >> >> - if (locked) >> + VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src); >> + >> + if (flags & RMP_LOCKED) >> rmap_walk_locked(dst, &rwc); >> else >> rmap_walk(dst, &rwc); >> @@ -934,7 +988,7 @@ static int writeout(struct address_space *mapping, struct folio *folio) >> * At this point we know that the migration attempt cannot >> * be successful. >> */ >> - remove_migration_ptes(folio, folio, false); >> + remove_migration_ptes(folio, folio, 0); >> >> rc = mapping->a_ops->writepage(&folio->page, &wbc); >> >> @@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio *src, >> struct list_head *ret) >> { >> if (page_was_mapped) >> - remove_migration_ptes(src, src, false); >> + remove_migration_ptes(src, src, 0); >> /* Drop an anon_vma reference if we took one */ >> if (anon_vma) >> put_anon_vma(anon_vma); >> @@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, >> lru_add_drain(); >> >> if (old_page_state & PAGE_WAS_MAPPED) >> - remove_migration_ptes(src, dst, false); >> + remove_migration_ptes(src, dst, 0); >> >> out_unlock_both: >> folio_unlock(dst); >> @@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, >> >> if (page_was_mapped) >> remove_migration_ptes(src, >> - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); >> + rc == MIGRATEPAGE_SUCCESS ? dst : src, 0); >> >> unlock_put_anon: >> folio_unlock(dst); >> diff --git a/mm/migrate_device.c b/mm/migrate_device.c >> index 8d687de88a03..9cf26592ac93 100644 >> --- a/mm/migrate_device.c >> +++ b/mm/migrate_device.c >> @@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns, >> continue; >> >> folio = page_folio(page); >> - remove_migration_ptes(folio, folio, false); >> + remove_migration_ptes(folio, folio, 0); >> >> src_pfns[i] = 0; >> folio_unlock(folio); >> @@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long *src_pfns, >> dst = src; >> } >> >> - remove_migration_ptes(src, dst, false); >> + remove_migration_ptes(src, dst, 0); >> folio_unlock(src); >> >> if (folio_is_zone_device(src)) >> -- >> 2.43.5 > > Best Regards, > Yan, Zi