From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42272C4167B for ; Tue, 5 Dec 2023 12:52:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFA556B0078; Tue, 5 Dec 2023 07:52:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CAA286B007E; Tue, 5 Dec 2023 07:52:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC01F6B0080; Tue, 5 Dec 2023 07:52:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ABE846B0078 for ; Tue, 5 Dec 2023 07:52:44 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7EF43160141 for ; Tue, 5 Dec 2023 12:52:44 +0000 (UTC) X-FDA: 81532753848.13.82CA315 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 6448CA0011 for ; Tue, 5 Dec 2023 12:52:42 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701780762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1DpMF9u9LT3f4T4qNMW6sTPN+qjWTgWbSid1qtWB+o4=; b=H4i2qMGvfE6vlczFdyWkvN6i/X1rRJZubX1tHalq5Hc6WohUZryvdM1uu/czc1dXDJXh4H SJ8qusKv+B3IEyWjqJ+oqo6hEYAkheRExVD4Pw2i9ibmG94ydIaI8U5BX+zjAw+zevzDQJ aWnhv/2RUWV2HCrdlaoVuBgPXU4acXY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701780762; a=rsa-sha256; cv=none; b=t80G2eSPzLYHvGZIe4oLwzU/6xqnsuRgWBJysLr3ysYwyFjQZn0Y33s9yxzcm3ORFaFZYu F1HrLW8bKvABzsKgejLuz0sWfN4Mue4/AI6KHcK9aggsDVx/S9WhrBtZstxKvjrMX6u3Bm CeR2ITHeSnNhPK+u1TJedUP7lgmihfM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 24428139F; Tue, 5 Dec 2023 04:53:28 -0800 (PST) Received: from [10.57.73.130] (unknown [10.57.73.130]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E5BAA3F5A1; Tue, 5 Dec 2023 04:52:39 -0800 (PST) Message-ID: Date: Tue, 5 Dec 2023 12:52:38 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 23/39] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Content-Language: en-GB To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu References: <20231204142146.91437-1-david@redhat.com> <20231204142146.91437-24-david@redhat.com> From: Ryan Roberts In-Reply-To: <20231204142146.91437-24-david@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6448CA0011 X-Stat-Signature: 9ofuz475b5kpxedhecccx9udou476a8i X-Rspam-User: X-HE-Tag: 1701780762-304660 X-HE-Meta: U2FsdGVkX1/7UX95bqhuJ9K1yC9jBnxGa+Kzj8puvmfKmT0DtkBVgZ43tnKUNDOxesxwtcwi04gG2GfB+uN6rLLVCd/MDerROoDq0SvoK58EwGmEvzS2i1HMaYcNWeji5S5ofh5JQ9eExo4Is+Bn1OSaIx5f8SqGp8GgBQwgRDUeqoffEGt/qGHdUn+Do9d0WytHfgYMOM51qkejqkJtVQbxN07ggukZrYp3vCpUTHJu3vE3Eok/82jpkDL7SvsUwn/4W12Twk4m8Zl0vztkWiphY8vrXk+XjYkxueTRl0VZekWydfdz1q7SQDK+RPau43a0vmaLTi/qx0hitInJmfww1Cj7L6HUj3fOtpoIPq6ZpuR0RCRDju+3kKAXY3p6DDcfsSeH/pn/JO7algPmDG8XZXCBuVXYyxPHp7GMKFS+5TjLDZHb8BJ25vrURcJUAG57zavcONfBRrOfjuCMbZ6xelukoRqzXwY40+dSt9UI224NKegb1bBGJUXCKgnpLQ5X8J/aUOqJCbofuwKqCIOKc+6OKQ9Q8/SNd/8Wd0pRNGprD9sDc3CuFo+/iXy9nXq+owgDgjosfH712F6jnvel2Ngapywegpmda5JRL4ACLne7BR8pfCIZo2aukmvkM568qOKT6manm72GKK+ll+qioKsU3+zj0DwREQhYuLuqQI9iqi70Y/RY84vJFAZsRnP9759VFqyYQ1mvvy/OhjGihbvAH0i1bCFXOTpK55etUJiRvN+O7U1uEEIc4lOARlWIDwQDHqUZiRwViRT4YyUUM/EglRs1LpWVPEqG5kb4QanzpYQijTCoKL4boikPZz+Y77L1LIsG86+i6qsWrtb3fAiBBgJ3mD6qIDc/6/+S42kx+wRqaIX73kYsrYiSPcoxSrcZaC5v/L66i40ZadANpopVudrmswRPn3QmmZvP1ax9cV9zCiKD1cPtpFAJwV6Fnxu49W+Qt87glhR cqcG3ZTE 1wyTwA1W0eBHFjwVA87SCwUy5B6R+vvJrvvvPHj9lfZJ/kOBfoElJnOZmlSxD6YyraHgOszKiOE8kVW288noVSuCarHjttsJ8Txsnh8EzE0MT+jjppGTiun8fqVqEwc/3/uf6euZh31WK2c6awLnF9oDkwxDYrAtqlnIM9Te/07RXXeVUyu/Dn3p98A8DEHynYZfu+9SCDXfS+uxVFqPv4eio3kTQZyv8o7Um8UsDMZiSRrFHf97Dc8cHKL8kOq2i1S8v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 04/12/2023 14:21, David Hildenbrand wrote: > Let's mimic what we did with folio_add_file_rmap_*() and > folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() > next. > > Make the compiler always special-case on the granularity by using > __always_inline. > > We're adding folio_remove_rmap_ptes() handling right away, as we want to > use that soon for batching rmap operations when unmapping PTE-mapped > large folios. > > Signed-off-by: David Hildenbrand > --- > include/linux/rmap.h | 6 ++++ > mm/rmap.c | 76 ++++++++++++++++++++++++++++++++++++-------- > 2 files changed, 68 insertions(+), 14 deletions(-) > > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > index 017b216915f19..dd4ffb1d8ae04 100644 > --- a/include/linux/rmap.h > +++ b/include/linux/rmap.h > @@ -241,6 +241,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, > struct vm_area_struct *); > void page_remove_rmap(struct page *, struct vm_area_struct *, > bool compound); > +void folio_remove_rmap_ptes(struct folio *, struct page *, unsigned int nr, > + struct vm_area_struct *); > +#define folio_remove_rmap_pte(folio, page, vma) \ > + folio_remove_rmap_ptes(folio, page, 1, vma) > +void folio_remove_rmap_pmd(struct folio *, struct page *, > + struct vm_area_struct *); > > void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, > unsigned long address, rmap_t flags); > diff --git a/mm/rmap.c b/mm/rmap.c > index 3587225055c5e..50b6909157ac1 100644 > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -1463,25 +1463,36 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, > bool compound) > { > struct folio *folio = page_folio(page); > + > + if (likely(!compound)) > + folio_remove_rmap_pte(folio, page, vma); > + else > + folio_remove_rmap_pmd(folio, page, vma); > +} > + > +static __always_inline void __folio_remove_rmap(struct folio *folio, > + struct page *page, unsigned int nr_pages, > + struct vm_area_struct *vma, enum rmap_mode mode) > +{ > atomic_t *mapped = &folio->_nr_pages_mapped; > - int nr = 0, nr_pmdmapped = 0; > - bool last; > + int last, nr = 0, nr_pmdmapped = 0; nit: you're being inconsistent across the functions with signed vs unsigned for page counts (e.g. nr, nr_pmdmapped) - see __folio_add_rmap(), __folio_add_file_rmap(), __folio_add_anon_rmap(). I suggest pick one and stick to it. Personally I'd go with signed int (since that's what all the counters in struct folio that we are manipulating are, underneath the atomic_t) then check that nr_pages > 0 in __folio_rmap_sanity_checks(). > enum node_stat_item idx; > > - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); > - VM_BUG_ON_PAGE(compound && !PageHead(page), page); > + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); > > /* Is page being unmapped by PTE? Is this its last map to be removed? */ > - if (likely(!compound)) { > - last = atomic_add_negative(-1, &page->_mapcount); > - nr = last; > - if (last && folio_test_large(folio)) { > - nr = atomic_dec_return_relaxed(mapped); > - nr = (nr < COMPOUND_MAPPED); > - } > - } else if (folio_test_pmd_mappable(folio)) { > - /* That test is redundant: it's for safety or to optimize out */ > + if (likely(mode == RMAP_MODE_PTE)) { > + do { > + last = atomic_add_negative(-1, &page->_mapcount); > + if (last && folio_test_large(folio)) { > + last = atomic_dec_return_relaxed(mapped); > + last = (last < COMPOUND_MAPPED); > + } > > + if (last) > + nr++; > + } while (page++, --nr_pages > 0); > + } else if (mode == RMAP_MODE_PMD) { > last = atomic_add_negative(-1, &folio->_entire_mapcount); > if (last) { > nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); > @@ -1517,7 +1528,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, > * is still mapped. > */ > if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) folio_test_pmd_mappable() -> folio_test_large() Since you're converting this to support batch PTE removal, it might as well also support smaller-than-pmd too? I currently have a patch to do this same change in the multi-size THP series. > - if (!compound || nr < nr_pmdmapped) > + if (mode == RMAP_MODE_PTE || nr < nr_pmdmapped) > deferred_split_folio(folio); > } > > @@ -1532,6 +1543,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, > munlock_vma_folio(folio, vma); > } > > +/** > + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio > + * @folio: The folio to remove the mappings from > + * @page: The first page to remove > + * @nr_pages: The number of pages that will be removed from the mapping > + * @vma: The vm area from which the mappings are removed > + * > + * The page range of the folio is defined by [page, page + nr_pages) > + * > + * The caller needs to hold the page table lock. > + */ > +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, > + unsigned int nr_pages, struct vm_area_struct *vma) > +{ > + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); > +} > + > +/** > + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio > + * @folio: The folio to remove the mapping from > + * @page: The first page to remove > + * @vma: The vm area from which the mapping is removed > + * > + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) > + * > + * The caller needs to hold the page table lock. > + */ > +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, > + struct vm_area_struct *vma) > +{ > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); > +#else > + WARN_ON_ONCE(true); > +#endif > +} > + > /* > * @arg: enum ttu_flags will be passed to this argument > */