From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EFFAC54E67 for ; Wed, 20 Mar 2024 11:10:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 797356B0082; Wed, 20 Mar 2024 07:10:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 747046B0083; Wed, 20 Mar 2024 07:10:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60E536B0088; Wed, 20 Mar 2024 07:10:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 50CA66B0082 for ; Wed, 20 Mar 2024 07:10:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DCBDB41215 for ; Wed, 20 Mar 2024 11:10:38 +0000 (UTC) X-FDA: 81917149356.30.6D94829 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id BED531C0027 for ; Wed, 20 Mar 2024 11:10:36 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710933037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rIyIqnmXdSr8DwknXJE+unrQTQXvO03fiEVWJ94D/pQ=; b=wsQsTSbuyXhNfrNe5NXCZmMIkzUIutNd4DVp+Egmev8Q3X4trdexi2Au7tkPvqiUcok4EA JITFL5TF2AzZZSxeq19uly2naKYilbpzLWCBRC9qxPgnpFxJ/jyUdwd82EAVfozXswqBch l8NNOTdrlb+Dl4JaZ2uB3JlXAPYAngw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710933037; a=rsa-sha256; cv=none; b=bB+4IaxfiF27WmOQNYuMdkBp/fg86sV6+o98hHlmfCtFaTg+NjKOHHONQrD41RpKf8HcZk yFM+MvFqE8QzgSLgJbkKNlav/8XW1Sc3+6e8G3zkgxb5D7YS/bZkaPJc2pnA30nZj3qKzK /pKhs9ajIWOG4oFoK7gtewiF1ALx0Sk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 24E461007; Wed, 20 Mar 2024 04:11:07 -0700 (PDT) Received: from [10.57.72.78] (unknown [10.57.72.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5A9573F762; Wed, 20 Mar 2024 04:10:30 -0700 (PDT) Message-ID: <42ae8cdf-2352-4bb9-9b22-aed92a2c9930@arm.com> Date: Wed, 20 Mar 2024 11:10:28 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 2/6] mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache() Content-Language: en-GB To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang , Barry Song <21cnbao@gmail.com>, Chris Li Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240311150058.1122862-1-ryan.roberts@arm.com> <20240311150058.1122862-3-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: <20240311150058.1122862-3-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: BED531C0027 X-Rspam-User: X-Stat-Signature: 89n8ps6fejomdhbhxe361csy7br9n9db X-Rspamd-Server: rspam03 X-HE-Tag: 1710933036-851453 X-HE-Meta: U2FsdGVkX18T94ZdSWCv3LBGyEUBVJV9o+ej9mL5v6CuRmWNvomI7T8aCOimITVGmk5V9OBwjiHgBYq9V5+4SP+xEmW4IzZglRQxSM671QQ8HfrL+UpRk3tj/QzjJgPxo5vZY9Wi+PMAefPL3MfyRzxVfyd8KxprKTMadjUBIIa0fUeLSumORHfz/p8aPKXxn6DS1fjdRBnxPcY/1hl0G7G1CoIsy4CNs65CcIGBYtpy3Kil0dctbgI/cFzWtOZVjQM/iaHI3BTF/43M9+yhmk3A/Dxzm1S0TbXEk8OrLXbXKagPay7cA3zkW/zCAlGlnzGZ5ZmW+EOHYiWiExgCRZaQnctdybjDgxOC2xGZbjGTYE20nAngv4tjSFRwYLTPOax0gq++wAFkWatSO64VRgmvV1c6LmMnNhK0X1tzqCPQpbT4782XWgIZCw9oHKvRa90s840QNTZ550a5YMUII10Ueclf7f2yPzinbKqfIX8SVoFDwdGc7axRCxqWXYF/zIJaPClxAUPmXGyKgcy8Fww76jMRgSvDQH0TZ/zCYpgyxmGDpH7ui7xgbduulPScFR9WYsLDH1Jnfrne58mn94xYnO8DkWEq249+UbyWqRkh1bnxOUfWZBSbq+Wc21Vz9seNqMDDtJTlAzpPdgcPf1p3tPAcZkwSdxwT6kkB1nM57bETekwMxTw8buPumF0m2YCdeI7xFqFz97gYOxkZdBerZd1x+EB3ejRDdhn1AqA9O/tSK+Twk3l93otrJXu5uUsaoQXKSFKRt98n9ir/nttCvSyV8iLud3/k9OK0un90CNY8qtwFZj6dj25bbPx7s5l8ZnLzyX2EgjDhT5cx1pTyjnUjL/wppMBPhgUO09oK6O+AzIeye3s4WFHtKVsvWTrgRmscrvbZ80xgAQggUCnmaZRc2GDAy7+acnoqDdxfrNOo9Z6kNPEHh+o62S40w6YGPnSz1kNJvypPBW8 IpClfJ6v vQN2IB58g08eMydS7v1JuoIfJ7ZvrLOANz/HUeoXWEKda9iaUn1RDRNQtp81IITMd1WdMhMYPvOsb5loxaP+U4qHxfLXroAK21HCTQVxnTM8Bu3+SSvJp49BdQhnnM0F6aiB/Fuz2neSvOINpCylBriED7tzso2GK+34eAgxEM1+3fBPfNco7fBTu0ku0SY5stGGR2nHJTtk/KODYv2i9TM3zKrs3zz+WIHGHgFhl1B5XAh6mqe0cqsLYoeL78dNexWvr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi David, I hate to chase, but since you provided feedback on a couple of the other patches in the series, I wondered if you missed this one? It's the one that does the batching of free_swap_and_cache(), which you suggested in order to prevent needlessly taking folio locks and refs. If you have any feedback, it would be appreciated, otherwise I'm planning to repost as-is next week (nobody else has posted comments against this patch either) as part of the updated series. Thanks, Ryan On 11/03/2024 15:00, Ryan Roberts wrote: > Now that we no longer have a convenient flag in the cluster to determine > if a folio is large, free_swap_and_cache() will take a reference and > lock a large folio much more often, which could lead to contention and > (e.g.) failure to split large folios, etc. > > Let's solve that problem by batch freeing swap and cache with a new > function, free_swap_and_cache_nr(), to free a contiguous range of swap > entries together. This allows us to first drop a reference to each swap > slot before we try to release the cache folio. This means we only try to > release the folio once, only taking the reference and lock once - much > better than the previous 512 times for the 2M THP case. > > Contiguous swap entries are gathered in zap_pte_range() and > madvise_free_pte_range() in a similar way to how present ptes are > already gathered in zap_pte_range(). > > While we are at it, let's simplify by converting the return type of both > functions to void. The return value was used only by zap_pte_range() to > print a bad pte, and was ignored by everyone else, so the extra > reporting wasn't exactly guaranteed. We will still get the warning with > most of the information from get_swap_device(). With the batch version, > we wouldn't know which pte was bad anyway so could print the wrong one. > > Signed-off-by: Ryan Roberts > --- > include/linux/pgtable.h | 28 +++++++++++++++ > include/linux/swap.h | 12 +++++-- > mm/internal.h | 48 +++++++++++++++++++++++++ > mm/madvise.c | 12 ++++--- > mm/memory.c | 13 +++---- > mm/swapfile.c | 78 ++++++++++++++++++++++++++++++----------- > 6 files changed, 157 insertions(+), 34 deletions(-) > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index 85fc7554cd52..8cf1f2fe2c25 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -708,6 +708,34 @@ static inline void pte_clear_not_present_full(struct mm_struct *mm, > } > #endif > > +#ifndef clear_not_present_full_ptes > +/** > + * clear_not_present_full_ptes - Clear consecutive not present PTEs. > + * @mm: Address space the ptes represent. > + * @addr: Address of the first pte. > + * @ptep: Page table pointer for the first entry. > + * @nr: Number of entries to clear. > + * @full: Whether we are clearing a full mm. > + * > + * May be overridden by the architecture; otherwise, implemented as a simple > + * loop over pte_clear_not_present_full(). > + * > + * Context: The caller holds the page table lock. The PTEs are all not present. > + * The PTEs are all in the same PMD. > + */ > +static inline void clear_not_present_full_ptes(struct mm_struct *mm, > + unsigned long addr, pte_t *ptep, unsigned int nr, int full) > +{ > + for (;;) { > + pte_clear_not_present_full(mm, addr, ptep, full); > + if (--nr == 0) > + break; > + ptep++; > + addr += PAGE_SIZE; > + } > +} > +#endif > + > #ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH > extern pte_t ptep_clear_flush(struct vm_area_struct *vma, > unsigned long address, > diff --git a/include/linux/swap.h b/include/linux/swap.h > index 4a8b6c60793a..f2b7f204b968 100644 > --- a/include/linux/swap.h > +++ b/include/linux/swap.h > @@ -481,7 +481,7 @@ extern int swap_duplicate(swp_entry_t); > extern int swapcache_prepare(swp_entry_t); > extern void swap_free(swp_entry_t); > extern void swapcache_free_entries(swp_entry_t *entries, int n); > -extern int free_swap_and_cache(swp_entry_t); > +extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); > int swap_type_of(dev_t device, sector_t offset); > int find_first_swap(dev_t *device); > extern unsigned int count_swap_pages(int, int); > @@ -530,8 +530,9 @@ static inline void put_swap_device(struct swap_info_struct *si) > #define free_pages_and_swap_cache(pages, nr) \ > release_pages((pages), (nr)); > > -/* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ > -#define free_swap_and_cache(e) is_pfn_swap_entry(e) > +static inline void free_swap_and_cache_nr(swp_entry_t entry, int nr) > +{ > +} > > static inline void free_swap_cache(struct folio *folio) > { > @@ -599,6 +600,11 @@ static inline int add_swap_extent(struct swap_info_struct *sis, > } > #endif /* CONFIG_SWAP */ > > +static inline void free_swap_and_cache(swp_entry_t entry) > +{ > + free_swap_and_cache_nr(entry, 1); > +} > + > #ifdef CONFIG_MEMCG > static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) > { > diff --git a/mm/internal.h b/mm/internal.h > index a3e19194079f..8dbb1335df88 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -11,6 +11,8 @@ > #include > #include > #include > +#include > +#include > #include > > struct folio_batch; > @@ -174,6 +176,52 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, > > return min(ptep - start_ptep, max_nr); > } > + > +/** > + * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries > + * @start_ptep: Page table pointer for the first entry. > + * @max_nr: The maximum number of table entries to consider. > + * @entry: Swap entry recovered from the first table entry. > + * > + * Detect a batch of contiguous swap entries: consecutive (non-present) PTEs > + * containing swap entries all with consecutive offsets and targeting the same > + * swap type. > + * > + * max_nr must be at least one and must be limited by the caller so scanning > + * cannot exceed a single page table. > + * > + * Return: the number of table entries in the batch. > + */ > +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, > + swp_entry_t entry) > +{ > + const pte_t *end_ptep = start_ptep + max_nr; > + unsigned long expected_offset = swp_offset(entry) + 1; > + unsigned int expected_type = swp_type(entry); > + pte_t *ptep = start_ptep + 1; > + > + VM_WARN_ON(max_nr < 1); > + VM_WARN_ON(non_swap_entry(entry)); > + > + while (ptep < end_ptep) { > + pte_t pte = ptep_get(ptep); > + > + if (pte_none(pte) || pte_present(pte)) > + break; > + > + entry = pte_to_swp_entry(pte); > + > + if (non_swap_entry(entry) || > + swp_type(entry) != expected_type || > + swp_offset(entry) != expected_offset) > + break; > + > + expected_offset++; > + ptep++; > + } > + > + return ptep - start_ptep; > +} > #endif /* CONFIG_MMU */ > > void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, > diff --git a/mm/madvise.c b/mm/madvise.c > index 44a498c94158..547dcd1f7a39 100644 > --- a/mm/madvise.c > +++ b/mm/madvise.c > @@ -628,6 +628,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, > struct folio *folio; > int nr_swap = 0; > unsigned long next; > + int nr, max_nr; > > next = pmd_addr_end(addr, end); > if (pmd_trans_huge(*pmd)) > @@ -640,7 +641,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, > return 0; > flush_tlb_batched_pending(mm); > arch_enter_lazy_mmu_mode(); > - for (; addr != end; pte++, addr += PAGE_SIZE) { > + for (; addr != end; pte += nr, addr += PAGE_SIZE * nr) { > + nr = 1; > ptent = ptep_get(pte); > > if (pte_none(ptent)) > @@ -655,9 +657,11 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, > > entry = pte_to_swp_entry(ptent); > if (!non_swap_entry(entry)) { > - nr_swap--; > - free_swap_and_cache(entry); > - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); > + max_nr = (end - addr) / PAGE_SIZE; > + nr = swap_pte_batch(pte, max_nr, entry); > + nr_swap -= nr; > + free_swap_and_cache_nr(entry, nr); > + clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); > } else if (is_hwpoison_entry(entry) || > is_poisoned_swp_entry(entry)) { > pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); > diff --git a/mm/memory.c b/mm/memory.c > index f2bc6dd15eb8..25c0ef1c7ff3 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -1629,12 +1629,13 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > folio_remove_rmap_pte(folio, page, vma); > folio_put(folio); > } else if (!non_swap_entry(entry)) { > - /* Genuine swap entry, hence a private anon page */ > + max_nr = (end - addr) / PAGE_SIZE; > + nr = swap_pte_batch(pte, max_nr, entry); > + /* Genuine swap entries, hence a private anon pages */ > if (!should_zap_cows(details)) > continue; > - rss[MM_SWAPENTS]--; > - if (unlikely(!free_swap_and_cache(entry))) > - print_bad_pte(vma, addr, ptent, NULL); > + rss[MM_SWAPENTS] -= nr; > + free_swap_and_cache_nr(entry, nr); > } else if (is_migration_entry(entry)) { > folio = pfn_swap_entry_folio(entry); > if (!should_zap_folio(details, folio)) > @@ -1657,8 +1658,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > pr_alert("unrecognized swap entry 0x%lx\n", entry.val); > WARN_ON_ONCE(1); > } > - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); > - zap_install_uffd_wp_if_needed(vma, addr, pte, 1, details, ptent); > + clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); > + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); > } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); > > add_mm_rss_vec(mm, rss); > diff --git a/mm/swapfile.c b/mm/swapfile.c > index df1de034f6d8..ee7e44cb40c5 100644 > --- a/mm/swapfile.c > +++ b/mm/swapfile.c > @@ -130,7 +130,11 @@ static inline unsigned char swap_count(unsigned char ent) > /* Reclaim the swap entry if swap is getting full*/ > #define TTRS_FULL 0x4 > > -/* returns 1 if swap entry is freed */ > +/* > + * returns number of pages in the folio that backs the swap entry. If positive, > + * the folio was reclaimed. If negative, the folio was not reclaimed. If 0, no > + * folio was associated with the swap entry. > + */ > static int __try_to_reclaim_swap(struct swap_info_struct *si, > unsigned long offset, unsigned long flags) > { > @@ -155,6 +159,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, > ret = folio_free_swap(folio); > folio_unlock(folio); > } > + ret = ret ? folio_nr_pages(folio) : -folio_nr_pages(folio); > folio_put(folio); > return ret; > } > @@ -895,7 +900,7 @@ static int scan_swap_map_slots(struct swap_info_struct *si, > swap_was_freed = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY); > spin_lock(&si->lock); > /* entry was freed successfully, try to use this again */ > - if (swap_was_freed) > + if (swap_was_freed > 0) > goto checks; > goto scan; /* check next one */ > } > @@ -1572,32 +1577,63 @@ bool folio_free_swap(struct folio *folio) > return true; > } > > -/* > - * Free the swap entry like above, but also try to > - * free the page cache entry if it is the last user. > - */ > -int free_swap_and_cache(swp_entry_t entry) > +void free_swap_and_cache_nr(swp_entry_t entry, int nr) > { > - struct swap_info_struct *p; > - unsigned char count; > + unsigned long end = swp_offset(entry) + nr; > + unsigned int type = swp_type(entry); > + struct swap_info_struct *si; > + unsigned long offset; > > if (non_swap_entry(entry)) > - return 1; > + return; > > - p = get_swap_device(entry); > - if (p) { > - if (WARN_ON(data_race(!p->swap_map[swp_offset(entry)]))) { > - put_swap_device(p); > - return 0; > - } > + si = get_swap_device(entry); > + if (!si) > + return; > > - count = __swap_entry_free(p, entry); > - if (count == SWAP_HAS_CACHE) > - __try_to_reclaim_swap(p, swp_offset(entry), > + if (WARN_ON(end > si->max)) > + goto out; > + > + /* > + * First free all entries in the range. > + */ > + for (offset = swp_offset(entry); offset < end; offset++) { > + if (!WARN_ON(data_race(!si->swap_map[offset]))) > + __swap_entry_free(si, swp_entry(type, offset)); > + } > + > + /* > + * Now go back over the range trying to reclaim the swap cache. This is > + * more efficient for large folios because we will only try to reclaim > + * the swap once per folio in the common case. If we do > + * __swap_entry_free() and __try_to_reclaim_swap() in the same loop, the > + * latter will get a reference and lock the folio for every individual > + * page but will only succeed once the swap slot for every subpage is > + * zero. > + */ > + for (offset = swp_offset(entry); offset < end; offset += nr) { > + nr = 1; > + if (READ_ONCE(si->swap_map[offset]) == SWAP_HAS_CACHE) { > + /* > + * Folios are always naturally aligned in swap so > + * advance forward to the next boundary. Zero means no > + * folio was found for the swap entry, so advance by 1 > + * in this case. Negative value means folio was found > + * but could not be reclaimed. Here we can still advance > + * to the next boundary. > + */ > + nr = __try_to_reclaim_swap(si, offset, > TTRS_UNMAPPED | TTRS_FULL); > - put_swap_device(p); > + if (nr == 0) > + nr = 1; > + else if (nr < 0) > + nr = -nr; > + nr = ALIGN(offset + 1, nr) - offset; > + } > } > - return p != NULL; > + > +out: > + put_swap_device(si); > } > > #ifdef CONFIG_HIBERNATION