From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25A64C4167B for ; Tue, 5 Dec 2023 13:13:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B34CB6B006E; Tue, 5 Dec 2023 08:13:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE3FF6B0082; Tue, 5 Dec 2023 08:13:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FAAC6B0083; Tue, 5 Dec 2023 08:13:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 924F36B006E for ; Tue, 5 Dec 2023 08:13:03 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 529888017A for ; Tue, 5 Dec 2023 13:13:03 +0000 (UTC) X-FDA: 81532805046.10.02BF415 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 0B4624001E for ; Tue, 5 Dec 2023 13:13:00 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701781981; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HznDrp0DTHAI0i3W3IvBFeR/diazMAfaLUibUqFOx8U=; b=sYwSwex5VIZ05AXFWd18FSdPSBqzKlwDg0G10QsqbDdsWNcuD+omgIflpzQ/eZCIbw/60M Czemz2nu3smacp8mfrEGD/LVOiaLq8GKdG+jRgXUyqPBNVt5jR2P/OsSyuyeLNeNZuQaw/ b8hr1vQHjQ6cinJllJ3+u7NiElWcwGo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701781981; a=rsa-sha256; cv=none; b=L09Ns3XIQlNSwr33Cc2SImR0EfvqMZ+wQn/HHshBFRpMPnImw1XqYPJraGbdABYhyhgfuz 5M4FizJB9Djn8WleP47AKnciBxgXFxOjAxuh3mgaPoQRoRn77NsL4xL45el0NgJWmMRtvh 81i+oZY3DmxFLMAFlwgrZWw1rsL8x1g= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7DD77139F; Tue, 5 Dec 2023 05:13:46 -0800 (PST) Received: from [10.57.73.130] (unknown [10.57.73.130]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 73CD43F5A1; Tue, 5 Dec 2023 05:12:58 -0800 (PST) Message-ID: Date: Tue, 5 Dec 2023 13:12:56 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 34/39] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Content-Language: en-GB To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu References: <20231204142146.91437-1-david@redhat.com> <20231204142146.91437-35-david@redhat.com> From: Ryan Roberts In-Reply-To: <20231204142146.91437-35-david@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: aixs4ou6kmgodstxp3we1snx4qc8pcbg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 0B4624001E X-Rspam-User: X-HE-Tag: 1701781980-507171 X-HE-Meta: U2FsdGVkX1/rNyaF+SJX9TYLXTKx4GkWftowyfvoVDfRFMpHdNf0846looCaAQAHjtbf/ISwmOig+lMRxntM75iS/DAw6egVQsh0jS0m0GjKQEVk3gRwfAbztf0ktykIV0F36oiWmQ4QvAr11Pvapzl73/NgXHQY8dEtfYS90fqfNn1BEjD4lWzocen9eBU+3+SZsNGgXso3up7Wzfdy/IYK6OJfu0EkJ2yPvCIXBrzD4cveSFMeDvdz9wz34DIPxlY85SEz+BQACcgSSAm+5RUk1Z854msJIMRkE9SeTdQK/up+B/HOp31TPP7WMAzNRkOKLKoE1BK9t9n9S0epmG25MOVBMhxwAJUE7JcTPUkCRq30Y1XoetdOAavyIUBztvroTPCu6rXuK1Ipekqlats3p7IJSmOUQfGVGYbo1JNw8uOB7VPWo/Gn9Ele+FEGJptGUHoewOi89K8UV3RoZMhygLcSr/eBwL3KX12QqIUW3i5sFypT+udkiNq2hFaIYKGfgnBXchuFuBkmZdvvGPHpr+blcH0ixjIPo3AGpX7gkbTyVVPeThXWBbMBUBJjhL1pXoEIpGmZjxA9n6AC5yejWjXCDutz+Qz4GqW+zeqg0EniE+3quVOlxBRNN/WOW0QKU5O9acRkcvEoRvvEpPcdcWLBNMIkA5YKsHb2B61B39bfg0urgsT5bciJm8ug+uipH+p20xPxiDdJXrPCOv+7fPxq0PMz+TaZZZ0Se0xvCYgab690n3SFTsG1xWRP4dMjazu6PEPZXfhfFseFkDumgd03kl2C/wc17rzYQOdfetIXbxpHXRF708e+ZzUdBQnn+m3T5QN9iVL2agJFlzt26BGk0CS3BIM8tGH1Sp73TQDzV0GqdnXwDeMH25c77t9mYODEG2v1bxmm5f3JSQsXEZyOD7teWPwgpY0uxGat2y7FzYzVjuWY4T7IZq2ksTxdpWK4z6/hehYdCp9 5BP1jsLP i5E0VoBTpGbyfJA2NIIrN8UpL2S2gpObCA1ga5qKb+D9aGGtw8MiRFdq9QKoJmcPsIlzATx6BniU/Ql8CGhhM33bSUzM0MMaJ6uytt/zzmVMKZ7W/8CXVANkBAs1ip8W0HFRJZ80GdGMnMNjyBh+Gud6n8HaJz6VlaVFCTrRjEs+9WhdbN+KX8BCibfM9Lu1FsGWx8eASZwAoUg6HAWUtn92Lhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 04/12/2023 14:21, David Hildenbrand wrote: > The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, > remove them. > > Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap > baching during fork() soon. > > Signed-off-by: David Hildenbrand > --- > include/linux/mm.h | 6 -- > include/linux/rmap.h | 145 +++++++++++++++++++++++++++++-------------- > 2 files changed, 100 insertions(+), 51 deletions(-) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index 24c1c7c5a99c0..f7565b35ae931 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -1964,12 +1964,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, > return folio_maybe_dma_pinned(folio); > } > > -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, > - struct page *page) > -{ > - return folio_needs_cow_for_dma(vma, page_folio(page)); > -} > - > /** > * is_zero_page - Query if a page is a zero page > * @page: The page to query > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > index 21d72cc602adc..84439f7720c62 100644 > --- a/include/linux/rmap.h > +++ b/include/linux/rmap.h > @@ -354,68 +354,123 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, > #endif > } > > -static inline void __page_dup_rmap(struct page *page, bool compound) > +static inline int __folio_try_dup_anon_rmap(struct folio *folio, __always_inline? > + struct page *page, unsigned int nr_pages, > + struct vm_area_struct *src_vma, enum rmap_mode mode) > { > - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); > + int i; > > - if (compound) { > - struct folio *folio = (struct folio *)page; > + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); > > - VM_BUG_ON_PAGE(compound && !PageHead(page), page); > - atomic_inc(&folio->_entire_mapcount); > - } else { > - atomic_inc(&page->_mapcount); > + /* > + * No need to check+clear for already shared PTEs/PMDs of the folio. > + * This includes PTE mappings of (order-0) KSM folios. > + */ > + if (likely(mode == RMAP_MODE_PTE)) { Presumbly if __always_inline then the compiler will remove this if/else and just keep the part indicated by mode? In which case "likely" is pretty useless? Same for all similar sites in the other patches. > + for (i = 0; i < nr_pages; i++) { > + if (PageAnonExclusive(page + i)) > + goto clear; > + } > + } else if (mode == RMAP_MODE_PMD) { > + if (PageAnonExclusive(page)) > + goto clear; > } > + goto dup; > + > +clear: > + /* > + * If this folio may have been pinned by the parent process, > + * don't allow to duplicate the mappings but instead require to e.g., > + * copy the subpage immediately for the child so that we'll always > + * guarantee the pinned folio won't be randomly replaced in the > + * future on write faults. > + */ > + if (likely(!folio_is_device_private(folio) && > + unlikely(folio_needs_cow_for_dma(src_vma, folio)))) > + return -EBUSY; > + > + if (likely(mode == RMAP_MODE_PTE)) { > + for (i = 0; i < nr_pages; i++) Do you really need to reset i=0 here? You have already checked that lower pages are shared in the above loop, so can't you just start from the first exclusive page here? > + ClearPageAnonExclusive(page + i); > + } else if (mode == RMAP_MODE_PMD) { > + ClearPageAnonExclusive(page); > + } > + > +dup: > + __folio_dup_rmap(folio, page, nr_pages, mode); > + return 0; > } > > /** > - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped > - * anonymous page > - * @page: the page to duplicate the mapping for > - * @compound: the page is mapped as compound or as a small page > - * @vma: the source vma > + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range > + * of a folio > + * @folio: The folio to duplicate the mappings of > + * @page: The first page to duplicate the mappings of > + * @nr_pages: The number of pages of which the mapping will be duplicated > + * @src_vma: The vm area from which the mappings are duplicated > * > - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. > + * The page range of the folio is defined by [page, page + nr_pages) > * > - * Duplicating the mapping can only fail if the page may be pinned; device > - * private pages cannot get pinned and consequently this function cannot fail. > + * The caller needs to hold the page table lock and the > + * vma->vma_mm->write_protect_seq. > + * > + * Duplicating the mappings can only fail if the folio may be pinned; device > + * private folios cannot get pinned and consequently this function cannot fail. > + * > + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in > + * the parent and the child. They must *not* be writable after this call. > + * > + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. > + */ > +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, > + struct page *page, unsigned int nr_pages, > + struct vm_area_struct *src_vma) > +{ > + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, > + RMAP_MODE_PTE); > +} > +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ > + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) > + > +/** > + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range > + * of a folio > + * @folio: The folio to duplicate the mapping of > + * @page: The first page to duplicate the mapping of > + * @src_vma: The vm area from which the mapping is duplicated > + * > + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) > + * > + * The caller needs to hold the page table lock and the > + * vma->vma_mm->write_protect_seq. > * > - * If duplicating the mapping succeeds, the page has to be mapped R/O into > - * the parent and the child. It must *not* get mapped writable after this call. > + * Duplicating the mapping can only fail if the folio may be pinned; device > + * private folios cannot get pinned and consequently this function cannot fail. > + * > + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in > + * the parent and the child. They must *not* be writable after this call. > * > * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. > */ > +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, > + struct page *page, struct vm_area_struct *src_vma) > +{ > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, > + RMAP_MODE_PMD); > +#else > + WARN_ON_ONCE(true); > +#endif > +} > + > static inline int page_try_dup_anon_rmap(struct page *page, bool compound, > struct vm_area_struct *vma) > { > - VM_BUG_ON_PAGE(!PageAnon(page), page); > - > - /* > - * No need to check+clear for already shared pages, including KSM > - * pages. > - */ > - if (!PageAnonExclusive(page)) > - goto dup; > - > - /* > - * If this page may have been pinned by the parent process, > - * don't allow to duplicate the mapping but instead require to e.g., > - * copy the page immediately for the child so that we'll always > - * guarantee the pinned page won't be randomly replaced in the > - * future on write faults. > - */ > - if (likely(!is_device_private_page(page) && > - unlikely(page_needs_cow_for_dma(vma, page)))) > - return -EBUSY; > + struct folio *folio = page_folio(page); > > - ClearPageAnonExclusive(page); > - /* > - * It's okay to share the anon page between both processes, mapping > - * the page R/O into both processes. > - */ > -dup: > - __page_dup_rmap(page, compound); > - return 0; > + if (likely(!compound)) > + return folio_try_dup_anon_rmap_pte(folio, page, vma); > + return folio_try_dup_anon_rmap_pmd(folio, page, vma); > } > > /**