From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EE23C001DF for ; Wed, 26 Jul 2023 16:32:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A02396B0071; Wed, 26 Jul 2023 12:32:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B2516B0072; Wed, 26 Jul 2023 12:32:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87C908D0001; Wed, 26 Jul 2023 12:32:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 796556B0071 for ; Wed, 26 Jul 2023 12:32:17 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 39B73C02B6 for ; Wed, 26 Jul 2023 16:32:17 +0000 (UTC) X-FDA: 81054305514.05.7A4959E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 0F285180027 for ; Wed, 26 Jul 2023 16:32:11 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690389132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jFrC6C+qOdQR9zLvGnFuG5+frQZsON5WtLz64L2QeAI=; b=OFy0KlK31An5uH0EJHhKppDDVP18Qq0kyU0rZjtJA8t3Cvxb7u4mqKnmnCDhuykYOTX6Cz aQO9ZDiuRnLyCNZyfJ1HJ1Bq2Jt+HSIYNkLs/0xojFs1YQ+eB0E0Dl6Ot5pIEd9xFhpVmx UGOugvpOtOv8eW139414dxgHHy1uLy0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690389132; a=rsa-sha256; cv=none; b=nKxColaKTGVMkFfV3a+kk1EcISJ7Z8G5N8BjYl0fuQxUKruyLQG0PVfV+F8ljitTzSfCsl tj0WzSEoSBV3fZrUtdlb93ZQcIPuXwxotZKghLzuAhHXNMND1uGGny4zp6TPiZctK7+H5M vGHKxskfAkinMs/q5mfTlPh61zvX93Y= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D0F6814BF; Wed, 26 Jul 2023 09:32:53 -0700 (PDT) Received: from [10.57.77.6] (unknown [10.57.77.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 482143F67D; Wed, 26 Jul 2023 09:32:09 -0700 (PDT) Message-ID: <188251d8-32c4-846f-8ca9-8a7c635d6172@arm.com> Date: Wed, 26 Jul 2023 17:32:07 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v3 3/3] mm: Batch-zap large anonymous folio PTE mappings To: Nathan Chancellor Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan , linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230720112955.643283-1-ryan.roberts@arm.com> <20230720112955.643283-4-ryan.roberts@arm.com> <20230726161942.GA1123863@dev-arch.thelio-3990X> From: Ryan Roberts In-Reply-To: <20230726161942.GA1123863@dev-arch.thelio-3990X> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 0F285180027 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ackxy3q8yk3991utfxby83x14kxrn9dm X-HE-Tag: 1690389131-778247 X-HE-Meta: U2FsdGVkX1/BdhoMIVrNTWn+u9rqFfMDxWsswAKKVUzD7YsM9n90uWOyrA0WSNILMLIOD9uI2IuCNPh003AsrxjS/F/+VA9AjAI6kXNrna/wjGci3kKRcbjs2EOyN9O+6DXW9ir+sDkKHhOMVWG3pNXByU3XTjCtJ3hZaUrF3aKhfGppY5dcF7Z4Yne91Do5enjNuV4AJm/ASt3XRI6ojnVbUuH7MRxQHOKsOHn6m+JV11076UM+iXEnWUp5vRKXgcubOHCZMUXdwobkQPrYlKJ5m0rXnFjLOgWKifQkUUXdkMcDyjCAsBbOpxEP4JV4JxixVEyOwXSc6EIKaOlgOsRvop1uCzUaF2Yut94FEW1HdwUMkp2TZaZ+3pyEMet+JDUV4mZcid1bHlBmxsEPo0MWZRpnml6WCod8RO/gnB0uBJ1mjp/+f09fH/2ALv+etP4i1/3UKQRj8iDpcGm8WSUfnGyfRRIv7dWpgIimzRogSPvkPR0agi3aBl4/EnqBw6zMguyi29jaI7UTICNfsydxaJZ3J4IDoJns7Au/XvQFSYTUpILhUqkfzqy4rgChxNUO8wWv8OWHOrKqz1aGJoW6St/BrK4UycDC8hToS+d8SA2xcvQ9kVCQSUmWkVrlJx2QHnt/3WjD54TJJe+qV4V0oaMDFS9zH3lElDYhEZ0EAITHFhLd/fhm9MJQNG3diGE6xp5upJZT+pn7FVJFYqj3+hVSoX4DgBt6BGv6Uhl3KIAHDXLmyhpXnm+g5aCoc7Nby6+rx2CXx6podM+abCYFxjavxN6bsaK98d0bcfgrPC6hpd0iXubD/v6T8igorzLf4tLGboEIC0InHyAc4EOJCO2GuD38eev88qGFDG/4MS1TGWoc9UfGddHtLQHNexn3xzKQKNectHcGzLTEOoQcbUxvq1pcN5o8lYUTDYFflRaY2yRCZdfLstaBBdYRi/ZPfWEmteAjDeQL8Ev wfAWzK6A K9NfZlqGpaVE+bC8lQFia448N+Kvkm7B2UfjMs3R6xH4ChIUEl6N7KJP5vp8/6O/BKRp+l/Eal7bK6m/sra+HSYVQPlAZRroX2S3T48KF19eRwVH6r5/f7yxZtEd3fgBlKoL2/BLbWYHRzhlx2Pt//uznf1QhV1IUyrsofD/tB+0yUPc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 26/07/2023 17:19, Nathan Chancellor wrote: > Hi Ryan, > > On Thu, Jul 20, 2023 at 12:29:55PM +0100, Ryan Roberts wrote: >> This allows batching the rmap removal with folio_remove_rmap_range(), >> which means we avoid spuriously adding a partially unmapped folio to the >> deferred split queue in the common case, which reduces split queue lock >> contention. >> >> Previously each page was removed from the rmap individually with >> page_remove_rmap(). If the first page belonged to a large folio, this >> would cause page_remove_rmap() to conclude that the folio was now >> partially mapped and add the folio to the deferred split queue. But >> subsequent calls would cause the folio to become fully unmapped, meaning >> there is no value to adding it to the split queue. >> >> Signed-off-by: Ryan Roberts >> --- >> mm/memory.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 120 insertions(+) >> >> diff --git a/mm/memory.c b/mm/memory.c >> index 01f39e8144ef..189b1cfd823d 100644 >> --- a/mm/memory.c >> +++ b/mm/memory.c >> @@ -1391,6 +1391,94 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, >> pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); >> } >> >> +static inline unsigned long page_cont_mapped_vaddr(struct page *page, >> + struct page *anchor, unsigned long anchor_vaddr) >> +{ >> + unsigned long offset; >> + unsigned long vaddr; >> + >> + offset = (page_to_pfn(page) - page_to_pfn(anchor)) << PAGE_SHIFT; >> + vaddr = anchor_vaddr + offset; >> + >> + if (anchor > page) { >> + if (vaddr > anchor_vaddr) >> + return 0; >> + } else { >> + if (vaddr < anchor_vaddr) >> + return ULONG_MAX; >> + } >> + >> + return vaddr; >> +} >> + >> +static int folio_nr_pages_cont_mapped(struct folio *folio, >> + struct page *page, pte_t *pte, >> + unsigned long addr, unsigned long end) >> +{ >> + pte_t ptent; >> + int floops; >> + int i; >> + unsigned long pfn; >> + struct page *folio_end; >> + >> + if (!folio_test_large(folio)) >> + return 1; >> + >> + folio_end = &folio->page + folio_nr_pages(folio); >> + end = min(page_cont_mapped_vaddr(folio_end, page, addr), end); >> + floops = (end - addr) >> PAGE_SHIFT; >> + pfn = page_to_pfn(page); >> + pfn++; >> + pte++; >> + >> + for (i = 1; i < floops; i++) { >> + ptent = ptep_get(pte); >> + >> + if (!pte_present(ptent) || pte_pfn(ptent) != pfn) >> + break; >> + >> + pfn++; >> + pte++; >> + } >> + >> + return i; >> +} >> + >> +static unsigned long try_zap_anon_pte_range(struct mmu_gather *tlb, >> + struct vm_area_struct *vma, >> + struct folio *folio, >> + struct page *page, pte_t *pte, >> + unsigned long addr, int nr_pages, >> + struct zap_details *details) >> +{ >> + struct mm_struct *mm = tlb->mm; >> + pte_t ptent; >> + bool full; >> + int i; >> + >> + for (i = 0; i < nr_pages;) { >> + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); >> + tlb_remove_tlb_entry(tlb, pte, addr); >> + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); >> + full = __tlb_remove_page(tlb, page, 0); >> + >> + if (unlikely(page_mapcount(page) < 1)) >> + print_bad_pte(vma, addr, ptent, page); >> + >> + i++; >> + page++; >> + pte++; >> + addr += PAGE_SIZE; >> + >> + if (unlikely(full)) >> + break; >> + } >> + >> + folio_remove_rmap_range(folio, page - i, i, vma); >> + >> + return i; >> +} >> + >> static unsigned long zap_pte_range(struct mmu_gather *tlb, >> struct vm_area_struct *vma, pmd_t *pmd, >> unsigned long addr, unsigned long end, >> @@ -1428,6 +1516,38 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, >> page = vm_normal_page(vma, addr, ptent); >> if (unlikely(!should_zap_page(details, page))) >> continue; >> + >> + /* >> + * Batch zap large anonymous folio mappings. This allows >> + * batching the rmap removal, which means we avoid >> + * spuriously adding a partially unmapped folio to the >> + * deferrred split queue in the common case, which >> + * reduces split queue lock contention. >> + */ >> + if (page && PageAnon(page)) { >> + struct folio *folio = page_folio(page); >> + int nr_pages_req, nr_pages; >> + >> + nr_pages_req = folio_nr_pages_cont_mapped( >> + folio, page, pte, addr, end); >> + >> + nr_pages = try_zap_anon_pte_range(tlb, vma, >> + folio, page, pte, addr, >> + nr_pages_req, details); >> + >> + rss[mm_counter(page)] -= nr_pages; >> + nr_pages--; >> + pte += nr_pages; >> + addr += nr_pages << PAGE_SHIFT; >> + >> + if (unlikely(nr_pages < nr_pages_req)) { >> + force_flush = 1; >> + addr += PAGE_SIZE; >> + break; >> + } >> + continue; >> + } >> + >> ptent = ptep_get_and_clear_full(mm, addr, pte, >> tlb->fullmm); >> tlb_remove_tlb_entry(tlb, pte, addr); >> -- >> 2.25.1 >> > > After this change in -next as commit 904d9713b3b0 ("mm: batch-zap large > anonymous folio PTE mappings"), I see the following splats several times > when booting Debian's s390x configuration (which I have mirrored at [1]) > in QEMU (bisect log below): Thanks for the bug report and sorry for the inconvenience. I'm going to need a little time to figure out a build environment for s390x and get it reproducing. Hopefully I can come back to you tomorrow with a fix. Thanks, Ryan