From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 808A8C77B72 for ; Fri, 14 Apr 2023 14:25:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19555900004; Fri, 14 Apr 2023 10:25:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1459F6B0075; Fri, 14 Apr 2023 10:25:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDA39900004; Fri, 14 Apr 2023 10:25:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DD5516B0071 for ; Fri, 14 Apr 2023 10:25:10 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7FC7FC0205 for ; Fri, 14 Apr 2023 14:25:10 +0000 (UTC) X-FDA: 80680218780.30.2195752 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf30.hostedemail.com (Postfix) with ESMTP id 821B080006 for ; Fri, 14 Apr 2023 14:25:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=BQzfiDtP; spf=pass (imf30.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681482308; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dV/mcZq3W0H+ZVjv4/WnTWEPlbKKMR+weOFNfLWMeuE=; b=gtQJEWi+bSFpoVrl3SjWKQvlVKKpzBpciApfvB/odLdA92RWqL/LdVSN1jtn3COrxDzve8 8EEa+aaob/A7QYqmo/kdnFpoEDtXrAfZxGJgyZcSYiazh2wLUoQYrInFsNwHolf/Vu6/7b WjHz5+IbutklV8JHEta9XP+mMY2KpOE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=BQzfiDtP; spf=pass (imf30.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681482308; a=rsa-sha256; cv=none; b=Vg1w4/4WQJCbqp1wkMjgukfJn6aPiagRw2UJ/WlrNCu5zXb5MM4/laj0jbf8F5BzzvZxsM Aeqe8HM9iiYAOu8nP81wCtkz2iMGiDCSAWSIn3hpjeWcgx2ABPBqQbXxHGFlpZ+bKIjrZ/ +OdIi2u0hwDcb+V6aFGFvODLMRoWoS0= Received: by mail-pl1-f179.google.com with SMTP id i8so9459320plt.10 for ; Fri, 14 Apr 2023 07:25:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681482307; x=1684074307; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dV/mcZq3W0H+ZVjv4/WnTWEPlbKKMR+weOFNfLWMeuE=; b=BQzfiDtPhE/M68lDOGSaU8+eH4GXZloBn+mcpK/aB1N6LYOdAjEHlkaXhXyCfvnuvb CEpHvh+6L7eTtlX4yjqbuSmzGdE8fDpHFFYGnZbcZ6MNJT9ylYo2GZtipXJ+6+UzOHTP D9tDXMnJaFlxK7VzjklsrLI+Av/83wf+aOZbs0pt3RSvdic9m6nNNQPQNVFeZpQ+CdQC 995Zkm9AnPgmLkqehhkRv+/kwi2qGfOnf/V+BnSYKMp7WXLrh660WuQiac7blN5+7+zM OxDurW2yAjeOkZJcY9VilooGc5d9qjlhXDXtnE3QoGRPpuKLJN+X+vr5d2OKSx411qOk mH+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681482307; x=1684074307; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dV/mcZq3W0H+ZVjv4/WnTWEPlbKKMR+weOFNfLWMeuE=; b=N0u7duEmYkPEdBqcSNPpAF6KziIZ/gehQxI6a3bX9GwE33bvQUyzv9VihUF335Yq/B OFIaulq1uAY39riIfCktDFny4Al1zOgLfR2OdXCc5XwXD8AXiBWxzNWtjl84WPd73j4b KWyt+/c5psxvCQvlKLGYrCGoKlzY4D3HN+3uvebPZCs1mB/St/NI93HSj5zbfILCaKgG OdGW4nOx6igBRZSkAERkLcmQqRpzpsJox6AUSh8f0q+IFBs2rc9FHfo9BeF3/1HMHnPu 6DO+zsRy8+I9fvJ/jRMe0cLrpetI/A1561Kj1RHMA7urZ+hMYTGVHBMzq7SEjAQe+Os7 qvVQ== X-Gm-Message-State: AAQBX9dlUf68U4QlAOsnvytNkb+BvgJl8Q3tgUycsRXNhv6LkFNwyhVF RktjfPAbG0zRHoTxQ/rZdok= X-Google-Smtp-Source: AKy350aLkOiAuG932gYmw+BNYfvRwYLr6hT7jeWvEtEaIuQt7YrDBI1fGEqKpu+mP4tyXM4ARraeiA== X-Received: by 2002:a17:902:ce89:b0:19f:2dff:21a4 with SMTP id f9-20020a170902ce8900b0019f2dff21a4mr3285427plg.16.1681482307277; Fri, 14 Apr 2023 07:25:07 -0700 (PDT) Received: from strix-laptop.. (2001-b011-20e0-1499-8303-7502-d3d7-e13b.dynamic-ip6.hinet.net. [2001:b011:20e0:1499:8303:7502:d3d7:e13b]) by smtp.googlemail.com with ESMTPSA id h7-20020a17090ac38700b0022335f1dae2sm2952386pjt.22.2023.04.14.07.24.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Apr 2023 07:25:06 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , "Matthew Wilcox (Oracle)" , Christophe Leroy , John Hubbard , Nadav Amit , Barry Song , Pasha Tatashin Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Peter Zijlstra , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Ian Rogers , Adrian Hunter , Yu Zhao , Steven Barrett , Juergen Gross , Peter Xu , Kefeng Wang , Tong Tiangen , Christoph Hellwig , "Liam R. Howlett" , Yang Shi , Vlastimil Babka , Alex Sierra , Vincent Whitchurch , Anshuman Khandual , Li kunyu , Liu Shixin , Hugh Dickins , Minchan Kim , Joey Gouly , Chih-En Lin , Michal Hocko , Suren Baghdasaryan , "Zach O'Keefe" , Gautam Menghani , Catalin Marinas , Mark Brown , "Eric W. Biederman" , Andrei Vagin , Shakeel Butt , Daniel Bristot de Oliveira , "Jason A. Donenfeld" , Greg Kroah-Hartman , Alexey Gladkov , x86@kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng Subject: [PATCH v5 05/17] mm: Handle COW-ed PTE during zapping Date: Fri, 14 Apr 2023 22:23:29 +0800 Message-Id: <20230414142341.354556-6-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230414142341.354556-1-shiyn.lin@gmail.com> References: <20230414142341.354556-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: kqufhjzb8cfjg53dksux7smg3hchic1s X-Rspam-User: X-Rspamd-Queue-Id: 821B080006 X-Rspamd-Server: rspam06 X-HE-Tag: 1681482308-537703 X-HE-Meta: U2FsdGVkX1/Nx7U4EaJ3uNewlYPMicxIgBNYP9uNIgou4EjBbrRqFL6ElDwCEJ2QYVh6dtKbcXIhCBXBOe01+kRdGfZ9tjqu792waU5TzeN18lOZXwxl9k4cYQQJ0O70Nm8VDC8C+bHng4YzmlCjSJEHZ84VNrH0mnYqipGL4CyHoN4NOgLMDnsR7Jju9n3NIdR+hlrQPQfzKtHGW4N/7sdD+xnt0A0cToOCHrRVl2o4gOoxfN5aACnCfrHldDvHDMWUjHUtOK0ZEnZA0zDYOdzGGNJWZC8fJ595rKllkiBKwfXJ2httFnzLMqVez9BmigNAeuTHtPAnvckjhV4pulWOBiFPZLod5oPL0MeCxQWildyN4hRPFFcLChrbqt1Eft0wmnKei/beOissVyGS4sPQERA7DJBGoAE4QNV++qGoBHB7r4RSrsGwav5+B2qT8jqqv1cEFtnxlIM/2n45LhbedhMbp4t4i9NxZ5yjn9N3GoAhbqYn9y7TKwbQCj63kDVoAWI03ccC/gXw8W92fa+LQw7Gl+oOFO+YC1qqtcvUBbNUUZJeK74ulnyu3xZrnBNSXh+FnDfW6P2RDT66JfU0GJNhWSpZkSa06odo1Pvz0by1SuKu/gCo/vQ261mCQSx5AbsT2eecmFb1+cioBxP6SZNKkEhayV2cZGKC4KSvoeoUuM/Ldy84I+VdR0ymefMeLeDKYRO8ElkL5zzeir7VAfg3i0fsRGigntTtJDdrlPX/ujsz7jfL0cRXawaQvs2grnaAdoLdID0mRoCmAvOgZutbx4wogT7gvl6DDtZ4XcW8yQ5QT9RrmQiklQIY3lcgC0A7ZzdQp/ev3VC4VXWr9ja3392mJokwwUZvNGKTCXBamCu8wmawB9w2UDB0yvfhD6IDAnNsNWPNZXyAKFw9MX3wykSdV+8k0pCmMpwkBCgJexQLu8HmH+92NRLg9ZPY6vRum1ny/QDW9HN 2aM+RNDe vMk6BvPNjmbJw8rsVGu8SyyM9pxJnX0JqcI0j5H8LSakQT9YaV2Q2wz/0cIjT0RTByR4SQzenphUamGfgApwKm1luqzIhy364/Us1KXssKZ10xQU/oRDpPggs63Guo/HwgnGLobhqlz3B/ahuipyK9dY1GUso1zYHOEsdGTo5/k/S++qmzmakbBAHlNk0d7MgtPFlXIiWzMMAIibGrfXHiprDlz+4TeDla/hbNfqV5hiGzLiK5RhG5gOs59zSzvLJ63O0TxG9OoHoFd5Lz+2dTG+K3LaXgKaKk+BFnfcLbpEi9cpoAHoAEoUwVIZ8+Tv72KDPNFRv4fjvdkj6lF63AIjPppSYbF8wBhg4rquJCLK9KRHFdV59mwRCYP1EpIaal8B6VG9FFGYXoJqX7J+pltuNuis9/fc3pXx2Y3c9TfwkBh2oNE2i+OQrnm9PLtVjUwHhrv8ylxAE2nEY9OlxcH+h0mSX45MSMrpMakBOHJ75cFq4WdIWKxdwpm3UP57JyX2nmAeHY1gU3dKS/ISopfWoa8Ifo72OdIGriOTNjdBMaqo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To support the zap functionally for COW-ed PTE, we need to zap the entire PTE table each time instead of partially zapping pages. Therefore, if the zap range covers the entire PTE table, we can handle de-account, remove the rmap, etc. However we shouldn't modify the entries when there are still someone references to the COW-ed PTE. Otherwise, if only the zapped process references to this COW-ed PTE, we just reuse it and do the normal zapping. Signed-off-by: Chih-En Lin --- mm/memory.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 87 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f8a87a0fc382..7908e20f802a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -192,6 +192,12 @@ static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, pmd = pmd_offset(pud, addr); do { next = pmd_addr_end(addr, end); +#ifdef CONFIG_COW_PTE + if (test_bit(MMF_COW_PTE, &tlb->mm->flags)) { + if (!pmd_none(*pmd) && !pmd_write(*pmd)) + VM_WARN_ON(cow_pte_count(pmd) != 1); + } +#endif if (pmd_none_or_clear_bad(pmd)) continue; free_pte_range(tlb, pmd, addr); @@ -1656,6 +1662,7 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, #define ZAP_PTE_INIT 0x0000 #define ZAP_PTE_FORCE_FLUSH 0x0001 +#define ZAP_PTE_IS_SHARED 0x0002 struct zap_pte_details { pte_t **pte; @@ -1681,9 +1688,13 @@ zap_present_pte(struct mmu_gather *tlb, struct vm_area_struct *vma, if (unlikely(!should_zap_page(details, page))) return 0; - ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + if (pte_details->flags & ZAP_PTE_IS_SHARED) + ptent = ptep_get(pte); + else + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (!(pte_details->flags & ZAP_PTE_IS_SHARED)) + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); if (unlikely(!page)) return 0; @@ -1767,8 +1778,10 @@ zap_nopresent_pte(struct mmu_gather *tlb, struct vm_area_struct *vma, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (!(pte_details->flags & ZAP_PTE_IS_SHARED)) { + pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + } } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1785,6 +1798,36 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, .flags = ZAP_PTE_INIT, .pte = &pte, }; +#ifdef CONFIG_COW_PTE + unsigned long orig_addr = addr; + + if (test_bit(MMF_COW_PTE, &mm->flags) && !pmd_write(*pmd)) { + if (!range_in_vma(vma, addr & PMD_MASK, + (addr + PMD_SIZE) & PMD_MASK)) { + /* + * We cannot promise this COW-ed PTE will also be zap + * with the rest of VMAs. So, break COW PTE here. + */ + break_cow_pte(vma, pmd, addr); + } else { + /* + * We free the batched memory before we handle + * COW-ed PTE. + */ + tlb_flush_mmu(tlb); + end = (addr + PMD_SIZE) & PMD_MASK; + addr = addr & PMD_MASK; + start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (cow_pte_count(pmd) == 1) { + /* Reuse COW-ed PTE */ + pmd_t new = pmd_mkwrite(*pmd); + set_pmd_at(tlb->mm, addr, pmd, new); + } else + pte_details.flags |= ZAP_PTE_IS_SHARED; + pte_unmap_unlock(start_pte, ptl); + } + } +#endif tlb_change_page_size(tlb, PAGE_SIZE); again: @@ -1828,7 +1871,16 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, */ if (pte_details.flags & ZAP_PTE_FORCE_FLUSH) { pte_details.flags &= ~ZAP_PTE_FORCE_FLUSH; - tlb_flush_mmu(tlb); + /* + * With COW-ed PTE, we defer freeing the batched memory until + * after we have actually cleared the COW-ed PTE's pmd entry. + * Since, if we are the only ones still referencing the COW-ed + * PTe table after we have freed the batched memory, the page + * table check will report a bug with anon_map_count != 0 in + * page_table_check_zero(). + */ + if (!(pte_details.flags & ZAP_PTE_IS_SHARED)) + tlb_flush_mmu(tlb); } if (addr != end) { @@ -1836,6 +1888,36 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, goto again; } +#ifdef CONFIG_COW_PTE + if (pte_details.flags & ZAP_PTE_IS_SHARED) { + start_pte = pte_offset_map_lock(mm, pmd, orig_addr, &ptl); + if (!pmd_put_pte(pmd)) { + pmd_t new = pmd_mkwrite(*pmd); + set_pmd_at(tlb->mm, addr, pmd, new); + /* + * We are the only ones who still referencing this. + * Clear the page table check before we free the + * batched memory. + */ + page_table_check_pte_clear_range(mm, orig_addr, *pmd); + pte_unmap_unlock(start_pte, ptl); + /* free the batched memory and flush the TLB. */ + tlb_flush_mmu(tlb); + free_pte_range(tlb, pmd, addr); + } else { + pmd_clear(pmd); + pte_unmap_unlock(start_pte, ptl); + mm_dec_nr_ptes(tlb->mm); + /* + * Someone still referencing to the table, + * we just flush TLB here. + */ + flush_tlb_range(vma, addr & PMD_MASK, + (addr + PMD_SIZE) & PMD_MASK); + } + } +#endif + return addr; } -- 2.34.1