From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE737C77B72 for ; Fri, 14 Apr 2023 14:25:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6EC0C280001; Fri, 14 Apr 2023 10:25:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C3326B0075; Fri, 14 Apr 2023 10:25:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58B4A280001; Fri, 14 Apr 2023 10:25:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 486C36B0071 for ; Fri, 14 Apr 2023 10:25:29 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 15FE9C033C for ; Fri, 14 Apr 2023 14:25:29 +0000 (UTC) X-FDA: 80680219578.04.E264052 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf13.hostedemail.com (Postfix) with ESMTP id 0135420008 for ; Fri, 14 Apr 2023 14:25:26 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ZUkBFFgD; spf=pass (imf13.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681482327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TCxBEbp3Vh2LecugBLRL5yp+Vpy2XMoG5rUe/amD8/I=; b=BeWQtXn4mmnocpLYc717MsogQgCpL30MfnHKP+q8l1fZBFP0waQbaY9EekCRd50163fFd9 ftNCCE9vwrgmS5kYiVh+lhXdeq9xkuYsJWlDPgxqIEFcyquKr4+W5kKsmW5LCJQjSvCOYT gCr0VCMWGSkLnWNnu7bdxv8+YiuDLOE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ZUkBFFgD; spf=pass (imf13.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681482327; a=rsa-sha256; cv=none; b=AqDuDRquyuQI2ZJkUz0Ojti3LfA3caPAJ3jDt8/qAqfERmfD9JX10fOj4+AnaM2fFO27SW 8VLBhe7qqtNRVUmW9SifUlUh5vV4KyCg3m4v7naItHuIoBjsMJgPdlZLTWmPuxBiRL4Fcj 0tEyO0ucKIsutnWzSQkz64H/HYijWz8= Received: by mail-pj1-f42.google.com with SMTP id my14-20020a17090b4c8e00b0024708e8e2ddso7877318pjb.4 for ; Fri, 14 Apr 2023 07:25:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681482326; x=1684074326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TCxBEbp3Vh2LecugBLRL5yp+Vpy2XMoG5rUe/amD8/I=; b=ZUkBFFgDZMzq0ei7OkZmTW87uX2xk5y2kJKyAYLp3lpS89qPRHDKJgz7tCSi6O1V6k GrFbOBQFX/XypGH/c1yklfosrEHoIqzQ10mKCvEBfVArBFpSNqifnArvfV57Wofxfked tQSdXPx2lkO9j+I9wB/qaUE15663ceimdY4la8QuwN52VLu4kI7OSpxGktPP3xTt05op 8/JPrXs0WD8ikj7mOCQTPzq3BQuATZD6Bj6rr4p7YEsHpV8Rg7fuN426Vq3Yjlwn5Htj 98xjWssljzirK0y11ZY7ToIsvcWt1GPlCV+66dUlZIWt1/ldVbpqyb0Vt9r3pd3avAS7 t9Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681482326; x=1684074326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TCxBEbp3Vh2LecugBLRL5yp+Vpy2XMoG5rUe/amD8/I=; b=lcBzc3ng3OUcbVTicPVwUxZ8xqu2IcHa0aX3e8bt9dMjcu2gKU0MAoNTUlc6aUBkEr ajrfxszv0Vmvt6AXs2BxV5q6FqmqmNkJBad1Oitjcv775m7txG8Un4Z7tT0HO1+LlGPj /F3ntC19GQnTHdSWsEJSDcTPDe+Fzn3oC4ap7odRkkPu8CrP6qCj5bOQSMC6G2KA7wnv WR4BI/c5Iyl+K8U2S3gSoYlCpnO1uqucIsZ+xJ9RghK6w/nQTKu61ysaWSP5GnNAjos8 LFmQss4q1176G3tKf1sU01QIfgCA3b5PQexKd/seLQTAww/f0yd7L604p0XsLloIqq3k boNw== X-Gm-Message-State: AAQBX9cgzb3Ev1dyTH84F391bIcYrEeYRDN+lG691a7LKnpQjaVuPxKW y1axTCKBL11bLP/0c1dm2jk= X-Google-Smtp-Source: AKy350apXLIP0kt5VCJQRQireZ4xb8EqK6nTeakZbNZ/7tzb6r5mGG/H+ghRbHSVVMAlJe/5ZJjFlQ== X-Received: by 2002:a17:90b:788:b0:246:896a:408d with SMTP id l8-20020a17090b078800b00246896a408dmr5981639pjz.14.1681482325813; Fri, 14 Apr 2023 07:25:25 -0700 (PDT) Received: from strix-laptop.. (2001-b011-20e0-1499-8303-7502-d3d7-e13b.dynamic-ip6.hinet.net. [2001:b011:20e0:1499:8303:7502:d3d7:e13b]) by smtp.googlemail.com with ESMTPSA id h7-20020a17090ac38700b0022335f1dae2sm2952386pjt.22.2023.04.14.07.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Apr 2023 07:25:25 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , "Matthew Wilcox (Oracle)" , Christophe Leroy , John Hubbard , Nadav Amit , Barry Song , Pasha Tatashin Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Peter Zijlstra , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Ian Rogers , Adrian Hunter , Yu Zhao , Steven Barrett , Juergen Gross , Peter Xu , Kefeng Wang , Tong Tiangen , Christoph Hellwig , "Liam R. Howlett" , Yang Shi , Vlastimil Babka , Alex Sierra , Vincent Whitchurch , Anshuman Khandual , Li kunyu , Liu Shixin , Hugh Dickins , Minchan Kim , Joey Gouly , Chih-En Lin , Michal Hocko , Suren Baghdasaryan , "Zach O'Keefe" , Gautam Menghani , Catalin Marinas , Mark Brown , "Eric W. Biederman" , Andrei Vagin , Shakeel Butt , Daniel Bristot de Oliveira , "Jason A. Donenfeld" , Greg Kroah-Hartman , Alexey Gladkov , x86@kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng Subject: [PATCH v5 07/17] mm/khugepaged: Break COW PTE before scanning pte Date: Fri, 14 Apr 2023 22:23:31 +0800 Message-Id: <20230414142341.354556-8-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230414142341.354556-1-shiyn.lin@gmail.com> References: <20230414142341.354556-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0135420008 X-Stat-Signature: b313mfhs8c7ec6eopt3bu6asejgsnmu4 X-HE-Tag: 1681482326-945990 X-HE-Meta: U2FsdGVkX1/fXCGuYNO++7hdG7BuPYekUYujcjGomx5jHd3sMjbxTLHYR3t6R/U7H091DAgSnS0I7Mywhb+c6XpVh4fkUWAlVBDBP9wgC/ECLML8ZzkoukuU+s3e2qZJsvS6J74vnlVCiZ1q2eQTDZroA+1m9liHHR2RFLwa43A4WENkcIwnlae6FrQ26Db6xVFWm6Xv3FMADanJKe9uA/ikW85+h17KDaFXFO7X0Xaphcx6gb6/heuoLKUC2p2KImaC4a6mUWbrnCrpqn1U7K/FhdYFTdSV3mPPZ9w3kwf64aqAlR7aqlqoNHBlpQStauVKxx03YkARNpV4GUlOI9rr9YWVnzpJqkj9LWdMIZEKl2AhU1b3l9zFGYuryVg36Nt9hOAkDUpEdhBf9CcV7Q14ebnRm0BpNR8/1JQ/6uQZw5u27Oa3amjHQ8JdFXVL7NJ5qDCWfps3aupRTMgZ20ovJypzCM5d7ri9xUCS3SzTyUrJbbYkSxJC3BdWVKTDiAuzgDC7VLva4ClCbKWbjQ9sX1F7c78iukESmhjaO0mx1dy+bfpw75OjXN/H3UxGCeYuGssSpCwtyJUHxwLIIYxGknqZLoh++lhQsttPzKA4lyxMRjk3a+VnTtX2rCJF00Bu6FlAax9vyLIo9u5aeP/3DKilNp1bkbHVrhXI/gaCHbIMsAGlcfqpOT8lh4OJiFdj6iQ15pMf+8qEHKZkGgDO5O4LYPGKROsQZ9DHpqHsAF1ey70kSVpu2jWG3EAkSV+lvlVjlHLkl7fy4/ipfYWiN4ygDyJud33BG0Y0DpFP02Rll9P2oDP0w+WhcsIDIYb3YwPuxuftkCt42tDmTEazYtJNmucIvtccTpe02R+QXlku9pjzxVIPWuLAKofOkY6Ky7JB2i4aziZTPBqn13fEGO6f/zZ8WmLluK83Ly5vXLckI/IRId8wDkaLekMTTzAwyUvHNZVse/WUW1a AZjjFZ4d 3c2UUqJE1gu+scDPi0lJZW2gC/K3VFgLWOsw5B97Fg1B235WCZzx+ShgfsIsqg6CoWYBAIVIa9xFbITsFGd2TZ1sZujmSHdvI64baUP9SAzqHYcaIjV6NCKYdoaahwk3wetAYEXxPk3uSxVcNSOh3DkCQKkZnEfStY0vUhPx/eLnHAMkKPRHHHrznXuz/7e7Fyw6ea5BHEvhP47guzLGO62pA+TH9MwFwAWlr/Af29KtfjJzsrVITNltW5RB0ocg37bj6zTtAOLwxhihQOEpPhhszydTEE0ys0tbNUjiSCcfwb7pDDInBsPcuJ07MdJkyncU/G7AfK9DEc0NywxCwV8yLNgsLlfCjr6S0ySCQZtX73mGWdnyH3Ry72ERyyx5QoP6f/WTkSTGRSIZ+LHkqCR2XH0aBgGSabP5vtqHwHp/UcukHWaEh8S8p+/B+6yDi+wIk45Pz/jI2L1OWGjBp2pY2pzBw+1yE1bTwVSBRoClf2WubktyVA4aNMNBqk+SpYGyh9fjkO19ehZl4fiUkSgIdRNj7++EAkqMa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We should not allow THP to collapse COW-ed PTE. So, break COW PTE before collapse_pte_mapped_thp() collapse to THP. Also, break COW PTE before khugepaged_scan_pmd() scan PTE. Signed-off-by: Chih-En Lin --- include/trace/events/huge_memory.h | 1 + mm/khugepaged.c | 35 +++++++++++++++++++++++++++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 3e6fb05852f9..5f2c39f61521 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -13,6 +13,7 @@ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_PMD_NONE, "pmd_none") \ EM( SCAN_PMD_MAPPED, "page_pmd_mapped") \ + EM( SCAN_COW_PTE, "cowed_pte") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 92e6f56a932d..3020fcb53691 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -31,6 +31,7 @@ enum scan_result { SCAN_PMD_NULL, SCAN_PMD_NONE, SCAN_PMD_MAPPED, + SCAN_COW_PTE, SCAN_EXCEED_NONE_PTE, SCAN_EXCEED_SWAP_PTE, SCAN_EXCEED_SHARED_PTE, @@ -886,7 +887,7 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, return SCAN_PMD_MAPPED; if (pmd_devmap(pmde)) return SCAN_PMD_NULL; - if (pmd_bad(pmde)) + if (pmd_write(pmde) && pmd_bad(pmde)) return SCAN_PMD_NULL; return SCAN_SUCCEED; } @@ -937,6 +938,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, pte_unmap(vmf.pte); continue; } + if (break_cow_pte(vma, pmd, address)) + return SCAN_COW_PTE; ret = do_swap_page(&vmf); /* @@ -1049,6 +1052,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + /* We should already handled COW-ed PTE. */ + VM_WARN_ON(test_bit(MMF_COW_PTE, &mm->flags) && !pmd_write(*pmd)); + anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1159,6 +1165,13 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + + /* Break COW PTE before we collapse the pages. */ + if (break_cow_pte(vma, pmd, address)) { + result = SCAN_COW_PTE; + goto out; + } + pte = pte_offset_map_lock(mm, pmd, address, &ptl); for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { @@ -1217,6 +1230,10 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } + /* + * If we only trigger the break COW PTE, the page usually + * still in COW mapping, which it still be shared. + */ if (page_mapcount(page) > 1) { ++shared; if (cc->is_khugepaged && @@ -1512,6 +1529,11 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* We shouldn't let COW-ed PTE collapse. */ + if (break_cow_pte(vma, pmd, haddr)) + goto drop_hpage; + VM_WARN_ON(test_bit(MMF_COW_PTE, &mm->flags) && !pmd_write(*pmd)); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1717,6 +1739,11 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + if (test_bit(MMF_COW_PTE, &mm->flags) && + !pmd_write(*pmd)) { + result = SCAN_COW_PTE; + goto unlock_next; + } collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); @@ -2154,6 +2181,11 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, swap = 0; memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + if (break_cow_pte(find_vma(mm, addr), NULL, addr)) { + result = SCAN_COW_PTE; + goto out; + } + rcu_read_lock(); xas_for_each(&xas, page, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, page)) @@ -2224,6 +2256,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } rcu_read_unlock(); +out: if (result == SCAN_SUCCEED) { if (cc->is_khugepaged && present < HPAGE_PMD_NR - khugepaged_max_ptes_none) { -- 2.34.1