From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB19BC636CC for ; Tue, 7 Feb 2023 04:01:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45C9D6B0071; Mon, 6 Feb 2023 23:01:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E5D46B0073; Mon, 6 Feb 2023 23:01:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238796B0074; Mon, 6 Feb 2023 23:01:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0CEA46B0071 for ; Mon, 6 Feb 2023 23:01:50 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CE06280B1C for ; Tue, 7 Feb 2023 04:01:49 +0000 (UTC) X-FDA: 80439147138.16.8D7E171 Received: from mail-vk1-f180.google.com (mail-vk1-f180.google.com [209.85.221.180]) by imf28.hostedemail.com (Postfix) with ESMTP id 0AF78C0005 for ; Tue, 7 Feb 2023 04:01:47 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PLGORcIX; spf=pass (imf28.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.221.180 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675742508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vnJphn8h/teaY51bYztcFGncAa/2yVuimAdfonZwh2I=; b=e+VqKu10AQ1Ovdk9XthPDgcWdPTCPPS5rgMIfN9/4CQ0WTr9pe+hbCf7ABHp3e6MgBXzcy 2tBK11bn/1h2HWtZXayF0Tb3lDM81gZ1w44YAUquwi4QOk538kNd67+UL6FuHKDH6bBPkI EfMDJ85rBwo5l3Bl9fRUlEDwlO23Bio= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PLGORcIX; spf=pass (imf28.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.221.180 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675742508; a=rsa-sha256; cv=none; b=dPAq0Sh4d02dRE15LJ6JH4AesmVDb9iuwAJ0LPoyM+9Ql9Apx9T+WSBh0tJgO9+7seMI3l IydyVOlbVLm5A+4AD4gjpGiJ6llR4iWvkEts0kEwaZO5d3wcdRlAI+hPjPnyBvxO6+vPJF 0OFjRsnzbX3OvWgF4GRJ+Fon9jlOBlA= Received: by mail-vk1-f180.google.com with SMTP id v5so7220584vkc.10 for ; Mon, 06 Feb 2023 20:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vnJphn8h/teaY51bYztcFGncAa/2yVuimAdfonZwh2I=; b=PLGORcIX7Z6FO2uh+mQhLmU6Q65FKoCoeEVpBOahKass5lqk/ZUcoT3F2MndhZwPvD 0mcdoaI0U+baKWJ4JeOTlEZMvmaGel6EF3KYGvL0ma0HMwuPbMS/eNIBMp+3cjBI0CAV NPA6xJUzYQHhqdAsh/yDBz3rnThh7PF6IO6hLIZ3nUcpqLHo1AWOMnsnXd1o1hcMqQcd pgEDrwE4PnU5B1H8IA44sCgFJTPz/tBTbDkXemjFx+Vaa57KvKgs/iHYnRn357d7G2JT LQ3Urkg90KMHhySnNKDmJ+igImMdxMsMTgz5oKSQDPZl1mNYYAqTz+y9YQ4gDmZ0jN2o Be1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vnJphn8h/teaY51bYztcFGncAa/2yVuimAdfonZwh2I=; b=xH556CQIZpfcgMFug6vydYn96kIjn1YON/PztNPSi/9F9MQO1X1bLDX16nASjr1Ice FX8+/nXoUQdPIoZfOYLlYVEeNRnumv2TQX9mBtbRDMO2z3LmB9bjEEUYktS5InDwpI8B EX4FoK3Yf1f8428udU/jgWQ8+wYWuIu69fuI19cbSjii7F47xtKG22VQcb29hi5KjARa nXlTunv+PyEy/GAzefnFn/P9IzdHB6mG+oicPql90dZ8JGKxkWBmnmK3PsUUshwCuyFD 5ucMAaJe6Eynv7sATm4rnujYXaE4+hiCah3Sq+biTJS9FOnz1LdB0UscYdNMKP4S/APq cgyw== X-Gm-Message-State: AO0yUKXr+NijSo/2ugVd0i+JNJcBZFBySDbBELVjicOwl/6SOlCotk/6 q17jdzNgZdE0q75cSy7WsPBFxz/bQwQ5S5yi X-Google-Smtp-Source: AK7set+8EGcTNLLEym88LMDaQznwmnIZe5fv2xZTmJeysKbk4P1tuC9B1rImnpJu2fsuK/TfhwXLRA== X-Received: by 2002:a17:902:c40f:b0:196:e8e:cd28 with SMTP id k15-20020a170902c40f00b001960e8ecd28mr1564019plk.15.1675742016151; Mon, 06 Feb 2023 19:53:36 -0800 (PST) Received: from strix-laptop.hitronhub.home ([123.110.9.95]) by smtp.googlemail.com with ESMTPSA id q4-20020a170902b10400b0019682e27995sm7647655plr.223.2023.02.06.19.53.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 19:53:35 -0800 (PST) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , "Matthew Wilcox (Oracle)" , Christophe Leroy , John Hubbard , Nadav Amit , Barry Song Cc: Steven Rostedt , Masami Hiramatsu , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Yang Shi , Peter Xu , Vlastimil Babka , "Zach O'Keefe" , Yun Zhou , Hugh Dickins , Suren Baghdasaryan , Pasha Tatashin , Yu Zhao , Juergen Gross , Tong Tiangen , Liu Shixin , Anshuman Khandual , Li kunyu , Minchan Kim , Miaohe Lin , Gautam Menghani , Catalin Marinas , Mark Brown , Will Deacon , Vincenzo Frascino , Thomas Gleixner , "Eric W. Biederman" , Andy Lutomirski , Sebastian Andrzej Siewior , "Liam R. Howlett" , Fenghua Yu , Andrei Vagin , Barret Rhoden , Michal Hocko , "Jason A. Donenfeld" , Alexey Gladkov , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng , Chih-En Lin Subject: [PATCH v4 05/14] mm/khugepaged: Break COW PTE before scanning pte Date: Tue, 7 Feb 2023 11:51:30 +0800 Message-Id: <20230207035139.272707-6-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230207035139.272707-1-shiyn.lin@gmail.com> References: <20230207035139.272707-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0AF78C0005 X-Stat-Signature: rsjuz7omywofyowx8kqtuffczimdq5jb X-HE-Tag: 1675742507-858737 X-HE-Meta: U2FsdGVkX1/hqTQ41nEUg1oQ3Z7oIBI/dHV6OztwLF4NuJ8mRz4PMeFzF7ZIRJYjvG7USCgdq6XdEr3tT3zG6DnSEsrZd5557fGQVOfp2Rf0qP2anCajMKqRaX9+D9ew7Inaf/9rz34zwiCQ+Rf9/J4jGvsI5Vj/5Dd3u9z5uumrpQmA046rJDdoWospUVm5MboA+kXQ4nRLA0Q7t2xupcnzaTHRLjY8pErviVKUQCDSEvPqXHxB1df60oKpp/6YRSHmL4sA71oMh3jHFRYHgMEXXyIG7cZRRjoRJ3OE6QUE5U5cR8eGIa0om9+cLuE2HgZCIpEOgXycei9btpRqxoPiyV2MPQLL5jDhIH3XKx7dJPV+V3WVKwEQF+etx0VK8XSGy3WXlJlPn7Z/RJ9CeT/nXFaKFiogBCHOSQzrQdWdYXA9Rvbty3qmcqwsgESBP03lX5LuoAGjiMitHIbrt0yYe2XoGjIbiSj+1V3PnPyxJnG1/ZijJieVSoXjij8hE7m88x2ehI4ukyYroSR9H6NCnBs87dAG/wCN5YaLxuI1YMt2wZzu04eNiFoUswpaoNhCrjRVbCX7ZfL23T36knCRcNqzpqGPVKfnAdgC6EU58XsVl8sbpUe5Oa0BA4wuLi+Y1aZYuP0xkYDRKarUnPSdf+B2frBIv+7pXIrXKOGguFzi/hLCSKP0mTTCmHF4Q85Vt9XXgvo53etAJ1SZNQP4I5mwGOShwT0gECIe3WyUTc7D2Jt+Jjqyxpwo5s7Yxdz6sCJP3pirQLwUbljn0ejkBYvb0Y1E5Ep+h88f4h2Y/aVINo1x1ibmXucFvpE7oCccqvBwaWo/jV6SJxcsljhDzxhiUw7kQpwasyzSPJfMcvsr38vJ2J48+b9NmbbqB4YW+7yW5mq6+4aOkZFh8ZS/hL3fsyO8VZAv+1U8gwt1G63PHdpOWo6F21hIHmv1kUEOd4mGCqrZWzqfcD2 mP/6WeL1 RjX4aGutgYj4cO9r/2/JlOgS9HuFwSpT9yuwaYsGZcwNR1VcGF4QC8svsX2GMR0aU3wTK0Z50Mwir9tb+xgql6OTImOr5p/My7H2weDZx9I84Jc4KyP1w2vftpqZrqzp9oCCIhOojiqU7vlCpknISPj7Q6ngDEhnS0YDEos2/vztrX0WjWC3C0wS6XEfEEYXvqDqnz5tNa/nrOZLtnvFtL5LsibguJ4nMLX3+3Ef/I8OxyuATE+djB3rplMHCudJrUEPImbTTQaVuqduTVk5AoeaK+atEXsKZjY+pXlj8FhrQUXORWAfzh0awmDPUIrBEBeNk40vU9mJ3mSY+fd3EIanyogil2vYY+ifxU/gTGqULvGDks1+Zjx8OHkGWsPTIT2+wmbU+D/Hs9IY6MGJ7uLA31JfiYhpedKwNmKhylUgRwxCqFZv4Mlpd1bg1p4TAesSgwI20ECjCD12op+cPAZ/cAKYeFJISD/uCnPDEk7qlmfRmWFjgJ9/q1Qpc9UYJHrTAgqEX3vCKkETJpob08sT3o6jXnOTIUlPdmP1suW9iVPR57YPawVUnuA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We should not allow THP to collapse COW-ed PTE. So, break COW PTE before collapse_pte_mapped_thp() collapse to THP. Also, break COW PTE before khugepaged_scan_pmd() scan PTE. Signed-off-by: Chih-En Lin --- include/trace/events/huge_memory.h | 1 + mm/khugepaged.c | 35 +++++++++++++++++++++++++++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 3e6fb05852f9..5f2c39f61521 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -13,6 +13,7 @@ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_PMD_NONE, "pmd_none") \ EM( SCAN_PMD_MAPPED, "page_pmd_mapped") \ + EM( SCAN_COW_PTE, "cowed_pte") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 90acfea40c13..1cddc20318d5 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -31,6 +31,7 @@ enum scan_result { SCAN_PMD_NULL, SCAN_PMD_NONE, SCAN_PMD_MAPPED, + SCAN_COW_PTE, SCAN_EXCEED_NONE_PTE, SCAN_EXCEED_SWAP_PTE, SCAN_EXCEED_SHARED_PTE, @@ -875,7 +876,7 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, return SCAN_PMD_MAPPED; if (pmd_devmap(pmde)) return SCAN_PMD_NULL; - if (pmd_bad(pmde)) + if (pmd_write(pmde) && pmd_bad(pmde)) return SCAN_PMD_NULL; return SCAN_SUCCEED; } @@ -926,6 +927,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, pte_unmap(vmf.pte); continue; } + if (break_cow_pte(vma, pmd, address)) + return SCAN_COW_PTE; ret = do_swap_page(&vmf); /* @@ -1038,6 +1041,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + /* We should already handled COW-ed PTE. */ + VM_WARN_ON(test_bit(MMF_COW_PTE, &mm->flags) && !pmd_write(*pmd)); + anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, @@ -1148,6 +1154,13 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + + /* Break COW PTE before we collapse the pages. */ + if (break_cow_pte(vma, pmd, address)) { + result = SCAN_COW_PTE; + goto out; + } + pte = pte_offset_map_lock(mm, pmd, address, &ptl); for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { @@ -1206,6 +1219,10 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } + /* + * If we only trigger the break COW PTE, the page usually + * still in COW mapping, which it still be shared. + */ if (page_mapcount(page) > 1) { ++shared; if (cc->is_khugepaged && @@ -1501,6 +1518,11 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* We shouldn't let COW-ed PTE collapse. */ + if (break_cow_pte(vma, pmd, haddr)) + goto drop_hpage; + VM_WARN_ON(test_bit(MMF_COW_PTE, &mm->flags) && !pmd_write(*pmd)); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1706,6 +1728,11 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + if (test_bit(MMF_COW_PTE, &mm->flags) && + !pmd_write(*pmd)) { + result = SCAN_COW_PTE; + goto unlock_next; + } collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); @@ -2143,6 +2170,11 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, swap = 0; memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + if (break_cow_pte(find_vma(mm, addr), NULL, addr)) { + result = SCAN_COW_PTE; + goto out; + } + rcu_read_lock(); xas_for_each(&xas, page, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, page)) @@ -2213,6 +2245,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } rcu_read_unlock(); +out: if (result == SCAN_SUCCEED) { if (cc->is_khugepaged && present < HPAGE_PMD_NR - khugepaged_max_ptes_none) { -- 2.34.1