From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA611C27C55 for ; Mon, 10 Jun 2024 12:06:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E7EC6B009A; Mon, 10 Jun 2024 08:06:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6983D6B009B; Mon, 10 Jun 2024 08:06:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539206B009C; Mon, 10 Jun 2024 08:06:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 359696B009A for ; Mon, 10 Jun 2024 08:06:35 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E99A7A0F21 for ; Mon, 10 Jun 2024 12:06:34 +0000 (UTC) X-FDA: 82214851908.09.FA7AA6B Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf26.hostedemail.com (Postfix) with ESMTP id 1E729140010 for ; Mon, 10 Jun 2024 12:06:32 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bIOyAINm; spf=pass (imf26.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718021193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YOOpP3mhn3ktm+AdbLZ/EN2VmROk6QDShkqGnjdgrDU=; b=8KqC0xVrZIsJzGD9FV6l9tS4PCSH+E9ZmgSr4/ndIrHHaW5JrdrjeU5Bzjaw5gB0rNAMzP Su8wY0dRlSIPSZgMxP8QqmrtWu7llQjlw0omfenlchtgoKzvXWQbbNYf/nz5cNYem5BiVV ULwYGFjTD+Mb/qgbM9i2AXA70bUmo5Q= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bIOyAINm; spf=pass (imf26.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718021193; a=rsa-sha256; cv=none; b=M4XzowstxvD55l4SU1WtgIfxcNYWlxetXogbvdN9XbSlcokMWmmT7eCR3GzvE/8MhjyoGg J8sjYDj8iv/3rhlWKTWP5XTmiLm5b9I0KWJwoYdfMrJGUNwNlJjuXxWsL+zIDXJ2MJdMUP 8pnPGZzkBzB8IyF8Z9obuT80DMgdDyg= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-70417a6c328so2080135b3a.1 for ; Mon, 10 Jun 2024 05:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718021192; x=1718625992; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YOOpP3mhn3ktm+AdbLZ/EN2VmROk6QDShkqGnjdgrDU=; b=bIOyAINmlIWF1nAc2T8vuI13etDEY0vcr9liOKAbRWDtmNdQ+Z6H3/DZF5++s3TaKs Ph6Sabru4nypa2KC4IktK+M05w8YUNBKl82E6hq2CNZQOnnCem327xNg6Ck+u35DcJKY 7lI9vu9qm84/l0ZbU2Ma08Cw7TTznRMcqByW14lqOG91/tWAUudvjHkaKPUoM6m9FsFt 7CbhBuiftFs+UEb4cEpjua3Z2eObfpdE3e8KV9gF5hzPXwgAL00JmkMTtOBbeq+vUw9O pZBVrYWWExSWWA9f2MUPaKqATKJtJE3GOBw0yT0MGJPlUZxTsr2jpNqtubDT5HYzsJrb 8ltg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718021192; x=1718625992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YOOpP3mhn3ktm+AdbLZ/EN2VmROk6QDShkqGnjdgrDU=; b=Mc0ADIQT6ur+G0zhZVOLP6fjexse63JAJVT29pOUTRQXWSVoC1jTCp9PJSuybwbOxQ BI8lni8PqycFxOX8VFpUm2PtzrKTs5Pa+BqPNC1PZjKBJ0DG0gKFlVtfVLZrHNEBGVmP Thk8B3p6MVzTEV26MZLvu7cTsvsnQ20E8RWMn06Y7Ua82/p1dM5t2M2w6JgSftRJ9qh1 0Mwf8zuzcUj9aP48C9XyXGHmz1h32ZpfZQe1gSEpMgi3E4iOIZBES9RnpibdpM9+1mzV auSKMwKnPYpHjiN210Vkv/TXODOuKyx2cjKaY0j+5Dpem0rAcp8/yhagRh2vN3k1ww7Y PaCw== X-Forwarded-Encrypted: i=1; AJvYcCWuLsqT0oFNW3qVZTS9NnDMROOxKeVzCv/SSocuL3EmOkCErMpg8A+6sdBshiBcWf92lQF039kw6ORWdvekjNAqNZo= X-Gm-Message-State: AOJu0YyyhoNmJjPgYSxgbiZJcRurDdMUqwjwAC4lMZxhMudI/5zOi3d5 y/SdVlorBtjVnGUEeLF4H/+bn36GLd5ERFnf0IEpOEv8W+OhSdyc X-Google-Smtp-Source: AGHT+IHuuLA3IkOVWnemUAiXi6DwNGqf6nD+qxo5av/4Kb4vZP8/N6Adb31SMqw2zaGmXf2b8MhZIQ== X-Received: by 2002:a05:6a20:8423:b0:1af:7646:fc14 with SMTP id adf61e73a8af0-1b2f151f841mr9176853637.0.1718021191815; Mon, 10 Jun 2024 05:06:31 -0700 (PDT) Received: from LancedeMBP.lan.lan ([2403:2c80:6::304c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7043aa52899sm2182250b3a.25.2024.06.10.05.06.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jun 2024 05:06:31 -0700 (PDT) From: Lance Yang To: ioworker0@gmail.com Cc: 21cnbao@gmail.com, akpm@linux-foundation.org, baolin.wang@linux.alibaba.com, david@redhat.com, fengwei.yin@intel.com, libang.li@antgroup.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, maskray@google.com, mhocko@suse.com, minchan@kernel.org, peterx@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com, sj@kernel.org, songmuchun@bytedance.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiehuan09@gmail.com, ziy@nvidia.com, zokeefe@google.com Subject: [PATCH v7 3/4] mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop Date: Mon, 10 Jun 2024 20:06:18 +0800 Message-Id: <20240610120618.66520-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240610120209.66311-1-ioworker0@gmail.com> References: <20240610120209.66311-1-ioworker0@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: t3g8g5pebpjpm9xj14t97mti7ojruxn8 X-Rspamd-Queue-Id: 1E729140010 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1718021192-357395 X-HE-Meta: U2FsdGVkX1+Y9FGSilJeTdTW31rKEMLczH4GJs/eT8QrLFVi9k54i7907GY6a0J3CeGl4YemT2mAENM+sBlccdzh6+b5Om/aCeqX5AGhCnz8qzqYWGn/vcWZJs2uJ9UJaXQEsVvXB/3q5tCUNVxIIDgrLdbNwHr9i6Z/sKLE+gKvzLQC8IttU97El3AQ8PNhkbUiOT0hk3RL9ZqQJA3JvP2JaWE9rNahVRLgR3wDpInHrMVfKRAMVlEBeuGbbtwaGIyabEe87rYRCoRUT+I5b5acYkbQggYuYRTkxFH/SnNfbUqKlBP4Bq68em0sw8JYJNGrRkC9vS5WlmPs6JIHEpmvo5vVGQsuypxcRkoacTbw5PIojRuAij7BuGuwOUVCXXJD27+A9gANBZLagGp/UXtDQD3Tfn6NYYHrSAIM1A0ZJyEOGm297UM/gshMJfwaJtzUs8AQDMaOxZt/LLgLCfq129+dBFAchXqNrPY/id5S7I+tiBmoKBSidUj4RVnbYLabXIIzQuCPFVWLniwuMm4FrjyGh83rQkqfrQ+yzsq7h2fvcj/626PC4N2hPtUcJs4G2HezndC8jkscZk6h/sNUwC0WN94D5w4N2zXrNSztWBdIk39uGJsenFHXNx0RaHMfmTW3iSFw3ED0NPvwsQvG4PBu9R5AgGdy/1JORm3OG1QHw8wh+omLoizPPkDOPsU2oTLX3qc6dOPcMCtLA6YqoItq1Q/MW8cX1s60uFqAYR39eo7ZzpfNFtof2ImwRXwVve+drzIl0herNJNGH8dH8wzvSRMYUgQkMwzfkpdGDy1tKJd3RnUPEs/qg+RvV++REDVDs2djhQQTVdaEPi7CFf0KnB6iv3mPyOQK+XG753iTuLp+GpXGUk/yqYDLmlivHi3Gs0L2uMy+VpSd3nqB7GYElAjnC3vJDuRvKoHqXNSBPmyEzNUHMXpRheS7b0IP/s5vf03U6iXKsa/ JZMECETr WuGHqBn6TXCkzXKs7Mogx1rFtOA+JAQGVGGOrk4R5IzdVMoIshCYO+WXpx6u6w4JIfNk97IoGnUI9k6u6+rr00L/VrSXj2GtUUfT90JPO7YaIjwANaNZpFGZRfZQP8vKaUIdVL8LubFQiC47vFT447Wftc9/q9HufoUo5kIQe2jeqchhaOTrdoJ1ek9ZfhL8O3qqqhJmYuGGSH7Wz8o9JkHDlQ0FWbUGLaw071zEOay+o5udTzvFUY6gX94QrKRz1x/5Dm+GHKLnbymxGlJZqYuGBfKAVZdpGSvg8nUrqCkGx6WrEHw3P2zzC6eaCHUP8RGvjvaSj9A2UlJse1oGnGdaWCtU84zqBu0VtN9hB7IbeIjcUPscjiX/5JzQmyeRCZhYPBQsDu30wSc71/o60EFA9LmeOXEjCX5O06obgqpYJHc2KYrhinZS+g1EJ+btJbykrU+5M/RkIRCTbMZereEuWKuLy1itQU6zwtF7uwDjdPMJZYPqH6F06cg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for supporting try_to_unmap_one() to unmap PMD-mapped folios, start the pagewalk first, then call split_huge_pmd_address() to split the folio. Suggested-by: David Hildenbrand Suggested-by: Baolin Wang Signed-off-by: Lance Yang --- include/linux/huge_mm.h | 6 ++++++ mm/huge_memory.c | 42 +++++++++++++++++++++-------------------- mm/rmap.c | 21 +++++++++++++++------ 3 files changed, 43 insertions(+), 26 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 088d66a54643..4670c6ee118b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -415,6 +415,9 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, bool freeze, struct folio *folio); + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline bool folio_test_pmd_mappable(struct folio *folio) @@ -477,6 +480,9 @@ static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio) {} static inline void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, bool freeze, struct folio *folio) {} +static inline void split_huge_pmd_locked(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmd, + bool freeze, struct folio *folio) {} #define split_huge_pud(__vma, __pmd, __address) \ do { } while (0) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e6d26c2eb670..d2697cc8f9d4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2581,6 +2581,27 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pmd_populate(mm, pmd, pgtable); } +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmd, bool freeze, struct folio *folio) +{ + VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio)); + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); + VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); + VM_BUG_ON(freeze && !folio); + + /* + * When the caller requests to set up a migration entry, we + * require a folio to check the PMD against. Otherwise, there + * is a risk of replacing the wrong folio. + */ + if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || + is_pmd_migration_entry(*pmd)) { + if (folio && folio != pmd_folio(*pmd)) + return; + __split_huge_pmd_locked(vma, pmd, address, freeze); + } +} + void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio) { @@ -2592,26 +2613,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); ptl = pmd_lock(vma->vm_mm, pmd); - - /* - * If caller asks to setup a migration entry, we need a folio to check - * pmd against. Otherwise we can end up replacing wrong folio. - */ - VM_BUG_ON(freeze && !folio); - VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); - - if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || - is_pmd_migration_entry(*pmd)) { - /* - * It's safe to call pmd_page when folio is set because it's - * guaranteed that pmd is present. - */ - if (folio && folio != pmd_folio(*pmd)) - goto out; - __split_huge_pmd_locked(vma, pmd, range.start, freeze); - } - -out: + split_huge_pmd_locked(vma, range.start, pmd, freeze, folio); spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } diff --git a/mm/rmap.c b/mm/rmap.c index ddffa30c79fb..b77f88695588 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1640,9 +1640,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (flags & TTU_SYNC) pvmw.flags = PVMW_SYNC; - if (flags & TTU_SPLIT_HUGE_PMD) - split_huge_pmd_address(vma, address, false, folio); - /* * For THP, we have to assume the worse case ie pmd for invalidation. * For hugetlb, it could be much worse if we need to do pud @@ -1668,9 +1665,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); while (page_vma_mapped_walk(&pvmw)) { - /* Unexpected PMD-mapped THP? */ - VM_BUG_ON_FOLIO(!pvmw.pte, folio); - /* * If the folio is in an mlock()d vma, we must not swap it out. */ @@ -1682,6 +1676,21 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; } + if (!pvmw.pte && (flags & TTU_SPLIT_HUGE_PMD)) { + /* + * We temporarily have to drop the PTL and start once + * again from that now-PTE-mapped page table. + */ + split_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, + false, folio); + flags &= ~TTU_SPLIT_HUGE_PMD; + page_vma_mapped_walk_restart(&pvmw); + continue; + } + + /* Unexpected PMD-mapped THP? */ + VM_BUG_ON_FOLIO(!pvmw.pte, folio); + pfn = pte_pfn(ptep_get(pvmw.pte)); subpage = folio_page(folio, pfn - folio_pfn(folio)); address = pvmw.address; -- 2.33.1