From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C59EEC54798 for ; Tue, 27 Feb 2024 13:52:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5919F6B0169; Tue, 27 Feb 2024 08:52:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 541066B016C; Tue, 27 Feb 2024 08:52:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 430736B016D; Tue, 27 Feb 2024 08:52:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 34FF96B0169 for ; Tue, 27 Feb 2024 08:52:44 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 089AAA07E0 for ; Tue, 27 Feb 2024 13:52:44 +0000 (UTC) X-FDA: 81837724248.03.91073E3 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) by imf24.hostedemail.com (Postfix) with ESMTP id 18F64180007 for ; Tue, 27 Feb 2024 13:52:41 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="Kg/SiZf2"; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf24.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709041962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KTx0R6BPzRbPHHXux5hHxOtyW6q21tS4nAQfi/BVpWw=; b=EtNKWA5/G0RKfaNhTK7KiYiZJEF8Ya0NGJt+w9mBR/4dLAfHeM7YTkM50SAr7ot6hoIaA7 1hmelkRin6Il+LPM9KVGTlhNdjKsPificJClx0mJZEZW0kc66yibrY5EafRdD3wf58+aI3 NJmbjLUTSEa5FdJsIMqWLsS765+s5k0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="Kg/SiZf2"; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf24.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709041962; a=rsa-sha256; cv=none; b=casA7hjUdX2VUqSLVQyRIT2zHkJI7OJwWyd7Ds4FpbleB9CZGj4bVg5lH21X6hOOGAfFmC NO78kEoUaFHW9K126ep5VpYh26xvSZEhxbBgjP4JXRL6QgGnWiZSG4ZahkTDoEB8eW2XQg srO+mKoIuTwvti7dflxOZlVAuRZ/Mj0= DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1709041957; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=KTx0R6BPzRbPHHXux5hHxOtyW6q21tS4nAQfi/BVpWw=; b=Kg/SiZf23lS4yDTRuJvpuC9JVjUivT/Yqw5FhawV7FrnW/YhQupRgEOSUa/eKGRnTHK1DroTRfWABW5Pb2KnKJN5lRnF+QEWJ8der1hIcOUqRGIKqghsOdRjEpx8oZUJbwboswoZ2fukomAFWheCUdJ1HemEwn8TXnBdMGR7ceI= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R711e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0W1MvL.U_1709041956; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W1MvL.U_1709041956) by smtp.aliyun-inc.com; Tue, 27 Feb 2024 21:52:36 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: muchun.song@linux.dev, osalvador@suse.de, david@redhat.com, linmiaohe@huawei.com, naoya.horiguchi@nec.com, mhocko@kernel.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] mm: hugetlb: make the hugetlb migration strategy consistent Date: Tue, 27 Feb 2024 21:52:26 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: wcqmj5jz3si7oy4becw8ky95nyi4eo5z X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 18F64180007 X-HE-Tag: 1709041961-952455 X-HE-Meta: U2FsdGVkX18hRw/82WJ3XEJj3GEKVpsssM6OOBaai82dwt874tVnHhyNBPul+sStKZiE0inmtIGfnDOTH2RpRZuUaGPkEcLV86GjhU3LN0cfz2OhsZJcQHgo0SNLKQyhgVdXTyCmwEX+zi4+6UcYLBihKlZ0TwmXmWED+YYiBam+yW3HfJvimuZlvEC6+hulvhS0FtuUQltiQnvnVpGy6IirxAs+QbNK5931rF4/5KV1AUdqI7ekcpsTcT2YJmmjeSmsSvKLu3cQj+qeTAA6CnKkeps761U9STxe3BTj18LY4+WjR2c7PqX6nbLn/QtxcN5K3K0XzGgIuFPk6QZoOqE8py9eogQ2fudtdkNrrLZdzBeQtG+/+F+TDQXK0EWXmuxYK4brOdzBfd7NS3caK9DzM6DBKDBXluKcIySJSEkSLweHUTcoPImgDBfI3Q78OJthrGc3lm+cBuLkJegDRU6b9PdquRDNmRcWStTHH7CRvZNhkv+fSF0B+VBy5jjYML3tUFf1wJGLS+sHECMu4cgcjOcf+zrXA6rpAKzLEl9RVHrRS8BQg0nWuODANxoaQdyjq2jIT8rNaw4RNjgdOY8SRB1MwLfHUgjmiCuDWLPCCsVwmlbtMZ1LZG91QuajOoFrdl9xXscCht/Er8/JZl0qQE+izy+YUmu120Jft96qBiO71S5RtTIinvGixS84Mns8SgS12G1aC/WHaY19YJuE7kPVI0dcl6ZzxN24SLz2K80/3U3GtX9eIkgZiKmfKKaE0h2auGXISkm1amhG/oMjTwlNIGcgFJDMdoIXnHEuiAirForB21TZhCj1NAriLda4HecHcAEEDop9bmnIZCBcLjUMsMsR8CNiDviviKXVv4E4XDSdNEdkPI+FGqYDDa/K6GLQ3YgnMQp5GzvCKZ9c4wHHzKsf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As discussed in previous thread [1], there is an inconsistency when handing hugetlb migration. When handling the migration of freed hugetlb, it prevents fallback to other NUMA nodes in alloc_and_dissolve_hugetlb_folio(). However, when dealing with in-use hugetlb, it allows fallback to other NUMA nodes in alloc_hugetlb_folio_nodemask(), which can break the per-node hugetlb pool and might result in unexpected failures when node bound workloads doesn't get what is asssumed available. To make hugetlb migration strategy more clear, we should list all the scenarios of hugetlb migration and analyze whether allocation fallback is permitted: 1) Memory offline: will call dissolve_free_huge_pages() to free the freed hugetlb, and call do_migrate_range() to migrate the in-use hugetlb. Both can break the per-node hugetlb pool, but as this is an explicit offlining operation, no better choice. So should allow the hugetlb allocation fallback. 2) Memory failure: same as memory offline. Should allow fallback to a different node might be the only option to handle it, otherwise the impact of poisoned memory can be amplified. 3) Longterm pinning: will call migrate_longterm_unpinnable_pages() to migrate in-use and not-longterm-pinnable hugetlb, which can break the per-node pool. But we should fail to longterm pinning if can not allocate on current node to avoid breaking the per-node pool. 4) Syscalls (mbind, migrate_pages, move_pages): these are explicit users operation to move pages to other nodes, so fallback to other nodes should not be prohibited. 5) alloc_contig_range: used by CMA allocation and virtio-mem fake-offline to allocate given range of pages. Now the freed hugetlb migration is not allowed to fallback, to keep consistency, the in-use hugetlb migration should be also not allowed to fallback. 6) alloc_contig_pages: used by kfence, pgtable_debug etc. The strategy should be consistent with that of alloc_contig_range(). Based on the analysis of the various scenarios above, determine whether fallback is permitted according to the migration reason in alloc_migrate_hugetlb_folio(). [1] https://lore.kernel.org/all/6f26ce22d2fcd523418a085f2c588fe0776d46e7.1706794035.git.baolin.wang@linux.alibaba.com/ Signed-off-by: Baolin Wang --- include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++---- mm/mempolicy.c | 2 +- mm/migrate.c | 2 +- 4 files changed, 39 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 77b30a8c6076..fa122dc509cf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -747,7 +747,7 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, - nodemask_t *nmask, gfp_t gfp_mask); + nodemask_t *nmask, gfp_t gfp_mask, int reason); int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping, pgoff_t idx); void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, @@ -1065,7 +1065,7 @@ static inline struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, static inline struct folio * alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, - nodemask_t *nmask, gfp_t gfp_mask) + nodemask_t *nmask, gfp_t gfp_mask, int reason) { return NULL; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 418d66953224..e8eb08bbc688 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2567,13 +2567,38 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, } static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, - int nid, nodemask_t *nmask) + int nid, nodemask_t *nmask, int reason) { struct folio *folio; + bool allowed_fallback = false; if (hstate_is_gigantic(h)) return NULL; + if (gfp_mask & __GFP_THISNODE) + goto alloc_new; + + /* + * Note: the memory offline, memory failure and migration syscalls will + * be allowed to fallback to other nodes due to lack of a better chioce, + * that might break the per-node hugetlb pool. While other cases will + * set the __GFP_THISNODE to avoid breaking the per-node hugetlb pool. + */ + switch (reason) { + case MR_MEMORY_HOTPLUG: + case MR_MEMORY_FAILURE: + case MR_SYSCALL: + case MR_MEMPOLICY_MBIND: + allowed_fallback = true; + break; + default: + break; + } + + if (!allowed_fallback) + gfp_mask |= __GFP_THISNODE; + +alloc_new: folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); if (!folio) return NULL; @@ -2621,7 +2646,7 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h, /* folio migration callback function */ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, - nodemask_t *nmask, gfp_t gfp_mask) + nodemask_t *nmask, gfp_t gfp_mask, int reason) { spin_lock_irq(&hugetlb_lock); if (available_huge_pages(h)) { @@ -2636,7 +2661,7 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, } spin_unlock_irq(&hugetlb_lock); - return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); + return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask, reason); } /* @@ -6653,7 +6678,13 @@ static struct folio *alloc_hugetlb_folio_vma(struct hstate *h, gfp_mask = htlb_alloc_mask(h); node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); - folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask); + /* + * This is used to allocate a temporary hugetlb to hold the copied + * content, which will then be copied again to the final hugetlb + * consuming a reservation. Set the migrate reason to -1 to indicate + * that breaking the per-node hugetlb pool is not allowed in this case. + */ + folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask, -1); mpol_cond_put(mpol); return folio; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 98ceb12e0e17..436e817eeaeb 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1228,7 +1228,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, h = folio_hstate(src); gfp = htlb_alloc_mask(h); nodemask = policy_nodemask(gfp, pol, ilx, &nid); - return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp); + return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp, MR_MEMPOLICY_MBIND); } if (folio_test_large(src)) diff --git a/mm/migrate.c b/mm/migrate.c index bde63010a3cf..0c2b70800da3 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2022,7 +2022,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private) gfp_mask = htlb_modify_alloc_mask(h, gfp_mask); return alloc_hugetlb_folio_nodemask(h, nid, - mtc->nmask, gfp_mask); + mtc->nmask, gfp_mask, mtc->reason); } if (folio_test_large(src)) { -- 2.39.3