From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 968FBE95A9B for ; Mon, 9 Oct 2023 15:19:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 301498D0083; Mon, 9 Oct 2023 11:19:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 28A238D0031; Mon, 9 Oct 2023 11:19:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 104CD900002; Mon, 9 Oct 2023 11:19:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D823F8D0083 for ; Mon, 9 Oct 2023 11:19:15 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9A7411CA8A6 for ; Mon, 9 Oct 2023 15:19:15 +0000 (UTC) X-FDA: 81326281470.08.AFF2623 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf12.hostedemail.com (Postfix) with ESMTP id ACCE840019 for ; Mon, 9 Oct 2023 15:19:13 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRhM3dgH; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696864753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xEViuWF/QhMtAn6zDhENFWi+HKxgrVT/QGI1gB+0GRQ=; b=pxCxWFaRy1Rlda14mbtovW6LObLQeRWSNKJbpG+pT1I4hwD1ZTfIvdXlLEUD7fF8Gp6gHi fqeXAEfZT15PZK7F093dLWd9ov6OIi1C9vtM+C90lhJnFu4c5TPaCOBCO1gvkYGc7s2XbT YHIFur6xJpACaUJtxjcoXz0gCOLaj2E= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRhM3dgH; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696864753; a=rsa-sha256; cv=none; b=WVIbprZUBoHWx3+ZWpbe4bzWb2JRDwqkxmtELthYhSp1hbl6yfo1xnEpF0NOGdi97BkZ/7 TUoWYKTgqiGYaPXonP/zrIqXNrv9dzIfQ1dDElQf8FFjfenuZyj0cW1YrUyGJssdCjTGRd FVRx91NK9tEnJXo2HgIlLvAQxvLU4EM= Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-405505b07dfso33729275e9.0 for ; Mon, 09 Oct 2023 08:19:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1696864752; x=1697469552; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xEViuWF/QhMtAn6zDhENFWi+HKxgrVT/QGI1gB+0GRQ=; b=cRhM3dgH5uqpHvYPdjEVaE8dtIDOt5J7rB7wdRPWdfupbzm3BVf34qhygi+VAk9Hkm wQrrQYLxBnjNNjYslzRSKyEdT9hBu3x96tz9hG3IaM2YS+nCXFVpKRKcEwoe9zNHguqj xm9J01uX6xUoaOJqWjIb+OOgm4n+vNPK6wKHlEhjdfiGyF0TjGPBpf9dHI2nBxUMKZ1h xuUE2r2HN4m/9f/iy85KmCfs0tByIq7okaO2wYODno8NLP2Om0hOu9bPcjzAxLWFSAOm tAozIrrFVmL+YJS2pjOpqIp/v2ILey2xZSA03ds4Coy3s8XqA+OkptPk2uptiZxA0syP NBLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696864752; x=1697469552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xEViuWF/QhMtAn6zDhENFWi+HKxgrVT/QGI1gB+0GRQ=; b=d/2tjYqX4jNP+/lhbS7MBtydjZ67PpeYID0qtuMAuKsoodX3UK6r8L4IIU6hDGl87y TpSu/DQSBjjWY7xOMT3j7Lz5+DbkNm2ApHEXADK1KQL0V9aAw1ETFT4fZOOFDXLis+OL Ml1De1BMUBa3Y1DEVV6TbUHl1CDbkq2Mx3PSRZWIiMFgmiQL6iTKtxkd1KHm1YYmGzq+ 2QDNweYkmVRmNe08Kuc+ACudEerAG+J6IVHttF7q9hKLGHWtAlYagAGP6yNkaGat9jxW u/K9O+MD9s9eBr5ooQ0nmaOf8t7O5Cvy5AmL+1h2DzZVVSZV7S53H38SuvR9rQbkAnGl hScg== X-Gm-Message-State: AOJu0YyFcbzjOKi41090qb08az7MINh46xF8nS4HdgVnq8AZ2yUmvRIF VgC1uZIY4T9sNmqEmzlXhMSaS5/vhwjpyBkOWCo= X-Google-Smtp-Source: AGHT+IHI7PFMZNOa3RxMFsQks2ktjm8AOtNoVtfhv8G2DShsq9TsPpUNhU+TN3KPpAar2zRhbyCKjw== X-Received: by 2002:a05:600c:5010:b0:407:4126:f71c with SMTP id n16-20020a05600c501000b004074126f71cmr5828847wmr.6.1696864751989; Mon, 09 Oct 2023 08:19:11 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:c1b1:1479:6449:a1ff]) by smtp.gmail.com with ESMTPSA id r2-20020a05600c320200b0040641ce36a8sm7075008wmp.1.2023.10.09.08.19.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 08:19:11 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, mike.kravetz@oracle.com, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [PATCH 1/1] hugetlb_vmemmap: use folio argument for hugetlb_vmemmap_* functions Date: Mon, 9 Oct 2023 16:18:30 +0100 Message-Id: <20231009151830.2248885-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231009151830.2248885-1-usama.arif@bytedance.com> References: <20231009151830.2248885-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: ACCE840019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 7p3tzu4ob14yad98iedqx7gb5ftzhp64 X-HE-Tag: 1696864753-789477 X-HE-Meta: U2FsdGVkX18VRSgb54u3Q4mnDFSOrNsSXKdTBMRyVo5+Xbe3CbgmpxttKH8w7rnzF+YA4SRIVLkAJNcvDP75o4BHVjegsqPMFP5Kqk/oRSS0ZhfsqvtA4sH8INwnc9nWAydi6MO35U6VwRfn9+WUTDYB3tMEJ1VEVxc2hpwDKEr9hWtZFK0fEx5n3ndoQYnYyZYySBXvyhUNV/pRAxTJS7NGvisBnbQilWFvFLZV2V43oYtT4V1sxYkch03uOszNaQHp0knGD7EtGAqFnDEH0YefGKlBrJD0xlYmMz3FxSe695lhpCYhFeOrQPlbgtgKHBPGG38C0QPqhbbLCwO9PzO+GHtojiPlfGuDo0/x5IGwaC8CFtBjiJbGkohlPbE2OaJLfdQI70e3K8tfHyjcwQ4WivHaq1XGb9OgdDc21GEux9poI4o9GB2Cwpm6IfKwp86JjWa3h3i459nLxFjmyzQS1rue6kUL5Ijh2B7x/L5yd1/o0zM+J57fLOsZd2D222ckcRGZLcoTLedPKNlHf98KfCN/rMvshlI1zBwMaXPf811xlbxHoo/3GIX3VX5bkQR7DfJFmZad9++qIPdw+j4Tl28g4TgqkTpnXvnnjLP0ot1IOOeqaAEZZK9xV+Dnec5fW7+HiwToBnRu+fyIPOxzmTR/XK2xdryBZSF1V29ie+p1rw+aZuCT2XP8TDuta61s2AWjjUy6gzDEPqOt8Fa0a9f9UG2gDdIyttsoPgvz4SMqawFZyerkgBYnbxVh0jAYS+dKKV61SCcLSKLSayn/wDNZc+pPwCa6e1Dwe44ososrVOyt7jWM+wgI4s3Nn7Q6PxyAKIx2Vlv1ogD1vQjlcyUfGxURPvAF2uQPw0YD2PSMGoCg3Z31ZbIXPSEtuvoHB14/1Bo3/wcTej+0hv7eYhA8EUbDk5AKxu2nE6K2L+w6j302w3ZiOBtPWMvYAFrTcGRyKTz6avsRJge 4a57HM/x 16EqCDTGoEORMZ/EyhRWdyUaloYQGotUBxBPiUAJTMIsVBtqoIYR5XyStDd2Xe8mOv9ZEd1qprfBxidjsheuL11GC9wbwWOaWBVKTFo7e71mcggA3tOESd1QCH7MfkKcreCFjMXqMRk/YS2NopEJbSvY9EeGXy2nPaQkY2WFUiTQNCndYHlnGuhu14dzKwchcywbTX2UNo8weOkrLcRSBzKFTfYt1hqLqAZ+4J4GjrywoY8MPZJyYb0v9uKFEdx8RpsqJa2pJ86+dGeM2rCYoiQSh64+lju3rr68HLCcfC/CxTh3HQEYXHKtzodAynpdLzgUaJbj6PwUyGWdEByyYMlMDrgCplB3rnpufRMsyKZoCQsrBCWmLHpDwKCVwLDLI6ukPQJ4IptrPfPHUQhswX/8CXhvb1EvOrViyIZ9E518CqwZ2Q3C88vQ4jGmjUcjJQhKj/6a7OATqbFAnC1nHcmHy5j0JP5Y+Ojtf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most function calls in hugetlb.c are made with folio arguments. This brings hugetlb_vmemmap calls inline with them by using folio instead of head struct page. Head struct page is still needed within these functions. The set/clear/test functions for hugepages are also changed to folio versions. Signed-off-by: Usama Arif --- mm/hugetlb.c | 10 +++++----- mm/hugetlb_vmemmap.c | 42 ++++++++++++++++++++++-------------------- mm/hugetlb_vmemmap.h | 8 ++++---- 3 files changed, 31 insertions(+), 29 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b12f5fd295bb..73803d62066a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1606,7 +1606,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, * is no longer identified as a hugetlb page. hugetlb_vmemmap_restore * can only be passed hugetlb pages and will BUG otherwise. */ - if (clear_dtor && hugetlb_vmemmap_restore(h, &folio->page)) { + if (clear_dtor && hugetlb_vmemmap_restore(h, folio)) { spin_lock_irq(&hugetlb_lock); /* * If we cannot allocate vmemmap pages, just refuse to free the @@ -1749,7 +1749,7 @@ static void bulk_vmemmap_restore_error(struct hstate *h, * quit processing the list to retry the bulk operation. */ list_for_each_entry_safe(folio, t_folio, folio_list, lru) - if (hugetlb_vmemmap_restore(h, &folio->page)) { + if (hugetlb_vmemmap_restore(h, folio)) { list_del(&folio->lru); spin_lock_irq(&hugetlb_lock); add_hugetlb_folio(h, folio, true); @@ -1907,7 +1907,7 @@ static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) { init_new_hugetlb_folio(h, folio); - hugetlb_vmemmap_optimize(h, &folio->page); + hugetlb_vmemmap_optimize(h, folio); } static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) @@ -2312,7 +2312,7 @@ int dissolve_free_huge_page(struct page *page) * Attempt to allocate vmemmmap here so that we can take * appropriate action on failure. */ - rc = hugetlb_vmemmap_restore(h, &folio->page); + rc = hugetlb_vmemmap_restore(h, folio); if (!rc) { update_and_free_hugetlb_folio(h, folio, false); } else { @@ -3721,7 +3721,7 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) * passed hugetlb folios and will BUG otherwise. */ if (folio_test_hugetlb(folio)) { - rc = hugetlb_vmemmap_restore(h, &folio->page); + rc = hugetlb_vmemmap_restore(h, folio); if (rc) { /* Allocation of vmemmmap failed, we can not demote folio */ spin_lock_irq(&hugetlb_lock); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d2999c303031..84b5ac93b9e5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -495,14 +495,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct folio *folio, unsigned long flags) { int ret; + struct page *head = &folio->page; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; VM_WARN_ON_ONCE(!PageHuge(head)); - if (!HPageVmemmapOptimized(head)) + if (!folio_test_hugetlb_vmemmap_optimized(folio)) return 0; vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); @@ -518,7 +519,7 @@ static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, */ ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { - ClearHPageVmemmapOptimized(head); + folio_clear_hugetlb_vmemmap_optimized(folio); static_branch_dec(&hugetlb_optimize_vmemmap_key); } @@ -530,14 +531,14 @@ static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, * hugetlb_vmemmap_optimize()) vmemmap pages which * will be reallocated and remapped. * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. + * @folio: the folio whose vmemmap pages will be restored. * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * Return: %0 if @folio's vmemmap pages have been reallocated and remapped, * negative error code otherwise. */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +int hugetlb_vmemmap_restore(const struct hstate *h, struct folio *folio) { - return __hugetlb_vmemmap_restore(h, head, 0); + return __hugetlb_vmemmap_restore(h, folio, 0); } /** @@ -563,7 +564,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_for_each_entry_safe(folio, t_folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - ret = __hugetlb_vmemmap_restore(h, &folio->page, + ret = __hugetlb_vmemmap_restore(h, folio, VMEMMAP_REMAP_NO_TLB_FLUSH); if (ret) break; @@ -641,11 +642,12 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h } static int __hugetlb_vmemmap_optimize(const struct hstate *h, - struct page *head, + struct folio *folio, struct list_head *vmemmap_pages, unsigned long flags) { int ret = 0; + struct page *head = &folio->page; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; @@ -665,7 +667,7 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, * If there is an error during optimization, we will immediately FLUSH * the TLB and clear the flag below. */ - SetHPageVmemmapOptimized(head); + folio_set_hugetlb_vmemmap_optimized(folio); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -681,27 +683,27 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, vmemmap_pages, flags); if (ret) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - ClearHPageVmemmapOptimized(head); + folio_clear_hugetlb_vmemmap_optimized(folio); } return ret; } /** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * hugetlb_vmemmap_optimize - optimize @folio's vmemmap pages. * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. + * @folio: the folio whose vmemmap pages will be optimized. * - * This function only tries to optimize @head's vmemmap pages and does not + * This function only tries to optimize @folio's vmemmap pages and does not * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. + * can use folio_test_hugetlb_vmemmap_optimized(@folio) to detect if @folio's + * vmemmap pages have been optimized. */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +void hugetlb_vmemmap_optimize(const struct hstate *h, struct folio *folio) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0); + __hugetlb_vmemmap_optimize(h, folio, &vmemmap_pages, 0); free_vmemmap_page_list(&vmemmap_pages); } @@ -745,7 +747,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { - int ret = __hugetlb_vmemmap_optimize(h, &folio->page, + int ret = __hugetlb_vmemmap_optimize(h, folio, &vmemmap_pages, VMEMMAP_REMAP_NO_TLB_FLUSH); @@ -761,7 +763,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize(h, &folio->page, + __hugetlb_vmemmap_optimize(h, folio, &vmemmap_pages, VMEMMAP_REMAP_NO_TLB_FLUSH); } diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index a0dcf49f46ba..6a06dccd7ffa 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -18,11 +18,11 @@ #define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page)) #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +int hugetlb_vmemmap_restore(const struct hstate *h, struct folio *folio); long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list, struct list_head *non_hvo_folios); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) @@ -43,7 +43,7 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate return size > 0 ? size : 0; } #else -static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct folio *folio) { return 0; } @@ -56,7 +56,7 @@ static long hugetlb_vmemmap_restore_folios(const struct hstate *h, return 0; } -static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct folio *folio) { } -- 2.25.1