From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10A5BC5AD49 for ; Sun, 8 Jun 2025 07:36:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B8F06B0098; Sun, 8 Jun 2025 03:36:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 98FF16B0099; Sun, 8 Jun 2025 03:36:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85C9F6B009A; Sun, 8 Jun 2025 03:36:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 63AF46B0098 for ; Sun, 8 Jun 2025 03:36:27 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DE4101A0E8D for ; Sun, 8 Jun 2025 07:36:26 +0000 (UTC) X-FDA: 83531425572.07.D6D956F Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf21.hostedemail.com (Postfix) with ESMTP id F370C1C0005 for ; Sun, 8 Jun 2025 07:36:24 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IO1f+K09; spf=pass (imf21.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749368185; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Om/soDdlb1irFSyjr02L7f3XW+ZfYX9+/hWOK174JY=; b=stMMKgyOJcX2gNn/ocD4kWfIUyH134kuY9zd5uJtwerddZcEAtxNZcrq8FwnK+EV8+KgqB 6lq9i0W4gwNkHRhmypLEH761ukoS2djgOzuyANfgKrybxZfylGd0ELqbQXkAUwBqvZwu9W bzMVNEO+cIU1it9Ip9Qvd03HtWuKPng= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IO1f+K09; spf=pass (imf21.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749368185; a=rsa-sha256; cv=none; b=DxBIwuzSfpOhmkbhhAbTx9+iWNntVeGUHJxyX5Qjio18ETr9o+z5yKXKa/ilBcwtYDdOdk 06in2bx8AAHUD4W7Mw9boxzGJijMTKdntYKrV/jHrICihfWTSPgCkKYUa1nMg7olycO4Ms BE8siZa14VgEImhVeJJJGb0mEX0vF5A= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2353a2bc210so29178105ad.2 for ; Sun, 08 Jun 2025 00:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749368184; x=1749972984; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Om/soDdlb1irFSyjr02L7f3XW+ZfYX9+/hWOK174JY=; b=IO1f+K09R0bGQ4jCZ1AHcDHeN6LSPTW5q2IkBMt+Uh/0uYNNamPQkarVFots+nyd24 PuBn6RStgqBDIhZD1ZIpuOgaSPBl2tu2WGt/UlVsbBCvKCI7rjrfwWs0KU8y/FjDZeUq chRI3cWMZHx9DGDa3inU9oHHQAC6FNOMEiGfnY9GhR7LHAMnmTjCAEFjzFvijfyXyBFz zYtFlCsXNFOrJG5edRKf8r+MYUhDMiYD9Vi+6JRKBZGQMwQbaW683le4zKS0PouqwOxF 9m+tv16AXuKoDK2fgg3yoBh5e7gPWITdwIisIjlcMqpVGV3JncMnmxIk54gAzbw+eVKh dfTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749368184; x=1749972984; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Om/soDdlb1irFSyjr02L7f3XW+ZfYX9+/hWOK174JY=; b=RZdeXh4DP1HLluye0x72Li5Qh/ehYHm7GF5W5Ov2WGvuBJtgHJQB8IlLVVCFycJQ8O eI3IQYHSeHbDghNudGDY6VOrCcs7D//x8zh15M8VoYnG5G/gFUQ65c1XWuSk6vLVqSt6 jKGz4YrCuDYcRArzKq4hF2LOMPgqk1+HQoyCtacJG4FzvgBbxXGNBJMIKwYTj24u/mii 7s3biN3CSrpvDe5MLtT9ivO95mIyhOrmgsMYXgAvLBIkvJuuTBFEuWvfUbFI+1O1AjIq L4MUhYugboSoRblLhjHIJ33m8kdX8owkVOa/HSJISZmPC5A4Y4zZ0GFeUHhjLKSMojdl zz9g== X-Forwarded-Encrypted: i=1; AJvYcCX8BwHeJLd/2ZoCH+9SUireRSk/ALzA2Kb4c82h04FAa622UYjQMPfEoW6Y1W6qE6AP9O/dzqoMuQ==@kvack.org X-Gm-Message-State: AOJu0YxLBhIq2I9PEocMAQHTKQt74oUILYcqmheG4hTqVLf7N/yh7PzQ PWTk1jdDtGXIB3l14av2CG6poo2ZmfNOTyJPfDVIIWrKVihnLejtae/t X-Gm-Gg: ASbGnctFurcKKhROLde3yPfxF8Ej9l5SotcHZhTJUokHqb4BdN36btSF+x3wdsZPdYD YX9OTcT4qD9DJYAy4OfeEnIrq38fDuP6I2yrC6KCJS/lZppgoEV/TzUwtGt6N0tg+3pF5UjWIHt ZlRzlhm/BdpAN6kWwGpnapdR1J9R14vRh4oWlT8FDGci0+/TnVLSdDDrs74EfmLmuVbkOy9RzJr HpNPp6nci27V3LuigGNDhONfjGcLUchvSEcpiYf31Dz2g/HEcKYp7oy+/eNYnsMu3TcnXUmKON4 MA1ihYCJRscq7bV+E9CeajAOXBZYm/7+OeEGODFPxgc5XeKSPKf1E2d2Wx+aj6z0XhEtA0xBUFY = X-Google-Smtp-Source: AGHT+IEFc5S1UisQmcqD08/fo40gZbWFkNW0G+aVNGeaRx/g3WCuw3RQL35rBd2EOVk2O6yocWSgUw== X-Received: by 2002:a17:902:f686:b0:234:aa9a:9e0f with SMTP id d9443c01a7336-23601d21231mr136980265ad.23.1749368183703; Sun, 08 Jun 2025 00:36:23 -0700 (PDT) Received: from localhost.localdomain ([39.144.124.91]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-236035069c3sm35968135ad.234.2025.06.08.00.36.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 08 Jun 2025 00:36:23 -0700 (PDT) From: Yafang Shao To: akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, hannes@cmpxchg.org, usamaarif642@gmail.com, gutierrez.asier@huawei-partners.com, willy@infradead.org, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org Cc: bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [RFC PATCH v3 4/5] mm: thp: add bpf thp struct ops Date: Sun, 8 Jun 2025 15:35:15 +0800 Message-Id: <20250608073516.22415-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20250608073516.22415-1-laoar.shao@gmail.com> References: <20250608073516.22415-1-laoar.shao@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F370C1C0005 X-Stat-Signature: 4c4zejqpznjz9uoo86dyfeyotp3majp5 X-Rspam-User: X-HE-Tag: 1749368184-430622 X-HE-Meta: U2FsdGVkX1/IgyUbxthK5clNWmOyqULmW2Fr1efE9IKSTbC/ZgQkprCClK5LUxXd8y0cwWfMW7DN3rxElbRpgIZnmQDFMxkqAkZKEUqzUDvSKM5uL3XSBtjXK9vjDb7CQ5Evy6sNYgKK4FfeE84dXYfAhDXCMfQ0poMFRsFfFJ34/b3zqz9LIbCIKkFCh9ufnrwDzXD01RjAXfVGOe2jN92kXdD0LDmEERt172BwJ/SkjzYh000fJcsd9o120FG3NCKy2+jGBGpGnFD/MXieshg+wE9PgrVnLXIrcw8PZgDgbOdR2v1bJVIxFYoeaQVNzli4zJ3HFNt9AZ1epdwVK9W9/+r3En1WOw3A0rhsXb1O/TWOiG64+0MFqGDhUdECIYaJ72BQBbchBW96q4MSKRExdT3kOkQiFgSRVvWtz+Ck36PbC+t6l+CXKgInXe0ZW61RDD3oD3QUCJ8Od/RURZWvaIVLgSaF6ade+cI7K5Yo1ciXhkkLULbzeXugon7TNsLVbJIePf/KhhkcsaHrYRu+ZTMI555vEyVY0XQdlxeYwXayvCBbBlWK4diPzzv9YXSFCJc6z4ZZRI1j4qCFTJnNBfiiFdK+NBnpWR2stD+LofwzBGxxTmqs8gb5YcEe5ZFDWTuwUyiL7TJHcaYTwCpEIVhtpU3GTVFXMxY1DBqai3L9D0dIIruZ73S/OslTEuveAQLgRpeCludDpmoRjUrDeoX1wcxnNFLPc4ITWUZwso6nRhC5SCBcN6+1GKs6CGdNS7ZjaplyzZZNiwfnqPG26kF5fenyZMsB2k3JuI5D10OkCbJATU1NEXg8DxKoYiWDBFhMulZ5YqV+N2NBWvyrR+DS5qB0c6HuqQ+fc0cxYmoksSBQLDjDVb9sgjpB4UzN5DthaXFD4TBwNElrb0aYrYuGSJWSHezNea7RBJEOvrY8b2PDveu0i0WHhdkbpIpMSejUlDQZ2+Rr6pV mSN9mYUW AkK/C423lYc2NCMVmubJ8VN0Vr4MoRdWMp575GAroWzyt7yBrur+yCzxt0zIdifvff+EJLMIXb5r2oNgB4Sl0BxPMcYkz1r1MJI4H6Eqt4NMFMvmZ7kXvc0ERoCOUGYNfh6LXQ8rx0ZHkKDixWdm0oBy7jtwgA2xXUG4feZ+nWak3+65gOTdLS6Gi4QB6q76JiImtqymBZqKOi11sCM3DqTu+fbnoj6EvnaSSCj29tAvLGc5Y+4KzEAPgXAQLB8bj7UFTietddWNri5TVRVr3RVT2VhDzqR2U55YeDhl12Bj+Wi8uZfYVnqt11Sl1T7cGos66t2KGqI3utlBqnZ3E2ua1j03bNoRqEQorq0MnsAiHMuqD+JbfAq6g75iaho5528wyE3DJjE4Hfca004grcGR6oXGh4mEBVZunx8i0CZkCKzxXE+RvZPzD+JFz41u0/5YTjKIFK5leF6dsuicREXlTzupf82NgO9LF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new bpf_thp struct ops is introduced to provide finer-grained control over THP allocation policy. The struct ops includes two APIs for determining the THP allocator and reclaimer behavior: - THP allocator int (*allocator)(unsigned long vm_flags, unsigned long tva_flags); The BPF program returns either THP_ALLOC_CURRENT or THP_ALLOC_KHUGEPAGED, indicating whether THP allocation should be performed synchronously (current task) or asynchronously (khugepaged). The decision is based on the current task context, VMA flags, and TVA flags. - THP reclaimer int (*reclaimer)(bool vma_madvised); The BPF program returns either RECLAIMER_CURRENT or RECLAIMER_KSWAPD, determining whether memory reclamation is handled by the current task or kswapd. The decision depends on the current task and VMA flags. Signed-off-by: Yafang Shao --- include/linux/huge_mm.h | 13 +-- mm/Makefile | 3 + mm/bpf_thp.c | 184 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+), 10 deletions(-) create mode 100644 mm/bpf_thp.c diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 6a40ebf25f5c..0d02c9b56a85 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -54,6 +54,7 @@ enum transparent_hugepage_flag { TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG, TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG, + TRANSPARENT_HUGEPAGE_BPF_ATTACHED, /* BPF prog is attached */ }; struct kobject; @@ -192,16 +193,8 @@ static inline bool hugepage_global_always(void) #define THP_ALLOC_KHUGEPAGED (1 << 1) #define THP_ALLOC_CURRENT (1 << 2) -static inline int bpf_thp_allocator(unsigned long vm_flags, - unsigned long tva_flags) -{ - return THP_ALLOC_KHUGEPAGED | THP_ALLOC_CURRENT; -} - -static inline gfp_t bpf_thp_gfp_mask(bool vma_madvised) -{ - return 0; -} +int bpf_thp_allocator(unsigned long vm_flags, unsigned long tva_flags); +gfp_t bpf_thp_gfp_mask(bool vma_madvised); static inline int highest_order(unsigned long orders) { diff --git a/mm/Makefile b/mm/Makefile index 1a7a11d4933d..e5f41cf3fd61 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,6 +99,9 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +ifdef CONFIG_BPF_SYSCALL +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += bpf_thp.o +endif obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o diff --git a/mm/bpf_thp.c b/mm/bpf_thp.c new file mode 100644 index 000000000000..894d6cb93107 --- /dev/null +++ b/mm/bpf_thp.c @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include + +#define RECLAIMER_CURRENT (1 << 1) +#define RECLAIMER_KSWAPD (1 << 2) +#define RECLAIMER_BOTH (RECLAIMER_CURRENT | RECLAIMER_KSWAPD) + +struct bpf_thp_ops { + /** + * @allocator: Specifies whether the THP allocation is performed + * by the current task or by khugepaged. + * @vm_flags: Flags for the VMA in the current allocation context + * @tva_flags: Flags for the TVA in the current allocation context + * + * Rerurn: + * - THP_ALLOC_CURRENT: THP was allocated synchronously by the calling + * task's context. + * - THP_ALLOC_KHUGEPAGED: THP was allocated asynchronously by the + * khugepaged kernel thread. + * - 0: THP allocation is disallowed in the current context. + */ + int (*allocator)(unsigned long vm_flags, unsigned long tva_flags); + /** + * @reclaimer: Specifies the entity performing page reclaim: + * - current task context + * - kswapd + * - none (no reclaim) + * @vma_madvised: MADV flags for this VMA (e.g., MADV_HUGEPAGE, MADV_NOHUGEPAGE) + * + * Return: + * - RECLAIMER_CURRENT: Direct reclaim by the current task if THP + * allocation fails. + * - RECLAIMER_KSWAPD: Wake kswapd to reclaim memory if THP allocation fails. + * - RECLAIMER_ALL: Both current and kswapd will perform the reclaim + * - 0: No reclaim will be attempted. + */ + int (*reclaimer)(bool vma_madvised); +}; + +static struct bpf_thp_ops bpf_thp; + +int bpf_thp_allocator(unsigned long vm_flags, unsigned long tva_flags) +{ + int allocator; + + /* No BPF program is attached */ + if (!(transparent_hugepage_flags & (1<allocator; + bpf_thp.reclaimer = ops->reclaimer; + return 0; +} + +static void bpf_thp_unreg(void *kdata, struct bpf_link *link) +{ + clear_bit(TRANSPARENT_HUGEPAGE_BPF_ATTACHED, &transparent_hugepage_flags); + bpf_thp.allocator = NULL; + bpf_thp.reclaimer = NULL; +} + +static int bpf_thp_check_member(const struct btf_type *t, + const struct btf_member *member, + const struct bpf_prog *prog) +{ + return 0; +} + +static int bpf_thp_init_member(const struct btf_type *t, + const struct btf_member *member, + void *kdata, const void *udata) +{ + return 0; +} + +static int bpf_thp_init(struct btf *btf) +{ + return 0; +} + +static int allocator(unsigned long vm_flags, unsigned long tva_flags) +{ + return 0; +} + +static int reclaimer(bool vma_madvised) +{ + return 0; +} + +static struct bpf_thp_ops __bpf_thp_ops = { + .allocator = allocator, + .reclaimer = reclaimer, +}; + +static struct bpf_struct_ops bpf_bpf_thp_ops = { + .verifier_ops = &thp_bpf_verifier_ops, + .init = bpf_thp_init, + .check_member = bpf_thp_check_member, + .init_member = bpf_thp_init_member, + .reg = bpf_thp_reg, + .unreg = bpf_thp_unreg, + .name = "bpf_thp_ops", + .cfi_stubs = &__bpf_thp_ops, + .owner = THIS_MODULE, +}; + +static int __init bpf_thp_ops_init(void) +{ + int err = register_bpf_struct_ops(&bpf_bpf_thp_ops, bpf_thp_ops); + + if (err) + pr_err("bpf_thp: Failed to register struct_ops (%d)\n", err); + return err; +} +late_initcall(bpf_thp_ops_init); -- 2.43.5