From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C4D5C369DC for ; Wed, 30 Apr 2025 00:55:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB3576B00D8; Tue, 29 Apr 2025 20:55:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C3D9E6B00D9; Tue, 29 Apr 2025 20:55:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A670A6B00DB; Tue, 29 Apr 2025 20:55:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 827626B00D8 for ; Tue, 29 Apr 2025 20:55:25 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E49A916146E for ; Tue, 29 Apr 2025 23:39:00 +0000 (UTC) X-FDA: 83388699240.29.A3F91AD Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by imf22.hostedemail.com (Postfix) with ESMTP id 282D1C0002 for ; Tue, 29 Apr 2025 23:38:58 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eRT0Ud3U; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745969939; a=rsa-sha256; cv=none; b=zV/971LLQSKfZ3mviKfG3phJutYDlCwdlLytO/GSSR/DZze4pSMTsAJwChaTyXjdrjh7yk G9gA42icOaLQTnN9Jhr8FXTkZf6+WE1ibe6e5uKnHx4dpeEFgsKi6kHEOtOLdXlKDHh+t5 OehzzbdZn4A5xRnn5Va/QmXY7FEl2d0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eRT0Ud3U; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745969939; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VDaYNr62oZV/U6TDpPRjwWYHhp0Fz9Nf+NOXmppHJcs=; b=xGR6K6HRAGg5qaUCvlDEL5yC8BB5TWEIhx7O59pycx72m38lcGGgnSdtpL6u1K5DXWs+f2 70BvmmFaIogjKe050/+tfIfvNbfEUElbkFn33k3C2jQ8xssvVvf2SUqad5Nk1o/0Nhy0qJ HjejAqCMc4ZKBX/1+KrFMqJRg8nD6jg= Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-e732fadd0ebso3224293276.3 for ; Tue, 29 Apr 2025 16:38:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745969938; x=1746574738; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VDaYNr62oZV/U6TDpPRjwWYHhp0Fz9Nf+NOXmppHJcs=; b=eRT0Ud3UJRQnWRFVUWRLMC1of14PPZ5MejuDTGI44T5Ncqz/HuhZmapGgDdTAcyql9 fNjavUqbWbXZlNTp74slftJslceKF3BMjHrqIjhVcX3RyD2LU69xuB8hn3mtUcs2Siv7 3riBt5h91MGHj/E/ulMgZ+ZQ5Rh2/Fpizb80ZOxXg8g9/RpGK+qIvekcZWOP/+iVoa5j kuge6p/4dqugwCFIMeO2d8uGk8XJOdmq278Da/Vb41sB8PmC4JYrOx81A/+ScvboZN0x CR1KpEdzmowBsn0GMxY7HDW25YFpheiaIq/ZLpeVOQAA7icNVwFgygcli7TSJFKePxme ySUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745969938; x=1746574738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VDaYNr62oZV/U6TDpPRjwWYHhp0Fz9Nf+NOXmppHJcs=; b=cRyt1iYFo+gE5X5qw1apq0e7nR2cnTrLha+T2dQxemJLR4nUd0lkPIGZ/sFhJ9hOBI lY98TBwel7rxZhgugIbpeQWOHfqgbb8++iFDmgCUV57cnF+40icaC70GqLWadukcXYs9 xz7LhTveRV1Bodg6XUZEa1P4a0C3eio/zNpvWbzCOtoNrZzGIR2PHSHaCcEFfwkKLe4L YXU7IZH4zKnY9yir9vNBXLZNC1xoMYte6+CQ4oOMRIc8bxUMcyslRo7InxeO57Vk1t7J Q4Jp7BtzQgwUUbtEcZMN9frRUunz+Trsbsx+DZ/KvY61nxJpGXQPR4YHmXPEBm9AnLru DquQ== X-Gm-Message-State: AOJu0YzySiwYNWuDcmtS1S8EMPlJGs3/5Pp6AroSAabRi1Fka6encQQd 0dS0eL69Y2ZpZ6sjjtr7PaCKOs1nWVZwtecbUBT1Sd4Ut01kCdZ2Cy/ukQ== X-Gm-Gg: ASbGncvhYib2xorCsvrpUPqCxL2Hu/fdcIYs3FRqXxtLY/vN2fRQ1I57q62WaNDsOCc MCL4Obwy30gZp2HaFwd+hVLw12xXcanMY1qKG1J5SeiLgPZ+J6qEAY2Z6UgRr/eJFVHYc+xCMTw Po1ojM4jdm3QGlTobJA4mtUy0ROuPW2CRLmlBF1FZLUjh9JBLLpdUZT/bpG2CJteSRHSuXh38gZ FhogfWm9mdWLl9tudErufNlAXAtTDqDdpisD7XRBW/JHCfJfbj79Kcv2y7ywTkkAw/WWAWkqOCi N9VwwgZC297EMvulHMCRJ4PHZA+Gd9kc X-Google-Smtp-Source: AGHT+IHn6mOAWGkqEbsXafeS7GzpaJiDdkS+3cjQx5q2B086gbKX/FA06JCBoeWXljznIEoDhpFWCg== X-Received: by 2002:a05:6902:4607:b0:e73:3147:e7c with SMTP id 3f1490d57ef6-e73e7a5043amr1563404276.0.1745969938064; Tue, 29 Apr 2025 16:38:58 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:74::]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e7412f1386esm63843276.22.2025.04.29.16.38.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Apr 2025 16:38:57 -0700 (PDT) From: Nhat Pham To: linux-mm@kvack.org Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, hughd@google.com, yosry.ahmed@linux.dev, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, len.brown@intel.com, chengming.zhou@linux.dev, kasong@tencent.com, chrisl@kernel.org, huang.ying.caritas@gmail.com, ryan.roberts@arm.com, viro@zeniv.linux.org.uk, baohua@kernel.org, osalvador@suse.de, lorenzo.stoakes@oracle.com, christophe.leroy@csgroup.eu, pavel@kernel.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-pm@vger.kernel.org, peterx@redhat.com Subject: [RFC PATCH v2 09/18] swap: implement the swap_cgroup API using virtual swap Date: Tue, 29 Apr 2025 16:38:37 -0700 Message-ID: <20250429233848.3093350-10-nphamcs@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250429233848.3093350-1-nphamcs@gmail.com> References: <20250429233848.3093350-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 282D1C0002 X-Stat-Signature: n7gdokcft17hb35mg8b87igzff6qixqb X-HE-Tag: 1745969938-827079 X-HE-Meta: U2FsdGVkX19XQ7oFQdA7NWBCczk3j94gg6Iqt9Tz1OdjpVbXtE2iHrAD3lxBF4OieTpgbWGNH2bqIUiaaBPqS6dMFAkfCjqSNhFlxVmLYZJPzm92sTY6Yf7i5MWXiaJOZzOphucxhHHqada3PB72jNk+tadPeIoXbE9MnP/ocsQAWhONrfljGUapMMhYAHbicvtqDMuVA1LTPj+2DDjzNZWFFpkCygdx/7NcGNAizN/sLCRC29tkSbu3fWWInHYQb2TZfvUNuFA0ebDZRtpEQRH8eaFA4SKGNxW87jChOuredLYj+KbYg8Obeh+WnYTNd1AULvmY0YmDTRppFgB8SQp5Vc04KPNNZaG8iT+VFPZ+OP6kkGwPQ6AxQMlfSOmcWazxpSY05NkhSIQSx8B36HF8/yhbgMAhgXhyCr4XtXpizIwqIQ093T1nQ10ivpv7bsgB+vsjTSsM+mpUEcNffDUFU6gUXOOKxQ2LHnamFNq8xJZSls2EjSsXhd9xDV7d/DJHioOoi+jVt00BBwIvV8U3UIri7PvIeB81sye8tzLXZPTNT/3DedBgs5b/RxOpFNJZcFnIZ0eJaXaTtKwukHflfc5zEkOVXY5Kb338EMwOP5iLFxrMg4HnaeiSCwfa5/1fAE43KzTvMTllAWipIKwQarg3m5F6PJqGqxqZnsXjuEI6sjkGroFdi2gW3ysuPijOH+/7TD+iNe+cdB+iNCDzKKKYZpJRlmEyxh1PIyM9z/oTK6+r3sAUUmpE4a4VzEG2rEvsuY9W4EdTI9SivlvSTw02T3RKO2nI74wLc1YfVR0gULc6y57GMQ945vWTQhb6+Oq+wGCLezXbzpUEm2aFbXUSyQcmcQyl93JXbcgpuyZZeK5hkf+4asF4N/UlbyrTwy3FguHUUQigzyrh1G1Ihthr1xK3THVBoJK2RO7vRbhGTvRH87H8PrUsmP93U+Lhl9yN1Qavy1LjDZO PoP3IuOI 7JMXCINmTTlESc9/6Cbn2Cyu68s1jpWJHbBCPVyrCZWIyYDbSCgi5Uz4NUUtWAtWB7fTpVZHlF69GWJkRdlg12u6VZLS8BPeMpI5lQPWVc8qzuxbCQ214zVeO05vY8IBR17DPGc4bSyFJFeYxMUkUqm76v+AeNCfS4lEkcs9VcQjQjXxw12yMko2RbYdThXihp7njeC1ZMaU1cA8fr7/yuH7rJ21h4/ztH/6TEjk9wXaEGBz/u8/HSoL41Rvv3R1+XfMWMYx3vJeZhkKw9SlMzce8pJaSD6nfOQBa19uxGOhnH9ZiXoojCW3foa+nPeC8QG6RlOXUDReJaJkAynKKtdig7wDONFPoQ1leau3mGc1P8bZwLSqJYbyAJlOChWqheWadz3qrogPaUdqKxFAmMznZvCnnAtoID5Cj9Mas8BvDPtEEo8ZM39naT88+vr6yGIkQiIraoDlc9uLuzfWZKQ1nzZJjxq/FqcYQGkhKpzYB5wm6/P3+Ohv9IHtZ8gyz5Y5CpJ3SZnY843t3CU4wfwBGdO+pbDoZ2i+ofTtjiZnWvQU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we decouple a swap entry from its backing store via the virtual swap, we can no longer statically allocate an array to store the swap entries' cgroup information. Move it to the swap descriptor. Signed-off-by: Nhat Pham --- mm/Makefile | 2 ++ mm/vswap.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 1 deletion(-) diff --git a/mm/Makefile b/mm/Makefile index b7216c714fa1..35f2f282c8da 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -101,8 +101,10 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o ifdef CONFIG_SWAP +ifndef CONFIG_VIRTUAL_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif +endif obj-$(CONFIG_CGROUP_HUGETLB) += hugetlb_cgroup.o obj-$(CONFIG_GUP_TEST) += gup_test.o obj-$(CONFIG_DMAPOOL_TEST) += dmapool_test.o diff --git a/mm/vswap.c b/mm/vswap.c index 23a05c3393d8..3792fa7f766b 100644 --- a/mm/vswap.c +++ b/mm/vswap.c @@ -27,10 +27,14 @@ * * @slot: The handle to the physical swap slot backing this page. * @rcu: The RCU head to free the descriptor with an RCU grace period. + * @memcgid: The memcg id of the owning memcg, if any. */ struct swp_desc { swp_slot_t slot; struct rcu_head rcu; +#ifdef CONFIG_MEMCG + atomic_t memcgid; +#endif }; /* Virtual swap space - swp_entry_t -> struct swp_desc */ @@ -122,8 +126,10 @@ static swp_entry_t vswap_alloc(int nr) return (swp_entry_t){0}; } - for (i = 0; i < nr; i++) + for (i = 0; i < nr; i++) { descs[i]->slot.val = 0; + atomic_set(&descs[i]->memcgid, 0); + } xa_lock(&vswap_map); if (nr == 1) { @@ -352,6 +358,70 @@ swp_entry_t swp_slot_to_swp_entry(swp_slot_t slot) return entry ? (swp_entry_t){xa_to_value(entry)} : (swp_entry_t){0}; } +#ifdef CONFIG_MEMCG +static unsigned short vswap_cgroup_record(swp_entry_t entry, + unsigned short memcgid, unsigned int nr_ents) +{ + struct swp_desc *desc; + unsigned short oldid, iter = 0; + + XA_STATE(xas, &vswap_map, entry.val); + + rcu_read_lock(); + xas_for_each(&xas, desc, entry.val + nr_ents - 1) { + if (xas_retry(&xas, desc)) + continue; + + oldid = atomic_xchg(&desc->memcgid, memcgid); + if (!iter) + iter = oldid; + VM_WARN_ON(iter != oldid); + } + rcu_read_unlock(); + + return oldid; +} + +void swap_cgroup_record(struct folio *folio, unsigned short memcgid, + swp_entry_t entry) +{ + unsigned short oldid = + vswap_cgroup_record(entry, memcgid, folio_nr_pages(folio)); + + VM_WARN_ON(oldid); +} + +unsigned short swap_cgroup_clear(swp_entry_t entry, unsigned int nr_ents) +{ + return vswap_cgroup_record(entry, 0, nr_ents); +} + +unsigned short lookup_swap_cgroup_id(swp_entry_t entry) +{ + struct swp_desc *desc; + unsigned short ret; + + /* + * Note that the virtual swap slot can be freed under us, for instance in + * the invocation of mem_cgroup_swapin_charge_folio. We need to wrap the + * entire lookup in RCU read-side critical section, and double check the + * existence of the swap descriptor. + */ + rcu_read_lock(); + desc = xa_load(&vswap_map, entry.val); + ret = desc ? atomic_read(&desc->memcgid) : 0; + rcu_read_unlock(); + return ret; +} + +int swap_cgroup_swapon(int type, unsigned long max_pages) +{ + return 0; +} + +void swap_cgroup_swapoff(int type) {} +#endif /* CONFIG_MEMCG */ + int vswap_init(void) { swp_desc_cache = KMEM_CACHE(swp_desc, 0); -- 2.47.1