From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 270F3C25B75 for ; Tue, 4 Jun 2024 02:06:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA0A56B009A; Mon, 3 Jun 2024 22:06:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D280D6B009B; Mon, 3 Jun 2024 22:06:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAC7C6B009C; Mon, 3 Jun 2024 22:06:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9511D6B009A for ; Mon, 3 Jun 2024 22:06:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 52CA31C0E40 for ; Tue, 4 Jun 2024 02:06:21 +0000 (UTC) X-FDA: 82191566562.12.82643FC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 7D2FE140005 for ; Tue, 4 Jun 2024 02:06:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ekSlHce9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3mnZeZgcKCDsvrXkZerdlldib.Zljifkru-jjhsXZh.lod@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3mnZeZgcKCDsvrXkZerdlldib.Zljifkru-jjhsXZh.lod@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717466779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IFb3qAFC4K5HEmXTa3qV09z8mTyiAg8MnrqMh3hMESY=; b=MEbqqi3TU5EHZlxMH0zXG6pcnLRpQ/CqsxwgXlhTNHbdvMHRiUd28+QOGHpr2xG+C8uvC0 y1CQxQerkZVuOpRZL/3ghCo9/ZNwYtg2nD4kcs1plqCTomU0sGLZeqjRkSUCt1wM+SYdlq cRcdrccqD2ICAnXIBuUMGAmLG5oOdJU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717466779; a=rsa-sha256; cv=none; b=PF4evxn0tJfcoxNhzZI6fDHb+AJ2EcpbSnBSYRp9ovxGVN+peCUBre4SvbV1AlHuVAMMxP LbYCDBiuN1bizBSzWNJ8HVtlO+s16mEQOYLgjx3ya2GZS5qDVvZScaWF2BWXgq7Yf6/Brm WzRPMDpox6V4WcEyC7BFk8u0NaStKG0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ekSlHce9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3mnZeZgcKCDsvrXkZerdlldib.Zljifkru-jjhsXZh.lod@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3mnZeZgcKCDsvrXkZerdlldib.Zljifkru-jjhsXZh.lod@flex--yuanchu.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-627ee6339d6so85449087b3.3 for ; Mon, 03 Jun 2024 19:06:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466778; x=1718071578; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IFb3qAFC4K5HEmXTa3qV09z8mTyiAg8MnrqMh3hMESY=; b=ekSlHce9m5tUyP4QsQB/prVlVKf+EJvXGErIRChuRuGgCxNv3z6IEXb0XOvo6gHBRz ATwQFNtjvI5Bc7rp2sjTjnAMvDzAw0E0MrB63IJTtgx0fE55gxvucGuWpG4m2oLh86GE Yg2QjmaPVq3YBzadJ0yVYHHDjaH/LfGNkqenfoCyGCBeGZApu5yoIbDwcB+VfFKUZxsL aBVBk3tYG9WD8j+XLSxbvTD2K/VgzQ1EJGvueJ2Xm0mtd5CFx6Bs/W64LZJr+jYuSBEL BogUG+OufPolojSVen1LxzPBNymGNlVZsEZwqpSUEyF1aCo3vA7sr0xhJUdSY0PB8bzX zN8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466778; x=1718071578; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IFb3qAFC4K5HEmXTa3qV09z8mTyiAg8MnrqMh3hMESY=; b=s+EbRmgM954c6UN6Nn+rEfdeRm/gAJu/Hf3AaPUIxwhxIeU0eCOKZq7DtcmgA9MB73 8otbaUm/W7KSWv/p+91YjZKDk3WN+vHczOg5/6BzQYMsSXqImQv/u3OgenSDhO0dta5x ejIq9dxxifrEWZ1RAEJIz9fSb+tW9YKT3d0YpOsAg/aRgvV2dfiLg/1cI/gOUD8izPhP 4FgcRzbwNmHQ1saKYnSAtPJ2A3ppsP9K20bV5pK36o5swWYFty49k88uz8x5o8EFDxOT QqQW1KEKy+/DMlUf5FEPFQSujwVV+7okF2aa6AJVSO/zNFXlwoi0ti7WtvtBupvjmC95 w8Ow== X-Forwarded-Encrypted: i=1; AJvYcCUBmjqk7YIVnRCvHzPE3MRFmgipIPmZQ+aQcpqITJVHv0zoVvdq2paIHWvOd5tYeeX6gdW7mi9sgUOC+hczII/7ilY= X-Gm-Message-State: AOJu0YzTUWdHUcqNbH3tmxlRBjBRmTb9+w4klh8sb7+13nMp9GokwHkM WzqvbFywHvBbD8VORs+DSpCYcYy84bNdaSrKdEyyEEx5Iq+o+h4feiuDHMku3MrfV/c1rQzQtmB BzxwAOA== X-Google-Smtp-Source: AGHT+IERBr3rpHyGYQbtWTU2Cnqq5B/QCSj2QzI+uA89IMrXbKRLeHdRMrnyxldq/MA/nBeGIemY82Zq3hyI X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:6902:706:b0:dfa:6ea5:c8d5 with SMTP id 3f1490d57ef6-dfa73d9b762mr2674608276.10.1717466778461; Mon, 03 Jun 2024 19:06:18 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:47 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-7-yuanchu@google.com> Subject: [PATCH v2 6/8] mm: add kernel aging thread for workingset reporting From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: 9juuzma1t66snw5g6dyu5g734krjzn73 X-Rspamd-Queue-Id: 7D2FE140005 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717466779-954907 X-HE-Meta: U2FsdGVkX1+apf9SqDsFuS1EqzItzKJ9D8hf5kmQjoSRlIyPvd67glJWebUO9Llxvygav605yE1GGsCReF3a7hKvQzAvp85JmaszogITVEgSvrPyKNoVxkaaPcOu8jUZpv1gbnV7btBJLDLFtAo26XnqAvygKElQj6xFM9YzUV8xZEvsRbgF4B1vovaf+iKwa4HyOIB0FoaZIUERcAyi6qEy9aRTFq9jxAcvPsPSKzaSdGDsWDuFFLU0l/H+OO42vjYOxCdhmKLNdU+/dGolLacBGWg7gd20dKNTNJYkWK4sNp8sM2cwy7ayXhAKFn/+hdtJIldGfDHshDMXchZyMeoQzLMRRHxjdu1wTa0tEAgd5siX1TQanR2PQe0EAaRoOe3tOKVm/EQQu2/f4OSHNDr0RtS+6mAB2knmg8H1DV7pgttg2ZMzJEKY/lFpXl64yFe+lb5DQv8ba/hbS+Q7qkc+lNnCvTGOPDG+AIT1k//hWG+58xitQCIfv+MjmWSh9Lu5PssoTWc4W9TBDAA4kf75h+WJpI7bd4sN6Y4F/GHomRhCx5qIqkeFWUkGterO3tnujkx4sVYmTyX7IbeDrZkGflPQfggMvz4rC9MsQy86SGDsn99ERz30X54+9O8TJ8sVYHBY0DphE1KLSBgBL8AS3jxpQ8SzGmuYkB8xmej0lCqY5/DejP1Z9EuYrbtZq2UW7RPMnMigaWV290P7oNcpUsh23aOCcYe7Cju5WBvt3Y1K9qLrDcFkQs8HBFWruzfm2YemtGocoDE9xfVaHK7U0ZFOYLOPC9DZc/Ommo3ODPsJ/I8gBYTWeWHDx3o87AK85x5sty/KLqZTzKqmnxY5BAIMnqacXGwl2yOP33WKcVKIMN41TcYzJQiVA+3UZybVU8d7Poc1Ah9qL+mhhfFfyGxpeQIzFRl7DqRVUhS+EprkMMXCsv2j1xZFGWtFZ7vMSgeKv/GF4ZOkB2M isOxfS/j DRCJs39lXVOsNkoaoRbXPoKPyMHSxy29Ci+uDV8UuSYUBZ7yx/KPMF9GGxliLbdSbpEAHfDcXHBV6BLb/wGqmyH14jeuN77i28dfKlZkJTHQ2Gs1T/7WZz+FYCchTFb9rVxZ4Mrmt1OPspMbwue177aGgGe9TgePIo+8I/bTAkHLCr5n+3AXcESfymPCf1/G+P1MTkC0XpZT0gPxEZ2zN97/Y1OUjEfYInU+0S50IIvNxFUFXkl4P9ewG1HeNHGt99Bn2QseVEr/hgqfU7B4xjYzZLHkk2kdixJOo50iHWHbbapxJMnie/LDiVLIxhRakAHYfSNG3KjVva/D2Lze4/VuvmT88GSJUc2DYNGTaDCW3bqNO1Rw48IJac1P/bejtPqBmjo25S7U/mSYhBaH7WhF1X74dnWZkg+nD7vMzpqsDAOVEbgVmzWiOMDhiTj9QMD4Em40T9JkrYXTiUbd/O82cTwSzqJx0s9Em/KmrHqMxLqBnXXJd5Za0O1s1sWNKZiPcnoNhH9peWhul/dKGnR2jyWNLFY//al/tkxANiurUQMHYWvvXD78Qy+/dNM4Lj587wSZYMBkr4JZ/Z6rc9v+snuiJTe7mAt5frOEM6D+cO175htLh3AVXFtXR9bSGLCCKQu4LiQCDBuOPp5jOWtZYDA+KG2PNtwLyULfeed1acQieUsBzaqYimg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For reliable and timely aging on memcgs, one has to read the page age histograms on time. A kernel thread makes it easier by aging memcgs with valid refresh_interval when they can be refreshed, and also reduces the latency of any userspace consumers of the page age histogram. The kerne aging thread is gated behind CONFIG_WORKINGSET_REPORT_AGING. Debugging stats may be added in the future for when aging cannot keep up with the configured refresh_interval. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 11 ++- mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/memcontrol.c | 8 +- mm/workingset_report.c | 15 +++- mm/workingset_report_aging.c | 127 ++++++++++++++++++++++++++++++ 6 files changed, 162 insertions(+), 6 deletions(-) create mode 100644 mm/workingset_report_aging.c diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index ae412d408037..9294023db5a8 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -63,7 +63,16 @@ void wsr_remove_sysfs(struct node *node); * The next refresh time is stored in refresh_time. */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat); + struct pglist_data *pgdat, unsigned long *refresh_time); + +#ifdef CONFIG_WORKINGSET_REPORT_AGING +void wsr_wakeup_aging_thread(void); +#else /* CONFIG_WORKINGSET_REPORT_AGING */ +static inline void wsr_wakeup_aging_thread(void) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT_AGING */ + #else static inline void wsr_init_lruvec(struct lruvec *lruvec) { diff --git a/mm/Kconfig b/mm/Kconfig index 03927ed2adbd..f8ff41408b9c 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1258,6 +1258,12 @@ config WORKINGSET_REPORT This option exports stats and events giving the user more insight into its memory working set. +config WORKINGSET_REPORT_AGING + bool "Workingset report kernel aging thread" + depends on WORKINGSET_REPORT + help + Performs aging on memcgs with their configured refresh intervals. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index ed05af2bb3e3..e9c9048e1e09 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -97,6 +97,7 @@ obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o +obj-$(CONFIG_WORKINGSET_REPORT_AGING) += workingset_report_aging.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48cdc9422794..547a2161b7e2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7281,12 +7281,12 @@ static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, { unsigned int nid, msecs; struct wsr_state *wsr; + unsigned long old_interval; struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); if (ret < 0) return ret; - wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; mutex_lock(&wsr->page_age_lock); @@ -7305,9 +7305,13 @@ static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, wsr->page_age = NULL; } + old_interval = READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(msecs)); unlock: mutex_unlock(&wsr->page_age_lock); + if (ret > 0 && msecs && + (!old_interval || jiffies_to_msecs(old_interval) > msecs)) + wsr_wakeup_aging_thread(); return ret; } @@ -7358,7 +7362,7 @@ static int memory_ws_page_age_show(struct seq_file *m, void *v) if (!READ_ONCE(wsr->page_age)) continue; - wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + wsr_refresh_report(wsr, memcg, NODE_DATA(nid), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) goto unlock; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 72f2cad85a0d..15118c4aecb1 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -274,7 +274,7 @@ static void copy_node_bins(struct pglist_data *pgdat, } bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, unsigned long *refresh_time) { struct wsr_page_age_histo *page_age; unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); @@ -291,10 +291,14 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, goto unlock; if (page_age->timestamp && time_is_after_jiffies(page_age->timestamp + refresh_interval)) - goto unlock; + goto time; refresh_scan(wsr, root, pgdat, refresh_interval); copy_node_bins(pgdat, page_age); refresh_aggregate(page_age, root, pgdat); + +time: + if (refresh_time) + *refresh_time = page_age->timestamp + refresh_interval; unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; @@ -357,6 +361,7 @@ static ssize_t refresh_interval_store(struct kobject *kobj, unsigned int interval; int err; struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned long old_interval = 0; err = kstrtouint(buf, 0, &interval); if (err) @@ -378,9 +383,13 @@ static ssize_t refresh_interval_store(struct kobject *kobj, wsr->page_age = NULL; } + old_interval = READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); unlock: mutex_unlock(&wsr->page_age_lock); + if (!err && interval && + (!old_interval || jiffies_to_msecs(old_interval) > interval)) + wsr_wakeup_aging_thread(); return err ?: len; } @@ -470,7 +479,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) diff --git a/mm/workingset_report_aging.c b/mm/workingset_report_aging.c new file mode 100644 index 000000000000..91ad5020778a --- /dev/null +++ b/mm/workingset_report_aging.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Workingset report kernel aging thread + * + * Performs aging on behalf of memcgs with their configured refresh interval. + * While a userspace program can periodically read the page age breakdown + * per-memcg and trigger aging, the kernel performing aging is less overhead, + * more consistent, and more reliable for the use case where every memcg should + * be aged according to their refresh interval. + */ +#define pr_fmt(fmt) "workingset report aging: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DECLARE_WAIT_QUEUE_HEAD(aging_wait); +static bool refresh_pending; + +static bool do_aging_node(int nid, unsigned long *next_wake_time) +{ + struct mem_cgroup *memcg; + bool should_wait = true; + struct pglist_data *pgdat = NODE_DATA(nid); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + struct wsr_state *wsr = &lruvec->wsr; + unsigned long refresh_time; + + /* use returned time to decide when to wake up next */ + if (wsr_refresh_report(wsr, memcg, pgdat, &refresh_time)) { + if (should_wait) { + should_wait = false; + *next_wake_time = refresh_time; + } else if (time_before(refresh_time, *next_wake_time)) { + *next_wake_time = refresh_time; + } + } + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); + + return should_wait; +} + +static int do_aging(void *unused) +{ + while (!kthread_should_stop()) { + int nid; + long timeout_ticks; + unsigned long next_wake_time; + bool should_wait = true; + + WRITE_ONCE(refresh_pending, false); + for_each_node_state(nid, N_MEMORY) { + unsigned long node_next_wake_time; + + if (do_aging_node(nid, &node_next_wake_time)) + continue; + if (should_wait) { + should_wait = false; + next_wake_time = node_next_wake_time; + } else if (time_before(node_next_wake_time, + next_wake_time)) { + next_wake_time = node_next_wake_time; + } + } + + if (should_wait) { + wait_event_interruptible(aging_wait, refresh_pending); + continue; + } + + /* sleep until next aging */ + timeout_ticks = next_wake_time - jiffies; + if (timeout_ticks > 0 && + timeout_ticks != MAX_SCHEDULE_TIMEOUT) { + schedule_timeout_idle(timeout_ticks); + continue; + } + } + return 0; +} + +/* Invoked when refresh_interval shortens or changes to a non-zero value. */ +void wsr_wakeup_aging_thread(void) +{ + WRITE_ONCE(refresh_pending, true); + wake_up_interruptible(&aging_wait); +} + +static struct task_struct *aging_thread; + +static int aging_init(void) +{ + struct task_struct *task; + + task = kthread_run(do_aging, NULL, "kagingd"); + + if (IS_ERR(task)) { + pr_err("Failed to create aging kthread\n"); + return PTR_ERR(task); + } + + aging_thread = task; + pr_info("module loaded\n"); + return 0; +} + +static void aging_exit(void) +{ + kthread_stop(aging_thread); + aging_thread = NULL; + pr_info("module unloaded\n"); +} + +module_init(aging_init); +module_exit(aging_exit); -- 2.45.1.467.gbab1589fc0-goog