From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A75DCEB64D7 for ; Wed, 21 Jun 2023 18:17:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 470178D0006; Wed, 21 Jun 2023 14:17:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F9E78D0002; Wed, 21 Jun 2023 14:17:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C0688D0006; Wed, 21 Jun 2023 14:17:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1B4CC8D0002 for ; Wed, 21 Jun 2023 14:17:10 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DBDD51C80E6 for ; Wed, 21 Jun 2023 18:17:09 +0000 (UTC) X-FDA: 80927561778.02.282E886 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id D44684001C for ; Wed, 21 Jun 2023 18:17:07 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YrxzTYP2; spf=pass (imf11.hostedemail.com: domain of 3oj6TZAcKCJcPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3oj6TZAcKCJcPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687371427; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VxhcEKjfe7vCc52XMrwf8C6JOsMy8jrJM8NpygXtz0A=; b=nj9yYR4YhVyKp7r9t3FOD2Al0B1U8nWUGbydq00jHNpJL5VQ8lF0tWqGj9AX+YHx6MYs8J sP3C8Bl6E/1bedAcyecTUqGcvwDlTb5x2/EHxErUlkSz6wxMABDAeoJ4JBLceHPO4a1QNw UYiYmfQKijk45NOhMRLoIvKHwRIf0bY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687371427; a=rsa-sha256; cv=none; b=FO2XNpgXwg5U8xYdqNwWbzK8b+iCol+RKos5Kt/fdjKyW5pLJ5S/Oo9qMCqJ1hPcpIbqYz GUt+X4h6GpP67xHu+/5UxLYBpjef7H9WfuvxaT4BYglNQ+PNgOPl5DXwC6t/GcE6TUvwwv PGkiF+F4nR2/H4RAa/fdlHB+CyX3QU0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YrxzTYP2; spf=pass (imf11.hostedemail.com: domain of 3oj6TZAcKCJcPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3oj6TZAcKCJcPL1E38L7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-bffcd3cd439so1238083276.1 for ; Wed, 21 Jun 2023 11:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687371427; x=1689963427; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VxhcEKjfe7vCc52XMrwf8C6JOsMy8jrJM8NpygXtz0A=; b=YrxzTYP2Y/JUxuvH4119nDn3m669dvyeu1ioqqJFdsHRVZAj+ZA9EqTpltTHxG50zq qfOTrJc6zM1jH8E4+2c3lCHG6eE3+bnJMfrPN5dNe5p0C2a8FPj+SyHCvquxF7n79tcF VHmZC11UUpY3AyupbmMbK35VBgwGx2dy7P5HJ+del76VoLJE5CBkEqHVxN6YMZmEjiNW Nm/tK449uQplLimDFQlk1RDQFbICohvd+xoAZOn5/kUpZH7auwDHZzdUwPamBHfS4zje DibmTNCjMMTpWrlCnyh4PwuG3h/Y48tI7tHM9kAgnxNARzwXIjzj+7DB8CzJyW06rhtB RT1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687371427; x=1689963427; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VxhcEKjfe7vCc52XMrwf8C6JOsMy8jrJM8NpygXtz0A=; b=VF//n2z0QMaemcRYwLfVHMKR0l5wWUw4nItKiwkM3IlM8ZYAAvvATOY/us2dLq070o tNTQ6ErPp7G4/CoyAYHVSiBmEsEgqSfUd+rFjdHWtqtJXpK7rqRPGdJ3WJH1nl1piz4Q XGgSkO+mxp/fAgyZ14jFlRk5g6wyMwSh8MELxHWjFd5Efd9PP/Yhd/x1iYyAszuyaRXT seEh64/3KizAM6ptyCVYI4m6ZuSr6owcX1VXhlrb4ObyFnL2mJIZjSxCJdRDxgz2hNmB AP2xTx9xKj2mQPVqOG/SzJjD0RqR5XRDVrtrGIQyD20VO0MVOJuTyf/Si1n2vpcqVfnT rKGw== X-Gm-Message-State: AC+VfDxzlr/+kgfvqpGf35ePjJPdGgW95k4LtPqJNldjJ5/xhQmHRUVi oBbcNeOPE7qi3n77QtdxkmcEPWDjG8+z X-Google-Smtp-Source: ACHHUZ65jQ/dKsvSfi9erBP0SAOa4RBu0GZrXY0q5bDJnqAf/X8F8WR+vY/upGLS77avU0vNnyM+2Ijryv8A X-Received: from yuanchu.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:4024]) (user=yuanchu job=sendgmr) by 2002:a25:7710:0:b0:be4:aab4:789b with SMTP id s16-20020a257710000000b00be4aab4789bmr6836965ybc.8.1687371426885; Wed, 21 Jun 2023 11:17:06 -0700 (PDT) Date: Wed, 21 Jun 2023 18:04:50 +0000 In-Reply-To: <20230621180454.973862-1-yuanchu@google.com> Mime-Version: 1.0 References: <20230621180454.973862-1-yuanchu@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621180454.973862-3-yuanchu@google.com> Subject: [RFC PATCH v2 2/6] mm: add working set refresh threshold to rate-limit aggregation From: Yuanchu Xie To: Greg Kroah-Hartman , "Rafael J . Wysocki" , "Michael S . Tsirkin" , David Hildenbrand , Jason Wang , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Yu Zhao , Kefeng Wang , Kairui Song , Yosry Ahmed , Yuanchu Xie , "T . J . Alumbaugh" Cc: Wei Xu , SeongJae Park , Sudarshan Rajagopalan , kai.huang@intel.com, hch@lst.de, jon@nutanix.com, Aneesh Kumar K V , Matthew Wilcox , Vasily Averin , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-mm@kvack.org, cgroups@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: D44684001C X-Rspam-User: X-Stat-Signature: osa19x5tybxjhdsbn8dqf1jr3bk5g398 X-Rspamd-Server: rspam03 X-HE-Tag: 1687371427-866841 X-HE-Meta: U2FsdGVkX1/wwKKbzQhjHj4SuatdKqsWozvDUdD2Bk9gtd8VXDrD0rpOorH43OJhfsUK2Rupk0b8qQRYBJzQn5SMXvfUiXIEmmP6FLsz3Xb5JPlkCZqZrJrKyLhBG++rAlfrNGgHH6NKF3EpsOGTMFENOL4dSWCJ9KPDwMHPn8VTlrPlEtIbfL7gwPC6Hj3dyNF2wMzyVk3DcAbJbBMVufzZdZptR4u7k1gfo3mPE5kGGB0x5s5pV+Mj3ZYHPeumb5yxAmbiWqTpXvDMopiqAAdDpyynNS0z7F/3VDcRxVSa3y+sGm9amIQD3zHpZbUbTC66PUoinP+FkEi9QHsTCbMmCdHtNOieuroJjzbMKTsmJIcB+yJTcF2NHjUHGY9WCxGOF62bXeGla89t5b+u8MJQs0+Hh4b1dd0PeqJhyqJJM5oDLu/K0SfJTcdpuaBrrWAFh3gm4NLd2rOwg3Yy7+oB7wqz//icS+loFa6vSw5UYsnEiz9JV8NJp2Pj0hcm6DSgfM2coOK4r9dg94YGAZktAIiYMcx/GI6lF/df+yv39xhOV2/3lwZ/SCFTUjcGN4JcuRA3cFWaIXXOBuZScOrf22MxStbtVueV6CZ3ivyBnPWr4h6O0jNnVXy0BRrfMAQoQjnkFCRvBdaQhomdS8BUjKW3sboYfKREZAtOZS4QRxqJ3dkbkkCLO0geNHPRgkk9essi/FtrnMyMOqnNVCZUzzxrFdScM0jTJ/oUHZd3uk8qwV2URlB+xvAjQmrQQk33bmfcFJKicUZRbYU0lxopbwBMf8c2gNOgG0qEIPlxrxDy2+NxN5giOQCsX/S4vyRibGWl3HR7atO7X3dYNaHnDVu/GiEMEzKfAQbZMdKGtsTKA1QGQRKu3IROln1LgoQ6g2trtrm2IQ8bQWJ5uhfwmEEM591o/MvV0zOwQuiCNTSAOqJzR+vVsXxdKEjwYhaY0nU2WqEMjiCc1N8 t+M0sIZa ibGr+8eRXSasAXlQk5zoXoCwG0CFH9fknzOKtpzoWuSON8KA4rejbNmyvU+k06u9iuvfPdD+klYD1FaSoctDBto+oiv4f3TTe1bFYqJDVDRz+QfM5YNf5g+EsugjMjHnv0MSOLHpl3/xxC1ZVgMbLHIMuUYeM0hDx1e24WobY5uK0DJIR9OZY/5cQ8+uBGLBVny1PTmXluJ+7EyG64IescufUJw57yrWdAMf8YEL8Gx8K2Iukv7Twyr1wfigrINHOIOjodD8GvEbD601OfjjbtwpVRMzQJQMtPvf0TPxcZYvf1amuEm5srk4ZHHtBAQFYOFbctKwCVUD7EyJ01o9vFr+2AmkIdNqgzMgtXKsqpRcF00Vudf/x7pRFCv1R8yAliblLNR8uFwdjDKIFPqDAPMMUWSf8liGmVfCTY1KzqVgmIEWPFw0J9x86TtKzRH4JavVhhHXYi2K1jPqAuy0a9WtJO2F2PJR+jz15HziPiW0yLlfGW+aebIqGQAJ6OHXII7lhFwEmg6cZhbd77CVA+Df6LAVVyi4RbTPwq9BG0WbgvA4m8UGE9OR79QDrj1mTNEIXjMtZ+08FUJzIE3hm8bN60kiAMd258JP0Sjb920xbyEGkoXFsRkjHiQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh threshold is a rate limiting factor to working set histogram reads. When a working set report is generated, a timestamp is noted, and the same report will be read until it expires beyond the refresh threshold, at which point a new report is generated. Signed-off-by: T.J. Alumbaugh Signed-off-by: Yuanchu Xie --- include/linux/mmzone.h | 1 + include/linux/wsr.h | 3 +++ mm/internal.h | 11 +++++++++ mm/vmscan.c | 39 +++++++++++++++++++++++++++++-- mm/wsr.c | 52 +++++++++++++++++++++++++++++++++++++++--- 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 96f0d8f3584e4..bca828a16a46b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -362,6 +362,7 @@ enum lruvec_flags { #ifndef __GENERATING_BOUNDS_H +struct node; struct lruvec; struct page_vma_mapped_walk; diff --git a/include/linux/wsr.h b/include/linux/wsr.h index fa46b4d61177d..a86105468c710 100644 --- a/include/linux/wsr.h +++ b/include/linux/wsr.h @@ -26,6 +26,8 @@ struct ws_bin { struct wsr { /* protects bins */ struct mutex bins_lock; + unsigned long timestamp; + unsigned long refresh_threshold; struct ws_bin bins[MAX_NR_BINS]; }; @@ -40,6 +42,7 @@ ssize_t wsr_intervals_ms_parse(char *src, struct ws_bin *bins); /* * wsr->bins needs to be locked + * refreshes wsr based on the refresh threshold */ void wsr_refresh(struct wsr *wsr, struct mem_cgroup *root, struct pglist_data *pgdat); diff --git a/mm/internal.h b/mm/internal.h index 88dba0b11f663..ce4757e7f8277 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -186,6 +186,17 @@ int folio_isolate_lru(struct folio *folio); void putback_lru_page(struct page *page); void folio_putback_lru(struct folio *folio); extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason); +int get_swappiness(struct lruvec *lruvec, struct scan_control *sc); +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, bool can_swap, + bool force_scan); + +/* + * in mm/wsr.c + */ +void refresh_wsr(struct wsr *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat, struct scan_control *sc, + unsigned long refresh_threshold); /* * in mm/rmap.c: diff --git a/mm/vmscan.c b/mm/vmscan.c index 150e3cd70c65e..66c5df2a7f65b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3201,7 +3201,7 @@ static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) return &pgdat->__lruvec; } -static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) +int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) { struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4402,7 +4402,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) spin_unlock_irq(&lruvec->lru_lock); } -static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, bool can_swap, bool force_scan) { bool success; @@ -5900,6 +5900,41 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen); +/****************************************************************************** + * working set reporting + ******************************************************************************/ + +#ifdef CONFIG_WSR +void wsr_refresh(struct wsr *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + unsigned int flags; + struct scan_control sc = { + .may_writepage = true, + .may_unmap = true, + .may_swap = true, + .reclaim_idx = MAX_NR_ZONES - 1, + .gfp_mask = GFP_KERNEL, + }; + + lockdep_assert_held(&wsr->bins_lock); + + if (wsr->bins->idle_age != -1) { + unsigned long timestamp = READ_ONCE(wsr->timestamp); + unsigned long threshold = READ_ONCE(wsr->refresh_threshold); + + if (time_is_before_jiffies(timestamp + threshold)) { + set_task_reclaim_state(current, &sc.reclaim_state); + flags = memalloc_noreclaim_save(); + refresh_wsr(wsr, root, pgdat, &sc, threshold); + memalloc_noreclaim_restore(flags); + set_task_reclaim_state(current, NULL); + } + } +} + +#endif /* CONFIG_WSR */ + #else /* !CONFIG_LRU_GEN */ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) diff --git a/mm/wsr.c b/mm/wsr.c index 1e4c0ce69caf7..ee295d164461e 100644 --- a/mm/wsr.c +++ b/mm/wsr.c @@ -125,8 +125,9 @@ static void collect_wsr(struct wsr *wsr, const struct lruvec *lruvec) } } -static void refresh_wsr(struct wsr *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) +void refresh_wsr(struct wsr *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat, struct scan_control *sc, + unsigned long refresh_threshold) { struct ws_bin *bin; struct mem_cgroup *memcg; @@ -146,6 +147,24 @@ static void refresh_wsr(struct wsr *wsr, struct mem_cgroup *root, do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + bool can_swap = get_swappiness(lruvec, sc); + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + unsigned long min_seq[ANON_AND_FILE] = { + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_ANON]), + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_FILE]), + }; + + mem_cgroup_calculate_protection(root, memcg); + if (!mem_cgroup_below_min(root, memcg) && refresh_threshold && + min_seq[!can_swap] + MAX_NR_GENS - 1 > max_seq) { + int gen = lru_gen_from_seq(max_seq); + unsigned long birth = + READ_ONCE(lruvec->lrugen.timestamps[gen]); + + if (time_is_before_jiffies(birth + refresh_threshold)) + try_to_inc_max_seq(lruvec, max_seq, sc, + can_swap, false); + } collect_wsr(wsr, lruvec); @@ -165,6 +184,32 @@ static struct wsr *kobj_to_wsr(struct kobject *kobj) return lruvec_wsr(mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))); } + +static ssize_t refresh_ms_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct wsr *wsr = kobj_to_wsr(kobj); + unsigned long threshold = READ_ONCE(wsr->refresh_threshold); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(threshold)); +} + +static ssize_t refresh_ms_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int msecs; + struct wsr *wsr = kobj_to_wsr(kobj); + + if (kstrtouint(buf, 0, &msecs)) + return -EINVAL; + + WRITE_ONCE(wsr->refresh_threshold, msecs_to_jiffies(msecs)); + + return len; +} + +static struct kobj_attribute refresh_ms_attr = __ATTR_RW(refresh_ms); + static ssize_t intervals_ms_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -227,7 +272,7 @@ static ssize_t histogram_show(struct kobject *kobj, struct kobj_attribute *attr, mutex_lock(&wsr->bins_lock); - refresh_wsr(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh(wsr, NULL, kobj_to_pgdat(kobj)); for (bin = wsr->bins; bin->idle_age != -1; bin++) len += sysfs_emit_at(buf, len, "%u anon=%lu file=%lu\n", @@ -245,6 +290,7 @@ static ssize_t histogram_show(struct kobject *kobj, struct kobj_attribute *attr, static struct kobj_attribute histogram_attr = __ATTR_RO(histogram); static struct attribute *wsr_attrs[] = { + &refresh_ms_attr.attr, &intervals_ms_attr.attr, &histogram_attr.attr, NULL -- 2.41.0.162.gfafddb0af9-goog