From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.9 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25131C433B4 for ; Mon, 5 Apr 2021 18:09:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF13061246 for ; Mon, 5 Apr 2021 18:09:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF13061246 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 52F9E6B0081; Mon, 5 Apr 2021 14:09:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 491306B0082; Mon, 5 Apr 2021 14:09:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 311D86B0083; Mon, 5 Apr 2021 14:09:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0156.hostedemail.com [216.40.44.156]) by kanga.kvack.org (Postfix) with ESMTP id EF9D26B0081 for ; Mon, 5 Apr 2021 14:09:10 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B2E92C5D1 for ; Mon, 5 Apr 2021 18:09:10 +0000 (UTC) X-FDA: 77999100060.10.7CFB922 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf06.hostedemail.com (Postfix) with ESMTP id A8676C0007C2 for ; Mon, 5 Apr 2021 18:09:10 +0000 (UTC) IronPort-SDR: fQ7Mk4QAOQJONhaVPE52khR9FnmM7iOw0ttK7nyo/MiiEVFWAX49PKpPxleIab4C9UO1cPZFHP dpzsIKxKn4sQ== X-IronPort-AV: E=McAfee;i="6000,8403,9945"; a="172968217" X-IronPort-AV: E=Sophos;i="5.81,307,1610438400"; d="scan'208";a="172968217" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2021 11:09:09 -0700 IronPort-SDR: zciwvXhVlJxANCS1RQuETFmvpI/SVAShGiAS1SOFk+fup6XS3eaAdNiHzq00W3PW/LzFQBiUp1 11xpUJFo26pQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,307,1610438400"; d="scan'208";a="448153935" Received: from skl-02.jf.intel.com ([10.54.74.28]) by fmsmga002.fm.intel.com with ESMTP; 05 Apr 2021 11:09:08 -0700 From: Tim Chen To: Michal Hocko Cc: Tim Chen , Johannes Weiner , Andrew Morton , Dave Hansen , Ying Huang , Dan Williams , David Rientjes , Shakeel Butt , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v1 07/11] mm: Account the total top tier memory in use Date: Mon, 5 Apr 2021 10:08:31 -0700 Message-Id: <9170c90b0f58dee05a2b2c1d3789d674df42ed65.1617642417.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A8676C0007C2 X-Stat-Signature: 8pykzgmedzyqhgorcaki3egag64u6cge Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mga17.intel.com; client-ip=192.55.52.151 X-HE-DKIM-Result: none/none X-HE-Tag: 1617646150-318228 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Track the global top tier memory usage stats. They are used as the basis = of deciding when to start demoting pages from memory cgroups that have excee= ded their soft limit. We start reclaiming top tier memory when the total top tier memory is low. Signed-off-by: Tim Chen --- include/linux/vmstat.h | 18 ++++++++++++++++++ mm/vmstat.c | 20 +++++++++++++++++--- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index e1a4fa9abb3a..a3ad5a937fd8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -139,6 +139,7 @@ static inline void vm_events_fold_cpu(int cpu) * Zone and node-based page accounting with per cpu differentials. */ extern atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS]; +extern atomic_long_t vm_toptier_zone_stat[NR_VM_ZONE_STAT_ITEMS]; extern atomic_long_t vm_numa_stat[NR_VM_NUMA_STAT_ITEMS]; extern atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS]; =20 @@ -175,6 +176,8 @@ static inline void zone_page_state_add(long x, struct= zone *zone, { atomic_long_add(x, &zone->vm_stat[item]); atomic_long_add(x, &vm_zone_stat[item]); + if (node_state(zone->zone_pgdat->node_id, N_TOPTIER)) + atomic_long_add(x, &vm_toptier_zone_stat[item]); } =20 static inline void node_page_state_add(long x, struct pglist_data *pgdat= , @@ -212,6 +215,17 @@ static inline unsigned long global_node_page_state(e= num node_stat_item item) return global_node_page_state_pages(item); } =20 +static inline unsigned long global_toptier_zone_page_state(enum zone_sta= t_item item) +{ + long x =3D atomic_long_read(&vm_toptier_zone_stat[item]); + +#ifdef CONFIG_SMP + if (x < 0) + x =3D 0; +#endif + return x; +} + static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) { @@ -325,6 +339,8 @@ static inline void __inc_zone_state(struct zone *zone= , enum zone_stat_item item) { atomic_long_inc(&zone->vm_stat[item]); atomic_long_inc(&vm_zone_stat[item]); + if (node_state(zone->zone_pgdat->node_id, N_TOPTIER)) + atomic_long_inc(&vm_toptier_zone_stat[item]); } =20 static inline void __inc_node_state(struct pglist_data *pgdat, enum node= _stat_item item) @@ -337,6 +353,8 @@ static inline void __dec_zone_state(struct zone *zone= , enum zone_stat_item item) { atomic_long_dec(&zone->vm_stat[item]); atomic_long_dec(&vm_zone_stat[item]); + if (node_state(zone->zone_pgdat->node_id, N_TOPTIER)) + atomic_long_dec(&vm_toptier_zone_stat[item]); } =20 static inline void __dec_node_state(struct pglist_data *pgdat, enum node= _stat_item item) diff --git a/mm/vmstat.c b/mm/vmstat.c index f299d2e89acb..b59efbcaef4e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -161,9 +161,11 @@ void vm_events_fold_cpu(int cpu) * vm_stat contains the global counters */ atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in= _smp; +atomic_long_t vm_toptier_zone_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_al= igned_in_smp; atomic_long_t vm_numa_stat[NR_VM_NUMA_STAT_ITEMS] __cacheline_aligned_in= _smp; atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS] __cacheline_aligned_in= _smp; EXPORT_SYMBOL(vm_zone_stat); +EXPORT_SYMBOL(vm_toptier_zone_stat); EXPORT_SYMBOL(vm_numa_stat); EXPORT_SYMBOL(vm_node_stat); =20 @@ -695,7 +697,7 @@ EXPORT_SYMBOL(dec_node_page_state); * Returns the number of counters updated. */ #ifdef CONFIG_NUMA -static int fold_diff(int *zone_diff, int *numa_diff, int *node_diff) +static int fold_diff(int *zone_diff, int *numa_diff, int *node_diff, int= *toptier_diff) { int i; int changes =3D 0; @@ -717,6 +719,11 @@ static int fold_diff(int *zone_diff, int *numa_diff,= int *node_diff) atomic_long_add(node_diff[i], &vm_node_stat[i]); changes++; } + + for (i =3D 0; i < NR_VM_ZONE_STAT_ITEMS; i++) + if (toptier_diff[i]) { + atomic_long_add(toptier_diff[i], &vm_toptier_zone_stat[i]); + } return changes; } #else @@ -762,6 +769,7 @@ static int refresh_cpu_vm_stats(bool do_pagesets) struct zone *zone; int i; int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] =3D { 0, }; + int global_toptier_diff[NR_VM_ZONE_STAT_ITEMS] =3D { 0, }; #ifdef CONFIG_NUMA int global_numa_diff[NR_VM_NUMA_STAT_ITEMS] =3D { 0, }; #endif @@ -779,6 +787,9 @@ static int refresh_cpu_vm_stats(bool do_pagesets) =20 atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] +=3D v; + if (node_state(zone->zone_pgdat->node_id, N_TOPTIER)) { + global_toptier_diff[i] +=3Dv; + } #ifdef CONFIG_NUMA /* 3 seconds idle till flush */ __this_cpu_write(p->expire, 3); @@ -846,7 +857,7 @@ static int refresh_cpu_vm_stats(bool do_pagesets) =20 #ifdef CONFIG_NUMA changes +=3D fold_diff(global_zone_diff, global_numa_diff, - global_node_diff); + global_node_diff, global_toptier_diff); #else changes +=3D fold_diff(global_zone_diff, global_node_diff); #endif @@ -868,6 +879,7 @@ void cpu_vm_stats_fold(int cpu) int global_numa_diff[NR_VM_NUMA_STAT_ITEMS] =3D { 0, }; #endif int global_node_diff[NR_VM_NODE_STAT_ITEMS] =3D { 0, }; + int global_toptier_diff[NR_VM_NODE_STAT_ITEMS] =3D { 0, }; =20 for_each_populated_zone(zone) { struct per_cpu_pageset *p; @@ -910,11 +922,13 @@ void cpu_vm_stats_fold(int cpu) p->vm_node_stat_diff[i] =3D 0; atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] +=3D v; + if (node_state(pgdat->node_id, N_TOPTIER)) + global_toptier_diff[i] +=3Dv; } } =20 #ifdef CONFIG_NUMA - fold_diff(global_zone_diff, global_numa_diff, global_node_diff); + fold_diff(global_zone_diff, global_numa_diff, global_node_diff, global_= toptier_diff); #else fold_diff(global_zone_diff, global_node_diff); #endif --=20 2.20.1