From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8BCFC83F1A for ; Fri, 11 Jul 2025 15:51:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CAA06B00A1; Fri, 11 Jul 2025 11:51:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77B996B00A3; Fri, 11 Jul 2025 11:51:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6917F6B00A4; Fri, 11 Jul 2025 11:51:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 599346B00A1 for ; Fri, 11 Jul 2025 11:51:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7F5AF8099E for ; Fri, 11 Jul 2025 15:51:31 +0000 (UTC) X-FDA: 83652423582.02.B7CEE69 Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) by imf25.hostedemail.com (Postfix) with ESMTP id B5C69A000D for ; Fri, 11 Jul 2025 15:51:29 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DuJCnggh; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752249090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=vPqNoVEMPHsyv6LMrHTUjHJt39CUZ7GoudiGKQX0R+8=; b=0yB8aLZExLuM90YRLg5osgz3xyifJpOCoh3k9+1trlMMb+2GWbsH+jwxIIkVx/tmNETYj0 U+USqEPyk8c4VHtlBjU0ZmDbKB3ayKlLafAMuXkWuPZJnBk6UeMEa2LU8MCDnKKtcsvewp tg03gmaX+8sUuTcvtG8unq2RRFuCUTU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DuJCnggh; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752249090; a=rsa-sha256; cv=none; b=e9cxtQI1vtFvB5yeeOyq09Jg1knI8syYMMS69FXlFgMHX19AsV3u7/l85IXSHyxIXrrGAe 5xTctUMu4ylmd1TShxqCX9o/dobhcSNt/6rRXYcphrfms6e454OC5pnzSA6B8rvZonoVrW vaY3sVtC2XnT2da+2cOENUpeBg6Ubh8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1752249087; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=vPqNoVEMPHsyv6LMrHTUjHJt39CUZ7GoudiGKQX0R+8=; b=DuJCngghyOuTGnjJgaqwiJyoGxHbZfB41026zWXq6K8PBseQzUFN9BeVxm8yp+avaiBv64 tcA+F0irPhBkZYEVlSzxcMXi9aESY6uGfCq/mO6G/iQmdW3vrf4v9WbYZ6Zl+YT8PzqsuC vfJmUXSDsqTVMKITXBm7/1nyg2f1TXM= From: Roman Gushchin To: Andrew Morton Cc: Johannes Weiner , Shakeel Butt , Lorenzo Stoakes , Michal Hocko , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Roman Gushchin Subject: [PATCH] mm: skip lru_note_cost() when scanning only file or anon Date: Fri, 11 Jul 2025 08:50:44 -0700 Message-ID: <20250711155044.137652-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B5C69A000D X-Stat-Signature: nnaz7p4qb513rxbhfxqmzqdntehq5nqk X-Rspam-User: X-HE-Tag: 1752249089-810635 X-HE-Meta: U2FsdGVkX18aJPGXS2eiJqTQaDHr3o12PCS8qEDk7OhsVQmhBdWImER/oecaiFPPi9I1qsI0eGnFudP2JOuVkT0Co2kAbbOeup3Rr3TrSkjyUq7o/iiANmVJttpp+9pf+k0dhgMRiD2ZR+KIfEpkTHpjacVZtkVthCunbr0OS1Rzv2TveRXo3A9iJonIWFfwF4Qltb4fEikgknPA621TxgbksbMNbO5Vjnfkh82aRcCuGojlhRHrPWy+O7AF2L3LPI0MGAKhMnkkMH+5msgrUMQx/dMSu6CA1bIAwa8YxD0IJuV8BVPpzFAwtQm9DM8HMtJLKGHLbwBjmT+3VdY9bVIeN5okzQoawvIlFaERTATOQ0S+77sVCSH9GrvSYbH+ItbLHAUhAWrqtDS5DFdPjZxtnvC8AiGBy5Zm5L4CGJf+2KIYD2slcZhVNt6P4b3yh5/5tZXWDXahe4RIv7DATNwQ1zj3TH1AulMviG3DxBdPI86BevjQMKv8X+seVWtbTDUI25ceLArJ35xgqnmXsWvsxuVrTH+l4C2t/3F3UHy1MSftF28DqDkvyw/vzXOgt/HSptsE3gJ+CoWHqFP3D4B7mC5W19h+9ypZpY/I+o3vIz+TWB+WYgmtb4rKUlHf85Fpcx+98jbzGPBhOeK3BofK9TPzA0BQLCFQQIs9r9rfpEhplJf9vauQOpyAPNaaY2wsPFdPWz8t+RAhRRGhRiLCA2aUktgOQJ8daThLuHwn5bvivMOnHY21eJtoAoIctTzwdKXqz677VgqxU4VAn41xOR0nKeUp63mqk+WhyeH2bsbG2mwUu9FOahTzSPOuWWjG0l+75DfByckhpo5crHUYqt3aSpNQDrhcTCJRiSxn7EWDzCyTpoaH1HTGgkL4UO/isYnCt1CRw9E+/m+/6k5SkyXWcnOiguqkAFOwmdeizLSx5ZpPobBVRgc158WcWGuGByVbxOxGZLBVRMk ocJbCiqo eNmRB1m5DlT6vQwX6jofx8eXRQEX2wOPoAbZKqePYFMzIASmOn9ubQGMi7TUctrnRUTmNVqzBQjMMqfL1e5ieMGa3zp6GP0EFBZIrOvnxyU0zZ+c7wcs/sNB9L9FQ/77x7XeB7dbvu4M0APXnL9mrjgwctUE1gMYS5um+sYvHFzGr0TmWusRbI1gwIxz7+UXvt4aXfAjkJ9zIf/uZpgGqQ7yUCyQy6ZzO60eAnTtEhD8+sV9FF2wP5pN5Oju2dYHczaU+wXVf5bbz3/lcUhgQlYi2/1N78vt5uxpwlsI3L8prU8+5us9gOFv1ZfUvpAe2ttpTSuh8WbbtpA8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: lru_note_cost() records relative cost of incurring io and cpu spent on lru rotations, which is used to balance the pressure on file and anon memory. The applied pressure is inversely proportional to the recorded cost of reclaiming, but only within 2/3 of the range (swappiness aside). This is useful when both anon and file memory is reclaimable, however in many cases it's not the case: e.g. there might be no swap, proactive reclaim can target anon memory specifically, the memory pressure can come from cgroup v1's memsw limit, etc. In all these cases recording the cost will only bias all following reclaim, also potentially outside of the scope of the original memcg. So it's better to not record the cost if it comes from the initially biased reclaim. lru_note_cost() is a relatively expensive function, which traverses the memcg tree up to the root and takes the lruvec lock on each level. Overall it's responsible for about 50% of cycles spent on lruvec lock, which might be a non-trivial number overall under heavy memory pressure. So optimizing out a large number of lru_note_cost() calls is also beneficial from the performance perspective. Signed-off-by: Roman Gushchin --- mm/vmscan.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c86a2495138a..33fd67eab0c9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -71,6 +71,13 @@ #define CREATE_TRACE_POINTS #include +enum scan_balance { + SCAN_EQUAL, + SCAN_FRACT, + SCAN_ANON, + SCAN_FILE, +}; + struct scan_control { /* How many pages shrink_list() should reclaim */ unsigned long nr_to_reclaim; @@ -90,6 +97,7 @@ struct scan_control { /* * Scan pressure balancing between anon and file LRUs */ + enum scan_balance scan_balance; unsigned long anon_cost; unsigned long file_cost; @@ -1988,6 +1996,17 @@ static int current_may_throttle(void) return !(current->flags & PF_LOCAL_THROTTLE); } +static bool scan_balance_biased(struct scan_control *sc) +{ + switch (sc->scan_balance) { + case SCAN_EQUAL: + case SCAN_FRACT: + return false; + default: + return true; + } +} + /* * shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages @@ -2054,7 +2073,9 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); spin_unlock_irq(&lruvec->lru_lock); - lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); + if (!scan_balance_biased(&sc)) + lru_note_cost(lruvec, file, stat.nr_pageout, + nr_scanned - nr_reclaimed); /* * If dirty folios are scanned that are not queued for IO, it @@ -2202,7 +2223,7 @@ static void shrink_active_list(unsigned long nr_to_scan, __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); spin_unlock_irq(&lruvec->lru_lock); - if (nr_rotated) + if (nr_rotated && !scan_balance_biased(&sc)) lru_note_cost(lruvec, file, 0, nr_rotated); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); @@ -2327,13 +2348,6 @@ static bool inactive_is_low(struct lruvec *lruvec, enum lru_list inactive_lru) return inactive * inactive_ratio < active; } -enum scan_balance { - SCAN_EQUAL, - SCAN_FRACT, - SCAN_ANON, - SCAN_FILE, -}; - static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) { unsigned long file; @@ -2613,6 +2627,8 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, calculate_pressure_balance(sc, swappiness, fraction, &denominator); out: + sc->scan_balance = scan_balance; + for_each_evictable_lru(lru) { bool file = is_file_lru(lru); unsigned long lruvec_size; -- 2.50.0