From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6706C3DA6E for ; Mon, 25 Dec 2023 07:28:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C52B8E0005; Mon, 25 Dec 2023 02:28:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 373E98E0001; Mon, 25 Dec 2023 02:28:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2149A8E0005; Mon, 25 Dec 2023 02:28:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0F5858E0001 for ; Mon, 25 Dec 2023 02:28:56 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C64778036B for ; Mon, 25 Dec 2023 07:28:55 +0000 (UTC) X-FDA: 81604513830.08.36CE57E Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf29.hostedemail.com (Postfix) with ESMTP id 0918D120006 for ; Mon, 25 Dec 2023 07:28:53 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KtT2V5qf; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703489334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ajjp7l7jDf3axBjq+hs31OFx0lpw7BRNQglDbs1yasM=; b=0h/DoH9awzBn2AMYVIuL8HntGWzHetdGgnz4j79Y7eCxZkz7F6ny0hMQUkZNSGwMGFmdNj xppK+3aQl1+ML2OHeJErSxVo0G5h/oPW13OyfHLpWvYwo3R6yoXqmYAKC4PWLt+RzAzoL9 9OV9MBklGgYw0+TgQwXa3WYj6wqiZU0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703489334; a=rsa-sha256; cv=none; b=mM/1zjc0OQ7SotFcFl48R+4J1ZRpVhA0O6sqaMzMYP44FyY4gSR2ekbE3OfHlCuN7Ixyaa jDUZaKIuqoD17bfWadaAFmHwrX2BkJFz5TLjCKjo1US3+LtPm7k3Oke0PgJMbWD3wgEr3g iLvlIxKVSzEAFtKYwTJV/0U5HlXhVbw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KtT2V5qf; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-40d3102d5d6so144215e9.0 for ; Sun, 24 Dec 2023 23:28:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1703489332; x=1704094132; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Ajjp7l7jDf3axBjq+hs31OFx0lpw7BRNQglDbs1yasM=; b=KtT2V5qfUkc3neNs+0iX2fcdfoYcY2LYsQ+YFF6m9u3LlpLNTLqHLbL7PekQUXpe03 n+DPAoDDnIy+Ys20tveSiOubLo+FHo4sITHJUA9EcrDWGzyEoVIkHSSfI8et1vxvpghl zSTFk9WpTHHkNNyz3BwWW5zB2Cx7bShU2WV3KSZ8uPz8wRSqQTeuyXrc9YDeelP2A1BA YMZHG2KMMwrkTDbnFMh1iXI1xEIql+58yz5GxFzIVJ+iqRka5P8/rKQE2seSjY+7RjZq ugZdLPwsE8ElaagRf/mLnuyzdyNR5VoWzO8vYtA+IfiuLv2Hvvv/RlXrr0Z7toftIflD QVFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703489332; x=1704094132; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ajjp7l7jDf3axBjq+hs31OFx0lpw7BRNQglDbs1yasM=; b=mX5kJF0gz1jLOlgPM+wokcp8i60nyPTXGlNKpkOncGAw9/4lSjSIMlBxZE1WyLZC95 LODKUmeaijew8R/QsZBGsizF1VpWzxpnuBAoxzAkhtaJKtGbAxR8eNyMdac7Jg6dCyMS c8xsMAcO2Lz4iAS2wQ9ANZhBjaT4FkvHr9nlkYRPcgTyawJqtezbKsL4fPSt+8npcStd iy4k6cm5B1aFW9EFCl/yLXt6np3HW63Yyv+XzSAc0vAzFB800atVG+RTfYJXHgYnV6Y4 WrHvxawZeikrkWQXhail3MjvjjsZCZMvBzg84oE5AtxrQcD/QRC/Hk9beb2F7SsyjHO3 Ul3Q== X-Gm-Message-State: AOJu0Ywf8A+jU5YBPztzvjGSDg5LA4HQ+OPuHSNT601DUjAAtXvt0kGU iecE4A9FaETv2FNPohw3oZZZmcM7h/53HHmRij63xcJiduk6zSMVjranl1LkKZnQFKo= X-Google-Smtp-Source: AGHT+IF1dnp4CPwUhRLefYQ1twjaGlaGeltP36VFyAHwaOCrUXuV5npLxVWr2JaZUFYqVDywk1e7/EmT/XeeZCf7mfs= X-Received: by 2002:a05:600c:744:b0:40d:2361:d0a2 with SMTP id j4-20020a05600c074400b0040d2361d0a2mr368118wmn.4.1703489332436; Sun, 24 Dec 2023 23:28:52 -0800 (PST) MIME-Version: 1.0 References: <20231222102255.56993-1-ryncsn@gmail.com> <20231222102255.56993-2-ryncsn@gmail.com> In-Reply-To: <20231222102255.56993-2-ryncsn@gmail.com> From: Yu Zhao Date: Mon, 25 Dec 2023 00:28:14 -0700 Message-ID: Subject: Re: [PATCH 1/3] mm, lru_gen: batch update counters on againg To: Kairui Song Cc: linux-mm@kvack.org, Andrew Morton , linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 0918D120006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: z5b7sxot6xd5db3gyprfoqztsu4eb6r8 X-HE-Tag: 1703489333-273520 X-HE-Meta: U2FsdGVkX18EH8W7U/VGQLoQFQdtoYq2eNS+QaEhft4KBFnRtNGtSHEisYaC5sL3B+RiC2RRmKFRZv2rqm74LY95B2W7RjcNHmiAZ56GpFJXDPR+4gud1tafAPPSwELFPj2xhR6Yp/Y54pqjR/L7HaKrsjXXYAls7kuwc6fDZeQ6hUC2sPHZ4Bd6rUpEsJoNLNcl88RwnGTKc5O/aGsiaBeFji8wijgYPL4tObScsaSxEpgge1fu9XWZxjhYUst3+rkMEX9z3R6iSvh/5mTtdM+xhXmwWRXgxnpQ1ejIWKOjdOyd0/8xaVPTBWeyY0vPz+VnDjtnuS39kFpOfIsjeizYnU8RRLg3LG6dnGlwuwW0ZuuFZYxoH/zVqRRWYHYGkRXbXBZrWnz7ty8VeJkAyEZrbWmJ3RaQuYVvC9QlcgK2eX5G/IgnxzVXGDaSK3DsfzzZL4aK+lQUxw1gy0tjSXDZ50bGpFCuWvfvn+sETYGKCR8UXucjTYm+f3I/F6MmCnd4xBYeL+ccxvBgd19ZEt/d2hqDsafYH4ybOJ0vXTYg1PX61f7rx1Ch6YiR5WlcGI7xRo4QE1Ai3SV/XUp6ZHW3crp5B7O0EBMzaCZJYMdSUhku2zmYBcHbTh56KI6WIIb8e/PmDilvS2ejc05UwJG/y72omiHz97vUYCGWKvMLZgIz8lAlJRXkMXapsPmdVGcJPKSc/mXNjXRgYxJDitKzHIvYiyqsfInk57UpXl54TJzVnXcO1lKQU0VrrsZ3ACbiQGPlNBfFQluqZc97XZMIUkUhsQp/MCRMbJcxeP3jDprOwfl4o4NAFNEjcv/vgpba9IrA1Q86S7K2rt9BJ1uwxFJj1h9D1qFlONZDkmZjiFLrACWoU8xHoQ8Li+oTw1oM3Tdt5KcCnwHOCzZbET7SdrqGKCe/g184dmvUn11xd7HLENJ6HXSSTJafILC3BVuknPqGFDL9k8Cc5DP Il4H7FuW GTLyC7ZbOnGD/2/aAGIYpxalv/MfkIaGFBlSa+KN4/MDB4dIRMn65c+C1D6i4xZ4YbnKdDD0CvDrtBt3uDIkwXDO6X9vehCluE1yhgMVj0+XewFaRLJ3laR8owEKPB6WHfiOjy5tPHAn2ARgyuDxutS91mo8xGIvj1PFoim9IuzF8iYqZrGQjKzGAdYjbb1+TsGyq2ICBfwaJrrMqhC9cOljvtZFlEh1JMZ4LJFA7jHXJqdXjs/hx69/OBi9bnA9VQVzaQFTwJunsa41Q2YXJkzERF2/0YG7MaUG9ca0MbqRRN2z9r/tCH0tWrUJ/ykvEhgX4Ad5YNWPaZfo5L8NONgXvKBRluII6EYuK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000809, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Dec 22, 2023 at 3:24=E2=80=AFAM Kairui Song wrot= e: > > From: Kairui Song > > When lru_gen is aging, it will update mm counters page by page, > which causes a higher overhead if age happens frequently or there > are a lot of pages in one generation getting moved. > Optimize this by doing the counter update in batch. > > Although most __mod_*_state has its own caches the overhead > is still observable. > > Tested in a 4G memcg on a EPYC 7K62 with: > > memcached -u nobody -m 16384 -s /tmp/memcached.socket \ > -a 0766 -t 16 -B binary & > > memtier_benchmark -S /tmp/memcached.socket \ > -P memcache_binary -n allkeys \ > --key-minimum=3D1 --key-maximum=3D16000000 -d 1024 \ > --ratio=3D1:0 --key-pattern=3DP:P -c 2 -t 16 --pipeline 8 -x 6 > > Average result of 18 test runs: > > Before: 44017.78 Ops/sec > After: 44687.08 Ops/sec (+1.5%) > > Signed-off-by: Kairui Song > --- > mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 55 insertions(+), 9 deletions(-) Usually most reclaim activity happens in kswapd, e.g., from the MongoDB benchmark (--duration=3D900): pgscan_kswapd 11294317 pgscan_direct 128 And kswapd always has current->reclaim_state->mm_walk. So the following should bring the vast majority of the improvement (assuming it's not noise) with far less code change: diff --git a/mm/vmscan.c b/mm/vmscan.c index 9dd8977de5a2..c06e00635d2b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3095,6 +3095,8 @@ static int folio_update_gen(struct folio *folio, int = gen) static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { int type =3D folio_is_file_lru(folio); + struct lru_gen_mm_walk *walk =3D current->reclaim_state->mm_walk; + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags); @@ -3116,7 +3118,10 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai new_flags |=3D BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + if (walk) + update_batch_size(walk, folio, old_gen, new_gen); + else + lru_gen_update_size(lruvec, folio, old_gen, new_gen); return new_gen; } @@ -3739,6 +3744,8 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) int prev, next; int type, zone; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + struct lru_gen_mm_walk *walk =3D current->reclaim_state->mm_walk; + restart: spin_lock_irq(&lruvec->lru_lock); @@ -3758,6 +3765,9 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) goto restart; } + if (walk && walk->batched) + reset_batch_size(lruvec, walk); + /* * Update the active/inactive LRU sizes for compatibility. Both sid= es of * the current max_seq need to be covered, since max_seq+1 can over= lap