From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED6DC3DA6E for ; Mon, 25 Dec 2023 06:58:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC2AD6B0081; Mon, 25 Dec 2023 01:58:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D73136B0082; Mon, 25 Dec 2023 01:58:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C39506B0083; Mon, 25 Dec 2023 01:58:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B3BF16B0081 for ; Mon, 25 Dec 2023 01:58:41 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8D6B4C034B for ; Mon, 25 Dec 2023 06:58:41 +0000 (UTC) X-FDA: 81604437642.22.B0A46D4 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf23.hostedemail.com (Postfix) with ESMTP id B0E6F140021 for ; Mon, 25 Dec 2023 06:58:38 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mBSEI+SI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703487518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0XpnMDmsH50s0lvPTiOZRWTMXCRnNPpXm9QclJ7i9l8=; b=4dB/6SUY6ggGaR0ym7iyXPNkfMYCjV05N1dnhKYpJPXStF+yy1uMbYLyf9d5nsR4Zo2bPf az9eQ2WGX9ha0dY/h9b6t3vuYyUm+xTd2jNPudCEP09q6pXu4WZaCUoSRAdq2/bIBZv0bj fU3Ifay1B3ep6jE25qb8jRU8ADEGDgg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mBSEI+SI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703487518; a=rsa-sha256; cv=none; b=MpU9+X7xRy8jf3nEeHMcnKHuFAViXOEBUNzJud9uGKEGa//oBDrql53zppzAJQI76/SQKx PL9SiFDuy+UQJ0bjYE33NxhOhtOMqGO16vsqdYOiuTfQOHOufwVeNpnnLDygRnWltave3F xAwtP2fUFRrzDkAZsJjXyc2dOP3OfkY= Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-40d3b2c8517so128065e9.1 for ; Sun, 24 Dec 2023 22:58:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1703487517; x=1704092317; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=0XpnMDmsH50s0lvPTiOZRWTMXCRnNPpXm9QclJ7i9l8=; b=mBSEI+SIz/dhJWBOHeFI7Hdf6NCPHtTj0reeFlQCCSEm2WDHNc5VNLRRSKV4bQNAoe D78qqd1cpiKe6QPl8CAn/7dFHXTkL/zkNkEUKHLd6pOTVGxKwx0AdFILiUbyiisEtv8H TWDkxmsziBWj9ZcOKUAvK8XL/c/CHtvWPcZT310I6JKfBcVCJJiFiDuCjovrNc/W2HnN KC6iMD9jmbvu8OlZqqFM8EkvlFx0XdsRVGq8lSbhFJute9O7+k4ixigVQcS0n0YUdypP anjgqslxK/iGWFyZcciXrgmh5I49KsvNitM95bxFaLaECiTeCiW0zCQx8ILkC/0WT9r5 twWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703487517; x=1704092317; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0XpnMDmsH50s0lvPTiOZRWTMXCRnNPpXm9QclJ7i9l8=; b=WXcqt32e6GX/vzFq/UilLph5uNRdnO4wYVmq9WUwqDW4760OKWuC8sdqWpB6KMU9n5 T9bieA5wwiPKu6URlzaUREaY+drqFbsQNgv2UQ8Gaz75/vaGnqbPyZQICDJi15wdPYvb OCp6O/RERHDt8HKYqymY7YP+x43UCpLoxXB4h2QYONK6CA2H2ChOQFTlfa2YSGM8R5aF r2eB/HSOcOg1WOP1hz/P/jQX4CW9ltkkgIblG8S6Jwt1Qs57Hs2tAhHY2As16mTCBVuo O2Twz85TiMndDhcd7AmNTc4aYVie+Pg5CJoaBNBpaWKjB+YjqoLdgkm/1V/1/VtZvY6K hiUw== X-Gm-Message-State: AOJu0YzhYe+Tdd/6vsocxXicSw2zwXuYS5E2KCk/mFb+IeNT5PfBixEd BtRa79l2D11t1nd/h8kWfEyoeyTBNyVgMX580l2VcK9eFxHS X-Google-Smtp-Source: AGHT+IHycYngDkHH5FBpbNfgus3DgvUjPd0CH9Lblf3/orXmT7bqv23O4ueiovKcuWCCutI9NeQ3EHMhEM8cGmDAPcw= X-Received: by 2002:a05:600c:229a:b0:40d:400c:1b1f with SMTP id 26-20020a05600c229a00b0040d400c1b1fmr341589wmf.5.1703487517174; Sun, 24 Dec 2023 22:58:37 -0800 (PST) MIME-Version: 1.0 References: <20231222102255.56993-1-ryncsn@gmail.com> <20231222102255.56993-3-ryncsn@gmail.com> In-Reply-To: <20231222102255.56993-3-ryncsn@gmail.com> From: Yu Zhao Date: Sun, 24 Dec 2023 23:58:00 -0700 Message-ID: Subject: Re: [PATCH 2/3] mm, lru_gen: move pages in bulk when aging To: Kairui Song Cc: linux-mm@kvack.org, Andrew Morton , linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B0E6F140021 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nm6hq1z67d8u9q896h3d5i8jh89zbd94 X-HE-Tag: 1703487518-584352 X-HE-Meta: U2FsdGVkX18X2TCfw9o7aD9CR8/g1FsoPO3ergGjGE283fPGLQeGiB3rWfHdk8ZGoj06Jt7vQGdCNxkdJVUKzzC3K7zRhqSabSq+9q/Wv8f2aZqCD3ZOJQ+oR9TT4lUWsq+JWNuH3R+wUF44Jg3lo59NgCsEKvHZzl6RlO264uEiIX0oP1uveF2ewIMFl4qLoFaz8Q4khXZpe2cxyfn+UyWszwSZFu9+PGkXdDtBh1iT+/5ZZOuYDa4s5BErS/rmJoMihISEYr5/ahpqCfaP5FP85K5jC/q6lnbg1QKLvR2zqnaPXwn+3bs1vSWMNyFsjRv8M0kIXPHlf3eBZmH36aPMibDTupH8nWce6dULwmfyYgqgeE0Hxb/RnVm8ZfEvg9T6nSfQ+nF0BHMgO3PdyHZ6IxdFUWeR1TSRS8LBMK1fXQrCKn7byAQrz7BjjX7y4iH0GG7yPgMaCsdzu8/Obcj5CZToPbJY+SSt/XTLAt2H06LtJA8hLIp7Uvacb7FSqz6KKTNgnfI64b0sfpehcYx1OhmOf8w2bCQW2MsXclpvOo8sEv2vgabUZRW4o+8zxvQbrA68pv5AJH+cmOraZPgcCXvG64PNxwl1jkbyQgjQry+4I83s4vYcbV9Vl4YHbXmrCKDm1Wmq8g7PZbkdb7MQTVSBWRslOoMF5Zhav2FbcLBmvpy91ZEb1+nM8s3caORqIh3VgEUiYZ1QpnXO3YPHe7sTohCBEcDCQ80MAQDH8EnEXFVtdvzL0JY8UD0gsGNjMcIe4APkH1PEJbQ1BVdJPG5Iu1sqq0nTKac+r+sa8Zh6pMKQLP4eLX/nGv0tft6yt2HF7qt1zCZSO49cai6ZbN0A2TC48B+0uz22ajtGnzzIBduWZoggGDoNbHxTQp3AKdxpD2hy2N11bBDOg5D+3JvKc7zSwONV2QFbRuwtztP3IV5t+WaEAekx91dMC6bjuoHu3ZjFErIHAAh Vqo9O0IP FpoTUG0kWn+0T5tyMi6EvGuu+yOqnWp3MfZuGHkrrcavtmzBZVLqS2TpTe/GrM8F+rero7S6gzylkQZRwos71ZQkjmgLjL/Ma66r5QlKMfox5r1ansY0o2wN00UHNAb0TPyMgp/xh9jhaw6wXmplXMrcNwywn8NymtlRE8iuw15mH3fPatW0szwvl8E+cLrmzXDsbVRFJ9/vipj1AJ8zTGKZwUmEwhuyzJEsqMS/y9sm+COqonJvhoYZ2bXqAzp2a98Okk1Xi0xbVUUNqDajTKwBtsKNcdnoRRsx9Jt885xgNdZTRDcmvwonuOFZY5vpq6S0w6IvP1bNVw5F2LDHtKVJOTqh67x3gPT9N X-Bogosity: Ham, tests=bogofilter, spamicity=0.092310, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Dec 22, 2023 at 3:24=E2=80=AFAM Kairui Song wrot= e: > > From: Kairui Song > > Another overhead of aging is page moving. Actually, in most cases, > pages are being moved to the same gen after folio_inc_gen is called, > especially the protected pages. So it's better to move them in bulk. > > This also has a good effect on LRU ordering. Currently when MGLRU > ages, it walks the LRU backward, and the protected pages are moved to > the tail of newer gen one by one, which reverses the order of pages in > LRU. Moving them in batches can help keep their order, only in a small > scope though due to the scan limit of MAX_LRU_BATCH pages. > > After this commit, we can see a performance gain: > > Tested in a 4G memcg on a EPYC 7K62 with: > > memcached -u nobody -m 16384 -s /tmp/memcached.socket \ > -a 0766 -t 16 -B binary & > > memtier_benchmark -S /tmp/memcached.socket \ > -P memcache_binary -n allkeys \ > --key-minimum=3D1 --key-maximum=3D16000000 -d 1024 \ > --ratio=3D1:0 --key-pattern=3DP:P -c 2 -t 16 --pipeline 8 -x 6 > > Average result of 18 test runs: > > Before: 44017.78 Ops/sec > After patch 1-2: 44810.01 Ops/sec (+1.8%) Was it tested with CONFIG_DEBUG_LIST=3Dy? Also, the (44810.01-44687.08)/44687.08=3D0.0027 improvement also sounded like a noise to me. > Signed-off-by: Kairui Song > --- > mm/vmscan.c | 84 ++++++++++++++++++++++++++++++++++++++++++++--------- > 1 file changed, 71 insertions(+), 13 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index e3b4797b9729..af1266129c1b 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -3102,9 +3102,46 @@ static int folio_update_gen(struct folio *folio, i= nt gen) > */ > struct gen_update_batch { > int delta[MAX_NR_GENS]; > + struct folio *head, *tail; > }; > > -static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int z= one, > +static void inline lru_gen_inc_bulk_finish(struct lru_gen_folio *lrugen, > + int bulk_gen, bool type, int z= one, > + struct gen_update_batch *batch= ) > +{ > + if (!batch->head) > + return; > + > + list_bulk_move_tail(&lrugen->folios[bulk_gen][type][zone], > + &batch->head->lru, > + &batch->tail->lru); > + > + batch->head =3D NULL; > +} > + > +/* > + * When aging, protected pages will go to the tail of the same higher > + * gen, so the can be moved in batches. Besides reduced overhead, this > + * also avoids changing their LRU order in a small scope. > + */ > +static void inline lru_gen_try_inc_bulk(struct lru_gen_folio *lrugen, st= ruct folio *folio, > + int bulk_gen, int gen, bool type,= int zone, > + struct gen_update_batch *batch) > +{ > + /* > + * If folio not moving to the bulk_gen, it's raced with promotion > + * so it need to go to the head of another LRU. > + */ > + if (bulk_gen !=3D gen) > + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); > + > + if (!batch->head) > + batch->tail =3D folio; > + > + batch->head =3D folio; > +} > + > +static void lru_gen_update_batch(struct lruvec *lruvec, int bulk_gen, bo= ol type, int zone, > struct gen_update_batch *batch) > { > int gen; > @@ -3112,6 +3149,8 @@ static void lru_gen_update_batch(struct lruvec *lru= vec, bool type, int zone, > struct lru_gen_folio *lrugen =3D &lruvec->lrugen; > enum lru_list lru =3D type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANO= N; > > + lru_gen_inc_bulk_finish(lrugen, bulk_gen, type, zone, batch); > + > for (gen =3D 0; gen < MAX_NR_GENS; gen++) { > int delta =3D batch->delta[gen]; > > @@ -3705,6 +3744,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, bool can_swap) > struct gen_update_batch batch =3D { }; > struct lru_gen_folio *lrugen =3D &lruvec->lrugen; > int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); > + int bulk_gen =3D (old_gen + 1) % MAX_NR_GENS; > > if (type =3D=3D LRU_GEN_ANON && !can_swap) > goto done; > @@ -3712,24 +3752,33 @@ static bool inc_min_seq(struct lruvec *lruvec, in= t type, bool can_swap) > /* prevent cold/hot inversion if force_scan is true */ > for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { > struct list_head *head =3D &lrugen->folios[old_gen][type]= [zone]; > + struct folio *prev =3D NULL; > > - while (!list_empty(head)) { > - struct folio *folio =3D lru_to_folio(head); > + if (!list_empty(head)) > + prev =3D lru_to_folio(head); > > + while (prev) { > + struct folio *folio =3D prev; > VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(foli= o), folio); > VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), f= olio); > VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != =3D type, folio); > VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D z= one, folio); > > + if (unlikely(list_is_first(&folio->lru, head))) > + prev =3D NULL; > + else > + prev =3D lru_to_folio(&folio->lru); > + > new_gen =3D folio_inc_gen(lruvec, folio, false, &= batch); > - list_move_tail(&folio->lru, &lrugen->folios[new_g= en][type][zone]); > + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new= _gen, type, zone, &batch); > > if (!--remaining) { > - lru_gen_update_batch(lruvec, type, zone, = &batch); > + lru_gen_update_batch(lruvec, bulk_gen, ty= pe, zone, &batch); > return false; > } > } > - lru_gen_update_batch(lruvec, type, zone, &batch); > + > + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch= ); > } > done: > reset_ctrl_pos(lruvec, type, true); > @@ -4240,7 +4289,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) > ***********************************************************************= *******/ > > static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struc= t scan_control *sc, > - int tier_idx, struct gen_update_batch *batch) > + int tier_idx, int bulk_gen, struct gen_update_batc= h *batch) > { > bool success; > int gen =3D folio_lru_gen(folio); > @@ -4283,7 +4332,7 @@ static bool sort_folio(struct lruvec *lruvec, struc= t folio *folio, struct scan_c > int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); > > gen =3D folio_inc_gen(lruvec, folio, false, batch); > - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zo= ne]); > + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, = zone, batch); > > WRITE_ONCE(lrugen->protected[hist][type][tier - 1], > lrugen->protected[hist][type][tier - 1] + delt= a); > @@ -4293,7 +4342,7 @@ static bool sort_folio(struct lruvec *lruvec, struc= t folio *folio, struct scan_c > /* ineligible */ > if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { > gen =3D folio_inc_gen(lruvec, folio, false, batch); > - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zo= ne]); > + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, = zone, batch); > return true; > } > > @@ -4367,11 +4416,16 @@ static int scan_folios(struct lruvec *lruvec, str= uct scan_control *sc, > LIST_HEAD(moved); > int skipped_zone =3D 0; > struct gen_update_batch batch =3D { }; > + int bulk_gen =3D (gen + 1) % MAX_NR_GENS; > int zone =3D (sc->reclaim_idx + i) % MAX_NR_ZONES; > struct list_head *head =3D &lrugen->folios[gen][type][zon= e]; > + struct folio *prev =3D NULL; > > - while (!list_empty(head)) { > - struct folio *folio =3D lru_to_folio(head); > + if (!list_empty(head)) > + prev =3D lru_to_folio(head); > + > + while (prev) { > + struct folio *folio =3D prev; > int delta =3D folio_nr_pages(folio); > > VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(foli= o), folio); > @@ -4380,8 +4434,12 @@ static int scan_folios(struct lruvec *lruvec, stru= ct scan_control *sc, > VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D z= one, folio); > > scanned +=3D delta; > + if (unlikely(list_is_first(&folio->lru, head))) > + prev =3D NULL; > + else > + prev =3D lru_to_folio(&folio->lru); > > - if (sort_folio(lruvec, folio, sc, tier, &batch)) > + if (sort_folio(lruvec, folio, sc, tier, bulk_gen,= &batch)) > sorted +=3D delta; > else if (isolate_folio(lruvec, folio, sc)) { > list_add(&folio->lru, list); > @@ -4401,7 +4459,7 @@ static int scan_folios(struct lruvec *lruvec, struc= t scan_control *sc, > skipped +=3D skipped_zone; > } > > - lru_gen_update_batch(lruvec, type, zone, &batch); > + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch= ); > > if (!remaining || isolated >=3D MIN_LRU_BATCH) > break; > -- > 2.43.0 >