From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB2A3C7EE23 for ; Tue, 23 May 2023 20:59:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 171326B0074; Tue, 23 May 2023 16:59:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12208900003; Tue, 23 May 2023 16:59:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00F79900002; Tue, 23 May 2023 16:59:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E4BC86B0074 for ; Tue, 23 May 2023 16:59:29 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A97FC80855 for ; Tue, 23 May 2023 20:59:29 +0000 (UTC) X-FDA: 80822735658.30.951846A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id E37E02000F for ; Tue, 23 May 2023 20:59:27 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=00x6wRrX; spf=pass (imf13.hostedemail.com: domain of 3LiltZAsKCF8H7IBPEPFKDEDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kaleshsingh.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3LiltZAsKCF8H7IBPEPFKDEDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684875567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=oqzY22DU8hsochaAkiR/zou3nxJkIdpTHuQYEZ4euM0=; b=WsJ0UdbVSxj+OL9r8bJLBnkMTMKNiSVw7O2YnzSDmN1fimL8IurLfh7U7q/Co5CEVj1NKQ lEFEpaoMDURyCddfxJSL39GzTWwBraQGPokTfl8KBwDXKTo1jgRH3z++yCmjPDmrxURBX7 yPkaMDGlvDmIUxPKcXJL3iamw/ABHMw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684875568; a=rsa-sha256; cv=none; b=jQCpVV+Jug5lqOAOKqzO1d5rn/4RTzcXC0fAGGVOkq3nG0orbx2Oklu1w7H4aLh9W05O+B JIhwY0gEZnUA3qcNab1bx36tf12tSywRkj/IfG5vr+7Ol/gU+XFYFMPRRgf7A5ID7WOwh0 uGgfF+qWSyYsf5d67+ThZ9YKZxEUxq8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=00x6wRrX; spf=pass (imf13.hostedemail.com: domain of 3LiltZAsKCF8H7IBPEPFKDEDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kaleshsingh.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3LiltZAsKCF8H7IBPEPFKDEDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kaleshsingh.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-ba8337ade1cso359679276.2 for ; Tue, 23 May 2023 13:59:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684875567; x=1687467567; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=oqzY22DU8hsochaAkiR/zou3nxJkIdpTHuQYEZ4euM0=; b=00x6wRrXLaQc4E3HL5TfDI5wMyAGjsMlESwMz+Ip4SzPcVjGJi/DmFI7yqcmjpIDXU pPCYsg34Ok7Qu3IWjFmttSP2QWAvyZvKR2As9PcQkIbMrWSB0Pqf43q9Fv05dgCWXA90 8L/zBV0+LK7j5OKIqtev3NQqSaoWx7+MDW/HMzhoLZWiADmByxFVuQRAUjGTQJ8yZ72u 07YFpJeOKKk+Y8azY6PMmooahKyJQF48a6TKVwhqgHWmHi5brLqbLRdX7yLsDCfoplvy gMa+0HxIHAn+ddNOkeS8dOuCqzS4SY/rdq3vouSURnGek+KmjC84DuvNK8RgrObbCYXG r2VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684875567; x=1687467567; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=oqzY22DU8hsochaAkiR/zou3nxJkIdpTHuQYEZ4euM0=; b=hoDhx6doBf27se11b98wYMwPwnwjPTSAZpPs1R6FQmVHdO5HT5dPem/DD0SeOfxgJE /J7MtIaiDkjBbjGdUmWDt5gygpyVfHhQ2W4P5b6moZO/L2utipcmrjT+y6gg5Qv7Q7i8 SH7+Gs0xds6iLu732mxN++P+aqQlW8LxsdDI9NywGqNOnnvA2mSkqM6hvrmn5gAdwFut CPRDf/CU/RelIX1E5d4XbCR29B5OyrxNFWCvdsv+/WZKVJSij5plM4ye7CZdAqCL/8Dh 1RN/iGObgUe4A1/NLJ+YUt+DcNpzLUGAMphggYKc8Ax3J7JhE5SNcXyD82OwGtSMNCpI b0Tg== X-Gm-Message-State: AC+VfDyuDpTRboLxhQ3yoTStT1n1nfyuyjm0/7OClUVh15cq9NLhANw/ Tna0L+7+mlD/CQ4vznbacq0658E5xeALoaw88g== X-Google-Smtp-Source: ACHHUZ6CXRzlSLLUEil2mIC1MEhVd9KbB1k8BRZTxoZezM54UzgFmX5aoe8mWHGRqKL8xAx7PpFCLEoULxy3kxBITA== X-Received: from kalesh.mtv.corp.google.com ([2620:15c:211:201:1c60:f27:f83c:ef25]) (user=kaleshsingh job=sendgmr) by 2002:a25:9786:0:b0:ba8:939b:3e04 with SMTP id i6-20020a259786000000b00ba8939b3e04mr6271389ybo.5.1684875566909; Tue, 23 May 2023 13:59:26 -0700 (PDT) Date: Tue, 23 May 2023 13:59:21 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230523205922.3852731-1-kaleshsingh@google.com> Subject: [PATCH] mm-unstable: Multi-gen LRU: Fix workingset accounting From: Kalesh Singh To: yuzhao@google.com, akpm@linux-foundation.org Cc: surenb@google.com, android-mm@google.com, kernel-team@android.com, Kalesh Singh , Charan Teja Kalla , Oleksandr Natalenko , "Jan Alexander Steffens (heftig)" , Brian Geffon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: m8x3kmka89ooboow98711mf5os3c6ni8 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: E37E02000F X-Rspam-User: X-HE-Tag: 1684875567-309282 X-HE-Meta: U2FsdGVkX1/ry0V9LNeAftve3YOFfFYUuloqOAvEulMqUVJQKB70mYIAOW8eO0/usmZf342n9MGjFD0qma/MagJk7N4z1RgLcR57tccMmgmNa4iuELapweVjQSD5AwQNXl0ZCe6UWLX6gEiwHNINs1C8/DNRKD+n+nt86c5aSPvvcUxNttqHTsTGv5JUwjKPSqLssaLohavA2C7KCa4tF8md5tJg6MMTfgP8CCLQXp9coHrZPiA99GIAdfEM0SuQ/fQbDCNo+KU23DFfmQ4kAWQTozaKqm+f/fIwlsLOhX/bh45HOXGuoJeclJ7BIaE4SbtFAVoNQI3SviwTzc/SQrZ0Kx/Pj6rESWYUcBFpnWFoOCjHyQph1niP8GWKfbPKp6UlL0ry3lRKe28VB6UlBV6L6QutEY0xfbkhh/e/Ro+5LAhk2BQgaUFgVo8Cwfy6wg05VsFQNxPLwUefdG3E5bdOSgd9eX1xZVnjsM1xyOMCTMhpXRJY5YIDsf54XiLpzREkNxEdNAn4LkHWyU8ezEnM3zrUxNGVDww8H3CT9nWsmI4bRUP1e3atNm4Kn19lO88yGw3Tuzs3dS8nelQr2cQtR+3ari4kDutU5JH9UrHZizVRMezvpuKjvuM5yjPcDMzJb9aUxy1fmBUEXiYKPOU2lea1SO2zLtx851Z4dcWjn0D48Cna6F3MRP2CCSh0z7J7AWl/Ga0i0oYkK0eIwJQ1ULaz7RSeSYLDHlh5QwOCYwi4hda0F3pLzsLf1SbwXORal5eu0rD6Tj4gd6SaFMMwzmm50w4YR0HDUmI5/xwm8W49lFzUezC60dp/ux7HZJVDh2ZSxXv/UiW4mcpO1G87w+TGdGDYGzYFJD84Z4DwA8hII2k+UrnKtLlVKSqTkgdJJiEwcl1IU8yC/ot0L2dbkR6zP8WnEaxKvrXpXRr3qhZfSKD2gItG3eZOlLyE+hwU6TYUag4MkCD2Ayy xcEn5/B+ SOzvZn/ro5LS8+UzxUL0m0JYdCqU1EQbsChxkL4y0OYKdCDhNZpHTih0EIvVKBTP4kkdmKpTvm65+VUGmXU6E8I+cWI6lq7i5sqDH/kvHhLXW/2MgjruCXc34nH2JcXTXjkzNO+IDiuULZxKbNkun6Sb+5YCUB+yag/S6nRkOzgfNTFONusMm7REtAVYrnFCw5S7j4dUj9sPi391QxwN5eLWDwL3iWShKizq7q5c1bMjiItSc7xMiup3quo8c9Aoe7K8MEdy/aKgXvBCXQi/j4xcZBEhnboWTrW8rlay1x6bRwK8iG0z/JfphtLmS+NPy6brFHvPD6xa7HPkzWCZHcR7FBh4ulR9S9nL2in4zAyW2EPkzXE6F+nsCB/5gXS8iIxKQGZEvfgzya7aAtczU5pIugrAwkRhqwJUivd2S9cqQaiPivQb/PPl+xlSqJc7VdOyjf1yb3ITOK2ajV+Ls0h5oKZa7pT8jIijKAJSe/yIT3QWCkc0wOPyHtD6V5y23rgbEm4asgXmW6dDBJPGxZqq8AjCEB5RrFFANlfLNjfIZiCpf7KqiNyQFoLWaLKrg9yVPsdYiJl8YvEevA3oWq9D6NEboKTiQJrisZQ/AZwx8QY3g2bvON/6QRQw1mFJLq2NX5Ih4skZQOMS2mw3jSZssmjleJENi37YkPetHoXgxo2mXe6qcFWy+uBXYWRBOhyADyu99A1FysU0eOiKpwRpOZLV8P8FxEoOHY0FuDkt9qiwqWrgAngBjAOcqrFsmFheltkikA4GkZ2c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Android app cycle workloads, MGLRU showed a significant reduction in workingset refaults although pgpgin/pswpin remained relatively unchanged. This indicated MGLRU may be undercounting workingset refaults. This has impact on userspace programs, like Android's LMKD, that monitor workingset refault statistics to detect thrashing. It was found that refaults were only accounted if the MGLRU shadow entry was for a recently evicted folio. However, recently evicted folios should be accounted as workingset activation, and refaults should be accounted regardless of recency. Fix MGLRU's workingset refault and activation accounting to more closely match that of the conventional active/inactive LRU. Fixes: ac35a4902374 ("mm: multi-gen LRU: minimal implementation") Cc: Yu Zhao Cc: Andrew Morton Reported-by: Charan Teja Kalla Signed-off-by: Kalesh Singh --- mm/vmscan.c | 1 - mm/workingset.c | 10 +++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index eeca83e28c9b..ef687f9be13c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4925,7 +4925,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); - __mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta); return true; } diff --git a/mm/workingset.c b/mm/workingset.c index 5796e927e6d7..4686ae363000 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -278,6 +278,7 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, static void lru_gen_refault(struct folio *folio, void *shadow) { + bool recent; int hist, tier, refs; bool workingset; unsigned long token; @@ -288,10 +289,13 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - if (!lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset)) + recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + if (lruvec != folio_lruvec(folio)) goto unlock; - if (lruvec != folio_lruvec(folio)) + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta); + + if (!recent) goto unlock; lrugen = &lruvec->lrugen; @@ -302,7 +306,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) tier = lru_tier_from_refs(refs); atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]); - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta); + mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta); /* * Count the following two cases as stalls: base-commit: cd255dae35f51ea33cb85d29720d6b4e5fa16365 -- 2.40.1.698.g37aff9b760-goog