From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B85AC38147 for ; Wed, 18 Jan 2023 00:18:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEC656B0075; Tue, 17 Jan 2023 19:18:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E284E6B0078; Tue, 17 Jan 2023 19:18:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC8356B007B; Tue, 17 Jan 2023 19:18:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B96C76B0075 for ; Tue, 17 Jan 2023 19:18:46 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 78FCA1404AA for ; Wed, 18 Jan 2023 00:18:46 +0000 (UTC) X-FDA: 80366009052.26.3EDF745 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id BD5F540012 for ; Wed, 18 Jan 2023 00:18:44 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="EX4UN/0e"; spf=pass (imf12.hostedemail.com: domain of 34zrHYwgKCJoN4FOG54OAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--talumbau.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34zrHYwgKCJoN4FOG54OAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--talumbau.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674001124; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WTRJVqBDkGUHWO7WMmajNVElkfX34x7nOoLfAt9+bEQ=; b=NGrJI1vuUNl/My827G8sqXUZugqItbuVbs03dZ9Kkn32IhkUKsnqTLW79Hea3inYVeaI6p Rye1mMrSq+R15HVQQ49tu2gtp8JyGMCCNSfoZSgO04k5qpSAtylX1Q46Cfj2YJHGnzsSIT Gxpbwr+wM1H2kETQtQ7YPjoLXbXIHW4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="EX4UN/0e"; spf=pass (imf12.hostedemail.com: domain of 34zrHYwgKCJoN4FOG54OAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--talumbau.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34zrHYwgKCJoN4FOG54OAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--talumbau.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674001124; a=rsa-sha256; cv=none; b=gbKb/krDeALRSPx1qLxR2he13m8RVf4RUlmB9KRuI0yNt+M27mp61BT+TmHNhSIz6QfPmZ 2b5ObGWYs2+C41C1nwiIjUrr6+9eGLszu+puztQRysmNBfgsCDVUe7ndEC6U7PEjyHNcOG A8tk9MEcnjaGS6cibdO5QErEkYG+f34= Received: by mail-yb1-f201.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so32259734ybr.11 for ; Tue, 17 Jan 2023 16:18:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WTRJVqBDkGUHWO7WMmajNVElkfX34x7nOoLfAt9+bEQ=; b=EX4UN/0eFUWIexqbQBRbpsGvA8KNCkS9OA2NXANeTD6ptgjqai35FyeOaIlNA+FBeT Cnm1APiNeCIgthHh2iELTziFYadXwmKI+My466IFRjTf/8YHjX/qU076qk38a1evmlWx 9cOBrA43RLYgGR0khB9BTTTnuGuKySesuafSPYRMecz6/nmWTUyMMYSArHGqmlT0Lqvp 18oLpRY+JKlwBTFpzRAjjAaW5oIyD/R7ryPOJioSA9lpHAZcpz2CvjjALChgEZLHXOM7 SoqMJT034sFZS2SsfMRRiWjeQ3w338ooV2rbafg3XxSq36dYpf7Y7Qlv8xp/0Z4MLaY/ 8dKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WTRJVqBDkGUHWO7WMmajNVElkfX34x7nOoLfAt9+bEQ=; b=RUSN3F4sbxUY0iehX09WMBkP0m6pDjjyC/6Eme0JoWH9kUN/h9uMrQjRHWPrHYQMBu mLQvQ3gO9S6a5orcbg/2QALS8e+FVvGjJHTKaLVs1bO5jFBEQ/68W7heWPWF51Qc3xs9 QqyRsKMEmRni1V4+w57EQZRNfrJRa6gBS33zL16koZiQ5I9CoXjNcgxCy8zjEjjekUJT /NkO8P/528Ukha/ffPT/sVth/+cDnVujSIN1jPj487jP6HFz4p54QMAl4NO8NCI/Bjh8 p/HHcBLwV4Gt1YPrwavyXqQJ570HtLKbFXb9UCxpQ31+Xph4988h3FitnWePsuwV07QH D70Q== X-Gm-Message-State: AFqh2kqiyVruasb38L3gdh138rCGbOxJi4H46GQ0LdXaPqQ8wrZvANuV kIbiprOlgq91AngY3esfx6Zgsa43OQsUew== X-Google-Smtp-Source: AMrXdXtD7bqi98Yh7h2wUwVBLNMYi2DQaFvYeFfYECMq032GE0MLbvHBZdI0aQprknrCs8TdSBeXA1M9DG39IQ== X-Received: from talumbau.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:90d]) (user=talumbau job=sendgmr) by 2002:a25:6089:0:b0:7b4:6882:d442 with SMTP id u131-20020a256089000000b007b46882d442mr579253ybb.429.1674001123956; Tue, 17 Jan 2023 16:18:43 -0800 (PST) Date: Wed, 18 Jan 2023 00:18:22 +0000 In-Reply-To: <20230118001827.1040870-1-talumbau@google.com> Mime-Version: 1.0 References: <20230118001827.1040870-1-talumbau@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230118001827.1040870-3-talumbau@google.com> Subject: [PATCH mm-unstable v1 2/7] mm: multi-gen LRU: section for rmap/PT walk feedback From: "T.J. Alumbaugh" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, "T.J. Alumbaugh" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BD5F540012 X-Stat-Signature: 45j1itcxr6e3dn68cybd5mm1ag3qxtgs X-Rspam-User: X-HE-Tag: 1674001124-892978 X-HE-Meta: U2FsdGVkX1+h+zNsQerqYqIK/V9oemPtGnJmekjQchp8OUZcL7BNQoeQyUWGKgyIW9KwgU8kRvvAAJ+AhHdFX8bQHu0vA8oF922ZAI3oD4d78t15dok0PSHtGbJ64PPUUL2aCOuWFgSPzlbYpooXX3ycrPbFw1Y1XkVqYj/vZqLULBK8iCRi1aQJStEwbW9gIJ4b5VNI8RYPukPiQQAB4BA9qLmOClzbVB5G9onjZ+qKpjhyyo00phM4gHZYUXVUZoSGZ4+7rU25qujW+D86mw6rbnd24kEvFJfeTZ3xTUEUK1mAD7OFzndoYdXWN3H9Iq7paGSEfbaK7mfF5JT44gvI9o4o5GfcLa8IKjhAkqUP8i52DsFekYK+U/pDaNhE0T473ZAzh6aD5nCyyusdnN6euvQ/vR/EZIyUQ629oGP1rdROj9EJjKcTL+ztRVH5IJXxWi+bZP6E2B804JuS47oNLbmiDziOGsher9QxckLNlTOM6O3KF0KgFgnp3BMaRoOHeP6cA+1rXGqOeQXjXNoL/QkLywMv03puksyWB/NaWUEHCwQFZT1CrSj8CkeNOVLFuWG2ttXlHmzmZcblociI+4gt2kSRKpPv+ss+A/8eZsuo0Q7L+LlQwYZZ+umh3wJ4noakrmZ8dI5RaR7GyhJNIYbyCfFsD+VwUjVwqbjmiMEGBY6thK+uOt6oypNEUIaSq3r8KVYIoM0RGoxI9aJ93HCnRt51U6rzJ52AzFo+7DrATNnm8eMZUJEALiwV67lBDeexTNSBHOmf8y1Lp2lwEozL8thcxNUZcwel8Y5u7LBCvhd79kGi61MIFdd3LSQ0gdprZWoT5ZcVbEUHu28LvKZHbfHgqoMuByBjybYIL6q/RarHTztbdmr57DwIB2oOcvxGgh3F4PIhkdUymErhRMhcQwi3a3sPD4ikwwyX5QEmrDpOAZG/J/6egpLBjFhXWZZA21IFyXjAbHF hjdM1q7l Vd0rwB8/RB+/UXXioxZOH45Cw/idR+L+gCyreg2mPykJ2aMMVanKl2SRuKm22ZDnp1W5PfctGoqOwAHK1VAKg3PPQc/v5qX0812IR0tICSbSBDGe5BjSyYFXfRp8bw5lukPjbNd2iu8U02RVz4Sb78sEy1OGXo+SppDzdRU5ThDh7dDGDhx1o9GriWs+BGD40rC60VN0hlu/YKniliDdpHB1caa3h3E4tuMQv2aklBFo2j286Z33f2SW0o/SFTlahE+GM0OSBeF1pSM/Rc2bB6fZJbVx47oLmUW7jOKV/A30F4etZMR/VVVAcEhuXZ+/lplTMjWYKWXdSJ38FY8F07IIkf+zDqZGB/eSpLZ5Y9UBlMnvDejuifZOlIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000100, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a section for lru_gen_look_around() in the code and the design doc. Signed-off-by: T.J. Alumbaugh --- Documentation/mm/multigen_lru.rst | 14 ++++++++++++++ mm/vmscan.c | 4 ++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/mm/multigen_lru.rst b/Documentation/mm/multigen_lru.rst index 6e1483e70fdc..bd988a142bc2 100644 --- a/Documentation/mm/multigen_lru.rst +++ b/Documentation/mm/multigen_lru.rst @@ -156,6 +156,20 @@ This time-based approach has the following advantages: and memory sizes. 2. It is more reliable because it is directly wired to the OOM killer. +Rmap/PT walk feedback +--------------------- +Searching the rmap for PTEs mapping each page on an LRU list (to test +and clear the accessed bit) can be expensive because pages from +different VMAs (PA space) are not cache friendly to the rmap (VA +space). For workloads mostly using mapped pages, searching the rmap +can incur the highest CPU cost in the reclaim path. + +``lru_gen_look_around()`` exploits spatial locality to reduce the +trips into the rmap. It scans the adjacent PTEs of a young PTE and +promotes hot pages. If the scan was done cacheline efficiently, it +adds the PMD entry pointing to the PTE table to the Bloom filter. This +forms a feedback loop between the eviction and the aging. + Summary ------- The multi-gen LRU can be disassembled into the following parts: diff --git a/mm/vmscan.c b/mm/vmscan.c index a741765896b6..eb9263bf6806 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4569,6 +4569,10 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) } } +/****************************************************************************** + * rmap/PT walk feedback + ******************************************************************************/ + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If -- 2.39.0.314.g84b9a713c41-goog