From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89EAEC25B75 for ; Wed, 29 May 2024 22:58:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3E6C6B00A6; Wed, 29 May 2024 18:58:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEE0B6B00A8; Wed, 29 May 2024 18:58:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B40636B00A9; Wed, 29 May 2024 18:58:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 938CA6B00A6 for ; Wed, 29 May 2024 18:58:07 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 439F6A201D for ; Wed, 29 May 2024 22:58:07 +0000 (UTC) X-FDA: 82172948214.02.1DAC2D6 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 805F840010 for ; Wed, 29 May 2024 22:58:05 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ED11OW5R; spf=pass (imf04.hostedemail.com: domain of 3_LJXZgYKCPktfbokdhpphmf.dpnmjovy-nnlwbdl.psh@flex--seanjc.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_LJXZgYKCPktfbokdhpphmf.dpnmjovy-nnlwbdl.psh@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717023485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lXQ4aAmXpFR+Rnn+LVSp/P9w+zocv3IAjl6jcVlXHAQ=; b=eFT28CY72B1OuCh7RPoHJ8bWugfjefvh+VxavVw+NhfQWzbZBBopHp3JLF77E/Mb1ERS95 dbNB03pRIPP0N8exq/M1NVun5DJxoDvxzec75bXXgAO1twOH/yYXTI49mt3XFPfyOC0ZQC z7RK+QfNxyedSApaNTYZfwQCNXrM024= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ED11OW5R; spf=pass (imf04.hostedemail.com: domain of 3_LJXZgYKCPktfbokdhpphmf.dpnmjovy-nnlwbdl.psh@flex--seanjc.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_LJXZgYKCPktfbokdhpphmf.dpnmjovy-nnlwbdl.psh@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717023485; a=rsa-sha256; cv=none; b=FwNMNe74uuhVz6o9vtR0MITWT/EaYn4vJyFq80I3ABvHR8rW7sbSdTMlczMUAOD3KE5zMx 3FRnwlh+VfFLA8fQzICgzLWYi/M2jgYZtSXXv/50qJf08SeYKvlAwtvj2AFrRXgEWxGYGL od0DOwpIdCA4r7ysJHILRK012CAdqXs= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-df4f32e9f46so374230276.0 for ; Wed, 29 May 2024 15:58:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717023484; x=1717628284; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=lXQ4aAmXpFR+Rnn+LVSp/P9w+zocv3IAjl6jcVlXHAQ=; b=ED11OW5RdYEr7qo8Us3af0zXbFl2iVkha+chQH5ZiOEAgMTBBodCc8ZJXJAPhw/0hJ ntTLm397W4y2Id8memeKefZrvq7Zkb4U5aN0rO12PIQwT/iWZWlIBuHP0fWelabXJr5n Mx+ZCOCKdsBiCRAQJ74GSvF4Wwe334r0zKSiD1BzNs16dm4qLcWiSfaiac62hLLxVC0z zg+fDAVRc+DUV6/ZjbyVtxZL+nn4wrjMkmcP3eM3BUAmZ8s6lySryYQN1mKz3qyZGosE MIkjiheUd50vbZVnAdppvLsTQ7lQVqkoai6alTKtzklDYmopQYjloJzPias8sQGv+StI ZMUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023484; x=1717628284; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=lXQ4aAmXpFR+Rnn+LVSp/P9w+zocv3IAjl6jcVlXHAQ=; b=pSN4VWk7I0XaDy1XpZ6e2Fsfxcz54a7t6PWdilq0f2JD+LEPuq7HmbPdNCFEwBUNjw NxNAUk4BWWl9OqCN+B324Sse8fxRL8RlxncJbYw1B0mCoP9qCdYc5adElVMR5DzLl4im lYowjgF5Ks3pDWO7J+jBOVUpssAS+bKUkjPsolsI5X7wsNUydEYWsstI0WlNvnH9thsQ 1Ggm8zf9I+qA62bEqdDso8hPcV/R7Ee3XwZMFky2K9Z3v52T8xtcA4UDm+CgJdsD3IUm D4IUMSkssOGycw1NWChGa6iJSkmxxTdrwGy5bx6/9jvOLXNcvhmpiw1q5b/Pl8wIpZ/t 2C/A== X-Forwarded-Encrypted: i=1; AJvYcCU2xRCmnMPmwPF2K6uV8SjqlUrlN/HxfuADvs/0ur5o4Z94AvGjdDtTe6Xz1Jut5mRnurax4hKzNLxkbnO0IgtJLPw= X-Gm-Message-State: AOJu0YyuV1SRXRhPrgE0OiRxHpefy1Cofkg0tG+NSrYViyStRXthYdtB cnkMDCVGMPFw7VQqUPw7ZmXV30SJDH18bgDgtKdPIf/g7NzhnNAoanGr4/4HVi0IlvYkuWbjcOq ioQ== X-Google-Smtp-Source: AGHT+IEcoxWAWGQljmQC1ObvPASoASZSWeSUzQr2Szq7dgxCnuAYyloQCwOCyj3B7twEp9K1BnNOHbqA4OU= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1007:b0:dfa:5838:b919 with SMTP id 3f1490d57ef6-dfa5a68828dmr142777276.10.1717023484293; Wed, 29 May 2024 15:58:04 -0700 (PDT) Date: Wed, 29 May 2024 15:58:02 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> Message-ID: Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging From: Sean Christopherson To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Oliver Upton , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: xip45usumo8as4rpx4tbc48wtqexm8ie X-Rspamd-Queue-Id: 805F840010 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717023485-389090 X-HE-Meta: U2FsdGVkX1+L50sGJgpxyF2aqveRjzfcbiG1YUWO1xNgrT11QxZbviAseT3P+GbjAUI24rfLKuS+CFFKGXciQHFz9fyaqpC16FMV2x8hRRw0W3A8/10sQgD8mmwNodwiU6ZsI3mWKuf2SOk590cuwO4oocjn0rZ4Zls2YeNy7JJok4DCBs0zRnHoZC+KHdET0zJtTcKrQyJgEtc2BEd1txFYbypvTNr/qZs2k71ghwGqma0kw84EZ7Un/DdI8VdU73a1ByjhSC6O7Tm+pKU8U1b8XCuqQahIHvZyFbW2ljy8qP5OTXfJn6mHeo1wyHjqDEOMYIV5cR1VSWQwFQ75fvFAQaDKJ0hv8JvwK8nSoH5hFrkliNt/FSDj0VKXK3kb1pwVnTCAahbAlUDlNQUySzDYMXJ74u3RKsLlMsHaIx1adSX1SHIBEkCf7SGLyscgBQLnmVh+kY0O3XeQlwIIe+/DcCxDs44bL4GUlDwYANgUJTnoUxI7wofgiSjYxmLCtdydB332crqrKvZxfxCttDrZx+Eyp3Tq71g0Y9timTlQSOp4RSNroKGjxDpJej5doCGPQMQ1j+5wC6E2yeb80ND7VGO62gT3d4+QG7NNX8ZAQZF3zuygD1MfRJbt0tlHEP+cRK224f+x27zo7HsJ8iCt+pIFB/DPoZZxoNzMwrhLrCq+O9YmF6blLnPUKGI+7TvmpODW9yS4bdQTFKi7djpqmwH2IznAvXTEkxk2JrSD1KgJ7Cd3hAahq5Go0M8SeSJzVKpQxXrV9BTvXJnfF1RHcHiZwTnNX5nAp7eYXVlCYpG/x9ZvgxzRMYxWWdTdjT/N2TTiM8Hm9hZAnxD58FcCp19Hbnr7PWgM08snylIZJwKPPOcmIbs4SLgsUxxNH9NoGobSwSsOkwpgNIxjFDLkm3izDtLfKSdk1p2gBar4GKuLZ3fSESy+7yW66MmoGQZvMU8SZ+P+/nCxosI Ndyu8UOz TcuK/XO1pXXHXK9P4sGk70/V/FRm8tV/h2WDFklFpZM20ORV9kD3XFkz5J/YJ8tO65G4KzdVUSPIT9jmki1US04DILsZ1CEWNbMvJ3r/x5VLeyvUgSURqxYj7woVjIOf4qKGzzdEXRy2k57Q5o9nrRihOPAaighBJQC4dPt5mIKjw8ZWGZCx+bQ0UI1KRC7JvL0HfFK5kk61mMt+mag7Q14s2GmN9l3wLS0z21HfJqB9m6Kpygr0TVoM28gwwO+XzY7fj1bUXqSX2euzTvAj8BLhVMLAyIkE8o4ImlMd8ftyeN6/tmdSozvtrnJky+uadh7ErtEKdOCR4sfKKbzFvNPhq14TYNOpsOQ2Q4L6Yr1qA2xvcUbzNXyHmfnStA/3vafvnAR650HZu6b5enyRiQLhb/gGdTrBQQhLBqMTD6604QHFAAN3joFBywxwSY+YLHHJIjp3hNjTGvWAizkKYr9dfq1sS9ra+DZKq/FgQi1i+JZuVEO9Ix6fW+EUh2+OIJiDfEuKoS0CLY9sDxPpmxUqiVnPH0Hj/+W25Ixs495RblO6cDaZe4mCCPCroG7EUiKtPZ/E2RYKmlfGdJsCmO4HhhefSUDCAW2ht9ZyO+BNIjTG3ymx4Z4sx3yIcE9WE/Wp5qc8qf33RRe8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, May 29, 2024, Yu Zhao wrote: > On Wed, May 29, 2024 at 3:59=E2=80=AFPM Sean Christopherson wrote: > > > > On Wed, May 29, 2024, Yu Zhao wrote: > > > On Wed, May 29, 2024 at 12:05=E2=80=AFPM James Houghton wrote: > > > > > > > > Secondary MMUs are currently consulted for access/age information a= t > > > > eviction time, but before then, we don't get accurate age informati= on. > > > > That is, pages that are mostly accessed through a secondary MMU (li= ke > > > > guest memory, used by KVM) will always just proceed down to the old= est > > > > generation, and then at eviction time, if KVM reports the page to b= e > > > > young, the page will be activated/promoted back to the youngest > > > > generation. > > > > > > Correct, and as I explained offline, this is the only reasonable > > > behavior if we can't locklessly walk secondary MMUs. > > > > > > Just for the record, the (crude) analogy I used was: > > > Imagine a large room with many bills ($1, $5, $10, ...) on the floor, > > > but you are only allowed to pick up 10 of them (and put them in your > > > pocket). A smart move would be to survey the room *first and then* > > > pick up the largest ones. But if you are carrying a 500 lbs backpack, > > > you would just want to pick up whichever that's in front of you rathe= r > > > than walk the entire room. > > > > > > MGLRU should only scan (or lookaround) secondary MMUs if it can be > > > done lockless. Otherwise, it should just fall back to the existing > > > approach, which existed in previous versions but is removed in this > > > version. > > > > IIUC, by "existing approach" you mean completely ignore secondary MMUs = that > > don't implement a lockless walk? >=20 > No, the existing approach only checks secondary MMUs for LRU folios, > i.e., those at the end of the LRU list. It might not find the best > candidates (the coldest ones) on the entire list, but it doesn't pay > as much for the locking. MGLRU can *optionally* scan MMUs (secondary > included) to find the best candidates, but it can only be a win if the > scanning incurs a relatively low overhead, e.g., done locklessly for > the secondary MMU. IOW, this is a balance between the cost of > reclaiming not-so-cold (warm) folios and that of finding the coldest > folios. Gotcha. I tend to agree with Yu, driving the behavior via a Kconfig may generate si= mpler _code_, but I think it increases the overall system complexity. E.g. distr= os will likely enable the Kconfig, and in my experience people using KVM with = a distro kernel usually aren't kernel experts, i.e. likely won't know that th= ere's even a decision to be made, let alone be able to make an informed decision. Having an mmu_notifier hook that is conditionally implemented doesn't seem = overly complex, e.g. even if there's a runtime aspect at play, it'd be easy enough= for KVM to nullify its mmu_notifier hook during initialization. The hardest pa= rt is likely going to be figuring out the threshold for how much overhead is too = much.