From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7986C4345F for ; Fri, 19 Apr 2024 21:07:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DD916B0095; Fri, 19 Apr 2024 17:07:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58D1B6B0096; Fri, 19 Apr 2024 17:07:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47DBD6B0098; Fri, 19 Apr 2024 17:07:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A8B76B0095 for ; Fri, 19 Apr 2024 17:07:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BFD88C159C for ; Fri, 19 Apr 2024 21:07:30 +0000 (UTC) X-FDA: 82027517460.29.443D0D3 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf09.hostedemail.com (Postfix) with ESMTP id 1C3C9140047 for ; Fri, 19 Apr 2024 21:07:03 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qUFqtB9n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of dmatlack@google.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=dmatlack@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713560824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vDccPRw3/PnxvI5b+QrmqH79otR7qScpzAtcfP56j+c=; b=Pq4+022IAbx3TaCmXs44Xf3Y+YyV64Q263NGVhjkn01UKhOmIGepVG8v0xEJPf27+fUd/M TTWgL6Jhy018jitZ5PfaEfu5T439CTQJa2BG15uoqZTjBwCIBTF9VLGRStJrhLrsvxOwk5 zABtWxNArAa1rSjHAi5U3Z8iwp1JP/U= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qUFqtB9n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of dmatlack@google.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=dmatlack@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713560824; a=rsa-sha256; cv=none; b=K5QMjMl3KMgGWO3cxupC6A7+LWJRrOtX3pWBLEoru6ZA7LONSDzrOUNc8kzemvImMkmDs+ reAvF6I+gSRsukVGK6wZi6Gis0Im32REwGHrXezXx/2CY4BDk4TMHkqDuT3EG9cGN23jok +phy1NqSGDbMOmRC5D8d6JRZgvuPQy4= Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6ed691fb83eso2100412b3a.1 for ; Fri, 19 Apr 2024 14:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713560823; x=1714165623; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=vDccPRw3/PnxvI5b+QrmqH79otR7qScpzAtcfP56j+c=; b=qUFqtB9n9BJf48jjEVl+AgdKt5SMibxthfoMeN95StaWLipimd9xiSaFJEV6cixRFS S4ZdXH1Zb2mPLeiN/3+pVdOJI0dplxQQ8EKqlBA0AVYEBlPj1MC2BSDUd1TZXHQAJu3+ QpUJmPpvdOSey9sUEJn530JjAFRbMyHE4niDc0QUvuBnsnIx3X5s7CEBv0AVOX6Q3Ha3 Ega75PqczX1/aNFjSLnhcuI4Qd2XDCFWa3IYRfdVc0I8Q/e3KVQc+0bnXZP2y4swNERJ fFz2KeHD41mukKgWxC/O+5qC65GLtOSw7facBm8a7FeTeBUP3SOxxLgbr8BPH+NrNuc9 aPHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713560823; x=1714165623; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vDccPRw3/PnxvI5b+QrmqH79otR7qScpzAtcfP56j+c=; b=QliG2J/OvVqQ83zLtT6Pt7m7YH9ucf85RgwNg3l7boSRUCTENi1KAee/4XafwgTF6Y ncPWuxcEWeZ1C7QXBNANgLxwComb9CwsoCJmltFJB49+/yT+tyavhDPIre2F8/3uNzkl rNdUmokfQkNQUwERYV521EhVWvC2WhJ3Q5835m4cLr16pTzQ2h9+kyC1ZUvRX5r1GMBb 69nPYxzVaIdg6JE0+7bEeqfF558O1AFU1ZaEKSuGpcwn+yROOYsdOXJv23gKZZwkyLPp sDLH6AwcZJxBkaEWeQuZqrOmMOxh7WxLV5yZWmX9Y3rRasS9x6/Rkyyf3Eccwib/KSox HQ+Q== X-Forwarded-Encrypted: i=1; AJvYcCVL4Wg1fMxtLPzeiLdgpAEdfuRuS1T6fY9Idtr2CA1QmTz30NuRp0E8inQrcVOK9xC0Bc+OhurK0cFnkUl0yh6ulqI= X-Gm-Message-State: AOJu0YwK84SJxG+OtmYdcq8hVR99/OyCGvQxM3VlFjnrCZd62jGk4sW3 2V6WNnRcz8OiagrifuJE2R7zBJP+6Crc+NHRKx5MRBQAyzB526nShC3KztXGlA== X-Google-Smtp-Source: AGHT+IHw/NlYfJI+uhC54L32I8Lr7TROJVo7EQ9Q27rPiTvl8C0kR92yS8I4f6VA8oYKT2j68BaI1g== X-Received: by 2002:a05:6a21:6da2:b0:1a3:63fa:f760 with SMTP id wl34-20020a056a216da200b001a363faf760mr4253434pzb.14.1713560822638; Fri, 19 Apr 2024 14:07:02 -0700 (PDT) Received: from google.com (210.73.125.34.bc.googleusercontent.com. [34.125.73.210]) by smtp.gmail.com with ESMTPSA id go20-20020a056a003b1400b006e6233563cesm3661623pfb.218.2024.04.19.14.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 14:07:01 -0700 (PDT) Date: Fri, 19 Apr 2024 14:06:56 -0700 From: David Matlack To: James Houghton Cc: Andrew Morton , Paolo Bonzini , Yu Zhao , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging Message-ID: References: <20240401232946.1837665-1-jthoughton@google.com> <20240401232946.1837665-6-jthoughton@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 1C3C9140047 X-Stat-Signature: xasn5toir3zs4ottfmtfdz3rkp81xqen X-HE-Tag: 1713560823-48613 X-HE-Meta: U2FsdGVkX1/N6MTv4mvztjFp2Ki2/tQftC+DlBClCxw6tPeO3tryNbszbcJyQSIBgXI+JqWyuOz/y4OAM1EgMJXDPbLUEz4cH/ZrakyNdM36yF5GsNblHWvmNn7KWxyPYRrzK+3ks3A8MFV/xt4vRpgkqfLfudUQtrD0T5/lyH9Z/0OLH8UkC9Qr7cGl+kTEAB9pMklibYPBttM65Tjxfzu6t/iaiRK2ZdIAhovcb5mQ54T60V7Rr1s39ekgKZn9XyFWx4CEjNanY7g1X92lrZz/IZsoY03jXK02fwTMLIYE6nKpOLu7IAVn9oFPYuo3ErHq40etfjorXfegKW9Mf8mmA2wK5b8XXNnl9jS7BmaRf3iezrvkG4bl7XhtbQA/DmnvfH2Kte0kxtN95Ugl7ogln7yMRed1ezjL84G5Rnu/sNs2vkR6DfcfoyBShJXnnZXGE5nDCOwuq0ZEyDdLNWGbvjTJdI45Fn2qU7RKi1GzMZ+PuCXbEcAcdxtSyW9OfoNPUMxW9m9vPtf+hKp1ufpR0dq1uDn3GQNvKp0W/v6uW3ei+Og4V8PZTwbvu/4i6DTvUGiY3R82E1ZxjanJo2w9iRXMXFdJ2KTbDiMggt//LQnT/CbciXFYFVeWPRVvD4DCJFJ8LVhDYgrVrZIqkBSLVns3V0LIC4iMLdghyfHw2VpZEkKBxOwyowLcSJpogkvtJs5/8bHqM5NiQqLxdM9ypY8dPl4MmW8B4Hq0Ups8Y27P1+Xovhbn4C8VbnoDttPhn84ZEGDAyMiNTuD95RCMTaaWlgHXyqpcjOPvOnSxWCv8nPsMSGbWKGk803Rtw7BS3IswIcWUo++rEvMeKe/oPof1dCr7BZ37ITa8PXd+oR2SquQuLGZS3Kl2l8djUXAYwctXpwJxoVs7fw25VF/LGtwlBZr4NCRoLkSqL5XnmMkEkG1HtBQOupsRYLRRVVL8WzyO9Wrr5I3H51B qZXUbsBh FC6u3jWMKpjw6mVrSyPDKs+tm+clijFWVifgKZsaLPFv3+lgYfzgUAIb/TirpC9rQz5S4NWi4nTDrLyhEE7Ns8asy88PSde4CqsfzB6qPdjyJbOGXSrZ5OcJDxeYnlqzLuBYXcC4sekMEQrDHnhwr58Yt1RVYvE/F/FBFjhYxwd3peDyI6zWsPvtHj4WVVs8ojEfdYa4lOVsStfL6PfCRyfNC6FezbFivrFlsPe4uwi+nu5bBEg7y/09x/iQU2eX34Mcp7T461ArzsEj/qM2d3tsVd0rnQODD1JcnsRDIG++KmB1ZGbCXEmlgHCcZlKCnCdRgrQXy7eQmIGzmZo9foE877qMdlWneUvHxrJZ+Y8OQI2egUCeIE79qhiLQ+gVxrbYjWNk0jUQWQ++RNMH0Y6o8Y0OZvN4w1Xe+e3mY2gRTE/rMGbFZ2eA+2N94urTCGGZv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-04-19 01:47 PM, James Houghton wrote: > On Thu, Apr 11, 2024 at 10:28 AM David Matlack wrote: > > On 2024-04-11 10:08 AM, David Matlack wrote: > > bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) > > { > > bool young = false; > > > > if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) > > young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); > > > > if (tdp_mmu_enabled) > > young |= kvm_tdp_mmu_age_gfn_range(kvm, range); > > > > return young; > > } > > > > bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) > > { > > bool young = false; > > > > if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm)) > > young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); > > > > if (tdp_mmu_enabled) > > young |= kvm_tdp_mmu_test_age_gfn(kvm, range); > > > > return young; > > > Yeah I think this is the right thing to do. Given your other > suggestions (on patch 3), I think this will look something like this > -- let me know if I've misunderstood something: > > bool check_rmap = !bitmap && kvm_memslot_have_rmaps(kvm); > > if (check_rmap) > KVM_MMU_LOCK(kvm); > > rcu_read_lock(); // perhaps only do this when we don't take the MMU lock? > > if (check_rmap) > kvm_handle_gfn_range(/* ... */ kvm_test_age_rmap) > > if (tdp_mmu_enabled) > kvm_tdp_mmu_test_age_gfn() // modified to be RCU-safe > > rcu_read_unlock(); > if (check_rmap) > KVM_MMU_UNLOCK(kvm); I was thinking a little different. If you follow my suggestion to first make the TDP MMU aging lockless, you'll end up with something like this prior to adding bitmap support (note: the comments are just for demonstrative purposes): bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; /* Shadow MMU aging holds write-lock. */ if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; } Then when you add bitmap support it would look something like this: bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { unsigned long *bitmap = range->arg.metadata->bitmap; bool young = false; /* SHadow MMU aging holds write-lock and does not support bitmap. */ if (kvm_memslots_have_rmaps(kvm) && !bitmap) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } /* TDM MMU aging is lockless and supports bitmap. */ if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); return young; } rcu_read_lock/unlock() would be called in kvm_tdp_mmu_age_gfn_range(). That brings up a question I've been wondering about. If KVM only advertises support for the bitmap lookaround when shadow roots are not allocated, does that mean MGLRU will be blind to accesses made by L2 when nested virtualization is enabled? And does that mean the Linux MM will think all L2 memory is cold (i.e. good candidate for swapping) because it isn't seeing accesses made by L2?