From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A7C8C02198 for ; Thu, 6 Feb 2025 03:37:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A403B6B0085; Wed, 5 Feb 2025 22:37:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F0A86B0088; Wed, 5 Feb 2025 22:37:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 891D76B0089; Wed, 5 Feb 2025 22:37:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6A0D56B0085 for ; Wed, 5 Feb 2025 22:37:11 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 18C711C86C0 for ; Thu, 6 Feb 2025 03:37:11 +0000 (UTC) X-FDA: 83088109062.23.ECC65F5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf07.hostedemail.com (Postfix) with ESMTP id 526894000E for ; Thu, 6 Feb 2025 03:37:09 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=j0gECEvA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3ZC6kZwsKCJUz193GA3NIC55DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--ackerleytng.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3ZC6kZwsKCJUz193GA3NIC55DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--ackerleytng.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738813029; a=rsa-sha256; cv=none; b=Xv7uR7ZYGrrkAPinuVPLiuKstr0nnhRZeDMV0Yr2y4M0PdiwMZMIiQAk8YxNwqWrvv868d Q91bK+Nc5pde6D5vHoSNTo7onB3SY0CnBzUKMcs9c1UxC92rQFrgAKo7z3e5ctXc1h+JHl xSrYlB28GFLII2auzaPJd3ry+1MSLZ4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=j0gECEvA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3ZC6kZwsKCJUz193GA3NIC55DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--ackerleytng.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3ZC6kZwsKCJUz193GA3NIC55DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--ackerleytng.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738813029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:dkim-signature; bh=pRUcWLN38Aa0aZPGEebbNawOjCOpEyDG26hlfVRLOOs=; b=RLOr7a4Onyr3CU9oXUR+oG8ibD8yIiRmPKmJEtJ7CBtYNDDNCa+mXcemBZFdbaKjNy1dvr xKoL7mAVA+S4ibcrL6ogXyT0wIqVggVTlBBL9gpF67r+0DqlhcZx/IzzZijoQJqGkUy+2w Fof965JDZjoex+aZTOg1c9e591eF9rY= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f9da17946fso1330300a91.3 for ; Wed, 05 Feb 2025 19:37:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738813028; x=1739417828; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:in-reply-to:date:from:to :cc:subject:date:message-id:reply-to; bh=pRUcWLN38Aa0aZPGEebbNawOjCOpEyDG26hlfVRLOOs=; b=j0gECEvASk5XAt70S7hbjp9nPDNcJ6fXAZiYgKGmU7AycxkVtusK6DF/ZG3gBu30Ly H6vIS8cqUNWKMKqeZGK3iqcsgemZo9faHOXU8aXeYAHhhtrzGm8TQpFGpSlSnWBTY0D1 SfdTw97BSw7dI3+u/Rn6xbL13dQCb0MYXZLoqhOY8v6Vhl1d2cnlS9QmD0BelUETFIsX H2iXDj2jcrzN1/1QZ0AB+udQOQKpuwKMCff0cp6oekMOorWotCT2+dsmL8HAGe1gn3Xr HP+p8y4cvw01uDnDMiu8Ns9KC6/qU/lMAmra4NNQMfB8hivSpElcH7y9nUB5UROD5rSo h8hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738813028; x=1739417828; h=cc:to:from:subject:message-id:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pRUcWLN38Aa0aZPGEebbNawOjCOpEyDG26hlfVRLOOs=; b=iENUf6Gg0HrNbCv0+0QqB4BIj3Q242RM47gBzzct6JNDd9tJriefofZ5av0HtTxSpt ucBOq7Kk05uXvAq+ZOZJo15sq4tQy6JjaiMLSdQ2mb50OSIcJsoXe+ZqBxdae5RwIppY urSj4A2TsyKOP3HPQnonqGVLxERikbSs7kF1vJ5IV6lN76eviur4RX9E3jcLPZQAIzye cnucLmTb3/3T/wJHxqCX07eW2NrXBeu5nlSk7q0Jia9mRTulMaJrPMnrRFbAONcfVMGs cVjfrprX5ojr4n6XMK5AvI5vHNSqGhI3us+YUuFI3UXstZ65HnakmSKwyUlWxFJMYQG6 pPsw== X-Forwarded-Encrypted: i=1; AJvYcCVH5YzNHk1Cwl38tmJ8FtwXbVJFXu5IAa3UGXVn4DZbdTZ1XUsVEJ32LiD93+x8etvXW+tGtvjoVg==@kvack.org X-Gm-Message-State: AOJu0YxcEhU43CMB2qcUOZYJjdLVS5s8t5rv8mQsUVJ7oYlgTqXTYXBl WFfZT1pmg4JKMIK2HJfFnFB3DVdCd1bd2sehFY3maNfp3GuYJp1b6RXxG1s9pzSNuX6j5+s5s5v +fjTR72VvIk0BqBVyRpImUA== X-Google-Smtp-Source: AGHT+IGhY4/pV+4mHiEaSq+to1TSQKDTUBZjG4PDt5LI9mDx6rl17K7kwWvRj+1UQyTkxkdUhDnW93d4pfLlxxgdZA== X-Received: from pjl7.prod.google.com ([2002:a17:90b:2f87:b0:2f9:d5f9:128f]) (user=ackerleytng job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:d44b:b0:2ee:c9b6:c267 with SMTP id 98e67ed59e1d1-2f9e075da23mr8852501a91.9.1738813028145; Wed, 05 Feb 2025 19:37:08 -0800 (PST) Date: Thu, 06 Feb 2025 03:37:06 +0000 In-Reply-To: <20250117163001.2326672-7-tabba@google.com> (message from Fuad Tabba on Fri, 17 Jan 2025 16:29:52 +0000) Mime-Version: 1.0 Message-ID: Subject: Re: [RFC PATCH v5 06/15] KVM: guest_memfd: Handle final folio_put() of guestmem pages From: Ackerley Tng To: Fuad Tabba Cc: kvm@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mm@kvack.org, pbonzini@redhat.com, chenhuacai@kernel.org, mpe@ellerman.id.au, anup@brainfault.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, seanjc@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, willy@infradead.org, akpm@linux-foundation.org, xiaoyao.li@intel.com, yilun.xu@intel.com, chao.p.peng@linux.intel.com, jarkko@kernel.org, amoorthy@google.com, dmatlack@google.com, yu.c.zhang@linux.intel.com, isaku.yamahata@intel.com, mic@digikod.net, vbabka@suse.cz, vannapurve@google.com, mail@maciej.szmigiero.name, david@redhat.com, michael.roth@amd.com, wei.w.wang@intel.com, liam.merwick@oracle.com, isaku.yamahata@gmail.com, kirill.shutemov@linux.intel.com, suzuki.poulose@arm.com, steven.price@arm.com, quic_eberman@quicinc.com, quic_mnalajal@quicinc.com, quic_tsoni@quicinc.com, quic_svaddagi@quicinc.com, quic_cvanscha@quicinc.com, quic_pderrin@quicinc.com, quic_pheragu@quicinc.com, catalin.marinas@arm.com, james.morse@arm.com, yuzenghui@huawei.com, oliver.upton@linux.dev, maz@kernel.org, will@kernel.org, qperret@google.com, keirf@google.com, roypat@amazon.co.uk, shuah@kernel.org, hch@infradead.org, jgg@nvidia.com, rientjes@google.com, jhubbard@nvidia.com, fvdl@google.com, hughd@google.com, jthoughton@google.com, tabba@google.com Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Queue-Id: 526894000E X-Rspamd-Server: rspam10 X-Stat-Signature: 6e1b1ek8de71e83fbkdsxwjcsx3e64jk X-HE-Tag: 1738813029-210070 X-HE-Meta: U2FsdGVkX1+yeYMoRSHTbmMjrOUUevsDKaWGkkx5QjGzLJh9ySDzWzcHKmvtZiBx115lE16jKF+k3aUlHa/SQiTM8fAWxqvjLs1MSccpg3l+pwR3gJyUhLD/q9ltLxzP8fvHeeTJmwUo267Rsx7ViVwZ9lw4dCqGVxTHA//+3fdB6ag40FOrjSyfak5i8KuVJKq851tfMy2uxHtneu2+XxZ0tSWfgNSME6sbjbFWTdBvSq01TJY1yAWmQVKiSdqHBKhGl3M7rcVLhl40jgmLJu9Eqi9dclIlBw/EUZtQ/BTkp+tpE3gSbcFrW9LVpuLvbY4xHNJ82Akwcj8AiTyyDF81CHQ49NtW+n7wso42FIC1eehtMcBCzfDcrxnIsc6L73pJSyBv4wDuu71JgOTwVki2F4oN6CsFr7dEevQiGjBqz1PgFLunCknczGQVHSPyXdbY78tiXPlS2o6FIOd/SQHMp/muRJ7WwjtT6jJGelyX5vJo4VEQ1NSYjLgFiv50d6fBVStVthjSGsyxykftJplfVIjIgBXAFtQB3Q9EAJ6Iluvk9Eyv/MW3ozGRx3eAVlaUjp4WAw3JiRAqBMJV0SjAWF0RYTd8v9y0LZxkbeQR2VTmEU8blP9ov6sZjzwkGOriqZm/JvvGwvW5cj0OBnJ1FRkIfDNfpwq3zna4eODrEJKg4K9ETo0AOdCMXi7jDUgouUzaKFrIqX4onAH5aT5pxFahvt27gOzQ4ESq90at1RNOGxfk61f6EZyEWl7UdYBZQslBy1u1W5tho8ZDxT9iyzajZt7HZx8U/y4ofH6o6unMvbmuDGkYeoWQbNzDBh4cB9c60KxcE4/IZxxKLDrQMnGF4q1d/KOWAcQqo70P1HZuN5nVHmeLA3QzeUcNeSMibOYgHy59ixxQOtbgCPPGbH+8e+aoFIpfQu0MiV92Gj0xQOWWN/VAhRyLYLX6WfRoEgG/kL7obAVTAf+ kP/YYofU DHuTy0mUv+6zy0s9u9Y8kIen5BHzs/QAuwfg4hMCJ8hrsaKjwVnzEFD9w0MnbocKrOt681VU8wLwJ9noqQjD/q3FYoxAp/w2mQGJgiHQgEyeRoFk/gN0aafDblo9pF16FCcY29SZLRfTZToUPVssk+IvCg6hEDRk3MzAArp9QaevY7lljgskMmvxstpNxLTOJSkcHgqkdZmvzcdHnx/QDI4LJSxddjAqVNzS6w9l2R/pkr1vCOFN3s4dM9PSkdg7Y18r07vrfBQtcITKLGqJjQsyuZnovK98hxFDyS8Gbtzp/Af3CidzUTd/cOG0+8SUpnImfDmfRJjFXcxmOrr3eatAkrHK+kZ9iKIDAnjzE2HSzvIeUoZn7qv8OsrVSaC83LH9az/PJTQ+tTXq/OihjTtbQkdeYCfqCLfoCcpvjggBuOPGAqV5SEopqVFbkyqRAwvw9v3oZ/608iXWhNP0f2FirdfWLkXkazTf6PoeG41RWdUORMpaZQp4rPPqDtmZzoiI7hdDV6aZ/k3BkYsU6i0Ku3hQq/gKQQgGPjxZZsEcWnK094RfGOv1hwjks2pgP7l45lpWoZKzu5WFcHYeV2MufwFgZ1ZAUscGvz0v063QB22eygpPq/tbd8VElik77Xam/ysnTfvw32qSj1QdsfjcJQrGHqaP7XC9+Y1vt/eNIeershwZl4A8XxsJ3GyFY76jcx57qoYRXy2PbaNt4xLo63q0Jh0/9fgLg6RZBYcGqN8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Fuad Tabba writes: > Before transitioning a guest_memfd folio to unshared, thereby > disallowing access by the host and allowing the hypervisor to > transition its view of the guest page as private, we need to be > sure that the host doesn't have any references to the folio. > > This patch introduces a new type for guest_memfd folios, and uses > that to register a callback that informs the guest_memfd > subsystem when the last reference is dropped, therefore knowing > that the host doesn't have any remaining references. > > Signed-off-by: Fuad Tabba > --- > The function kvm_slot_gmem_register_callback() isn't used in this > series. It will be used later in code that performs unsharing of > memory. I have tested it with pKVM, based on downstream code [*]. > It's included in this RFC since it demonstrates the plan to > handle unsharing of private folios. > > [*] https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/guestmem-6.13-v5-pkvm > --- > include/linux/kvm_host.h | 11 +++ > include/linux/page-flags.h | 7 ++ > mm/debug.c | 1 + > mm/swap.c | 4 + > virt/kvm/guest_memfd.c | 145 +++++++++++++++++++++++++++++++++++++ > 5 files changed, 168 insertions(+) > > diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h > index 84aa7908a5dd..63e6d6dd98b3 100644 > --- a/include/linux/kvm_host.h > +++ b/include/linux/kvm_host.h > @@ -2574,6 +2574,8 @@ int kvm_slot_gmem_clear_mappable(struct kvm_memory_slot *slot, gfn_t start, > gfn_t end); > bool kvm_slot_gmem_is_mappable(struct kvm_memory_slot *slot, gfn_t gfn); > bool kvm_slot_gmem_is_guest_mappable(struct kvm_memory_slot *slot, gfn_t gfn); > +int kvm_slot_gmem_register_callback(struct kvm_memory_slot *slot, gfn_t gfn); > +void kvm_gmem_handle_folio_put(struct folio *folio); > #else > static inline bool kvm_gmem_is_mappable(struct kvm *kvm, gfn_t gfn, gfn_t end) > { > @@ -2615,6 +2617,15 @@ static inline bool kvm_slot_gmem_is_guest_mappable(struct kvm_memory_slot *slot, > WARN_ON_ONCE(1); > return false; > } > +static inline int kvm_slot_gmem_register_callback(struct kvm_memory_slot *slot, gfn_t gfn) > +{ > + WARN_ON_ONCE(1); > + return -EINVAL; > +} > +static inline void kvm_gmem_handle_folio_put(struct folio *folio) > +{ > + WARN_ON_ONCE(1); > +} > #endif /* CONFIG_KVM_GMEM_MAPPABLE */ > > #endif > diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h > index 6615f2f59144..bab3cac1f93b 100644 > --- a/include/linux/page-flags.h > +++ b/include/linux/page-flags.h > @@ -942,6 +942,7 @@ enum pagetype { > PGTY_slab = 0xf5, > PGTY_zsmalloc = 0xf6, > PGTY_unaccepted = 0xf7, > + PGTY_guestmem = 0xf8, > > PGTY_mapcount_underflow = 0xff > }; > @@ -1091,6 +1092,12 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) > FOLIO_TEST_FLAG_FALSE(hugetlb) > #endif > > +#ifdef CONFIG_KVM_GMEM_MAPPABLE > +FOLIO_TYPE_OPS(guestmem, guestmem) > +#else > +FOLIO_TEST_FLAG_FALSE(guestmem) > +#endif > + > PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc) > > /* > diff --git a/mm/debug.c b/mm/debug.c > index 95b6ab809c0e..db93be385ed9 100644 > --- a/mm/debug.c > +++ b/mm/debug.c > @@ -56,6 +56,7 @@ static const char *page_type_names[] = { > DEF_PAGETYPE_NAME(table), > DEF_PAGETYPE_NAME(buddy), > DEF_PAGETYPE_NAME(unaccepted), > + DEF_PAGETYPE_NAME(guestmem), > }; > > static const char *page_type_name(unsigned int page_type) > diff --git a/mm/swap.c b/mm/swap.c > index 6f01b56bce13..15220eaabc86 100644 > --- a/mm/swap.c > +++ b/mm/swap.c > @@ -37,6 +37,7 @@ > #include > #include > #include > +#include > > #include "internal.h" > > @@ -103,6 +104,9 @@ static void free_typed_folio(struct folio *folio) > case PGTY_offline: > /* Nothing to do, it's offline. */ > return; > + case PGTY_guestmem: > + kvm_gmem_handle_folio_put(folio); > + return; > default: > WARN_ON_ONCE(1); > } > diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c > index d1c192927cf7..722afd9f8742 100644 > --- a/virt/kvm/guest_memfd.c > +++ b/virt/kvm/guest_memfd.c > @@ -387,6 +387,28 @@ enum folio_mappability { > KVM_GMEM_NONE_MAPPABLE = 0b11, /* Not mappable, transient state. */ > }; > > +/* > + * Unregisters the __folio_put() callback from the folio. > + * > + * Restores a folio's refcount after all pending references have been released, > + * and removes the folio type, thereby removing the callback. Now the folio can > + * be freed normaly once all actual references have been dropped. > + * > + * Must be called with the filemap (inode->i_mapping) invalidate_lock held. > + * Must also have exclusive access to the folio: folio must be either locked, or > + * gmem holds the only reference. > + */ > +static void __kvm_gmem_restore_pending_folio(struct folio *folio) > +{ > + if (WARN_ON_ONCE(folio_mapped(folio) || !folio_test_guestmem(folio))) > + return; > + > + WARN_ON_ONCE(!folio_test_locked(folio) && folio_ref_count(folio) > 1); > + > + __folio_clear_guestmem(folio); > + folio_ref_add(folio, folio_nr_pages(folio)); > +} > + > /* > * Marks the range [start, end) as mappable by both the host and the guest. > * Usually called when guest shares memory with the host. > @@ -400,7 +422,31 @@ static int gmem_set_mappable(struct inode *inode, pgoff_t start, pgoff_t end) > > filemap_invalidate_lock(inode->i_mapping); > for (i = start; i < end; i++) { > + struct folio *folio = NULL; > + > + /* > + * If the folio is NONE_MAPPABLE, it indicates that it is > + * transitioning to private (GUEST_MAPPABLE). Transition it to > + * shared (ALL_MAPPABLE) immediately, and remove the callback. > + */ > + if (xa_to_value(xa_load(mappable_offsets, i)) == KVM_GMEM_NONE_MAPPABLE) { > + folio = filemap_lock_folio(inode->i_mapping, i); > + if (WARN_ON_ONCE(IS_ERR(folio))) { > + r = PTR_ERR(folio); > + break; > + } > + > + if (folio_test_guestmem(folio)) > + __kvm_gmem_restore_pending_folio(folio); > + } > + > r = xa_err(xa_store(mappable_offsets, i, xval, GFP_KERNEL)); > + > + if (folio) { > + folio_unlock(folio); > + folio_put(folio); > + } > + > if (r) > break; > } > @@ -473,6 +519,105 @@ static int gmem_clear_mappable(struct inode *inode, pgoff_t start, pgoff_t end) > return r; > } > I think one of these functions to restore mappability needs to be called to restore the refcounts on truncation. Without doing this, the refcounts on the folios at truncation time would only be the transient/speculative ones, and truncating will take off the filemap refcounts which were already taken off to set up the folio_put() callback. Should mappability can be restored according to GUEST_MEMFD_FLAG_INIT_MAPPABLE? Or should mappability of NONE be restored to GUEST and mappability of ALL left as ALL? > +/* > + * Registers a callback to __folio_put(), so that gmem knows that the host does > + * not have any references to the folio. It does that by setting the folio type > + * to guestmem. > + * > + * Returns 0 if the host doesn't have any references, or -EAGAIN if the host > + * has references, and the callback has been registered. > + * > + * Must be called with the following locks held: > + * - filemap (inode->i_mapping) invalidate_lock > + * - folio lock > + */ > +static int __gmem_register_callback(struct folio *folio, struct inode *inode, pgoff_t idx) > +{ > + struct xarray *mappable_offsets = &kvm_gmem_private(inode)->mappable_offsets; > + void *xval_guest = xa_mk_value(KVM_GMEM_GUEST_MAPPABLE); > + int refcount; > + > + rwsem_assert_held_write_nolockdep(&inode->i_mapping->invalidate_lock); > + WARN_ON_ONCE(!folio_test_locked(folio)); > + > + if (folio_mapped(folio) || folio_test_guestmem(folio)) > + return -EAGAIN; > + > + /* Register a callback first. */ > + __folio_set_guestmem(folio); > + > + /* > + * Check for references after setting the type to guestmem, to guard > + * against potential races with the refcount being decremented later. > + * > + * At least one reference is expected because the folio is locked. > + */ > + > + refcount = folio_ref_sub_return(folio, folio_nr_pages(folio)); > + if (refcount == 1) { > + int r; > + > + /* refcount isn't elevated, it's now faultable by the guest. */ > + r = WARN_ON_ONCE(xa_err(xa_store(mappable_offsets, idx, xval_guest, GFP_KERNEL))); > + if (!r) > + __kvm_gmem_restore_pending_folio(folio); > + > + return r; > + } > + > + return -EAGAIN; > +} > + > +int kvm_slot_gmem_register_callback(struct kvm_memory_slot *slot, gfn_t gfn) > +{ > + unsigned long pgoff = slot->gmem.pgoff + gfn - slot->base_gfn; > + struct inode *inode = file_inode(slot->gmem.file); > + struct folio *folio; > + int r; > + > + filemap_invalidate_lock(inode->i_mapping); > + > + folio = filemap_lock_folio(inode->i_mapping, pgoff); > + if (WARN_ON_ONCE(IS_ERR(folio))) { > + r = PTR_ERR(folio); > + goto out; > + } > + > + r = __gmem_register_callback(folio, inode, pgoff); > + > + folio_unlock(folio); > + folio_put(folio); > +out: > + filemap_invalidate_unlock(inode->i_mapping); > + > + return r; > +} > + > +/* > + * Callback function for __folio_put(), i.e., called when all references by the > + * host to the folio have been dropped. This allows gmem to transition the state > + * of the folio to mappable by the guest, and allows the hypervisor to continue > + * transitioning its state to private, since the host cannot attempt to access > + * it anymore. > + */ > +void kvm_gmem_handle_folio_put(struct folio *folio) > +{ > + struct xarray *mappable_offsets; > + struct inode *inode; > + pgoff_t index; > + void *xval; > + > + inode = folio->mapping->host; > + index = folio->index; > + mappable_offsets = &kvm_gmem_private(inode)->mappable_offsets; > + xval = xa_mk_value(KVM_GMEM_GUEST_MAPPABLE); > + > + filemap_invalidate_lock(inode->i_mapping); > + __kvm_gmem_restore_pending_folio(folio); > + WARN_ON_ONCE(xa_err(xa_store(mappable_offsets, index, xval, GFP_KERNEL))); > + filemap_invalidate_unlock(inode->i_mapping); > +} > + > static bool gmem_is_mappable(struct inode *inode, pgoff_t pgoff) > { > struct xarray *mappable_offsets = &kvm_gmem_private(inode)->mappable_offsets;