From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1FD9C433EF for ; Tue, 5 Apr 2022 18:03:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BDAE6B0072; Tue, 5 Apr 2022 14:03:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06D8D6B0073; Tue, 5 Apr 2022 14:03:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E50166B0074; Tue, 5 Apr 2022 14:03:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id D49BE6B0072 for ; Tue, 5 Apr 2022 14:03:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8ABA28248D52 for ; Tue, 5 Apr 2022 18:03:28 +0000 (UTC) X-FDA: 79323597696.21.82F8A7D Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf19.hostedemail.com (Postfix) with ESMTP id 18A481A0033 for ; Tue, 5 Apr 2022 18:03:27 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id o5-20020a17090ad20500b001ca8a1dc47aso3370118pju.1 for ; Tue, 05 Apr 2022 11:03:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=U3PyY0pRKQhXeOMKJejZ2jWEMHgtC5U/1t6Griz2ZtE=; b=KGXUBKKkDKDhZlvZKiQELbSWKwFyjZ9pqbjywy++hvUawyC+vx+Kiwb6Gb6jEV0Wuc giwl5qnSM5u8wdE1F5+llR4K2Ticwza1m/ioW1C0ItJqonPtmvm5nhpHrTL3N6QT1Rl3 Wtn49OopVnhw8g8P+PKXrUXFRKcYdC8fjuouVVIXctNCy1B+qGpT4pX2jNYpbQ1t2mP6 xMe4g7fYtlOfWMZCvSnpCUHEZXd84dkEWXSH6ZD1YzTcBCMO32gius56GNlGsSmncOvV KQlOe+jigYhK1S2TKkrRmuV21s9fTFFu+nLTpzm/DTIX7AvACXAolik1vLnEB78gFwNe bYpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=U3PyY0pRKQhXeOMKJejZ2jWEMHgtC5U/1t6Griz2ZtE=; b=zk55ompCpDfNNKV6amUYN0PTYmmJJKWTyCB/ASrPqrFRLHjlr00bGT+daKSwXlPpgn FmEqtnSFmsJpHCCTDL2RKRdacEoaJl3GGSV2NpgkEZwKUjaobh21+tBl2Zbh55s6SjPX jnZAXfknSJ9iv37257+Q8OGm8s8utBynDd9KCql7jWbMTtWupE10wMgP8+YB3d7Yc2wl WKm+ljOTOcRl9KxK412HTIvOm0G5ZbcOTESEgeJJSM9LH/y13CuAxX6zF8WzZFpoEXEK vNpUerCsMAtA28Ptpdk7chyz8S0Ettdm1AFdVYIFTwYTSm9xzwdX9KIJ9O4haOu3307w TH8g== X-Gm-Message-State: AOAM531BxPEpyGvDAnnaiBGfN8LQofjqI7SZHO15/DdVBBW9FUhLvDD2 /UpdTuGyOG3VjKwlPyzb3TSrWQ== X-Google-Smtp-Source: ABdhPJyStH5/Nz6FqwbLd9JjuWL5wIJ/TCBBrc+CUyEhlKkOsXFCG0zdV80i6DTnhSTJUvObFVL14A== X-Received: by 2002:a17:903:288:b0:156:a6b5:80d4 with SMTP id j8-20020a170903028800b00156a6b580d4mr4934454plr.98.1649181806567; Tue, 05 Apr 2022 11:03:26 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id d21-20020a056a0024d500b004fb0e7c7c3bsm17524046pfv.161.2022.04.05.11.03.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Apr 2022 11:03:25 -0700 (PDT) Date: Tue, 5 Apr 2022 18:03:21 +0000 From: Sean Christopherson To: Quentin Perret Cc: Andy Lutomirski , Steven Price , Chao Peng , kvm list , Linux Kernel Mailing List , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Linux API , qemu-devel@nongnu.org, Paolo Bonzini , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , the arch/x86 maintainers , "H. Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Mike Rapoport , "Maciej S . Szmigiero" , Vlastimil Babka , Vishal Annapurve , Yu Zhang , "Kirill A. Shutemov" , "Nakajima, Jun" , Dave Hansen , Andi Kleen , David Hildenbrand , Marc Zyngier , Will Deacon Subject: Re: [PATCH v5 00/13] KVM: mm: fd-based approach for supporting KVM guest private memory Message-ID: References: <88620519-029e-342b-0a85-ce2a20eaf41b@arm.com> <80aad2f9-9612-4e87-a27a-755d3fa97c92@www.fastmail.com> <83fd55f8-cd42-4588-9bf6-199cbce70f33@www.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KGXUBKKk; spf=pass (imf19.hostedemail.com: domain of seanjc@google.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=seanjc@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 18A481A0033 X-Stat-Signature: fb5memh7nzrre9z9enuosn1sigo3adoa X-HE-Tag: 1649181807-859568 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Apr 05, 2022, Quentin Perret wrote: > On Monday 04 Apr 2022 at 15:04:17 (-0700), Andy Lutomirski wrote: > > >> - it can be very useful for protected VMs to do shared=>private > > >> conversions. Think of a VM receiving some data from the host in a > > >> shared buffer, and then it wants to operate on that buffer without > > >> risking to leak confidential informations in a transient state. In > > >> that case the most logical thing to do is to convert the buffer back > > >> to private, do whatever needs to be done on that buffer (decrypting a > > >> frame, ...), and then share it back with the host to consume it; > > > > > > If performance is a motivation, why would the guest want to do two > > > conversions instead of just doing internal memcpy() to/from a private > > > page? I would be quite surprised if multiple exits and TLB shootdowns is > > > actually faster, especially at any kind of scale where zapping stage-2 > > > PTEs will cause lock contention and IPIs. > > > > I don't know the numbers or all the details, but this is arm64, which is a > > rather better architecture than x86 in this regard. So maybe it's not so > > bad, at least in very simple cases, ignoring all implementation details. > > (But see below.) Also the systems in question tend to have fewer CPUs than > > some of the massive x86 systems out there. > > Yep. I can try and do some measurements if that's really necessary, but > I'm really convinced the cost of the TLBI for the shared->private > conversion is going to be significantly smaller than the cost of memcpy > the buffer twice in the guest for us. It's not just the TLB shootdown, the VM-Exits aren't free. And barring non-trivial improvements to KVM's MMU, e.g. sharding of mmu_lock, modifying the page tables will block all other updates and MMU operations. Taking mmu_lock for read, should arm64 ever convert to a rwlock, is not an option because KVM needs to block other conversions to avoid races. Hmm, though batching multiple pages into a single request would mitigate most of the overhead. > There are variations of that idea: e.g. allow userspace to mmap the > entire private fd but w/o taking a reference on pages mapped with > PROT_NONE. And then the VMM can use mprotect() in response to > share/unshare requests. I think Marc liked that idea as it keeps the > userspace API closer to normal KVM -- there actually is a > straightforward gpa->hva relation. Not sure how much that would impact > the implementation at this point. > > For the shared=>private conversion, this would be something like so: > > - the guest issues a hypercall to unshare a page; > > - the hypervisor forwards the request to the host; > > - the host kernel forwards the request to userspace; > > - userspace then munmap()s the shared page; > > - KVM then tries to take a reference to the page. If it succeeds, it > re-enters the guest with a flag of some sort saying that the share > succeeded, and the hypervisor will adjust pgtables accordingly. If > KVM failed to take a reference, it flags this and the hypervisor will > be responsible for communicating that back to the guest. This means > the guest must handle failures (possibly fatal). > > (There are probably many ways in which we can optimize this, e.g. by > having the host proactively munmap() pages it no longer needs so that > the unshare hypercall from the guest doesn't need to exit all the way > back to host userspace.) ... > > Maybe there could be a special mode for the private memory fds in which > > specific pages are marked as "managed by this fd but actually shared". > > pread() and pwrite() would work on those pages, but not mmap(). (Or maybe > > mmap() but the resulting mappings would not permit GUP.) Unless I misunderstand what you intend by pread()/pwrite(), I think we'd need to allow mmap(), otherwise e.g. uaccess from the kernel wouldn't work. > > And transitioning them would be a special operation on the fd that is > > specific to pKVM and wouldn't work on TDX or SEV. To keep things feature agnostic (IMO, baking TDX vs SEV vs pKVM info into private-fd is a really bad idea), this could be handled by adding a flag and/or callback into the notifier/client stating whether or not it supports mapping a private-fd, and then mapping would be allowed if and only if all consumers support/allow mapping. > > Hmm. Sean and Chao, are we making a bit of a mistake by making these fds > > technology-agnostic? That is, would we want to distinguish between a TDX > > backing fd, a SEV backing fd, a software-based backing fd, etc? API-wise > > this could work by requiring the fd to be bound to a KVM VM instance and > > possibly even configured a bit before any other operations would be > > allowed. I really don't want to distinguish between between each exact feature, but I've no objection to adding flags/callbacks to track specific properties of the downstream consumers, e.g. "can this memory be accessed by userspace" is a fine abstraction. It also scales to multiple consumers (see above).