From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A72B6C433F5 for ; Mon, 25 Apr 2022 13:44:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 134E26B0074; Mon, 25 Apr 2022 09:44:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E41D6B0075; Mon, 25 Apr 2022 09:44:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC66A6B0078; Mon, 25 Apr 2022 09:44:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id D6E466B0074 for ; Mon, 25 Apr 2022 09:44:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id A3C7261916 for ; Mon, 25 Apr 2022 13:44:30 +0000 (UTC) X-FDA: 79395521100.16.E9D597F Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf03.hostedemail.com (Postfix) with ESMTP id 2E5E620036 for ; Mon, 25 Apr 2022 13:44:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1650894269; x=1682430269; h=date:from:to:cc:subject:message-id:reply-to:references: mime-version:in-reply-to; bh=11l8nWOAh8jVYp4pTM08c8ajsmoXROtLceTxJ8NiSMU=; b=QPNS/iLbqr3YI9OyOKS9OxAq+6ORZRGqzQc8+nZ4zWdbGGKBWLnOXH4B 69ZvpQLFcdfdD/iz5uFzT0p6oUg+kexGTu/Daff88YoKtOy1D1zYG3Hzb kwPL7n+Lw8tPk3EcvFXJw0hGR2NNVQUpTFW/8PDp8j+vmrEcJzCioPSMV UpK/V2ZAXrtLyVKNR2xlbqZeE70XP5VZgg+h1Z1PJVGiOCtjQsH1BJX23 YjxCM9zi15+bApTkh6uYUx4SETDC+0CK4CtFRq+EU/+AwwVrh0lAB3Xbj YvVSfdTnUDTA8mJcbPCbnTeHRN0hQsbN55WLJ9erAN9ApkbmR1GAR9uEu g==; X-IronPort-AV: E=McAfee;i="6400,9594,10327"; a="265414234" X-IronPort-AV: E=Sophos;i="5.90,288,1643702400"; d="scan'208";a="265414234" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Apr 2022 06:44:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,288,1643702400"; d="scan'208";a="704562650" Received: from chaop.bj.intel.com (HELO localhost) ([10.240.192.101]) by fmsmga001.fm.intel.com with ESMTP; 25 Apr 2022 06:44:19 -0700 Date: Mon, 25 Apr 2022 21:40:51 +0800 From: Chao Peng To: Andy Lutomirski Cc: Sean Christopherson , Quentin Perret , Steven Price , kvm list , Linux Kernel Mailing List , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Linux API , qemu-devel@nongnu.org, Paolo Bonzini , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , the arch/x86 maintainers , "H. Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Mike Rapoport , "Maciej S . Szmigiero" , Vlastimil Babka , Vishal Annapurve , Yu Zhang , "Kirill A. Shutemov" , "Nakajima, Jun" , Dave Hansen , Andi Kleen , David Hildenbrand , Marc Zyngier , Will Deacon Subject: Re: [PATCH v5 00/13] KVM: mm: fd-based approach for supporting KVM guest private memory Message-ID: <20220425134051.GA175928@chaop.bj.intel.com> Reply-To: Chao Peng References: <80aad2f9-9612-4e87-a27a-755d3fa97c92@www.fastmail.com> <83fd55f8-cd42-4588-9bf6-199cbce70f33@www.fastmail.com> <20220422105612.GB61987@chaop.bj.intel.com> <3b99f157-0f30-4b30-8399-dd659250ab8d@www.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3b99f157-0f30-4b30-8399-dd659250ab8d@www.fastmail.com> X-Stat-Signature: 7oymcid5qhxn415oj1ca8u4fq8cxk5ty Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="QPNS/iLb"; spf=none (imf03.hostedemail.com: domain of chao.p.peng@linux.intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=chao.p.peng@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2E5E620036 X-HE-Tag: 1650894265-687863 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Apr 24, 2022 at 09:59:37AM -0700, Andy Lutomirski wrote: > > > On Fri, Apr 22, 2022, at 3:56 AM, Chao Peng wrote: > > On Tue, Apr 05, 2022 at 06:03:21PM +0000, Sean Christopherson wrote: > >> On Tue, Apr 05, 2022, Quentin Perret wrote: > >> > On Monday 04 Apr 2022 at 15:04:17 (-0700), Andy Lutomirski wrote: > > Only when the register succeeds, the fd is > > converted into a private fd, before that, the fd is just a normal (shared) > > one. During this conversion, the previous data is preserved so you can put > > some initial data in guest pages (whether the architecture allows this is > > architecture-specific and out of the scope of this patch). > > I think this can be made to work, but it will be awkward. On TDX, for example, what exactly are the semantics supposed to be? An error code if the memory isn't all zero? An error code if it has ever been written? > > Fundamentally, I think this is because your proposed lifecycle for these memfiles results in a lightweight API but is awkward for the intended use cases. You're proposing, roughly: > > 1. Create a memfile. > > Now it's in a shared state with an unknown virt technology. It can be read and written. Let's call this state BRAND_NEW. > > 2. Bind to a VM. > > Now it's an a bound state. For TDX, for example, let's call the new state BOUND_TDX. In this state, the TDX rules are followed (private memory can't be converted, etc). > > The problem here is that the BOUND_NEW state allows things that are nonsensical in TDX, and the binding step needs to invent some kind of semantics for what happens when binding a nonempty memfile. > > > So I would propose a somewhat different order: > > 1. Create a memfile. It's in the UNBOUND state and no operations whatsoever are allowed except binding or closing. OK, so we need invent new user API to indicate UNBOUND state. For memfd based, it can be a new feature-neutral flag at creation time. > > 2. Bind the memfile to a VM (or at least to a VM technology). Now it's in the initial state appropriate for that VM. > > For TDX, this completely bypasses the cases where the data is prepopulated and TDX can't handle it cleanly. For SEV, it bypasses a situation in which data might be written to the memory before we find out whether that data will be unreclaimable or unmovable. This sounds a more strict rule to avoid semantics unclear. So userspace needs to know what excatly happens for a 'bind' operation. This is different when binds to different technologies. E.g. for SEV, it may imply after this call, the memfile can be accessed (through mmap or what ever) from userspace, while for current TDX this should be not allowed. And I feel we still need a third flow/operation to indicate the completion of the initialization on the memfile before the guest's first-time launch. SEV needs to check previous mmap-ed areas are munmap-ed and prevent future userspace access. After this point, then the memfile becomes truely private fd. > > > ---------------------------------------------- > > Now I have a question, since I don't think anyone has really answered it: how does this all work with SEV- or pKVM-like technologies in which private and shared pages share the same address space? I sounds like you're proposing to have a big memfile that contains private and shared pages and to use that same memfile as pages are converted back and forth. IO and even real physical DMA could be done on that memfile. Am I understanding correctly? For TDX case, and probably SEV as well, this memfile contains private memory only. But this design at least makes it possible for usage cases like pKVM which wants both private/shared memory in the same memfile and rely on other ways like mmap/munmap or mprotect to toggle private/shared instead of fallocate/hole punching. > > If so, I think this makes sense, but I'm wondering if the actual memslot setup should be different. For TDX, private memory lives in a logically separate memslot space. For SEV and pKVM, it doesn't. I assume the API can reflect this straightforwardly. I believe so. The flow should be similar but we do need pass different flags during the 'bind' to the backing store for different usages. That should be some new flags for pKVM but the callbacks (API here) between memfile_notifile and its consumers can be reused. > > And the corresponding TDX question: is the intent still that shared pages aren't allowed at all in a TDX memfile? If so, that would be the most direct mapping to what the hardware actually does. Exactly. TDX will still use fallocate/hole punching to turn on/off the private page. Once off, the traditional shared page will become effective in KVM. Chao > > --Andy