From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 365B3C433F5 for ; Sat, 8 Oct 2022 16:15:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3507B6B0072; Sat, 8 Oct 2022 12:15:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FF5C6B0073; Sat, 8 Oct 2022 12:15:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 157C56B0074; Sat, 8 Oct 2022 12:15:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F12666B0072 for ; Sat, 8 Oct 2022 12:15:21 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B9983160AF3 for ; Sat, 8 Oct 2022 16:15:21 +0000 (UTC) X-FDA: 79998282042.26.972EAD7 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 3104C100016 for ; Sat, 8 Oct 2022 16:15:20 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 283906068C; Sat, 8 Oct 2022 16:15:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A207C433D6; Sat, 8 Oct 2022 16:15:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1665245719; bh=NIgNigZexN067z2ad+UvtpUHM+JSZc0JYM0evS6yNx0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=VAGeVVDf6KIOhTWVugZOJTuKPzDIFSGBDrGjIzZCQaYbWM1BU4NBt3yHdh/p+QB8Z Mx6iHaLe+DEGh+bDFLK83w/Rr0LUpsfWq5pCWNpvvFsJhEEnvolwGToq7kVMYm16C7 QcypgNRE4zg87Wqm9iJT8rOYJDbQxSGv57JHugrmcHqPoIxRdFRiHZJqRPFSyCDMtn VuP1L+Zw3IgC/rMVlFF4fXoapnhUapUQB/vBk7S49rXo0meL1hSUbsDRuB/MFvpsc1 IrKiCJ6IemXXuaxmEzEV+cS22e0yEOJl6KXc6pQBJg1bl7a6cOxdRgEbitVsWeCajN r4dGxRV6nD9YQ== Date: Sat, 8 Oct 2022 19:15:13 +0300 From: Jarkko Sakkinen To: Sean Christopherson Cc: Chao Peng , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, qemu-devel@nongnu.org, Paolo Bonzini , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H . Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Shuah Khan , Mike Rapoport , Steven Price , "Maciej S . Szmigiero" , Vlastimil Babka , Vishal Annapurve , Yu Zhang , "Kirill A . Shutemov" , luto@kernel.org, jun.nakajima@intel.com, dave.hansen@intel.com, ak@linux.intel.com, david@redhat.com, aarcange@redhat.com, ddutile@redhat.com, dhildenb@redhat.com, Quentin Perret , Michael Roth , mhocko@suse.com, Muchun Song , wei.w.wang@intel.com Subject: Re: [PATCH v8 2/8] KVM: Extend the memslot to support fd-based private memory Message-ID: References: <20220915142913.2213336-1-chao.p.peng@linux.intel.com> <20220915142913.2213336-3-chao.p.peng@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665245721; a=rsa-sha256; cv=none; b=uxCyznOyRA6v2G4NzZkQy1+JkINANctKvZgw4lzS7roy4fy1VwXSwvDBqjOOU7prGfMIWb +qQ34DIQbSk+cewE9ZuJFXAn3LrBWB75hWfHTAiljlJLNmdGqfA095YdXKz96i3WCLELfe LO+e3nzWV4z3nqWoG+b/nqB2liyFEFM= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VAGeVVDf; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf05.hostedemail.com: domain of jarkko@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jarkko@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665245721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/UsMdOtVkUvX57Nnk7pCVAkg+XmQ5yH9N7x2AknFg5E=; b=Xz/peh7PS7M5ygyvAeui69RtHouFzswm7jK+297g+jYvczKkrFh5UZVp0t6UY0M6aQt/pc yHUefj+JogDBBK84zKM4wO7HqN53C+PS4SrFI7/tg3FBqz1AIPkaqoSw/YAcrzcrSpzZKN NXE+nnSiM1XQiFx9JPaQdGmavdXhn8I= X-Stat-Signature: z9hztfpxqqq837yhkjwqu8op1aij7k4r X-Rspamd-Queue-Id: 3104C100016 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VAGeVVDf; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf05.hostedemail.com: domain of jarkko@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jarkko@kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1665245720-192126 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Oct 08, 2022 at 12:54:32AM +0300, Jarkko Sakkinen wrote: > On Fri, Oct 07, 2022 at 02:58:54PM +0000, Sean Christopherson wrote: > > On Fri, Oct 07, 2022, Jarkko Sakkinen wrote: > > > On Thu, Oct 06, 2022 at 03:34:58PM +0000, Sean Christopherson wrote: > > > > On Thu, Oct 06, 2022, Jarkko Sakkinen wrote: > > > > > On Thu, Oct 06, 2022 at 05:58:03PM +0300, Jarkko Sakkinen wrote: > > > > > > On Thu, Sep 15, 2022 at 10:29:07PM +0800, Chao Peng wrote: > > > > > > > This new extension, indicated by the new flag KVM_MEM_PRIVATE, adds two > > > > > > > additional KVM memslot fields private_fd/private_offset to allow > > > > > > > userspace to specify that guest private memory provided from the > > > > > > > private_fd and guest_phys_addr mapped at the private_offset of the > > > > > > > private_fd, spanning a range of memory_size. > > > > > > > > > > > > > > The extended memslot can still have the userspace_addr(hva). When use, a > > > > > > > single memslot can maintain both private memory through private > > > > > > > fd(private_fd/private_offset) and shared memory through > > > > > > > hva(userspace_addr). Whether the private or shared part is visible to > > > > > > > guest is maintained by other KVM code. > > > > > > > > > > > > What is anyway the appeal of private_offset field, instead of having just > > > > > > 1:1 association between regions and files, i.e. one memfd per region? > > > > > > > > Modifying memslots is slow, both in KVM and in QEMU (not sure about Google's VMM). > > > > E.g. if a vCPU converts a single page, it will be forced to wait until all other > > > > vCPUs drop SRCU, which can have severe latency spikes, e.g. if KVM is faulting in > > > > memory. KVM's memslot updates also hold a mutex for the entire duration of the > > > > update, i.e. conversions on different vCPUs would be fully serialized, exacerbating > > > > the SRCU problem. > > > > > > > > KVM also has historical baggage where it "needs" to zap _all_ SPTEs when any > > > > memslot is deleted. > > > > > > > > Taking both a private_fd and a shared userspace address allows userspace to convert > > > > between private and shared without having to manipulate memslots. > > > > > > Right, this was really good explanation, thank you. > > > > > > Still wondering could this possibly work (or not): > > > > > > 1. Union userspace_addr and private_fd. > > > > No, because userspace needs to be able to provide both userspace_addr (shared > > memory) and private_fd (private memory) for a single memslot. > > Got it, thanks for clearing my misunderstandings on this topic, and it > is quite obviously visible in 5/8 and 7/8. I.e. if I got it right, > memblock can be partially private, and you dig the shared holes with > KVM_MEMORY_ENCRYPT_UNREG_REGION. We have (in Enarx) ATM have memblock > per host mmap, I was looking into this dilated by that mindset but makes > definitely sense to support that. For me the most useful reference with this feature is kvm_set_phys_mem() implementation in privmem-v8 branch. Took while to find it because I did not have much experience with QEMU code base. I'd even recommend to mention that function in the cover letter because it is really good reference on how this feature is supposed to be used. BR, Jarkko