From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B71A0C28D13 for ; Wed, 8 Jun 2022 00:55:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E934E6B0072; Tue, 7 Jun 2022 20:55:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E42FD6B0073; Tue, 7 Jun 2022 20:55:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0B006B0074; Tue, 7 Jun 2022 20:55:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BEDAF6B0072 for ; Tue, 7 Jun 2022 20:55:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 910B921429 for ; Wed, 8 Jun 2022 00:55:58 +0000 (UTC) X-FDA: 79553251596.30.848779F Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf12.hostedemail.com (Postfix) with ESMTP id 18AE14003D for ; Wed, 8 Jun 2022 00:55:57 +0000 (UTC) Received: by mail-oi1-f176.google.com with SMTP id q11so10843589oih.10 for ; Tue, 07 Jun 2022 17:55:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wFH16JpjBRikHN1IAO48ehtoxfzCGCzN3oM9EiMa6X8=; b=l4i+k+6Abr5HeudzU075zO0r/BLzbacv0V6STNZIEAEBluaeTcRvx1TChTOuxyCASs uAUgVwmDgoEyG6k0dhZ8NFmw5OEbd2l74yvSxXxWA53CBaw9OoBV3iEq26EPYk4y+7IT qKn9JDYz2OP806c9cvPpDahmxZAZDvrVhTSIayulr7dYNanhmDiOChse+RJNE6HykCsj 9xuaaLmPGA8/BJDWN8cpUzOPbGbAyNe0shOzPx2NRgDBxX7ug5Bu5rZe5CkKdYsGZDwp RBwBbso17/gWhg2hEdQ9VBPL7L2J3NplhFzwna8YW9Inmvn2hE+jo34OGkRp4w12iAUj sThA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wFH16JpjBRikHN1IAO48ehtoxfzCGCzN3oM9EiMa6X8=; b=cRWifyyIeajd1q2y8lVQK4ypiSEERtW+sOEs4Xz2rKDm+nE5Kv6NuqQMXoTe8kK2H2 8znTb2wyiRMpmKKTtu5FgEKUBkqy92VmDuxzCHMjV+BEBnEWDpknNlyPRAo37sgWEZNr 92OLIuDfvVH7gJ8rYOVYkKQzOxgRvlAwRVkVs3O4YxuuozM7v2cx28koITxQcF0KQNAg 3EFQEq6QEYcbfnpwkfnzKbBu4DOGOHVqKJrOncRxXFvddofg4XQqyYJ7tDpwBkSicF9l upGaNFcLbt+mi/1SBjYBgTkSOBz1DSoly59W4XLiCLXx2jB8kW5r1KAOH7jK93aOVGnu Xf1w== X-Gm-Message-State: AOAM530O7AU/rrmOmQw+UZWlSrRfKTrlPmmJDlKed8tOOoB1kDeRFSyJ CY2M+XwpaGSkKj/XYazh2U7gPofja290WtdYVl76dw== X-Google-Smtp-Source: ABdhPJwjBSZgp55zoUZOFIm3kjCTD4NqCF0p7muhSr/F2gMJemxqIIRloSiqM46aTGwoS/mB8l6y90gRDZltreZ8pZ8= X-Received: by 2002:a05:6808:bd3:b0:32e:400d:1eb1 with SMTP id o19-20020a0568080bd300b0032e400d1eb1mr1047136oik.110.1654649757116; Tue, 07 Jun 2022 17:55:57 -0700 (PDT) MIME-Version: 1.0 References: <20220519153713.819591-1-chao.p.peng@linux.intel.com> <20220607065749.GA1513445@chaop.bj.intel.com> In-Reply-To: <20220607065749.GA1513445@chaop.bj.intel.com> From: Marc Orr Date: Tue, 7 Jun 2022 17:55:46 -0700 Message-ID: Subject: Re: [PATCH v6 0/8] KVM: mm: fd-based approach for supporting KVM guest private memory To: Chao Peng Cc: Vishal Annapurve , kvm list , LKML , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, qemu-devel@nongnu.org, Paolo Bonzini , Jonathan Corbet , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86 , "H . Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Mike Rapoport , Steven Price , "Maciej S . Szmigiero" , Vlastimil Babka , Yu Zhang , "Kirill A . Shutemov" , Andy Lutomirski , Jun Nakajima , Dave Hansen , Andi Kleen , David Hildenbrand , aarcange@redhat.com, ddutile@redhat.com, dhildenb@redhat.com, Quentin Perret , Michael Roth , mhocko@suse.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: dutf6515f4oyeb4n6sbap5e7tfwhn33p X-Rspam-User: Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=l4i+k+6A; spf=pass (imf12.hostedemail.com: domain of marcorr@google.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=marcorr@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 18AE14003D X-HE-Tag: 1654649757-355231 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Jun 7, 2022 at 12:01 AM Chao Peng wrote: > > On Mon, Jun 06, 2022 at 01:09:50PM -0700, Vishal Annapurve wrote: > > > > > > Private memory map/unmap and conversion > > > --------------------------------------- > > > Userspace's map/unmap operations are done by fallocate() ioctl on the > > > backing store fd. > > > - map: default fallocate() with mode=0. > > > - unmap: fallocate() with FALLOC_FL_PUNCH_HOLE. > > > The map/unmap will trigger above memfile_notifier_ops to let KVM map/unmap > > > secondary MMU page tables. > > > > > .... > > > QEMU: https://github.com/chao-p/qemu/tree/privmem-v6 > > > > > > An example QEMU command line for TDX test: > > > -object tdx-guest,id=tdx \ > > > -object memory-backend-memfd-private,id=ram1,size=2G \ > > > -machine q35,kvm-type=tdx,pic=no,kernel_irqchip=split,memory-encryption=tdx,memory-backend=ram1 > > > > > > > There should be more discussion around double allocation scenarios > > when using the private fd approach. A malicious guest or buggy > > userspace VMM can cause physical memory getting allocated for both > > shared (memory accessible from host) and private fds backing the guest > > memory. > > Userspace VMM will need to unback the shared guest memory while > > handling the conversion from shared to private in order to prevent > > double allocation even with malicious guests or bugs in userspace VMM. > > I don't know how malicious guest can cause that. The initial design of > this serie is to put the private/shared memory into two different > address spaces and gives usersapce VMM the flexibility to convert > between the two. It can choose respect the guest conversion request or > not. For example, the guest could maliciously give a device driver a private page so that a host-side virtual device will blindly write the private page. > It's possible for a usrspace VMM to cause double allocation if it fails > to call the unback operation during the conversion, this may be a bug > or not. Double allocation may not be a wrong thing, even in conception. > At least TDX allows you to use half shared half private in guest, means > both shared/private can be effective. Unbacking the memory is just the > current QEMU implementation choice. Right. But the idea is that this patch series should accommodate all of the CVM architectures. Or at least that's what I know was envisioned last time we discussed this topic for SNP [*]. Regardless, it's important to ensure that the VM respects its memory budget. For example, within Google, we run VMs inside of containers. So if we double allocate we're going to OOM. This seems acceptable for an early version of CVMs. But ultimately, I think we need a more robust way to ensure that the VM operates within its memory container. Otherwise, the OOM is going to be hard to diagnose and distinguish from a real OOM. [*] https://lore.kernel.org/all/20210820155918.7518-1-brijesh.singh@amd.com/ > > Chao > > > > Options to unback shared guest memory seem to be: > > 1) madvise(.., MADV_DONTNEED/MADV_REMOVE) - This option won't stop > > kernel from backing the shared memory on subsequent write accesses > > 2) fallocate(..., FALLOC_FL_PUNCH_HOLE...) - For file backed shared > > guest memory, this option still is similar to madvice since this would > > still allow shared memory to get backed on write accesses > > 3) munmap - This would give away the contiguous virtual memory region > > reservation with holes in the guest backing memory, which might make > > guest memory management difficult. > > 4) mprotect(... PROT_NONE) - This would keep the virtual memory > > address range backing the guest memory preserved > > > > ram_block_discard_range_fd from reference implementation: > > https://github.com/chao-p/qemu/tree/privmem-v6 seems to be relying on > > fallocate/madvise. > > > > Any thoughts/suggestions around better ways to unback the shared > > memory in order to avoid double allocation scenarios? I agree with Vishal. I think this patch set is making great progress. But the double allocation scenario seems like a high-level design issue that warrants more discussion.