From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C814C4332F for ; Wed, 23 Nov 2022 22:57:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0A356B007D; Wed, 23 Nov 2022 17:57:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB9406B007E; Wed, 23 Nov 2022 17:57:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C81046B0080; Wed, 23 Nov 2022 17:57:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B93146B007D for ; Wed, 23 Nov 2022 17:57:34 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 985281A023E for ; Wed, 23 Nov 2022 22:57:34 +0000 (UTC) X-FDA: 80166220428.08.6979A12 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf08.hostedemail.com (Postfix) with ESMTP id 5CF8B160013 for ; Wed, 23 Nov 2022 22:57:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669244253; x=1700780253; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=nbcy9sYYQ21ddpqNbtcQLkzP7qnkCuH1u1wrwKLYh4k=; b=g2kFxdUtDmuEsee/Ot8QrubqNpplMREpa6R9eFcRyKy3aWGxIBs0EyZV L48klBNGJ5X9R4MNODWeWi1FGgmQym6fRIs5GNbV56esyaCqVEZkLQpu+ Z9jGiw/EC9gR/LZSqrSvDBEklviGkbcp6RYNW4uEY8p5/oOSn1CcrdUYs 4sS0z0qarOzO9L7ln8j6mUc9rzJUqL4ZqU8+Dg1DD+BeNW/xarvbzojVT af2J+Zr25nrliWNrzfik2D5eXBwtIRKrY9S7rqsIAfPphTW597vCNYIre L3HQwhROsFSDyzmG2f0JWqboTYlZqAgEP0SZTeHmCrrAyf+PBbud5/6Rf A==; X-IronPort-AV: E=McAfee;i="6500,9779,10540"; a="311809769" X-IronPort-AV: E=Sophos;i="5.96,187,1665471600"; d="scan'208";a="311809769" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2022 14:57:31 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10540"; a="710749796" X-IronPort-AV: E=Sophos;i="5.96,187,1665471600"; d="scan'208";a="710749796" Received: from vcbudden-mobl3.amr.corp.intel.com (HELO [10.212.129.67]) ([10.212.129.67]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2022 14:57:29 -0800 Message-ID: <74723e2b-3094-d04b-aed7-2789268b00ab@intel.com> Date: Wed, 23 Nov 2022 14:57:28 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [PATCH v7 13/20] x86/virt/tdx: Allocate and set up PAMTs for TDMRs Content-Language: en-US To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, kirill.shutemov@linux.intel.com, ying.huang@intel.com, reinette.chatre@intel.com, len.brown@intel.com, tony.luck@intel.com, peterz@infradead.org, ak@linux.intel.com, isaku.yamahata@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669244254; a=rsa-sha256; cv=none; b=RP/A5xx5bdBqMmN+RmmF5LLuU7F+RZlgvvGf1CvSAJYcmAkJd02y+Qrv10mWWk/Qi352B1 RZeO7ceXT0anAfoKDueBGP9h1Cld9oyWLjO8oicr7EwvzZsjqbqQzRv5nhizE3hGd84YqR rAPIqERAqz54SA/p3LivaBYOpJJSQos= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=g2kFxdUt; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf08.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669244254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cLgfMR9ud9YNT7dOpkzr5L4KsNS2CGMnqytCFFk7UhM=; b=sTR/0tUFLn6reqVRhzGJ7tNU9w8hUyc9rhje94xIVRfmE57x51JLMPOrXX5972DpieX+e9 P1z0nlmkHxPZbr/vjbQK49+bsB/BZoWHv0AQpvQVRbu2hlAxvMvskGRhTmIal6nevMt4np K+W7R8+8HaGN6YunsQ8rQCvnxPQwF/w= Authentication-Results: imf08.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=g2kFxdUt; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf08.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com X-Stat-Signature: 5xcxkh7q563e4ocx68tbtr915prgew1h X-Rspamd-Queue-Id: 5CF8B160013 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1669244253-741308 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 11/20/22 16:26, Kai Huang wrote: > The TDX module uses additional metadata to record things like which > guest "owns" a given page of memory. This metadata, referred as > Physical Address Metadata Table (PAMT), essentially serves as the > 'struct page' for the TDX module. PAMTs are not reserved by hardware > up front. They must be allocated by the kernel and then given to the > TDX module. ... during module initialization. > TDX supports 3 page sizes: 4K, 2M, and 1G. Each "TD Memory Region" > (TDMR) has 3 PAMTs to track the 3 supported page sizes. Each PAMT must > be a physically contiguous area from a Convertible Memory Region (CMR). > However, the PAMTs which track pages in one TDMR do not need to reside > within that TDMR but can be anywhere in CMRs. If one PAMT overlaps with > any TDMR, the overlapping part must be reported as a reserved area in > that particular TDMR. > > Use alloc_contig_pages() since PAMT must be a physically contiguous area > and it may be potentially large (~1/256th of the size of the given TDMR). > The downside is alloc_contig_pages() may fail at runtime. One (bad) > mitigation is to launch a TD guest early during system boot to get those > PAMTs allocated at early time, but the only way to fix is to add a boot > option to allocate or reserve PAMTs during kernel boot. FWIW, we all agree that this is a bad permanent way to leave things. You can call me out here as proposing that this wart be left in place while this series is merged and is a detail we can work on afterword with new module params, boot options, Kconfig or whatever. > TDX only supports a limited number of reserved areas per TDMR to cover > both PAMTs and memory holes within the given TDMR. If many PAMTs are > allocated within a single TDMR, the reserved areas may not be sufficient > to cover all of them. > > Adopt the following policies when allocating PAMTs for a given TDMR: > > - Allocate three PAMTs of the TDMR in one contiguous chunk to minimize > the total number of reserved areas consumed for PAMTs. > - Try to first allocate PAMT from the local node of the TDMR for better > NUMA locality. > > Also dump out how many pages are allocated for PAMTs when the TDX module > is initialized successfully. ... this helps answer the eternal "where did all my memory go?" questions. > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig > index b36129183035..b86a333b860f 100644 > --- a/arch/x86/Kconfig > +++ b/arch/x86/Kconfig > @@ -1960,6 +1960,7 @@ config INTEL_TDX_HOST > depends on KVM_INTEL > depends on X86_X2APIC > select ARCH_KEEP_MEMBLOCK > + depends on CONTIG_ALLOC > help > Intel Trust Domain Extensions (TDX) protects guest VMs from malicious > host and certain physical attacks. This option enables necessary TDX > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c > index 57b448de59a0..9d76e70de46e 100644 > --- a/arch/x86/virt/vmx/tdx/tdx.c > +++ b/arch/x86/virt/vmx/tdx/tdx.c > @@ -586,6 +586,187 @@ static int create_tdmrs(struct tdmr_info *tdmr_array, int *tdmr_num) > return 0; > } > > +/* > + * Calculate PAMT size given a TDMR and a page size. The returned > + * PAMT size is always aligned up to 4K page boundary. > + */ > +static unsigned long tdmr_get_pamt_sz(struct tdmr_info *tdmr, int pgsz) > +{ > + unsigned long pamt_sz, nr_pamt_entries; > + > + switch (pgsz) { > + case TDX_PS_4K: > + nr_pamt_entries = tdmr->size >> PAGE_SHIFT; > + break; > + case TDX_PS_2M: > + nr_pamt_entries = tdmr->size >> PMD_SHIFT; > + break; > + case TDX_PS_1G: > + nr_pamt_entries = tdmr->size >> PUD_SHIFT; > + break; > + default: > + WARN_ON_ONCE(1); > + return 0; > + } > + > + pamt_sz = nr_pamt_entries * tdx_sysinfo.pamt_entry_size; > + /* TDX requires PAMT size must be 4K aligned */ > + pamt_sz = ALIGN(pamt_sz, PAGE_SIZE); > + > + return pamt_sz; > +} > + > +/* > + * Pick a NUMA node on which to allocate this TDMR's metadata. > + * > + * This is imprecise since TDMRs are 1G aligned and NUMA nodes might > + * not be. If the TDMR covers more than one node, just use the _first_ > + * one. This can lead to small areas of off-node metadata for some > + * memory. > + */ > +static int tdmr_get_nid(struct tdmr_info *tdmr) > +{ > + struct tdx_memblock *tmb; > + > + /* Find the first memory region covered by the TDMR */ > + list_for_each_entry(tmb, &tdx_memlist, list) { > + if (tmb->end_pfn > (tdmr_start(tdmr) >> PAGE_SHIFT)) > + return tmb->nid; > + } Aha, the first use of tmb->nid! I wondered why that was there. > + > + /* > + * Fall back to allocating the TDMR's metadata from node 0 when > + * no TDX memory block can be found. This should never happen > + * since TDMRs originate from TDX memory blocks. > + */ > + WARN_ON_ONCE(1); That's probably better a pr_warn() or something. A backtrace and all that jazz seems a bit overly dramatic for this. > + return 0; > +} The rest of this actually looks fine. It's nearing ack'able state.