From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFF72EB64DC for ; Tue, 11 Jul 2023 11:42:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5409E6B0075; Tue, 11 Jul 2023 07:42:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F1166B0078; Tue, 11 Jul 2023 07:42:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BA746B007B; Tue, 11 Jul 2023 07:42:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2DE1C6B0075 for ; Tue, 11 Jul 2023 07:42:20 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DD77C1201FF for ; Tue, 11 Jul 2023 11:42:19 +0000 (UTC) X-FDA: 80999142798.10.BB48CB8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 7E56320020 for ; Tue, 11 Jul 2023 11:42:16 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KCqyOyuI; spf=pass (imf03.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689075736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TVNlN8Fu7HKmMU86W9Jarak9bhN5uqIsTvRRP5Jak7k=; b=TaX4d9T1ze81zl02dsEFAAQECfZHbKVBm0ldZfaSvFBfeRT+L4OoVJylE0r5qw1s6JnWfd M+HBl6tuNaajGJVLiruMhOLBjvkZRXnSLCKTQPULkikcT5W7LH242wQYsho7eIr3ssp6Ld 5nUeBEijyGp7qrSLvG9Nqp3Dy3p6VRg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689075736; a=rsa-sha256; cv=none; b=ERPnj/9BxCCNhbIUWJBLhJSNAN1DuIzvkxXYtVxdNe0t811UO82duVp8jDB1Yy9k5hUnZY FpXxihjXH6KDKczY2k/fDJrejccSRPMw3gT8lRDM3EmcwRU1flX6903v5CQednG9qp+Ja/ djYOfAvuhw4Rskv6B5XMmhByCT8vWwo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KCqyOyuI; spf=pass (imf03.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689075735; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TVNlN8Fu7HKmMU86W9Jarak9bhN5uqIsTvRRP5Jak7k=; b=KCqyOyuInjXCSkaL4exPNgn7PQOZypf/+FHH7G2zdBGhg7OKMBk1DDZ6odFtaQu9yDHQmj DgZptEP7c5WkImYHMbW4/bAhdFHCD7jr/jXcsmlVi2x/aNsiaC5UFlRGcoh6vnSCKzMi0d 4rD9VSLgA26BPfc5rY/EzQ9iK5wMT8w= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-497-y1wXHdfiO9eQDDQKVoWYHw-1; Tue, 11 Jul 2023 07:42:14 -0400 X-MC-Unique: y1wXHdfiO9eQDDQKVoWYHw-1 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-3141325af78so3458822f8f.3 for ; Tue, 11 Jul 2023 04:42:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689075733; x=1691667733; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TVNlN8Fu7HKmMU86W9Jarak9bhN5uqIsTvRRP5Jak7k=; b=L+o/fK7kY9OYMWizHAXPEDO1jSpboEFnHwAwqneCkP1BejuCHNEyuAYhQtcq9y/19J 5n5k87ELc0dQX+u/tDm7pnhhHVbU1e19qBCelV0amAzZUFX3ofeyd++w9T9TgdclBPLZ 2nUICObZyBMae/QeaJyHkAXkYBKO35a0aFaY9i2q6RAvqsAyp2R0XmqRMnR3C+LbOxK7 Nv7Y/Uhz6hKmCMGlVis/KPNN5UKb0rOn2/bSWTDjAjnIScO7UjCIYbTELqjaN6bXWriC 15MWjZaZGMy/rlJSzSC9VK3ZzOdpN8rPoXyzXvCco64iV33eWmH5QZ9Jtl8Y/j5z27gp CqFg== X-Gm-Message-State: ABy/qLYhjNuGyqjglUQr6tezZwy7Oq40bIZDzX70jsejygpsTShrzrdS APreKukhA4wBWhNyWEovaX+JdayXaTqyLcIQoJmOg2Rv7LpV0FYV8IPANM7KCNyxJm3RlZ4suKl ZdqlQdRcmki8= X-Received: by 2002:a05:6000:370:b0:313:f9a0:c530 with SMTP id f16-20020a056000037000b00313f9a0c530mr16858086wrf.52.1689075733284; Tue, 11 Jul 2023 04:42:13 -0700 (PDT) X-Google-Smtp-Source: APBJJlG930FaW9+FN5ODUew7AowqA5AQd10iiEj46Z4R/Iy1KQe+FGD8mJ1gLZStu1NIkLAOOBxXqA== X-Received: by 2002:a05:6000:370:b0:313:f9a0:c530 with SMTP id f16-20020a056000037000b00313f9a0c530mr16858053wrf.52.1689075732814; Tue, 11 Jul 2023 04:42:12 -0700 (PDT) Received: from ?IPV6:2003:cb:c745:4000:13ad:ed64:37e6:115d? (p200300cbc745400013aded6437e6115d.dip0.t-ipconnect.de. [2003:cb:c745:4000:13ad:ed64:37e6:115d]) by smtp.gmail.com with ESMTPSA id s15-20020adff80f000000b00313f9a0c521sm2045310wrp.107.2023.07.11.04.42.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Jul 2023 04:42:12 -0700 (PDT) Message-ID: <68a8cba5-55b6-4c8a-f4e6-d528fc839285@redhat.com> Date: Tue, 11 Jul 2023 13:42:10 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v12 12/22] x86/virt/tdx: Allocate and set up PAMTs for TDMRs To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, dave.hansen@intel.com, kirill.shutemov@linux.intel.com, tony.luck@intel.com, peterz@infradead.org, tglx@linutronix.de, bp@alien8.de, mingo@redhat.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, ashok.raj@intel.com, reinette.chatre@intel.com, len.brown@intel.com, ak@linux.intel.com, isaku.yamahata@intel.com, ying.huang@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, nik.borisov@suse.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: <85ea233226ec7a05e8c5627a499e97ea4cbd6950.1687784645.git.kai.huang@intel.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <85ea233226ec7a05e8c5627a499e97ea4cbd6950.1687784645.git.kai.huang@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: fm8batfj5xt1wp56hagdzxe1k7idu34p X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7E56320020 X-Rspam-User: X-HE-Tag: 1689075736-946928 X-HE-Meta: U2FsdGVkX18+cYr9T1d1YVaB6ljGGmrRg1J+IX9eIpCpHlz6wVk7l2o3lH3igxqNifhPMJgcCWP4nbun9BvZ5upIDDymO8kahaiQ5lr/H5szA7OhVkiiUc8XhIf9xz0+kjED7Cs2SuMQ4Q7HQn81YNIqFT/qdHA4QXgyzG1ZlJe4F4Yb+kOZYnu5pRvUsqg/qMRycjgbBAByqz+8qNjMOQyDS4bwhdyQEw1tm4+iEl9lY+vhF3WQd9T55ZPfwbK6dV30eETOznjXst8XKI0yVdxegXqLFyXMNfdjDS+MRkTvJG109V+eP8atg5/6egLS7SL+orDC+4IAo801JmX4vFMVMpu/Z3cX2yK5rwHkqVmuB5+7h0x/DUW5wfqOHny2+1yr9TX7xNwt8Jb9n6w07sGLLIEzx5/QGiFKdX3/2VgRUduNsz7zKvl8L4ECL89dlsEwB54QHhYlHB8AnpJlKi2wwiT22c+DaVz5utRwubG9F8o3rpcfUr+N/ZsLn3hTJ7xOVqz3nIlgBJBWdg4IJGVRJ+yW3HR78QgnRkiDrITG1rJidfDnZ6rD0udlyN0poBW8Vpm9mFGjq1dOCc2t8uyGuRzZU0SdeLejI2wAkUTw43CNlHIZiCocMn7H4axuFzfX6lB+biRInNoIAQnGn5AOkXDGT6/PViJOI+sdxZVxfpCr/uiRBkAlNTkfoDBvfXKYKsux4lnLtFJN6j/Ge/0Ffcd93iq2gaKB6nyZsPpwuqnDVv8tV0uWQGbOlkRjtSlj3wytlKzE7+iftkM29DZDSjYwbLcjwRTrSvUyxz557OsR+BE82lYATPM3ZRVQY5BU5IHsE2xk1uLFIARcDEUCSMg3djMeSIK/7FSl3AqLL8MTo1rfg9tW6SqkW/ETOCbHxxNf8DsuVXGWukuQkh/3csNFd6r3b/YX7JIgPQSrNx5qzBCRFg4j4KzOsU2hRONIpWobdeYQlcBZCVj nMEQ+QB3 mQiYr3lFbZCsitfKSPHQS4WvXsY1Poay8RSYF6XRh3bpbI6iqLRgGbuU1W8TyGiYaYsPpMx2Z789zMCbWRk6hoo+rD4WJ0WppqJ0hK1d6bAsTZZ+ydKnr0xUolo5sLOqqyl+xGN9XqlhsdnFnTqXSKWhzfDJIZDLguF6eCA72g348EDq9MI+jo7R6A5LQB68PEhWcpthROoHcgbJtOTUpoAx9zmSRqQdJMdEH/8pUjLBCBk05C/kBqYJQy/Uz+WvrbQfdLqq7r0TRBFZfiOz/OkjqgQJJYN1fJS21wSI/z8AszwnJaYclvIPDwSknyPgoSSz+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 26.06.23 16:12, Kai Huang wrote: > The TDX module uses additional metadata to record things like which > guest "owns" a given page of memory. This metadata, referred as > Physical Address Metadata Table (PAMT), essentially serves as the > 'struct page' for the TDX module. PAMTs are not reserved by hardware > up front. They must be allocated by the kernel and then given to the > TDX module during module initialization. > > TDX supports 3 page sizes: 4K, 2M, and 1G. Each "TD Memory Region" > (TDMR) has 3 PAMTs to track the 3 supported page sizes. Each PAMT must > be a physically contiguous area from a Convertible Memory Region (CMR). > However, the PAMTs which track pages in one TDMR do not need to reside > within that TDMR but can be anywhere in CMRs. If one PAMT overlaps with > any TDMR, the overlapping part must be reported as a reserved area in > that particular TDMR. > > Use alloc_contig_pages() since PAMT must be a physically contiguous area > and it may be potentially large (~1/256th of the size of the given TDMR). > The downside is alloc_contig_pages() may fail at runtime. One (bad) > mitigation is to launch a TDX guest early during system boot to get > those PAMTs allocated at early time, but the only way to fix is to add a > boot option to allocate or reserve PAMTs during kernel boot. > > It is imperfect but will be improved on later. > > TDX only supports a limited number of reserved areas per TDMR to cover > both PAMTs and memory holes within the given TDMR. If many PAMTs are > allocated within a single TDMR, the reserved areas may not be sufficient > to cover all of them. > > Adopt the following policies when allocating PAMTs for a given TDMR: > > - Allocate three PAMTs of the TDMR in one contiguous chunk to minimize > the total number of reserved areas consumed for PAMTs. > - Try to first allocate PAMT from the local node of the TDMR for better > NUMA locality. > > Also dump out how many pages are allocated for PAMTs when the TDX module > is initialized successfully. This helps answer the eternal "where did > all my memory go?" questions. > > Signed-off-by: Kai Huang > Reviewed-by: Isaku Yamahata > Reviewed-by: Dave Hansen > --- > > v11 -> v12: > - Moved TDX_PS_NUM from tdx.c to (Kirill) > - "<= TDX_PS_1G" -> "< TDX_PS_NUM" (Kirill) > - Changed tdmr_get_pamt() to return base and size instead of base_pfn > and npages and related code directly (Dave). > - Simplified PAMT kb counting. (Dave) > - tdmrs_count_pamt_pages() -> tdmr_count_pamt_kb() (Kirill/Dave) > > v10 -> v11: > - No update > > v9 -> v10: > - Removed code change in disable_tdx_module() as it doesn't exist > anymore. > > v8 -> v9: > - Added TDX_PS_NR macro instead of open-coding (Dave). > - Better alignment of 'pamt_entry_size' in tdmr_set_up_pamt() (Dave). > - Changed to print out PAMTs in "KBs" instead of "pages" (Dave). > - Added Dave's Reviewed-by. > > v7 -> v8: (Dave) > - Changelog: > - Added a sentence to state PAMT allocation will be improved. > - Others suggested by Dave. > - Moved 'nid' of 'struct tdx_memblock' to this patch. > - Improved comments around tdmr_get_nid(). > - WARN_ON_ONCE() -> pr_warn() in tdmr_get_nid(). > - Other changes due to 'struct tdmr_info_list'. > > v6 -> v7: > - Changes due to using macros instead of 'enum' for TDX supported page > sizes. > > v5 -> v6: > - Rebase due to using 'tdx_memblock' instead of memblock. > - 'int pamt_entry_nr' -> 'unsigned long nr_pamt_entries' (Dave/Sagis). > - Improved comment around tdmr_get_nid() (Dave). > - Improved comment in tdmr_set_up_pamt() around breaking the PAMT > into PAMTs for 4K/2M/1G (Dave). > - tdmrs_get_pamt_pages() -> tdmrs_count_pamt_pages() (Dave). > > - v3 -> v5 (no feedback on v4): > - Used memblock to get the NUMA node for given TDMR. > - Removed tdmr_get_pamt_sz() helper but use open-code instead. > - Changed to use 'switch .. case..' for each TDX supported page size in > tdmr_get_pamt_sz() (the original __tdmr_get_pamt_sz()). > - Added printing out memory used for PAMT allocation when TDX module is > initialized successfully. > - Explained downside of alloc_contig_pages() in changelog. > - Addressed other minor comments. > > > --- > arch/x86/Kconfig | 1 + > arch/x86/include/asm/tdx.h | 1 + > arch/x86/virt/vmx/tdx/tdx.c | 215 +++++++++++++++++++++++++++++++++++- > arch/x86/virt/vmx/tdx/tdx.h | 1 + > 4 files changed, 213 insertions(+), 5 deletions(-) > > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig > index 2226d8a4c749..ad364f01de33 100644 > --- a/arch/x86/Kconfig > +++ b/arch/x86/Kconfig > @@ -1959,6 +1959,7 @@ config INTEL_TDX_HOST > depends on KVM_INTEL > depends on X86_X2APIC > select ARCH_KEEP_MEMBLOCK > + depends on CONTIG_ALLOC > help > Intel Trust Domain Extensions (TDX) protects guest VMs from malicious > host and certain physical attacks. This option enables necessary TDX > diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h > index d8226a50c58c..91416fd600cd 100644 > --- a/arch/x86/include/asm/tdx.h > +++ b/arch/x86/include/asm/tdx.h > @@ -24,6 +24,7 @@ > #define TDX_PS_4K 0 > #define TDX_PS_2M 1 > #define TDX_PS_1G 2 > +#define TDX_PS_NR (TDX_PS_1G + 1) > > /* > * Used to gather the output registers values of the TDCALL and SEAMCALL > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c > index 2ffc1517a93b..fd5417577f26 100644 > --- a/arch/x86/virt/vmx/tdx/tdx.c > +++ b/arch/x86/virt/vmx/tdx/tdx.c > @@ -221,7 +221,7 @@ static int tdx_get_sysinfo(struct tdsysinfo_struct *sysinfo, > * overlap. > */ > static int add_tdx_memblock(struct list_head *tmb_list, unsigned long start_pfn, > - unsigned long end_pfn) > + unsigned long end_pfn, int nid) > { > struct tdx_memblock *tmb; > > @@ -232,6 +232,7 @@ static int add_tdx_memblock(struct list_head *tmb_list, unsigned long start_pfn, > INIT_LIST_HEAD(&tmb->list); > tmb->start_pfn = start_pfn; > tmb->end_pfn = end_pfn; > + tmb->nid = nid; > > /* @tmb_list is protected by mem_hotplug_lock */ > list_add_tail(&tmb->list, tmb_list); > @@ -259,9 +260,9 @@ static void free_tdx_memlist(struct list_head *tmb_list) > static int build_tdx_memlist(struct list_head *tmb_list) > { > unsigned long start_pfn, end_pfn; > - int i, ret; > + int i, nid, ret; > > - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) { > + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { > /* > * The first 1MB is not reported as TDX convertible memory. > * Although the first 1MB is always reserved and won't end up > @@ -277,7 +278,7 @@ static int build_tdx_memlist(struct list_head *tmb_list) > * memblock has already guaranteed they are in address > * ascending order and don't overlap. > */ > - ret = add_tdx_memblock(tmb_list, start_pfn, end_pfn); > + ret = add_tdx_memblock(tmb_list, start_pfn, end_pfn, nid); > if (ret) > goto err; Why did you decide to defer remembering the nid as well? I'd just move that part to the patch that adds add_tdx_memblock(). -- Cheers, David / dhildenb