From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A82EC63797 for ; Fri, 6 Jan 2023 21:53:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6D5C8E0002; Fri, 6 Jan 2023 16:53:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF53E8E0001; Fri, 6 Jan 2023 16:53:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BCF98E0002; Fri, 6 Jan 2023 16:53:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 89B1B8E0001 for ; Fri, 6 Jan 2023 16:53:45 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F5B9120F0C for ; Fri, 6 Jan 2023 21:53:45 +0000 (UTC) X-FDA: 80325726810.25.3510ED2 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf06.hostedemail.com (Postfix) with ESMTP id 6DEF718000A for ; Fri, 6 Jan 2023 21:53:42 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=D0IaFSnD; spf=pass (imf06.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673042023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aVgaWWMITeFqFkYRhCPvuQUJAySpmdgbnuKPkfwH4L8=; b=4RK+/es4CYD5iQGr11JX2vN/sSMm/Efs0Qh8M3M9vPDNsbOR2wYAp3fpj81pFoR8H/S78G muIXzSY/pdCEgb6BsX6K+JV09PHsDGBLC5YxTHTy9aU6vLVT2DzXN88+1VNr+Rf4E9dFSD j1e/b30Nzbr3hcl0WVlAN7+VnoWfUuk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=D0IaFSnD; spf=pass (imf06.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673042023; a=rsa-sha256; cv=none; b=iCPntECbdjKof0OqWCAHv67m4fsHAF65+24PBYKhYIIVydyNtUGVkiy09aYAqbnjoJhV8D 3Rx2Leev35r27C5fY7Fh2Vpx1zhTRR+VOkWggwfxh4I2PQChoskBd1hOWA/5tUfxoIIP0z TTLs71pMKIoVkjRN9c6flouv8YRf+AY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673042022; x=1704578022; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=700/gsUIjT1YnoH0baKzs/gmLcJb6q0I2gHbT2Y3JXA=; b=D0IaFSnDMvaZrWi4dkaY2dqSqLdQGJGw1b162uc98hSgskY6EFK/RGwv mFVKkxAxUFvYU1sbctUKDzYz5Mk/zZRCcqKgZYPD8Jsi3GEY09nGAG3rG igE3eBbVXIvieWv0Y9c9dp25EX8L+nyPz4PUNzsGErIOwB9d2SUjqgvy1 pq81kpYqSFvg+zDYBPv/fV1pUQWdijFQZa/fSswAsoTRVtSoGpY8mcXWS /8fiDKQBzjBP8BMfY8iLQIHMLEc/DR9Y9VpuHWFIrD/5YKI66usYicFeQ L5B+14FfGkjkJuGi2RYZwgIf+GgZNproqa7bQwjYwOTOhg/94QLjXzkRB Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10582"; a="320277677" X-IronPort-AV: E=Sophos;i="5.96,306,1665471600"; d="scan'208";a="320277677" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jan 2023 13:53:40 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10582"; a="830033783" X-IronPort-AV: E=Sophos;i="5.96,306,1665471600"; d="scan'208";a="830033783" Received: from xiangyuy-mobl.amr.corp.intel.com (HELO [10.212.251.186]) ([10.212.251.186]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jan 2023 13:53:40 -0800 Message-ID: <50ad132f-463f-3b61-6aab-b96a57b6f3a1@intel.com> Date: Fri, 6 Jan 2023 13:53:40 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [PATCH v8 10/16] x86/virt/tdx: Allocate and set up PAMTs for TDMRs Content-Language: en-US To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, peterz@infradead.org, tglx@linutronix.de, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, kirill.shutemov@linux.intel.com, ying.huang@intel.com, reinette.chatre@intel.com, len.brown@intel.com, tony.luck@intel.com, ak@linux.intel.com, isaku.yamahata@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: <76a17c574da18c10ad7a4f96e010697aaa5f7c04.1670566861.git.kai.huang@intel.com> From: Dave Hansen In-Reply-To: <76a17c574da18c10ad7a4f96e010697aaa5f7c04.1670566861.git.kai.huang@intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: t7d4s9ybgbwminokk64ur6r6fs6rkxx3 X-Rspam-User: X-Rspamd-Queue-Id: 6DEF718000A X-Rspamd-Server: rspam06 X-HE-Tag: 1673042022-302665 X-HE-Meta: U2FsdGVkX19SnEU9rFqnpsWQVGTd5K/fzRIZv2P+GHTBN+3QxHC9MO5v2EVverOHxFZ9tL4U+BmvKUIix7yc7IAljXsT2KQ5fYPptAuredNdjc/Gxy9xs8XHj+orZYzUDiBxxVDPgCL6xSvCauatZSBxLh2hjjhohwDu/hhg1VjzpSKiIPQIRfEiw+TKt30/UpkmpJ2A9F/DwWo/mD3e8flv8gdPO6t1RjBQLLfznXqBOZUQ+KfhFgjIyDDoEHK3SCDqVjbXoMxZRq9q4+QWwQdfmyedQNFsMC1OK/wuqnnU77MGBfLTHK23YWcz5pOwvlkEWuOzYYcmqU3FSP/eeH2Be6ZLEosVB0Cn09jlqC26iZbePXs4WtkMH9wADzxssWwwUvWzOAEJgd65LVhM+iUZTwGbpPXHOf3QOPo7JqrdDSDucJBIcPewpXPIur4FW6fP4rKbBkCorLBiyoR9I3kG/lR1qLNuAIMvgYjHayCOJ+W4II+Y1GvXLgMOQvVct1iziuobA2Z8xLzy0wa6g7o9rD77T4kjspGhXnjgCldKJh9U8ToTPP9Ft6MxlbtouEL7ZCstfmYY5cXVegNM6CBvQmeP1LirU2XGQf1DzPVRtZww2BYrZdydsKcjsTy96TH6Sx+4uzWPZ1T3H6D0YhrJaIh6Pai2dJlLyzm6xfPKvEEiAHHlMhRln6wOGqcnHlXgvWzLGLr+JQKehpjaMuNtEoJg1H+i/iVhXiNisyT9gm7DSZeGJe4mKtrq/I1RZANif06V5UAOwPZyyqsZSMxR6FUUi2iK0RmvBhU3yAjMo7GNsJybkVYGr8pR1Kb2ADCA7LXEaXRvHePUNaKJ2pyQhE0Xum7GCHFvk0ouL1k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Looks good so far. > +/* > + * Allocate PAMTs from the local NUMA node of some memory in @tmb_list > + * within @tdmr, and set up PAMTs for @tdmr. > + */ > +static int tdmr_set_up_pamt(struct tdmr_info *tdmr, > + struct list_head *tmb_list, > + u16 pamt_entry_size) > +{ > + unsigned long pamt_base[TDX_PS_1G + 1]; > + unsigned long pamt_size[TDX_PS_1G + 1]; Nit: please define a TDX_PS_NR rather than open-coding this. > + unsigned long tdmr_pamt_base; > + unsigned long tdmr_pamt_size; > + struct page *pamt; > + int pgsz, nid; > + > + nid = tdmr_get_nid(tdmr, tmb_list); > + > + /* > + * Calculate the PAMT size for each TDX supported page size > + * and the total PAMT size. > + */ > + tdmr_pamt_size = 0; > + for (pgsz = TDX_PS_4K; pgsz <= TDX_PS_1G ; pgsz++) { > + pamt_size[pgsz] = tdmr_get_pamt_sz(tdmr, pgsz, > + pamt_entry_size); This alignment is wonky. Should be way over here: > + pamt_entry_size); > + tdmr_pamt_size += pamt_size[pgsz]; > + } > + > + /* > + * Allocate one chunk of physically contiguous memory for all > + * PAMTs. This helps minimize the PAMT's use of reserved areas > + * in overlapped TDMRs. > + */ > + pamt = alloc_contig_pages(tdmr_pamt_size >> PAGE_SHIFT, GFP_KERNEL, > + nid, &node_online_map); > + if (!pamt) > + return -ENOMEM; > + > + /* > + * Break the contiguous allocation back up into the > + * individual PAMTs for each page size. > + */ > + tdmr_pamt_base = page_to_pfn(pamt) << PAGE_SHIFT; > + for (pgsz = TDX_PS_4K; pgsz <= TDX_PS_1G; pgsz++) { > + pamt_base[pgsz] = tdmr_pamt_base; > + tdmr_pamt_base += pamt_size[pgsz]; > + } > + > + tdmr->pamt_4k_base = pamt_base[TDX_PS_4K]; > + tdmr->pamt_4k_size = pamt_size[TDX_PS_4K]; > + tdmr->pamt_2m_base = pamt_base[TDX_PS_2M]; > + tdmr->pamt_2m_size = pamt_size[TDX_PS_2M]; > + tdmr->pamt_1g_base = pamt_base[TDX_PS_1G]; > + tdmr->pamt_1g_size = pamt_size[TDX_PS_1G]; > + > + return 0; > +} > + > +static void tdmr_get_pamt(struct tdmr_info *tdmr, unsigned long *pamt_pfn, > + unsigned long *pamt_npages) > +{ > + unsigned long pamt_base, pamt_sz; > + > + /* > + * The PAMT was allocated in one contiguous unit. The 4K PAMT > + * should always point to the beginning of that allocation. > + */ > + pamt_base = tdmr->pamt_4k_base; > + pamt_sz = tdmr->pamt_4k_size + tdmr->pamt_2m_size + tdmr->pamt_1g_size; > + > + *pamt_pfn = PHYS_PFN(pamt_base); > + *pamt_npages = pamt_sz >> PAGE_SHIFT; > +} > + > +static void tdmr_free_pamt(struct tdmr_info *tdmr) > +{ > + unsigned long pamt_pfn, pamt_npages; > + > + tdmr_get_pamt(tdmr, &pamt_pfn, &pamt_npages); > + > + /* Do nothing if PAMT hasn't been allocated for this TDMR */ > + if (!pamt_npages) > + return; > + > + if (WARN_ON_ONCE(!pamt_pfn)) > + return; > + > + free_contig_range(pamt_pfn, pamt_npages); > +} > + > +static void tdmrs_free_pamt_all(struct tdmr_info_list *tdmr_list) > +{ > + int i; > + > + for (i = 0; i < tdmr_list->nr_tdmrs; i++) > + tdmr_free_pamt(tdmr_entry(tdmr_list, i)); > +} > + > +/* Allocate and set up PAMTs for all TDMRs */ > +static int tdmrs_set_up_pamt_all(struct tdmr_info_list *tdmr_list, > + struct list_head *tmb_list, > + u16 pamt_entry_size) > +{ > + int i, ret = 0; > + > + for (i = 0; i < tdmr_list->nr_tdmrs; i++) { > + ret = tdmr_set_up_pamt(tdmr_entry(tdmr_list, i), tmb_list, > + pamt_entry_size); > + if (ret) > + goto err; > + } > + > + return 0; > +err: > + tdmrs_free_pamt_all(tdmr_list); > + return ret; > +} > + > +static unsigned long tdmrs_count_pamt_pages(struct tdmr_info_list *tdmr_list) > +{ > + unsigned long pamt_npages = 0; > + int i; > + > + for (i = 0; i < tdmr_list->nr_tdmrs; i++) { > + unsigned long pfn, npages; > + > + tdmr_get_pamt(tdmr_entry(tdmr_list, i), &pfn, &npages); > + pamt_npages += npages; > + } > + > + return pamt_npages; > +} > + > /* > * Construct a list of TDMRs on the preallocated space in @tdmr_list > * to cover all TDX memory regions in @tmb_list based on the TDX module > @@ -506,15 +702,19 @@ static int construct_tdmrs(struct list_head *tmb_list, > if (ret) > goto err; > > + ret = tdmrs_set_up_pamt_all(tdmr_list, tmb_list, > + sysinfo->pamt_entry_size); > + if (ret) > + goto err; > /* > * TODO: > * > - * - Allocate and set up PAMTs for each TDMR. > * - Designate reserved areas for each TDMR. > * > * Return -EINVAL until constructing TDMRs is done > */ > ret = -EINVAL; > + tdmrs_free_pamt_all(tdmr_list); > err: > return ret; > } > @@ -574,6 +774,11 @@ static int init_tdx_module(void) > * Return error before all steps are done. > */ > ret = -EINVAL; > + if (ret) > + tdmrs_free_pamt_all(&tdmr_list); > + else > + pr_info("%lu pages allocated for PAMT.\n", > + tdmrs_count_pamt_pages(&tdmr_list)); > out_free_tdmrs: > /* > * Free the space for the TDMRs no matter the initialization is Other than the two nits: Reviewed-by: Dave Hansen