From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61639C54EBC for ; Tue, 10 Jan 2023 16:18:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD6188E0002; Tue, 10 Jan 2023 11:18:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B869F8E0001; Tue, 10 Jan 2023 11:18:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4E3A8E0002; Tue, 10 Jan 2023 11:18:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 93C328E0001 for ; Tue, 10 Jan 2023 11:18:53 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 736E51A0DCE for ; Tue, 10 Jan 2023 16:18:53 +0000 (UTC) X-FDA: 80339398146.26.1565ED9 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf18.hostedemail.com (Postfix) with ESMTP id 534E91C0006 for ; Tue, 10 Jan 2023 16:18:50 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WB2YSRgw; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf18.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673367531; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ynb0/sJsFeb+dYt3CpHaWPUhhfZsXisAfcbNTaPxH+c=; b=uJptbKKxERE6EuvYEwQrYh/8zIQ61kC+Xd8beBtA0YTAnDwuFl1DbHUYrnbevZAz9C3/PV GMPE+uBNsTP7kMcr51ztRO+u9Y5kOW1G0zD1skMJre9fyy7LpkzzHn/SQdW0r4hHQIVxQB 3upwObmluZHtoHFxN7BHzbVacor5wF4= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WB2YSRgw; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf18.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=dave.hansen@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673367531; a=rsa-sha256; cv=none; b=LEss6MP8cO48FkOqd6Z+FQSwUlpqjN5YUKSPL2/l4mtnS8CsPT2uOovDwdIddjJjzrS79o kjlvAUGIpEOtlLofYKs7ILJ54cS+voIYAboAANDiPZqO3Ev753/rpQHOuQp58JJldorID7 9yYKBWKjsQaSUFFo5BAoN+u/Iht2o9I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673367530; x=1704903530; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=HzAV4h4PwTOTMezU8bvkGgKpyYb8PNekRFDWu1NI97I=; b=WB2YSRgw013m8eOEdUrTYJvRleYeIBMpmXcftoEfaWY+q2SHKLgHWlso l1gRt5FnD7ntANfkt2dX2KvRxXJHWoHCoH7DIbJS4QJLyihB+5+RmDEaV +MVTPzVssjc+IZ9TDi58TuTzKi/5z5jy99BDyYCr2l4ybp+IWHXuLtzEH rEeI+MsJYrL7Pm/XcmhZeQE3Ygn42uCa0BDmZYiHcpRiGROtfpiUge7fZ WtTsxzpl00LwbuklDutUOSOcHgB/lHMeLaT6wow0rcxuD4sp8OSzIaZHl 88fH0rULwB42n/KSTx5vleBkcZ5YupztXEMpWiDi1xI5BKS2QeeuZ5oIH Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="320874901" X-IronPort-AV: E=Sophos;i="5.96,315,1665471600"; d="scan'208";a="320874901" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jan 2023 08:18:48 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="725596337" X-IronPort-AV: E=Sophos;i="5.96,315,1665471600"; d="scan'208";a="725596337" Received: from svenka7-mobl1.amr.corp.intel.com (HELO [10.209.63.27]) ([10.209.63.27]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jan 2023 08:18:47 -0800 Message-ID: <16f23950-2a27-29de-c0b4-e5f2d927c8b4@intel.com> Date: Tue, 10 Jan 2023 08:18:46 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [PATCH v8 07/16] x86/virt/tdx: Use all system memory when initializing TDX module as TDX memory Content-Language: en-US To: "Huang, Kai" , "kvm@vger.kernel.org" , "linux-kernel@vger.kernel.org" Cc: "Luck, Tony" , "bagasdotme@gmail.com" , "ak@linux.intel.com" , "Wysocki, Rafael J" , "kirill.shutemov@linux.intel.com" , "Christopherson,, Sean" , "Chatre, Reinette" , "pbonzini@redhat.com" , "linux-mm@kvack.org" , "Yamahata, Isaku" , "tglx@linutronix.de" , "Shahar, Sagi" , "imammedo@redhat.com" , "Gao, Chao" , "Brown, Len" , "peterz@infradead.org" , "sathyanarayanan.kuppuswamy@linux.intel.com" , "Huang, Ying" , "Williams, Dan J" References: <8aab33a7db7a408beb403950e21f693b0b0f1f2b.1670566861.git.kai.huang@intel.com> From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 534E91C0006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: hng8mps4uwz6u34uod4i39xej85g9i9a X-HE-Tag: 1673367530-621004 X-HE-Meta: U2FsdGVkX1/ajUev2jlVz3ObuL2gTvKHhsjbWboaG+BPTR4uglkz0+k6G8KX0QdZ5gKTn0QaxkMKGDO9aWvZr1clubNmXldPahPkZl89lIKYLYmvPwQpDLtzs3B/COKX4JvEg4bla9T1ArafsYSFlIGLDCVDgjrmOYRvxhAXNjMHFO7SOkpxv1sL0l9oWyDY3lmRkVAIIFlFvvYhLlcxzUpC7btVdnqkkDsapH5rFsJRTE6BRQ+s16CScy/+EEvxd8Ay9kgRpEYQ2mt1fc9dAFINaGPDLw7rEiW6AlYDrDo4zw6+/Bo6jm38vugMZiuslczDC0TQ2mnuM63DjKGDV42K4dmCPTVB+ti1sJ4toig63ZHEB7Cr2G8abmZKA0AzwCrzftJg71NmWAjy1ntizH2b/JmNBKqk4epP5x2Zkqh4ndv6F7WH6tXfKQNqC0YVLnfIQrQbX7Ku+paOAn9LesPv4HiXEVdh+76ySN1YXs3tHKdb0F+VugiGQogQq7qMlLU5QJg9eKxCUb9uwc+qx+1ZUSg2ZULS1ju4BYOsKYUoVVZ/bOtYbkU0RLBjMUyUB8QrrY2cUpSAXHeOkLoENoB4WMyI0iaf9Hp0P9bFyDoV/gJXmyCMqn4T9qd4ng8I0f7f0hFsbpasKTlp9BQoZkEsQrCkqUiwAoGvbvh7adeyOqLu8rJVWWsy1wg9zfzsUzC2TCdMvcs9gRRIOyxjiWTWTaZTJ/DnxcXGBonIZHNya4FhvyJg8D7gqhCnjtj7YC6nbjWLey90zY4gCJ5vcT4gVwocg/US1vJ/YtLoJUSxMolV180LzZsJhnHrIyiM/J8bD5Z5n9ImgPMjZkS0MSVWZ7AEkhHmn5BtEbkA2ohEOqo+EamaTlr1w6Dgi/NH5ZQ8DVLtuZs427n8I6454Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 1/10/23 04:09, Huang, Kai wrote: > On Mon, 2023-01-09 at 08:51 -0800, Dave Hansen wrote: >> On 1/9/23 03:48, Huang, Kai wrote: >>>>>>> This can also be enhanced in the future, i.e. by allowing adding non-TDX >>>>>>> memory to a separate NUMA node. In this case, the "TDX-capable" nodes >>>>>>> and the "non-TDX-capable" nodes can co-exist, but the kernel/userspace >>>>>>> needs to guarantee memory pages for TDX guests are always allocated from >>>>>>> the "TDX-capable" nodes. >>>>> >>>>> Why does it need to be enhanced? What's the problem? >>> >>> The problem is after TDX module initialization, no more memory can be hot-added >>> to the page allocator. >>> >>> Kirill suggested this may not be ideal. With the existing NUMA ABIs we can >>> actually have both TDX-capable and non-TDX-capable NUMA nodes online. We can >>> bind TDX workloads to TDX-capable nodes while other non-TDX workloads can >>> utilize all memory. >>> >>> But probably it is not necessarily to call out in the changelog? >> >> Let's say that we add this TDX-compatible-node ABI in the future. What >> will old code do that doesn't know about this ABI? > > Right. The old app will break w/o knowing the new ABI. One resolution, I > think, is we don't introduce new userspace ABI, but hide "TDX-capable" and "non- > TDX-capable" nodes in the kernel, and let kernel to enforce always allocating > TDX guest memory from those "TDX-capable" nodes. That doesn't actually hide all of the behavior from users. Let's say they do: numactl --membind=6 qemu-kvm ... In other words, take all of this guest's memory and put it on node 6. There lots of free memory on node 6 which is TDX-*IN*compatible. Then, they make it a TDX guest: numactl --membind=6 qemu-kvm -tdx ... What happens? Does the kernel silently ignore the --membind=6? Or does it return -ENOMEM somewhere and confuse the user who has *LOTS* of free memory on node 6. In other words, I don't think the kernel can just enforce this internally and hide it from userspace. >> Is there something fundamental that keeps a memory area that spans two >> nodes from being removed and then a new area added that is comprised of >> a single node? >> Boot time: >> >> | memblock | memblock | >> <--Node=0--> <--Node=1--> >> >> Funky hotplug... nothing to see here, then: >> >> <--------Node=2--------> > > I must have missed something, but how can this happen? > > I had memory that this cannot happen because the BIOS always allocates address > ranges for all NUMA nodes during machine boot. Those address ranges don't > necessarily need to have DIMM fully populated but they don't change during > machine's runtime. Is your memory correct? Is there evidence, or requirements in any specification to support your memory?