From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51DF0CEB2CD for ; Mon, 30 Sep 2024 23:54:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB4BE28003F; Mon, 30 Sep 2024 19:54:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C6369280036; Mon, 30 Sep 2024 19:54:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2B1528003F; Mon, 30 Sep 2024 19:54:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 909E4280036 for ; Mon, 30 Sep 2024 19:54:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0D557A0BF9 for ; Mon, 30 Sep 2024 23:54:50 +0000 (UTC) X-FDA: 82623062340.19.853A192 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) by imf30.hostedemail.com (Postfix) with ESMTP id C64398000D for ; Mon, 30 Sep 2024 23:54:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JBMF4F1d; spf=pass (imf30.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.9 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727740362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kAzMJnNy86lExR9ZHQIp8VxDULQGxuF6O7XGjHSfEGQ=; b=X4mqNABKm97Al0Fz5BR5eQRdI7obsSPp4EJE/OsnO+j5rB740SfbMTCBzOiegLmY0L7xQ+ T+8rVTDDADstl1j+S2kFXAuf52gyFaBqMF0zdGWPSXEIu9uSTnwGzWyEz5kGPNFJBa19pW 4mkJ8qMJ4b4yq9MaUjpyPvfEcY/S9lM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727740362; a=rsa-sha256; cv=none; b=nAQPJ+Gd6n02iUfV9Gi3rAKngNUNcrt4Ll2dBSyFg2ifGuxVKhRGOWvagAvi31oMqj/pvg ESo/eah2cVhtk16+UeoV4/VLDZN7QVrskYmaddUOUy1A6af31jUEEsOtuInUz5LjXKtY51 0dCilbz4m7l1sUK0K7nS1GIoQWwBUQY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JBMF4F1d; spf=pass (imf30.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.9 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727740488; x=1759276488; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=jnLEbV10e8rDNV6NIwnCNnnDROhh8eyNMS6Se3TBvOk=; b=JBMF4F1dOEQfz+SuUAWJ/pW6idgzXKfDiLhXF62haAdS+qrvnRz/kZI8 ZW/imL/7+4z1nix31r++U+LangzSdw5erEYM01flEqp7nVx+sYMdJAeN/ 9Ax/ZLpGZqm51rDhSxU6unuALeXgVofLWoOdJzWmPx0crNBQF+YSa8iHy wiEz2rQK6yrpWVKgVH4tL+DXjvjEgiK9Yh/3nxATOACgOPDoen9Rd2I4/ 2OgutWNNXltHPaAIrLm9NhhkxJRpYNvtfz8yGjvSjY+RX0/1GYMMDPEmE 1DZ4ia6SI7f6AwDyv2StLuKPM00JS4NsUsRmmuPo6vF4Vd+/aan/zU6zQ w==; X-CSE-ConnectionGUID: JcC1z1HPQayF8SSFGOQXzQ== X-CSE-MsgGUID: NHUbjHB7TeuBOkHu9Kx08Q== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37527747" X-IronPort-AV: E=Sophos;i="6.11,166,1725346800"; d="scan'208";a="37527747" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 16:54:46 -0700 X-CSE-ConnectionGUID: LuEfMUGrTyqBT1bjMosr1g== X-CSE-MsgGUID: fdGZ+36IR3So8V9oC2Z4Tg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,166,1725346800"; d="scan'208";a="73593323" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 16:54:43 -0700 From: "Huang, Ying" To: David Hildenbrand Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "Kirill A . Shutemov" , x86@kernel.org, Andrew Morton , Oscar Salvador , linux-coco@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dan Williams , Kai Huang , "H. Peter Anvin" , Andy Lutomirski Subject: Re: [PATCH] tdx, memory hotplug: Check whole hot-adding memory range for TDX In-Reply-To: (David Hildenbrand's message of "Mon, 30 Sep 2024 10:58:09 +0200") References: <20240930055112.344206-1-ying.huang@intel.com> Date: Tue, 01 Oct 2024 07:51:09 +0800 Message-ID: <8734lgpuoi.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: C64398000D X-Stat-Signature: mnh4n1dgmwck8mwo19ny18rxwis66urc X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727740487-33536 X-HE-Meta: U2FsdGVkX191IFkOT6HLyLFKY1sgPo+HGf9JPcYbIR7UG0aKZfxWzyoc0+dVcsZFQRk5j51TDQr0qt5BNTgdVqrUROWUrYjngnhroCA+XXrgd5xG8UDKKCqAl2BSmS6gfX442Ep246oA24FwdN1HUDCB7V47zJdEMbCOl5GjMAxHPRKIkz8Nb7X3XckW/muk996/6R6cH7IiWA1MfFy0S6QVfFcMOoQKja/hO/rr+kxrHmAG8ctVBxPFn4Agi0aiBsdAQ3FZIqSN8jclyphOL5lbOAjXeEQsIJMKzMlAC6qWFwybcos1C4+FcRsXuPca/DSQNi1zJ9HtfvOieBr3/ZZLX245jyqVqIOwO8oOfbcBQQeMVpSKmkQMAWkASfIHZ5xf9m1hSPufZ3R3DY5zN6xhxmzslySbv8Y3SI5K+a/Sq3yaa9pVcY5H5bJ8n4X8+DyG1d4sTtlXa8feEZf9MRSPwYfeId5jJmONpGvTFcq9NhsS2hbp/NRBVHzCbvMPGhb+Yx9nw5eYxHMUQrCUxKr5lmGtM9oV2PKf6/Ht5aDSY8RqtGqODam/5oH8ZlYxmF3C8zHl2MxCeDBhKmYJ5y5QPZMPtw6uQT++U9qhq/4RopCReqmxtIdQ4AwrLAugbfmat+UGPnkHTQp4XcKHShhoL8/x7vemv6a7ulkNRRbvZtx12+TPJrDPkZqcNPtP6C0ZZmXvgcJO0N23cnFpbJ4kCYF8LNvj5nXyIsLT8u467jyqr1doPOGPkon475rp3pGk10A7RU1h8Z5wrSYEZops4aVqWplB8zLIuGax4RBcUnQ29ZztblZKRYC3cviTVt38OuGbpClfZdxWKTFPu9LbgfgJgmwC0wPWfwX1UQW7Le6gE2KyoQEyK4mhJnNLq5KmbehuxPTL7HCnqzVtUrfrMphikWjKKU8Qb1mu+S1ZbCGXE568NI8cM+VbmBQaKBtit//fP/FnbKtaHDo CKTvM5FY LzaACdTDsd3HvW/3pqFIXLkZbCjYWwHOjs57AxK5+RCA3qUoP3d8QPH2DYxhLHcKVQoNHgFh0MHm4gz3wmYsxv2cmQczsWk+hyq/ADcmGdazy9amRilB81rAaFggBG/sfz/6VCaF8LUKvTiBmGvneBZEnS+lYAKoOPmUlhzd8kbRC58z5YBXfAN13hfy2J+09nYlVgdlFJ6ZpXGpKiBcGHdjcItT5RcbIRQkx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, David, Thanks a lot for comments! David Hildenbrand writes: > On 30.09.24 07:51, Huang Ying wrote: >> On systems with TDX (Trust Domain eXtensions) enabled, memory ranges >> hot-added must be checked for compatibility by TDX. This is currently >> implemented through memory hotplug notifiers for each memory_block. >> If a memory range which isn't TDX compatible is hot-added, for >> example, some CXL memory, the command line as follows, >> $ echo 1 > /sys/devices/system/node/nodeX/memoryY/online >> will report something like, >> bash: echo: write error: Operation not permitted >> If pr_debug() is enabled, the error message like below will be shown >> in the kernel log, >> online_pages [mem 0xXXXXXXXXXX-0xXXXXXXXXXX] failed >> Both are too general to root cause the problem. This will confuse >> users. One solution is to print some error messages in the TDX memory >> hotplug notifier. However, memory hotplug notifiers are called for >> each memory block, so this may lead to a large volume of messages in >> the kernel log if a large number of memory blocks are onlined with a >> script or automatically. For example, the typical size of memory >> block is 128MB on x86_64, when online 64GB CXL memory, 512 messages >> will be logged. > > ratelimiting would likely help here a lot, but I agree that it is > suboptimal. > >> Therefore, in this patch, the whole hot-adding memory range is >> checked >> for TDX compatibility through a newly added architecture specific >> function (arch_check_hotplug_memory_range()). If rejected, the memory >> hot-adding will be aborted with a proper kernel log message. Which >> looks like something as below, >> virt/tdx: Reject hot-adding memory range: 0xXXXXXXXX-0xXXXXXXXX >> for TDX compatibility. >> > The target use case is to support CXL memory on TDX enabled systems. >> If the CXL memory isn't compatible with TDX, the whole CXL memory >> range hot-adding will be rejected. While the CXL memory can still be >> used via devdax interface. > > I'm curious, why can that memory be used through devdax but not > through the buddy? I'm probably missing something important :) Because only TDX compatible memory can be used for TDX guest. The buddy is used to allocate memory for TDX guest. While devdax will not be used for that. >> This also makes the original TDX memory hotplug notifier useless, so >> delete it. > > The online-notifier would even be too late when used with the > memmap-on-memory feature I assume, as we might be touching that memory > even before being able to call memory online notifiers. This should be OK. Because we will not use the memory for TDX guest in this way. > One way to handle that would be to switch to the MEM_PREPARE_ONLINE > notifier, but it's still called per-memory block. > > Nothing jumped at me, so > > Acked-by: David Hildenbrand Thank you very much! -- Best Regards, Huang, Ying