From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2033FD1CDB6 for ; Tue, 22 Oct 2024 08:35:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60A9F6B007B; Tue, 22 Oct 2024 04:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BAC86B0082; Tue, 22 Oct 2024 04:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 482126B0083; Tue, 22 Oct 2024 04:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 29A966B007B for ; Tue, 22 Oct 2024 04:35:30 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 793701C46C7 for ; Tue, 22 Oct 2024 08:35:11 +0000 (UTC) X-FDA: 82700578170.20.7D1C03F Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) by imf29.hostedemail.com (Postfix) with ESMTP id 8941A120008 for ; Tue, 22 Oct 2024 08:35:05 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EQTLHK0w; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.18 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729585962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5xRFWoOKK4TFSyk7YQIKCrs6m1hpnJMc6jB80oyDR2k=; b=Xgyj9nhb+jEpBYKdt/Jrgstz+sBKzX6Kz+yJTODyhuE5TrQDXB6H6FKiCCCvhzNvCw793i KF3uT3dwEBz4qPSoGIpy+o7z/ebXNKIU/ALaREhFNtMLIaw4QrcaUP73YfwpcaAL+TONl3 wZt8FGc9RjaJy4cG8KQa6DDg17udEzM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729585962; a=rsa-sha256; cv=none; b=ZZjy3RISo3sq2Lx7JPfFU2UGix0W+YJ5tnmE9aLzL2GOjxCYsVAp6XMYzv8oSTk1JS2FgW YrhpCT+vPPDu3yNMgGhaz7lL1KXMC+i0gayHh/jaqR968s45KTtLGS060Pfx4YRuBjyH6F P8LXx3HotAIaPsYsjhYH+UnIKFf1FdE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=EQTLHK0w; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.18 as permitted sender) smtp.mailfrom=ying.huang@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729586127; x=1761122127; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=aJGyiVV1Vx+wkCJPog3Ga3tOsSFFzIoJWEjdxQipbM8=; b=EQTLHK0wBZfZVJyFySSvbj5X2Sw5aTfIvRZLcmgdTg3sjK4IKSdjrI54 Gy45kyH0NnYxzziAMlgdRwF3AnQYsfaUsmz/MC9PPeIFgjj4N5Hrg0hd4 tkHrFavnUzbdifqszSnK5oEQqck6Z9HFL9yyUb+71cpFZXpAk5pAx9Lj4 wxNQYTtbWI6wV70weR0WxcAB60mBsUrwc5DQ6pDwhQVm1PaRtmlrYYAbU UV78wtvEsnKK4t0Z8vzigXZR3BpROCGUrcehCHctpeNsQR/MA66CtygL1 2khAXAufy2wwlCBnTYaqHEAcIZQqzlPAnDEBm4IindCn+YMxhb14+3beB Q==; X-CSE-ConnectionGUID: rRCFgkp3S5+AUcuoNRHdvA== X-CSE-MsgGUID: LZ6liqxVSriAJBlv/rTAeQ== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="29260033" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="29260033" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2024 01:35:25 -0700 X-CSE-ConnectionGUID: 7gXu7vSUTKqgcCNx5pA5ow== X-CSE-MsgGUID: iWFLw6k0QJqYXX+yMmEUGA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,222,1725346800"; d="scan'208";a="80602371" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2024 01:35:22 -0700 From: "Huang, Ying" To: Oscar Salvador Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "Kirill A . Shutemov" , x86@kernel.org, linux-coco@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dan Williams , Kai Huang , David Hildenbrand , "H. Peter Anvin" , Andy Lutomirski Subject: Re: [PATCH -V3] x86/tdx, memory hotplug: Check whole hot-adding memory range for TDX In-Reply-To: (Oscar Salvador's message of "Tue, 22 Oct 2024 09:49:06 +0200") References: <20241022031617.159969-1-ying.huang@intel.com> Date: Tue, 22 Oct 2024 16:31:48 +0800 Message-ID: <87r0884jyj.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8941A120008 X-Stat-Signature: 4mkzubxou36afnnsy4gonnmceyewnabr X-Rspam-User: X-HE-Tag: 1729586105-917800 X-HE-Meta: U2FsdGVkX1/MgaGfvCLXRNSK7pBHNL9gQKfZc7MBoOti/jlbiZzCFX6oM9g5XMrIIyXeBZT+OMfShjdDcdEUuqVPv6qZwgJex74MMnlAImUMoj9LDgC7NbXwhX86gWntd80R4OcpXAarlPR1bkoFf7deuTwEOdmrh98IRDiyWNgs2ogdIkFeUYonVXJQqymnIJroATUJtz+oz8vut+yhrX7eXBjWZ1fvjMQNmutBbpwHanOtBHeImgSP3buklxmixBjrrsrkWzqRZvO7oGQGYuTHLNKCTBd32NvtWcs+1muGcvV6X7+Ctdox6LCTf3/1Up3Y5UYu9Kis8PpiJ/yMJjM2Mscn2YYG2hr4065VDi+gYo23V4TEMMyDiBgRV3r5mai876d9Ytu87A5kMLl5mE9IDOKiaGoYgmNuc9O4MiwndaGfs4VWBKoeMzD2VevBNNyS32VjURiEgmcyhjVaECfTnQtUq3Z1qcehbsAOM4RTdEY4Ik8rLUsUUXlJ04q74/2RR4vx4D68N4qDrPVxOdIegaONVV9yHVNJXdWOxt/y3MO4u+3WYDMQXrH/A8dB2K7EKNCOICfmv/pdRmgRGhzmHbRzOhJ37nO9ZtPpRDjrh2Ul3Q2PnsdYukUePZWGnefXuvujgo+vQj/OBB45D2xYNMliNskzYgWx/a98e/SonT7Fx+5L3r/7yYb/HfKzqTK5q5pU63fbmA31LW1R3+ihuGlgVG57hg+Cdw37RLFREKq6CyHeTlVAvUZ6oCaTQgK5Bz1QsBN94VYxxzNOJOrS7xjtTaMks67Nsj84D1NPNcYaZLfcgz8jkKiXxzGBCfbulQ+YTgBUNPV2wZxRk85PjnhAEl694yvcEEXgcxMDIUokYp/sUXvPJ2P4cyCtNlM6cHQgkYCfgEjx1Q759hV9o14UyypxoUBY1+WwLPfakAzF8Xu0Xmv+tz0mWRSMyK+Zy5kCWRG0dnjCv2J wb1XUQX3 XC8VfngoPzXoPuk7ezMovU7SwlcVyuVPATpVNHJiMCBuyrDFtqffDw3SRvugoFJdy7MjIJb7WmBk93LSq+jDAzM8RP1EsLXvU+enuixu1BwFy98LliSMCNAzSQs8XIrVVlH0rO6g+5FGXO8HWV7TMVDmAw6WUR8kxAZiq4fB0TCBKyvLlXLUV1m79ZpfzeQ+g+5sE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, Oscar, Oscar Salvador writes: > On Tue, Oct 22, 2024 at 11:16:17AM +0800, Huang Ying wrote: >> On systems with TDX (Trust Domain eXtensions) enabled, current kernel >> checks the TDX compatibility of the hot-added memory ranges through a >> memory hotplug notifier for each memory_block. If a memory range >> which isn't TDX compatible is hot-added, for example, some CXL memory, >> the command line as follows, >> >> $ echo 1 > /sys/devices/system/node/nodeX/memoryY/online >> >> will report something like, >> >> bash: echo: write error: Operation not permitted >> >> If pr_debug() is enabled, current kernel will show the error message >> like below in the kernel log, >> >> online_pages [mem 0xXXXXXXXXXX-0xXXXXXXXXXX] failed >> >> Both are too general to root cause the problem. This may confuse >> users. One solution is to print some error messages in the TDX memory >> hotplug notifier. However, kernel calls memory hotplug notifiers for >> each memory block, so this may lead to a large volume of messages in >> the kernel log if a large number of memory blocks are onlined with a >> script or automatically. For example, the typical size of memory >> block is 128MB on x86_64, when online 64GB CXL memory, 512 messages >> will be logged. >> >> Therefore, this patch checks the TDX compatibility of the whole >> hot-adding memory range through a newly added architecture specific >> function (arch_check_hotplug_memory_range()). If this patch rejects >> the memory hot-adding for TDX compatibility, it will output a kernel >> log message like below, >> >> virt/tdx: Reject hot-adding memory range: 0xXXXXXXXX-0xXXXXXXXX for TDX compatibility. >> >> The target use case is to support CXL memory on TDX enabled systems. >> If the CXL memory isn't compatible with TDX, the kernel will reject >> the whole CXL memory range. While the CXL memory can still be used >> via devdax interface. >> >> This also makes the original TDX memory hotplug notifier useless, so >> this patch deletes it. >> >> Signed-off-by: "Huang, Ying" > > Acked-by: Oscar Salvador Thanks! > One question below: > > ... > >> +int tdx_check_hotplug_memory_range(u64 start, u64 size) >> { >> - struct memory_notify *mn = v; >> - >> - if (action != MEM_GOING_ONLINE) >> - return NOTIFY_OK; >> + u64 start_pfn = PHYS_PFN(start); >> + u64 end_pfn = PHYS_PFN(start + size); >> >> /* >> * Empty list means TDX isn't enabled. Allow any memory >> - * to go online. >> + * to be hot-added. >> */ >> if (list_empty(&tdx_memlist)) >> - return NOTIFY_OK; >> + return 0; >> >> /* >> * The TDX memory configuration is static and can not be >> - * changed. Reject onlining any memory which is outside of >> + * changed. Reject hot-adding any memory which is outside of >> * the static configuration whether it supports TDX or not. >> */ >> - if (is_tdx_memory(mn->start_pfn, mn->start_pfn + mn->nr_pages)) >> - return NOTIFY_OK; >> + if (is_tdx_memory(start_pfn, end_pfn)) >> + return 0; >> >> - return NOTIFY_BAD; >> + pr_info("Reject hot-adding memory range: %#llx-%#llx for TDX compatibility.\n", >> + start, start + size); > > Why not using pr_err() here? > > I was checking which kind of information level we use when failing at > hot-adding memory, and we seem to be using pr_err(), and pr_debug() when > onlining/offlining. > > Not a big deal, and not saying it is wrong, but was just wondering the reasoning > behind. TBH, I have no strong opinion about which log level is more appropriate. IMHO, it shouldn't be pr_debug() to make it easy for users to root cause the hot-adding failure. And, it appears too harsh to use pr_err(), because there's no program error, etc. So, I think that something in-between is more appropriate. That is, pr_warn(), pr_notice, or pr_info(). In them, I prefer pr_info() a little. -- Best Regards, Huang, Ying