From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8826ECF34C1 for ; Thu, 3 Oct 2024 20:40:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAF8A6B03E0; Thu, 3 Oct 2024 16:40:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6B36B0442; Thu, 3 Oct 2024 16:40:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79B8B6B0441; Thu, 3 Oct 2024 16:40:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 572626B03E0 for ; Thu, 3 Oct 2024 16:40:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0416A1C3D07 for ; Thu, 3 Oct 2024 20:39:59 +0000 (UTC) X-FDA: 82633457760.14.36E0087 Received: from mail-ua1-f47.google.com (mail-ua1-f47.google.com [209.85.222.47]) by imf05.hostedemail.com (Postfix) with ESMTP id 3CA95100006 for ; Thu, 3 Oct 2024 20:39:58 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BZle/GSv"; spf=pass (imf05.hostedemail.com: domain of shy828301@gmail.com designates 209.85.222.47 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727987826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yPDOQS4g3jiSC7koeugiFipgPQOubT+R5rh1868TSLk=; b=6Y7XGxugEIilfpQk/uS1eIUSKo9iIQwV4N32L3SbVOJ73lNuBPQHyjB3x8OuRlCkt6zQgF v4uWRW69wSNR4IAsslHh5wpUP+txDRaVSMfOFVAW82rdFkd1OweVkPVDbBzUyx6P6XS9Y3 6tDWv84lJZc6i1hcII0dVfpxGO/BIcU= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BZle/GSv"; spf=pass (imf05.hostedemail.com: domain of shy828301@gmail.com designates 209.85.222.47 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727987826; a=rsa-sha256; cv=none; b=BQkOQsYCxDMWJvRsvvi+ZBzck3EJoJwNtWsOu92z2jZrCgnEUVuiFmq+5hWMzgx9X9AHeR YvXtMDuq2GWIl7JAk0WgdM/9RLU23xUhyvXp5y+mAfmSkYxMojZDeESNlA9M9b25Wal4+t xbL3VPhwOltGkpDGMgORZI1lQkvots0= Received: by mail-ua1-f47.google.com with SMTP id a1e0cc1a2514c-84e808f3c3cso386342241.2 for ; Thu, 03 Oct 2024 13:39:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727987997; x=1728592797; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=yPDOQS4g3jiSC7koeugiFipgPQOubT+R5rh1868TSLk=; b=BZle/GSvLuc0p1JNj7+dSp4RpzgO0ZmcVijg27GQ6CJ+tw79f4tH/4VUZSE7oJkgNS /DjkV43v7ImWULczoeQSNziJnBD34Hto0bQGR6hZW8f95pP1QgO0fczGgm5+o3kYDe7C /XIFNfswdeV1AioNS4hBWsmCnr/O0YOvkY3oeQweErePyVcyEKa+nobs2Km3CV39908K EnlFsryJ2TrBdaUvDjWhyJElNIdiOtRs3tZvxAe9An2gdoR67slJjyJSOZP8hGQWle26 rTBhOEke0N83wNGzP5q6+p+NP6wN7nZEv6zrFPC409guu7jHSuE+7pQ3SS6GA7ZGRozf DDeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727987997; x=1728592797; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yPDOQS4g3jiSC7koeugiFipgPQOubT+R5rh1868TSLk=; b=r24CasFMl+2J6pRdBEbKC9PN64+kSwGs0Yz+FB/+TCwq4+clegUXpDmRA5zYgubxmd TfzTdu9DVny62pHA/3wlRA9ghXg6g7muGnpC12JO4YrYgr4wm1AUv4rqFiNwpkV5uOnv Yu3IKFo2xLZYNv5EeGYzkyA/bsz4DtC4cttfupgOuS9EAeTHJSRmnW1y9bB0gzWaKPhz vftcEdq+KdIiWXgpQ8oEXI6Y6D97qhcUALQ+fVqOb/VxBjgnyidQ720FsbjzXH0rn5Yf 3YCtgkRTXDNSGdW3GvCvN8ZcTha6aN17YsL19NFLagSAPaAlMCD0UH0zoHd+fiJsGCRq pbiA== X-Forwarded-Encrypted: i=1; AJvYcCU5ppJzvY0Mbhw+jiVvqTfs8Gc8hkB7NCy4ylkI2G5/IbF53QCM7Xl3y1gHXFLeLlrZXSf0KTL1nQ==@kvack.org X-Gm-Message-State: AOJu0Yy/2GliNcClmsErdkM7lYZG4B2xZC5EWpG1iv/0x4mMLpG7MKUt mvzBTtt7Ik5UFw5f7UZNOQSlMUo34HTTJm2zuKuA/ue4C5MC0Sy3NGgnGaTfYzzvG5XSw0XxxNf uoM4hxZBFz8ZLHFNmly+mbLynvuI= X-Google-Smtp-Source: AGHT+IFFz+Yln8L/nNoifUwD/XeJ5mUA5108ak9GrdaULvEzgln+g9PdDUAcb0H7lTFYiTWuSV+s0QF4kwDd0y+d7cE= X-Received: by 2002:a05:6102:c14:b0:49b:cfbc:63ac with SMTP id ada2fe7eead31-4a4057441ccmr754716137.6.1727987997227; Thu, 03 Oct 2024 13:39:57 -0700 (PDT) MIME-Version: 1.0 References: <20240930055112.344206-1-ying.huang@intel.com> <8734lgpuoi.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <8734lgpuoi.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Yang Shi Date: Thu, 3 Oct 2024 13:39:44 -0700 Message-ID: Subject: Re: [PATCH] tdx, memory hotplug: Check whole hot-adding memory range for TDX To: "Huang, Ying" Cc: David Hildenbrand , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "Kirill A . Shutemov" , x86@kernel.org, Andrew Morton , Oscar Salvador , linux-coco@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dan Williams , Kai Huang , "H. Peter Anvin" , Andy Lutomirski Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3CA95100006 X-Stat-Signature: h5zayi89x1aq3wqg19b3w57ei1g84osd X-Rspam-User: X-HE-Tag: 1727987998-271745 X-HE-Meta: U2FsdGVkX1+bukMgMmF9Yohgw6KYDggLlUTzPj8jJTol743lGFyl/olRKAtVEBVGJMf7jK7hWrZKBiubU/CDrHvyfS7GAiNOXj2679QsgjehNYHtDfcT6+0SVrnQWnGl0aosJcm54dSbnkjW08dYC1vlCvp679DDjyxtDejOORtfNTplSO9Kdm40J+ypAs1ChNMPosMMY+OvnSVaW7dfS7Lzj25G3WQ15WiL0gjEjz5USxM3Pj9JJ15zKC8ANKcBBj8Jj3YCMyJjXVD1zK7/zLtchCpPwJhBY5rBtQZ8yJUpmrPkDSwJXwwzlIHnlKqHU03gI55+OnzAXtdpHhCjlvry0s7cw5LmXSqD5Hyv+PFg+hX85grUK8PdJ3zcbnYLk01iKetd66QyEKLRVvTQBLLAeX0J6iE5Hioh6K0a1NnxCMK3HUtGumxEytjaYH7mLz+ExNxLsredImLWKzGpy9UN9SCnsWTUc7QyMrEFwXUs41Qmr58abngL/MjJ2oK/b0N2Q6SFjHaM1FuWZ1ZVBCsRh8veEAMjm8lOZU9rZkGpkD3N67Ppigk9QK1XgU51ppRsdTgGyvza2amwcNCT0/VkSuV/q+JC7MBQyisPTGM5bmlS+G5fFbOF17joBVEaHNo9APl9eBWuhRK65KJ+Cf13Vr3hExpAQpBbcPC8Nut2docL0if9jxyX7k2fRwuJ0HaA2sLPOUTJxMq5LmevVNELjAbzB6i6Nau74UUrSuhGvau1BGGcEl43IABWaEsUiJ+DZ2/iaBqOxhEj4EsqFdvoicKBD2kv/FZ49Du5Z6TrvObH6C6R0RVf8mWvjjMm0AcxcXFQwB+gwf2FKvKXpweahnLCePnh74xMY2T/TFyk3DPgAVh0sQJMeEh4Yv7ETWoZUwtBH+27NXzqYOqLigwfSAYrSNyMTty1o81SKQj2dftfXfjQ63W7dqS3pEtlduRhyYm/EOoFqu/Ij0r ujxZPxb4 j0QLBSg7xa1sxwRcKWrMkgjA00Pk4HE7IUJ99FBhlMw3W8bgiQbxB5m7WcUg9sI/oHiXcq6GSda0RZlrHuRRrTTB7dLXGyg3/tvT/YD5U+5UNnHiXXBkfjetba2kXhE5ZAg9LSjeUjFVJRT4tezUN8PdOCfoKRWJZatrVkX27qWCFD9Gt3aZEEmilZIzfqQ5x34cjHK2RPY7fGjJTfKl+atUG7uAoClMHP3/7S4nXrqVeFjMlD2l7C8F/7Za0jmEiSrCgsIdoxLr7bVzkH53Hk5thbSbxWOioZHZ6vM3bu7xzlTT02pc2ac7MCdxUGH5JOrnwuRfpAljy0CnvCGHs5j0+Cu04IVbfk9xegYnOPZ53z64VIw6nqCB0nw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Sep 30, 2024 at 4:54=E2=80=AFPM Huang, Ying = wrote: > > Hi, David, > > Thanks a lot for comments! > > David Hildenbrand writes: > > > On 30.09.24 07:51, Huang Ying wrote: > >> On systems with TDX (Trust Domain eXtensions) enabled, memory ranges > >> hot-added must be checked for compatibility by TDX. This is currently > >> implemented through memory hotplug notifiers for each memory_block. > >> If a memory range which isn't TDX compatible is hot-added, for > >> example, some CXL memory, the command line as follows, > >> $ echo 1 > /sys/devices/system/node/nodeX/memoryY/online > >> will report something like, > >> bash: echo: write error: Operation not permitted > >> If pr_debug() is enabled, the error message like below will be shown > >> in the kernel log, > >> online_pages [mem 0xXXXXXXXXXX-0xXXXXXXXXXX] failed > >> Both are too general to root cause the problem. This will confuse > >> users. One solution is to print some error messages in the TDX memory > >> hotplug notifier. However, memory hotplug notifiers are called for > >> each memory block, so this may lead to a large volume of messages in > >> the kernel log if a large number of memory blocks are onlined with a > >> script or automatically. For example, the typical size of memory > >> block is 128MB on x86_64, when online 64GB CXL memory, 512 messages > >> will be logged. > > > > ratelimiting would likely help here a lot, but I agree that it is > > suboptimal. > > > >> Therefore, in this patch, the whole hot-adding memory range is > >> checked > >> for TDX compatibility through a newly added architecture specific > >> function (arch_check_hotplug_memory_range()). If rejected, the memory > >> hot-adding will be aborted with a proper kernel log message. Which > >> looks like something as below, > >> virt/tdx: Reject hot-adding memory range: 0xXXXXXXXX-0xXXXXXXXX > >> for TDX compatibility. > >> > The target use case is to support CXL memory on TDX enabled systems. > >> If the CXL memory isn't compatible with TDX, the whole CXL memory > >> range hot-adding will be rejected. While the CXL memory can still be > >> used via devdax interface. > > > > I'm curious, why can that memory be used through devdax but not > > through the buddy? I'm probably missing something important :) > > Because only TDX compatible memory can be used for TDX guest. The buddy > is used to allocate memory for TDX guest. While devdax will not be used > for that. Sorry for chiming in late. I think CXL also faces the similar problem on the platform with MTE (memory tagging extension on ARM64). AFAIK, we can't have MTE on CXL, so CXL has to stay as dax device if MTE is enabled. We should need a similar mechanism to prevent users from hot-adding CXL memory if MTE is on. But not like TDX I don't think we have a simple way to tell whether the pfn belongs to CXL or not. Please correct me if I'm wrong. I'm wondering whether we can find a more common way to tell memory hotplug to not hot-add some region. For example, a special flag in struct resource. off the top of my head. No solid idea yet, I'm definitely seeking some advice. > > >> This also makes the original TDX memory hotplug notifier useless, so > >> delete it. > > > > The online-notifier would even be too late when used with the > > memmap-on-memory feature I assume, as we might be touching that memory > > even before being able to call memory online notifiers. > > This should be OK. Because we will not use the memory for TDX guest in > this way. > > > One way to handle that would be to switch to the MEM_PREPARE_ONLINE > > notifier, but it's still called per-memory block. > > > > Nothing jumped at me, so > > > > Acked-by: David Hildenbrand > > Thank you very much! > > -- > Best Regards, > Huang, Ying >