From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A77DC636D4 for ; Mon, 13 Feb 2023 12:13:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A41D76B0071; Mon, 13 Feb 2023 07:13:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F2086B0073; Mon, 13 Feb 2023 07:13:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 892486B0078; Mon, 13 Feb 2023 07:13:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 79E256B0071 for ; Mon, 13 Feb 2023 07:13:34 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 232A68022A for ; Mon, 13 Feb 2023 12:13:34 +0000 (UTC) X-FDA: 80462159148.30.22B78EA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id F095E40004 for ; Mon, 13 Feb 2023 12:13:31 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZxnYoGNx; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676290412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+6v0ooLyeNfzQ7xBgMjxn5omUuyxiJes4sQHKrDWMfg=; b=bYOhrRYU7jaUKSYd7z2URryqK6v8M86IeAR4nkykdwcvnzU9D13Gd9xRL2dALNN65jbaPd AsNt7/VH3MBM9mFMTj47JsPMqaq/hQio2G9DPbNJhoXQQbsNK3NlJBBjDT/BGw3NZ/skJ/ d6caShR4gDy6LUyNBM8nvRsVs/EkBMg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZxnYoGNx; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676290412; a=rsa-sha256; cv=none; b=uieW56QXfOCgv8/PbDcwccBCflkYMvsbDefffFADS5CxYjA3WqkWWCu5bZxmGSVtiRbW4w VcqEA7VdnXD8EuNpnWbhr1Sc10jjJDIPtKaVSbgnp9iVRMfV6tT/vkIzyrT1yPZb+QqxnV 3DafXeSpTFYC6JZfH7nfS5JKYwSDl8A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676290411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+6v0ooLyeNfzQ7xBgMjxn5omUuyxiJes4sQHKrDWMfg=; b=ZxnYoGNxzmzptVPE43HFzbNn0Pec9BvV5ZqZtdDyTlANJ/SvUuC31OY63gt010QlvWMKRv VXbEImFXEAtWaHAgX57L9Y0M7UxsOXJssXULgEJoNYK0Fhwr5UZZtmQdt7JZTPheBJbRDg f+RB1UqkhJz98RhZIw/dIu0XmfmRlFk= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-533-iHCXgFA7O-m5pPcpvNr7uw-1; Mon, 13 Feb 2023 07:13:30 -0500 X-MC-Unique: iHCXgFA7O-m5pPcpvNr7uw-1 Received: by mail-wr1-f69.google.com with SMTP id p7-20020a5d48c7000000b002c53d342406so2229026wrs.2 for ; Mon, 13 Feb 2023 04:13:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+6v0ooLyeNfzQ7xBgMjxn5omUuyxiJes4sQHKrDWMfg=; b=FBSoYXd7kjnAZYHijhpzkBujgbRQuum+um0TuJGiAQ01uYNb711kTXM8gDlRk3xlx9 bVWttMe0BUmYnDhEG07a8vvlIoBZpms7BnOmaxQyH9IY+R7+GrPtjJtoXv3zK7QsM32a eVmLMavMx82JYsq1LUdR9chT+PUqwOJbByveoHOoJ8UwQUfPuGJSZgk4Yk6fBVZJk/FO LpxVIZ1rqw/Lpu1oziKZBU2kAlMUECPkaLVQDdEm7+EGg9HkJ8i2xr0mAqWVCdu31PLQ lNmGiuq+QsdlC+0w+lsfqkFu8fQ/4wK+qgYoijMF5OqTZKaEQ6h+y039Q+Oz865vjR/h zZjg== X-Gm-Message-State: AO0yUKXSrPwv27AMvQZp+CQt0ZWYpLuduF4AN6F/+bKxmiz5iaFaEJZR FixTJXYK6x8l13kJys8uXnQvPwEZIbti7Fs93cq75caRpldZbcl1g5HJkPY5/26MhboNbcbUMS8 h5OnUUqbYIEM= X-Received: by 2002:a05:600c:807:b0:3e0:47:66cc with SMTP id k7-20020a05600c080700b003e0004766ccmr19097472wmp.23.1676290409319; Mon, 13 Feb 2023 04:13:29 -0800 (PST) X-Google-Smtp-Source: AK7set9UKPGUs5FOatxbxbL/MB4NwZDNS48v4t9kXrFWtKl/EbrfjKpJI1CK4NzxGM8p+c9u9Ic45g== X-Received: by 2002:a05:600c:807:b0:3e0:47:66cc with SMTP id k7-20020a05600c080700b003e0004766ccmr19097458wmp.23.1676290409076; Mon, 13 Feb 2023 04:13:29 -0800 (PST) Received: from ?IPV6:2003:cb:c705:6d00:5870:9639:1c17:8162? (p200300cbc7056d00587096391c178162.dip0.t-ipconnect.de. [2003:cb:c705:6d00:5870:9639:1c17:8162]) by smtp.gmail.com with ESMTPSA id g10-20020a05600c310a00b003e1e8d794e1sm4939993wmo.13.2023.02.13.04.13.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 13 Feb 2023 04:13:28 -0800 (PST) Message-ID: Date: Mon, 13 Feb 2023 13:13:27 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Subject: Re: [PATCH 00/18] CXL RAM and the 'Soft Reserved' => 'System RAM' default To: Dan Williams , linux-cxl@vger.kernel.org Cc: Kees Cook , stable@vger.kernel.org, Dave Hansen , Michal Hocko , linux-mm@kvack.org, linux-acpi@vger.kernel.org References: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F095E40004 X-Rspam-User: X-Stat-Signature: gucndzt97kuhywx7bdbc66ziois86wk8 X-HE-Tag: 1676290411-199360 X-HE-Meta: U2FsdGVkX19b4XvFnLztHIYxfqiw8qeGYDA11UaEf6UCVNRoWpnk97fe6a5+Oz4IZZEAbZrTve3+DWY68C3FGjZcjck1rEYqJKBDLKamrviAOXcMAtPB64iE/hLMgw3ihCfYL9TSqkoIO9tZwsbH17RYj7IVV7oRH98HPGOiM/Y9sgOgzMzcTJATTA3xl6sgynRYXPgdDquJL5k+z1cQEc9X5Ld8qHtZAPwZiY6vtCR/1Fza0sD9RYD81Fn8a1nYp8IQUt5+ZSKGZHxcJBj4jG9wPOucZyxDMp8kpin012oqzZezrvj5plnntqvyCbwHiQUA7LIBpOoh3+qbG5hUFJ88gdvrUVvZmyge9pp5+2HfjYYKRmXtF2G764i34JoZ6wClj/6EgR7M6VNCokX5byyTOLF4Ser8uVaB1YoWAcJDjsk3Iw7PXEbPDCAdxMEYcLS/v7n+xV2svBaR2+CZ6IfI+lLp2EQahzVgbEE6YHIYFuoO0QKNTXsVxIn/DVgI3oRexyfF148oH1cxWmoa/pumFxrY4ey/fu5rmfyhELBgtUIDrNWet/qv7Ysb2Q9U7eEzeTxksv5TvLUC4H0SnSIfmTlHnpqio/YQmOC430z+kzMVKGlXE2BKDFrAjPO/wdcxhzoSREwWwFFPVJQ3k/zpFUBrgJ4gpcHVyEemerRBG6Ltn6qN1yvVr3MV+uArdpO4Fz/TFbNPrvu0A9Y8CpmWAAgIEHv866LFPe40T+KieWc8jGm+bN4ZEhbQnUwWEiwZeytRwhIb0gZgHGiEq7jVg02J9yjeiNTLKGJQaeoCIVvlv4tAwCayDwyCjMgcLfhj+03KLZydEUrZDZpi/b53S1mngT6MFO6Wzt5XYC4G5rxMVqeCz7YNrCE9m6mecx5KLXFw74XMDMfUvRV2SXh7hOSSJYIyurLkyBgHvk+CSoPnftXrcAxyjtHuoj6akRDjjbkuqhKHs1uNluM ZkCiAPvG jLEurtKzTQy7isBf9lwPKt9DRJxTl4zZiVwJCn5wH07V6/K/s6x7JhNtUvXbrVVxgvoEkFtjNyZtVNfqOV4dkzc1TzoVOq6mvQzgH+j0nvMnicSn4JTCVqhyWKdtmZ7dVh8cbSyHyIgc54C0d7SIpN5auRj5hsXDQCik5PVTJrhSdyT5ad8PfNLzCBKQgxfqjWSfx3j1g+x23gb41r5JWhGLAHVFUuu1C0e3LTF+mAtlFPhYLGfxmtT8nfqh7WmcSTvkjXawsRF9acj4Puk7s8YTnVFaiKc8IlVuEhr6ZgSXrt5waV9fmTsBY50BWhvZrfFfLu7Azd3IhYPcMaGIJLmHDUmv2+RxkJDoc4Zuz1X6hoOBC1U3yUcfcjUq+oYTms7XRyOgpvB45V36Vq8KwjUpVoN21yjou6Be4jgmrtO3TOVGE93xvRsy/VXFmx82pwjM0I5IeytQaxE0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 06.02.23 02:02, Dan Williams wrote: > Summary: > -------- > > CXL RAM support allows for the dynamic provisioning of new CXL RAM > regions, and more routinely, assembling a region from an existing > configuration established by platform-firmware. The latter is motivated > by CXL memory RAS (Reliability, Availability and Serviceability) > support, that requires associating device events with System Physical > Address ranges and vice versa. > > The 'Soft Reserved' policy rework arranges for performance > differentiated memory like CXL attached DRAM, or high-bandwidth memory, > to be designated for 'System RAM' by default, rather than the device-dax > dedicated access mode. That current device-dax default is confusing and > surprising for the Pareto of users that do not expect memory to be > quarantined for dedicated access by default. Most users expect all > 'System RAM'-capable memory to show up in FREE(1). > > > Details: > -------- > > Recall that the Linux 'Soft Reserved' designation for memory is a > reaction to platform-firmware, like EFI EDK2, delineating memory with > the EFI Specific Purpose Memory attribute (EFI_MEMORY_SP). An > alternative way to think of that attribute is that it specifies the > *not* general-purpose memory pool. It is memory that may be too precious > for general usage or not performant enough for some hot data structures. > However, in the absence of explicit policy it should just be 'System > RAM' by default. > > Rather than require every distribution to ship a udev policy to assign > dax devices to dax_kmem (the device-memory hotplug driver) just make > that the kernel default. This is similar to the rationale in: > > commit 8604d9e534a3 ("memory_hotplug: introduce CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE") > > With this change the relatively niche use case of accessing this memory > via mapping a device-dax instance can be achieved by building with > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=n, or specifying > memhp_default_state=offline at boot, and then use: > > daxctl reconfigure-device $device -m devdax --force > > ...to shift the corresponding address range to device-dax access. > > The process of assembling a device-dax instance for a given CXL region > device configuration is similar to the process of assembling a > Device-Mapper or MDRAID storage-device array. Specifically, asynchronous > probing by the PCI and driver core enumerates all CXL endpoints and > their decoders. Then, once enough decoders have arrived to a describe a > given region, that region is passed to the device-dax subsystem where it > is subject to the above 'dax_kmem' policy. This assignment and policy > choice is only possible if memory is set aside by the 'Soft Reserved' > designation. Otherwise, CXL that is mapped as 'System RAM' becomes > immutable by CXL driver mechanisms, but is still enumerated for RAS > purposes. > > This series is also available via: > > https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.3/cxl-ram-region > > ...and has gone through some preview testing in various forms. > My concern would be that in setups with a lot of CXL memory (soft-reserved), having that much offline memory during boot might make the kernel run out of memory. After all, offline memory consumes memory for the memmap. Is the assumption that something like that cannot happen because we'll never ever have that much soft-reserved memory? :) Note that this is a concern only applies when not using auto-onlining in the kernel during boot, which (IMHO) is or will be the default in the future. -- Thanks, David / dhildenb