From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82D62C021AA for ; Wed, 19 Feb 2025 16:14:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6F7828023F; Wed, 19 Feb 2025 11:14:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D1FDF28023C; Wed, 19 Feb 2025 11:14:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBFE828023F; Wed, 19 Feb 2025 11:14:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9CB6828023C for ; Wed, 19 Feb 2025 11:14:27 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6AC08B71D4 for ; Wed, 19 Feb 2025 16:14:25 +0000 (UTC) X-FDA: 83137191690.28.18036E6 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf06.hostedemail.com (Postfix) with ESMTP id 63CDA180015 for ; Wed, 19 Feb 2025 16:14:23 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=oObrwiG0; spf=pass (imf06.hostedemail.com: domain of gourry@gourry.net designates 209.85.160.180 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739981663; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lAEwvi0GBeXV5su+o42Sjw6+fX6vESKuWe1ByEzBoCM=; b=uWNiqjcUEERhf7YNILB/J8C6J/7vDln+j5Y6vEWVtiz5FQfmh/j/Os6iDzhETNFQ1Q3Ida bH/3y+AEUvQNmmzJR+O/gd3kSYNImPI/LrDiYF/qVHUvi1WeivLEAF87k6DXIyYE7QaCqY pRqQB6GrUIYPqn5Q+QKDqc4Ge7K5aBo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=oObrwiG0; spf=pass (imf06.hostedemail.com: domain of gourry@gourry.net designates 209.85.160.180 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739981663; a=rsa-sha256; cv=none; b=kQbSQcSiJhtl0016Hq1M/ZID9BamnfG0twpQWkVlLgU7GEC2vuf76QTP4jmMrOPI3t6twY ymwKSW0QYHOjb9IoEflXertnbyrvSmEC6S2ZHib9XuD5lkAUkkFb18AODZxydbx0p5YSqg 07Wx3iw5pozC+p0hDfKou4mgmRfczcw= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-47210a94356so3765781cf.3 for ; Wed, 19 Feb 2025 08:14:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1739981662; x=1740586462; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=lAEwvi0GBeXV5su+o42Sjw6+fX6vESKuWe1ByEzBoCM=; b=oObrwiG0Fqf4Jog9io8+3mgouBEtS9CXk8ZOX/1ZlDfN/98xaSHUZIO8tyOFOlCq58 UcHdZ/+zVElMyGDj+boKd/wFLxC8G+5R4kqcOro/V7A2q7Snb9IdTMMtNgSUhFjL11cp 2fDsX7Q3EtperK1Wh7hU7/qneNghbYCPs9HhHtMCce/KHfY5iscLg3VfWlC2F/0y2mhX Nt2xO5mlPl1Sd9m4kiKfFLRiA+ITm8zrjI9dkPnlVhQAPiHmX6niBhlNnVZRyG4yt6FS U9A4vrorr+lwpXFwXUYEaOUWUXOYn3kiCEUAyM7cP9jjUaZRVy9iyv6ANFtsLlBUAOPA ZLXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739981662; x=1740586462; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=lAEwvi0GBeXV5su+o42Sjw6+fX6vESKuWe1ByEzBoCM=; b=eI5GeNy+BOQ5LXgbimEcy9lh5e/7jrHe/qMiVRR46rDwuS6DF6QgyernQz59xAJdnA xFhW2tFjyztyZPSKfa3pnAflyQCP58hE+zrslCAUFIcp+jrtUcYIcFKr3A7DYuoZj1Ud 2ZaXOOIil3SD1BkYIU+DiNXH1i3uh9KOrG+yON/mizIOGNxfh/JvXG5aQ6mK4tu+j1gD HXKuWq8A/FfIdC7HGF7QsbeWifFyzvfF6RcVTqbRF4ijvFJPxKE+s1na+aoSVGzURl3q i4zIM3mhKFWWfzGzCmW7mDgL8/ARY8f7QAJn/kWNgXuTaVZAm6Y8AVgqjQZQh9FrsPhm CjEw== X-Forwarded-Encrypted: i=1; AJvYcCXTmeJjhsgn1NJkfG3zjxk6Rh1CHohc9XIFUD74BDwzT2vPtxP9orJ5w2j+0fX430mbYB+iUZ8Etg==@kvack.org X-Gm-Message-State: AOJu0Yz1fgIWluyL8Cd0DewLehjSC/+Ag2GY/kF2P0fTpbYOIqh3S0bM sisQ5lkEhKjikgz9oPLMyNABYlRcBTdmG7rw297oThe6V7JOgy/bkcotEPEuKUY= X-Gm-Gg: ASbGncuRmKjpcTVNsCvXF+dT8HZJTPLq7owhBEKJVZDu/1PYJSL22/2GXWuVkgyxkZ5 /Em0kJItOB2gsu3AAUPH56apM4CDgjZ0Nzryb/zzCo8OGHN5Z0X6CXBp5TVgqRFxgNX1gTr4Lj6 GOu0Kbu8agWeCKewwCsEwMapsu38V2dcA2Ynp9yTPVZgnhiss7rdW8RfJTD/Hzj3rs4Ai0gg6ms BYUmr+2G1wS9A2wyLaYecBSRfNk2X+2Ja1uxhr7IHug+ty+sHFwZ2XjooPWXIoLuQN4wjc1xRON 2qNd7uWax9yMX7Jxn0W4eI+aKDwYgMBiHL6HmCy2Rl3XtF4IBzRg9N3NYuVgoIWZQzcISSpXuA= = X-Google-Smtp-Source: AGHT+IE6/v4wshXDrZEpD/MYfeRM134ORVrgYeRnyAz4JVMUV+BS4h0wbgOKih0/uNtmsb8QaJlVKw== X-Received: by 2002:a05:622a:260e:b0:471:fc9e:8de2 with SMTP id d75a77b69052e-471fc9e9085mr118660761cf.8.1739981660311; Wed, 19 Feb 2025 08:14:20 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4720624239csm14384451cf.11.2025.02.19.08.14.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 08:14:19 -0800 (PST) Date: Wed, 19 Feb 2025 11:14:17 -0500 From: Gregory Price To: David Hildenbrand Cc: Yang Shi , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug Message-ID: References: <1b4c6442-a2b0-4290-8b89-c7b82a66d358@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 63CDA180015 X-Stat-Signature: pbgmmi3sskx9qkbpaebu7apyqdnourju X-Rspam-User: X-HE-Tag: 1739981663-756283 X-HE-Meta: U2FsdGVkX1/gTVlsGcNwT4cGyenPRZZ8f4lylUfTUuYHj1QLBkWZ6EDWA8tJ+9Aj7smUswiR6pI8gJ2cUE9GK0HlrH/SyptZUnOoHjXawmX6W+wlAfnxRgpbfAeJWICFhk6kFPnWjLGzlnWy5hj6t7VFqriynFrzOmZLG+Lwsa4Fm/VgQurVm9xcFzN34f3TdKoojxlQ8kxaaAdBh4Ir50JqnHmH1KOugpjFklPZhm7iFm3JNM3Ng56oo7UDvhLku0zwyzjTRluY1rrDph3rMb5AKFTMxXRkpBlBjQ39SofVijaDdBdmU8aFhlUa3joo/RS41m0eNH+mHK/fcQAtqY56y3k7mEZswYmkdlWWmNCYwpibzWv0I267RPY6g72eJAnaqvlc5GAdhCFwGRRFxhTIDRuFfDxmegVIHv6ftsEULqmgCNvsbBsMd083bg4NMD4Iw4GVflUUMjpkA5jdYOfikGB6TWD77m+PinGeK0AiyCGbDzsX6vA0OrMLSG4JqYAzkgmgJLEmzxcrdAgw1YW2AXVUmnUlLqhtrms/hhR8ZHIep/9WDh7yvXpvDBk9La4rceEVbRWkz8ltkdYyP42Gx6a8UvA5d44/yOV+Nqizl17oLjGE4t0GHhhD8G+Me6aUmZbcPd0dv6gOECyXBVbov8QZaP6qgAFXhz/v0WRPQUcEWS2RCgULieGOvMnpBjffAAxMHLZB+jgOgIe7c7y+AP/pMPmdW4p8raynByIN2iiXWro8rDB7inL7kaWg2lHRnDlsHT58zEjhtmXPpDtJP8H5FAqYVg8qbNH00UjFFCpWDPEhrBlyhp0q+kKzRy5h+mrsZB1GhsLrkYEVVIoAfISDlhtMt0tgbyyfauOIWkau5kIj+8qwf3buE2ijujR87IrdNauOoDDr0qxhGNhg41uYPvthTulD0fY7zqidd92W3ugTLd8+4uLhxx8R31yQnApQnQP+5F6UCF4 gVu6r7hO 0L9VMjnbGRc9grRByIbHY+Bhjm0KMD1JRK11ZJyp08GXLzZuVM59rdD037382p0tYjSgWenEg3VR8PwxOR8UONfdgfm4HsU0B7VOlsWnsfmX26jCcIocZRHvl6Dx1vKnbiyemKrq22Tj1NaItA6AX0mzp952QOABs+IPyOw6XZnggDcnZfV4EyFNK8SG/cc5Uko+zGPZsuJY3Mwywa3TqAFX2vY9Nq5RJWZhjRyfScElKIv5fogyMnlD1ePJen8IWtaUoQpXcB26Ln4cnow4y63sgARRteAWkaW7c2D6nC9Qsx4IBqVIHLbWhkn0SyQ4iB0P87HjkueImj89E1bYouWsdT2rOPep7sdkPoe9i8RVqDifdLmKGj/0wrDAKLUTJbN+LhiD6mu8Gl2H5HwXaShk8Rw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Feb 19, 2025 at 09:53:07AM +0100, David Hildenbrand wrote: ... snip ... > > Would the following make it better or worse? > > The question is whether you want that memory to be hidden from MemTotal > (carveout?) or treated just like allocated memory (allocation?). > > If you treat the memmap as "just a memory allocation after early boot" > and "memap_on_memory" telling you to allocate that memory from the > hotplugged memory instead of the buddy, then "carveout" > might be more of an internal implementation detail to achieve that memory > allocation. > Probably this is fine either way, but I'll poke at the patch you provided and let you know. > Right, it only currently works with ZONE_NORMAL, because 1 GiB pages are > considered unmovable in practice (try reliably finding a 1 GiB area to > migrate the memory to during memory unplug ... when these hugetlb things are > unswappable etc.). > > I documented it under https://www.kernel.org/doc/html/latest/admin-guide/mm/memory-hotplug.html > > "Gigantic pages are unmovable, resulting in user space consuming a lot of unmovable memory." > Ah, I'm so used to seeing "1GiB Huge Pages" I missed that parts of the kernel refer to them as "gigantic". Completely missed this line in the docs. So a subtle choice being made by onlining memory into zone movable is the exclusion of this memory region from being used for gigantic pages (for now, assuming it never changes). This is good to know. > > I appreciate the clarifications here, sorry for the incorrect info and > > the increasing confusing. > > No worries. If you have ideas on what to improve in the memory hotplug > docs, please let me know! > I think that clears up most of my misconceptions. The end-goal of this series is essentially 2-4 "basic choices" for onlining CXL memory - the implicit decisions being made - while identifying some ambiguity. For example: driver-setup into ZONE_MOVABLE w/o memmap_on_memory means 1) ZONE_NORMAL page-struct use 2) no gigantic page support 3) limited kernel allocation support 4) changeable configuration without reboot (in theory) 5) Additioanl ras capabilities. Thanks again for you patience and help as I work through this. ~Gregory