From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33B2CC021AA for ; Tue, 18 Feb 2025 20:25:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AED328019D; Tue, 18 Feb 2025 15:25:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E7B628019B; Tue, 18 Feb 2025 15:25:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661A328019D; Tue, 18 Feb 2025 15:25:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 07D2828019B for ; Tue, 18 Feb 2025 15:25:43 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E6B644B44C for ; Tue, 18 Feb 2025 20:25:42 +0000 (UTC) X-FDA: 83134196124.09.8D740BD Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf27.hostedemail.com (Postfix) with ESMTP id C390A4000D for ; Tue, 18 Feb 2025 20:25:40 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=uLMSWeZD; spf=pass (imf27.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739910341; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=22K3K8H/Z4VEtlmzFLZqURw4WdVHs3LP2poxDfotsZE=; b=eh/NOiYnPfWlYM9e21IJjgr6K1biOdp0TbyxhgrbX/t3Z4FgQ2ytR3qKqoecXBT1+GYFni +UJR/ceYSH+a2ODNc33aZpdGyL19eYRN5ld2WBME3dKAjlcKK8B6YIu17iYrRRdPohickn +0WbqJxUzaIhLbpjxjNaGs/sLmT33zk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=uLMSWeZD; spf=pass (imf27.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739910341; a=rsa-sha256; cv=none; b=W9QBrGjJYvSPinknQds1ETTGk6f+fiA1W32G7QwCsUGncmzWABkSz3vq+QigDTGZjBCPLJ miFROrfRCyIMi1ie+75GA8HQRGUpzgHLX296WR5M9dAVUi3IRlKRgyezlBbBOEfW+IJDlw GhF1tR1Z+fk9nAsYBkOpWCJBaskkPAM= Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-6e6827984b2so24955166d6.1 for ; Tue, 18 Feb 2025 12:25:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1739910339; x=1740515139; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=22K3K8H/Z4VEtlmzFLZqURw4WdVHs3LP2poxDfotsZE=; b=uLMSWeZDX6a9349IlC/Oyo+kGWJOQ47lJ9w0Y7Zp6WVQ+t16RbCEn8BduaFw+TqW3B 6Bf1P4NhR/EKG2yQypr49tplEC5Buv17pIoOK5gXhosXFWh0Cn5f3tV91W5fdXcxzN0A YTwp1rpTHa+FAu4UU2qSdaJrM/6elshTXeEYoHScSEkmR/JsaRRCiD28hg3LVp2/EZWd z15u8t77pZ87f3ddr3+gPuXbXXQJwbbZxQth46VxiLqMEhzsbn3Xy/XKtzOsMfSdlzlr KAbv5oj9mUX5USdeE98Ksd4w4Q8WTLKNdYqNiqR4cZJAmt/PJZhhOdB4faIBeuJLCs91 BkEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739910339; x=1740515139; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=22K3K8H/Z4VEtlmzFLZqURw4WdVHs3LP2poxDfotsZE=; b=fCb5ExOX27Us7i7uaaQpJw/SUJ3v8hCmV3K6y7SBkh8jEMRZJNsbHCi+6Iu8Fic2GM yeTQjE9XTSESRaH5zVSBOE9I8sg7L5dX0MQqaU/aBKRn6qt6j1NLHsLh01Ih8hd54byY BZf6sENk4YasBfUleaZf40TJOL4/XGKM0FmRkiZA+Y1TTXMVlgUae1EuuPGeRweCQzWX ox/+c+sCzTOqUZPHEymAgQPxaLBsCQGts0Fsr0prqgKWbZKG5kcXH4vghBlcNzakTMHn G0oIMSJuOy7eablVPB3Xz/jEPWOMTCM0poKQr5A11PHj1aWDOFOL0shN72ZHe79/h5dA Fh6g== X-Forwarded-Encrypted: i=1; AJvYcCWUaMwAu7uo9HZpXwKp28gWsSoSueS6339crEIEejBCFbzXRbNeQ5kG/XbcrxVEBw8I9c9xWlBQtg==@kvack.org X-Gm-Message-State: AOJu0YysieUrInqwLAEJf/FQjL2kHKEp63FGppCzOWC+2VhFaPJ92YDB +gximQcIYHfl/9Ub7cwZSXWsrqm9TKM7Coi9bSuWyyhUihgK1onXlvLDBUGFUrc= X-Gm-Gg: ASbGncsmH3hOk71sq2oUylAtz9Ye7aDWFz/1/MdygM1knvscYHG8mKq4+gXEJrYmToX 3PtEiHAx8NwKyMudWUqxanLo1m8AACddLkCfGIqrms0JjCwMnDDFnKtlPxeZKBQ49pAdZhhZp+A XDsCl6jim0aIR3FTY7ksraYeDLIKqm9lweRD7mOIo7mC8lUzuR+ulywKv/qFYurvmZ+kkpE7pSi omJZv1XisRWcXSv+JUajuHBOAU4xckvWvu7tBYugk0Sh3wwLvvaFPLQCWMQZdM+OgHJqMfT0Lsb QifwyOlnI8JNo9nGJ2axZux7/cbGUH/z0N5IOpun6YAZQSRoysdXrOqUjPns+5h+gB6gtRipQw= = X-Google-Smtp-Source: AGHT+IEI19wWOEv8YZ18w+TjkiABH7btIaZS7NvAL6Op0w+BcszJCj70Cd8D/1/PYxxqGY2625dWkA== X-Received: by 2002:a05:6214:300e:b0:6e6:5bb2:c1a6 with SMTP id 6a1803df08f44-6e66cd064f7mr220329296d6.28.1739910339666; Tue, 18 Feb 2025 12:25:39 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c09a14d45dsm308921585a.10.2025.02.18.12.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2025 12:25:38 -0800 (PST) Date: Tue, 18 Feb 2025 15:25:37 -0500 From: Gregory Price To: David Hildenbrand Cc: Yang Shi , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug Message-ID: References: <1b4c6442-a2b0-4290-8b89-c7b82a66d358@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1b4c6442-a2b0-4290-8b89-c7b82a66d358@redhat.com> X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C390A4000D X-Stat-Signature: hk3if5dpr54eprkc6apx5yxhirts6f7k X-HE-Tag: 1739910340-905689 X-HE-Meta: U2FsdGVkX18NKM/qCBTTwckweX39SjfJaUB4iVm+fMmwoHBlMHGAgYDS72B9zlxnun1rhec7THZaFZz0HrUuTr/3x08FtDYX8GvwqP/N/KVsndI7EUY2KzHTiRqphoNb5C5yr/dyE+Kx+1p2ek3fA6MYVQ/b4tAy90ZcaSpDQRK69oxwwM7K93H5lWJTZJgavi14VvPtaL+xJVR+F4LQJMjxQfcm9/c4nzG1wybu2sZ0mz7ohZ4zddWnMOaWIpcgQip0hhcpKeGTGfe8Tj7EaHb7sztZdd/866bnRJ8H4kY8BgtSnCg7AAps7gewSS+0bRixpaPU9kemBA2vklMGolGIyCgUJO3GAnw4q99M8H3wR1TxPmHW3RtlvVVoRMeVRchS4tDG2L71YeaY7Aj6ymU1HYetcnriAMVkQzfm43sUmEjafVxegxlBJ8ndzF3vOzz/V/aSuZvuLa00hdTCmYJqR+aUYn3a7iU/WIJxUcoQIipWaIw1GtBjtWesG6Jg1KG+KrX/Dz7bK0pck4J9NBJqBTkklbLsfF6hGS4IzUBASmVX3gNWAjtrSmrSBdiaAD3lari5CwHGOF1Q4LOpdbTjW5Ptqm1cNJLecpz0xwerVkTLUiiGoEmxVJ6eiyAkeHwixESc6p6gqvuAe6i5POTvb/zS+vUfx+YSbufoF2ophMVRS0iM7CqjkxvAG/AuSLzS47FdEOK7ZL4/cOdImRCsSYsINu5nGSi4xgEslXrR4v9bUa3ips0s6ly9PT/fsisDUJBKvfAYTG5HVkDzExBQW9SPpHigzkdWIVgjuw/+CcfGzTEv1a7QkGkBx7QQjsLpH/XPXpQeE5gqjgoN5TwdbuVKoG6ycwSCHtRGuE56HR3EUSQJ0UbHMVnsT3+AaPZCPP1wp/y+p87pghSj1Ps4MaWR6/tz0RfxL1KG3M8wa6Y+bjNzVzvH7Bht+MKVZMSb6SBb7EjWFZWlGcs +UUM5qCo VzVt/TgBq4zbc/oBP71EnhCm7AyF/kYF4MT8Njug0odU8ss0XQA4S/FEzFYwsCbisSBk5gT2naQ82vuVY6cYSde6R9bl4N6jqlNdX1FH1LvcsRceJlyDBlIZkz92jifcbz0KiYQAO4L5gEchQefpKnJs+owv4qRzHhCj0qrI9Tu8lMAjnrA83p6BI7Jkay+KAoQ9GfK5MZaBbe/NzZOQ7ri114ML4gK7UCvYS0+QHOB5A7lNo99Yh2Nj2oIWMHJEo8Z0S6AJ3az0ppcGySiPU9lvuB7laaNIkXScgIlFW+QimK2qIXQWn0G8m5WYpLUxlBeU7rCYbCzAnJv+pTUKBZpHxhnFC/KXSpYPoeeXSo0CqSx/QLd0j3NT2E9tdUuRHcjrRv/4UOWxkqXAnEQuuor3ITQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001144, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 18, 2025 at 08:25:59PM +0100, David Hildenbrand wrote: > On 18.02.25 19:04, Gregory Price wrote: > > Hm? > > If you enable memmap_on_memory, we will place the memmap on that carved-out > region, independent of ZONE_NORMAL/ZONE_MOVABLE etc. It's the "altmap". > > Reason that we can place the memmap on a ZONE_MOVABLE is because, although > it is "unmovable", we told memory offlining code that it doesn't have to > care about offlining that memmap carveout, there is no migration to be done. > Just offline the block (memmap gets stale) and remove that block (memmap > gets removed). > > If there is a reason where we carve out the memmap and *not* use it, that > case must be fixed. > Hm, I managed to trace down the wrong path on this particular code. I will go back and redo my tests to sanity check, but here's what I would expect to see: 1) if memmap_on_memory is off, and hotplug capacity (node1) is zone_movable - then zone_normal (node0) should have N pages accounted in nr_memmap_pages 1a) when dropping these memory blocks, I should see node0 memory use drop by 4GB - since this is just GFP_KERNEL pages. 2) if memmap_on_memory is on, and hotplug capacity (node1) is zone_movable - then each memory block (256MB) should appear as 252MB (-4MB of 64-byte page structs). For 256GB (my system) I should see a total of 252GB of onlined memory (-4GB of page struct) 2a) when dropping these memory blocks, I should see node0 memory use stay the same - since it was vmemmap usage. I will double check that this isn't working as expected, and i'll double check for a build option as well. stupid question - it sorta seems like you'd want this as the default setting for driver-managed hotplug memory blocks, but I suppose for very small blocks there's problems (as described in the docs). :thinking: - is it silly to suggest maybe a per-driver memmap_on_memory setting rather than just a global setting? For CXL capacity, this seems like a no-brainer since blocks can't be smaller than 256MB (per spec). ~Gregory