From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F44AC021B1 for ; Thu, 20 Feb 2025 17:50:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBE04280310; Thu, 20 Feb 2025 12:50:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B6E452802C1; Thu, 20 Feb 2025 12:50:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3603280310; Thu, 20 Feb 2025 12:50:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 84DE52802C1 for ; Thu, 20 Feb 2025 12:50:24 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 58B13122617 for ; Thu, 20 Feb 2025 17:50:23 +0000 (UTC) X-FDA: 83141062326.12.647508B Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by imf19.hostedemail.com (Postfix) with ESMTP id 6B3751A0016 for ; Thu, 20 Feb 2025 17:50:21 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CNyC57QK; spf=pass (imf19.hostedemail.com: domain of shy828301@gmail.com designates 209.85.208.53 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740073821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ACbXZSJEeD7MSU+0ygnOG3CFUz1SHUqmj4uHUBUExvo=; b=qxpH//FdQKzpAnblJRa4ggqR36igP6G/9vHzuWd0cSzqurYFdDxDDx/K5bk/LKtIkfvYjm dO8frLL68G/fc/PAv3kpY1aGHcKTEFqi9EOg7+wQMJ1L7CgBjmLvA32tbRXV7c48VfV3gb x9ZLWT2x7yohnY9fXXpnPtFevOc9L6U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740073821; a=rsa-sha256; cv=none; b=enKJNJR92ptNm7o9136IZpY7zCER8c8J9NMuTQYfk9heAAGEGCB4tsIluxtssx2U1pYqx0 SBJr0gsOVcpnUtSto/1McGZtqzGwAGqQxWnUSabuuc89mgFQetoDqTY3Wa0WRdVfExD1+P 3pfrxaSPkN81QrbWwDxmwS8CHQJlT4k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CNyC57QK; spf=pass (imf19.hostedemail.com: domain of shy828301@gmail.com designates 209.85.208.53 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-5e04064af07so584064a12.0 for ; Thu, 20 Feb 2025 09:50:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740073820; x=1740678620; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ACbXZSJEeD7MSU+0ygnOG3CFUz1SHUqmj4uHUBUExvo=; b=CNyC57QKendYMURLNV1OPkSkZrQW5pwE0qOOVPBbn/xVsk0WRv3Rlnii3hjA7Uu68Q R0BbvxabUor8K93+6/+3SSUN94WOJq7+UeqwRACW57BpR7psK/2T9I0awYYPHSdah7O7 Ky6MAdlerbApC2UDNsQKpWWMIXN/B7Xn/Z8Jv8MAGbnUY+ZVjC0fxd+r1m72w1kRvm0P Efee75TbXlgQq1CAXowl27MBiHSfW8dZbMOSTin6ZPCElcFB2TiAWxeCXE8/2UJ/Th5d YdUPK01JUO+wQPNF+oe/TGZ6ByySnc9tiA8aqng4zaLZdLl3p7vtGpOLRqugHz317UmS PNgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740073820; x=1740678620; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ACbXZSJEeD7MSU+0ygnOG3CFUz1SHUqmj4uHUBUExvo=; b=G/csK2jmA+l/NpaHN5O4YsMqDEOeXwgqaL6nlVJC+c7wNd4Qt7gtfP8WrUTfrhsqxO sJHQYWMoVG7GF2V1YBjgND3ujY3Uu6rZZ1AtOg95k8HAZ/eKXYZUZl/K8vU/7Jp3ZESR eay8G9L4xz4QL0dZLI7kl1bb1WhE+QW/iPqeT5Cl8QlfdmbATs2lFYdSJOzlrWjLIzBd vcLOuCaO45Kfbx4K6aQhgX4iyxKnjlHsHVSqfxjLKdR5mM/wc88E+D/+TsLCclkaEBb0 XrhWhkts7Hukz+SHaG7GN8NiJKf1SWq8ZOJQR0MLjGaD7N58P10VvfaWM9gwMiB/+XMm u6WQ== X-Forwarded-Encrypted: i=1; AJvYcCXonEG04PbBr+uOT7avhsaT54qduXIyYHpzgR8zJ5zNan5G2i05V4S9HkfjFKKT/TxABeqaUFfbCw==@kvack.org X-Gm-Message-State: AOJu0YxyVkJ1ezKNhhPsCgU54raa4FEZP+Fkk1doCDdnsmBzVuBwaJB5 knzdilEeWD/yCWVwdg/VSM0bjOol4oRh6l0j3/JIdpWIUPa6ursY8oAFZsuKWJ+2+BzdHjWu9YJ B7GqPv2d3c8wu/hdbIL/ZMwWgV9o= X-Gm-Gg: ASbGncvitpiE4oRXu5fMmxppDoD7LZqiiN1i97V83SX0cft4ydIGfnzIE3cttU58yNG bxaVS3hshewzIc1xz1+ZRmoXqxaQAchQP3inW8rwQYRTwaVYSPZwsYyLGAKDdHRrXX577gPS6QR I= X-Google-Smtp-Source: AGHT+IEoWXbWwTVuK2Nwv99J648YCts8QbcZyhvyRX5LPXJIREs0N43ohOkCkOhIjhQm5MU2a34de0JWy5oegIyzsg0= X-Received: by 2002:a05:6402:268a:b0:5d9:a62:32b with SMTP id 4fb4d7f45d1cf-5e0360440a0mr26146606a12.7.1740073819559; Thu, 20 Feb 2025 09:50:19 -0800 (PST) MIME-Version: 1.0 References: <1b4c6442-a2b0-4290-8b89-c7b82a66d358@redhat.com> In-Reply-To: From: Yang Shi Date: Thu, 20 Feb 2025 09:50:07 -0800 X-Gm-Features: AWEUYZkw36f3OEiXrlZX_cyPCOYpVkyNjb2lSz_Gn-SXQJshZlbv5FIQbCvDvP0 Message-ID: Subject: Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug To: David Hildenbrand Cc: Gregory Price , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Queue-Id: 6B3751A0016 X-Rspamd-Server: rspam07 X-Stat-Signature: d3tnhqzonnf4k7anqxx6ngxscejjgha1 X-HE-Tag: 1740073821-546696 X-HE-Meta: U2FsdGVkX1+gZxNbSI/oZxBjTMaoRiQ4tptFSRViQfWd+Skl40OnU3yvxdtwn5xRxS5pNyx3OmI8JlrVtxtyLnrYQYA/86mFIgGynXFL1R/Y/mia1szysJBNz/ZDxXUui2zy2NQJ/cWYC5HLaZcFz9Vimnea+0UlDMXVGIZO8VG36Hld5rSKNyPPj4yD92uEG9rQDHR2Di1DGBsMWMUnLG8AIFQwKctisC89PKiygrk/GzYwQHRfc3/354Wm8Yz8XYvmDJm6zbDbmaoBfnWY93+7hSH8uzcUgnP5oQR2kvAFonCRs0rVnfwWFl7WwKVGIrq0mHWjaHvDIUw9qnKXSVM3FaOALecR7RbJTSD2RY8DkSL3cCS8WAbYVKdnEeljXqdfe4r/3wnBhcp89ksbkrmvaYNqnmajCqMVzB3KC34eQ2DQGJimzQuDlbnwS0Ui+Amerf+oxb/kb6IXoqkwnpnn+lP+pQhoRFtjn6s5ubTeHY0rMSB2PEGc4EdhR4ngTadJ24W2s+ARc8frTDdpV9nDxMXzp5p07vyp0sCckygH41MWzDrQOSRYckJ/K+pCeycswhxP39VxYJoa44umSwHbx6Hq65DL4PneueivejFIvgQ3II1kf9AaK0llHrnRTp0B8lVA0zF0HNc1ikkEQp4Bfi5vh+4jo3Kpvl+CG+DXIE/SVpD+G9Z2gA7Bf4M9hfzLqo1H97KMo0lZ94ScFj34uo+PSgt6NqraQ3kkaFYZvJ5FvCAxEfJbe9p7OU9QNvM6chGAfKZYwIVa52atcacrV2NQA/BUkx6sjBV3kH/DqPDsZAksIPyEE77Hm9ProsQqJBtfqGB/MjlsX40q5OgIl25jdjIRj7SqCF+ft3Eyeg+Vk130i1XUc3/f/qEctRyDtQ3gFooUgemZ+Bl4nf5RZCx8TIZQ+hZMIsuR8W3XUvQP5pUVwAy07Oxlxv2mft0oawV790xezKOSfnp rYi09dT3 uiMIplc6gkn29S3jCIX4TMECVb4Gh0BTVsG13BtTGn8N7ZK6tdJN0OTYxxtRPHo8yZ0sH8XskvAKI+QF5KPcZM1b9fp78SAt+6n9NVOix8zcb9aLlJBjFbdiQGUMMGNyELgeMQyvQnFvN7O7+f9GkQfeUb6vQDyo2JeO5H1jDxskferw83d4r70yvou+ezCEcbYCXnVkBib55LeV4nvph0o6wNoZfrPoPDGcduHOmbdGEnLBa9Hkf65wafe8Aa6o37X4PCgBNQR0mA5U4bkrFzotiLvavPTMQ4axh5oS+4ak+bdCnn6piMCQIn2W2yKBM5W7l X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > > > > > 2) if memmap_on_memory is on, and hotplug capacity (node1) is > > zone_movable - then each memory block (256MB) should appear > > as 252MB (-4MB of 64-byte page structs). For 256GB (my system) > > I should see a total of 252GB of onlined memory (-4GB of page struct) > > In memory_block_online(), we have: > > /* > * Account once onlining succeeded. If the zone was unpopulated, it is > * now already properly populated. > */ > if (nr_vmemmap_pages) > adjust_present_page_count(pfn_to_page(start_pfn), mem->group, > nr_vmemmap_pages); > > So we'd add the vmemmap pages to > * zone->present_pages > * zone->zone_pgdat->node_present_pages > > (mhp_init_memmap_on_memory() moved the vmemmap pages to ZONE_MOVABLE) > > However, we don't add them to > * zone->managed_pages > * totalram pages > > /proc/zoneinfo would show them as present but not managed. > /proc/meminfo would not include them in MemTotal > > We could adjust the latter two, if there is a problem. > (just needs some adjust_managed_page_count() calls) > > So yes, staring at MemTotal, you should see an increase by 252 MiB right now. > > > > > 2a) when dropping these memory blocks, I should see node0 memory use > > stay the same - since it was vmemmap usage. > > Yes. > > > > > I will double check that this isn't working as expected, and i'll double > > check for a build option as well. > > > > stupid question - it sorta seems like you'd want this as the default > > setting for driver-managed hotplug memory blocks, but I suppose for > > very small blocks there's problems (as described in the docs). > > The issue is that it is per-memblock. So you'll never have 1 GiB ranges > of consecutive usable memory (e.g., 1 GiB hugetlb page). Regardless of ZONE_MOVABLE or ZONE_NORMAL, right? Thanks, Yang > > > > > :thinking: - is it silly to suggest maybe a per-driver memmap_on_memory > > setting rather than just a global setting? For CXL capacity, this seems > > like a no-brainer since blocks can't be smaller than 256MB (per spec). > > I thought we had that? See MHP_MEMMAP_ON_MEMORY set by dax/kmem. > > IIRC, the global toggle must be enabled for the driver option to be considered. > > -- > Cheers, > > David / dhildenb >