From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2254AEB64D7 for ; Fri, 16 Jun 2023 07:54:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0A3C6B0074; Fri, 16 Jun 2023 03:54:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A92598E0001; Fri, 16 Jun 2023 03:54:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90BA56B007E; Fri, 16 Jun 2023 03:54:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7B9E86B0074 for ; Fri, 16 Jun 2023 03:54:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4146B1A0BE1 for ; Fri, 16 Jun 2023 07:54:31 +0000 (UTC) X-FDA: 80907848742.06.130734A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 150DE100009 for ; Fri, 16 Jun 2023 07:54:28 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8FWd4ZW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686902069; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LCwXsPaPd7Dj5AMUAfbDa6LnAHzyeLCA6sw8/b3X0Ho=; b=3mmfszu67bifmw5LJDP7m9PNPpK3eoaN+Y2l8hVh29G1Wh7VHblZ8GxeJpckN4hILXasT/ FigWs7+wLC92W07Hh64DJzURc4Tg2uKz8bJ77QDvftLVTp/YAhzsUi3AdxcihswwUX3V36 7ktXaGyYAaohqqNjeOcsBahRrVdxg64= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8FWd4ZW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686902069; a=rsa-sha256; cv=none; b=Km10GfqT93nrTCpODrnmo3YE3efrvmmMDodTyW+fKU1Cb6teLxAztZMPeo6drqrlsZYMYg dKNvs9YTnZnwXYU9Qauuz0n3hASn+NtFUoB3aMtyh25AI0l4ui1b6JUFcyJcRKVeQiBcz0 FQw4c3gBATIULPr5ac/HIR3+ogNg7VY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686902068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LCwXsPaPd7Dj5AMUAfbDa6LnAHzyeLCA6sw8/b3X0Ho=; b=h8FWd4ZWoLbKP3a0FFusPZdqiw1rPe46zHP7CFBmwPKefJPR9/ilnBvzQLiwFwMn2GRjT0 vqvIwSjHBuB4KP8bnlIFtTtJaG7ZhfzJqXTfnROj3Mdzt/CH1VF4lgnFEH3wkBTxYOhSif KdLSpdOwsypn1rIsydeHSGHoE2f4oXQ= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-412-SoviRVVCOECS0YzRrlCvGA-1; Fri, 16 Jun 2023 03:54:25 -0400 X-MC-Unique: SoviRVVCOECS0YzRrlCvGA-1 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-3f7ef0e0292so1265855e9.3 for ; Fri, 16 Jun 2023 00:54:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686902064; x=1689494064; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LCwXsPaPd7Dj5AMUAfbDa6LnAHzyeLCA6sw8/b3X0Ho=; b=Lt74C9czA3XzSV5X7OUBpKOF3LtYMTthTezKIGm6A03i5N8bGnekICx79YX2pWoLHd ecaeMZg8LDk79IrPJ/RJiv+znciI8HXaxcFJrJ6WTKyPvGNeGbUqX0Lw1o8BdM57mfSb Dm/ji7qFa3IsUaP9nA0hnOmWH77SCODnfxA3jRn/cpFbh4Wnumkd5Zj/ovOW1F4CbI3/ MKbmYv2cJgBP558h0OyW9yUaTEml9rAtEsoxvBByWVrXyyvoTpRXPj7pHaIzNVmyQkcR 6j87lj0LMBucUCNtCEXBwdHeZYV2opMSd9T8oGFZKonzl/RiFe5RUyRnmW7anJpok/q0 QRiw== X-Gm-Message-State: AC+VfDzDP+fXi2MPAysUmRcR3ZEoYSxVOg1L7X4Fz4chNc1ZsVnto2tV KqJb9zMmy+vDtzKp4GAc8NNI5CQYZOkxsLvD/cUY9aBQHQOhYptV8JZ2wMHkX0nnnTfB9T29SxT lgfBVX2fukbU= X-Received: by 2002:a7b:ce85:0:b0:3f8:c5a6:7a8d with SMTP id q5-20020a7bce85000000b003f8c5a67a8dmr1006465wmj.12.1686902064528; Fri, 16 Jun 2023 00:54:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ57hUFJkB9ARA1uPg7u8sBxFriDIGC6XvjqrvDb1WUHap8aii1g0Wt8HyS0VoMZ5iYl6OcUQw== X-Received: by 2002:a7b:ce85:0:b0:3f8:c5a6:7a8d with SMTP id q5-20020a7bce85000000b003f8c5a67a8dmr1006449wmj.12.1686902064164; Fri, 16 Jun 2023 00:54:24 -0700 (PDT) Received: from ?IPV6:2003:cb:c707:9800:59ba:1006:9052:fb40? (p200300cbc707980059ba10069052fb40.dip0.t-ipconnect.de. [2003:cb:c707:9800:59ba:1006:9052:fb40]) by smtp.gmail.com with ESMTPSA id o8-20020a05600c378800b003f195d540d9sm1431039wmr.14.2023.06.16.00.54.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 16 Jun 2023 00:54:23 -0700 (PDT) Message-ID: Date: Fri, 16 Jun 2023 09:54:22 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Vishal Verma , "Rafael J. Wysocki" , Len Brown , Andrew Morton , Oscar Salvador , Dan Williams , Dave Jiang Cc: linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, Huang Ying , Dave Hansen References: <20230613-vv-kmem_memmap-v1-0-f6de9c6af2c6@intel.com> <20230613-vv-kmem_memmap-v1-3-f6de9c6af2c6@intel.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH 3/3] dax/kmem: Always enroll hotplugged memory for memmap_on_memory In-Reply-To: <20230613-vv-kmem_memmap-v1-3-f6de9c6af2c6@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 150DE100009 X-Stat-Signature: xjyrduhyjamjg9fetpiewqf8mset9g55 X-HE-Tag: 1686902068-578510 X-HE-Meta: U2FsdGVkX1+U/P7WT8Rf4t7E9u8zgSD9cPH++CIriNg1Gj834o9ieINApVxg8OuGsRjppVIclCkBaOXepZsOkhXwz55E3ABOshoEqqjzE2TcAQsZ+jcUnCP8bTW1hjM6E7PjxWOefcPsoQSnRWhZYBqhVWjpx6OtiRqAhszlOaCXLBha3nYEBcsU9FrejP4pK6Ax0iAJ+p1hWiRBZk9DJ0YjtXGD8jDxoqrI25Fw2elilO5m0W1TqZkgBDCln5o07/jvpHJSWj4FSmlAkpCZm+uq5Zryyp/psccP2hR4q3JBTB5NBgM9UMwzTnxLEa89rKM54I+kTSxScoFRLRg08JmTC/rOWmKF70Xht7rIkMLOILMMqYyKU+FjpIVpnYJFzxj0u1S2st8xwuS36ZDNefMXLzWjttODDq6y1lI5QmiHIvwxfO5wEp7krJ1YgiPaeRiq9iWuOtwFVH5Jzh8lAdFMRNfLL3gI43si+HGxRlbjw0KB9ePC6Nw3hi9eg7EJ60RAApeZPsbruvGWYQUghXFnSavT+7V5nWUet/XHQ24vyYqM9MO7d+Lh/ITvu33pZBAAFtlQP07HA90jyEXlhsdYswQQ/qSi3ROn+uBQatdjNAEuZS02AaqAvjiS5o+2WZwKZoY6rEbYGyIhVz4afGl7VUzcEgpRmSDLl9OkxDcD/qIcI82+53H3zGuZcjYh7sMQF3Ak/trm9zwXeSmB6RHP9gR3l46m4SpR7UqJBvR2RpOJ1y6wJlhO/tlo0KRTCCW0uSYnv/uN18YnJ0TI67Fmq4iMlAd/p1qm46wtgkR8nmh9ylBn/gjxj0KGuGwlVyTUS0c1plR8qzuxUTJB3fr7KqkO2e5uzhdFzW4TLO4r3+5gIxGQrkG+yZBMbeG6EcRfwc799NNEdx5G3U4+AKsIoRfioAAFnSS4ksFG5rHeZNQOCmLW6d4n7T0HRhrnXoCf84QqzrznJS6u2pr gTNZoNVo WOR5VxUbqOBJ2fADg0yYH8TbADOEkyMxaail814an/zcOU9Zjf326fCjZTGjeJ7QXbYoBeDRj+Htq2jG9ba/PCDbmsGfaLJfA+bHD7IemhS5flDM3xODJuesBttmwHcJ5IlfamWEdoTusYNqBrRct861rUz3yiDxzX+B2SUkqKdlEzcSxNu60eBHdf34ssGnYzGOSmz0v35OUUpP+SqeiTMYa34r+EJKsWYC79Og8iv28XuJaYXwOgIYDpY0emOIycO35MfKXpSf4E6vE4I/THIcgktVELtJOqgULfC4CJZ0ameV2l8Im3eGcVnDULlDN5f2luSYc98VRF4x82p0DWQMOlPbTWad+S5H1hC91lGPjOQEs/6+iB6zO+jkcIqRxQs8MrGTw4gwkEC7cAICgeZiaoZK6FynRX34uJ3FJRWuF8qc6MjI2Y8eLzh3tNZROqsJ3UiQGdloyh8WhlptoL5jTHzk69sJzkjJmV02p/4acdj+dCsNYQ5W5ysMfJGvVE5NP58SxIry63nF7Pfg1gj8odktGa1IDwtrJi/Koo0eHBSCiRxwR2gJLiuqEZnKsbfRJpWVOk8v/miYGEfd9dv/XZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 16.06.23 00:00, Vishal Verma wrote: > With DAX memory regions originating from CXL memory expanders or > NVDIMMs, the kmem driver may be hot-adding huge amounts of system memory > on a system without enough 'regular' main memory to support the memmap > for it. To avoid this, ensure that all kmem managed hotplugged memory is > added with the MHP_MEMMAP_ON_MEMORY flag to place the memmap on the > new memory region being hot added. > > To do this, call add_memory() in chunks of memory_block_size_bytes() as > that is a requirement for memmap_on_memory. Additionally, Use the > mhp_flag to force the memmap_on_memory checks regardless of the > respective module parameter setting. > > Cc: "Rafael J. Wysocki" > Cc: Len Brown > Cc: Andrew Morton > Cc: David Hildenbrand > Cc: Oscar Salvador > Cc: Dan Williams > Cc: Dave Jiang > Cc: Dave Hansen > Cc: Huang Ying > Signed-off-by: Vishal Verma > --- > drivers/dax/kmem.c | 49 ++++++++++++++++++++++++++++++++++++------------- > 1 file changed, 36 insertions(+), 13 deletions(-) > > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c > index 7b36db6f1cbd..0751346193ef 100644 > --- a/drivers/dax/kmem.c > +++ b/drivers/dax/kmem.c > @@ -12,6 +12,7 @@ > #include > #include > #include > +#include > #include "dax-private.h" > #include "bus.h" > > @@ -105,6 +106,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) > data->mgid = rc; > > for (i = 0; i < dev_dax->nr_range; i++) { > + u64 cur_start, cur_len, remaining; > struct resource *res; > struct range range; > > @@ -137,21 +139,42 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) > res->flags = IORESOURCE_SYSTEM_RAM; > > /* > - * Ensure that future kexec'd kernels will not treat > - * this as RAM automatically. > + * Add memory in chunks of memory_block_size_bytes() so that > + * it is considered for MHP_MEMMAP_ON_MEMORY > + * @range has already been aligned to memory_block_size_bytes(), > + * so the following loop will always break it down cleanly. > */ > - rc = add_memory_driver_managed(data->mgid, range.start, > - range_len(&range), kmem_name, MHP_NID_IS_MGID); > + cur_start = range.start; > + cur_len = memory_block_size_bytes(); > + remaining = range_len(&range); > + while (remaining) { > + mhp_t mhp_flags = MHP_NID_IS_MGID; > > - if (rc) { > - dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n", > - i, range.start, range.end); > - remove_resource(res); > - kfree(res); > - data->res[i] = NULL; > - if (mapped) > - continue; > - goto err_request_mem; > + if (mhp_supports_memmap_on_memory(cur_len, > + MHP_MEMMAP_ON_MEMORY)) > + mhp_flags |= MHP_MEMMAP_ON_MEMORY; > + /* > + * Ensure that future kexec'd kernels will not treat > + * this as RAM automatically. > + */ > + rc = add_memory_driver_managed(data->mgid, cur_start, > + cur_len, kmem_name, > + mhp_flags); > + > + if (rc) { > + dev_warn(dev, > + "mapping%d: %#llx-%#llx memory add failed\n", > + i, cur_start, cur_start + cur_len - 1); > + remove_resource(res); > + kfree(res); > + data->res[i] = NULL; > + if (mapped) > + continue; > + goto err_request_mem; > + } > + > + cur_start += cur_len; > + remaining -= cur_len; > } > mapped++; > } > Maybe the better alternative is teach add_memory_resource()/try_remove_memory() to do that internally. In the add_memory_resource() case, it might be a loop around that memmap_on_memory + arch_add_memory code path (well, and the error path also needs adjustment): /* * Self hosted memmap array */ if (mhp_flags & MHP_MEMMAP_ON_MEMORY) { if (!mhp_supports_memmap_on_memory(size)) { ret = -EINVAL; goto error; } mhp_altmap.free = PHYS_PFN(size); mhp_altmap.base_pfn = PHYS_PFN(start); params.altmap = &mhp_altmap; } /* call arch's memory hotadd */ ret = arch_add_memory(nid, start, size, ¶ms); if (ret < 0) goto error; Note that we want to handle that on a per memory-block basis, because we don't want the vmemmap of memory block #2 to end up on memory block #1. It all gets messy with memory onlining/offlining etc otherwise ... -- Cheers, David / dhildenb