From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9574BEB64D7 for ; Fri, 23 Jun 2023 08:41:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EB868D0002; Fri, 23 Jun 2023 04:41:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C2A68D0001; Fri, 23 Jun 2023 04:41:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA5648D0002; Fri, 23 Jun 2023 04:41:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D75428D0001 for ; Fri, 23 Jun 2023 04:41:19 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A2724140DCD for ; Fri, 23 Jun 2023 08:41:19 +0000 (UTC) X-FDA: 80933368278.30.5A8844A Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf05.hostedemail.com (Postfix) with ESMTP id 41072100007 for ; Fri, 23 Jun 2023 08:41:16 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=sSIyXqaZ; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf05.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687509676; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wbDpukB6EcoDMgMhtdtYou3jl/+Z9bf22pAkaqy4JPQ=; b=t5m6Sarc9i6+I11DYSYsveS6fDdXFv7HVaySYhxQd9OltJ9WvrnsRmj1M7XnjAStn52LUK aPnbnUbOgerIaTdsdYTiBORWNKMFKFic9Nz6har60znFU4YpclzlAdPwqN+T7YhmsuK4zL /7YnpULhYcVl9KZQwHXqNzw+i9DneLs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=sSIyXqaZ; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf05.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687509676; a=rsa-sha256; cv=none; b=MtlTSceMTJSWivo5M18AqUs11FQ5nmi+vDc0ICgQGuqF848KF26XsuUCASCjzYKY6MzN3R r6IIA0Qz8zbo3hDsZzHZaBDTyB5bcu4mqVW2w1wATlMyZlaC1lSKtpHGz4U+z3MNjIea0z q53Po/IYOchjSZlO4whqtzp2CDEOF8E= Received: from pps.filterd (m0353727.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35N8ei0q008734; Fri, 23 Jun 2023 08:41:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : mime-version : content-type; s=pp1; bh=wbDpukB6EcoDMgMhtdtYou3jl/+Z9bf22pAkaqy4JPQ=; b=sSIyXqaZfghlh5i9uh2w1gT9JP5qu19yqZSo1hyyi6+8hJFsl+5XbyVcFKpsWiAoy8Pe 81nfIiZCXFeAke0Vmit4I6pCd1a5eY0vHZcIa7cRLmBCtKkXd/MrM10Jxd4lFkVfoDjb 8mAy4gX65hwxqO0rNSOSxHL+1i85R7Ahbxv1RW8y/Gsq0Wrf2mvPPahMUZ9vKA6qDjvw 8WMePMuxXeNodKHVGDKoZIISjsXpfw5FQEf+jScxoemZ0IFMX3DEktdGbq2GWGu+LKw3 YRNPvt8WpUM+wGk2Gbm8BgnrJw2AvFdPGBQkRaH+sAXW+CzyFQXB/mem2GNqNr0gZBc2 DQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rd7upre2c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Jun 2023 08:41:09 +0000 Received: from m0353727.ppops.net (m0353727.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35N8erGS009097; Fri, 23 Jun 2023 08:41:09 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rd7upre1u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Jun 2023 08:41:09 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35N7dHUt001270; Fri, 23 Jun 2023 08:41:07 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3r94f6e0gq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Jun 2023 08:41:07 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35N8f7D342991886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 23 Jun 2023 08:41:07 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 113FB58068; Fri, 23 Jun 2023 08:41:07 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CF3ED58056; Fri, 23 Jun 2023 08:41:01 +0000 (GMT) Received: from skywalker.linux.ibm.com (unknown [9.43.18.202]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Fri, 23 Jun 2023 08:41:01 +0000 (GMT) X-Mailer: emacs 29.0.91 (via feedmail 11-beta-1 I) From: "Aneesh Kumar K.V" To: Vishal Verma , "Rafael J. Wysocki" , Len Brown , Andrew Morton , David Hildenbrand , Oscar Salvador , Dan Williams , Dave Jiang Cc: linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, Huang Ying , Dave Hansen , Vishal Verma Subject: Re: [PATCH 1/3] mm/memory_hotplug: Allow an override for the memmap_on_memory param In-Reply-To: <20230613-vv-kmem_memmap-v1-1-f6de9c6af2c6@intel.com> References: <20230613-vv-kmem_memmap-v1-0-f6de9c6af2c6@intel.com> <20230613-vv-kmem_memmap-v1-1-f6de9c6af2c6@intel.com> Date: Fri, 23 Jun 2023 14:10:58 +0530 Message-ID: <87mt0qy3j9.fsf@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: -2aAWK_DDa6XyaRL594jM6M3_jR34vSM X-Proofpoint-GUID: af9dV7ZApH8KD2C7zgQKTGsabPIJtEn6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-06-23_03,2023-06-22_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 spamscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 clxscore=1011 suspectscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306230076 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 41072100007 X-Stat-Signature: 3dyy8588pkcc66zwca91dcsxz6znetnz X-Rspam-User: X-HE-Tag: 1687509676-982592 X-HE-Meta: U2FsdGVkX19s+yMla8boOu6px4TsQBg5Cg6c6WRkThS4lmfpBSUKlCQuYWF+Wdf9gtwm4aUIhNrnjXe0wBLNJGxqZadn133j5izTTkSqFSALeGIswWJK9CsBfoq1F6C4C7P9czcTEhGpG1yw7q1jKe71d1CODlPocidPIoep12Q1L6fqBlp6FvEnXGWX7khiotSM7M0rFkf1u/VnVYlUuKw+UJUIXKz4HKGhPPsbMIH43XBPFzmUNETQBoZBRP6LoPs3qe0bh4jfpn1I+khE2FzviRODTNu2iyi4bhv7++htzffjFxTtcEmB+SwiLh+UBaeSKjk33/sJbszREcZJrof8Uyz9S4EmrnZBlqy388dk4I5mnEyF7B45I+aYt3FicKUea7NY2u11OSUC/aKJDw5AZWpbga7ejJKewJvG8GFz7RU4EIil17t4bjmt+nM3IHHIcSYdMTTmijB7bkALlS/aJc4kslmoNhRhz8MJ9Y3HHLYs282XU6TiCt9fC0vYLfYJim+Vguv4FIuUDIuzEQIW6RRtlzMNqDrzxoqwZoHAwwf/iS8fofheqQt3gSQ8kTM6U0+lyL8hwDe75ObO1G5FsklhVpyJxN/7MFQH3TlXjStZG+w/QDJUyYFs5wM1enKOCRW86C7/4XWbW1AnARFfu0m0SoC4bSNTiG4tTV05R4+3zNSenD/EbSY8mv+frDfRQxB6rnudyeIFRrPzHW7YLS5lwcmybFHV7kybg50NfSe+OGDYinqq22oTyZJkBLE4JqEmT1b+5q/G/b6O6tH1c3vJkqJigQEd/bgMxpI/heSHa9fFNRxRd/iAs2CQxS5m5XWYpADNHLndzs7w5UwD4j+ZrAgrGXiJsjEwwe+qGB6ZcuF7xXTsgk+MkvsplQEB1A39gQZdVAe2a55nqKfcy1awOo2EW7R/baNk3SVva1nuuAgdychC6A9UJ+slGHR3SG/Qd+8iRwooMjT XubPaECf tRzJBLa+32ICoQhtyWmNQh4A8F+jyw0+SAogryy5oWwzyl9vLlayyki5K5+ZdDNmF8QwU23Xq58NIHNWvCJls/2q1zsPf0FFtGWdWnsqvjMwSIiFNphlI2eOw5he9+htTlha6eQAXAnpt3Sql/L/FjXSWQk/2ie2sA/psCR+gSWUbHnBVchDH+w4yLJYFC7RoCUl2OP/7j+s0W9LJWdA5Nn0dzj24qGCcljixRlKrvlChL6fbRNc+e2OwK/rtHQNf3k4ABKM/0bUH8moMusZwj+7pPL8SaKgZ0SR7NANnzgvFJxlsflPGTSqhZJFT686jo3xW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Vishal Verma writes: > For memory hotplug to consider MHP_MEMMAP_ON_MEMORY behavior, the > 'memmap_on_memory' module parameter was a hard requirement. > > In preparation for the dax/kmem driver to use memmap_on_memory > semantics, arrange for the module parameter check to be bypassed via the > appropriate mhp_flag. > > Recall that the kmem driver could contribute huge amounts of hotplugged > memory originating from special purposes devices such as CXL memory > expanders. In some cases memmap_on_memory may be the /only/ way this new > memory can be hotplugged. Hence it makes sense for kmem to have a way to > force memmap_on_memory without depending on a module param, if all the > other conditions for it are met. > > The only other user of this interface is acpi/acpi_memoryhotplug.c, > which only enables the mhp_flag if an initial > mhp_supports_memmap_on_memory() test passes. Maintain the existing > behavior and semantics for this by performing the initial check from > acpi without the MHP_MEMMAP_ON_MEMORY flag, so its decision falls back > to the module parameter. > > Cc: "Rafael J. Wysocki" > Cc: Len Brown > Cc: Andrew Morton > Cc: David Hildenbrand > Cc: Oscar Salvador > Cc: Dan Williams > Cc: Dave Jiang > Cc: Dave Hansen > Cc: Huang Ying > Signed-off-by: Vishal Verma > --- > include/linux/memory_hotplug.h | 2 +- > drivers/acpi/acpi_memhotplug.c | 2 +- > mm/memory_hotplug.c | 24 ++++++++++++++++-------- > 3 files changed, 18 insertions(+), 10 deletions(-) > > diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h > index 9fcbf5706595..c9ddcd3cad70 100644 > --- a/include/linux/memory_hotplug.h > +++ b/include/linux/memory_hotplug.h > @@ -358,7 +358,7 @@ extern struct zone *zone_for_pfn_range(int online_type, int nid, > extern int arch_create_linear_mapping(int nid, u64 start, u64 size, > struct mhp_params *params); > void arch_remove_linear_mapping(u64 start, u64 size); > -extern bool mhp_supports_memmap_on_memory(unsigned long size); > +extern bool mhp_supports_memmap_on_memory(unsigned long size, mhp_t mhp_flags); > #endif /* CONFIG_MEMORY_HOTPLUG */ > > #endif /* __LINUX_MEMORY_HOTPLUG_H */ > diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c > index 24f662d8bd39..119d3bb49753 100644 > --- a/drivers/acpi/acpi_memhotplug.c > +++ b/drivers/acpi/acpi_memhotplug.c > @@ -211,7 +211,7 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) > if (!info->length) > continue; > > - if (mhp_supports_memmap_on_memory(info->length)) > + if (mhp_supports_memmap_on_memory(info->length, 0)) > mhp_flags |= MHP_MEMMAP_ON_MEMORY; > result = __add_memory(mgid, info->start_addr, info->length, > mhp_flags); > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c > index 8e0fa209d533..bb3845830922 100644 > --- a/mm/memory_hotplug.c > +++ b/mm/memory_hotplug.c > @@ -1283,15 +1283,21 @@ static int online_memory_block(struct memory_block *mem, void *arg) > return device_online(&mem->dev); > } > > -bool mhp_supports_memmap_on_memory(unsigned long size) > +bool mhp_supports_memmap_on_memory(unsigned long size, mhp_t mhp_flags) > { > unsigned long nr_vmemmap_pages = size / PAGE_SIZE; > unsigned long vmemmap_size = nr_vmemmap_pages * sizeof(struct page); > unsigned long remaining_size = size - vmemmap_size; > > /* > - * Besides having arch support and the feature enabled at runtime, we > - * need a few more assumptions to hold true: > + * The MHP_MEMMAP_ON_MEMORY flag indicates a caller that wants to force > + * memmap_on_memory (if other conditions are met), regardless of the > + * module parameter. drivers/dax/kmem.c is an example, where large > + * amounts of hotplug memory may come from, and the only option to > + * successfully online all of it is to place the memmap on this memory. > + * > + * Besides having arch support and the feature enabled at runtime or > + * via the mhp_flag, we need a few more assumptions to hold true: > * > * a) We span a single memory block: memory onlining/offlinin;g happens > * in memory block granularity. We don't want the vmemmap of online > @@ -1315,10 +1321,12 @@ bool mhp_supports_memmap_on_memory(unsigned long size) > * altmap as an alternative source of memory, and we do not exactly > * populate a single PMD. > */ > - return mhp_memmap_on_memory() && > - size == memory_block_size_bytes() && > - IS_ALIGNED(vmemmap_size, PMD_SIZE) && > - IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); > + > + if ((mhp_flags & MHP_MEMMAP_ON_MEMORY) || mhp_memmap_on_memory()) > + return size == memory_block_size_bytes() && > + IS_ALIGNED(vmemmap_size, PMD_SIZE) && > + IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); > + return false; > } > > /* > @@ -1375,7 +1383,7 @@ int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags) > * Self hosted memmap array > */ > if (mhp_flags & MHP_MEMMAP_ON_MEMORY) { > - if (!mhp_supports_memmap_on_memory(size)) { > + if (!mhp_supports_memmap_on_memory(size, mhp_flags)) { > ret = -EINVAL; > goto error; > } > I was working on enabling memmap_on_memory for ppc64 and came across this. Can we do this as below? commit d55eddf71032cad3e057d8fa4c61ad2b8e05aaa8 Author: Aneesh Kumar K.V Date: Fri Jun 23 12:11:31 2023 +0530 mm/hotplug: Allow to force enable memmap on memory In some cases like DAX kmem, we want to enable memmap on memory even if we have memory_hotplug.memmap_on_memory disabled. This patch enables such usage. DAX kmem can use MHP_MEMMAP_ON_MEMORY directly to request for memmap on memory. Signed-off-by: Aneesh Kumar K.V diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c index 24f662d8bd39..4d0096fc4cc2 100644 --- a/drivers/acpi/acpi_memhotplug.c +++ b/drivers/acpi/acpi_memhotplug.c @@ -211,8 +211,7 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) if (!info->length) continue; - if (mhp_supports_memmap_on_memory(info->length)) - mhp_flags |= MHP_MEMMAP_ON_MEMORY; + mhp_flags |= get_memmap_on_memory_flags(); result = __add_memory(mgid, info->start_addr, info->length, mhp_flags); diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 9fcbf5706595..f7f534084393 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -359,6 +359,13 @@ extern int arch_create_linear_mapping(int nid, u64 start, u64 size, struct mhp_params *params); void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); +bool mhp_memmap_on_memory(void); #endif /* CONFIG_MEMORY_HOTPLUG */ +static inline unsigned long get_memmap_on_memory_flags(void) +{ + if (mhp_memmap_on_memory()) + return MHP_MEMMAP_ON_MEMORY; + return 0; +} #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index eb18d1ac7e84..85d29909fd89 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -50,7 +50,7 @@ static bool memmap_on_memory __ro_after_init; module_param(memmap_on_memory, bool, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); -static inline bool mhp_memmap_on_memory(void) +bool mhp_memmap_on_memory(void) { return memmap_on_memory; } @@ -1316,10 +1316,9 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * altmap as an alternative source of memory, and we do not exactly * populate a single PMD. */ - return mhp_memmap_on_memory() && - size == memory_block_size_bytes() && - IS_ALIGNED(vmemmap_size, PMD_SIZE) && - IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); + return size == memory_block_size_bytes() && + IS_ALIGNED(vmemmap_size, PMD_SIZE) && + IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); } /*