From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EC6CEB64D9 for ; Thu, 6 Jul 2023 09:19:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC7238D0002; Thu, 6 Jul 2023 05:19:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E77FC8D0001; Thu, 6 Jul 2023 05:19:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D403B8D0002; Thu, 6 Jul 2023 05:19:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C51E78D0001 for ; Thu, 6 Jul 2023 05:19:06 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7F0DA40434 for ; Thu, 6 Jul 2023 09:19:06 +0000 (UTC) X-FDA: 80980637892.16.126816D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 2134A20005 for ; Thu, 6 Jul 2023 09:19:03 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XeK+OYXp; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688635144; a=rsa-sha256; cv=none; b=5hYr4JEbbsl24Goubh3tvBxZD0nZkMnQGAyj6tOJvuyY+XbiuhwgyqapgUXPOiNF/H2UEH TdEC7RU6TdI/78705dyBNeg9J9wiTjmPlEaqqXbkf99aS5rkjMfNAroFBzXmcTxlxG/x7n S2GWXZO2XV7pWJPC5YqZbpY8xkIlQ0w= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XeK+OYXp; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688635144; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RPebE8QWnPzwlWbjblsdxk7DEkuF3S+UJHZ3Izk2Tco=; b=zySJYobGLL0H8pCKVZFluv/pqEwzW3NOt5JVtoAzzm/UdrpVZR7yHEEmHFM+JQ4tWUZUfA YTtPjl3Cj5Csm8q4Njl9RHt3gwXwAIaIT8Li1j5moHvpwdIe7lZMUC3xVAZFk3KjAGXwxD a8akl+MV/SCWf1OPqynOiJx/SsQPhcA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688635143; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RPebE8QWnPzwlWbjblsdxk7DEkuF3S+UJHZ3Izk2Tco=; b=XeK+OYXpdVG9FUewgn5J1DTDuYRIsIhCSXLM0kFIAi6IvrgB4rkvPt5oWlINt5y+Pupv6k ovztRhY9asVcx//NdtaO278JOq0Whkeg+Asp4h0PYMPuZNuo5/NKAfg7nHRgDq3Mcl/w/q TeMQT39LKyo3fdfJF3fW8SV1ex+CZB4= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-Zhrr1qBDOMOxdjF75XLxHg-1; Thu, 06 Jul 2023 05:19:01 -0400 X-MC-Unique: Zhrr1qBDOMOxdjF75XLxHg-1 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-4fb7b4be07bso444218e87.1 for ; Thu, 06 Jul 2023 02:19:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688635140; x=1691227140; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RPebE8QWnPzwlWbjblsdxk7DEkuF3S+UJHZ3Izk2Tco=; b=VG/228L4wC34MhCuniFxhVJv2TE0By2rKUCqZZNG8VTFMqBhJGxopo9/cNOxUvY5Lx qS6EmZOzZeYpL9f+pG/LUdV6cI/UnUIIdkilPxj1krv4FX+T1jPqjMpBXh3J8vqLTOyM EP08NUWhsZSQ4OIvGw7TckqoAUfIB2fXI/gl4FblXyU77+XA+MH/etI3Z6x3vKLGKjHy XhIrmTgixNwBYuBz6oCsZ0BpgljxlwkD9o8AGQgbplkrIyUTs67Zk4x/fPeI1fk3s2IU 1KNdHVFwvO3EMJsQqch89Q7aTDN1L/ts68qJ+xBPRwGGVOPX6elI7aGiMGI5DF0343P3 4bHQ== X-Gm-Message-State: ABy/qLZpHnrQXV1MOpuMyUyHZMbSkoUMATOtc74ixkZBniM86cwjx88j hG58YGVKDYmcwLCUmeqJQ4z7KyXCCjFxlhcz7qmRvNCkpdn7khnx6fyCIvBtEjlez2LZuBblfW8 k0Oj99FUmM1c= X-Received: by 2002:a19:5f1c:0:b0:4f8:67aa:4f03 with SMTP id t28-20020a195f1c000000b004f867aa4f03mr1059298lfb.1.1688635140394; Thu, 06 Jul 2023 02:19:00 -0700 (PDT) X-Google-Smtp-Source: APBJJlFCniOQLT9lvwUgCkA8BLQRCj3UZt9TtN428IpglPeLw3ODUKsTJ4WV8Tyv+ejt6GaRMR7aiw== X-Received: by 2002:a19:5f1c:0:b0:4f8:67aa:4f03 with SMTP id t28-20020a195f1c000000b004f867aa4f03mr1059284lfb.1.1688635140041; Thu, 06 Jul 2023 02:19:00 -0700 (PDT) Received: from ?IPV6:2a09:80c0:192:0:5dac:bf3d:c41:c3e7? ([2a09:80c0:192:0:5dac:bf3d:c41:c3e7]) by smtp.gmail.com with ESMTPSA id r13-20020a05600c458d00b003fa9a00d74csm5350128wmo.3.2023.07.06.02.18.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 06 Jul 2023 02:18:59 -0700 (PDT) Message-ID: <72488b8a-8f1e-c652-ab48-47e38290441f@redhat.com> Date: Thu, 6 Jul 2023 11:18:58 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: "Aneesh Kumar K.V" , linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Michal Hocko , Vishal Verma References: <20230706085041.826340-1-aneesh.kumar@linux.ibm.com> <20230706085041.826340-2-aneesh.kumar@linux.ibm.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v2 1/5] mm/hotplug: Embed vmem_altmap details in memory block In-Reply-To: <20230706085041.826340-2-aneesh.kumar@linux.ibm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2134A20005 X-Stat-Signature: 5a9nefkphf5qrei65m1fxkq6kkh7376i X-Rspam-User: X-HE-Tag: 1688635143-68251 X-HE-Meta: U2FsdGVkX1/cwTq3eqTFn5gh1TILJ6aVDHCWgo0A3LEaiPMg0GowXNlI82eRB1lxDnooJhBW3OjpXriihMHYvWKGbphT9qfH71AYRi+Z8DfmtSJwWIh3WX7knRygYs2yPa9AEnZuki9iWeAmtuE3yfXmCFt787lFj4z8OZO8pcUhNO659UVhos1C3cJXlzihG0pVJJUceLI/DI6Qna5AYlZSi7KKO5Bb3tK7DNd983DdbCXtIF5QUmwQKU4M33yWQ78s0PfAgVRaRLS5h2B4DQPkXi+PRM2END6fOKbYmbrbLPpP23WobSy0arX8r2shpWmQBDQuO8s3jCgcVbpy4zieEN8JipIRgZXM5g00BQsP0A12sHTvZLg6aWvfpn8fsOkKRDtfYBJObjpyvLcoBfoAPdzEw+bsY6H1RYcG1U6cDfMfq8abP9Bwy3wqJZ6IoW/JyOjHueFFRJaK5n1hoQs/8l1okoWwQ1JyzG0q/0YqvUYRFpSvunjwkbj7j5/2ten/uSQwP3LJLtPDoxHupeWEkvOu3tv29ReO8lErISPl0xjqIMTVe6N/hiqaEPLcvKaOAOmgsUkhbecxmpuDS/81EAfJa3sD36g1HiJojJOi9uxhwxriMfuFLQk30Be7cZhXgcKAafS4pByK8tV1SnMg4NIsoQkkxF4OjOcY1KXCdhygo95iDfpiR1LrK+/qEOzGTZap3HrOZcWucfJrdXw690DExpW/rNyu/7MyaRTt8f9CnhS2m8Ec/BIbSOMjotRUNH3fq8Fa9bkV1FDS5FL8JSvWUqUQKRProCFI4MaBl4+1ssBdeX3D6I1UipMmUerBMWWYqLzP9GaMNvdSk+/kJYiRYTWjEU8in12z3/mQSBWi1M+jUPIQ1vDpXKEJDm/VFv3ar6ZyZgnAeAlPXBCwFzMi6ppzcGOuFcn2r+XbOiMOYUylG/8YT10w1Gn0TCAMXzBmXCog4L7c2wo iNHo09YF gfggPJwvzgxSfiy2UKq/efwVzrDEhwoNOg1n4iq13YqsybjME9ES4gwGQB/Aa1FDhqPvvZqySi4yK9jx8A2uyEUWeaPkUIdRUGbSgYgJRQn5kfU1f1py0yM7GxPz8FA12h3t9wj8LARwIG4ADs/cTMGMNllJ4HHqk1pqGpYFemjY5mpHzZkXTtYS1t+6lGfYS4mlZuyRnAGGjH1K8u2dEknmAGM7Q2datwALHcB071ozkCv7Ioa0zmbtAdI86LScIRxf+bRFWx4axuIuLjAfxUFzNeFkw8P1A9+hPAYIYc7GRajcXheKOKTjmWr1rt8e/APmHoodKM8I9l0UdXS6/PB/o5QhX0vbbgAofC1zFhvFOtFG6D0Q61QYFsgnGt4dV68f62IH5yqSNJB6wlmy2nwrH5hDreksAyf3jAGrrLN0vPU92pNEcR0uLwoalDHfnRcDP52M8eLD9GI4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 06.07.23 10:50, Aneesh Kumar K.V wrote: > With memmap on memory, some architecture needs more details w.r.t altmap > such as base_pfn, end_pfn, etc to unmap vmemmap memory. Can you elaborate why ppc64 needs that and x86-64 + aarch64 don't? IOW, why can't ppc64 simply allocate the vmemmap from the start of the memblock (-> base_pfn) and use the stored number of vmemmap pages to calculate the end_pfn? To rephrase: if the vmemmap is not at the beginning and doesn't cover full apgeblocks, memory onlining/offlining would be broken. [...] > > +/** > + * struct vmem_altmap - pre-allocated storage for vmemmap_populate > + * @base_pfn: base of the entire dev_pagemap mapping > + * @reserve: pages mapped, but reserved for driver use (relative to @base) > + * @free: free pages set aside in the mapping for memmap storage > + * @align: pages reserved to meet allocation alignments > + * @alloc: track pages consumed, private to vmemmap_populate() > + */ > +struct vmem_altmap { > + unsigned long base_pfn; > + const unsigned long end_pfn; > + const unsigned long reserve; > + unsigned long free; > + unsigned long align; > + unsigned long alloc; > +}; Instead of embedding that, what about conditionally allocating it and store a pointer to it in the "struct memory_block"? In the general case as of today, we don't have an altmap. > + > struct memory_block { > unsigned long start_section_nr; > unsigned long state; /* serialized by the dev->lock */ > @@ -77,11 +94,7 @@ struct memory_block { > */ > struct zone *zone; > struct device dev; > - /* > - * Number of vmemmap pages. These pages > - * lay at the beginning of the memory block. > - */ > - unsigned long nr_vmemmap_pages; > + struct vmem_altmap altmap; > struct memory_group *group; /* group (if any) for this block */ > struct list_head group_next; /* next block inside memory group */ > #if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_MEMORY_HOTPLUG) > @@ -147,7 +160,7 @@ static inline int hotplug_memory_notifier(notifier_fn_t fn, int pri) > extern int register_memory_notifier(struct notifier_block *nb); > extern void unregister_memory_notifier(struct notifier_block *nb); > int create_memory_block_devices(unsigned long start, unsigned long size, [...] > static int check_cpu_on_node(int nid) > @@ -2036,9 +2042,8 @@ EXPORT_SYMBOL(try_offline_node); > > static int __ref try_remove_memory(u64 start, u64 size) > { > - struct vmem_altmap mhp_altmap = {}; > + int ret; > struct vmem_altmap *altmap = NULL; > - unsigned long nr_vmemmap_pages; > int rc = 0, nid = NUMA_NO_NODE; > > BUG_ON(check_hotplug_memory_range(start, size)); > @@ -2060,24 +2065,16 @@ static int __ref try_remove_memory(u64 start, u64 size) > * We only support removing memory added with MHP_MEMMAP_ON_MEMORY in > * the same granularity it was added - a single memory block. > */ > + ^ unrealted change? -- Cheers, David / dhildenb