From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22F8DC001DF for ; Thu, 3 Aug 2023 09:24:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87F61280221; Thu, 3 Aug 2023 05:24:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82EA32801EB; Thu, 3 Aug 2023 05:24:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CEF6280221; Thu, 3 Aug 2023 05:24:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5A55F2801EB for ; Thu, 3 Aug 2023 05:24:16 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2615EC1090 for ; Thu, 3 Aug 2023 09:24:16 +0000 (UTC) X-FDA: 81082257312.24.7FD584D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id AF5CA80011 for ; Thu, 3 Aug 2023 09:24:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AGIl5hWo; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691054653; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3WIFpcDecyPBPZDJa2C/HSA+GqxTetHp56fbdGINRyM=; b=lz5RUxDE1TyoXM3pTxW33Y2YflgFnL65pdN2f196SrBEpU1F4m+oT4BweoLnoybDRPtker DjCwQBe5AljWgtBCVey8wzdUxQMVO2CHhYBNNM1C0bCAoxSf0Tz6z3GWa/P2uQk/62pWtW zEJQDTsoyfvUGyXHtPqOpdmGzgnlTZU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AGIl5hWo; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691054653; a=rsa-sha256; cv=none; b=tW+OWq8oMY/QQ3FUBlgFuplX2/jW+T2MmU0kOm966dCGTqRowSUvqxJ5SAxaq9n0vVGxVc 1sFtdguSlSDKVfBx2zgr9FTiBQXy5CEzKIXZmDdLcwJD6FgObWot3LmoOpt2kCMH2yZFIL kaNCDNsrSa0Jp+vjaFDTy3VdI+JVx/w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691054652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3WIFpcDecyPBPZDJa2C/HSA+GqxTetHp56fbdGINRyM=; b=AGIl5hWoiIE3IjzbBY0o7q9DeXF6xvkc1n3RF1r0tMbXKA2xyc/ydPFTtqwksGOqA9HN8A 9ZsTPG1KOC8Rdg1YX5dlF0gVFPOBqVFwSslSAU+twm88GN/j98BBnvYxuplvAA3fJyeHB5 sN4uyD3CHtaOdwLUV89lUVhVKJUw0gk= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-258-zFLMppQBOJGLrKfKxbWdNg-1; Thu, 03 Aug 2023 05:24:11 -0400 X-MC-Unique: zFLMppQBOJGLrKfKxbWdNg-1 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-3fd2dec82a6so5005375e9.3 for ; Thu, 03 Aug 2023 02:24:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691054650; x=1691659450; h=content-transfer-encoding:in-reply-to:subject:organization:from :content-language:references:cc:to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3WIFpcDecyPBPZDJa2C/HSA+GqxTetHp56fbdGINRyM=; b=amt1R3cuUEpC0RDu1xiHIBU/pSXlchjIErDzTpwCziHFJ0wNHEY/NBJkBZ9vdYkPJO b3+xI8PjnXR+fpEboJCsabnNRKf+liNfOXONxFjxejnRfzxqME7qRK2YStKWE1fTbhQf NZb47RgvnAsINpn3yx07hG+vr4e0hwU+r6na/Izd01yS5OfHyjkJ7wU907LpZ66hX6DF rULMVrnboHw9lHeByenDR4haDKx6eSqUXrGnZj1kr3Q2VeFozscfwzSPFjjCjRyXGfLl wzNFlDAfgX4aZ8XwcCJNgbAuf0KBXaEIsa7ynlMBJIzIfX7uA3ffZp5YkSgPvIw10C3D c3TA== X-Gm-Message-State: ABy/qLZBgnG2eJvG12PW8p3S9ZhOhIRnuCvEo+pUaeRxJm4l/62D2yFs qRQyvTv3lUitzHtvxTiBW96axz4dSevLUOPKM2jCM/ABAr8w4Pk0pxjwhfcE2iESFuOeqkkDmed 3tmMfc20LhPs= X-Received: by 2002:a5d:4847:0:b0:313:eb29:4436 with SMTP id n7-20020a5d4847000000b00313eb294436mr7133221wrs.67.1691054650401; Thu, 03 Aug 2023 02:24:10 -0700 (PDT) X-Google-Smtp-Source: APBJJlFjX9fBwtEKHvbthMHsENAzcm9c2T2DJFmneGeXmLu4/R/dytL9uj33Y0dSxRui3+JSN4Xrog== X-Received: by 2002:a5d:4847:0:b0:313:eb29:4436 with SMTP id n7-20020a5d4847000000b00313eb294436mr7133199wrs.67.1691054649977; Thu, 03 Aug 2023 02:24:09 -0700 (PDT) Received: from ?IPV6:2003:cb:c718:9a00:a5f5:5315:b9fa:64df? (p200300cbc7189a00a5f55315b9fa64df.dip0.t-ipconnect.de. [2003:cb:c718:9a00:a5f5:5315:b9fa:64df]) by smtp.gmail.com with ESMTPSA id i15-20020adffdcf000000b003145559a691sm21291061wrs.41.2023.08.03.02.24.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 03 Aug 2023 02:24:09 -0700 (PDT) Message-ID: <1c6a74f0-85e9-5299-1520-9068e842b1a5@redhat.com> Date: Thu, 3 Aug 2023 11:24:08 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 To: Michal Hocko Cc: Aneesh Kumar K V , linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu, Oscar Salvador , Vishal Verma References: <20230801044116.10674-1-aneesh.kumar@linux.ibm.com> <20230801044116.10674-8-aneesh.kumar@linux.ibm.com> <31305ab7-1e65-80aa-ee91-9190c8f67430@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v7 7/7] mm/memory_hotplug: Enable runtime update of memmap_on_memory parameter In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: AF5CA80011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: g64juythojn9fb9nmt7r9i6ua3fqu3g3 X-HE-Tag: 1691054653-708066 X-HE-Meta: U2FsdGVkX1+eiKJaVgpvuWCHs6MWNgKmjEc+junSySsrdG2VfJDdY1z8tC3MXIJWjbNvLJ13yujx6WNA+nAlX5V5gPIr5K4XuLMcazJw8iW8mZOJDrAWpYoV14iM/k2sSQFVjUuLCC3mzcCXnlG7q37S6oktb1TODW1wmogWnUFup0LP1Ma5JtWd9qizpSvhD5SxcVaqh3/MBTJ4jGobFwbpFOAfkxSeJaxLfteF12Dxl915vGuOeZ89PqBk+I7FS+qG8JNX1IaqBybFEBWSda7WmjhDsyKvy0gmhR+mPI6lBknbc9IkCchW5b/bCAwbWEBbQaunei3pMi1SZelhsACm6OlyW99QJsYri+PcJVNizoNPZf7T8wqwTiKfxW/ntklVBC539bJm6IRD22SY7u49jf0syO1wyoAXo0+ThkYmcFuo80wGhSvysme35/4ygzcpRi3sC3BVTS9vXV8JyDFtb3QCMsl93/R/Lrtc4LT6jj/8VxHS0u9/4bDbHlhjP6SV9Lr20SN5UqdmDejyXZqNxD8E/bJrixaIzs7P+Cn5D+gmJQomlB5NGy8LqZThYnTEZH3FqGngkYmm5UIlWtBd7sBtCg/l60V61prmEIR1BBo77ytwLFE2MzZpnIFKaqlMvNvj/tZa2pMfgLdfBs6AoeP/W9vkmZPyHtkyhB35871icRcW1xenXh+w9PzvrCoq615v2i2cRFGhRDza/sgPXAVDtk2TX1Z+bDnuZ753AJX+zSwHjxAOxyDHrBC/ekQwZApfundfeuHjNwF4tbdDuIrwNAsfwuNrPIXGBRG0u5tHjINWqDhVM4sRSRrk/ygZuoGGV/NcSF0q+zBhA9ki5DGlrlPKLOjikPsORPJ77YIwhSm9Ax6XCYDfLXFQoXc3V09CH8kPBOoZ54dWEplHTg6g+7FOzPgsOnVo3/K4DrT8Mij3UnzqfTwx+ZeyD9BViT6VLn2bKyo2TWx K5T0+1Ds tVc4cVFUQbJ7VwEnKngNQoAcaM5XEtX0S2io92n8n7RqAmOxccRzwug2S6Iz/t3N7EEYZ6V45EafvxbWFsB5M7Wohs/bWncV/8ggKpqEvxul+nkn7hsFEKTwNPaqO4iw4LF6rmoFbNQExYfg7tYfi9SwI46WjUmCL0JEWdS5uUqcipCDUAxawF2F/MAIGAwyjUH8dLZedrgNUEJKS/QQxwlZ8jCmJaGjpfb+Su+qmyPytDZtP094kWLXCHDnGmjj2kZIlnPbJexFpAtZBojLTa2sPDiHt6tcMSPlL7AYi/OayECVM2mW684bNEw+Rf4lOSX3HBm4Ea+jJ1VJfgLfBWbN797gpxUyQ7yDBN3yAkdskXwO0UWdsacKjLEzTRqjtMlKM56IvGFt0SCulYkFKZa0hqZ+YbhZVUwYG4XTgwoWoVqAZToSiRjqSZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 03.08.23 10:52, Michal Hocko wrote: > On Wed 02-08-23 18:59:04, Michal Hocko wrote: >> On Wed 02-08-23 17:54:04, David Hildenbrand wrote: >>> On 02.08.23 17:50, Michal Hocko wrote: >>>> On Wed 02-08-23 10:15:04, Aneesh Kumar K V wrote: >>>>> On 8/1/23 4:20 PM, Michal Hocko wrote: >>>>>> On Tue 01-08-23 14:58:29, Aneesh Kumar K V wrote: >>>>>>> On 8/1/23 2:28 PM, Michal Hocko wrote: >>>>>>>> On Tue 01-08-23 10:11:16, Aneesh Kumar K.V wrote: >>>>>>>>> Allow updating memmap_on_memory mode after the kernel boot. Memory >>>>>>>>> hotplug done after the mode update will use the new mmemap_on_memory >>>>>>>>> value. >>>>>>>> >>>>>>>> Well, this is a user space kABI extension and as such you should spend >>>>>>>> more words about the usecase. Why we could live with this static and now >>>>>>>> need dynamic? >>>>>>>> >>>>>>> >>>>>>> This enables easy testing of memmap_on_memory feature without a kernel reboot. >>>>>> >>>>>> Testing alone is rather weak argument to be honest. >>>>>> >>>>>>> I also expect people wanting to use that when they find dax kmem memory online >>>>>>> failing because of struct page allocation failures[1]. User could reboot back with >>>>>>> memmap_on_memory=y kernel parameter. But being able to enable it via sysfs makes >>>>>>> the feature much more useful. >>>>>> >>>>>> Sure it can be useful but that holds for any feature, right. The main >>>>>> question is whether this is worth maintaing. The current implementation >>>>>> seems rather trivial which is an argument to have it but are there any >>>>>> risks long term? Have you evaluated a potential long term maintenance >>>>>> cost? There is no easy way to go back and disable it later on without >>>>>> breaking some userspace. >>>>>> >>>>>> All that should be in the changelog! >>>>> >>>>> I updated it as below. >>>>> >>>>> mm/memory_hotplug: Enable runtime update of memmap_on_memory parameter >>>>> >>>>> Allow updating memmap_on_memory mode after the kernel boot. Memory >>>>> hotplug done after the mode update will use the new mmemap_on_memory >>>>> value. >>>>> >>>>> It is now possible to test the memmap_on_memory feature easily without >>>>> the need for a kernel reboot. Additionally, for those encountering >>>>> struct page allocation failures while using dax kmem memory online, this >>>>> feature may prove useful. Instead of rebooting with the >>>>> memmap_on_memory=y kernel parameter, users can now enable it via sysfs, >>>>> which greatly enhances its usefulness. >>>> >>>> >>>> I do not really see a solid argument why rebooting is really a problem >>>> TBH. Also is the global policy knob really the right fit for existing >>>> hotplug usecases? In other words, if we really want to make >>>> memmap_on_memory more flexible would it make more sense to have it per >>>> memory block property instead (the global knob being the default if >>>> admin doesn't specify it differently). >>> >>> Per memory block isn't possible, due to the creation order. >> >> I am not sure I follow. Could you elaborate more? > > Must have been a tired brain. Now I see what you mean of course. vmemmap > is allocated at the time the range is registered and therefore memory > block sysfs created. So you are right that it is too late in some sense > but I still believe that this would be doable. The memmap_on_memory file Exactly. > would be readable only when the block is offline and it would reallocate > vmemmap on the change. Makes sense? Are there any risks? Maybe pfn > walkers? The question is: is it of any real value such that it would be worth the cost and risk? One of the primary reasons for memmap_on_memory is that *memory hotplug* succeeds even in low-memory situations (including, low on ZONE_NORMAL situations). So you want that behavior already when hotplugging such devices. While there might be value to relocate it later, I'm not sure if that is really worth it, and it does not solve the main use case. For dax/kmem memory hotplug this is controlled by user space, so user space can easily configure it. For DIMMs it's just a hotplug event that triggers all that in the kernel, and one has to plan ahead (e.g., set the global kernel parameter). Besides, devices like virtio-mem really cannot universally support memmap_on_memory. -- Cheers, David / dhildenb