From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56EB5C433EF for ; Fri, 24 Sep 2021 08:17:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0ED0860F41 for ; Fri, 24 Sep 2021 08:17:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0ED0860F41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5EECC6B006C; Fri, 24 Sep 2021 04:17:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 59FEA900002; Fri, 24 Sep 2021 04:17:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 466636B0073; Fri, 24 Sep 2021 04:17:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 368A16B006C for ; Fri, 24 Sep 2021 04:17:53 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E40FD32625 for ; Fri, 24 Sep 2021 08:17:52 +0000 (UTC) X-FDA: 78621763584.18.4A7A4D8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 6FCA3102FAD7 for ; Fri, 24 Sep 2021 08:17:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632471471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hv0DavdoyFzFPDaDYsQZiKG5xroiwpLoHl3JpBTrMIc=; b=XulkcWNl5r2rELkLnvxypXqEcU01x2WQqA5KHruYO3FIY8GW56MMk00GW4Z0kxRKSpKpdm JsWeil4PlP5ZmgpZUnIWYn3GG64EWT+W26QLOoqXvH2TQ9ucQcm50NyCE11HUeHdRfrDB3 cK0thNl+gU0po+M06sPhWXiw2QTxmt0= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-170-JAbyP0c2OcqalEuNKOZyeA-1; Fri, 24 Sep 2021 04:17:49 -0400 X-MC-Unique: JAbyP0c2OcqalEuNKOZyeA-1 Received: by mail-wr1-f69.google.com with SMTP id s14-20020adff80e000000b001601b124f50so7397972wrp.5 for ; Fri, 24 Sep 2021 01:17:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:to:cc:references:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=Hv0DavdoyFzFPDaDYsQZiKG5xroiwpLoHl3JpBTrMIc=; b=AyIbGSOg17aYxIM9c8IzzssTpFT728pxdcW4O9IjihqGhqswLVaxE7PuN/qu395zfN 1sjiDYSvaYioFDZ3Tl5SXpkKN3lVYZCGXskpdGd60qAGx1xY9nSPC97+rF3cfBG2/sTa pMWonOqdhpEc/AgMjmpM0mtN1VchPApB0Qm4DdhuqGkdyTVmdloHmP7mz6Xtrk4+Bg8b Nwp92BWT3EWyiB+B+HR+Qso22qjOdbPUXf49ZyQ5gUxo+SLmxVI+qdefyjDFIFSZGU2A wARSAj++doP5nXFdm3ragP4XVd2OpWhtI5Spo/n2gcrpFWhm4Wb06gA0erVanBGTxmgu E5pQ== X-Gm-Message-State: AOAM531RuzTt9biqoyp2OCCHXa1Jv4IS6p5/92dhIb5zw9rOI2mGB5zU dtO2D05QFbLWHcmKbE5lHket2MCSlr0t0NT9ScqToOfi2Wzai5E85Rn2HZLblJSE1oQJsQpol3U B+K8RdB7SfEw= X-Received: by 2002:a1c:f405:: with SMTP id z5mr671657wma.33.1632471467847; Fri, 24 Sep 2021 01:17:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzrH74aZKFo1LlKFHQ42orrvv8hwQA4S5bh26q5ompTUH91rPUY1MEp2TJJUur1lv+fSj85rg== X-Received: by 2002:a1c:f405:: with SMTP id z5mr671634wma.33.1632471467602; Fri, 24 Sep 2021 01:17:47 -0700 (PDT) Received: from [192.168.3.132] (p5b0c61fc.dip0.t-ipconnect.de. [91.12.97.252]) by smtp.gmail.com with ESMTPSA id f1sm7642302wri.43.2021.09.24.01.17.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Sep 2021 01:17:47 -0700 (PDT) To: Florian Fainelli , Chris Goldsworthy , Catalin Marinas , Will Deacon , Andrew Morton Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sudarshan Rajagopalan , Doug Berger References: <595d09279824faf1f54961cef52b745609b05d97.1632437225.git.quic_cgoldswo@quicinc.com> <6eb8319d-acba-b69a-5db3-5dca9ef426e8@gmail.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [RFC] arm64: mm: update max_pfn after memory hotplug Message-ID: <41789cad-76c6-0ea5-4aa1-3e4a52acff86@redhat.com> Date: Fri, 24 Sep 2021 10:17:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <6eb8319d-acba-b69a-5db3-5dca9ef426e8@gmail.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XulkcWNl; spf=none (imf13.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6FCA3102FAD7 X-Stat-Signature: afpz8mpfrsbjwpi9k5kck5qza6ye4y9s X-HE-Tag: 1632471472-17820 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 24.09.21 04:47, Florian Fainelli wrote: >=20 >=20 > On 9/23/2021 3:54 PM, Chris Goldsworthy wrote: >> From: Sudarshan Rajagopalan >> >> After new memory blocks have been hotplugged, max_pfn and max_low_pfn >> needs updating to reflect on new PFNs being hot added to system. >> >> Signed-off-by: Sudarshan Rajagopalan >> Signed-off-by: Chris Goldsworthy >> --- >> arch/arm64/mm/mmu.c | 5 +++++ >> 1 file changed, 5 insertions(+) >> >> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c >> index cfd9deb..fd85b51 100644 >> --- a/arch/arm64/mm/mmu.c >> +++ b/arch/arm64/mm/mmu.c >> @@ -1499,6 +1499,11 @@ int arch_add_memory(int nid, u64 start, u64 siz= e, >> if (ret) >> __remove_pgd_mapping(swapper_pg_dir, >> __phys_to_virt(start), size); >> + else { >> + max_pfn =3D PFN_UP(start + size); >> + max_low_pfn =3D max_pfn; >> + } >=20 > This is a drive by review, but it got me thinking about your changes a = bit: >=20 > - if you raise max_pfn when you hotplug memory, don't you need to lower > it when you hot unplug memory as well? The issue with lowering is that you actually have to do some search to=20 figure out the actual value -- and it's not really worth the trouble.=20 Raising the limit is easy. With memory hotunplug, anybody wanting to take a look at a "struct page"=20 via a pfn has to do a pfn_to_online_page() either way. That will fail if=20 there isn't actually a memmap anymore because the memory has been=20 unplugged. So "max_pfn" is actually rather a hint what maximum pfn to=20 look at, and it can be bigger than it actually is. The a look at the example usage in fs/proc/page.c:kpageflags_read() pfn_to_online_page() will simply fail and stable_page_flags() will=20 indicate a KPF_NOPAGE. Just like we would have a big memory hole now at the end of memory. >=20 > - suppose that you have a platform which maps physical memory into the > CPU's address space at 0x00_4000_0000 (1GB offset) and the kernel boots > with 2GB of DRAM plugged by default. At that point we have not > registered a swiotlb because we have less than 4GB of addressable > physical memory, there is no IOMMU in that system, it's a happy world. > Now assume that we plug an additional 2GB of DRAM into that system > adjacent to the previous 2GB, from 0x00_C0000_0000 through > 0x14_0000_0000, now we have physical addresses above 4GB, but we still > don't have a swiotlb, some of our DMA_BIT_MASK(32) peripherals are goin= g > to be unable to DMA from that hot plugged memory, but they could if we > had a swiotlb. That's why platforms that hotplug memory should indicate the maximum=20 possible PFN via some mechanism during boot. On x86-64 (and IIRC also=20 arm64 now), this is done via the ACPI SRAT. And that's where "max_possible_pfn" and "max_pfn" differ. See=20 drivers/acpi/numa/srat.c:acpi_numa_memory_affinity_init(): max_possible_pfn =3D max(max_possible_pfn, PFN_UP(end - 1));$ Using max_possible_pfn, the OS can properly setup the swiotlb, even=20 thought it wouldn't currently be required when just looking at max_pfn. I documented that for virtio-mem in https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html "swiotlb and DMA memory". >=20 > - now let's go even further but this is very contrived. Assume that the > firmware has somewhat created a reserved memory region with a 'no-map' > attribute thus indicating it does not want a struct page to be created > for a specific PFN range, is it valid to "blindly" raise max_pfn if tha= t > region were to be at the end of the just hot-plugged memory? no-map means that no direct mapping is to be created, right? We would=20 still have a memmap IIRC, and the pages are PG_reserved. Again, I think this is very similar to just having no-map regions like=20 random memory holes within the existing memory layout. What Chris proposes here is very similar to=20 arch/x86/mm/init_64.c:update_end_of_memory_vars() called during=20 arch_add_memory()->add_pages() on x86-64. --=20 Thanks, David / dhildenb