From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C824C369CB for ; Wed, 23 Apr 2025 08:18:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6002A6B00B0; Wed, 23 Apr 2025 04:18:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 586896B00A9; Wed, 23 Apr 2025 04:18:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38C726B00AF; Wed, 23 Apr 2025 04:18:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1089E6B00A9 for ; Wed, 23 Apr 2025 04:18:50 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 00481817FF for ; Wed, 23 Apr 2025 08:18:50 +0000 (UTC) X-FDA: 83364607662.22.B4AEA22 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf13.hostedemail.com (Postfix) with ESMTP id EBC782000D for ; Wed, 23 Apr 2025 08:18:48 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=e16xgsqx; spf=none (imf13.hostedemail.com: domain of BATV+ca0b9ff444d258c14c24+7913+infradead.org+dwmw2@desiato.srs.infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=BATV+ca0b9ff444d258c14c24+7913+infradead.org+dwmw2@desiato.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745396329; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/QxP+QorPaa2Tzcn88O9030NShTMbuO4EdTafyQOEYk=; b=z/vCzA5nre+IoiZJajpPskzNo8HTgfmRSbTjG/Yzk0X6hR098TruW2UXhfCHQTP2g6X1kI OgNBxb81TYiZOhHXLEVKJUUAT/MS7QMCaeww8JTJjPi0XLxmd+876ebGiERbR/f58zgEmo LhDQUCPrzvJMnZMs3vbUytHfYwWB/F0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=e16xgsqx; spf=none (imf13.hostedemail.com: domain of BATV+ca0b9ff444d258c14c24+7913+infradead.org+dwmw2@desiato.srs.infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=BATV+ca0b9ff444d258c14c24+7913+infradead.org+dwmw2@desiato.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745396329; a=rsa-sha256; cv=none; b=AsPT8IwaeoAmaf5jl93jAefpo7YFWgb4J4hC9Fih8n91A76agb1bvbiKAQ8yCt6wQP+qQr bAdpTmdMPgRcLcuIKwlUIDAzBDMbyIt2xKE1CvJJe8pU2rrNaqvpWL1n6OBzVl931eeYK/ EZRpaz1k642jqLXF+JLhLrTB+UEx89Y= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=/QxP+QorPaa2Tzcn88O9030NShTMbuO4EdTafyQOEYk=; b=e16xgsqxWp/TFTt4kTtzrCKnsJ lhB5OpLgmwmq6A6S+tw2fer+PblQ1YUkcu0i5egxL0+8r3b5PmrA/a5EXzxWJgUOHKYqur9a7XRRl bHXtVlkMg+p6YW+K3Udfo3Cihy8v8s6jmwqIY870hNx2vGf57l3Dxh1hFRv/47QP6zA4aafI3jG7p SVmvF4AVUYDwsKAlBCNEyNko0oapQo7CGPMWew9i1X4mKmIwZcZx40bWktuo+oCSLNTMwtBpjKioj h5SD6M33U8PryDacRer2acWhLp42i+MpHPYn9l2cmOsTCKIFKERDqRTIqDyy8BcnR5S5r89RbNS0w RS/a/g+A==; Received: from [2001:8b0:10b:1::ebe] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1u7VJR-0000000BOih-1XXI; Wed, 23 Apr 2025 08:18:34 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.98.1 #2 (Red Hat Linux)) id 1u7VJO-00000002YNt-1NB4; Wed, 23 Apr 2025 09:18:30 +0100 From: David Woodhouse To: Mike Rapoport Cc: Andrew Morton , "Sauerwein, David" , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Marc Zyngier , Mark Rutland , Mike Rapoport , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ruihan Li Subject: [PATCH v3 3/7] mm: Implement for_each_valid_pfn() for CONFIG_SPARSEMEM Date: Wed, 23 Apr 2025 08:52:45 +0100 Message-ID: <20250423081828.608422-4-dwmw2@infradead.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250423081828.608422-1-dwmw2@infradead.org> References: <20250423081828.608422-1-dwmw2@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: EBC782000D X-Stat-Signature: mua3qwxryr3ou1hrmapbkka7qden4ghf X-HE-Tag: 1745396328-663760 X-HE-Meta: U2FsdGVkX1/nlWHvg9V5lLo2WpzLXjFJsi0z5ePmIc4Dt3CNpT29nxrck/KKQfh/DS3SuHA0N9H0KJGa4Qyt8o50uYgKAjxFWRrN/vW6Mc+dz/Ffk/+iThoIAnVBGESSI0SiAAlInfIpH06t8Nf7L+Vss/y22vI85X/Q8Qv0WiaL2lAIZoPVrVvbypvEnKxQxaNXNlvWVuXAkb9aWWu3WpDWQyGevHPSfba4VMy+cQsWhllAja+b32cQ8CeRWfFfDXrTGtNxRUWFokYZx7w54fv9Fk7l+3MIr+ri7zC3Am891gnPBcfojDkS07p1llC8gbAeg40oBYCdiv3WuXSoDxCE3za12Ix0mBOlCE6SFOMkIWN9VkZKDzJxrWwOrZDz4iewx+m6LlNu1aDL+L0SzSzCogFB4+IxPyS+rBUgzl7/KDmxkw9N601YR9f8IsvRabx+85hRdGN1tD1w8a88J0cIA72gQsGPO+X+rbLvRQZv4k1lev4DAk+j+Xq9cqYVRNb/p+MZeTzkb9QQJov+segDVyRQ166NSFDQTWOtEaje+mJYW2VBtoxWFz/XF8leDJXbGre8YCSjmGkdUeU3h5T+ar8HxyapVMVTuydY1X3s8FdCX0/vzUIiL3T32CvxmmLE9m1TYbkfMJz2yI78GQXhwE00hMCBD5pWeMK+vW3svkzSWKhoKtPY3z7iI3GFuagfeFtcXK5cqy8rJKlYYjFq8hsttkcwQrTj8PDsNJisCdOvNBW02aD72/Jr9jo9V5/wNntXcEt6oij/9EhGlwvp8q3ZgObaCJyKGT5u/RN7YwKH+90z/Zlv62jR6u+zRuKDQscVYvfOaBkgQW+5ih+oy5wVvJFgeUHYi4820cD7o4/cjM47tz/YmPtcafl7a+XNhrFJtRWtPa16P4kOgNnf+eS3G0pxioj+OckGVeruki6vL2L0wbC7+MDSB+EV50fHUNpnSRf357z1Ipo zexzo13U tUsbEAKPGC5RlkKvhXiIdpCDtgtX3d2D1deer6d8vNeo045+PrCk9emQQ9Dq0N8f0L9hXzW5YLDHxon00DcTSXzId4tN1IfBQRi6QUDe7nSLiJorczyMQ2xzlU8mJBwK2yoo2+Jw+zT1pVsI4NCFWZVMOwHcxQ9lUV4FCV0lgQURjwhpjeC3vNHvTuPTxJj/5ofIMkOUpGJ08GnpeTe53upgpHG+fCSV9I2wgL5L/Hy5bKFApxM7afJbXnb8h34rirNmNtPYOwlBZp4SibJj/L+LXmZA/vffR0nuVtWqtXel7wNkVc1eVvAdDD2QkJztvbWE1itkh0VXkcLvTtPAri//NSF19h6fJZ8Jual6ARy/zTWx2w9DZaaKkc7ZZrq3eO282iO77YJS+zsJrbiRXVZIFphvZnhmf8ulpyTqb1U15JkoVUwZx5CbFV43uJtdGADeQvq50UW1ZsYuXiupJZGBjuvvpxMXMH5z2a+Yo3YKdbZJ/mLmWJxhePWFjPXct+fB5gGF3PFHo0nAfuTchXt6fiRUpeHiUAZI/u1jfR1oqNokXGHIBbV2/BA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: David Woodhouse Implement for_each_valid_pfn() based on two helper functions. The first_valid_pfn() function largely mirrors pfn_valid(), calling into a pfn_section_first_valid() helper which is trivial for the !VMEMMAP case, and in the VMEMMAP case will skip to the next subsection as needed. Since next_valid_pfn() knows that its argument *is* a valid PFN, it doesn't need to do any checking at all while iterating over the low bits within a (sub)section mask; the whole (sub)section is either present or not. Note that the VMEMMAP version of pfn_section_first_valid() may return a value *higher* than end_pfn when skipping to the next subsection, and first_valid_pfn() happily returns that higher value. This is fine. Signed-off-by: David Woodhouse Previous-revision-reviewed-by: Mike Rapoport (Microsoft) --- include/asm-generic/memory_model.h | 26 ++++++++-- include/linux/mmzone.h | 78 ++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 5 deletions(-) diff --git a/include/asm-generic/memory_model.h b/include/asm-generic/memory_model.h index 74d0077cc5fa..044536da3390 100644 --- a/include/asm-generic/memory_model.h +++ b/include/asm-generic/memory_model.h @@ -31,12 +31,28 @@ static inline int pfn_valid(unsigned long pfn) } #define pfn_valid pfn_valid +static inline bool first_valid_pfn(unsigned long *pfn) +{ + /* avoid include hell */ + extern unsigned long max_mapnr; + unsigned long pfn_offset = ARCH_PFN_OFFSET; + + if (*pfn < pfn_offset) { + *pfn = pfn_offset; + return true; + } + + if ((*pfn - pfn_offset) < max_mapnr) + return true; + + return false; +} + #ifndef for_each_valid_pfn -#define for_each_valid_pfn(pfn, start_pfn, end_pfn) \ - for ((pfn) = max_t(unsigned long, (start_pfn), ARCH_PFN_OFFSET); \ - (pfn) < min_t(unsigned long, (end_pfn), \ - ARCH_PFN_OFFSET + max_mapnr); \ - (pfn)++) +#define for_each_valid_pfn(pfn, start_pfn, end_pfn) \ + for (pfn = max_t(unsigned long, start_pfn, ARCH_PFN_OFFSET); \ + pfn < min_t(unsigned long, end_pfn, ARCH_PFN_OFFSET + max_mapnr); \ + pfn++) #endif /* for_each_valid_pfn */ #endif /* valid_pfn */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 230a29c2ed1a..dab1d31477d7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2075,11 +2075,37 @@ static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) return usage ? test_bit(idx, usage->subsection_map) : 0; } + +static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn) +{ + struct mem_section_usage *usage = READ_ONCE(ms->usage); + int idx = subsection_map_index(*pfn); + unsigned long bit; + + if (!usage) + return false; + + if (test_bit(idx, usage->subsection_map)) + return true; + + /* Find the next subsection that exists */ + bit = find_next_bit(usage->subsection_map, SUBSECTIONS_PER_SECTION, idx); + if (bit == SUBSECTIONS_PER_SECTION) + return false; + + *pfn = (*pfn & PAGE_SECTION_MASK) + (bit * PAGES_PER_SUBSECTION); + return true; +} #else static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) { return 1; } + +static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn) +{ + return true; +} #endif void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, @@ -2128,6 +2154,58 @@ static inline int pfn_valid(unsigned long pfn) return ret; } + +/* Returns end_pfn or higher if no valid PFN remaining in range */ +static inline unsigned long first_valid_pfn(unsigned long pfn, unsigned long end_pfn) +{ + unsigned long nr = pfn_to_section_nr(pfn); + + rcu_read_lock_sched(); + + while (nr <= __highest_present_section_nr && pfn < end_pfn) { + struct mem_section *ms = __pfn_to_section(pfn); + + if (valid_section(ms) && + (early_section(ms) || pfn_section_first_valid(ms, &pfn))) { + rcu_read_unlock_sched(); + return pfn; + } + + /* Nothing left in this section? Skip to next section */ + nr++; + pfn = section_nr_to_pfn(nr); + } + + rcu_read_unlock_sched(); + return end_pfn; +} + +static inline unsigned long next_valid_pfn(unsigned long pfn, unsigned long end_pfn) +{ + pfn++; + + if (pfn >= end_pfn) + return end_pfn; + + /* + * Either every PFN within the section (or subsection for VMEMMAP) is + * valid, or none of them are. So there's no point repeating the check + * for every PFN; only call first_valid_pfn() the first time, and when + * crossing a (sub)section boundary (i.e. !(pfn & ~PFN_VALID_MASK)). + */ + if (pfn & (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP) ? + PAGE_SUBSECTION_MASK : PAGE_SECTION_MASK)) + return pfn; + + return first_valid_pfn(pfn, end_pfn); +} + + +#define for_each_valid_pfn(_pfn, _start_pfn, _end_pfn) \ + for ((_pfn) = first_valid_pfn((_start_pfn), (_end_pfn)); \ + (_pfn) < (_end_pfn); \ + (_pfn) = next_valid_pfn((_pfn), (_end_pfn))) + #endif static inline int pfn_in_present_section(unsigned long pfn) -- 2.49.0