From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C68EEE49A3 for ; Wed, 23 Aug 2023 13:16:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B9E228005F; Wed, 23 Aug 2023 09:16:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06AE128005D; Wed, 23 Aug 2023 09:16:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E750828005F; Wed, 23 Aug 2023 09:16:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D827128005D for ; Wed, 23 Aug 2023 09:16:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B5ADB160120 for ; Wed, 23 Aug 2023 13:16:44 +0000 (UTC) X-FDA: 81155419128.12.4DF2114 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 037A080003 for ; Wed, 23 Aug 2023 13:16:42 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SLpmhuiLqoeOoKiHs0XPEs55KygN5SxAbZ2C12O4yl0=; b=7DSQgNE1AoOoI8Mfb1FHri17C7xMaHkqp1rsKw8+gMbW93fCNDs3PmgN313GP6WnO2s7Vj 9hXKhg3BE07XwFKceU70Wp9LhmM7QP+XQFNduXdI96CKSzVJcElybfNEaIKjicFLJl4huu GFLNYzF8sgDvtIN/n2xZ/85/TvKWpVo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796603; a=rsa-sha256; cv=none; b=ljkS6xXLzrIXkngmm4Pt4hzubLNiV7K49EKqZ1UoXJ9DZEg1dSH1UTKNHxCM4sxjWi9mCv AshIuVYWA//KDH5Hk6hZJIJs+wh+7J/Wt4JqaJA1I6aEp1izlhmtw98qbLBYpmLN92HpSa QJ/dJpzo+NmtaCjhVfUQcUuRafg2lwc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EB6C41692; Wed, 23 Aug 2023 06:17:22 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2EA123F740; Wed, 23 Aug 2023 06:16:36 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 24/37] mm: page_alloc: Teach alloc_contig_range() about MIGRATE_METADATA Date: Wed, 23 Aug 2023 14:13:37 +0100 Message-Id: <20230823131350.114942-25-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 037A080003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 4xr1e9ze3xcqyrzmwqjjemnjmoexu4ft X-HE-Tag: 1692796602-250729 X-HE-Meta: U2FsdGVkX19DQroeERV3AHsjzKzBo5XLyongmtbFb4N1NLRwdfa0AgZkLLrU9is0QkgP5sGB73ubEuncU3DeElCURSIVCTcImnpvaqMVOd1J8dVWjlIvxH0NoQxTs+3ltSaz3R6LhG2zK0nKR7pHGCrjzV43PFyQIuVzjA5Rfmgd9ydh4dJXMXfx4Vk2UuJpfJx9+ottMSsfOnzwkPn3d0GYAyDkt0NR+B4deX/vlQjyk2XOanFX5BR1shp+GD+9SM0HiKoOBFuB5Nwmaj7YU84Azue+aeJWblMQwRM3ikMGxtMF3TSkAwjEu76FnQBJLCur+WKYZJaRdwQTmChopENT3HnfesUOb2cqIuTYUjDTvdjA73okslWFASZdcmOl2XqHXqC1zjmrkvRSrUCSkV08BYxGK4fTdBiTsLfly6XTUYxWhR6/48/T1+zwy1XmhpIX+361aESSEDruET+czOjTV63VSVsmtL/K1Jme2Czf6xmHtbWZaI2M3blHhGWxkYmA60EK7eDaxWvSQAonSgYve+cd8UMJvsINQzd9DM6RxQRzOAulLsHOB5EFkelRDyJ3GQWAkwhb/96T2tebcHzwYxIMbOLX6mUQZmbJ9ZnVDPSjpIK8mhyOtqIOVaTC2QgaOPfimGSBu6o/iWvCJDJ5k0K2rUrFthUbQ413hvlFiRlvUZ1cwE4EKNqVTPf0PRi74faLU3T1kRnwXa+E8Z+hbFlJXM9XVrrZPEJE7/LW9KCFCpwhe5UXsQSZHSXIv/cmyloi4ZwAkz56AHhlzrmXf8LtrQ1auXmGZTYU0VQtSKMmYliN/HntO88eOKwdpLMNwUQsuaRBux7wFMXQZzn1nGns90M0oJeRWbPXW5n2O12yHEoXDZJWBbEUGUQdXkTHqZSVkjqJy4k9rqWBM+P42iPyTJmS1cWwiLAE5GksIh3dNyEgh7gdPu3IqD5ZCq5u8ZAGv7yQePvMTEM mavI9/+L n2BsA0jE/6uBu5NNlJEb9qijfOB0wMhwitIGaG3Ik5zrqmw1PXhYzor50bjq05DZB5irLlzVD8S65ZIMxPZayEXGMdTX7vLH6nONkftNg+TBiVgZq33B+x0CVZCLPjCAQmd86qZM6hj9QWcjHzttR5ymt/qZS3rL0KPFHqSikTvCOVdv06wf+UL/OGRpKmE7489m7f2of5QhOm2c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() allocates a contiguous range of physical memory. Metadata pages in use for data will have to be migrated and then taken from the free lists when they are repurposed to store tags, and that will be accomplished by calling alloc_contig_range(). The first step in alloc_contig_range() is to isolate the requested pages. If the pages are part of a larger huge page, then the hugepage must be split before the pages can be isolated. Add support for metadata pages in isolate_single_pageblock(). __isolate_free_page() checks the WMARK_MIN watermark before deleting the page from the free list. alloc_contig_range(), when called to allocate MIGRATE_METADATA pages, ends up calling this function from isolate_freepages_range() -> isolate_freepages_block(). As such, take into account the number of free metadata pages when checking the watermark to avoid false negatives. Signed-off-by: Alexandru Elisei --- mm/compaction.c | 4 ++-- mm/page_alloc.c | 9 +++++---- mm/page_isolation.c | 19 +++++++++++++------ 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index af2ee3085623..314793ec8bdb 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -46,7 +46,7 @@ static inline void count_compact_events(enum vm_event_item item, long delta) #define count_compact_events(item, delta) do { } while (0) #endif -#if defined CONFIG_COMPACTION || defined CONFIG_CMA +#if defined CONFIG_COMPACTION || defined CONFIG_CMA || defined CONFIG_MEMORY_METADATA #define CREATE_TRACE_POINTS #include @@ -1306,7 +1306,7 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn, return ret; } -#endif /* CONFIG_COMPACTION || CONFIG_CMA */ +#endif /* CONFIG_COMPACTION || CONFIG_CMA || CONFIG_MEMORY_METADATA */ #ifdef CONFIG_COMPACTION static bool suitable_migration_source(struct compact_control *cc, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 911d3c362848..1adaefa22208 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2624,7 +2624,8 @@ int __isolate_free_page(struct page *page, unsigned int order) * exists. */ watermark = zone->_watermark[WMARK_MIN] + (1UL << order); - if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) + if (!zone_watermark_ok(zone, 0, watermark, 0, + ALLOC_CMA | ALLOC_FROM_METADATA)) return 0; __mod_zone_freepage_state(zone, -(1UL << order), mt); @@ -6246,9 +6247,9 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either - * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks - * in range must have the same migratetype and it must - * be either of the two. + * #MIGRATE_MOVABLE, #MIGRATE_CMA or #MIGRATE_METADATA). + * All pageblocks in range must have the same migratetype + * and it must be either of the three. * @gfp_mask: GFP mask to use during compaction * * The PFN range does not have to be pageblock aligned. The PFN range must diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 6599cc965e21..bb2a72ce201b 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -52,6 +52,13 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e return page; } + if (is_migrate_metadata_page(page)) { + if (is_migrate_metadata(migratetype)) + return NULL; + else + return page; + } + for (pfn = start_pfn; pfn < end_pfn; pfn++) { page = pfn_to_page(pfn); @@ -396,7 +403,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = head_pfn + nr_pages; continue; } -#if defined CONFIG_COMPACTION || defined CONFIG_CMA +#if defined CONFIG_COMPACTION || defined CONFIG_CMA || defined CONFIG_MEMORY_METADATA /* * hugetlb, lru compound (THP), and movable compound pages * can be migrated. Otherwise, fail the isolation. @@ -466,7 +473,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = outer_pfn; continue; } else -#endif +#endif /* CONFIG_COMPACTION || CONFIG_CMA || CONFIG_MEMORY_METADATA */ goto failed; } @@ -495,10 +502,10 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, * @gfp_flags: GFP flags used for migrating pages that sit across the * range boundaries. * - * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in - * the range will never be allocated. Any free pages and pages freed in the - * future will not be allocated again. If specified range includes migrate types - * other than MOVABLE or CMA, this will fail with -EBUSY. For isolating all + * Making page-allocation-type to be MIGRATE_ISOLATE means free pages in the + * range will never be allocated. Any free pages and pages freed in the future + * will not be allocated again. If specified range includes migrate types other + * than MOVABLE, CMA or METADATA, this will fail with -EBUSY. For isolating all * pages in the range finally, the caller have to free all pages in the range. * test_page_isolated() can be used for test it. * -- 2.41.0