From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25A0EE49B0 for ; Wed, 23 Aug 2023 13:17:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E62528006B; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BDC128005D; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6858628006B; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 55FEE28005D for ; Wed, 23 Aug 2023 09:17:13 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 30340B0D00 for ; Wed, 23 Aug 2023 13:17:13 +0000 (UTC) X-FDA: 81155420346.11.C206B02 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 6E68F10001A for ; Wed, 23 Aug 2023 13:17:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692796631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tgGYsPuAcTk/paK1LuxSZMiq6gyvIL+TuHYCaCWFD+c=; b=AEGVqOOPQLVHHh0B5sVcsH3qI2N6MymrjWhCyW+SGlrU4daYXo6ymWONNlFAyRPtD4vnts J51GkBnVAFj+1wjZ8CY6fIc3NzeTu543Mkk/TI3lMkHcSrrq+21PtHLStRXMOBsnX+Lx/2 dSFRqE6KGsJNP8ro32pGFc97vTcB27Y= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692796631; a=rsa-sha256; cv=none; b=NV8bZ8xXIa7U3aZWig3ravoCp/15pipXXAZaAA+wh7P8E0AO+tZ3JV0W8B0Q6L9Ubq2LyS 6+dgOsi5l2FBj+xtusBIQlWTjc59M/6Fw7DmollStrPfuxI6VKoD/02Fz278gQ4Q0Dysk7 ybJlbFpMPoqMIHuufA/c1fJukRrmYmA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5D19016F3; Wed, 23 Aug 2023 06:17:51 -0700 (PDT) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C9D2F3F740; Wed, 23 Aug 2023 06:17:04 -0700 (PDT) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC 28/37] mm: sched: Introduce PF_MEMALLOC_ISOLATE Date: Wed, 23 Aug 2023 14:13:41 +0100 Message-Id: <20230823131350.114942-29-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823131350.114942-1-alexandru.elisei@arm.com> References: <20230823131350.114942-1-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 6E68F10001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3tfi1hznh1ccwqxbhc8q6a9s3wrinyre X-HE-Tag: 1692796631-764338 X-HE-Meta: U2FsdGVkX18ZRE0e3QobrHY2JY5mYmpCIuWnm2wB0N4vfbscbf4pogw9oYz4rvf3w0AnGHzgoHeBqsE0JUU5dU7lPpPz+CkJodCUBqWeP6j8Y5A9hyExvrdxpMbw6jGs0x0qR+wU6eV4ov0s5leAnSR+1UgMxQUrujdk6lDRgMD2vwtVaWUyVo5xTIDk1EwqyNeHNr+s0s3+7YhOaL8yJqB94mFnWVLOq2x5vWElH/sgRps+GuIAV5iehBmx4YAd5fAU4RW58xdo6m67dZ+G83UV3NQDJ4poSWsO9broiQS+Ok3IvZwc/GpxbOL/jO9jm2JnBcuhqklTvJFcd7a/HJAZ/FPqlhAiiQlm69a8BzAhx3VSaz52U6ktGGno5xk7rZZjPd1Dduqx0f+s5uCDAyByej80fgrEErG7+KZvmYIiabRH1vnFdS+kmVQxpsEsRJveHqGjq969sy4eABG7ansg1GOTszyA8qCP4CruHtUHxf47AGTQ4Z+bOWzqN+lJH+xArcaoCE+7tcQKXByDJ0IYgJ+WMx7rlK8V58hcHLp5TfmT3Ny1e6iL+Hw4HMlXK7/f/PmTIWQaa/2F9ShHkRNQsyCqwOcUuJW01Xj+qM7NtGKiTdqFhUfwwM3mlVbjiyFMgC+iYx2/sbNoUZDN3Ayz6DFfLg9ePr6hfy01XSK0FGXzPQOUazLuUeFr5Ixdmzc68xzNLjWS8bH3sckeqHPuIUx6bQBj8UGSnE5n9Y7C040eKwUhsM1lihxRdTsOP3q0UOX8ESZWsJdU1HY2z0f6iHnPHPxdWXnq4PUHtzx3sRTIEqFYfT1FqVjJQmrxkrgyRFd4sP+majCPT7xg3fDNIo3VrJlAeQdOWgj1w+dCtVcva6n0wMkRnKxOEKU9O/rpVKaXmhCdJSi4U4pQr5vWNbe6grWluXwm1a2lqSEaqUv+wVBZ2YjSlUwZsPtehN/UJjWG43Ju8nYlR2G xOqBJSIg FAFaj8VYEU7RbSS0jKY5/jChh67WnbThDm23OHygNIXmMkUzFZBaaysSrAscP1jynfYX9WecUpi0vm/ZjDTxCfXuWZglbQtRkarvNfHFPbikuJ88YuAwASNaXHhoSkgPePE/Csd8vnnvV73kyZORS+X1+KujreYr0iwrcgts1ExZ1FuTdvBYTVbEhlbwQgZ1lx2HSFgMk+5WO3Bk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On arm64, when reserving tag storage for an allocated page, if the tag storage is in use, the tag storage must be migrated before it can be reserved. As part of the migration process, the tag storage block is first isolated. Compaction also isolates the source pages before migrating them. If the target for compaction requires metadata pages to be reserved, those metadata pages might also need to be isolated, which, in rare circumstances, can lead to the threshold in too_many_isolated() being reached, and isolate_migratepages_pageblock() will get stuck in an infinite loop. Add the flag PF_MEMALLOC_ISOLATE for the current thread, which makes too_many_isolated() ignore the threshold to make forward progress in isolate_migratepages_pageblock(). For consistency, the similarly named function too_many_isolated() called during reclaim has received the same treatment. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 5 ++++- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 13 +++++++++++++ mm/compaction.c | 3 +++ mm/vmscan.c | 3 +++ 5 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 1ab875be5f9b..ba316ffb9aef 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -505,9 +505,9 @@ static int order_to_num_blocks(int order) int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) { unsigned long start_block, end_block; + unsigned long flags, cflags; struct tag_region *region; unsigned long block; - unsigned long flags; int i, tries; int ret = 0; @@ -539,6 +539,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) } xa_unlock_irqrestore(&tag_blocks_reserved, flags); + cflags = memalloc_isolate_save(); for (block = start_block; block < end_block; block += region->block_size) { /* Refcount incremented above. */ if (tag_storage_block_is_reserved(block)) @@ -566,6 +567,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) for (i = 0; i < (1 << order); i++) set_bit(PG_tag_storage_reserved, &(page + i)->flags); + memalloc_isolate_restore(cflags); mutex_unlock(&tag_blocks_lock); return 0; @@ -581,6 +583,7 @@ int reserve_metadata_storage(struct page *page, int order, gfp_t gfp) } xa_unlock_irqrestore(&tag_blocks_reserved, flags); + memalloc_isolate_restore(cflags); mutex_unlock(&tag_blocks_lock); count_vm_events(METADATA_RESERVE_FAIL, region->block_size); diff --git a/include/linux/sched.h b/include/linux/sched.h index 609bde814cb0..a2a930cab31a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1734,7 +1734,7 @@ extern struct pid *cad_pid; #define PF_USED_MATH 0x00002000 /* If unset the fpu must be initialized before use */ #define PF_USER_WORKER 0x00004000 /* Kernel thread cloned from userspace thread */ #define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */ -#define PF__HOLE__00010000 0x00010000 +#define PF_MEMALLOC_ISOLATE 0x00010000 /* Ignore isolation limits */ #define PF_KSWAPD 0x00020000 /* I am kswapd */ #define PF_MEMALLOC_NOFS 0x00040000 /* All allocation requests will inherit GFP_NOFS */ #define PF_MEMALLOC_NOIO 0x00080000 /* All allocation requests will inherit GFP_NOIO */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 8d89c8c4fac1..8db491208746 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -393,6 +393,19 @@ static inline void memalloc_pin_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; } +static inline unsigned int memalloc_isolate_save(void) +{ + unsigned int flags = current->flags & PF_MEMALLOC_ISOLATE; + + current->flags |= PF_MEMALLOC_ISOLATE; + return flags; +} + +static inline void memalloc_isolate_restore(unsigned int flags) +{ + current->flags = (current->flags & ~PF_MEMALLOC_ISOLATE) | flags; +} + #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); /** diff --git a/mm/compaction.c b/mm/compaction.c index 314793ec8bdb..fdb75316f0cc 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -778,6 +778,9 @@ static bool too_many_isolated(struct compact_control *cc) unsigned long active, inactive, isolated; + if (current->flags & PF_MEMALLOC_ISOLATE) + return false; + inactive = node_page_state(pgdat, NR_INACTIVE_FILE) + node_page_state(pgdat, NR_INACTIVE_ANON); active = node_page_state(pgdat, NR_ACTIVE_FILE) + diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..912ebb6003a0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2453,6 +2453,9 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, if (current_is_kswapd()) return 0; + if (current->flags & PF_MEMALLOC_ISOLATE) + return 0; + if (!writeback_throttling_sane(sc)) return 0; -- 2.41.0