From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31688C71136 for ; Wed, 18 Jun 2025 00:25:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A22346B0088; Tue, 17 Jun 2025 20:25:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D2F66B0089; Tue, 17 Jun 2025 20:25:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 910276B008A; Tue, 17 Jun 2025 20:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 84CA86B0088 for ; Tue, 17 Jun 2025 20:25:51 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2F42E5F829 for ; Wed, 18 Jun 2025 00:25:51 +0000 (UTC) X-FDA: 83566628502.15.8B2FE11 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf17.hostedemail.com (Postfix) with ESMTP id 5EE3D40010 for ; Wed, 18 Jun 2025 00:25:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=BORxnhy4; spf=pass (imf17.hostedemail.com: domain of akpm@linux-foundation.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750206349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7sEkj7aGDwQDVVg23N/FFxM/bs1udvTPnnnaar23mSc=; b=Z9i55ysSY/sGoX3JUdo3GmFMLaJd+W+YBwxRbp7PmGN2tnSXB+f216VejFEvilXqw2ju12 0hG4SjfipMbLooqTpsJ2gIU+g2TMILXvlnDB7kO2mtW/PoUnFlDKRcUaUp6TYgVjV0sVQe hscOCBUN4ZqfnWkeMed7Y0y1pqHzQR0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=BORxnhy4; spf=pass (imf17.hostedemail.com: domain of akpm@linux-foundation.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750206349; a=rsa-sha256; cv=none; b=GUIULjrBQFeGU6DnAyX7+G6qGHM/czlapNzCz0d8SXHAswDsLqfE1UauBoiJMsRHhmOzeq X5lQ8oyh66DtXo8r1KQ2N2XOsCTSYK+xdgIADkr80DN+2IiI7fJtSy2buy5QeufxO0TVyo 6XSQlSNCTmtx5PTiHiauTy0XzX9tPdU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 9953EA4361B; Wed, 18 Jun 2025 00:25:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C8E5C4CEE3; Wed, 18 Jun 2025 00:25:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1750206348; bh=grNSdOh7xltwFNxLMxeS/rpygzEfyl+lh904ZKR+HhA=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=BORxnhy4BuuGukVUZdMqQ65TPjm3pQsgu5w7kMNTfGP2+P951TGchMDUjzcyuxtWZ H3gALD8kHa9AFEL69dJ1QJVmLPvCYEJMr8/9LIf7Ko4b7PBR3NTcUQ9NCxnSX6uv0p pX33s74HN622kx50CUGLJDgrqIXHzQAWzBbczZak= Date: Tue, 17 Jun 2025 17:25:47 -0700 From: Andrew Morton To: Zhiguo Jiang Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, opensource.kernel@vivo.com Subject: Re: [PATCH] mm: rt-threads retry mempool allocation without delay Message-Id: <20250617172547.25af99b0f195379f6d6df9f8@linux-foundation.org> In-Reply-To: <20250617091044.1062-1-justinjiang@vivo.com> References: <20250617091044.1062-1-justinjiang@vivo.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam11 X-Rspam-User: X-Rspamd-Queue-Id: 5EE3D40010 X-Stat-Signature: begkgx3794wcauueccdeb3abexwxr38y X-HE-Tag: 1750206349-263286 X-HE-Meta: U2FsdGVkX19j0zWTUgsgKHEFXpBb9261ly57L0LmSwkYVI+rf8AJ4211XRs0tuJvBzw3CkzCJUe0jbcFN0RYexCYtMudSMoCbfbyBkW7EQT/YP7IlJjZOhVLYM1SsKabWDdxhtJLDPxDYsC+fHCiZW/XKbRvbpxt7+L9XFQJHI/spb73Ru3glSYBaAzLMC7b6Mux0fYR+nqquTHGYu0qdoG2p8L2bULAVLzmhc12Qb1IXk/D7VZHMXFvbaHjy7QkgSjDYO12scZLRn8UgTl7JKcKTdXJO4jC0Gg7HzWGJbFDYtOot+Qzl1li3Qgg6UfJg0+fcz6jYMqzN/R7KsZ5A7lWI8iFu9hUF+oEIJ3ysCsyX3kUNwKS77XKLW3I29wT21R856jImAoCCNnMBRH7Bo90+PsEFAdwdklm4xcWw4G6j8H4h30BY6yWJUXJTz1+Yzql0TR8sEwng22WJ3ZtjbqGft17tEFjZ7xhtbYbGikFq6FyEbm8sKIeBAYSHc+qqqL23i4Y3YVzstb0+kDYURS1c80GHvou170+uvCHoWWi7CIxxqcAa9MDEQq61QOmoY45xpJmfmnKeBBV31GInIHpRh+4mYlehrG12ocfEpbS1ic4RMRaWbvbv6tg0OVV3JzDK7ZjybUqXP4jszQuCfKDKR7a3u803Yeb765Dedsy4RMZTZZv5mhULvU5sXP/cUg2QU/R/I0JP7QFDwmtYYeaZrTxO8ECt/2YinckmraggZGHLH8vHEFu1/q8gdbxatolzxuIL5xk6TRDZEZelJ1yU4AG3RSBmxgBjJgGZWg779xx1+lHTTmOcFrnduT0U3t8Qsz8Jh3jiJ9DRApTO8KN+Tz/h2CdaZC22IlFAjCJVL/P6NlLyUfm+tiAFXxTLcs6zl5LgcWD51LTWndKXOL0EWnYRPTllYdDqtNutCQ+5s8qzZr1YvhJGfI8r1NfRSmVhwCvXjpUgisGVd7 hIymJfM8 uZCpO4TCewKSGSOSmy7fKcaPf2X5zXSmtQpYWd/bXy+KwF8jPyz8GKIhHF1/edgxzKkg7lfzNZuDFP8uirh8PK9U4GPxh11feISwN9ifS4HTKrSmI+vSyvVuquAATLWN6+M96K6SKa26HmbJbYbGZMVGMGZ/YR4gX2NuHFMkgD9RzmKdChaMSqIc1qtn61M+h/fgxvWZIODPZvWZ2IuO9WJ1Mo/IXDVqujxYEQlCJS0hwbcNVmhkfe5ey6VowFyeVgp69zFV0AaaUdokGnqtb9jXjS+Xs1gZt/KU2yAWGuqYGoAGl3qzKyySJQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 17 Jun 2025 17:10:44 +0800 Zhiguo Jiang wrote: > The real-time(rt) threads are delayed for 5 seconds in mempool_alloc, > which will seriously affect the timeliness of front-end applications > and the user experience lag issues. Oh God, do we really do that? Yes we do! I'm surprised this wasn't reported some time over the intervening 13 years. Yes, a hard-coded 5 second delay might be a slight problem in a realtime kernel. > The real-time(rt) threads should retry mempool allocation without > delay and in order to obtain the required memory resources as soon as > possible. Well, does this actually work in your testing? I guess it can improve the situation, some of the time. If it's a uniprocessor non-preemptible then perhaps interrupt-time writeback completion might save us, otherwise it's time to hit the power button. > The following example shows that the real-time(rt) QoSCoreThread > prio=98 blocks 5 seconds in mempool_alloc, seriously affecting the > user experience. > > Running process: system_server (pid 2245) > Running thread: QoSCoreThread 2529 > State: Uninterruptible Sleep - Block I/O > Start: 12,859.616 ms > Systrace Time: 100,063.057104 > Duration: 5,152.591 ms > On CPU: > Running instead: kswapd0 > Args: {kernel callsite when blocked:: "mempool_alloc+0x130/0x1e8"} > > QoSCoreThread-2529 ( 2245) [000] d..2. 100063.057104: sched_switch: > prev_comm=QoSCoreThread prev_pid=2529 prev_prio=000255001000098 > prev_state=D ==> next_comm=kswapd0 next_pid=107 > next_prio=000063310000120 > [GT]ColdPool#14-23937 ( 23854) [000] dNs2. 100068.209675: sched_waking: > comm=QoSCoreThread pid=2529 prio=98 target_cpu=000 > [GT]ColdPool#14-23937 ( 23854) [000] dNs2. 100068.209676: > sched_blocked_reason: pid=2529 iowait=1 caller=mempool_alloc+0x130/0x1e8 > [GT]ColdPool#14-23937 ( 23854) [000] dNs3. 100068.209695: sched_wakeup: > comm=QoSCoreThread pid=2529 prio=98 target_cpu=000 > [GT]ColdPool#14-23937 ( 23854) [000] d..2. 100068.209732: sched_switch: > prev_comm=[GT]ColdPool#14 prev_pid=23937 prev_prio=000003010342130 > prev_state=R ==> next_comm=QoSCoreThread next_pid=2529 > next_prio=000255131000098 Do you have a call trace for these stalls? I'm interested to see who is calling mempool_alloc() here. Perhaps a suitable solution is to teach the caller(s) to stop passing __GFP_DIRECT_RECLAIM and to handle the NULL return. > --- a/mm/mempool.c > +++ b/mm/mempool.c > @@ -18,6 +18,7 @@ > #include > #include > #include > +#include > #include "slab.h" > > #ifdef CONFIG_SLUB_DEBUG_ON > @@ -386,7 +387,7 @@ void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) > void *element; > unsigned long flags; > wait_queue_entry_t wait; > - gfp_t gfp_temp; > + gfp_t gfp_temp, gfp_src = gfp_mask; > > VM_WARN_ON_ONCE(gfp_mask & __GFP_ZERO); > might_alloc(gfp_mask); > @@ -433,6 +434,16 @@ void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) > return NULL; > } > > + /* > + * We will try to direct reclaim cyclically, if the rt-thread "synchronously" > + * is without __GFP_NORETRY. > + */ > + if (!(gfp_src & __GFP_NORETRY) && current->prio < MAX_RT_PRIO) { > + spin_unlock_irqrestore(&pool->lock, flags); > + gfp_temp = gfp_src; > + goto repeat_alloc; > + } > + > /* Let's wait for someone else to return an element to @pool */ > init_wait(&wait); > prepare_to_wait(&pool->wait, &wait, TASK_UNINTERRUPTIBLE);