From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05BE2C0219B for ; Tue, 11 Feb 2025 19:04:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CE846B007B; Tue, 11 Feb 2025 14:04:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47D6E6B0082; Tue, 11 Feb 2025 14:04:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36C406B0085; Tue, 11 Feb 2025 14:04:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 192D96B007B for ; Tue, 11 Feb 2025 14:04:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A3A931C805A for ; Tue, 11 Feb 2025 19:04:29 +0000 (UTC) X-FDA: 83108589858.28.F431B6C Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) by imf24.hostedemail.com (Postfix) with ESMTP id C8627180009 for ; Tue, 11 Feb 2025 19:04:27 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="qHYgq/Vx"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739300668; a=rsa-sha256; cv=none; b=Tuj1XUpSfNC/re2OJLEJNL/ZeZOi6tEw4FLAblJaBbZNzcuOyQVTfRnESSp07Aj/nU6E03 eldkd37jThc3+0sE2a7dQDEe84tXjWoKKrUrkkcM4FEjLz9bPGw8om/3qrIcq5Asic0WmV S+orADIDyICDxHbomoMhxXtNb2a+Fr4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="qHYgq/Vx"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739300668; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LZcMl8Wvgu43xkeb3Z4P/CBQfkD+2Pi53RCHqGlIMtk=; b=L0ixCS/19vX4RynYlKv4D/3DTKEMGO1O4yqjVlQRBtHJNRzCUeSZI/NmKg5y+P53v4Jgh/ BopHqiIVTWnFJ9kmk96+Ob5Cetw33In0umfSr2n0wanLLC9c/J3vp29mDHlgF3+2ple/Rm O+f8wqnc9Xd2Dx5G+dCq58M66aR5zVE= Date: Tue, 11 Feb 2025 11:04:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1739300665; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=LZcMl8Wvgu43xkeb3Z4P/CBQfkD+2Pi53RCHqGlIMtk=; b=qHYgq/Vxdcl2ebxEnHDTf03kI7zycftVvjJeLHsqoj6LJyIUejJohwbUAqUYDXZF6xbGLz 7yswAXqAjvTIczfpEmO974KhH2s7AAwl5HcK2pJKtzMjoZ4s9jfYStUUWW5ukS0ty+zwNQ psLNTRqNDmTy5M0kokpxOIdqSL9xWp8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Chen Ridong Cc: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH] memcg: avoid dead loop when setting memory.max Message-ID: References: <20250211081819.33307-1-chenridong@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250211081819.33307-1-chenridong@huaweicloud.com> X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C8627180009 X-Stat-Signature: tawt7tizbapntxs35w3x79bdhzf6w4bt X-Rspam-User: X-HE-Tag: 1739300667-897878 X-HE-Meta: U2FsdGVkX18bVp1AeUU2Zbr4jFyRLVdWd8TCjyNyr4WTqsbJwsS8uPAxPKeG/SKbudPacPYPIybhsE5xX47sC8cTTCdWVUYJoe+Twmq8yp5jNqMUlBzQRJxzIV05KdZuWnlA7sp9Bz+DGVLsyqa74lGiwWncW1CFLQdb1FmwtEB0fVTBbSRpDAYV6MJhwSx+s2T8ouXAtLvQ7Jp6XvseA22TTHqZuQuCQtLVVxjB5LrzDRIh1a49SrGrnouGSIJY275DHxPmFn//+vJlXNZHBR2Gks5wLNpZwUPBgZv0wu9x+MrAfQIS98OorrG/uNjn1DvumrQyN72gJ0hRAsFY5nkzLQx0y1m7NwPIG1C25wfx4oK/goeK7nR5XJsfzs6XKDTXmJZ6N6mE5XATndleEQNyOuQ7PLBEag/xBQhe8PvPtSTrwmUt7IbnKddU0+Hd77cr3ppJ5OKj3jTc0G8RFvTL8G9bKtWH5f5wuoP5sfh2uIqAW8K6CjjL14phSnicxrdHD3Gj7+V4IvZHcOxccSJWiTTpGKnF/qS6A31pacCU/G/fBkBuT98GTJO9CXN+xSTZqlT23EaOnJ4H7hrPQsBQ3WEEWl97CsUVMFQ2PLHrJR2nPslvXc44LDmZpk1W8mnLyMJOTTG5o4gvlWeyqEo+gpnBy5m52D1BNFLbYy9sFoEdPNlgoAHBRTMa803d5tzNqnZ2GNZ/5fzPUojaTFHX8nHllO3dclIKrAJlcCcEy7qIvDU91y88QOLthcCIy0oiHlDrPu2xOxAVy/eXnnzcY95Hm4O2AvB7sOoFSZuk0KpqzA4pdsgvhlzOdfNeZ37LLzG0OV06VMQ9MYZFYNdU0aXIwuWSACvzK/FIKveFiS8aYIA95xR0MDVVL9lBV7BonvFh47xz4AfiOxOhn/tgFdvQkkAum6bk9QnL5uGGdZ73qwMbWKliI+6hEVE0EdLABtmY0uRFaELHjtg 0cWN9WpO ScbcborJbrzOfCOOF+jw/ZsmqEw6hjNpuAz2DL7H1HTHcxUxKNj2dK99BLhd+risoF3hvUay+/BunhGv5qMZ1TjHE9FiZpbxd34YPrQSagpfbEXdTZOLwMO3LON6D5ThSHwQ/dOB+RoqS/IYGLCxkNwQhmMPXFOGb/wjaEuBRhD+aOffcYIu93QeMK4hbRSDkFCPu5a3WiwD7K+UbzZPCJRYK460Mj/tKufrSk6SRyaLRoNw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 11, 2025 at 08:18:19AM +0000, Chen Ridong wrote: > From: Chen Ridong > > A softlockup issue was found with stress test: > watchdog: BUG: soft lockup - CPU#27 stuck for 26s! [migration/27:181] > CPU: 27 UID: 0 PID: 181 Comm: migration/27 6.14.0-rc2-next-20250210 #1 > Stopper: multi_cpu_stop <- stop_machine_from_inactive_cpu > RIP: 0010:stop_machine_yield+0x2/0x10 > RSP: 0000:ff4a0dcecd19be48 EFLAGS: 00000246 > RAX: ffffffff89c0108f RBX: ff4a0dcec03afe44 RCX: 0000000000000000 > RDX: ff1cdaaf6eba5808 RSI: 0000000000000282 RDI: ff1cda80c1775a40 > RBP: 0000000000000001 R08: 00000011620096c6 R09: 7fffffffffffffff > R10: 0000000000000001 R11: 0000000000000100 R12: ff1cda80c1775a40 > R13: 0000000000000000 R14: 0000000000000001 R15: ff4a0dcec03afe20 > FS: 0000000000000000(0000) GS:ff1cdaaf6eb80000(0000) > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000000000 CR3: 00000025e2c2a001 CR4: 0000000000773ef0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > PKRU: 55555554 > Call Trace: > multi_cpu_stop+0x8f/0x100 > cpu_stopper_thread+0x90/0x140 > smpboot_thread_fn+0xad/0x150 > kthread+0xc2/0x100 > ret_from_fork+0x2d/0x50 > > The stress test involves CPU hotplug operations and memory control group > (memcg) operations. The scenario can be described as follows: > > echo xx > memory.max cache_ap_online oom_reaper > (CPU23) (CPU50) > xx < usage stop_machine_from_inactive_cpu > for(;;) // all active cpus > trigger OOM queue_stop_cpus_work > // waiting oom_reaper > multi_cpu_stop(migration/xx) > // sync all active cpus ack > // waiting cpu23 ack > // CPU50 loops in multi_cpu_stop > waiting cpu50 > > Detailed explanation: > 1. When the usage is larger than xx, an OOM may be triggered. If the > process does not handle with ths kill signal immediately, it will loop > in the memory_max_write. > 2. When cache_ap_online is triggered, the multi_cpu_stop is queued to the > active cpus. Within the multi_cpu_stop function, it attempts to > synchronize the CPU states. However, the CPU23 didn't acknowledge > because it is stuck in a loop within the for(;;). > 3. The oom_reaper process is blocked because CPU50 is in a loop, waiting > for CPU23 to acknowledge the synchronization request. > 4. Finally, it formed cyclic dependency and lead to softlockup and dead > loop. > > To fix this issue, add cond_resched() in the memory_max_write, so that > it will not block migration task. > > Fixes: b6e6edcfa405 ("mm: memcontrol: reclaim and OOM kill when shrinking memory.max below usage") > Signed-off-by: Chen Ridong > --- > mm/memcontrol.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index 8d21c1a44220..16f3bdbd37d8 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -4213,6 +4213,7 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, > memcg_memory_event(memcg, MEMCG_OOM); > if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0)) Wouldn't it be more robust if we put an upper bound on the else case of above condition i.e. fix number of retries? As you have discovered there is a hidden dependency on the forward progress of oom_reaper and this check/code-path which I think is not needed. > break; > + cond_resched(); > } > > memcg_wb_domain_size_changed(memcg); > -- > 2.34.1 >