From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03F8CC0219B for ; Tue, 11 Feb 2025 09:02:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7978E6B0082; Tue, 11 Feb 2025 04:02:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 747E66B0085; Tue, 11 Feb 2025 04:02:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44E636B0088; Tue, 11 Feb 2025 04:02:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A070E6B0082 for ; Tue, 11 Feb 2025 04:02:53 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F24712133F for ; Tue, 11 Feb 2025 09:02:53 +0000 (UTC) X-FDA: 83107073826.06.6D8AF23 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by imf27.hostedemail.com (Postfix) with ESMTP id 3EFC540004 for ; Tue, 11 Feb 2025 09:02:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=N+FbLYnH; spf=pass (imf27.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739264571; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a/vcWSwoaK4yZZjIRRGK2GqLaY/dehgjARDfk7E0I8c=; b=jxKsO46fApkpqlXatEmn5W9tI+VuziUt1VA51RWS1aWd/76iQrR7tcqlyrj2mcd1AFtQxh 19KTn0GqCvKN7ZroHS1xH6pJjH6nDaC/y/UYN+KQrLFMn0X275/Kv2qG1RkJZDJNaPEoO4 gTVt+XyiOEXw1NDlRljkFqQwo63nvUQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=N+FbLYnH; spf=pass (imf27.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739264571; a=rsa-sha256; cv=none; b=U7wEfDdEXMdYaJyY791wTZQIM1e7Hwq8WQMXdMazN6/nGDnGnoni6T8GQlu7MH7ESb+vmp v11XP+xhdRXDNTSh0Uhn7vMMnzcpzgEpG4I7OZW3eyQOnEdfb6D1vF2uaBQEHUorZOOIKL dJXsYPj2KZM0YbvZY6Ut5jQllGBm4HE= Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-ab795ebaa02so661084766b.1 for ; Tue, 11 Feb 2025 01:02:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1739264569; x=1739869369; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=a/vcWSwoaK4yZZjIRRGK2GqLaY/dehgjARDfk7E0I8c=; b=N+FbLYnHk3wiU/flxgM4efZ0KhWj5xLj0iSJlLz4CeH+I7yHirKZRnK0q7mdfyDZWd jIY2qWeFyWwxzthOZcSJoEtxjl6cQPvI7932NlQU+/4enTvXehhBHssAyISM1w3pHUQJ dUmnoEsSS5v1imdmF5uHj8N8qG6g4W7hM80QtdETyJGvoIS0VEGwzAw5ra5WbX5MBUqQ iJsIDKK8oB03co5cRw6nqwoMTEBrbQ+GFjbG0FdQMWMCmQPW5f8xdidfo8sOWvIh9RBD 5L0dn3357MKDoE4QbrNZfB+4Vm6T3c4W6tGFA4TwcgysOf6IgMBv5mFVni58lzhq79qC GR5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739264569; x=1739869369; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=a/vcWSwoaK4yZZjIRRGK2GqLaY/dehgjARDfk7E0I8c=; b=R882FF9bFVgxr45THRx3qohztyvfoNL4uIIGlT05K84Uxur3C0tnTs8mFFkITWKVHr ckr+w9d13URqmw6ebLu7mY5DbRa4hg+ekxPucGOmM/1DgpFs9z5zCri+QGq7oyDjiSVY 7FFG+twGz1j0v2NA2ewHNujSeG0G0qBdt4ZGsjKawTwjrkIqj2nNBBbR7oMVwsi+mPWh PJK6EicNOD2/6UbCk5zNnJdIpLvZhf/wicWKfkiO8spPG1PcMYCtudvg3Qc1JGdAaFcy DejDL3mtEz3upx/DLffYd5gOD9wGu4i5jUhHa61kzaoEpx44CXGp0PWeaj09W3zOSnhe Uxiw== X-Forwarded-Encrypted: i=1; AJvYcCW8/lhKOPyPGP6pFxJVCjQI6sNcngD6njL6uLnYKVvZu6utT250+b8he+xZd91I85d2Lw03PR/lpQ==@kvack.org X-Gm-Message-State: AOJu0YziSfKLMDr5wzuVaGDKVCNvhasjsChQoUB/QCkGgLJDF1USNkbP zhLzhMe05KHXN01ruD71YW95qRKsMGHh5OFgJSpvUTQWYcnL01iSefcuFmvTQ88= X-Gm-Gg: ASbGncsI+rFoUAs81GaThj11yluwQRK3bWnAQ/VnNRn3ky/+nhji14q+PIdHVtTYaa1 dn3618Oi5zU+jJS0pUVl3OaW+jbjlVmINCQRCC+DiVbueoIJxuaBFe9wsfnYAhQoUpEWYW0nWvS IT+iDz4qQjtOMxjtpS+dGWj0iLrPeDs9143pUX4f4hTFMRb+otD8HgJL7nqCOFUPxTZZDWe7xX4 R30HsSiSc7ZazXPJdi6BvMgWUxRDoKGCa7FuGcEAjUZeviyNwRlQAWe0Pm9shxDIjjMZQd5UflW opUPcOuLPybv6yN5jbqgcz+dBhKf X-Google-Smtp-Source: AGHT+IHGiJT3lxoSX40Aod01sSIdr1IyEhGNC4ww2vOluNg3CLMKYJKm4clhF7IDwG5sMxFtt1ZuyA== X-Received: by 2002:a17:907:968b:b0:ab7:d801:86a7 with SMTP id a640c23a62f3a-ab7d80187cemr283211766b.3.1739264569562; Tue, 11 Feb 2025 01:02:49 -0800 (PST) Received: from localhost (109-81-84-135.rct.o2.cz. [109.81.84.135]) by smtp.gmail.com with UTF8SMTPSA id a640c23a62f3a-ab7c8b5a784sm294568966b.75.2025.02.11.01.02.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Feb 2025 01:02:49 -0800 (PST) Date: Tue, 11 Feb 2025 10:02:48 +0100 From: Michal Hocko To: Chen Ridong Cc: hannes@cmpxchg.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH] memcg: avoid dead loop when setting memory.max Message-ID: References: <20250211081819.33307-1-chenridong@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250211081819.33307-1-chenridong@huaweicloud.com> X-Rspam-User: X-Rspamd-Queue-Id: 3EFC540004 X-Stat-Signature: wtnjn6bhanpmdh8o1d1mdzcczmhnyzkq X-Rspamd-Server: rspam03 X-HE-Tag: 1739264571-681603 X-HE-Meta: U2FsdGVkX1/kfE0HKXs+8SdTpGgO78prf9ET3hZaEvHxt5OjBsd9nEeG+R/1SPvskAU8XcwRBJDa3s8hggVZ/WkIjE6BbBsQYekVOkk1AHpuePiXhbqKbWmnR0lmYSfzjyaCaYtf+sGwpnkq+JiFfAVkyEae2b4MB2DUEvCGhEmaazKKKGBmmCb58QREhpmHvs80k2CqBX1plwOM807CU3a127vo+DYsXMHQ/PQYL6sNsIa2YjXhs8B91VzQzAf9HyjZLNN2YVxNZ1913HfQgXotHtLh5NuK1Hfc7faB7LvJETtIziqP1cebSXPB27ItlH2jBwrUyBIKO3nn5221KcY0i8KAU7WOgVSWWa4BaNIRFovGKEbvE4Dg/ci5ppswSJW/LhLrbsFtMhV8dPx45zmr73Eolw577Ajp3+55DEXtWBDkuqD0UVQfnLfpnQbRUlOtpUibvvjV0wGgcgiSSGsS97n1rzDulVBj8FlLXfH7oftn7/Tmx5o+xry9T5Qtgh5ycxJV3d1UfzF3qPaN+eB/cAXDCYdbLqpxJDNDeh0Q9NqDf70BdsyrNLw6LbmNwZ7sJEzupiCtfXS6kzuTSSPT3P2sTYwEsD2ex9G2/g1gPNgI4LFt8wvR3IJj1Wp52GgMX9+6MyDWM2unIlTpEm9EXil1d3C416LgnL8AVjxSlNircipFTkkIFxu/jjy1Dvob110WSElAoBtBl7c65g8mUgDGGYjL/kW6BihOz4fPjg0qMVb1ZSPng1Q2JjKWw7zOhyDHnz66UdeLT2lihK692OwtB+1GyEOFUxQZqlqq09gEx0kkQsCAlBLKDd/fosYA7n5wazGXhIHfSp20rBLOtpn/NSeYJTczdzx5QOMxXiqReSMVZlJGO/BBMCnGECFrSU2uSMLnqgHF/5JfWeXXZt2N+U+1Pl6801ul5Pc/5DFBNfleEnLMy9QOIXl5YEjGvS5Qt0eOCiqcJ1B 6jOtNywS jRYsIvlM2j7s9y3XeI4qnXQc6kLVES+M11T5isFKjxF9g/EQusSc7Z+7aTYEadOoIO/MimHj0Th7b9RHyuHoquGq90pG30YT5+h03Y/gr35ohxx+MHBDDGGlsUJaVU/TC5vAu565/D7lTpgiIwTwSnD/YRoPaT5oMEorMmNuCnlIQynsDzdmtASbcyWExZqKZrzs0OEhF30jp2Qirk8qKDILiTKfrG3eSy5Lj1dB7M/4q30ZgCrJ7JEhL6SY+yIsHOddgegv4iqUfBKE8Imkzg2EdsQ1ydqigH7B4lipgo9/QiwvOsIJ3HR+4D0YgYegY/UDnP30U5ynYTgKroslxgYJWArxGdgFFK55zjyZYhz73HjTfpCatV5jE9W7fuo95VSlC44JmqAWMYMmfQ8vuMLoXcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue 11-02-25 08:18:19, Chen Ridong wrote: > From: Chen Ridong > > A softlockup issue was found with stress test: > watchdog: BUG: soft lockup - CPU#27 stuck for 26s! [migration/27:181] > CPU: 27 UID: 0 PID: 181 Comm: migration/27 6.14.0-rc2-next-20250210 #1 > Stopper: multi_cpu_stop <- stop_machine_from_inactive_cpu > RIP: 0010:stop_machine_yield+0x2/0x10 > RSP: 0000:ff4a0dcecd19be48 EFLAGS: 00000246 > RAX: ffffffff89c0108f RBX: ff4a0dcec03afe44 RCX: 0000000000000000 > RDX: ff1cdaaf6eba5808 RSI: 0000000000000282 RDI: ff1cda80c1775a40 > RBP: 0000000000000001 R08: 00000011620096c6 R09: 7fffffffffffffff > R10: 0000000000000001 R11: 0000000000000100 R12: ff1cda80c1775a40 > R13: 0000000000000000 R14: 0000000000000001 R15: ff4a0dcec03afe20 > FS: 0000000000000000(0000) GS:ff1cdaaf6eb80000(0000) > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000000000 CR3: 00000025e2c2a001 CR4: 0000000000773ef0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > PKRU: 55555554 > Call Trace: > multi_cpu_stop+0x8f/0x100 > cpu_stopper_thread+0x90/0x140 > smpboot_thread_fn+0xad/0x150 > kthread+0xc2/0x100 > ret_from_fork+0x2d/0x50 > > The stress test involves CPU hotplug operations and memory control group > (memcg) operations. The scenario can be described as follows: > > echo xx > memory.max cache_ap_online oom_reaper > (CPU23) (CPU50) > xx < usage stop_machine_from_inactive_cpu > for(;;) // all active cpus > trigger OOM queue_stop_cpus_work > // waiting oom_reaper > multi_cpu_stop(migration/xx) > // sync all active cpus ack > // waiting cpu23 ack > // CPU50 loops in multi_cpu_stop > waiting cpu50 > > Detailed explanation: > 1. When the usage is larger than xx, an OOM may be triggered. If the > process does not handle with ths kill signal immediately, it will loop > in the memory_max_write. Do I get it right that the issue is that mem_cgroup_out_of_memory which doesn't have any cond_resched so it cannot yield to stopped kthread? oom itself cannot make any progress because the oom victim is blocked as per 3). > 2. When cache_ap_online is triggered, the multi_cpu_stop is queued to the > active cpus. Within the multi_cpu_stop function, it attempts to > synchronize the CPU states. However, the CPU23 didn't acknowledge > because it is stuck in a loop within the for(;;). > 3. The oom_reaper process is blocked because CPU50 is in a loop, waiting > for CPU23 to acknowledge the synchronization request. > 4. Finally, it formed cyclic dependency and lead to softlockup and dead > loop. > > To fix this issue, add cond_resched() in the memory_max_write, so that > it will not block migration task. My first question was why this is not a problem in other allocation/charge paths but this one is different because it doesn't ever try to reclaim after MAX_RECLAIM_RETRIES reclaim rounds. We do have scheduling points in the reclaim path which are no longer triggered after we hit oom situation in this case. I was thinking about having a guranteed cond_resched when oom killer fails to find a victim but it seems the simplest fix for this particular corner case is to add cond_resched as you did here. Hopefully we will get rid of it very soon when !PREEMPT is removed. Btw. this could be a problem on a single CPU machine even without CPU hotplug as the oom repear won't run until memory_max_write yields the cpu. > Fixes: b6e6edcfa405 ("mm: memcontrol: reclaim and OOM kill when shrinking memory.max below usage") > Signed-off-by: Chen Ridong Acked-by: Michal Hocko > --- > mm/memcontrol.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index 8d21c1a44220..16f3bdbd37d8 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -4213,6 +4213,7 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, > memcg_memory_event(memcg, MEMCG_OOM); > if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0)) > break; > + cond_resched(); > } > > memcg_wb_domain_size_changed(memcg); > -- > 2.34.1 -- Michal Hocko SUSE Labs