From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28350E77187 for ; Wed, 18 Dec 2024 07:56:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD88B6B0085; Wed, 18 Dec 2024 02:56:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A86C56B0088; Wed, 18 Dec 2024 02:56:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94E276B0089; Wed, 18 Dec 2024 02:56:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 75D776B0085 for ; Wed, 18 Dec 2024 02:56:27 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EB9C9140C46 for ; Wed, 18 Dec 2024 07:56:26 +0000 (UTC) X-FDA: 82907321616.15.5F2B297 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf22.hostedemail.com (Postfix) with ESMTP id BE164C0004 for ; Wed, 18 Dec 2024 07:55:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=TuVDQ7Eh; spf=pass (imf22.hostedemail.com: domain of mhocko@suse.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734508563; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YQN0wm71xAkj0p1Q/NdDLgt+KxZdFC6f+ofVO1dsHQo=; b=UtRDpwfkD5KtmtBR7yuwH54/yse08qVb+KeqzpuHU4QVckmTLtkKm77tq1BIZGT2m3xxCM 0eCEoJrdWTwURKgsKoehS1oVG2t2GYdLNH1ZontXBdG480Dd1GtUcuSgMK9lSUFCGARNO/ c3MDeoDnJTn8LU2d1x/v8WKK8XThTNg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=TuVDQ7Eh; spf=pass (imf22.hostedemail.com: domain of mhocko@suse.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734508563; a=rsa-sha256; cv=none; b=RWkQublYhfptzxctPMW9kVC6lPwmy5iSnVZK8PL3qjsM/oshvLsKHcDF0kl+5OK3KIVCwV JhUcdjb3pcjR72jS7p10whewVOa7weOY+q5oBbvOJpH9ozPg2j63fYl/bIbLHQLdbmYpqt La0eeN9Mg5sckrzgr7Gsv3RjMdm7UKY= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5d4e2aa7ea9so3914799a12.2 for ; Tue, 17 Dec 2024 23:56:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1734508583; x=1735113383; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=YQN0wm71xAkj0p1Q/NdDLgt+KxZdFC6f+ofVO1dsHQo=; b=TuVDQ7Ehg6y1eMeo+Zrxx8iyhcJFlZWg1Nrh1B/hwt8Oa+fY9gMOAXXHJHQ13+u4o2 lipKgWE+r59II9yq0i/pwgvmDpbOfEEo+eFfj3UHp6N3bzXks5Ia9k5EdP4GojvXd7W7 3bztG2wJcqfM4MIJGxjpBCs38SSCGJyPTp9iu78XZqrMz/Mljd/UzJxDiVsPk6Ag8M+g 9bapJ+gCro0aGIxoYKJAqlMPoJhRsIkeLhGxcfS/Xins/1PI4HRuOhdFsjgsjClVITIV ZwzmgGRYuAFppERzyopoWsoVWwg3/+lBh2N+w3LMTVWdr+9emQ45vL48GCqAiLucn6Kf mxLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734508583; x=1735113383; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YQN0wm71xAkj0p1Q/NdDLgt+KxZdFC6f+ofVO1dsHQo=; b=pniWufWbbPXnDk5kfK7RhhY1YO5i4J1PrwUCT51QKE6bH19XnILlnnaCkwuhW64y/0 ebr/Ra5+VwHgFVOIoNn94GSNGD5/l4b5yK4VjSg2ERwE/ixdgzoGa9MmLrkomDLlG7pf 6rHLLioaNm4NGMfkxyekWM1jBqjCnXKllw5sS9ZuKq4uLm8Ftne4MS8cVD6rWeFk1iqb MszDG7U10bDdqmvZC8o+l9cgC9ubFXwXwx/LYZZtiJ13dcslvrQNabbi2sPIpwidmKgr 0eJUNrfzSMyHQEo1ZaHUBBfKRDlHvN7jy+nrtmaJOOAg6jWAlB2tD7qwCPuDX1cV9Qzl H63g== X-Forwarded-Encrypted: i=1; AJvYcCVHYQCfHBDOI93qROp16qI+w9y1i/a9/u1osrxOKn2+TlRpTlsqlTNYnrN3vo1k4R3jH8PgttGbXg==@kvack.org X-Gm-Message-State: AOJu0YzU3J/ZnQAZOETG/ODVwoVelTdAalfoPgLOBSGsOKp1oqy73MzC y2hARMUFaTtJbZ4pJ3Gfc6Jri8Tq46ZonyXKQgp0gJoAT3xxhAW/CGTz0+O+dzGgg/ubTCCX0lR 1 X-Gm-Gg: ASbGncssI4XKUTBHzHOtf7B5fJnZJ6IuRocKhlR1XGwRmXmwS+XZ+pn4R1kWB8DJrGw t69/7cn2ME/xgtB48UKxrMvLjZbWZvopiJ138UJCAAZLuPTITm3Tl/AuoUftWLDy5Hze3UNsBcC GMwaXkCHV0+qkjiIZ7Cimho4cqz7xJG8Nog4l2t8EjeOz6VVRC4YRTlE1mBe76COr1l1baGV91i 4JyYPOXTVVcdOP3BzsN5ZUP/nFpO+SkUwVUGsLg4msTCBmtnu2lxdccFotwpaP7CpE= X-Google-Smtp-Source: AGHT+IG9nTkGC+v0t17oJVqtodNfTCv4+bFZC7mXaRTkfBgwdIfAwM2qjG+K5/ZQ0hmsRv1dwYdgbA== X-Received: by 2002:a17:906:3145:b0:aa6:b4b3:5923 with SMTP id a640c23a62f3a-aabf47baa14mr132241166b.33.1734508583140; Tue, 17 Dec 2024 23:56:23 -0800 (PST) Received: from localhost (109-81-89-64.rct.o2.cz. [109.81.89.64]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5d652ae1127sm5304774a12.42.2024.12.17.23.56.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 23:56:22 -0800 (PST) Date: Wed, 18 Dec 2024 08:56:21 +0100 From: Michal Hocko To: Chen Ridong Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH v1] memcg: fix soft lockup in the OOM process Message-ID: References: <20241217121828.3219752-1-chenridong@huaweicloud.com> <872c5042-01d6-4ff3-94bc-8df94e1e941c@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <872c5042-01d6-4ff3-94bc-8df94e1e941c@huaweicloud.com> X-Rspamd-Server: rspam05 X-Stat-Signature: yga3yzy4aq6hksoqimmgxba4k4wto16m X-Rspamd-Queue-Id: BE164C0004 X-Rspam-User: X-HE-Tag: 1734508551-217154 X-HE-Meta: U2FsdGVkX1/60yHdchgIpxl3Bx64dg6kCglWVSBB1M/na5w43twqLxu5p38cckrEodxPV81TtmsHh1H3Ye2I1aW1OVeLZesHHMzEOJoDv//NcFE6FhFpmO6MZddGxtk9JjPoOaXgdrXzwihmkmZUtJsIdZfmHBppYCRhmNfFFmRNPGZaxwDogx2gr3IMxdEmDXwOdeoP49mDCuWVOOTrStMdhpgg1lAGNc/QeH3ZTk0Vi7+CKNpaaMKMvCxQBIQ3dQyn3ayk2D7ZQOCclcv78h54MNccDvp1r/glx+L2PLg5NBnRkr5HeiDsWEqCbKDlqtIz7QE5u4NWx68edTmWR0gpBhDF4VRGD8Qa883vwKc5U3qry/u/xqlxCgH1XdeW4EsYW3eyfW2j2wOjh5fP4ITdmBUOJf8cZklerOh382s4oisC9I+T5SD6NINnJJwq6HkGyG3OJrIe9U3fA9rqsZvy1//8oMY3oOwIpozatSFiEpVLZ/vBLnwb483Y4zChy/pJ9OY4z17b9oAoLTEoK4dXJaa4dJEeu2qCnzsGfqRq0X4VPeQjxgd09sWCVaSz4yFUsyGYwj00szOEAsDHAAP+2Kz/cXyIQ7E2hI6Fjdzk3HThBMz8wr8WfGV51TcYPG1miLB03b7QBhl442Y2LUsVS8/JS4Lf7NdtJ7hpkQZJFrdMHsL0sWF4gylg1A4IVjh6WiwdkwcuQZGAa7Enw45AJP4D+2ElwI+j6ocPR/Xw8o1F37x/zvjEOIAkLlkEe2uPqyiuVQ6CWL+00bdUXIg315lGCtwfpFV0FZIPMWmzx1sfnMs0XPD1VzpBaFXSD3fKN8xyT8kZPfa29kr7C4Sw0Itys/r3s8LP1SFmD106nia1tEMkkoQP9YS2m3qYEMrJRsv48OnOhWQpIxvM/V1yglfvrwKlbchNKTTggponBL/2xTgkzhHCq6OvT1skbhFCv7DFyDDxWcDV/OM gX8IrDdr jp57MjIjeEcVYzbjkNXdKGpPTb77p/1VxUwWCr0zVvG0NWMEuZAxL8p5mzbgjQJ8ya9fFtHS0uAh8IJQa4ujgHqfUCrdken7irN55mdlE36sY+jR8cZGd1rKpStv9zkyIa7QW3kCBNWCD6+KpxWcPhh6v7pMIPXbXpBzFmNSBoZhfmrD2DAjIC//cSTHnHeyecsdRk+ygjozbIKRLNiS1two+6YScnah8/QIYSYSD86HgcS1C/RfABjy3dNnE7Ea3PQZ9Phvbjcp19j3oLJicMd7iZlHQSu5QPrHsi8Kw7Sl0Xy+18w1BVLQDIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.007944, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed 18-12-24 15:44:34, Chen Ridong wrote: > > > On 2024/12/17 20:54, Michal Hocko wrote: > > On Tue 17-12-24 12:18:28, Chen Ridong wrote: > > [...] > >> diff --git a/mm/oom_kill.c b/mm/oom_kill.c > >> index 1c485beb0b93..14260381cccc 100644 > >> --- a/mm/oom_kill.c > >> +++ b/mm/oom_kill.c > >> @@ -390,6 +390,7 @@ static int dump_task(struct task_struct *p, void *arg) > >> if (!is_memcg_oom(oc) && !oom_cpuset_eligible(p, oc)) > >> return 0; > >> > >> + cond_resched(); > >> task = find_lock_task_mm(p); > >> if (!task) { > >> /* > > > > This is called from RCU read lock for the global OOM killer path and I > > do not think you can schedule there. I do not remember specifics of task > > traversal for crgoup path but I guess that you might need to silence the > > soft lockup detector instead or come up with a different iteration > > scheme. > > Thank you, Michal. > > I made a mistake. I added cond_resched in the mem_cgroup_scan_tasks > function below the fn, but after reconsideration, it may cause > unnecessary scheduling for other callers of mem_cgroup_scan_tasks. > Therefore, I moved it into the dump_task function. However, I missed the > RCU lock from the global OOM. > > I think we can use touch_nmi_watchdog in place of cond_resched, which > can silence the soft lockup detector. Do you think that is acceptable? It is certainly a way to go. Not the best one at that though. Maybe we need different solution for the global and for the memcg OOMs. During the global OOM we rarely care about latency as the whole system is likely to struggle. Memcg ooms are much more likely. Having that many tasks in a memcg certainly requires a further partitioning so if configured properly the OOM latency shouldn't be visible much. But I am wondering whether the cgroup task iteration could use cond_resched while the global one would touch_nmi_watchdog for every N iterations. I might be missing something but I do not see any locking required outside of css_task_iter_*. -- Michal Hocko SUSE Labs