From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6299FE77184 for ; Sat, 21 Dec 2024 07:28:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6711F6B008C; Sat, 21 Dec 2024 02:28:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 620586B0092; Sat, 21 Dec 2024 02:28:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50EAB6B0093; Sat, 21 Dec 2024 02:28:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 340616B008C for ; Sat, 21 Dec 2024 02:28:18 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DEC991607AF for ; Sat, 21 Dec 2024 07:28:17 +0000 (UTC) X-FDA: 82918136742.23.4072BDF Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) by imf09.hostedemail.com (Postfix) with ESMTP id C21EC140009 for ; Sat, 21 Dec 2024 07:27:51 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=NnBBh5MA; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf09.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734766063; a=rsa-sha256; cv=none; b=N+5EzPLJVsqgnnJXfcT/VO05sZyk4gcOOoipHPuKdfZWAYkxfxeU31Mrnzn+CwPOQSZOX5 uA4QbJK8SBDww9sthJpYmO6jTCtKzLR86x8qqah1S2cYRVhg8MroB6zONcH4YFDcfEfwDW cshI/G2H+4ERR/KTS54tIcsr+ImeKrk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=NnBBh5MA; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf09.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734766063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZfsGErZKjmGQuLFmm6/I8+C/KfUfqFh+6QS5YvZcZGo=; b=CPlcu0DSFkslkAzC99ALLQvRn4SnoNKtsv41aEfLZcLTBUJ/yM6vUtDXwyKuM3fjm4npCP q43XRf/Wj15QoVz2YcEjEDkbv7im0K5mnuHQY1xFnDNZNsyJiL6VIjvIHD0wWirUos2pdy cWncym4r3lSGbFGSjGyLBdiLF/m1Qow= Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a9a68480164so440445566b.3 for ; Fri, 20 Dec 2024 23:28:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1734766094; x=1735370894; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZfsGErZKjmGQuLFmm6/I8+C/KfUfqFh+6QS5YvZcZGo=; b=NnBBh5MA+btBJOilwOgXFNEzWDE4fN4ONIaquPfQC17bbR/O84h1a8J0RyUQgZDSho LEcUrjcyqpp2ADaTdAtsBy2cJ4H3TdRIxXBpyPydzHNd7U2+QDXHvz52YV/QMe+IE2KP 5PHUcmaZ7PannURIroM0c3rAm00xL0T8tblgvP60lIkxzSNXLzS1U/mJDAeTLn/mct0l 90nwLRNA7aF/6OmSs5BdusDOUdFW/gdB+9fbjMiyw4uHxibvWK9i1BiYzRpktmJLLlSZ wrARmvpzkvi7pmlbBuoa4EQ38oYrTLPNYbAFoQ28M98Ce4AhLltGYk9y15N7/NN9p0kx eBfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734766094; x=1735370894; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZfsGErZKjmGQuLFmm6/I8+C/KfUfqFh+6QS5YvZcZGo=; b=nDgcFLrAVKZd5OtEe5N55lbOp1fSyulGoJe6y7f7Oof21td1WYb2H4ovkJRUtS1rZM lH7zbX5ohVEtQMLCldkD5VBDqGGl8ex1xQ4JHAJVrioYc5FVgKeTN5vWsK2nUxl73zfM Xt+EEuziKhkY2ZZTM/jCqcOwiF1tFwG7yE7UcmwRc7IQN6FMJWlJCaw/LKIWjbKzQbIp vTmiuJYn35axG7qpjMBx6EsU5QPOONPUu0wQH5Q4PzUmD1eOgbk0JE7py5jZ3Ewq85Nh aI4dcUrA3Gzp2Lq0LmIWYXTePWtY3DGm0740bAn8NSc/+2rtChitOCThYVNHYQg6Jiiw lDVg== X-Forwarded-Encrypted: i=1; AJvYcCXjeelb2zlLiigEFuMg2wAmw+hUcuraIGGjuR2+O3j6ZZLHg52haP0HrGJDtU91amcoaVjX52JdzA==@kvack.org X-Gm-Message-State: AOJu0Yzui+AM209MT577bkHMmDA4cp9E34loPX+8HW4QmA+sgbU8ZI4r xybnqwO8kOBVt6bF/nxVjxhnmR59pSi+AjCqPF48BB+1nvl+irsslF7aKSfQQN8= X-Gm-Gg: ASbGncvUeXK5aTo9uhAawVjC+wSRH08ezmKzJbNyPwHMnlIQiAd3eg7G/53dS80h0mG ePo+LBJSfSdYbZvrayTOy/8qtOgr0GD8PrGJxHrpVEmAusN/1BJYHFVtCIwQTf38V2wISTAdKZ5 l46N+LizgXOeLHpayy2s/XCHfskT/CRjQXaPOwoaaPKzS4rBiwDGwIu7LhgWJPG2+xJKZdTrv63 xXqZSLuKE/6ReHW+G1A5N7uVaGM3RqzUS5Qzq3PaDVmuljfrJdzAkW5obcVSSLz X-Google-Smtp-Source: AGHT+IFHzrzntAhwZIjd/yc/QVtZxcJtWRq2cNvhXhZieIi32xjE4dR2rFr7xEc0wZQva/G+/Hck9g== X-Received: by 2002:a17:907:7208:b0:aa5:43c4:da78 with SMTP id a640c23a62f3a-aac33685dfemr503281866b.51.1734766094208; Fri, 20 Dec 2024 23:28:14 -0800 (PST) Received: from localhost (109-81-88-1.rct.o2.cz. [109.81.88.1]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aac0efe46b3sm252247466b.99.2024.12.20.23.28.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Dec 2024 23:28:13 -0800 (PST) Date: Sat, 21 Dec 2024 08:28:13 +0100 From: Michal Hocko To: Andrew Morton Cc: Chen Ridong , hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH v2] memcg: fix soft lockup in the OOM process Message-ID: References: <20241220103123.3677988-1-chenridong@huaweicloud.com> <20241220144734.05d62ef983fa92e96e29470d@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20241220144734.05d62ef983fa92e96e29470d@linux-foundation.org> X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C21EC140009 X-Stat-Signature: mjtsobg4aprnz55ozqiqhh8gbz3wi15c X-Rspam-User: X-HE-Tag: 1734766071-167582 X-HE-Meta: U2FsdGVkX1+oRzvDxsdAFnNngx1iUBaRZ8m0UO3xSaDJ023gvbmlRYAHNTpLNkiBAGIrEbGqaEIQINJ98PL6umGPXtLEjkgKH/LrV5/2WarlLozduIxmVvDSZbFLVf3Wm5YyWefL34UE2vESJqOu17nRDRF2WYJWXWRboPldqECfVNhKDEP6o6/yvxhGGocUl4NiqJRDQMj4RpBbsADMmuQ/37b7caOGTlhrZ2WhJSSTulrtafZxCDaT+PW1jVl3Yy/kh1tTaIqQQBdPWKNmopLJKxFVvQ1VdIVsMBQi9D6NexOVFBorLVP/xy5pBeaHfOuuF0yIIT3mUFQ57LWl/bTUTZk6ZebYbMBpsci6HEd8LiMGrzIpBRyR1oMm54MbX8Hd3OkbuHyhoruKTNg5nn3wq1Da1ylxl0OYYMTAb5Lh+bpfUNUSIDlUtEeDWaWj8QaCbyCgukPA+pJIIqE7Xnc2pWmZXuXQe7RTXJiL9ghNgj/rtUeLxw0Qxcx8A+LXIfgUvXJRg/e83wmw8u5zmOwhCVWvrXb0dH6YEJu1nqJh3L8fa5aAGKrpfsYL2rPHa5ZmnLub39uCF7h5AWCUoTj8TFbcmUKjXi92x4VanqU+5iG7ChbKEPpcjSuSYqo34UI1mj+Xl6q4UgiVGnQzEM/DJpU2nOYRsa5DtKjz3UgtNHu0uzGOPJ1opieTFTiI6W4g5+fv1fYclTfBavoy88B9VeB+NFpk0Y6XvseogIUjllJ+KQpfYgVenQLRzHesYQXngJqCecCZCIkIET/EpdCeTtAC1xUKodwyIL9kx1eIQmlq2ieAVXVDmFZ0NPV3wVI9QStgzk1NN+l6MkRv32cRHvaZ9vVQUYaiA7jSLygJWNMOfu7ZdoK0jw8QJuzEq//++srinh3t3ZB/ri9ZAvR81FwuWYqq4S4jWWVdGmDNgLdnt5+9i+Cs0cVCA1bmN1G9uR+yitM53jce6rz uqQR+sgt Qp99JY4cdQ37otJ424+LsGDUZ0ySvDWzMm/LSaTfpGQ5hodWE4+1ZEqybq+p1uj7b7cbqAnlnIc0VajC7qYRCNOQz1/A8cOnenU0U7l25lQ9TNCkny92WbiDM3UaW1Ftr7Zj4skgcoREhm2eKXIBNx1WvSp9sGcQ6yo1MaFKPRSd/RxFnvn1SBFUvrqdUU9bWGfgS9Xmp9qv6QBa0iRG4xtKBTPyG3g22P3mnCttyqBkexD2KShyxH8vLkypv8tuWSVl8t0Zc6SYLGwiAvg/PqJh58+8VFnLnxbxg9pcMItRKPdH6VidS4jHo7W4bODtjgqOgMeeIIoOzTjJdXr+cTKA1FhlwsFq4LDv3jooVKnHaGvY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000281, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri 20-12-24 14:47:34, Andrew Morton wrote: > On Fri, 20 Dec 2024 10:31:23 +0000 Chen Ridong wrote: > > > From: Chen Ridong > > > > A soft lockup issue was found in the product with about 56,000 tasks were > > in the OOM cgroup, it was traversing them when the soft lockup was > > triggered. > > > > ... > > > > This is because thousands of processes are in the OOM cgroup, it takes a > > long time to traverse all of them. As a result, this lead to soft lockup > > in the OOM process. > > > > To fix this issue, call 'cond_resched' in the 'mem_cgroup_scan_tasks' > > function per 1000 iterations. For global OOM, call > > 'touch_softlockup_watchdog' per 1000 iterations to avoid this issue. > > > > ... > > > > --- a/include/linux/oom.h > > +++ b/include/linux/oom.h > > @@ -14,6 +14,13 @@ struct notifier_block; > > struct mem_cgroup; > > struct task_struct; > > > > +/* When it traverses for long time, to prevent softlockup, call > > + * cond_resched/touch_softlockup_watchdog very 1000 iterations. > > + * The 1000 value is not exactly right, it's used to mitigate the overhead > > + * of cond_resched/touch_softlockup_watchdog. > > + */ > > +#define SOFTLOCKUP_PREVENTION_LIMIT 1000 > > If this is to have potentially kernel-wide scope, its name should > identify which subsystem it belongs to. Maybe OOM_KILL_RESCHED or > something. > > But I'm not sure that this really needs to exist. Are the two usage > sites particularly related? Yes, I do not think this needs to pretend to be a more generic mechanism to prevent soft lockups. The number of iterations highly depends on the operation itself. > > > enum oom_constraint { > > CONSTRAINT_NONE, > > CONSTRAINT_CPUSET, > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 5c373d275e7a..f4c12d6e7b37 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -1161,6 +1161,7 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, > > { > > struct mem_cgroup *iter; > > int ret = 0; > > + int i = 0; > > > > BUG_ON(mem_cgroup_is_root(memcg)); > > > > @@ -1169,8 +1170,11 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, > > struct task_struct *task; > > > > css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it); > > - while (!ret && (task = css_task_iter_next(&it))) > > + while (!ret && (task = css_task_iter_next(&it))) { > > ret = fn(task, arg); > > + if (++i % SOFTLOCKUP_PREVENTION_LIMIT) > > And a modulus operation is somewhat expensive. This is a cold path used during OOM. While we can make it more optimal I doubt it matters in practice so we should aim at readbility. I do not mind either way, I just wanted to note that this is not performance sensitive. > > Perhaps a simple > > /* Avoid potential softlockup warning */ > if ((++i & 1023) == 0) > > at both sites will suffice. Opinions might vary... > -- Michal Hocko SUSE Labs