From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF162E77188 for ; Mon, 23 Dec 2024 02:38:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 575A26B007B; Sun, 22 Dec 2024 21:38:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 525806B0082; Sun, 22 Dec 2024 21:38:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ECCD6B0083; Sun, 22 Dec 2024 21:38:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1BB946B007B for ; Sun, 22 Dec 2024 21:38:01 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 955451A0AEF for ; Mon, 23 Dec 2024 02:38:00 +0000 (UTC) X-FDA: 82924662492.13.F42AF1D Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf13.hostedemail.com (Postfix) with ESMTP id 3B2B62000C for ; Mon, 23 Dec 2024 02:37:19 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734921439; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6yTSF4Wq4y+FxFu9NFpQ7c45VyG/kWFAP73jyZCJTf4=; b=6lJlzEVaZJt28b2Akb+Iyei7863h+Lieweivl9WgWFhwhE4MwAujudJ/kY17FfUmFTz/xO k9I6mtXydHvyrv/NW/TAobieY23X0YhEUTvGAZrq5QIX4Mgby+DhW7AojPmtDTipvgTU7Y 72Mmme49j1499jNYgaFlo5fZ+yUrXcU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734921439; a=rsa-sha256; cv=none; b=2bQ8OLUIkGDZmbMJf5eGv8T4PM3HbLnyx2kk+QbBLvDYRxNSGiCpWB9jJ3/dGAORXcK0Gp AP5WjTPv2h5cL9CBG9LVIEO8/Zx84dvx2lwZJnTOPTZH19bmwNIrbUKtcg1e5hEZPLXmZD s8BWYFtUq08u+L4m7AV1lELlvsj8nNk= Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4YGhy20NYpz4f3kvt for ; Mon, 23 Dec 2024 10:37:30 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id BE83B1A0196 for ; Mon, 23 Dec 2024 10:37:50 +0800 (CST) Received: from [10.67.109.79] (unknown [10.67.109.79]) by APP1 (Coremail) with SMTP id cCh0CgCXP7H9zGhnOl_7FA--.63463S2; Mon, 23 Dec 2024 10:37:50 +0800 (CST) Message-ID: Date: Mon, 23 Dec 2024 10:37:49 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] memcg: fix soft lockup in the OOM process To: Michal Hocko , Andrew Morton Cc: hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com References: <20241220103123.3677988-1-chenridong@huaweicloud.com> <20241220144734.05d62ef983fa92e96e29470d@linux-foundation.org> <6319a3df-b19c-4e35-a46c-c4a5ea003a6b@huaweicloud.com> Content-Language: en-US From: Chen Ridong In-Reply-To: <6319a3df-b19c-4e35-a46c-c4a5ea003a6b@huaweicloud.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-CM-TRANSID:cCh0CgCXP7H9zGhnOl_7FA--.63463S2 X-Coremail-Antispam: 1UD129KBjvJXoWxGryUKw1fury7GFykJr4Dtwb_yoW5uFy3pF ZrWa1Uta1rJryUXr12v34Iq3s2y3yfWr1UXrn8Jr12kr9xKr1xGr1UKr15CFyrZF1akFyS vr4jv34xurWDA3JanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUv0b4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcVAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0En4kS 14v26r4a6rW5MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I 8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVW8ZVWr XwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x 0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_ Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU0 bAw3UUUUU== X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3B2B62000C X-Rspam-User: X-Stat-Signature: pq77zneiwnbtmanxxac1ej734817jr4g X-HE-Tag: 1734921439-244155 X-HE-Meta: U2FsdGVkX19s1VYImU/4PAigeQAS67ta4J6/rtKhQ6lVPS02q6BnGEBW9D6u8glwzqw1Ndo/txEQQ/H+GFLtVttTZS2Eg1gLLBwd/j/x2mMEMxKFniamwCBy46OEDq8fhEc4VwYBgvsH+RcjjmlEH8oftDtSBQ14RW4gLPXbz8ktF4Kj+5iRgdoqhENlFny9KrYmgStBKVghhF/479iT/6V0wkIQSLEA+MuQaPOIOChlCKyHICSKTugm+uWXVbysElLrZHzXftCEERYslIrDnchK1Z7Cm4OyGD4/WDAm4XLSc800hGa3geEuw3nikrxOhoeYK1H3F1Njm1aD0PM8XwVFR25sfFnriAc4bcccYasdTlnN28BSvwv1JcimpzZCSCoJspSBHXCiBdPlowAGFawV7CybCgZZctM+AGj1StUFI6lI04SUgcJzYKHPFL678Je/00a8Oyzxo4xyA4MSeM9EVnBQVflzs+Z7zphz0iMdoNYCZoFY+RV4QvqAr7s3bVzD2kyw0rOuIEv2AsburrNrCziZWmRH4E+IkbPgBTtAjDQloUOjUkF4QoGxlC5m+yKlSsWz+FPHVRlEGBxS1w0XPg06tRYrgEm4+dq4COYHHziy8jfPx9ggaxBSc4RNuCvuF5SG037uUzvbHkig6jjo+Fl/iCSRnPygbbx4ofDHYZB03CWtUxQ3hSpj1cisEtA7ZDpi5QnrqBSsHikKrv7YTwZqbJAZk1vWi/laOrJ/juUb1mScsfXB2GVmtfPtO0BdpXE8gMkbq2uDz2j+fj46j6tF/QyX9kW/i+JNPQv/nopEhFIYPd6gOaa3BDLfkDrJtZhKCnMmn4grEEG3C1vSvBowGKebfri4q0MsgC0IOtnnpbBZ8At5zkIXHARw/Ij74hQR+lk8Y2YYby+MZiFmC08tM3CDsuAlGQCn9ywrSa8of2b48LhksmAAEzW0LclISASXJgjgffsjVga 1Ev3AJNi +PWrQDYciCFf5zLxmIKKlfHzynSrs4oZkUEHwt+6o6ick1AhUnydYHC36CntrgoViD+jwKd8ZnKVBGu8QNeh5UudMv6wI6fP6fiZ2hT69z3/N64Q8RNaVYLGW9BaAEkEaenaW8YzC/CzP8Pz1NVxGSPZgTVS1mTMvnn6AoRd6QI31dg+kWNpxvCE5F+zcaaXcj20nRMHURAYC9hQlZjk4pTb6hiQ9+YMeRRDBYuB/EGGpf8w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/12/23 10:23, Chen Ridong wrote: > > > On 2024/12/21 15:28, Michal Hocko wrote: >> On Fri 20-12-24 14:47:34, Andrew Morton wrote: >>> On Fri, 20 Dec 2024 10:31:23 +0000 Chen Ridong wrote: >>> >>>> From: Chen Ridong >>>> >>>> A soft lockup issue was found in the product with about 56,000 tasks were >>>> in the OOM cgroup, it was traversing them when the soft lockup was >>>> triggered. >>>> >>>> ... >>>> >>>> This is because thousands of processes are in the OOM cgroup, it takes a >>>> long time to traverse all of them. As a result, this lead to soft lockup >>>> in the OOM process. >>>> >>>> To fix this issue, call 'cond_resched' in the 'mem_cgroup_scan_tasks' >>>> function per 1000 iterations. For global OOM, call >>>> 'touch_softlockup_watchdog' per 1000 iterations to avoid this issue. >>>> >>>> ... >>>> >>>> --- a/include/linux/oom.h >>>> +++ b/include/linux/oom.h >>>> @@ -14,6 +14,13 @@ struct notifier_block; >>>> struct mem_cgroup; >>>> struct task_struct; >>>> >>>> +/* When it traverses for long time, to prevent softlockup, call >>>> + * cond_resched/touch_softlockup_watchdog very 1000 iterations. >>>> + * The 1000 value is not exactly right, it's used to mitigate the overhead >>>> + * of cond_resched/touch_softlockup_watchdog. >>>> + */ >>>> +#define SOFTLOCKUP_PREVENTION_LIMIT 1000 >>> >>> If this is to have potentially kernel-wide scope, its name should >>> identify which subsystem it belongs to. Maybe OOM_KILL_RESCHED or >>> something. >>> >>> But I'm not sure that this really needs to exist. Are the two usage >>> sites particularly related? >> >> Yes, I do not think this needs to pretend to be a more generic mechanism >> to prevent soft lockups. The number of iterations highly depends on the >> operation itself. >> > > Thanks， I will update. > >>> >>>> enum oom_constraint { >>>> CONSTRAINT_NONE, >>>> CONSTRAINT_CPUSET, >>>> diff --git a/mm/memcontrol.c b/mm/memcontrol.c >>>> index 5c373d275e7a..f4c12d6e7b37 100644 >>>> --- a/mm/memcontrol.c >>>> +++ b/mm/memcontrol.c >>>> @@ -1161,6 +1161,7 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, >>>> { >>>> struct mem_cgroup *iter; >>>> int ret = 0; >>>> + int i = 0; >>>> >>>> BUG_ON(mem_cgroup_is_root(memcg)); >>>> >>>> @@ -1169,8 +1170,11 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, >>>> struct task_struct *task; >>>> >>>> css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it); >>>> - while (!ret && (task = css_task_iter_next(&it))) >>>> + while (!ret && (task = css_task_iter_next(&it))) { >>>> ret = fn(task, arg); >>>> + if (++i % SOFTLOCKUP_PREVENTION_LIMIT) >>> >>> And a modulus operation is somewhat expensive. >> >> This is a cold path used during OOM. While we can make it more optimal I >> doubt it matters in practice so we should aim at readbility. I do not >> mind either way, I just wanted to note that this is not performance >> sensitive. >> > > I think '(++i & 1023)' is much better, I will update. > Thank you all gays. > So sorry for wrong spelling Sorry sorry. Thank you all guys. Best regards Ridong > Best regards > Ridong > >>> >>> Perhaps a simple >>> >>> /* Avoid potential softlockup warning */ >>> if ((++i & 1023) == 0) >>> >>> at both sites will suffice. Opinions might vary... >>> >> >