From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C1EAE77187 for ; Wed, 18 Dec 2024 10:22:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E284E6B0082; Wed, 18 Dec 2024 05:22:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB1976B0083; Wed, 18 Dec 2024 05:22:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2A776B0085; Wed, 18 Dec 2024 05:22:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9CD1D6B0082 for ; Wed, 18 Dec 2024 05:22:47 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4A3FF1C7C14 for ; Wed, 18 Dec 2024 10:22:47 +0000 (UTC) X-FDA: 82907690544.11.EE1C96D Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by imf09.hostedemail.com (Postfix) with ESMTP id 90431140004 for ; Wed, 18 Dec 2024 10:22:23 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b="E/ypeLPv"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf09.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734517335; a=rsa-sha256; cv=none; b=KCssbGSvXV/zG+scNEECgvthnzfQAvrEK9K46aws3HPkpf+lXuwa1mbke1Dd9OaGZv9DKd PphSgnSQbzqvkInOsHUvRE1mzB9s8tPEKXRNqn2e5COZwyeXivQL2H3KTY3pDF9fiB7Uyn Im8OdCX6jUk8QagPDAW1LzjSPKr+0Bc= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b="E/ypeLPv"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf09.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734517335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oYOKzNBnl4zkAfK+4bxUQampaW88vV82YML9h86Utj4=; b=cFWYZSFbhHMMAbXXQOh1Ok3hInS5Bkc+0/fMpWhlwEwOA+msTMdCwgZ9mQ7zH3He1QTbFQ 187ZmvKbFg8QZBeh+Z7bud9pPJCSiyqsHYPLW5lr9O3lpLQJgiXpq6eA3VvLNSG2CZxuyk /KHXiwIwXBwV8izlc9OeePHpsp+jdPg= Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-aa679ad4265so100207666b.0 for ; Wed, 18 Dec 2024 02:22:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1734517363; x=1735122163; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=oYOKzNBnl4zkAfK+4bxUQampaW88vV82YML9h86Utj4=; b=E/ypeLPvAIXj0EmzFehyQDnwpGEgpeBODTKAGMawEJ1XLoM9UjKjDnNDZlnkAKNnzF Y/ygKWUHB8/ffiTHggCpq+fFIpnZF7qywAuliXTCvuH92GOSfyqzxDVUkJAfFNSARTOQ T/dsyMkIAtmMHjQeIyFr/PHwgDUxqyumII6LfqAbfagEjm5KItC4bK/zmqUY0U3oUo7y F0zmW1eJn3bfdvsl+UokgJKzjfti/kkTY74UDK/012zVjgsaWdWUTXvYCl+N2Bwz1ESM 0I5LklUM8CQ17WXsitHKfNb9SOIGN8wIFjJRp1QoyUhqDBkY709fvazRTyU3Xq04pD3v pc3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734517363; x=1735122163; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=oYOKzNBnl4zkAfK+4bxUQampaW88vV82YML9h86Utj4=; b=JiQzDpvN2Pimx00yZ3VJPKM9BCwjVQSJRmYQ1M64P5OBVEB73H7hihb/6ilqiG1hPz ZobBAJnpZUNSUSLVbk29ZSOXWNnTw0Avz+CtRCCqgG3BDEuLGX0zaR2HSPsLow6oYCrp eqk+5IFqep2nC5PP6OXPUIeijSphxQPE70bF5G6LavQFYWytqKaxetnJ8uKWh49VQcZV V1AuYTOGcECYVPCAk+R9W467PHU8H2qvCBkShXPFZuQOV1FrSngS97KcEt2hYiOg3GZW e7A7VeUacv872BmnbwIy8GO2DnflMtED4L4vW8sl6jZuRK2M6oJK6v5CNlind79H5ePg QzPw== X-Forwarded-Encrypted: i=1; AJvYcCWMr4ZTmnE3NbpeRlx8XSpYb7O4bqQQkqLZtrkStKhftWxfk6J4pndyYYKcMQgEYn4aShKMwduWAw==@kvack.org X-Gm-Message-State: AOJu0YxGeaScfCcfJHKVK8rVnpkVTSz7Byam67cgcgZL2B9TYpEd0q2g Ah6HRBM61avqy2spuXuRhZuN8p5gAy+qMsTQpZm4rsxug8TLoc/GyaP/NX9b3Zg= X-Gm-Gg: ASbGncurUZrITDxyWkipt9PA/fiSc8HX1Z6Is26Rr01kbqOOgpV5aHVjeCiO01+Orz2 sY8fyyrc297lyPEOtf91S7KPdNHO7Em7S0jC1c6fyzLUqnqXXNliGtgi5KQpANon9tknJ83WmSG WD5yyUG3RPUaH5WqPfWHe9PKZq+MptO1TfVGAhhRzD/UApfsBR/e0dCD8eq7KDjgm5HUDYkAYgP eiYbUZsbOhXZE+3coagErRUvbf0yCSi3dFsznVWA9XNEQ+Aev8PWg9n/mPo+1OJ/Kc= X-Google-Smtp-Source: AGHT+IEHMf20e4IOkXuBAj5Qk+7IMn4EhZCqjZWbJBo6BUh8KaMpOPPLLrs0IiDTRqJYNXF2qFtlWw== X-Received: by 2002:a17:906:c093:b0:aab:edc2:ccef with SMTP id a640c23a62f3a-aabedc2cdffmr301073466b.2.1734517363519; Wed, 18 Dec 2024 02:22:43 -0800 (PST) Received: from localhost (109-81-89-64.rct.o2.cz. [109.81.89.64]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aab9638ecfesm548407766b.146.2024.12.18.02.22.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 02:22:43 -0800 (PST) Date: Wed, 18 Dec 2024 11:22:42 +0100 From: Michal Hocko To: Chen Ridong Cc: Tejun Heo , akpm@linux-foundation.org, hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH v1] memcg: fix soft lockup in the OOM process Message-ID: References: <20241217121828.3219752-1-chenridong@huaweicloud.com> <872c5042-01d6-4ff3-94bc-8df94e1e941c@huaweicloud.com> <02f7d744-f123-4523-b170-c2062b5746c8@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <02f7d744-f123-4523-b170-c2062b5746c8@huaweicloud.com> X-Stat-Signature: yxb4wddkb6mhpn75jamnz4x4eqkt15t6 X-Rspam-User: X-Rspamd-Queue-Id: 90431140004 X-Rspamd-Server: rspam08 X-HE-Tag: 1734517343-202467 X-HE-Meta: U2FsdGVkX1/1dGkWhXTCfdIKmWkAS54Gbxo3klqnBXl9mOKjk5VfNPlMjpQOTSLamzpzhQdG9oAqomtzXPXivKorrbhpj02k5CgkkJ3n7cQh6l2Sv6SV2bCk4ltzK9+1QSVCenVkn2ZY11st6HXZXj14gRXXwxaYhmJXv3Xk7MPtRZixFolcgzQSNN2FXxyLaKGxwOfzJzFt1Rwoc4kvRucCNG4xR+8UScg1J3cpxaohoke/QUdr794p2hhaacKXpftQEecGvjjuIo5D9tGQOqDO2drKKkCgJVx9YRiPT/hGTGFWTdIdg4wJF5lz7zERoSQVGm6CUPFf4veAmTBEyXuJkW9iESZ7oB89Rj1vkQL8bqiQ1FJF4R7Vf7ko/gNQ72p9jS3sOUnXydgKZGbli+3Dc3A0wYAqyJ49SefHHsS030c5/Nh1yHaQynpFX9W8qOigLCiRhdPqlNSAoPvduMKUXqr5+iUsdjDN1VMTNxwMXdx6vVoaRXNW2B+LgwvTYBYWyrgUmGBTjo5E2XIeVg6y+NDRlmZxy6dqZhI0T75as5dadBTv1pQmhZot0SQSTfTRlBDewPtGxwzcVCfoOCQNau/aFI68IzV6ED3p+CQppwDlOcd/jBpC3vhhGd8ONF057a+6dwrAFx506tPX6cSjPki4XLaTpM+p3tH/e8Y3fs4rnvORUDNhde6vq59OTydLi08bgkHKuCrg12d8DMgTmTF3OS61O67Sgu6EDbX+Z6OfHPESp6WYyFFxIRqa3tMBQ6KyWwWD1zGFEenkJmb75DEWgwzXNFw2lF8BDZbGKuYnIkkLWVWfQEOylKnBwJ5g+BfMVx0eHd7UISQBFH2FtNgDQkXtQso5Eb3f/E1+owtm37LH6IxzvXPF0J1afGj6ppcL1+9dbti8rBFmRgQfwPUSvsavmr6mv3vBrq7jdAlt5KlIDAirsLZN39MQ1Bdj8eDIfUD3Io2KmHU 1I6Qg5Ny gOPnZt+qK2jDLgKr1uKXBhhH8K1UlcelZ9BSnVI/FaWDQH4DKBaKjLukOust6TL/zg4rpIWAVyGTfM8IJQreoq7UYGsJ/jOGPxr9hntar326s4BY9frnNZIIHeMpQ1iK9Un+QD5rS+GeRzZMKPo4MlSYeTu3yTdsuXbWEZfrjLdh5cEVTch3YU+oazFevTKq8L4E/TfFNeUNfu85ya11arj3ejld9ac1PyLtSO0LXspbuUTe5uDeytMHcyCvkJOj9gkp8GCyiVJPdbdhZ+D57FkL0fy4v7TPqSezAwf9xZR9knHA4vtrA4DfAlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000024, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed 18-12-24 17:00:38, Chen Ridong wrote: > > > On 2024/12/18 15:56, Michal Hocko wrote: > > On Wed 18-12-24 15:44:34, Chen Ridong wrote: > >> > >> > >> On 2024/12/17 20:54, Michal Hocko wrote: > >>> On Tue 17-12-24 12:18:28, Chen Ridong wrote: > >>> [...] > >>>> diff --git a/mm/oom_kill.c b/mm/oom_kill.c > >>>> index 1c485beb0b93..14260381cccc 100644 > >>>> --- a/mm/oom_kill.c > >>>> +++ b/mm/oom_kill.c > >>>> @@ -390,6 +390,7 @@ static int dump_task(struct task_struct *p, void *arg) > >>>> if (!is_memcg_oom(oc) && !oom_cpuset_eligible(p, oc)) > >>>> return 0; > >>>> > >>>> + cond_resched(); > >>>> task = find_lock_task_mm(p); > >>>> if (!task) { > >>>> /* > >>> > >>> This is called from RCU read lock for the global OOM killer path and I > >>> do not think you can schedule there. I do not remember specifics of task > >>> traversal for crgoup path but I guess that you might need to silence the > >>> soft lockup detector instead or come up with a different iteration > >>> scheme. > >> > >> Thank you, Michal. > >> > >> I made a mistake. I added cond_resched in the mem_cgroup_scan_tasks > >> function below the fn, but after reconsideration, it may cause > >> unnecessary scheduling for other callers of mem_cgroup_scan_tasks. > >> Therefore, I moved it into the dump_task function. However, I missed the > >> RCU lock from the global OOM. > >> > >> I think we can use touch_nmi_watchdog in place of cond_resched, which > >> can silence the soft lockup detector. Do you think that is acceptable? > > > > It is certainly a way to go. Not the best one at that though. Maybe we > > need different solution for the global and for the memcg OOMs. During > > the global OOM we rarely care about latency as the whole system is > > likely to struggle. Memcg ooms are much more likely. Having that many > > tasks in a memcg certainly requires a further partitioning so if > > configured properly the OOM latency shouldn't be visible much. But I am > > wondering whether the cgroup task iteration could use cond_resched while > > the global one would touch_nmi_watchdog for every N iterations. I might > > be missing something but I do not see any locking required outside of > > css_task_iter_*. > > Do you mean like that: I've had something like this (untested) in mind diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..37abc94abd2e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1167,10 +1167,14 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, for_each_mem_cgroup_tree(iter, memcg) { struct css_task_iter it; struct task_struct *task; + unsigned int i = 0 css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it); - while (!ret && (task = css_task_iter_next(&it))) + while (!ret && (task = css_task_iter_next(&it))) { ret = fn(task, arg); + if (++i % 1000) + cond_resched(); + } css_task_iter_end(&it); if (ret) { mem_cgroup_iter_break(memcg, iter); diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 1c485beb0b93..3bf2304ed20c 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -430,10 +430,14 @@ static void dump_tasks(struct oom_control *oc) mem_cgroup_scan_tasks(oc->memcg, dump_task, oc); else { struct task_struct *p; + unsigned int i = 0; rcu_read_lock(); - for_each_process(p) + for_each_process(p) { + if (++i % 1000) + touch_softlockup_watchdog(); dump_task(p, oc); + } rcu_read_unlock(); } } -- Michal Hocko SUSE Labs