From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E89BCE77184 for ; Thu, 19 Dec 2024 07:57:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E4C06B0093; Thu, 19 Dec 2024 02:57:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 394736B0095; Thu, 19 Dec 2024 02:57:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25C586B0098; Thu, 19 Dec 2024 02:57:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 06B396B0093 for ; Thu, 19 Dec 2024 02:57:07 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A500EA1511 for ; Thu, 19 Dec 2024 07:57:06 +0000 (UTC) X-FDA: 82910952138.18.969E3CB Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by imf13.hostedemail.com (Postfix) with ESMTP id DBB7820003 for ; Thu, 19 Dec 2024 07:56:32 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=F4uekDB7; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf13.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734595009; a=rsa-sha256; cv=none; b=C+DrFVKU1R9355N8SalCl7fZU5n4hOKaOhM21fvgOXFW5ouWwaBn92ZOYJpajXZM6jeROL OMrq+Qnki6h1UmQrSQVOrW9cBOaGGCxYZZViqEVt4wKSCiWeuANdnuTpQgZj6VtuQ1wXNy HkvG2IYvKutJdsXOEx2r6qqwjVyLqN8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=F4uekDB7; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf13.hostedemail.com: domain of mhocko@suse.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734595009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JEx8BhRUkkgkswMHP6Lmd6KXutJJvSq42c8tbR5W3Eo=; b=UKzQeA2TZ8LjYidSsGogHBlJpkfREuDNsnRJMDG8iec0wu4DMv72VZzyUQjRiuy7hy+2Nj rdEbIFqFwFjbNnrvp+xMDOncUGfAmdLgLyEO0UGStrEbhE2MEnqVoqy8r22SnlGv1M4p8d 8PvOMsPg0khDBhsEcIvAip4TqcbmVJc= Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-aa6c0d1833eso96967166b.1 for ; Wed, 18 Dec 2024 23:57:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1734595023; x=1735199823; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=JEx8BhRUkkgkswMHP6Lmd6KXutJJvSq42c8tbR5W3Eo=; b=F4uekDB7jXeDkUaY/cdActsK/CNMaCSHcSnpIEeGQvF9Yfz9FamJeYSzWESGnwGxOv O69LAz/+0rxPKNx9pyySRfmSV1Y4hYUm22PQK0NvU45Br9LELpx/roPQye/MSfl55lgs t2tha7ZVKOZIE3wM3Ereb8CJqfZA20JtnEG85aNsgaIJnjjJnzd30KSj/eOvRsMPf3A8 coE3U/h5+s/MwPPLJnWTd+ltyofc/NMmGBhUZBVFcx5P6Xc0OCu2ZA8F3fF62BL/0QTw v0SGUCxjGYLksAVfW7DmzEHx0ssRp8mjUxylSOMBQk88ZkGMvB9l5DVuNvF9gfxI9K8R FSnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734595023; x=1735199823; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=JEx8BhRUkkgkswMHP6Lmd6KXutJJvSq42c8tbR5W3Eo=; b=ETsj7Zgn3fLeJOeThOhtMFk6LBCtl2OmcexKWsWtXM+PzX92W49aMrymBl28cyuIXa nobcTYfBkmeVMKKcNSmkjpc8aQxUrDGpnYeesD65dBEyoXXKrS0AqwfgBJGPfMdv6c5f 8fT65ga+jAxb9Wzo/5me9SextgDdRnRvnDzpgMld+LSuGe0dUbQdurYc+iYf5HmZD1b4 oj5jmBV/wTKlOu3G/j8bdjdZRN6K+Mq/E0pIZlADDmSzZkFBNlC+dZTE5mNXwKjvzxbg aYsvzN7NZ7THnjuavZWC41XLG9AxuNpz59R50YoV74HnUs4KhJC3p/Qu5APLE94gOf9t oxMA== X-Forwarded-Encrypted: i=1; AJvYcCXmPlwwESbUotMoc7p6gDbZdkRfM2zv7Sf4yztOV1Qj9HLL81NXfYPEdCEwa42EvJnehd2r0xXxJw==@kvack.org X-Gm-Message-State: AOJu0YyZp7iOoRKM0XVEauvwxL7WpDCxXjeANCuPUv6cCdBfqH4rWOUn COV+EDfENSDXCYC8Ve0DJI2zujkIoVrtmlpTxdRjdbrtlyYPj1ZPHBoX3ivAfAo= X-Gm-Gg: ASbGncvOwy6Kk7TeKBY7VdxkGIXiOq9qW93HNJkkztDh6Sfgwe3DB8PuLSi5inbaY1i v0tD7Nvgv9LeFles9he7x8nlQJmAx+MFKF7d759MbzX/JRcg8+M1fS/rWp0Ccf9Gt8B7RFmkeqJ QcJ/R0i+0ItGF3zNKDvimVaLvD2xUMXQquxNkrM8fdtwvtgMGoaPpaq8fBPFtxPNeM0RkMMOS3t 4RxaylsiwPDJ9h+wHqr1kLDB3pLPzDsDmvZwAK69K56nhYLR6l/zKyJKNYZGvXd X-Google-Smtp-Source: AGHT+IFObg+BLSYVoCnWrejO/CfDsPWEnBJkisYPPT/XL+9HTYPC53Xic1aTpZCg0MxeBKFnmi/9Og== X-Received: by 2002:a17:907:9555:b0:aab:a0d4:ec9f with SMTP id a640c23a62f3a-aabf4758f2amr406629166b.21.1734595023224; Wed, 18 Dec 2024 23:57:03 -0800 (PST) Received: from localhost (109-81-88-1.rct.o2.cz. [109.81.88.1]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aac0e89537dsm37406766b.54.2024.12.18.23.57.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 23:57:02 -0800 (PST) Date: Thu, 19 Dec 2024 08:57:01 +0100 From: Michal Hocko To: Chen Ridong Cc: Tejun Heo , akpm@linux-foundation.org, hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: Re: [PATCH v1] memcg: fix soft lockup in the OOM process Message-ID: References: <20241217121828.3219752-1-chenridong@huaweicloud.com> <872c5042-01d6-4ff3-94bc-8df94e1e941c@huaweicloud.com> <02f7d744-f123-4523-b170-c2062b5746c8@huaweicloud.com> <7d7b3c01-4977-41fa-a19c-4e6399117e8e@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7d7b3c01-4977-41fa-a19c-4e6399117e8e@huaweicloud.com> X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DBB7820003 X-Stat-Signature: nfphurqbrg9itwixu7qz3isnd1e4qswt X-HE-Tag: 1734594992-65636 X-HE-Meta: U2FsdGVkX1+e9TLwf7tP6XFxG7adJsnqUDkXQ3OqhqT36bQUT4M8/EejzJUMWxyhA0RjOLAifQ2QzNnBsTyXLpxGbGi3xHSAZp0kIB3TV9fzHKThlrRtUF8stV2Q8XzDdsIhE2BFrWHlX0gY6U6ZpV7GuC8KxZ1R7dXjE/+MOahEqxCyM/pJebI4r1DdDywA4+Ap1fvsmz2D8abyhFh0NOZJGuZaMwbxlJoAMOKUV4o9y8qnYAWniQNh9mw/1k5sqxRXzfUxLxBrXY0M7riLZOvo7OCuyfmF5+J5oKGaB2ShlAqmqE6Xu+K32W9EcpcwhSrUn6QiLtfS96eHDwJ3WbF4KTp70YIav0yPYecBEFyXfdvgIzzgI0WBrTH0Pv7ZBG/nvxTheuXJRKg9rPl543M5qfwQT984MZntdiwuKAWc0/YQ4wKMuTCGhCHC77GUZkf8ipG8fz7q/IFGg5Y1Xt6bqhCZ29J7wFHMteWBdIx9WC5pCkEgCE+CntcrB2BVSkKP0E5sK3GLcQsi7WnklqLhezIKEdj9qT4bff/fT1V01w+eR2rudVwNGZbBrHGOXpJ+li6muQ+wYYm8pknqFvlqauffZQ02lNUXjLJPRH9xSmugKbO5N5vTlO+Mi/fzHFsWzxoDDWNAd6JDaLgb7UyaTiUQTjGlK31JaYXbmRWzDKtb8qqo0Ek6+VP1OUa3v+BjyCZIIv8AbD5kFENYYMuTqyEdSHq+n1XvZHwksdn1OTJudzg0Gmx3oB1ceVNj2w7zdBrwZu4RgJmF1mvvhd4yqNOdciwQT4qQ5krFc/QfKeYVaIxtscKte+IQMhf9eFfmxlhWLi+7a/wpmLXAf0kvrQ09E0FzMj4QYbju0j+sqwzNTfCeB92fMYWyP+q4mctsR5gD0yrYObR+h6eKsBN7Cr/G/hhe/lSGaCxAdNZXkrZf84wMr/SjWHoYf0OqeclbKcuTZ7o1/55PcF2 tUYs1xIi rFPkfumKy+TrGRMnnnoqeGeQJ1jgLfX/pWztVrk8+5kGYdN7N6eNUPep+y5smPdpyxGeQgdjV9A4pqkk+fzk/VzwJqssi82LNTq7UWBb2ztYXOaFTKjknvRfRggx56AbHNRL1pkn7ofRxEnWocxAsT90/NJwNyrkld9HEAg3vd4vEyTz9UJJutVM6i1bq02UF5K8KkwlT5WJPF18SCxh41XmdNWeVkaAlXDuUlQm+3t0VLy4T7daceRR4jQWd4cvZEI5HDS4uR0/xE2bq2p1CKdJGeyap6xYjW/Pq0yfXDRrRWwS+a2TU32haDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000017, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu 19-12-24 09:27:52, Chen Ridong wrote: > > > On 2024/12/18 18:22, Michal Hocko wrote: > > On Wed 18-12-24 17:00:38, Chen Ridong wrote: > >> > >> > >> On 2024/12/18 15:56, Michal Hocko wrote: > >>> On Wed 18-12-24 15:44:34, Chen Ridong wrote: > >>>> > >>>> > >>>> On 2024/12/17 20:54, Michal Hocko wrote: > >>>>> On Tue 17-12-24 12:18:28, Chen Ridong wrote: > >>>>> [...] > >>>>>> diff --git a/mm/oom_kill.c b/mm/oom_kill.c > >>>>>> index 1c485beb0b93..14260381cccc 100644 > >>>>>> --- a/mm/oom_kill.c > >>>>>> +++ b/mm/oom_kill.c > >>>>>> @@ -390,6 +390,7 @@ static int dump_task(struct task_struct *p, void *arg) > >>>>>> if (!is_memcg_oom(oc) && !oom_cpuset_eligible(p, oc)) > >>>>>> return 0; > >>>>>> > >>>>>> + cond_resched(); > >>>>>> task = find_lock_task_mm(p); > >>>>>> if (!task) { > >>>>>> /* > >>>>> > >>>>> This is called from RCU read lock for the global OOM killer path and I > >>>>> do not think you can schedule there. I do not remember specifics of task > >>>>> traversal for crgoup path but I guess that you might need to silence the > >>>>> soft lockup detector instead or come up with a different iteration > >>>>> scheme. > >>>> > >>>> Thank you, Michal. > >>>> > >>>> I made a mistake. I added cond_resched in the mem_cgroup_scan_tasks > >>>> function below the fn, but after reconsideration, it may cause > >>>> unnecessary scheduling for other callers of mem_cgroup_scan_tasks. > >>>> Therefore, I moved it into the dump_task function. However, I missed the > >>>> RCU lock from the global OOM. > >>>> > >>>> I think we can use touch_nmi_watchdog in place of cond_resched, which > >>>> can silence the soft lockup detector. Do you think that is acceptable? > >>> > >>> It is certainly a way to go. Not the best one at that though. Maybe we > >>> need different solution for the global and for the memcg OOMs. During > >>> the global OOM we rarely care about latency as the whole system is > >>> likely to struggle. Memcg ooms are much more likely. Having that many > >>> tasks in a memcg certainly requires a further partitioning so if > >>> configured properly the OOM latency shouldn't be visible much. But I am > >>> wondering whether the cgroup task iteration could use cond_resched while > >>> the global one would touch_nmi_watchdog for every N iterations. I might > >>> be missing something but I do not see any locking required outside of > >>> css_task_iter_*. > >> > >> Do you mean like that: > > > > I've had something like this (untested) in mind > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 7b3503d12aaf..37abc94abd2e 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -1167,10 +1167,14 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, > > for_each_mem_cgroup_tree(iter, memcg) { > > struct css_task_iter it; > > struct task_struct *task; > > + unsigned int i = 0 > > > > css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it); > > - while (!ret && (task = css_task_iter_next(&it))) > > + while (!ret && (task = css_task_iter_next(&it))) { > > ret = fn(task, arg); > > + if (++i % 1000) > > + cond_resched(); > > + } > > css_task_iter_end(&it); > > if (ret) { > > mem_cgroup_iter_break(memcg, iter); > > Thank you for your patience. > > I had this idea in mind as well. > However, there are two considerations that led me to reconsider it: > > 1. I wasn't convinced about how we should call cond_resched every N > iterations. Should it be 1000 or 10000? Sure, there will likely not be any _right_ value. This is mostly to mitigate the overhead of cond_resched which is not completely free. Having a system with 1000 tasks is not completely uncommon and we do not really need cond_resched now. More importantly we can expect cond_resched will eventually go away with the PREEMPT_LAZY (or what is the current name of that) so I wouldn't overthink this. > 2. I don't think all callers of mem_cgroup_scan_tasks need cond_resched. > Only fn is expensive (e.g., dump_tasks), and it needs cond_resched. At > least, I have not encountered any other issue except except when fn is > dump_tasks. See above. I wouldn't really overthink this. -- Michal Hocko SUSE Labs