From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4B95C02183 for ; Fri, 17 Jan 2025 06:59:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C8106B0082; Fri, 17 Jan 2025 01:59:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 276E16B0083; Fri, 17 Jan 2025 01:59:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13F6F6B0085; Fri, 17 Jan 2025 01:59:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E96676B0082 for ; Fri, 17 Jan 2025 01:59:24 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 636811616CB for ; Fri, 17 Jan 2025 06:59:24 +0000 (UTC) X-FDA: 83016042648.28.82136EB Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf15.hostedemail.com (Postfix) with ESMTP id 23EC6A0006 for ; Fri, 17 Jan 2025 06:59:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf15.hostedemail.com: domain of chenridong@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=chenridong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737097162; a=rsa-sha256; cv=none; b=TLr6V3x5/lZtgPxulLlBvGO8kesBGdbGongCUgWgEoGv72csbd5MYLwdm/4fy61R2JFLvv arQ3mTc2I0wmJ3HshwC69tQI9/L71aWahLNFyRk9iUxSRU8ZfUmow/sbng17hLzDXyCZfP 9A/oEk3QiX4BEXL9Fu+1w5xpFQ86f84= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf15.hostedemail.com: domain of chenridong@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=chenridong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737097162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mHqcthQPpNBpuDy89YciDdr2cH2Nt9SewDq4X39ZKNU=; b=wPwPXolVIYbDp9DeB2FxJnqRudni1JilCT8bPkPqtM9mqoBpDcUkxLP4U8wxllh43QJ7nK g3b/KsmKlavRl929nfDrvFMyJLsyqdwvVD+zk4bkfS64J1//5EHrpf5hcB8z1pKLTsXmjJ h3L6GzaKHOdnjswcbrnT4M6QVYKKFWQ= Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4YZ9V31bHmz1W49p; Fri, 17 Jan 2025 14:55:23 +0800 (CST) Received: from kwepemd100013.china.huawei.com (unknown [7.221.188.163]) by mail.maildlp.com (Postfix) with ESMTPS id 69E3B180101; Fri, 17 Jan 2025 14:59:16 +0800 (CST) Received: from [10.67.109.79] (10.67.109.79) by kwepemd100013.china.huawei.com (7.221.188.163) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.34; Fri, 17 Jan 2025 14:59:15 +0800 Message-ID: Date: Fri, 17 Jan 2025 14:59:14 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3] memcg: fix soft lockup in the OOM process To: , Chen Ridong CC: Vlastimil Babka , Michal Hocko , Andrew Morton , , , , , , , , , , RCU , , , , References: <20241224025238.3768787-1-chenridong@huaweicloud.com> <1ea309c1-d0f8-4209-b0b0-e69ad4e986ae@suse.cz> <58caaa4f-cf78-4d0f-af31-8a9277b6ebf5@huaweicloud.com> <20250113194546.3de1af46fa7a668111909b63@linux-foundation.org> <0b6a3935-8b6c-4d11-bacc-31c1ba15b349@huaweicloud.com> <81294730-1cd9-4793-b886-9ababe29d071@paulmck-laptop> Content-Language: en-US From: chenridong In-Reply-To: <81294730-1cd9-4793-b886-9ababe29d071@paulmck-laptop> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.67.109.79] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemd100013.china.huawei.com (7.221.188.163) X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 23EC6A0006 X-Stat-Signature: nybqadeo9f18conbz547qj437wweor9q X-Rspam-User: X-HE-Tag: 1737097160-513796 X-HE-Meta: U2FsdGVkX1+Pj3ZevY3KFI2q94nVLB2T6q4HhA8U3gsB4hbX7XwlBmNWFCRydhBYbI9fGSXKfrgQC10snjRtARCs6Ot5wocVy+HANn+lR2kC7ALJVYKYjvJe5ONg2ez20sjihT5PfWZMB8d0SsPVo6atlpZZqfDcC3zkNgGypBNAq2MoMeUE4aUyOQLXxWsX95kuOyFe5KVuZPI1R4Ki3wDqI8kE0tn/WP4//mYNRjcT38jw7bJ294ZjzfQuKG2wHYOB/2phdxVWkeDUR0TjHOwz6bA7DI+HOR8xRxXmv2hG3sDsVBUwWSBMxsM0zuNfxN1kXAd9lRckk5UCzbo+nzeZ9swb1x2M37ZTxaiFifHgouJm1eqqgvGH3oP5bRYHPAww0staYTnVYVuBCgVsZrLt9EAW7fMUffM7++oqwolJ2j8B/JUbaZoxsQyRxlohTeds1TR+HQB+/TbjJ5lEvyyb6coywLL7pJHFaElVJQALp4ASfbvsKVVTsF+ul7Msp5v5ey+Iqr4bLdf3RRUrqPVMNJea2xu+JUjCcLmf03rAxz8EjR0Hfu+n3mJSiDRijKjY3wk4E1LRFFctQ+wwEsmxkfMkDEykGmNOCLbtqYPd03XKIcYHm49lJGjsPP8AKuZ6Y8HXdP3jj2+UWe0pgl2kq3PLJf/pB79Zvbt0K8FoNVe7RqbtM4dayuGp+eYluKnyqFDu3SSKcFMGwN8VOUBY1pYUpzlCwhIkuUI6FClVuek0viKKEt5KmpUSRiaX01PtaSJb5WynIDDvFiWmzp6lMfi1zad4g71e02tClvwkejZsaKwaxw27S9AaFeV6vbXSGTsq6aSlUAx3Sa2Tc/CqmUa2s/EVEV6xTKg6qjtIbsC/6OfpPKUT0oUo6xIZXXNdaccfVcqfVOar+PyyHrL3LRyg9luO9GQF2jzkyjDVQNi6taCIBEwwnwpI3MXW5jz4pD3Dn7KfKN11JrH +hazf0CU PHscc0bpf4/hk194v1B3sPy7QOSmlHZC1TgvZ+3XkcmvvScKEd5aWpfLPzoQDbCAcL96q3/g0wx4AfGhLebXSeqGwP01To2YfwxnXrWqrqve+ISfybp5RrRwhWatk/THB1ANxgk3YNGKSgjiNPHXIO1Q06SAnRqSKG87hYqOJ9cSup0PAF3mNgxgKnm3wKTXkOjW/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2025/1/15 2:42, Paul E. McKenney wrote: > On Tue, Jan 14, 2025 at 08:13:37PM +0800, Chen Ridong wrote: >> >> >> On 2025/1/14 17:20, Vlastimil Babka wrote: >>> On 1/14/25 09:40, Michal Hocko wrote: >>>> On Mon 13-01-25 19:45:46, Andrew Morton wrote: >>>>> On Mon, 13 Jan 2025 14:51:55 +0800 Chen Ridong wrote: >>>>> >>>>>>>> @@ -430,10 +431,15 @@ static void dump_tasks(struct oom_control *oc) >>>>>>>> mem_cgroup_scan_tasks(oc->memcg, dump_task, oc); >>>>>>>> else { >>>>>>>> struct task_struct *p; >>>>>>>> + int i = 0; >>>>>>>> >>>>>>>> rcu_read_lock(); >>>>>>>> - for_each_process(p) >>>>>>>> + for_each_process(p) { >>>>>>>> + /* Avoid potential softlockup warning */ >>>>>>>> + if ((++i & 1023) == 0) >>>>>>>> + touch_softlockup_watchdog(); >>>>>>> >>>>>>> This might suppress the soft lockup, but won't a rcu stall still be detected? >>>>>> >>>>>> Yes, rcu stall was still detected. >>> >>> "was" or "would be"? I thought only the memcg case was observed, or was that >>> some deliberate stress test of the global case? (or the pr_info() console >>> stress test mentioned earlier, but created outside of the oom code?) >>> >> >> It's not easy to reproduce for global OOM. Because the pr_info() console >> stress test can also lead to other softlockups or RCU warnings(not >> causeed by OOM process) because the whole system is struggling.However, >> if I add mdelay(1) in the dump_task() function (just to slow down >> dump_task, assuming this is slowed by pr_info()) and trigger a global >> OOM, RCU warnings can be observed. >> >> I think this can verify that global OOM can trigger RCU warnings in the >> specific scenarios. > > We do have a recently upstreamed rcutree.csd_lock_suppress_rcu_stall > kernel boot parameter that causes RCU CPU stall warnings to suppress > most of the output when there is an ongoing CSD-lock stall. > > Would it make sense to do something similar when the system is in OOM, > give or take the traditional difficulty of determining exactly when OOM > starts and ends? > > 1dd01c06506c ("rcu: Summarize RCU CPU stall warnings during CSD-lock stalls") > > Thanx, Paul > I prefer to just drop it. Unlike memcg OOM, global OOM doesn't usually happen. Although it 'verified' that the RCU warning can be observed, we haven't encountered it in practice. Besides, other RCU warnings may also be observed during global OOM, and it's difficult to circumvent all the warnings. Best regards, Ridong >>>>>> For global OOM, system is likely to struggle, do we have to do some >>>>>> works to suppress RCU detete? >>>>> >>>>> rcu_cpu_stall_reset()? >>>> >>>> Do we really care about those? The code to iterate over all processes >>>> under RCU is there (basically) since ever and yet we do not seem to have >>>> many reports of stalls? Chen's situation is specific to memcg OOM and >>>> touching the global case was mostly for consistency reasons. >>> >>> Then I'd rather not touch the global case then if it's theoretical? It's not >>> even exactly consistent, given it's a cond_resched() in the memcg code (that >>> can be eventually automatically removed once/if lazy preempt becomes the >>> sole implementation), but the touch_softlockup_watchdog() would remain, >>> while doing only half of the job? >> >>