From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85DE4C433DF for ; Wed, 26 Aug 2020 10:41:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F2AA220786 for ; Wed, 26 Aug 2020 10:41:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2AA220786 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7BC406B0003; Wed, 26 Aug 2020 06:41:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76AE86B0005; Wed, 26 Aug 2020 06:41:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6817D6B0008; Wed, 26 Aug 2020 06:41:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id 524D66B0003 for ; Wed, 26 Aug 2020 06:41:33 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0EFC2180AD802 for ; Wed, 26 Aug 2020 10:41:33 +0000 (UTC) X-FDA: 77192378466.21.fight56_4e1193f27063 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id D4AC4180442C3 for ; Wed, 26 Aug 2020 10:41:32 +0000 (UTC) X-HE-Tag: fight56_4e1193f27063 X-Filterd-Recvd-Size: 4038 Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com [47.88.44.36]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 26 Aug 2020 10:41:31 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R561e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04407;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0U6vGXYm_1598438478; Received: from xunleideMacBook-Pro.local(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0U6vGXYm_1598438478) by smtp.aliyun-inc.com(127.0.0.1); Wed, 26 Aug 2020 18:41:18 +0800 Reply-To: xlpang@linux.alibaba.com Subject: Re: [PATCH] mm: memcg: Fix memcg reclaim soft lockup To: Michal Hocko Cc: Johannes Weiner , Andrew Morton , Vladimir Davydov , linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <1598426822-93737-1-git-send-email-xlpang@linux.alibaba.com> <20200826081102.GM22869@dhcp22.suse.cz> From: xunlei Message-ID: <99efed0e-050a-e313-46ab-8fe6228839d5@linux.alibaba.com> Date: Wed, 26 Aug 2020 18:41:18 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <20200826081102.GM22869@dhcp22.suse.cz> Content-Type: text/plain; charset=gbk X-Rspamd-Queue-Id: D4AC4180442C3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2020/8/26 =CF=C2=CE=E74:11, Michal Hocko wrote: > On Wed 26-08-20 15:27:02, Xunlei Pang wrote: >> We've met softlockup with "CONFIG_PREEMPT_NONE=3Dy", when >> the target memcg doesn't have any reclaimable memory. >=20 > Do you have any scenario when this happens or is this some sort of a > test case? It can happen on tiny guest scenarios. >=20 >> It can be easily reproduced as below: >> watchdog: BUG: soft lockup - CPU#0 stuck for 111s![memcg_test:2204] >> CPU: 0 PID: 2204 Comm: memcg_test Not tainted 5.9.0-rc2+ #12 >> Call Trace: >> shrink_lruvec+0x49f/0x640 >> shrink_node+0x2a6/0x6f0 >> do_try_to_free_pages+0xe9/0x3e0 >> try_to_free_mem_cgroup_pages+0xef/0x1f0 >> try_charge+0x2c1/0x750 >> mem_cgroup_charge+0xd7/0x240 >> __add_to_page_cache_locked+0x2fd/0x370 >> add_to_page_cache_lru+0x4a/0xc0 >> pagecache_get_page+0x10b/0x2f0 >> filemap_fault+0x661/0xad0 >> ext4_filemap_fault+0x2c/0x40 >> __do_fault+0x4d/0xf9 >> handle_mm_fault+0x1080/0x1790 >> >> It only happens on our 1-vcpu instances, because there's no chance >> for oom reaper to run to reclaim the to-be-killed process. >> >> Add cond_resched() in such cases at the beginning of shrink_lruvec() >> to give up the cpu to others. >=20 > I do agree that we need a cond_resched but I cannot say I would like > this patch. The primary reason is that it doesn't catch all cases when > the memcg is not reclaimable. For example it wouldn't reschedule if the > memcg is protected by low/min. What do you think about this instead? >=20 > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 99e1796eb833..bbdc38b58cc5 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2617,6 +2617,8 @@ static void shrink_node_memcgs(pg_data_t *pgdat, = struct scan_control *sc) > =20 > mem_cgroup_calculate_protection(target_memcg, memcg); > =20 > + cond_resched(); > + > if (mem_cgroup_below_min(memcg)) { > /* > * Hard protection. >=20 > This should catch both cases. I even have a vague recollection that > somebody has proposed something in that direction but I cannot remember > what has happened with that patch. >=20 It's the endless "retry" in try_charge() that caused the softlockup, and I think mem_cgroup_protected() will eventually return MEMCG_PROT_NONE, and shrink_node_memcgs() will call shrink_lruvec() for memcg self-reclaim cases, so it's not a problem here. But adding cond_resched() at upper shrink_node_memcgs() may eliminate potential similar issues, I have no objection with this approach. I tested it and works well, will send v2 later.