From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CCD8C433E2 for ; Tue, 8 Sep 2020 14:28:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DFBCF206B5 for ; Tue, 8 Sep 2020 14:28:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="TcMeF22U" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFBCF206B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 44D496B0055; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FF166B005A; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ECF48E0001; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 0D97A6B0055 for ; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C5C7A824805A for ; Tue, 8 Sep 2020 14:28:52 +0000 (UTC) X-FDA: 77240125704.17.man13_0406b6a270d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 9BE8D18027F98 for ; Tue, 8 Sep 2020 14:25:11 +0000 (UTC) X-HE-Tag: man13_0406b6a270d5 X-Filterd-Recvd-Size: 6364 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Tue, 8 Sep 2020 14:25:10 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id m8so4861057pgi.3 for ; Tue, 08 Sep 2020 07:25:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qbyfb7IENIIZnzycuSlXwlchWqptmqZvtFIJsiTST2Q=; b=TcMeF22UNZIEg9d0gzz98TwFeIxyqfPvLagxomEInN8Jz1RMp7VDdtXA6EehKzjSG8 /5SlRkTXBSFEzI7pP7AjUsnj7qEH/1FwaqMYorliQ5tbjGHarxMvO8K2yyl3kx+Odxn2 3p+pVF9186L5mOvCvvk4+cuQxqF1uHjnnMNSMl/4X7ONNFqdgPB5cc4Rdoy/RJMqBX7L bEIWU1IN9uRC1oJ6e3XY9xep4JvDEaHn65KJdAK/LLiIaIQAFmKTXm5LoKEfec4Q8zPo Af1hYqV97gBqQ2vTs0XQpUYzWpFqhNlJegb6cSioxGUw8S/cvolkKosVgz5eZAOnnSmN NcTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qbyfb7IENIIZnzycuSlXwlchWqptmqZvtFIJsiTST2Q=; b=srnT08vT431akJaXG9V+039st1da71AWGn5ipEWDMavS+hqCa1aiRclltY5kH0jdZO vWBn/hByT4EnuR5VsA1CYsEn0+VJ6SnhpEMh3NV3Qv0qxjlKXsfBDCFrub6FyCuAtsbC MnyJzLU6qyDv1OOeYc4yYe5myz6rHvZx8/7CsbBaA3UaoNCz1k1LQoBMdMA/v7x9xH/w 5vg3YSQUzYbF4GzK17zeZTplcI6JkeoCFYEo6vZvBcUkmtpH4sqZSUbm/yjrZCERE00L sIX8XFspEEdKbI8YHIQYeih4DAMfo9Bd0fN04TIWRbedan499eZD+SEGysWLoCYllgL/ X2hw== X-Gm-Message-State: AOAM532gsqtUSeBa56gcVuv6bEAqT/4b2lOddTIwB52P8UNACdTa73uB PMtqlinba5ykLBAUsRTuhaGJmw== X-Google-Smtp-Source: ABdhPJxgxtlU8ucp+kFiZL0rAJPS9+qutAhqRqo5Pl2vMcjnCNUKCP13XLhDLlCxt6b4VxJLHDqNIQ== X-Received: by 2002:aa7:8e85:: with SMTP id a5mr23961248pfr.96.1599575109593; Tue, 08 Sep 2020 07:25:09 -0700 (PDT) Received: from localhost.localdomain ([103.136.220.67]) by smtp.gmail.com with ESMTPSA id v26sm6120436pgo.83.2020.09.08.07.25.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Sep 2020 07:25:08 -0700 (PDT) From: zangchunxin@bytedance.com To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chunxin Zang , Muchun Song Subject: [PATCH] mm/vmscan: fix infinite loop in drop_slab_node Date: Tue, 8 Sep 2020 22:24:56 +0800 Message-Id: <20200908142456.89626-1-zangchunxin@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Rspamd-Queue-Id: 9BE8D18027F98 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chunxin Zang On our server, there are about 10k memcg in one machine. They use memory very frequently. When I tigger drop caches=EF=BC=8Cthe process will infin= ite loop in drop_slab_node. There are two reasons: 1.We have too many memcgs, even though one object freed in one memcg, the sum of object is bigger than 10. 2.We spend a lot of time in traverse memcg once. So, the memcg who traversed at the first have been freed many objects. Traverse memcg nex= t time, the freed count bigger than 10 again. We can get the following info through 'ps': root:~# ps -aux | grep drop root 357956 ... R Aug25 21119854:55 echo 3 > /proc/sys/vm/drop_cach= es root 1771385 ... R Aug16 21146421:17 echo 3 > /proc/sys/vm/drop_cach= es root 1986319 ... R 18:56 117:27 echo 3 > /proc/sys/vm/drop_caches root 2002148 ... R Aug24 5720:39 echo 3 > /proc/sys/vm/drop_caches root 2564666 ... R 18:59 113:58 echo 3 > /proc/sys/vm/drop_caches root 2639347 ... R Sep03 2383:39 echo 3 > /proc/sys/vm/drop_caches root 3904747 ... R 03:35 993:31 echo 3 > /proc/sys/vm/drop_caches root 4016780 ... R Aug21 7882:18 echo 3 > /proc/sys/vm/drop_caches Use bpftrace follow 'freed' value in drop_slab_node: root:~# bpftrace -e 'kprobe:drop_slab_node+70 {@ret=3Dhist(reg("bp")); = }' Attaching 1 probe... ^B^C @ret: [64, 128) 1 | = | [128, 256) 28 | = | [256, 512) 107 |@ = | [512, 1K) 298 |@@@ = | [1K, 2K) 613 |@@@@@@@ = | [2K, 4K) 4435 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@= @| [4K, 8K) 442 |@@@@@ = | [8K, 16K) 299 |@@@ = | [16K, 32K) 100 |@ = | [32K, 64K) 139 |@ = | [64K, 128K) 56 | = | [128K, 256K) 26 | = | [256K, 512K) 2 | = | In one drop caches action, only traverse memcg once maybe is better. If user need more memory, they can do drop caches again. Signed-off-by: Chunxin Zang Signed-off-by: Muchun Song --- mm/vmscan.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b6d84326bdf2..9d8ee2ae5824 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -699,17 +699,12 @@ static unsigned long shrink_slab(gfp_t gfp_mask, in= t nid, =20 void drop_slab_node(int nid) { - unsigned long freed; + struct mem_cgroup *memcg =3D NULL; =20 + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); do { - struct mem_cgroup *memcg =3D NULL; - - freed =3D 0; - memcg =3D mem_cgroup_iter(NULL, NULL, NULL); - do { - freed +=3D shrink_slab(GFP_KERNEL, nid, memcg, 0); - } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL)) !=3D NULL); - } while (freed > 10); + shrink_slab(GFP_KERNEL, nid, memcg, 0); + } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL)) !=3D NULL); } =20 void drop_slab(void) --=20 2.11.0