From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2A71C43461 for ; Tue, 15 Sep 2020 04:24:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0AC2620771 for ; Tue, 15 Sep 2020 04:24:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="uGJMXjqh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AC2620771 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3711E90000C; Tue, 15 Sep 2020 00:24:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FA44900007; Tue, 15 Sep 2020 00:24:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C32B90000C; Tue, 15 Sep 2020 00:24:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 01AB0900007 for ; Tue, 15 Sep 2020 00:24:13 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B410B3626 for ; Tue, 15 Sep 2020 04:24:13 +0000 (UTC) X-FDA: 77264003586.15.war77_2405e3c2710e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 840731814B0C1 for ; Tue, 15 Sep 2020 04:24:13 +0000 (UTC) X-HE-Tag: war77_2405e3c2710e X-Filterd-Recvd-Size: 7974 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Sep 2020 04:24:12 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id u21so1555212ljl.6 for ; Mon, 14 Sep 2020 21:24:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=PBQszRvBcpddfdSGm+HLooLgRnTgwy/19XVpfGXCozI=; b=uGJMXjqhCGd3SZkZNGGSyYMd0UpuNBHn8EY307GGWrsURCr0IYnhfzsd2LsuILvVBD nhq31oHLmEj/x0agiZelOjAa82Zkloe/gbYmGcT3kE9IM0Hhrp6587dA0g/NkeDoK06y TJeLCow/s0tr43DsoN+ZLLg46X7CxtZfFmIz8eJlSxPha0T5zdywm9w5bs5swQtjLLYZ y3e6jC6HvnM8gdSm1CzmzsZgVa6ZxhNA91F84gVmB/ebZl4uJJ0I8eqi2HgnLrXTO4rb Mwr4SK3ZUcus4de2TKsJzq5HcVnLnxNfzHOtptW/vSuSjsxdyMmo9VQId0s1ZAAEWDD3 I4Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=PBQszRvBcpddfdSGm+HLooLgRnTgwy/19XVpfGXCozI=; b=oR1N05AvcRTreWagg7vVACr+yagzlTryxS6L+vfGGdkA2G1GiJPhYPD92GKx8aJt4n PUoT7s40odqsH9CsdNV3ElSIVMfJ0YIEBs3yB9jQOHn/cyVLyJmbJ3e73Z3PHm1Uo8tn QdE7bc5HwnyKZDbFeqCQjecgl4gKXH6S/iXLx4lfOoTNpkLWqUbg6MHkIYFX276ZF+vu +zPzZnLTGMymDpo2v1pIoie2dzRhC1TLnL9VO737ddo9TMlUz4Gcb/mh3QnI6Zaov22v 7GHP8F+5JNKclSkmjBkk5j20+fNU1taXWzDLrZlt9iIuOAEIJkcEt6ycOTHlN2NOBXXj m7Fg== X-Gm-Message-State: AOAM530R6P/aAZRjKxztQXyXMlf4uI8NJGUFtF0CGgu36J//XmOoeBJ9 9WS1fPj0Dgngx4TuqoRz9ofHlASjldmiesTYBfELNA== X-Google-Smtp-Source: ABdhPJy5vhhyUJ1ZCfBYqoNmC8bZgbT/qPaLlMhIvu1fHG+IKPeYfzky1LiSmZ5e/eIWs/dXZjluz6WJBbSRmjQHwgY= X-Received: by 2002:a2e:b8d1:: with SMTP id s17mr5818687ljp.222.1600143850965; Mon, 14 Sep 2020 21:24:10 -0700 (PDT) MIME-Version: 1.0 References: <20200909152047.27905-1-zangchunxin@bytedance.com> <20200914093032.GG16999@dhcp22.suse.cz> <20200914134713.GS16999@dhcp22.suse.cz> <20200914151736.GA16999@dhcp22.suse.cz> In-Reply-To: <20200914151736.GA16999@dhcp22.suse.cz> From: Chunxin Zang Date: Tue, 15 Sep 2020 12:23:59 +0800 Message-ID: Subject: Re: [External] Re: [PATCH v2] mm/vmscan: fix infinite loop in drop_slab_node To: Michal Hocko Cc: Andrew Morton , Linux Memory Management List , LKML , Muchun Song Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 840731814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Sep 14, 2020 at 11:17 PM Michal Hocko wrote: > > On Mon 14-09-20 23:02:15, Chunxin Zang wrote: > > On Mon, Sep 14, 2020 at 9:47 PM Michal Hocko wrote: > > > > > On Mon 14-09-20 21:25:59, Chunxin Zang wrote: > > > > On Mon, Sep 14, 2020 at 5:30 PM Michal Hocko wrot= e: > > > > > > > > > The subject is misleading because this patch doesn't fix an infin= ite > > > > > loop, right? It just allows the userspace to interrupt the operat= ion. > > > > > > > > > > > > > > Yes, so we are making a separate patch follow Vlastimil's > > > recommendations. > > > > Use double of threshold to end the loop. > > > > > > That still means the changelog needs an update > > > > > > > The patch is already merged in Linux-next branch. Can I update the > > changelog now? > > Yes. Andrew will refresh it. He doesn't have a git tree which would > prevent rewriting the patch. > > > This is my first patch, please forgive me :) > > No worries. The mm patch workflow is rather different from others. > > > > > On Thu, Sep 10, 2020 at 1:59 AM Vlastimil Babka wr= ote: > > > > > > From: Chunxin Zang > > > > > > > > > > > ... > > > > > - IMHO it's still worth to bail out in your scenario even without= a > > > > > signal, e.g. > > > > > by the doubling of threshold. But it can be a separate patch. > > > > > Thanks! > > > > > ... > > > > > > > > > > > > > > > > On Wed 09-09-20 23:20:47, zangchunxin@bytedance.com wrote: > > > > > > From: Chunxin Zang > > > > > > > > > > > > On our server, there are about 10k memcg in one machine. They u= se > > > memory > > > > > > very frequently. When I tigger drop caches=EF=BC=8Cthe process = will infinite > > > loop > > > > > > in drop_slab_node. > > > > > > > > > > Is this really an infinite loop, or it just takes a lot of time t= o > > > > > process all the metadata in that setup? If this is really an infi= nite > > > > > loop then we should look at it. My current understanding is that = the > > > > > operation would finish at some time it just takes painfully long = to get > > > > > there. > > > > > > > > > > > > > Yes, it's really an infinite loop. Every loop spends a lot of tim= e. In > > > > this time, > > > > memcg will alloc/free memory, so the next loop, the total of 'fre= ed' > > > > always bigger than 10. > > > > > > I am still not sure I follow. Do you mean that there is somebody > > > constantly generating more objects to reclaim? > > > > > > > Yes, this is my meaning. :) > > > > > > > Maybe we are just not agreeing on the definition of an infinite loop = but > > > in my book that means that the final condition can never be met. Whil= e a > > > busy adding new object might indeed cause drop caches to loop for a l= ong > > > time this is to be expected from that interface as it is supposed to > > > drop all the cache and that can grow during the operation. > > > -- > > > > > > > Because I have 10k memcg , all of them are heavy users of memory. > > During each loop, there are always more than 10 reclaimable objects > > generating, so the > > condition is never met. > > 10k or any number of memcgs shouldn't really make much of a difference. > Except for the time the scan adds. Fundamentally we are talking about > freed objects and whether they are on the global or per memcg lists > should result in a similar behavior. > > > The drop cache process has no chance to exit the > > loop. > > Although the purpose of the 'drop cache' interface is to release all > > caches, we still need a > > way to terminate it, e.g. in this case, the process took too long to ru= n . > > Yes, this is perfectly understandable. Having a bail out on fatal signal > is a completely reasonable thing to do. I am mostly confused by your > infinite loop claims and what the relation of this patch to it. I would > propose this wording instead > > " > We have observed that drop_caches can take a considerable amount of > time (). Especially when there are many memcgs involved > because they are adding an additional overhead. > > It is quite unfortunate that the operation cannot be interrupted by a > signal currently. Add a check for fatal signals into the main loop > so that userspace can control early bailout. > " I understand what you mean. Yes, you are right, The patch only provides a way for users to terminate the process, not to reduce time consumption, not improve the situation about loops. I will update the description of the patch in v3. Thanks so much :) > or something along those lines. > > > > > root 357956 ... R Aug25 21119854:55 echo 3 > /proc/sys/vm/drop_ca= ches > > -- > Michal Hocko > SUSE Labs Best wishes Chunxin