From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CEDDC43461 for ; Mon, 14 Sep 2020 15:02:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6AE9B20732 for ; Mon, 14 Sep 2020 15:02:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="KvOdDKg4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AE9B20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F267C6B0055; Mon, 14 Sep 2020 11:02:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED6AD6B005C; Mon, 14 Sep 2020 11:02:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC6C76B0062; Mon, 14 Sep 2020 11:02:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id C85906B0055 for ; Mon, 14 Sep 2020 11:02:44 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 7DED68249980 for ; Mon, 14 Sep 2020 15:02:44 +0000 (UTC) X-FDA: 77261983848.28.peace14_550a53227109 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 862FC6D76 for ; Mon, 14 Sep 2020 15:02:41 +0000 (UTC) X-HE-Tag: peace14_550a53227109 X-Filterd-Recvd-Size: 10784 Received: from mail-lf1-f67.google.com (mail-lf1-f67.google.com [209.85.167.67]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Sep 2020 15:02:32 +0000 (UTC) Received: by mail-lf1-f67.google.com with SMTP id w11so13809836lfn.2 for ; Mon, 14 Sep 2020 08:02:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sFuhovjKCTRsvGjSz5u+cehrwOvylz6xZAbCE+XaW8E=; b=KvOdDKg4RMxgB9XFyAiGfOkSS3HBU8cT12Rs7PjCawQ5zphgFBBw9+3ItxRQtbmfjc 7fSHNCpoLZH2dFzFQPLUab7rx5n7FdtGLHhm7FycJywuK0K0kc503aM8qkEdPaaCvdKi fpJtjpay00pckVJ1QZ7uVPcIQb8zAySoigeMKADQLvPXP5+r7HV16kHJAlHwjBK1d1fw K2T0N3GMpNGnJtJV3ly9fO7+hF6sPo454tpT2QxL3beuZBoFDDQSxB9BJMf3utXYY3Au OQOWMaFKvCqK4rctCQSYHitOItJKQWbE4kXP5AtNEOciHtL5kI5KIIb5E9GVicL2PeFK iJ1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sFuhovjKCTRsvGjSz5u+cehrwOvylz6xZAbCE+XaW8E=; b=n4uM0BSfxvV/t6tLmZOtCxCNs4yKIKH095MbIiTBC9waGmAk+YT6txpkBnANEqeuut sxsdWiQDXchDgHntFgoR2zFVfryaFlUxVyzRoREBSvwmAUCp3HMc+oMnIz+7nTTkVG+V 9EXMCQxXm2GDlHxJzCnirUQOa+1p8YQbYos0aL36T3eYF0VmxarwJl52cHIgVlju92b8 kRbmumRilNt+r/Jzq4uv+40Wca1GoYa374/xW6cWHrKkRit9mEwnov1JzhCXPl582Slj NHLadoj5vjvfpPWLuKJ0DhB+U5Rm69dtNMXkiCZ9jhkVGPtbApMeKxbMmzH+ioQ9/5fg HoWA== X-Gm-Message-State: AOAM530lNDbBwN10SDSNCWYppCgK+QfWvRy+tn0IRMXLC31BZM/zRSJB sBS2V6FgdymIgqDCg9xhiKGUU6ZMKtGc6DaSjfPTHA== X-Google-Smtp-Source: ABdhPJyF99pclW1+VYBNYug4H3Xq+MLOSPZ+/siOmcWJCahydqM80jnOb3rnf81dsTP554rLStZgVTBr7+D7ukdnCUk= X-Received: by 2002:a19:404:: with SMTP id 4mr4233912lfe.343.1600095746571; Mon, 14 Sep 2020 08:02:26 -0700 (PDT) MIME-Version: 1.0 References: <20200909152047.27905-1-zangchunxin@bytedance.com> <20200914093032.GG16999@dhcp22.suse.cz> <20200914134713.GS16999@dhcp22.suse.cz> In-Reply-To: <20200914134713.GS16999@dhcp22.suse.cz> From: Chunxin Zang Date: Mon, 14 Sep 2020 23:02:15 +0800 Message-ID: Subject: Re: [External] Re: [PATCH v2] mm/vmscan: fix infinite loop in drop_slab_node To: Michal Hocko Cc: Andrew Morton , Linux Memory Management List , LKML , Muchun Song Content-Type: multipart/alternative; boundary="00000000000094de7805af475254" X-Rspamd-Queue-Id: 862FC6D76 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --00000000000094de7805af475254 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Sep 14, 2020 at 9:47 PM Michal Hocko wrote: > On Mon 14-09-20 21:25:59, Chunxin Zang wrote: > > On Mon, Sep 14, 2020 at 5:30 PM Michal Hocko wrote: > > > > > The subject is misleading because this patch doesn't fix an infinite > > > loop, right? It just allows the userspace to interrupt the operation. > > > > > > > > Yes, so we are making a separate patch follow Vlastimil's > recommendations. > > Use double of threshold to end the loop. > > That still means the changelog needs an update > The patch is already merged in Linux-next branch. Can I update the changelog now? This is my first patch, please forgive me :) > > > On Thu, Sep 10, 2020 at 1:59 AM Vlastimil Babka wrote: > > > > From: Chunxin Zang > > > > > > > ... > > > - IMHO it's still worth to bail out in your scenario even without a > > > signal, e.g. > > > by the doubling of threshold. But it can be a separate patch. > > > Thanks! > > > ... > > > > > > > > On Wed 09-09-20 23:20:47, zangchunxin@bytedance.com wrote: > > > > From: Chunxin Zang > > > > > > > > On our server, there are about 10k memcg in one machine. They use > memory > > > > very frequently. When I tigger drop caches=EF=BC=8Cthe process will= infinite > loop > > > > in drop_slab_node. > > > > > > Is this really an infinite loop, or it just takes a lot of time to > > > process all the metadata in that setup? If this is really an infinite > > > loop then we should look at it. My current understanding is that the > > > operation would finish at some time it just takes painfully long to g= et > > > there. > > > > > > > Yes, it's really an infinite loop. Every loop spends a lot of time. I= n > > this time, > > memcg will alloc/free memory, so the next loop, the total of 'freed' > > always bigger than 10. > > I am still not sure I follow. Do you mean that there is somebody > constantly generating more objects to reclaim? > Yes, this is my meaning. :) > Maybe we are just not agreeing on the definition of an infinite loop but > in my book that means that the final condition can never be met. While a > busy adding new object might indeed cause drop caches to loop for a long > time this is to be expected from that interface as it is supposed to > drop all the cache and that can grow during the operation. > -- > Because I have 10k memcg , all of them are heavy users of memory. During each loop, there are always more than 10 reclaimable objects generating, so the condition is never met. The drop cache process has no chance to exit the loop. Although the purpose of the 'drop cache' interface is to release all caches, we still need a way to terminate it, e.g. in this case, the process took too long to run . root 357956 ... R Aug25 21119854:55 echo 3 > /proc/sys/vm/drop_caches --00000000000094de7805af475254 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Mon, Sep 14, 2020 at 9:47 PM Michal Ho= cko <mhocko@suse.com> wrote:
On Mon 14-09-20 2= 1:25:59, Chunxin Zang wrote:
> On Mon, Sep 14, 2020 at 5:30 PM Michal Hocko <mhocko@suse.com> wrote:
>
> > The subject is misleading because this patch doesn't fix an i= nfinite
> > loop, right? It just allows the userspace to interrupt the operat= ion.
> >
> >
> Yes,=C2=A0 so we are making a separate patch follow Vlastimil's re= commendations.
> Use double of threshold to end the loop.

That still means the changelog needs an update

The patch is already merged in Linux-next branch.=C2=A0 Can I updat= e the changelog now?=C2=A0=C2=A0
This is my first patch, please f= orgive me :)
=C2=A0

> On Thu, Sep 10, 2020 at 1:59 AM Vlastimil Babka <vbabka@suse.cz> wrote:
> > > From: Chunxin Zang <zangchunxin@bytedance.com>
> > >
> > ...
> > - IMHO it's still worth to bail out in your scenario even wit= hout a
> > signal, e.g.
> > by the doubling of threshold. But it can be a separate patch.
> > Thanks!
> > ...
>
>
>
> On Wed 09-09-20 23:20:47, zangchunxin@bytedance.com wrote:
> > > From: Chunxin Zang <zangchunxin@bytedance.com>
> > >
> > > On our server, there are about 10k memcg in one machine. The= y use memory
> > > very frequently. When I tigger drop caches=EF=BC=8Cthe proce= ss will infinite loop
> > > in drop_slab_node.
> >
> > Is this really an infinite loop, or it just takes a lot of time t= o
> > process all the metadata in that setup? If this is really an infi= nite
> > loop then we should look at it. My current understanding is that = the
> > operation would finish at some time it just takes painfully long = to get
> > there.
> >
>
> Yes,=C2=A0 it's really an infinite loop.=C2=A0 Every loop spends a= lot of time. In
> this time,
> memcg will alloc/free memory,=C2=A0 so the next loop, the total of=C2= =A0 'freed'
> always bigger than 10.

I am still not sure I follow. Do you mean that there is somebody
constantly generating more objects to reclaim?

Yes, this is my meaning. :)
=C2=A0
Maybe we are just not agreeing on the definition of an infinite loop but in my book that means that the final condition can never be met. While a busy adding new object might indeed cause drop caches to loop for a long time this is to be expected from that interface as it is supposed to
drop all the cache and that can grow during the operation.
--
=C2=A0
Because I have 10k memcg , all of them= are heavy users of memory. =C2=A0
During each loop, there are always mo= re than 10 reclaimable objects generating, so the
condition is never me= t. The drop cache process has no chance to exit the loop.
Although the p= urpose of the 'drop cache' interface is to release all caches, we s= till need a
way to terminate it, e.g. in this case, the process took to= o long to run .

=C2=A0 root =C2=A0357956 ... R =C2=A0 =C2=A0Aug25 21119854:55 ec= ho 3 > /proc/sys/vm/drop_caches
--00000000000094de7805af475254--