From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40B49C4727F for ; Tue, 22 Sep 2020 08:07:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CB97023A32 for ; Tue, 22 Sep 2020 08:07:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ce1QZHfP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB97023A32 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 035646B00E3; Tue, 22 Sep 2020 04:07:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F26F96B00E6; Tue, 22 Sep 2020 04:07:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E169D6B00E9; Tue, 22 Sep 2020 04:07:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id C63A26B00E3 for ; Tue, 22 Sep 2020 04:07:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 89EF8181AEF07 for ; Tue, 22 Sep 2020 08:07:08 +0000 (UTC) X-FDA: 77289966936.29.rose27_120f62a2714c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 6476F18086CC7 for ; Tue, 22 Sep 2020 08:07:08 +0000 (UTC) X-HE-Tag: rose27_120f62a2714c X-Filterd-Recvd-Size: 8895 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 22 Sep 2020 08:07:07 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id r25so18641096ioj.0 for ; Tue, 22 Sep 2020 01:07:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dI2SOSDVGxObAVcXLG7drnzqQGI8RgnuHUnXIEvu+us=; b=ce1QZHfPYr5zB923i5Vse/ct0CsPz2mKEtK2BbKoGHWENzw/eroXjVr04Ewvb2hGJS QftNRfFEtfhs6OKUpC2ikzJAlZsa5r8xvteqqh4LEdNI+fgTY/I/WgLziY0QpPKVLEkb GG21E8+R4F7QBCrhWRnnz9XX/jPxaMKK3QjvEXC/4OyyQ0XvGH3tTppXeBz54NRX2Clk P5Cj3fSL5NObe+IXN+Xaj5LGNyn3qX5hrv5tR12V1FQ4ExbmniCg+wgzL9F+GGvyzcj1 u9yPJ5iY6r126F19PpbOXJqj6zYd4X2giCbe9fUrSLDp/sXu3N7X8lCc5jGBkUojuD+w BG5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dI2SOSDVGxObAVcXLG7drnzqQGI8RgnuHUnXIEvu+us=; b=B99IoGFqbyC1HfN/j11MaHghwjgVvkWidlHqFZR5GZy3WMfleHrCmOwHDHVzpvjLyE y/I3ILy3EuRF65wCyyNPOt+XYn9IYseRwGXNQrp5dGk18VHiJmB45CX12V2s6PVUskWj cOPTNrw+uOwdsvR4xUBMsVu/1687hY0a+NAGX3JVX1l6+2Gx3gvcr6qAIVslUxQfIUE1 5cZuX/PZkWVWMwv8+0kCOQA35kDWOcgFl94FR5bkt4DGkp9XcqYexdg7GtcaDnH0IxhT lThxQ6C8y1pPv23XM6OCCX4A/ZfsFhn320DY/uAmYs/Vu+mFnENUXwskiW0TRzvftrh2 sTMA== X-Gm-Message-State: AOAM531r1f4p4v2VCbLXx9SghmLs0pFN0lxknwK3dFS6UOtgKna1SX27 5nttcLzhEls22OYS901P+Km8hlFa4gbGGmO5Xe4= X-Google-Smtp-Source: ABdhPJxagr4gGtyErOUVlTZvQUbPaIuBkq84e2wnan3wMmEaYz2iq80yB0igTr3V5Jt0SIm8xZQL9HWaX00/Y7JcFoo= X-Received: by 2002:a5e:dc08:: with SMTP id b8mr2463838iok.13.1600762027413; Tue, 22 Sep 2020 01:07:07 -0700 (PDT) MIME-Version: 1.0 References: <20200921080255.15505-1-zangchunxin@bytedance.com> <20200921081200.GE12990@dhcp22.suse.cz> <20200921110505.GH12990@dhcp22.suse.cz> <20200921113646.GJ12990@dhcp22.suse.cz> <20200922072733.GT12990@dhcp22.suse.cz> In-Reply-To: <20200922072733.GT12990@dhcp22.suse.cz> From: Yafang Shao Date: Tue, 22 Sep 2020 16:06:31 +0800 Message-ID: Subject: Re: [PATCH] mm/memcontrol: Add the drop_cache interface for cgroup v2 To: Michal Hocko Cc: zangchunxin@bytedance.com, Johannes Weiner , Vladimir Davydov , Andrew Morton , Tejun Heo , lizefan@huawei.com, Jonathan Corbet , Alexei Starovoitov , Daniel Borkmann , kafai@fb.com, Song Liu , Yonghong Song , andriin@fb.com, john.fastabend@gmail.com, kpsingh@chromium.org, Cgroups , linux-doc@vger.kernel.org, Linux MM , LKML , netdev , bpf@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 22, 2020 at 3:27 PM Michal Hocko wrote: > > On Tue 22-09-20 12:20:52, Yafang Shao wrote: > > On Mon, Sep 21, 2020 at 7:36 PM Michal Hocko wrote: > > > > > > On Mon 21-09-20 19:23:01, Yafang Shao wrote: > > > > On Mon, Sep 21, 2020 at 7:05 PM Michal Hocko wrote: > > > > > > > > > > On Mon 21-09-20 18:55:40, Yafang Shao wrote: > > > > > > On Mon, Sep 21, 2020 at 4:12 PM Michal Hocko wrote: > > > > > > > > > > > > > > On Mon 21-09-20 16:02:55, zangchunxin@bytedance.com wrote: > > > > > > > > From: Chunxin Zang > > > > > > > > > > > > > > > > In the cgroup v1, we have 'force_mepty' interface. This is very > > > > > > > > useful for userspace to actively release memory. But the cgroup > > > > > > > > v2 does not. > > > > > > > > > > > > > > > > This patch reuse cgroup v1's function, but have a new name for > > > > > > > > the interface. Because I think 'drop_cache' may be is easier to > > > > > > > > understand :) > > > > > > > > > > > > > > This should really explain a usecase. Global drop_caches is a terrible > > > > > > > interface and it has caused many problems in the past. People have > > > > > > > learned to use it as a remedy to any problem they might see and cause > > > > > > > other problems without realizing that. This is the reason why we even > > > > > > > log each attempt to drop caches. > > > > > > > > > > > > > > I would rather not repeat the same mistake on the memcg level unless > > > > > > > there is a very strong reason for it. > > > > > > > > > > > > > > > > > > > I think we'd better add these comments above the function > > > > > > mem_cgroup_force_empty() to explain why we don't want to expose this > > > > > > interface in cgroup2, otherwise people will continue to send this > > > > > > proposal without any strong reason. > > > > > > > > > > I do not mind people sending this proposal. "V1 used to have an > > > > > interface, we need it in v2 as well" is not really viable without > > > > > providing more reasoning on the specific usecase. > > > > > > > > > > _Any_ patch should have a proper justification. This is nothing really > > > > > new to the process and I am wondering why this is coming as a surprise. > > > > > > > > > > > > > Container users always want to drop cache in a specific container, > > > > because they used to use drop_caches to fix memory pressure issues. > > > > > > This is exactly the kind of problems we have seen in the past. There > > > should be zero reason to addre potential reclaim problems by dropping > > > page cache on the floor. There is a huge cargo cult about this > > > procedure and I have seen numerous reports when people complained about > > > performance afterwards just to learn that the dropped page cache was one > > > of the resons for that. > > > > > > > Although drop_caches can cause some unexpected issues, it could also > > > > fix some issues. > > > > > > "Some issues" is way too general. We really want to learn about those > > > issues and address them properly. > > > > > > > One use case in our production environment is that some of our tasks > > become very latency sensitive from 7am to 10am, so before these tasks > > become active we will use drop_caches to drop page caches generated by > > other tasks at night to avoid these tasks triggering direct reclaim. > > > > The best way to do it is to fix the latency in direct reclaim, but it > > will take great effort. > > What is the latency triggered by the memory reclaim? It should be mostly > a clean page cache right as drop_caches only drops clean pages. Or is > this more about [id]cache? Do you have any profiles where is the time > spent? > Yes, we have analyzed the issues in the direct reclaim, but that is not the point. The point is that each case may take us several days to analyze, while the user can't wait, so they will use drop_caches to workaround it until we find the solution. > > while drop_caches give us an easier way to achieve the same goal. > > It papers over real problems and that is my earlier comment about. > > > IOW, drop_caches give the users an option to achieve their goal before > > they find a better solution. > > You can achieve the same by a different configuration already. You can > isolate your page cache hungry overnight (likely a backup) workload into > its own memcg. You can either use an aggressive high limit during the > run or simply reduce the high/max limit after the work is done. > > If you cannot isolate that easily then you can lower high limit > temporarily before your peak workload. > Right, there're many better solutions if you have _enough_ time. But as I described above, the user needs to fix it ASAP before you find a better solution. > Really throwing all the page cache away just to prepare for a peak > workload sounds like a bad idea to me. You are potentially throwing > data that you have to spend time to refault again. If some tasks work at night but idle at day, while the others work at day but idle at night, it is not bad to drop all caches when you switch the workload. It may be useless to drop these caches, but these caches are really useless for the next workload. -- Thanks Yafang