From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3847C4727E for ; Wed, 30 Sep 2020 15:45:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 243D920739 for ; Wed, 30 Sep 2020 15:45:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NjAmdQm8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 243D920739 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3FD7B8E0003; Wed, 30 Sep 2020 11:45:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 387318E0001; Wed, 30 Sep 2020 11:45:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DA128E0003; Wed, 30 Sep 2020 11:45:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 015C58E0001 for ; Wed, 30 Sep 2020 11:45:32 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8869040FE for ; Wed, 30 Sep 2020 15:45:32 +0000 (UTC) X-FDA: 77320152504.07.scale48_2203d5827194 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 65AB41815560C for ; Wed, 30 Sep 2020 15:45:32 +0000 (UTC) X-HE-Tag: scale48_2203d5827194 X-Filterd-Recvd-Size: 8926 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 30 Sep 2020 15:45:30 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id k25so2021850ljk.0 for ; Wed, 30 Sep 2020 08:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WjGyYmsLAMjsI4Vd+JJ6aVnyc9ajioRKaKMFxKhxaik=; b=NjAmdQm8MXZaLnWlGnp/SJHX6VYj0DS/7Gez+83AyXBMoNvk+/jPA1SbFRqQcqtBwO Ok/o36HzVVkUY3ru2aZxBs+wGU4DRrhw+hiDOzyytPNaO5JSw/Wf6d8Rri3vItQ4G2EM /Ek+Q+OZpsDOEL19Frebd8dK6g78mhoubgOLsJkyY+EC+TnjsmNR7Gao6ZLzC/8UMC5l f5wnBiQ4/mjBsrjkH0LrsRRzwlK0f18AfRnxg+Pdie3iMZ6Orfb+K5jq2KEmj7u+r/fE dfZQL7UGluptFcNBHGODab7Xi3NvTFTVox6i6gTovbv4j5iaq+2W5lMbaA2aphOu5YS0 3uKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WjGyYmsLAMjsI4Vd+JJ6aVnyc9ajioRKaKMFxKhxaik=; b=PHbFk7eP2sda5+dGKIQSXWN2d+Ty8olemvybBGtX345JpnJW7y0xZ8gO9jhYw4Men3 93uAzkbTv4S/5n5C+yNoM3o3BN/620aMVgKtzLb0FvO2XJAekHDNe5TIdhulkr57se07 TjYh0tiyxzQVXvEUi/TotFYPQ0g1gqm8IWGE+JLbJpuBa4fGWuRCnda6Fn0zzx8ESB73 fFNNJAsx0JdmQxk8JlesZYNDgXvpOKDU/muMDmnubvXi7ZlFXnYuNU4qc/DQEeYhOtls YuWiVbgYDSCzI8oGKjY37fEuBh0pQTnZLQu2fekMrKfFnSXh9kgLe9BMnoN5avCYfkoP dIKw== X-Gm-Message-State: AOAM533xZdl5j2QXKOwnO5K4iRqZmBWkdh7GPoXpKLoz11ZKIHqk2KCb KvgXRhyt4I0ttt3EJq947UBweuCvRMg0o/Pe4y0BJg== X-Google-Smtp-Source: ABdhPJy7Kpt7Q45UX41ep+M5rkizj38eD5g+4Z6l7TCfVZ1MzlbY3Yz8+VarznH/8FMlSNQWRSVWMedxWqjPqxxv7Dk= X-Received: by 2002:a2e:98cf:: with SMTP id s15mr1169162ljj.446.1601480729013; Wed, 30 Sep 2020 08:45:29 -0700 (PDT) MIME-Version: 1.0 References: <20200909215752.1725525-1-shakeelb@google.com> <20200928210216.GA378894@cmpxchg.org> <20200929150444.GG2277@dhcp22.suse.cz> <20200929215341.GA408059@cmpxchg.org> In-Reply-To: <20200929215341.GA408059@cmpxchg.org> From: Shakeel Butt Date: Wed, 30 Sep 2020 08:45:17 -0700 Message-ID: Subject: Re: [PATCH] memcg: introduce per-memcg reclaim interface To: Johannes Weiner Cc: Michal Hocko , Roman Gushchin , Yang Shi , Greg Thelen , David Rientjes , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Andrew Morton , Linux MM , Cgroups , LKML Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 29, 2020 at 2:55 PM Johannes Weiner wrote: > > On Tue, Sep 29, 2020 at 05:04:44PM +0200, Michal Hocko wrote: > > On Mon 28-09-20 17:02:16, Johannes Weiner wrote: > > [...] > > > My take is that a proactive reclaim feature, whose goal is never to > > > thrash or punish but to keep the LRUs warm and the workingset trimmed, > > > would ideally have: > > > > > > - a pressure or size target specified by userspace but with > > > enforcement driven inside the kernel from the allocation path > > > > > > - the enforcement work NOT be done synchronously by the workload > > > (something I'd argue we want for *all* memory limits) > > > > > > - the enforcement work ACCOUNTED to the cgroup, though, since it's the > > > cgroup's memory allocations causing the work (again something I'd > > > argue we want in general) > > > > > > - a delegatable knob that is independent of setting the maximum size > > > of a container, as that expresses a different type of policy > > > > > > - if size target, self-limiting (ha) enforcement on a pressure > > > threshold or stop enforcement when the userspace component dies > > > > > > Thoughts? > > > > Agreed with above points. What do you think about > > http://lkml.kernel.org/r/20200922190859.GH12990@dhcp22.suse.cz. > > I definitely agree with what you wrote in this email for background > reclaim. Indeed, your description sounds like what I proposed in > https://lore.kernel.org/linux-mm/20200219181219.54356-1-hannes@cmpxchg.org/ > - what's missing from that patch is proper work attribution. > > > I assume that you do not want to override memory.high to implement > > this because that tends to be tricky from the configuration POV as > > you mentioned above. But a new limit (memory.middle for a lack of a > > better name) to define the background reclaim sounds like a good fit > > with above points. > > I can see that with a new memory.middle you could kind of sort of do > both - background reclaim and proactive reclaim. > > That said, I do see advantages in keeping them separate: > > 1. Background reclaim is essentially an allocation optimization that > we may want to provide per default, just like kswapd. > > Kswapd is tweakable of course, but I think actually few users do, > and it works pretty well out of the box. It would be nice to > provide the same thing on a per-cgroup basis per default and not > ask users to make decisions that we are generally better at making. > > 2. Proactive reclaim may actually be better configured through a > pressure threshold rather than a size target. > > As per above, the goal is not to be punitive or containing. The > goal is to keep the LRUs warm and move the colder pages to disk. > > But how aggressively do you run reclaim for this purpose? What > target value should a user write to such a memory.middle file? > > For one, it depends on the job. A batch job, or a less important > background job, may tolerate higher paging overhead than an > interactive job. That means more of its pages could be trimmed from > RAM and reloaded on-demand from disk. > > But also, it depends on the storage device. If you move a workload > from a machine with a slow disk to a machine with a fast disk, you > can page more data in the same amount of time. That means while > your workload tolerances stays the same, the faster the disk, the > more aggressively you can do reclaim and offload memory. > > So again, what should a user write to such a control file? > > Of course, you can approximate an optimal target size for the > workload. You can run a manual workingset analysis with page_idle, > damon, or similar, determine a hot/cold cutoff based on what you > know about the storage characteristics, then echo a number of pages > or a size target into a cgroup file and let kernel do the reclaim > accordingly. The drawbacks are that the kernel LRU may do a > different hot/cold classification than you did and evict the wrong > pages, the storage device latencies may vary based on overall IO > pattern, and two equally warm pages may have very different paging > overhead depending on whether readahead can avert a major fault or > not. So it's easy to overshoot the tolerance target and disrupt the > workload, or undershoot and have stale LRU data, waste memory etc. > > You can also do a feedback loop, where you guess an optimal size, > then adjust based on how much paging overhead the workload is > experiencing, i.e. memory pressure. The drawbacks are that you have > to monitor pressure closely and react quickly when the workload is > expanding, as it can be potentially sensitive to latencies in the > usec range. This can be tricky to do from userspace. > This is actually what we do in our production i.e. feedback loop to adjust the next iteration of proactive reclaim. We eliminated the IO or slow disk issues you mentioned by only focusing on anon memory and doing zswap. > So instead of asking users for a target size whose suitability > heavily depends on the kernel's LRU implementation, the readahead > code, the IO device's capability and general load, why not directly > ask the user for a pressure level that the workload is comfortable > with and which captures all of the above factors implicitly? Then > let the kernel do this feedback loop from a per-cgroup worker. I am assuming here by pressure level you are referring to the PSI like interface e.g. allowing the users to tell about their jobs that X amount of stalls in a fixed time window is tolerable. Seems promising though I would like flexibility for giving the resource to the per-cgroup worker. Are you planning to work on this or should I give it a try?