From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B99FAC4363A for ; Mon, 5 Oct 2020 15:39:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 49D082100A for ; Mon, 5 Oct 2020 15:39:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49D082100A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=canonical.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 487368E0003; Mon, 5 Oct 2020 11:39:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 437A38E0001; Mon, 5 Oct 2020 11:39:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34CB38E0003; Mon, 5 Oct 2020 11:39:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 0AAF38E0001 for ; Mon, 5 Oct 2020 11:39:53 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 83C483629 for ; Mon, 5 Oct 2020 15:39:53 +0000 (UTC) X-FDA: 77338282266.09.title64_570cdb8271bf Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 33725180AD811 for ; Mon, 5 Oct 2020 15:39:53 +0000 (UTC) X-HE-Tag: title64_570cdb8271bf X-Filterd-Recvd-Size: 5787 Received: from youngberry.canonical.com (youngberry.canonical.com [91.189.89.112]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Mon, 5 Oct 2020 15:39:52 +0000 (UTC) Received: from mail-wr1-f71.google.com ([209.85.221.71]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kPSac-0001Ru-S7 for linux-mm@kvack.org; Mon, 05 Oct 2020 15:39:50 +0000 Received: by mail-wr1-f71.google.com with SMTP id v5so4191696wrr.0 for ; Mon, 05 Oct 2020 08:39:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=mM+mXfJfYWo+rDTzms1HSoE47Buw2ePWt0ypNfOv5eA=; b=e1z2nn9+bm6KczsqsoWOlA1qCijwi6lAWgSjM/hm+FwGLY3dcxsAo+tORg7p7ByJAJ BUCxoIB7NLgTPHtkxXIgzOgQOaE4zSKmsAw55aGXb+k451mcbGliikjJ4v1iyhBJWfeG ZtA39RiNENcAS9GywnsJ1kYLAKDS13jolK20qXABXMT3DQ55Yg4i/4pYA6kveIUgU1iu KKsUlQEVo+FpNawLt1DlpTlu9SZk2SNzRvCvLct6Sk/VLvSe8WrSIt1gEEzxn35LMhph BAYCr+0ymynVD5WbkOghJ9A83RaX6SxwTNo2kr/9eNHhWU2dsbPfPABetiSvapWiFURa 7k1g== X-Gm-Message-State: AOAM531vqLW2zCiYsvflMcBSfWgjmCK83SLrTMnakVD2vs1JLC8FMwsg NE9vzPqzvME+c+IbvAUs4QfDLlgXFGGFDN95WE5TDlEGOS8SEEKy9V267gAysQqdhEhR+J37DT6 eCNt7L2rUwZfpO/sUFcXXtadUCTZ0 X-Received: by 2002:adf:f6cd:: with SMTP id y13mr18239088wrp.161.1601912390435; Mon, 05 Oct 2020 08:39:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyyR5jOPYrgRLM5RUrs8DlVxB9ydRU8TU1jx6BFSljPHmzKZvGegdovcj80QusTJb5Aun56xw== X-Received: by 2002:adf:f6cd:: with SMTP id y13mr18239067wrp.161.1601912390139; Mon, 05 Oct 2020 08:39:50 -0700 (PDT) Received: from localhost (host-79-36-133-218.retail.telecomitalia.it. [79.36.133.218]) by smtp.gmail.com with ESMTPSA id c1sm438226wru.49.2020.10.05.08.39.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 08:39:49 -0700 (PDT) Date: Mon, 5 Oct 2020 17:39:48 +0200 From: Andrea Righi To: Chris Down Cc: Michal Hocko , Vladimir Davydov , Li Zefan , Tejun Heo , Johannes Weiner , Andrew Morton , Luigi Semenzato , "Rafael J . Wysocki" , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org Subject: Re: [PATCH RFC v2] Opportunistic memory reclaim Message-ID: <20201005153948.GC783944@xps-13-7390> References: <20201005081313.732745-1-andrea.righi@canonical.com> <20201005112555.GA108347@chrisdown.name> <20201005135130.GA850459@xps-13-7390> <20201005144612.GB108347@chrisdown.name> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201005144612.GB108347@chrisdown.name> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 05, 2020 at 03:46:12PM +0100, Chris Down wrote: > Andrea Righi writes: > > senpai is focused at estimating the ideal memory requirements without > > affecting performance. And this covers the use case about reducing > > memory footprint. > > > > In my specific use-case (hibernation) I would let the system use as much > > memory as possible if it's doing any activity (reclaiming memory only > > when the kernel decides that it needs to reclaim memory) and apply a > > more aggressive memory reclaiming policy when the system is mostly idle. > > From this description, I don't see any reason why it needs to be implemented > in kernel space. All of that information is available to userspace, and all > of the knobs are there. > > As it is I'm afraid of the "only when the system is mostly idle" comment, > because it's usually after such periods that applications need to do large > retrievals, and now they're going to be in slowpath (eg. periodic jobs). True, but with memory.high there's the risk to trash some applications badly if I'm not reacting fast at increasing memory.high. However, something that I could definitely want to try is to move all the memory hogs to a cgroup, set memory.high to a very small value and then immediately set it back to 'max'. The effect should be pretty much the same as calling shrink_all_memory(), that is what I'm doing with my memory.swap.reclaim. > > Such tradeoffs for a specific situation might be fine in userspace as a > distribution maintainer, but codifying them in the kernel seems premature to > me, especially for a knob which we will have to maintain forever onwards. > > > I could probably implement this behavior adjusting memory.high > > dynamically, like senpai, but I'm worried about potential sudden large > > allocations that may require to respond faster at increasing > > memory.high. I think the user-space triggered memory reclaim approach is > > a safer solution from this perspective. > > Have you seen Shakeel's recent "per-memcg reclaim interface" patches? I > suspect they may help you there. Yes, Michal pointed out to me his work, it's basically the same approach that I'm using. I started this work with a patch that was hibernation specific (https://lore.kernel.org/lkml/20200601160636.148346-1-andrea.righi@canonical.com/); this v2 was the natural evolution of my previous work and I didn't notice that something similar has been posted in the meantime. Anyway, I already contacted Shakeel, so we won't duplicate the efforts in the future. :) Thanks for your feedback! -Andrea