From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BC33C433F5 for ; Thu, 5 May 2022 17:22:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A0F86B0071; Thu, 5 May 2022 13:22:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 527A46B0073; Thu, 5 May 2022 13:22:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C8706B0074; Thu, 5 May 2022 13:22:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 265C86B0071 for ; Thu, 5 May 2022 13:22:32 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 00D6860ADA for ; Thu, 5 May 2022 17:22:31 +0000 (UTC) X-FDA: 79432358544.26.DF66086 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) by imf16.hostedemail.com (Postfix) with ESMTP id C3EC51800A5 for ; Thu, 5 May 2022 17:22:24 +0000 (UTC) Received: by mail-io1-f51.google.com with SMTP id e194so5436824iof.11 for ; Thu, 05 May 2022 10:22:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gQnXS3VNRKp8D+Sy399ehEMoC4EMmzEQsUoIIjKOfiE=; b=d4JI27qlk5SZDB+9Hby0gjWbMNH+sVdVmxwOX2zgw/Nk7my7cLkbgm7NZoWj3XkB2y KMO+TPuio5uiGWmqhfq9JzL+kIAyxEV3CeexwH9g4HO5zMfYMiruSp57kZOAR2amZLTC GcBdeVLOjtDjVr7yyTbR8rZ+DVGSPAPFDnomvPCZ9mHcu/Afwycbkz61Ftk4qnUiyQhN mMnI0bEqP2Ew/G4Uk+7sOBKSQ9Ph0+ImEgo/R6YcJPMRSvmNCHT9kT45UYsJkhM+atpn EW9RhuudWO8MrVWpLgF0ALWhk7tG9FvWqcoApblpA0K57kKKGSLbl3ntzgB/iTHsrmsh Onfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gQnXS3VNRKp8D+Sy399ehEMoC4EMmzEQsUoIIjKOfiE=; b=4jhBbCaFWmBd0frE9LY+IQEGclNAiGq2lNnLAjVW2gOK+Xlln3H9APWiazNJQOaPAT YXwYNcP1AhjeBgeiFpEUnYCQl8uoMtHWwCv9dZ8At/UdDuGWyII94xyrmFI8daqbXrIV QOyPDDzUhY8A3Nb5uAE9Znt0kX1OynGRvWzX7FfdzIZ7PcCl16TeeAyTCTav1ZojHDsZ nSmsoH7y4aBJEjbtm2WeJr1EJdj+t7IG2WP74gfFFM7Y9dwjuq/7bS30F6qDk8Z6cCay rqzlEV5axFZP3x+7xUs3ndawvSx4O22SVid558XH03kx+adOQlTROIruKTsCUEsh1c8/ zw+Q== X-Gm-Message-State: AOAM532I2FYsZFmJBe2kkhn/b69HdvOt+sKZqtgCLQlLOZMfe7L8pnBo rfyCVVcGKwtt4nWcNB0qrfoGqVZjxjM6pPOEN+461w== X-Google-Smtp-Source: ABdhPJyU/TVy1a54cerLD0Mz6JYkMl67Widk270/OlZuuc4nwwznKjz3b6erYzakr0XGNIeYbGT0Rj8AoZjGpmSC7lw= X-Received: by 2002:a05:6638:1392:b0:32b:8496:2c83 with SMTP id w18-20020a056638139200b0032b84962c83mr6346025jad.136.1651771349324; Thu, 05 May 2022 10:22:29 -0700 (PDT) MIME-Version: 1.0 References: <20220505121329.GA32827@us192.sjc.aristanetworks.com> In-Reply-To: From: Ganesan Rajagopal Date: Thu, 5 May 2022 22:51:52 +0530 Message-ID: Subject: Re: [PATCH] mm/memcontrol: Export memcg->watermark via sysfs for v2 memcg To: Shakeel Butt Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Cgroups , Linux MM Content-Type: multipart/alternative; boundary="00000000000086f6ec05de46fc3b" X-Rspamd-Queue-Id: C3EC51800A5 X-Stat-Signature: 9t1owhedepgd7aa49p9nph53qchrddiw X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=arista.com header.s=google header.b=d4JI27ql; spf=none (imf16.hostedemail.com: domain of rganesan@arista.com has no SPF policy when checking 209.85.166.51) smtp.mailfrom=rganesan@arista.com; dmarc=pass (policy=reject) header.from=arista.com X-Rspamd-Server: rspam09 X-HE-Tag: 1651771344-915785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000102, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --00000000000086f6ec05de46fc3b Content-Type: text/plain; charset="UTF-8" On Thu, May 5, 2022 at 9:42 PM Shakeel Butt wrote: > On Thu, May 5, 2022 at 5:13 AM Ganesan Rajagopal > wrote: > > > > v1 memcg exports memcg->watermark as "memory.mem_usage_in_bytes" in > > *max_usage_in_bytes > Oops, thanks for the correction. > > sysfs. This is missing for v2 memcg though "memory.current" is exported. > > There is no other easy way of getting this information in Linux. > > getrsuage() returns ru_maxrss but that's the max RSS of a single process > > instead of the aggregated max RSS of all the processes. Hence, expose > > memcg->watermark as "memory.watermark" for v2 memcg. > > > > Signed-off-by: Ganesan Rajagopal > > Can you please explain the use-case for which you need this metric? > Also note that this is not really an aggregated RSS of all the > processes in the cgroup. So, do you want max RSS or max charge and for > what use-case? > We run a lot of automated tests when building our software and used to run into OOM scenarios when the tests run unbounded. We use this metric to heuristically limit how many tests can run in parallel using per test historical data. I understand this isn't really aggregated RSS, max charge works. We just need some metric to account for the peak memory usage. We don't need it to be super accurate because there's significant variance between test runs anyway. We conservatively use the historical max to limit parallelism. Since this metric is not exposed in v2 memcg, the only alternative is to poll "memory.current" which would be quite inefficient and grossly inaccurate. Ganesan --00000000000086f6ec05de46fc3b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, May 5, 2022 at 9:42 PM Shakeel But= t <shakeelb@google.com> wr= ote:
On Thu, May 5, 2022 at 5:13 AM Ganesan Rajagop= al <rganesan@ar= ista.com> wrote:
>
> v1 memcg exports memcg->watermark as "memory.mem_usage_in_byte= s" in

*max_usage_in_bytes

Oops,= thanks for the correction.
> sysfs. This is missing for v2 memcg t= hough "memory.current" is exported.
> There is no other easy way of getting this information in Linux.
> getrsuage() returns ru_maxrss but that's the max RSS of a single p= rocess
> instead of the aggregated max RSS of all the processes. Hence, expose<= br> > memcg->watermark as "memory.watermark" for v2 memcg.
>
> Signed-off-by: Ganesan Rajagopal <rganesan@arista.com>

Can you please explain the use-case for which you need this metric?
Also note that this is not really an aggregated RSS of all the
processes in the cgroup. So, do you want max RSS or max charge and for
what use-case?

We ru= n a lot of automated tests when building our software and used to=C2=A0
run into OOM scenarios when the tests run unbounded. W= e use this metric=C2=A0
to heuristically limit how = many tests can=C2=A0run in parallel using per test
= historical data.=C2=A0

I understand this isn't really aggregated RSS, max charge works. We ju= st=C2=A0
need some metric to account for the peak m= emory usage.=C2=A0 We don't need=C2=A0
it to be= super accurate because there's significant variance between test=C2=A0=
runs anyway. We conservatively use the historical = max to limit parallelism.

Since this metric is not exposed in v2 memcg, the only alternative is t= o=C2=A0
poll "memory.current" which would= be quite inefficient and grossly=C2=A0
inaccurate.=

Ganesan

=C2=A0
--00000000000086f6ec05de46fc3b--