From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A8BFE7F140 for ; Tue, 26 Sep 2023 22:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3E4C8D005A; Tue, 26 Sep 2023 18:14:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DEE568D0002; Tue, 26 Sep 2023 18:14:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB6C88D005A; Tue, 26 Sep 2023 18:14:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B7ED58D0002 for ; Tue, 26 Sep 2023 18:14:18 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 87DC5120842 for ; Tue, 26 Sep 2023 22:14:18 +0000 (UTC) X-FDA: 81280152996.01.694B511 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf07.hostedemail.com (Postfix) with ESMTP id 7F45640021 for ; Tue, 26 Sep 2023 22:14:16 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=JbtZ0AuR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf07.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695766456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V6U1BTUAT6DClxkN/2UVfkrSLiqNT4sEy4cS5lEl9ps=; b=wKIikSxKdKy5qQyR4mDe4ZCCM46Qwm5TqQu3c/5wnFiWN4OOxlOBtd2t+NQbxpVvnOkhZ1 17FkuQw+zI9AeCsjPPxo8q38AhA19Rc+ABPpsGGHu9pcXALACbQfg4UxJPtFZNyiR+2KRy 4T6bIUyisPzd//Z1QjMkm/2AOqqejFc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=JbtZ0AuR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf07.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695766456; a=rsa-sha256; cv=none; b=WQraKgQc0YrsZFdMxNx2ZFmKtHGA5DyQLdKiWNqvky846qm9wF7RfMhWLPegYw0y1noi64 G7pKioCNMaLOSKEXFvJdK+F8DJFDXwTyhlVTOZQhf+bL6Nf3a59OHnOvK5dmPyoAU54WIk 0+HMPyGqgM6FZWWG1jf3W6Z25Kfi9n4= Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-4180adafdc6so47318411cf.2 for ; Tue, 26 Sep 2023 15:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1695766455; x=1696371255; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=V6U1BTUAT6DClxkN/2UVfkrSLiqNT4sEy4cS5lEl9ps=; b=JbtZ0AuRXiG9TgI527iMXQ7HngF430rBKYwXjWYrbtriaK/+X4WXu8w8fIHgpZGAnX 8FwlkMbpCZ68bCiPGMytNOlKeRvfbs40CzJru9kNlzxn5O99T+0vJEA9BNWMKLrdgB/P vS+Jqahe/okcHLU9GTvgsgPp6LAh55r9QJvAsHCYDAUp8DOEE4XsDSr+ms2r1kFxkxJR UhrPxCHIoDTHEtZ+Mj4Sge4PJ2zZkjmmQj6MKSwD4KuVVQl/3JFRUu9SUxAzP8NV35HU Wg6s6P0zirPaJ/sydt8uKIO35ix9cCWL8oWvTfT0Kza2Qagj5J+zTYrg3JglhBRVTlXo uBrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695766455; x=1696371255; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V6U1BTUAT6DClxkN/2UVfkrSLiqNT4sEy4cS5lEl9ps=; b=sLFfECgbQhOcqsskxmpVY2ae3RYwrIk3GpjA76GnYJD2Bea8LbjCupiAFJButCk6fB dx9IkmNekxsJl6FQht2XlAXhIGF2kOXkTjEXfvyYWjxavwFzZW0Yf43EhGOU9vGWTpTx L/5Rt4vWexVVkIdf/HEB8sDYdWINYW5hiGM5tlG9Lm+E2YtdIy5B5dBBgx2+E1BaPVLO vQ0q8V9iqOOkeBYIxgqnZ1VnNBqZ3HS/bSmMDu0JRVlSYSC19vOkwwyAOYwzF5YTtMQF TmPh8X9syW1oR/JwTJw6IAnisVuFWcEfbIU9IPFjPI+Zi6+cOoG3ci/tgwE3SBAVEf0N WMNA== X-Gm-Message-State: AOJu0YxTc7kkEOHgCin/J38dhBFLVjnzRjDU/JkXWZdHOILMNxbyVjhL 65r6ogFd2Z5M9Mph/uUD1cRf5w== X-Google-Smtp-Source: AGHT+IHuFcm42Gy+fzyBeHYAYQ1YUWMoy9PZVM4+bSjc4g9c6JzrZSykDY5b9grfk1BkFubXWT3+Xw== X-Received: by 2002:ac8:5796:0:b0:412:1e4c:e858 with SMTP id v22-20020ac85796000000b004121e4ce858mr157531qta.36.1695766455534; Tue, 26 Sep 2023 15:14:15 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:ba06]) by smtp.gmail.com with ESMTPSA id br22-20020a05622a1e1600b004108c610d08sm4992236qtb.32.2023.09.26.15.14.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Sep 2023 15:14:15 -0700 (PDT) Date: Tue, 26 Sep 2023 18:14:14 -0400 From: Johannes Weiner To: Frank van der Linden Cc: Nhat Pham , akpm@linux-foundation.org, riel@surriel.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, tj@kernel.org, lizefan.x@bytedance.com, shuah@kernel.org, mike.kravetz@oracle.com, yosryahmed@google.com, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: Re: [PATCH 0/2] hugetlb memcg accounting Message-ID: <20230926221414.GD348484@cmpxchg.org> References: <20230926194949.2637078-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 7F45640021 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: uncggoupsicum9kshnjmgf85x8iikgyr X-HE-Tag: 1695766456-483789 X-HE-Meta: U2FsdGVkX18stk+4y+YdoDn3z7xRdf4SLc4Yl3WdeMbNwlme56H04Pouh3re2X+GMSNgTblzXydTP4gzvS0etretmi+pjLu8AKw7eyizs1QYOtE6dyuAR77DxLA3hHD1iSYjrw4hLeJQ61qkHMNjGt9it3Zj9SVP4vVa4EkTlK4vtarqi3wZbqFEvrYhfojoNGPqoZaNDFDZaFxr9bPgNL02QXpucZeY0qHUgQ97HwcW5Ep0X2IYmEslzkwZeuSv9RUukSFZt72Dy3gLTwRHB27wf8QZxUBmyc0HhnYUIIdC51dFA7BMZ/mzmvQLiB++6Fq8y1w0jwSpbHwGYvFppYEPSfkX5lX3qhrB+yiu9qxEuU9VUf9RUdqa3lpfbi5zqi6gdqHC1ocvhaq1z8cbohAOVbqpI73FeXda+/xV2u5VzlAkBF5yUH/j6jwHgNicGBrxxRZbbj7LX8PPhvgjHWMJRynKhCetk8n8KMxlmD5Q+2iD7KjlZ6lAdNdr7wQzfmpCOTPSDpHoTnLJTejwM3ngJfUPDJKpwsf/WCOCbKjkw8As+P9rtgwHc4EgDf+NQyxBpXGSz9V/mAloHAuDp/kTNZbR6K8kCXpv53u25ZcOkv2clLRVd/6vh5uubSKBRj5IxOGuoMTi/oEsv/Rn9+lKXcSw5RoXbSsIUGOjzUE/xeVsEz6Y3FrrhIfBjpJyrpTtoDdxKIq86c/+czorCh/8b4x1pdQ6A8lrb8CMJsq84vS0s0TzYA04L0VUIYJVnZ2kWsYYmvil6RxS0D3sFFOL7JLMxI7EuryqkIBaQa1otteMfRU5oQHsANm2sz6G50/jsnFXfgiVVbwQyOSFQtmwg8Xxex278cnPZZg8M2911+EVjsLtggVgeKFZhNfEbEmabulTL39xsGEHgcjwC30RX+6hncFLIdOIwzRaqM5Q1zusqx97FkAG36NSjCbYqCHDN0oCdUPtDE1FVuL QvI+H36R Own2R+e0Dq1NwpV1JgF/NhDcWWgT1WOUni7PaYhYEeZh4/glz55SZcHuh/ATLd1eQ4rDA7z/GbZvO4lVroJyts2GycIwedHVgVrTPBtk5q8KIus2BzP6hlKq9YuZvwSgCv+oEyIX81Jie5TjPx0i6y50m33ijUx12IspS0aIT/1YfMFIHe38b36lHyTDhnNSJk0CQc8vvs+cwNWgZWGgto1KGHMTklRhCxn8uo2vpn/MWc9zjhOhq5UrcMS8Dwr74OBtvY9gjDyjzcA2SCVNxzLvWUTI8WZdWdFPBykc13faQ2RCIikNrrrCakTvoqrHUnFhBVHdfHnYDOMQFQpRVCpuzKxel3HZttTFz5iqfxFA1+YmQkHpTlXAo1ykXkCMaFzmbVWn2/MgVT3zqguRSCDwRs77WuhPmPY71tooVdWQol8LrcX4wZtpYr5U2r3vr+x5Y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Frank, On Tue, Sep 26, 2023 at 01:50:10PM -0700, Frank van der Linden wrote: > On Tue, Sep 26, 2023 at 12:49 PM Nhat Pham wrote: > > > > Currently, hugetlb memory usage is not acounted for in the memory > > controller, which could lead to memory overprotection for cgroups with > > hugetlb-backed memory. This has been observed in our production system. > > > > This patch series rectifies this issue by charging the memcg when the > > hugetlb folio is allocated, and uncharging when the folio is freed. In > > addition, a new selftest is added to demonstrate and verify this new > > behavior. > > > > Nhat Pham (2): > > hugetlb: memcg: account hugetlb-backed memory in memory controller > > selftests: add a selftest to verify hugetlb usage in memcg > > > > MAINTAINERS | 2 + > > fs/hugetlbfs/inode.c | 2 +- > > include/linux/hugetlb.h | 6 +- > > include/linux/memcontrol.h | 8 + > > mm/hugetlb.c | 23 +- > > mm/memcontrol.c | 40 ++++ > > tools/testing/selftests/cgroup/.gitignore | 1 + > > tools/testing/selftests/cgroup/Makefile | 2 + > > .../selftests/cgroup/test_hugetlb_memcg.c | 222 ++++++++++++++++++ > > 9 files changed, 297 insertions(+), 9 deletions(-) > > create mode 100644 tools/testing/selftests/cgroup/test_hugetlb_memcg.c > > > > -- > > 2.34.1 > > > > We've had this behavior at Google for a long time, and we're actually > getting rid of it. hugetlb pages are a precious resource that should > be accounted for separately. They are not just any memory, they are > physically contiguous memory, charging them the same as any other > region of the same size ended up not making sense, especially not for > larger hugetlb page sizes. I agree that on one hand they're a limited resource, and some form of access control makes sense. There is the hugetlb cgroup controller that allows for tracking and apportioning them per-cgroups. But on the other hand they're also still just host memory that a cgroup can consume, which is the domain of memcg. Those two aren't mutually exclusive. It makes sense to set a limit on a cgroup's access to hugetlb. It also makes sense that the huge pages a cgroup IS using count toward its memory limit, where they displace file cache and anonymous pages under pressure. Or that they're considered when determining degree of protection from global pressure. This isn't unlike e.g. kernel memory being special in that it consumes lowmem and isn't reclaimable. This shows up in total memory, while it was also tracked and limited separately. (Separate control disappeared for lack of a good enforcement mechanism - but hugetlb has that.) The fact that memory consumed by hugetlb is currently not considered inside memcg (host memory accounting and control) is inconsistent. It has been quite confusing to our service owners and complicating things for our containers team. For example, jobs need to describe their overall memory size in order to match them to machines and co-locate them. Based on that parameter the container limits as well as protection (memory.low) from global pressure is set. Currently, there are ugly hacks in place to subtract any hugetlb quota from the container config - otherwise the limits and protection settings would be way too big if a large part of the host memory consumption isn't a part of it. This has been quite cumbersome and error prone. > Additionally, if this behavior is changed just like that, there will > be quite a few workloads that will break badly because they'll hit > their limits immediately - imagine a container that uses 1G hugetlb > pages to back something large (a database, a VM), and 'plain' memory > for control processes. I agree with you there. This could break existing setups. We've added new consumers to memcg in the past without thinking too hard about it, but hugetlb often makes up a huge portion of a group's overall memory footprint. And we *do* have those subtraction hacks in place that would then fail in the other direction. A cgroup mountflag makes sense for this to ease the transition.