From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A70A3C52D7C for ; Tue, 13 Aug 2024 05:10:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2057D6B0085; Tue, 13 Aug 2024 01:10:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 18E4F6B009A; Tue, 13 Aug 2024 01:10:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0076B6B00A0; Tue, 13 Aug 2024 01:10:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D38396B009E for ; Tue, 13 Aug 2024 01:10:00 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 89E1D1C24EF for ; Tue, 13 Aug 2024 05:10:00 +0000 (UTC) X-FDA: 82446045360.02.15235C1 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf11.hostedemail.com (Postfix) with ESMTP id C627F4000D for ; Tue, 13 Aug 2024 05:09:58 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nafxyq8z; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of rientjes@google.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723525787; a=rsa-sha256; cv=none; b=Szpt53hISD7QXiVZ9sMypMe0zCggMOWrXlyltnw1FyQWyz3K/5ix5fX5hM7MDodn70pZux Nhmr7spTDKV5HSMmJf4rUF7UVSI00x72r/SiEC/F/O1caEEZZ6UWz0Dzbehnx09npV+Lh5 PrsycuO1zJTZwtmmKFXhyCU5YVwHo6g= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nafxyq8z; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of rientjes@google.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723525787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v1QYFau/BoTzy76SOWuTODxBFqibSdOTdy65uJMl6Iw=; b=hpy2YJOVYummw6Ik4rzQYJi5eDS4KZi9mAJj3VvisG9DH/tReqgux9MZaZhjUvZP7l+Iqk a5VfP1Wop4XXU9qCcv6H6VNOgRxIOESGGOKl+yk1T/bfzOtyTbuFEnqup99SRK3VhUz6TZ 1U/aqBMdSIFYbZp0uhIAi79MEpU9S2Q= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1fd657c9199so34595ad.1 for ; Mon, 12 Aug 2024 22:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723525797; x=1724130597; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=v1QYFau/BoTzy76SOWuTODxBFqibSdOTdy65uJMl6Iw=; b=nafxyq8zDJFZjpqDb6QbLpT8zYCjFQtJZueO/q10PUxLnGkw+ugVRPRbfPSPZL8yDw NqKWeyZc9rLonN1yO9Eee3ghoFkoTYsKmOcq4mnIuYulXU/Yx1q8hVUAjISaMLLJLOQq 0xq0QZ6ABz9CzyIiMjsGmcMqdRBI/S9K3Nr5OocMZclwX1xM881ooXKILbSUZu+CYXgF dop8wVMcApoVlWh/x1+b0J28dKsU+JV3PderS/APC6Kn9F5mQqJP32EjJhFfec+/3Tkn gVONisB8yWvXbAUkwKKArMTZnoGISKo7UmRG7c14bbJUrfW+i7k2CT0dhe7S+g8tALz6 j2EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723525797; x=1724130597; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v1QYFau/BoTzy76SOWuTODxBFqibSdOTdy65uJMl6Iw=; b=Br3rlE1BmAxKOLwPKc+5pjbzLTzxCLkcpA1eGcrIV32fCEBHYljWLGs3ItMU3wfRbX KRP48HOMADT4cJ9O+TecdWve2hsY3BHr0suy1vQYqZ9E4WWoDj8YcBNmIrcUe7lSjnTf Nz700f5OaQiRbCiGOmptuJIWPphmVtz9L9mO0oUmfZdk7cFTu2f+W8T0O9sZR/ZzbNrf zTTdYZSH5rTxn4YVLJmx/qM298gsW9J/oLG6aTsYCqmas5j+F17ZMUNzFb4gMRv/2M9e 0nZKN1/FKn0uIaztfI9BYuuFy3zbWUUFM5WLL3i4wGN5+goV51g1+pORABy6M5/wcSwV AJlw== X-Gm-Message-State: AOJu0YyAaLZCzFUigx2oeCVNqoww2AI9lyCg0oK/aJF3VtpPeDPUXLfP LZV6h1JD9MVKd8/0OhtTuKTNBJq9pTLpy08gPglHNChAKt0syJ3Ebsw6FOhdgg== X-Google-Smtp-Source: AGHT+IENkwUd9JvkyJ8WZbppPJls1rnH7OlPA+oSb9ghodPLmXaV1doVf6u+fPEMV5OfZQX90Fh5Nw== X-Received: by 2002:a17:902:f948:b0:1f6:5bba:8ea3 with SMTP id d9443c01a7336-201cdc4fdb4mr591195ad.25.1723525797103; Mon, 12 Aug 2024 22:09:57 -0700 (PDT) Received: from [2620:0:1008:15:499:4e79:c57b:11f5] ([2620:0:1008:15:499:4e79:c57b:11f5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14ac10sm5161755ad.102.2024.08.12.22.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 22:09:56 -0700 (PDT) Date: Mon, 12 Aug 2024 22:09:55 -0700 (PDT) From: David Rientjes To: Kaiyang Zhao cc: linux-mm@kvack.org, cgroups@vger.kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org Subject: Re: [PATCH] mm,memcg: provide per-cgroup counters for NUMA balancing operations In-Reply-To: Message-ID: <284406af-56b8-4b66-750f-10f9d38cfac7@google.com> References: <20240809212115.59291-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Rspamd-Queue-Id: C627F4000D X-Rspamd-Server: rspam01 X-Stat-Signature: i7cxp1w47grqd5qpqhpi51s3t1uyufsi X-HE-Tag: 1723525798-681835 X-HE-Meta: U2FsdGVkX19b+LeXVey96pvXB6a0RGQaQR3/tGgmysCiKLXPq0P3rnDNa/8AhMaKftHrtedemDSOwxSbpazTfhxwNkoDXqYOavhmCH0VLK1LLrlOYHqi54+68+5rGFwbRZR24mkkEfs/xyXlcnC5l64oUwbkDn8yVC06YE1wB1DXiyzW6o2o/q7ugQvA4zmd50I5vvOiFbMUDiyd/N8F/7B7b1xAWdCOjSjOJvIGiD+54KJLARBJI6KSLtI/TEwE7i9UkEpKwJq8VaNgFLpCncXRp+6ckMJcfmo5gQBohmmowOIw08jx8QIatdjpwnO/fYa9vOkkjArX5PjKaNKjJGelx6TW/n6wLgILd4nsKxcwa6J66v5r+AB7JPp4VC6+rgpqX2AiAy66xuXWjztSXzzgayzlimjbRYt3uamiT17U9Pyo8M46z58FMU4e9/gTLCQWKxN1IpwzUgJUrXScPaWVDG7jVyUwPLiNDOVYg8XrGzULo13k/IuJyF6FXn5kRrOs7acAzyQ8zWaFCjhRMZfRI2k0lWen5mD5nL6eD/uXzA21HTY0S64ZD6K63/IuILwT2eBXKy640WB4sq02tQbcXsAdOCbO52YBGZHHOkktuuBlSKOXbPDulJRXusWBhxCJbwYWZ58+6xl9K5ePPkvBzy0CmY4JJ1odsy+OLJoHYbZlsySTS8zqRqLANnJmcSjuPaVKxytrbZe6Yi1Ypua2LYNSjXvzKSUqF/94cSSaD8KK3vgwZDAiux3ERv5WUwvhkvyIf0o4Azu8Wos9f9iQBFV8XZdSjYiLNoBkwtysTaXY6GcmAjrwxHX0zJzi+bkc5wI+W2+rO3q+qXeIKOBPgwYNWaNt0+ky7ttc6Y98UqOmcB10SbsUO5u0/lhFARLfU3ODXczwhekH6cW34dgACcsy0K5pCwfJSATTGZ8ULiCgQJ0nFL1fvdTRcso2Gd1SgxWprxv6J3u7XFA NGRql9uN +8FZtXJVPHbj0pTpPKOnoMZfiQNvFuLuuXzxt2AtNr4y0fD9aib/CfmxpgREdF/xs/xLqu4pzTKSMkK1VmvNpMOlRxAN988P9FWSH3ma6vz2c3HmTYbqDgfJXW1WMC2CVS7J5IrD4eyC0ZzD3oo44sN1NKFC1Q0ICu5XKY8cg3Hc2pjzzkUmOY85QYDKl207upeGCm6KhD0Fv7vM+9T/wh5P0tfaTVg5qV1sqhc1Kt9Z19vnNDnxq1RowhFny34ObAL95emy2x0uQxk25pGzYZERVV+6j3xA6b2BpI4mOSgkqfhV3KHwSLrvHJzD/S+tWb6pee8rq0ckhnDzu5Z7JfHakBMmiVeSspVtv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000237, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 12 Aug 2024, Kaiyang Zhao wrote: > On Sun, Aug 11, 2024 at 01:16:53PM -0700, David Rientjes wrote: > > Hi Kaiyang, have you considered per-memcg control over NUMA balancing > > operations as well? > > > > Wondering if that's the direction that you're heading in, because it would > > be very useful to be able to control NUMA balancing at memcg granularity > > on multi-tenant systems. > > > > I mentioned this at LSF/MM/BPF this year. If people believe this is out > > of scope for memcg, that would be good feedback as well. > > Yes that's exactly where we are heading -- per-cgroup control of NUMA > balancing operations in the context of memory tiering with CXL memory, > by extending the concept of memory.low and memory.high. The use case is > enabling a fair share of top tier memory across containers. > Thanks Kaiyang, that will be very useful to test out, looking forward to seeing the patches! Does this include top-tier specific memory limits as well? And is your primary motivation the promotion path through NUMA Balancing or are you also looking at demotion to develop a comprehensive policy for memory placement using these limits? > I'm collaborating with Meta on this, and we already have an implementation > and some experiments done. The patches will go out soon. If others have > thoughts on this, please chime in. > I have lots of thoughts, but not sure if the primary motivation is around promotion only here :)