From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59B9FC369AB for ; Fri, 18 Apr 2025 20:30:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8057C6B0028; Fri, 18 Apr 2025 16:30:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B4006B0029; Fri, 18 Apr 2025 16:30:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A4BD6B002A; Fri, 18 Apr 2025 16:30:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4D9CE6B0028 for ; Fri, 18 Apr 2025 16:30:14 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5B8FE80BC3 for ; Fri, 18 Apr 2025 20:30:15 +0000 (UTC) X-FDA: 83348306790.06.0D0E161 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) by imf24.hostedemail.com (Postfix) with ESMTP id 63416180007 for ; Fri, 18 Apr 2025 20:30:13 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=qBoDblmB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745008213; a=rsa-sha256; cv=none; b=4UGko/yal8T1UR9FS+W7WYcrz1j7xStorAp+ddOU5VC7olBmf/XgZ96uqykCsEp8ozMFku NeoZa7nLtq9SQcX5eUh6HxAQCX43gYQV3dsDdjTXDBaAj6qrrfZWBMSSbFyq7CzQmmaVXI wV8xJxDzc22pB+ZeMyvFWLQDZM/lL28= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=qBoDblmB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745008213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4WmhYDIZ6fnmE1nZMjBHdeec4YLIkfuJIi+dcBY3/dw=; b=xZiobiX059PZb5qDeEbst1Tf2kIH/X0gmqUjZXF77f8hpGmehVNx6UN5HvXzA9vs+Jn8Kj a3e1i4RY4fedcjApxTOJYiMPKXieUNLmwROsMIqSpXXOtbkofKZIYXHfcTdi1STwxvNt8a elJwFff/r6R56prX2dV0TVJoGSnPzzU= Date: Fri, 18 Apr 2025 13:30:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1745008211; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4WmhYDIZ6fnmE1nZMjBHdeec4YLIkfuJIi+dcBY3/dw=; b=qBoDblmBrSQUhfhMjZb2eP/1R3TbszseqFKH92NRDM7XvtY8ANXVQQ5o/0g5M6Km/UnncF Ed3pIN4u71xCC78Md/Hrb4wueiiGUzAn7zdNMzpt/Lo068/JU3Rrh28EpSAQtNAgPFRpje 1Eo0E1VqXSI5S1Qkis/grttdz9I6A9I= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Greg Thelen Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Yosry Ahmed , Tejun Heo , Michal =?utf-8?Q?Koutn=C3=BD?= , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: Re: [PATCH] memcg: introduce non-blocking limit setting interfaces Message-ID: References: <20250418195956.64824-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 63416180007 X-Stat-Signature: k6ir5qwfaqw5ujn9km79tn7sa94uexqs X-Rspam-User: X-HE-Tag: 1745008213-729456 X-HE-Meta: U2FsdGVkX18mfEcYblZo4bPxWgkkuNaUO5qrI2FcSqgHfFKEGSVFmFx9uQAUTY0KMm9vvIr7niQwEl+QjXMvrSruxXHQacjWSgX4Whte6cpARISI61F/SgdQYH+SjCS0RhuErP59yMynzUpnB9emB36cqST6+UxuqwHMDYjjhPDmEo4dKyavUt/bf5yTrPTS/tSsgiR+3uTZf0sK2O4mLuky/PcwwiiFtjKItiLjJ1KG+UEESr6UVRnlD6A9lz/m3SQKsawiSwxb/m9g850AlhjAH7Y4bgqFdWUQROGFLk6vl7MvfJlopuFDxgeLg0IAhWAr//V7jEmw1qGmX/vTUh0f1uOzuBGFpRasxIQc806VPoqOgzzmF0moRHVvjh0zT4vA0Qtcu+Lisuxnll0NpD7VMnM+V4Ft1XDkhNaVfKEzuhwZES4nYH/BNJv+s4HcoSNNwFfQLBTph8rXquPA7OeySEfpfV/ucecmkuRyd8K/+eKoiUf0SOyutfhZSOvVx5NmLQ6qoJPioi79B6zn8d282WYdwRnGFbYCbvS6qQt3gOfKz99ArTf5LQMH6gmQBHTVnHDiYjkyytHigLk/4bmaDMzboGf1zsxhmnqqDUEaaK0lN/8qwtCj3jV/vnGfvzJCn6vm18b5ProF4hLzKVlWEfmCcB37rn21PPmWyWa+Y++PO6Sgifcqqxo0dXMaswXPt3Eebz8KZcaCW6wZ2wcuANdvToakcFkrw1iwFxyqhaGJRMNaZndJFOgH7W1zxmQJTskgMUMLEPP72Gn682zbKAt3SujqIRjLBt+xSyBrecuARQLBc0FAqLFX4Y6aiujnB20BEP96QwiDfpjF4zX6SmphJ3nKPU7gbYlajOowgPfoez95jWryQNf1bZugA9SsmRgKYCUg2kFiyC+ArsSCLgD68jMBwrBtyI8tic2mZNKW7WEGvuCgmqqS1jwuG+g4PosatxmhwkXhkyD 6RS8xTgs wvaJRdrmxHAeNsUf0Xi2HCSNfNQfL/Up1NISuclrcqLVHU7nJclBui1hCi8DvtFemjWqQlURKN73ySoRbwUxspkaVsHOhj6Y2M36SJw9FAeouCEvsNB8bKaVtcgYizx+n4J6IVwO9RTvnnJKlueiQ4KT2HkuqHZA1ZEduG3RuUA3tgP8rXqCYrcVEEfsOTn8Fb7BKN+NriJQmdjI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 18, 2025 at 01:18:53PM -0700, Greg Thelen wrote: > On Fri, Apr 18, 2025 at 1:00 PM Shakeel Butt wrote: > > > > Setting the max and high limits can trigger synchronous reclaim and/or > > oom-kill if the usage is higher than the given limit. This behavior is > > fine for newly created cgroups but it can cause issues for the node > > controller while setting limits for existing cgroups. > > > > In our production multi-tenant and overcommitted environment, we are > > seeing priority inversion when the node controller dynamically adjusts > > the limits of running jobs of different priorities. Based on the system > > situation, the node controller may reduce the limits of lower priority > > jobs and increase the limits of higher priority jobs. However we are > > seeing node controller getting stuck for long period of time while > > reclaiming from lower priority jobs while setting their limits and also > > spends a lot of its own CPU. > > > > One of the workaround we are trying is to fork a new process which sets > > the limit of the lower priority job along with setting an alarm to get > > itself killed if it get stuck in the reclaim for lower priority job. > > However we are finding it very unreliable and costly. Either we need a > > good enough time buffer for the alarm to be delivered after setting > > limit and potentialy spend a lot of CPU in the reclaim or be unreliable > > in setting the limit for much shorter but cheaper (less reclaim) alarms. > > > > Let's introduce new limit setting interfaces which does not trigger > > reclaim and/or oom-kill and let the processes in the target cgroup to > > trigger reclaim and/or throttling and/or oom-kill in their next charge > > request. This will make the node controller on multi-tenant > > overcommitted environment much more reliable. > > Would opening the typical synchronous files (e.g. memory.max) with > O_NONBLOCK be a more general way to tell the kernel that the user > space controller doesn't want to wait? It's not quite consistent with > traditional use of O_NONBLOCK, which would make operations to > fully succeed or fail, rather than altering the operation being requested. > But O_NONBLOCK would allow for a semantics of non-blocking > reclaim, if that's fast enough for your controller. > We actually thought about O_NONBLOCK but the challenge with that is how would the node controller knows if the underlying kernel has O_NONBLOCK implying no-reclaim/no-oom-kill feature. I don't think opening memory.max with O_NONBLOCK will fail today, so the node controller would still need to implement the complicated fork+set-limit+alarm logic until the whole fleet has moved away from older kernel. Also I have checked with systemd folks and they are not happy to implement that complicated fork+set-limit+alarm logic.