From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66ECFC369AB for ; Fri, 18 Apr 2025 22:07:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99E2A280002; Fri, 18 Apr 2025 18:07:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92881280001; Fri, 18 Apr 2025 18:07:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F217280002; Fri, 18 Apr 2025 18:07:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5C098280001 for ; Fri, 18 Apr 2025 18:07:41 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F22CE5F784 for ; Fri, 18 Apr 2025 22:07:40 +0000 (UTC) X-FDA: 83348552280.16.41718A3 Received: from out-178.mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) by imf16.hostedemail.com (Postfix) with ESMTP id E2B4A18000E for ; Fri, 18 Apr 2025 22:07:38 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=AuJTszNT; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745014059; a=rsa-sha256; cv=none; b=Lt6LuHSFvlFbkBbQBFT0WacwHW1F0ljjDickCuhn0fmTbjy4QzUhqYKPH5WyU/biTbTlt6 Kr0Hrm/0VUBZUyNyktRg1lW4lsqQCEx1yoPQT5FjQp3NEhu/EDV+mfNSoYw0kHl7n3e6AE 6xxErKXA9tjK0McRQAb/u0gf6ZwMdiY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=AuJTszNT; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745014059; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EHEzgdreTGKUMgd9RXTTlyD9jFkQ9z28zVdbJUd2dQE=; b=w9tMpCTJxv16nmTRqoaVPqLaVCB44zj/VqsKEYHlBMKOuv1/xXdvIusP/PSm5dW4Zd8W8X 1kEHVK2qTLoT0AWBOXUneNwoZdeOMigLFBwj7NWV14lZZy8aglm+pl2fw3Hj4SRqkXb1aa EQS42dGSpa3JvKKMNxmVnbMh0Ha5Les= Date: Fri, 18 Apr 2025 22:07:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1745014055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EHEzgdreTGKUMgd9RXTTlyD9jFkQ9z28zVdbJUd2dQE=; b=AuJTszNT23cDGu2206/Cfxh1goMTdF6tAkCsWWricd5ATNGlEj1+GbmZ5Ftk/bpGgnmqKm PUPMX68RFrqwegvdZsaMCWHqo57j12QSUhRb3xec/EyuI5eQD5akYEDLaQjCQWOe4cXvJt 3t6PrWwSmw+EPFAjhrRZfNgUASuXwCE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Shakeel Butt Cc: Greg Thelen , Andrew Morton , Johannes Weiner , Michal Hocko , Muchun Song , Yosry Ahmed , Tejun Heo , Michal =?iso-8859-1?Q?Koutn=FD?= , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: Re: [PATCH] memcg: introduce non-blocking limit setting interfaces Message-ID: References: <20250418195956.64824-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E2B4A18000E X-Stat-Signature: ixbq7zg9oknmokxem5xyi5yw9jcdn1sj X-Rspam-User: X-HE-Tag: 1745014058-698095 X-HE-Meta: U2FsdGVkX19n1GJtWKZ62biZ0HKqaV/QhbN1dqM5rbdAOUoYZ1POC1/OkNL/ygwqx6oKCF0SFtx6FQZFxjvdrHOhFIw33u+Eqr2QWJ0GLUyR3VZCVYGxFCP0BaZS8gZ/gavQIYn2HCp5D1/kjRd0GqV8kdu85sPwqUDVXtDDGkSuZzHJTM2RxIsKhfzpKzimCBwwpTwp9uj03Puz7giBRsFhld5TedEJFLdtex/W7aHrpF5LjvI7dv1+RH4ZaijeHXpOeV9+Mb89Sm9oqTXSA0HTu8ZhXmTaKb/z4xRAxllRH9LKuD2/ZSshMCUXJ+LgkzxfPUsTJh0cSmbn/n1+ITFtN8aLou7XQHdALfoJW3C6G4ZH3IcWa0LaoHPgVudvFsSyhTG/iutxohSmJMJWFBlYIMwAecb1+QFsqFXUU6UDgzwpFpDgONLds1mJc9hPCPjS2genfpm1Vn8SBOJZ5I4NeXKRL3UXeiSPc7us/D/iXw/cQ9D+DhwpYRN6j8Lm6PFrtqzY/fHcrPt5O2UuRfQsKFh62msJPNmOMUvAJtK8jtt5OvhRwSU4wJprwm0mzyChukoghUMsFz6T7vTLPGbWBaZSZO3cKUNCusurTTkmY0oLPC55qjvHv7ENUJMiv8SoC++g/iMbMxfVTZoDvipvOK+GLdWKSjtzgEswQOPogZniLGr7tgYFCWQQ+BYZEuRHksKqSZn1U0qFjgQtbLCrQmQc13B4iRA77CuYUOCozRQGi8X2KAm+O4qJI9AkBFfa1MjD4/Wk8sOWFO8j+P4MUSb/Z75y9gd4Kgri9A8UwGdp73CwZhGf/GGVOPvown2fAIVm6ng+qjuMghOwnzzXrdD7rHIazg6aDkisqm8RPNbgEx4Jz85gUvVtqetup+pISAtJT7ytjo6YzL8jJJRnGFdhyx7R0BdOJ0TiTxkZPGIe0qMVSUHxqfm7VULpsMQEjHIpGoQYKXkLNdL t5/1n/qx hNOLUYCzGJOwK5swCS4i92F5Gg9xBi1u9LwjCRBrSDCVM+6KZjHVZnTv2xRH+m2DoM0AW9mxaxeb6hLr6+mkejb7dWEmVTMpL3EqvUE4VfnQCb7Js2vizfPLzqZqiwSXbxB3/MXr+HofDY3qr2PvkLymOJ1EyjtQfNaCMNdsoUmqIdqwpWNMQ+B+A2akFp7GcKd5ZphsPXShu4q0FGM9z/Vqdr8t0EIbxh0dMOSvWBevX2E/KONCFJU9ZBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 18, 2025 at 01:30:03PM -0700, Shakeel Butt wrote: > On Fri, Apr 18, 2025 at 01:18:53PM -0700, Greg Thelen wrote: > > On Fri, Apr 18, 2025 at 1:00 PM Shakeel Butt wrote: > > > > > > Setting the max and high limits can trigger synchronous reclaim and/or > > > oom-kill if the usage is higher than the given limit. This behavior is > > > fine for newly created cgroups but it can cause issues for the node > > > controller while setting limits for existing cgroups. > > > > > > In our production multi-tenant and overcommitted environment, we are > > > seeing priority inversion when the node controller dynamically adjusts > > > the limits of running jobs of different priorities. Based on the system > > > situation, the node controller may reduce the limits of lower priority > > > jobs and increase the limits of higher priority jobs. However we are > > > seeing node controller getting stuck for long period of time while > > > reclaiming from lower priority jobs while setting their limits and also > > > spends a lot of its own CPU. > > > > > > One of the workaround we are trying is to fork a new process which sets > > > the limit of the lower priority job along with setting an alarm to get > > > itself killed if it get stuck in the reclaim for lower priority job. > > > However we are finding it very unreliable and costly. Either we need a > > > good enough time buffer for the alarm to be delivered after setting > > > limit and potentialy spend a lot of CPU in the reclaim or be unreliable > > > in setting the limit for much shorter but cheaper (less reclaim) alarms. > > > > > > Let's introduce new limit setting interfaces which does not trigger > > > reclaim and/or oom-kill and let the processes in the target cgroup to > > > trigger reclaim and/or throttling and/or oom-kill in their next charge > > > request. This will make the node controller on multi-tenant > > > overcommitted environment much more reliable. > > > > Would opening the typical synchronous files (e.g. memory.max) with > > O_NONBLOCK be a more general way to tell the kernel that the user > > space controller doesn't want to wait? It's not quite consistent with > > traditional use of O_NONBLOCK, which would make operations to > > fully succeed or fail, rather than altering the operation being requested. > > But O_NONBLOCK would allow for a semantics of non-blocking > > reclaim, if that's fast enough for your controller. +1 > > > > We actually thought about O_NONBLOCK but the challenge with that is how > would the node controller knows if the underlying kernel has O_NONBLOCK > implying no-reclaim/no-oom-kill feature. I don't think opening > memory.max with O_NONBLOCK will fail today, so the node controller would > still need to implement the complicated fork+set-limit+alarm logic > until the whole fleet has moved away from older kernel. Also I have > checked with systemd folks and they are not happy to implement that > complicated fork+set-limit+alarm logic. /sys/kernel/cgroup/features ?