From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76E05C4167B for ; Wed, 6 Dec 2023 08:46:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC86A6B009A; Wed, 6 Dec 2023 03:46:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E78876B009B; Wed, 6 Dec 2023 03:46:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D19CE6B009C; Wed, 6 Dec 2023 03:46:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BEE3A6B009A for ; Wed, 6 Dec 2023 03:46:03 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 826454010D for ; Wed, 6 Dec 2023 08:46:03 +0000 (UTC) X-FDA: 81535761006.14.ECBBE7E Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf23.hostedemail.com (Postfix) with ESMTP id 4B7E6140006 for ; Wed, 6 Dec 2023 08:46:01 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=F3nkpE2P; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf23.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.131 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701852361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hwIA41qwVke6sh1/FSzZs9q9NAcH6wFzAZcpVOEwDtg=; b=i25Vxn5WHhWp1A/dH3Eh9SflvVg0nvzkSvehmzsIAn0YkD4+dTO+WdUL6h3gpvWrI2KUBG /OijMWuB3tbgQ74IDZmE+3cX2vsfQJ62wv8fWcVmcpeSWZb7CCeRK7vp4E3vPeMLxBGvqB p0Xtk62PMiuwfrck+EWk2e4yrcpQTvg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=F3nkpE2P; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf23.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.131 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701852361; a=rsa-sha256; cv=none; b=iR9p6JnlT9Ztz2jOn7yBEM+Srjrx8WdbSa17S6w5GfupazTqBUt8Me7hcrOWGu3lQrUfuQ D4BrCWtDY0/qYWCF3Jqsi6qjlW/UEfrsVwdAg729aVWi5shgrcQB+O5iPS8vt+Y7qMKn9J EgHfrJXdIjcd5SMDkKrrdIPF5wTOaBs= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 714EA1FD03; Wed, 6 Dec 2023 08:45:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1701852359; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hwIA41qwVke6sh1/FSzZs9q9NAcH6wFzAZcpVOEwDtg=; b=F3nkpE2PwX4hk0Xxxgt+mJSvMVMZfgQN6BAAK4xuanoRacEXwVPgGTMCMARe2ltM1/eDQY CbOHJYIJvPIId67Ct911v689d00ph0QCWReF90MKFe7PRToKfTMKqu3mmOGqRQNIWMe5ry YjXK5TmQZ8ggVwgve7qQ+D+zgYiF/D8= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 53950136CD; Wed, 6 Dec 2023 08:45:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 7iNfEcc0cGWnMgAAD6G6ig (envelope-from ); Wed, 06 Dec 2023 08:45:59 +0000 Date: Wed, 6 Dec 2023 09:45:58 +0100 From: Michal Hocko To: David Finkel Cc: Muchun Song , core-services@vimeo.com, Jonathan Corbet , Roman Gushchin , Shakeel Butt , Shuah Khan , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Subject: Re: [PATCH] mm, memcg: cg2 memory{.swap,}.peak write handlers Message-ID: References: <20231204194156.2411672-1-davidf@vimeo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4B7E6140006 X-Stat-Signature: ig3xukhn5s5ojky1zz359edu4ddemmg4 X-Rspam-User: X-HE-Tag: 1701852361-456264 X-HE-Meta: U2FsdGVkX1+PKhXk9vZoey6S6do2qU2y1Vl0t5TXB24cKwuZfOvYOkDd9pC0D1uFIiGDLB7gYElk4uGyQqg2H5rLUsG6S8FqoFxPrKUZVhgNqWG1SiYtht3FgdavFadTRqJpL8jKSAnpumA0+LdTghEf9QvkREQiuYPf47xfYqTMZfm2v7PqGQpX+p6nvvLOTNGPn091Y+4aENTDoNVXBeLcWvhGTD+pZrTr7/7DoXLCfJoRHXpENdVYQf10fBHV0taeVzplUfg8nM/3EEyRVVgBlsobssY0ekiIs1r9bGmwIauxFndNbVvK75zMvFi5F9Om+AifvHc/m3W305r2k7wIuZh2fcpKnSnaQzwM8V6GmwbTLvnVtJqy4AySDAoACV8G9zE4g3mqOx5D9HOo2zofWQwri0Pg0Eif6+bI7tNoWVxygciahawMgaDqN47AMSqFfftdf7bFTts5k27XfgWT//W0q+s41wS81QgnbPViEv9Sxku4qnYa0dmixC790JUiloJla15wLgWfigd59xjesomnUmwYXq3PYkQUOfcPj8CE6MJqUEhybW0Qozx/D5VT9u78YKMlD6QprtdiuvxXJp72iHc9qOSQDnoaEDHVz+NSOjHdzNHgfbif5MBT6Fuq7ptoDx1rFHF89zPICMi3p64TlJivS8L1ecL/7uJ543DTLjO+2ZA+G1fnF4KsyuzKmveghOr8/bl6+UtpE9aQQTq+k/9ASeivL8yPCQccbuZB19AMCLtlqiLM1OCTWdyWBq/SFzXDCyxPBS2mAMMH5Dw2SzpxPGZ+9772BiRD2cNWQHeML6arsI2ncMhQnBUCujM/TjQ7HeK5R1uEfueNXIi1UOIkCXqefHPPFvtoaD3Nmy99VxpW+loil4iAhh6Iv459l0VLKEZDNq0R0yBAAkp7bwYqa8YG3nmDVfYWCz5LWetGwqh+1jEeo7sdnWhGlXi4TGgS9sWczn6 O1QFg3gE n0MEP1ITLXVBtse97tZeQeO2wQZAKwecLzWyYsk9b/+Jt/zc5gv0G6keXeo+6Lqjn0D+2IBvuu9ue+w3/YEwne2N1qOh0DMIdZcgk4Kz2M9icXS/gIT0Qs3sN5N+MIR05dV4OcncX6oE0oqQojGZNHfTGQ7ZxQ3042EPJiR9kjvsCDNQC/scnHC9BWeo8u3Ytw6FDlHAzrLilhT6PpLHzK60i9efNhqhNVPmfnd5qHJe9qaznEiofZws3KQ0MHK69F9/tenQIK2C4m3oGclcCRLrln8+PDDFKlhPFrWNY+wW/Bzw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue 05-12-23 11:00:47, David Finkel wrote: > On Tue, Dec 5, 2023 at 4:07 AM Michal Hocko wrote: > > > > This behavior is particularly useful for work scheduling systems that > > > need to track memory usage of worker processes/cgroups per-work-item. > > > Since memory can't be squeezed like CPU can (the OOM-killer has > > > opinions), these systems need to track the peak memory usage to compute > > > system/container fullness when binpacking workitems. > > > > I do not understand the OOM-killer reference here but I do understand > > that your worker reuses a cgroup and you want a peak memory consumption > > of a single run to better profile/configure the memcg configuration for > > the specific worker type. Correct? > > To a certain extent, yes. > At the moment, we're only using the inner memcg cgroups for > accounting/profiling, and using a > larger (k8s container) cgroup for enforcement. > > The OOM-killer is involved because we're not configuring any memory limits on > these individual "worker" cgroups, so we need to provision for > multiple workloads using > their peak memory at the same time to minimize OOM-killing. OK, that makes more sense now. Just be aware that you might under utilize your limit that way because peak memory can still be reclaimed. If you set the hard limit (memory.max) to the peak memory consumption you would get a very conservative configuration wihtout any memory reclaim. > In case you're curious, this is the job/queue-work scheduling system > we wrote in-house > called Quickset that's mentioned in this blog post about our new > transcoder system: > https://medium.com/vimeo-engineering-blog/riding-the-dragon-e328a3dfd39d Thanks! -- Michal Hocko SUSE Labs