From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43155C02187 for ; Thu, 16 Jan 2025 18:20:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8BCB280002; Thu, 16 Jan 2025 13:20:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3C1C280001; Thu, 16 Jan 2025 13:20:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A037F280002; Thu, 16 Jan 2025 13:20:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 80BAC280001 for ; Thu, 16 Jan 2025 13:20:37 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3FE54A10C2 for ; Thu, 16 Jan 2025 18:20:37 +0000 (UTC) X-FDA: 83014130514.17.534A823 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf25.hostedemail.com (Postfix) with ESMTP id 3BD5EA000E for ; Thu, 16 Jan 2025 18:20:35 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j40diDOK; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737051635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T9mGqQGdJHpgWwMiBdNaQnEkV/H58D4+6ntUtA4C2ds=; b=hvA71mha7nOC5BWqBvbPkr/hpyKLZOLcxul/KJjdr8w4aPdBEbYWUx3oIAVNoT7vNXDzjO ferqmGPG4V0QO9z1Ujkobn7vvw8ARpLAHFoBQ3wiDEY5xzvaKWOmSy6E890IBwYoXziXwV /+NLsid6fDeiKnRPu6vlbgnXCtnDTdg= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j40diDOK; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737051635; a=rsa-sha256; cv=none; b=coDz89Dk/BQ+rO8pEFe6djz4QUj+jDV9WEbY5dIgYT+fQfNWgwL1VT6R38e5JfSrDm6su5 KA+9N9pIHS4nhnfSxXL3WefW4vrwTmKdIMeVUPLSl554dDxH5RVYnCLGna2cmucR7w4vAi N7Hon4X9/ny0uTO7mqIIMb+pL8F0Zt4= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21c2f1b610dso5485145ad.0 for ; Thu, 16 Jan 2025 10:20:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737051634; x=1737656434; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=T9mGqQGdJHpgWwMiBdNaQnEkV/H58D4+6ntUtA4C2ds=; b=j40diDOKSMkW2oV2s3+2D5gRC6A5MthTk3z/6Zdhgzz5f/Pf1C4DEM8h4KSPFGwJz6 r+Vt6EZNrO2iXahJm9JD3VpFla14iRefX+GuGQjEbY+kjmufl0JWG2OMbaB1K7BnyciT CyDYZFJD7bEUnXAe0r0Cn5hvPd4aARSg8fIv+b4H7g5Q8bu2JGExjeaxJ8wHW+0DDjAn L8YzyGWIs5O8r+PBCS9oT4CLiyrtgvZA2kWjqIdzNuXav25nRKwISjPtd+hTtvOQ3IKZ +C0IvRyeeUSOsX8pFTxK7b24+zK0l9Ad9R/jyNkkPXFgkk3Xsux6rHNsrwbyKLjfLICn gXNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737051634; x=1737656434; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T9mGqQGdJHpgWwMiBdNaQnEkV/H58D4+6ntUtA4C2ds=; b=kJqMxrL/vQWTrskvYCqAhvjnP4KAEYAYdXxfVLmCg7UTQRh+YpFMC3OaGAwkKngEOL oMTnJrzMYVpX64v3U/uH8OtY3hyrzQwdXB6nfKRC46AOvfVpVywX2jcUMHEWbL1JGJgK AElztUHQObEQvANyCOoetnPRYEZ8vDyynp07lshCmTPG5Tec4dhOXVmLEYOVR22CDG6A qn1ZdBqK5J3LyqgthpwgtMRWKjiHeHE1qOGyamItRDOJaCi3veyjQ6YcHti7FXA8cCAE FKCmUBCx2OwfTfyi7NW8rMnhYCNhY6E+yfquUVn8K37JDppySPHQKYDTTdJZ23Q8rS9n ZhAQ== X-Forwarded-Encrypted: i=1; AJvYcCV5T7RZAEj5tn4coZ2n0xmw5MfrBRmMWZ7R2y9MOQ3RD+JXewFuOvQE8PFAavPTIzVqGaxPzwY5Uw==@kvack.org X-Gm-Message-State: AOJu0YyxqYMHCZv80eVbynRmvdiZL4v0IPshOe5t7Oxzslbrfj+SY5k+ NwE20KyQebtKIwlc/M6ITq/uiI5MFJSHKVX6dp0H+WLgLuU9G3i8 X-Gm-Gg: ASbGnct5UgjorM3Bqg/uacteydqZ4+qQZFzxC2sKkTMXdq0kZ2/GgzQxYjcMJAnFR5P rIU+KfnOXmR4Kj8Pkz9HjQiGQX2g7c3Eu/wJxNv1PRmNZc3fIP2mFrEStcT7NhzVDHK960fbA+8 heet18n/MaSvIZhM4/bjlgh/9Nf+iLETkru+HWdqjcagiG5g87Zg0tFMKjJXhBmZx4pPwHF/++l WchLR3hndWihrB7PY8afwAZupWIUpw1koLHpAGbYKFUZ3brwp0A8/loBwpiSpV3889hU7ZHcGlL QibagCKKSJLSiFUFX6qGMEDbI6M= X-Google-Smtp-Source: AGHT+IFHOx+zZgnYlRSi9+hTWBf4f17opgxm3PUVAuG5JYwdzZ7+IPOnNDQPIl3IG8aXjvFhQeycWw== X-Received: by 2002:a05:6a00:124a:b0:725:f376:f4ff with SMTP id d2e1a72fcca58-72d21f7bc52mr29015885b3a.13.1737051633910; Thu, 16 Jan 2025 10:20:33 -0800 (PST) Received: from ?IPV6:2a03:83e0:1151:15:583f:3b88:2f88:a4d8? ([2620:10d:c090:500::5:a19b]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72daba44453sm293830b3a.127.2025.01.16.10.20.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 16 Jan 2025 10:20:32 -0800 (PST) Message-ID: <3c078729-7455-4b7c-818f-bb8d293c00ec@gmail.com> Date: Thu, 16 Jan 2025 10:20:30 -0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/9 RFC] cgroup: separate rstat trees To: Yosry Ahmed Cc: Shakeel Butt , =?UTF-8?Q?Michal_Koutn=C3=BD?= , hannes@cmpxchg.org, akpm@linux-foundation.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Tejun Heo References: <20241224011402.134009-1-inwardvessel@gmail.com> <3wew3ngaqq7cjqphpqltbq77de5rmqviolyqphneer4pfzu5h5@4ucytmd6rpfa> <3348742b-4e49-44c1-b447-b21553ff704a@gmail.com> Content-Language: en-US From: JP Kobryn In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3BD5EA000E X-Stat-Signature: tek93if5kipg37nqhborpehgowcwjhtb X-Rspam-User: X-HE-Tag: 1737051635-16791 X-HE-Meta: U2FsdGVkX1/RvQN51VN+sdxrJfmzSDZc3VZd1SZn74/Sp2Fi/NehaJcnZNSUi1/Absyu3YMIUctgd40HLzUgvlKvHZt+0KQ2XvuRkC79EC+Ix3Hu4JLGl77idAUkD8YJcInazVXRlx6O5ClT7MTHhb0RKtsEwzEHrpFumaCuEndnVaJtO47630s9XqhoPTlKU271PPzsEGBB//D8rzwo09CVA/HOv/Bmb8KIUOKN8vFU4MdO1DO1uMYJlcfp3l2doPva8a4u6+Ejw4i8woUAQgQuhfTyovMQCwcU4H9Bt+XZD95sMPAKrfm+Qla8mD3tgtIfO3337zbWuPU2PhRhOB9xcgvH69Cux70NHHrEFD0Et3uVD8hCl8DzAx98wxt75WsgF4gk/uTUmW0J3XAQhBGMckb3b2VRbC9ops2ny0Qez0zf1En9/ANxzPIoaF6kYLOk43qCLjc7vq7XI7FRZnrXaukuU6PNkw4x4RFNkkUV58edE+8EecoayvkIh1VRXzxe7CIAh8792C0CU3CcgBkXFVWxwHklAhVhG7yCaMEHlzv9SPgMYZfHMRASr3P9jsYhUQjVRihbKrkjL64DCH74Q5tn2n/G4OfnMxnSWjhtQZDfnpSlK3dkTCFADGolVWP80hLt+O/xtj8jw32T0Gzk/mTG8YvSBdqH/Y2iZkCJB0rkVuiaJh4IYYXln51+qPgHGPYQ+hCGWk1LK/bXSHUrK2399mJiVra0KhEpVzlKfmdPp0YCU3etOm56y+lW5uyDfZfMBC+dASz8DZNMYt1Fm98IHTuaAsHHS2/Vr/dP+IfDg71L08OtpWYxJCkT3l4dxOYH0YXuXExSNJRdZAObFJfkZampZcqjVilXgpeAj5kpde9TbQJMZ/273FGKsG1jiKanFdp+nzGc2ZB6lAFDi3ClaGBi1tBdHP9EXpTvWNjRSNdbKKoY3T/xIhZhnlmJCSGCz/ZymSnSyNp zm+6R3sZ RLFxBHh3pcGAKRoL+0zrZPgmWI/fM5mayttAQg0SEPc09ErszsNI5Fby5BBPFCQxi4A4SUXuhgbHOQ5/IWA5RMe9QG8KPIt9wzjVBfC6hS6Gio/GFibEixEapYXt224THvrjgfhL8bxFLNeygpi+cjxplIwCTuxjtEGpdaokEu1mkvGAE7sRLP3uwBbpzUWivLHgxPOjI3itENK3zUrtvB7VgeZk3uEmgkMCEWXpt1l+UB+nPoxzAfLfvDTZunbnNcGPoAMC/bLOdgUkBn/mz8rF4NSa9wc7qwjJj/9wTRjcASvMKj1FF/98dal7w3GQH34ek7isIlI0J7HbulBftMLWES5+1w+tLFwcI63CC/ZVOInTU8G+upBXM8/JLYxUoq9uSFkvN7RW1AgWcW39C3pLHkSO3yvzMm9R4EAF187YD75k/xgXt0mUkubx5XDa/j5Qb0VgjqH2mL2zfi7HzFH3g0HxFC12sKXramEy9V6yyfb93wPRxQ7PTEg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 1/15/25 1:36 PM, Yosry Ahmed wrote: > On Wed, Jan 15, 2025 at 11:39 AM JP Kobryn wrote: >> >> Hi Yosry, >> >> On 1/14/25 5:39 PM, Yosry Ahmed wrote: >>> On Tue, Jan 14, 2025 at 5:33 PM JP Kobryn wrote: >>>> >>>> Hi Michal, >>>> >>>> On 1/13/25 10:25 AM, Shakeel Butt wrote: >>>>> On Wed, Jan 08, 2025 at 07:16:47PM +0100, Michal Koutný wrote: >>>>>> Hello JP. >>>>>> >>>>>> On Mon, Dec 23, 2024 at 05:13:53PM -0800, JP Kobryn wrote: >>>>>>> I've been experimenting with these changes to allow for separate >>>>>>> updating/flushing of cgroup stats per-subsystem. >>>>>> >>>>>> Nice. >>>>>> >>>>>>> I reached a point where this started to feel stable in my local testing, so I >>>>>>> wanted to share and get feedback on this approach. >>>>>> >>>>>> The split is not straight-forwardly an improvement -- >>>>> >>>>> The major improvement in my opinion is the performance isolation for >>>>> stats readers i.e. cpu stats readers do not need to flush memory stats. >>>>> >>>>>> there's at least >>>>>> higher memory footprint >>>>> >>>>> Yes this is indeed the case and JP, can you please give a ballmark on >>>>> the memory overhead? >>>> >>>> Yes, the trade-off is using more memory to allow for separate trees. >>>> With these patches the changes in allocated memory for the >>>> cgroup_rstat_cpu instances and their associated locks are: >>>> static >>>> reduced by 58% >>>> dynamic >>>> increased by 344% >>>> >>>> The threefold increase on the dynamic side is attributed to now having 3 >>>> rstat trees per cgroup (1 for base stats, 1 for memory, 1 for io), >>>> instead of originally just 1. The number will change if more subsystems >>>> start or stop using rstat in the future. Feel free to let me know if you >>>> would like to see the detailed breakdown of these values. >>> >>> What is the absolute per-CPU memory usage? >> >> This is what I calculate as the combined per-cpu usage. >> before: >> one cgroup_rstat_cpu instance for every cgroup >> sizeof(cgroup_rstat_cpu) * nr_cgroups >> after: >> three cgroup_rstat_cpu instances for every cgroup + updater lock for >> every subsystem plus one for base stats >> sizeof(cgroup_rstat_cpu) * 3 * nr_cgroups + >> sizeof(spinlock_t) * (CGROUP_SUBSYS_COUNT + 1) >> >> Note that "every cgroup" includes the root cgroup. Also, 3 represents >> the number of current rstat clients: base stats, memory, and io >> (assuming all enabled). > > On a config I have at hand sizeof(cgroup_rstat_cpu) is 160 bytes. > Ignoring the spinlock for a second because it doesn't scale with > cgroups, that'd be an extra 320 * nr_cgroups * nr_cpus bytes. On a > moderately large machine with 256 CPUs and 100 cgroups for example > that's ~8MB. > Revisiting the cgroup_rstat_cpu struct, there might be an opportunity to save some memory here. This struct has several cgroup_base_stat fields that are not useful to the actual subsystems (memory, io). So I'm considering a scenario where the existing cgroup_rstat_cpu is used for the base stats while a new lighter struct is used for others that maintains compatibility with the rstat infrastructure. >> >> As I'm writing this, I realize I might need to include the bpf cgroups >> as a fourth client and include this in my testing.