From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36A08E82CA5 for ; Wed, 27 Sep 2023 15:08:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F5666B0161; Wed, 27 Sep 2023 11:08:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A74E6B0163; Wed, 27 Sep 2023 11:08:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56E336B0165; Wed, 27 Sep 2023 11:08:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 470F16B0161 for ; Wed, 27 Sep 2023 11:08:57 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B49CD160D00 for ; Wed, 27 Sep 2023 15:08:56 +0000 (UTC) X-FDA: 81282709872.24.419C280 Received: from out-205.mta1.migadu.com (out-205.mta1.migadu.com [95.215.58.205]) by imf24.hostedemail.com (Postfix) with ESMTP id CC6A2180016 for ; Wed, 27 Sep 2023 15:08:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aLkDt9Aw; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.205 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Mx7CtH6cHJrUFBld37HGiK3WgeHY8tZGKc2c0Drcq2M=; b=M9JQ4Yn2grsSqUdAwIMu8Dyqz/q8+FRbLxbNyv6CETFsuNbceYZwBlpdy4sOGQ+aI0R3n5 o1aJsvrvHTplMd2HfZaLXM2xVetEk9ytJ9oVL0P5Ev9zbgC8Chmg3e37lRwDNnOWVOzFfQ 3IjE/AGnMuuh6noyVQNGHd8fU1BBsjw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aLkDt9Aw; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.205 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827335; a=rsa-sha256; cv=none; b=OEEMzOpcA7wkq9qOfRfkynVRR7STr7x1Y/K01z6eIA79V5XWX6NXNpQiSTYRqwv4kmkRFX x4LMenvAZdQlmgNlsxPzQzj851fGH+BfMNX/mqqKEH+1gCZPIE13U1xxD8n4cpe8W1ds5c ixjJu5iER34YFFVtSuHNC7nK+n8LWRk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Mx7CtH6cHJrUFBld37HGiK3WgeHY8tZGKc2c0Drcq2M=; b=aLkDt9Aw9FniFt3rZ02X3yb2uiZuG/fL/+LQZnLtgdENgBt2TpkfsBmplfKZfgORJ+Car3 FEM2u00QMiFqgQB6NVNoz46iFA+hTFjpudzfEev+erU6A6CGIannFcgCjhBHXP22g1XPGb mkbfVuaA4mWkfj14/0wiM4V+WP8fH+g= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 0/5] mm: improve performance of kernel memory accounting Date: Wed, 27 Sep 2023 08:08:27 -0700 Message-ID: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: CC6A2180016 X-Stat-Signature: npigg3bpshuxobzatotcjq4hqzq5ebqz X-HE-Tag: 1695827334-680316 X-HE-Meta: U2FsdGVkX18Bo2V6hHZ32FJX0AjmkB/vTCSmibQDEzIcg25hUtF2Y+RbhyK7TXzxThoSmqQ8uCJQ8HDnmBWMnbgURrRr62XNBTcRaVPy99yIc+FyLm7ixgvu4Kg6ewfv2dWAtH27d/AMr0pHhnDNK/GlteB5jyxG+A08VAZ9RFH3zs7ZhlHh+sRrHgAuABgvzz4pcK29hpdlVocmt6pgmAbpOo3vwXXFk9Cfta8TMLYP6AP9OoM7LvdJ6DbhSHWkSZ+hQ1fOuFtfTRQk898nkqFeeAXvVB0yUrisUNWulB1cbpJuWQMh7INStLd8NzLTeR0YjVzykMKbpXXvfDE4meq2RFvl1wTUpqS2BZcHAekvk0JZkq/xCUFGLzeaBQJc06q5Lh1ioEtWmKSSxaP0E/XtPncG1NlQdHTIx+srJmT6cMkrv/JJgcLT8IULaLyu/gIZjRumBgyzHXLuHzVAchpmVjh3vZhK6KJkC6LRG548U/tYj4PeJf3t4CxnOXRaES1w0DO/XsK414x9ymmQW123FX6CGyXbydK9V+Hy5sI6+Rr3gLn5/DBizjiK27HxDLKlgjypKZJjnKTCUtNHIAmYwEhrrRyQUn+MX2EJEP5JcYQpooANld3OwU1Uactho90sBrz7ydBmR4Nnhbu/M+N29RKJzq/iQDeIQTVpRDGZ9I+eARkEdGMUq1KmuKeuPWvw+c79JvbjwbFCbokTRb/Oj0s/6Ereb/oEzW2luO41SeNRi6EM2esMXSbXivTg4hpxaj8RYP4iYsAwdS3X/oqdhpJXLlaAW+/hv/X4pyPGFsTO3QV7ZjxNoe7p9lrLVSLl7mO2Uog/fItSV0543sJHu3kg6XHzDjyewM2pHajagpJ3UV1L9VR3uTNQ0Ig5cOW4h6QWRP/d4Yl+AiJEQiijGQxzsDi5Jfdq0ZHzOA8RFtBQx4ch3Cgl7+KvcTJ96JbcbjzCZCuqStudFWl NAh6rPpH a0HbNFVkfOBT93zG9J0Ew4jUTUZrog0SpWWoBAIuBWCAWQa2iRO28hcQLm5OuV+Pkjimq/ekl3dHd5S0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patchset improves the performance of the kernel memory accounting by ~30% as measured by a micro-benchmark [1]. The benchmark is very straightforward: 1M of 64 bytes-large kmalloc() allocations. Below are results with the disabled kernel memory accounting, the original state and with this patchset applied. | | Kmem disabled | Original | Patched | Delta | |-------------+---------------+----------+---------+--------| | User cgroup | 29764 | 84435 | 59385 | -29.6% | | Root cgroup | 29742 | 48425 | 31573 | -34.8% | As we can see, the patchset removes the majority of the overhead when there is no actual accounting (a task belongs to the root memory cgroup) and almost halves the accounting overhead. Overall it improves the speed of accounted allocations by ~30%. The main idea is to get rid of unnecessary memcg->objcg conversions and switch to a scope-based protection of objcgs, which eliminates extra operations with objcg reference counters under a rcu read lock. More details are provided in individual commit descriptions. -- [1]: static int memory_alloc_test(struct seq_file *m, void *v) { unsigned long i, j; void **ptrs; ktime_t start, end; s64 delta, min_delta = LLONG_MAX; ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL); if (!ptrs) return -ENOMEM; for (j = 0; j < 100; j++) { start = ktime_get(); for (i = 0; i < 1000000; i++) ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT); end = ktime_get(); delta = ktime_us_delta(end, start); if (delta < min_delta) min_delta = delta; for (i = 0; i < 1000000; i++) kfree(ptrs[i]); } kvfree(ptrs); seq_printf(m, "%lld us\n", min_delta); return 0; } -- Signed-off-by: Roman Gushchin (Cruise) Roman Gushchin (5): mm: kmem: optimize get_obj_cgroup_from_current() mm: kmem: add direct objcg pointer to task_struct mm: kmem: make memcg keep a reference to the original objcg mm: kmem: scoped objcg protection percpu: scoped objcg protection include/linux/memcontrol.h | 24 ++++- include/linux/sched.h | 4 + mm/memcontrol.c | 178 ++++++++++++++++++++++++++++++++----- mm/percpu.c | 8 +- mm/slab.h | 10 +-- 5 files changed, 187 insertions(+), 37 deletions(-) -- 2.42.0