From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D214EE49A8 for ; Mon, 21 Aug 2023 20:28:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FF6E94000D; Mon, 21 Aug 2023 16:28:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 187918E0012; Mon, 21 Aug 2023 16:28:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0010A94000D; Mon, 21 Aug 2023 16:28:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DC45D8E0012 for ; Mon, 21 Aug 2023 16:28:57 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AE5C11C9230 for ; Mon, 21 Aug 2023 20:28:57 +0000 (UTC) X-FDA: 81149250714.20.F39885D Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by imf29.hostedemail.com (Postfix) with ESMTP id E2BA7120016 for ; Mon, 21 Aug 2023 20:28:55 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="GS9/2BnU"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bg7FamlUBikKy1ZdTRl5d2sgxVAImmgfb9egeVVkijU=; b=Lybj7LW5uGdNqZ+torTZ9XudA8SL2Y8HfXa9YhCwP7eY6auzz3kSUOQcNFswY3fq4/DyvW 1eUzyMlyaFbJVwd1Vp/3W46AU5wCQ7pz3ncLfTu4W0lIwTFq+5l/e2qzrZ7pWXsGi8n5F/ Qifbp/VFBaVrHwVjB+QDe68J1U1wKXY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="GS9/2BnU"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649736; a=rsa-sha256; cv=none; b=A2+LOwJSP6jrAjcCJoM2mN98nuGMznlVa6U6oajBdLHDMWFzZmdz2F720rxsGAYoPqZ/3F tt1hMeu15nAngfrgE4qTeS1Gfho3SVBBHNoDF11E3KvjEud5dUQ+KG416dKV2gqic0dGIH 44L/xz/PPGEP5njl31RIUA+nLDSK/pY= Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-986d8332f50so502920566b.0 for ; Mon, 21 Aug 2023 13:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692649734; x=1693254534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bg7FamlUBikKy1ZdTRl5d2sgxVAImmgfb9egeVVkijU=; b=GS9/2BnUwu1b5bIuLG08EYaNXH+HEHCTChQxHxg55sLB8NOoGwX+vmAnKuiPbbDf9b mZ7LpTBMnC3a1Nnzgv3bbrPFZ9IRYV8A3nWgKv9m3THXrC6NdegX590Uq5kun9tMH5S5 5zlojrZmE82bf49ARL1iHeWmcDVQkaRvkbWrcnKAPyQYo3U4lCYpvLAjaa/YOB8Q01Zc u7K/ecFioIx+x8qXdfKm7gGhaPRhdHsGjU5rZaGLpBhTiv92bL2bScXiQDmWFMjvy85e Lh0w+bXwcDn6prStj+dkCeBnGR68a6gMpfE3b9sVu7ld7iurMfMYdHnlYp1I1OxpvK3l +Uig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692649734; x=1693254534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bg7FamlUBikKy1ZdTRl5d2sgxVAImmgfb9egeVVkijU=; b=Osrh3ZzBgdsyFu2LLyAcsBv2BOYBnCTKSpYxhq8OCxMSX/5sHFDuGvpqFLUxyK09uu NR69VPvKEi6gylvPiQnwgfrQ1W9OoZujGCObDsV0MJ8bzA1LaN5UdnAVVJlq7Spg9xa+ 23cTAdGevj1X1MQKtKcDRGzwV8vA4AlCuAQNijHC06cMRZxrqXI/Erkw/W/HwBH1PY7O akwDysd/K76o6EIMh8U2o4nds4kJ9bnvEiKvHjMjNZ+g9/ZS0rPnnEPQtKlC+dqsCYyI YqgAeg4maagexYpjf9Jc9FM3W0r5nlrsOtYS2PRGr9fQj6hO0pnYPamGKoVGe7tF8S2b 2XGw== X-Gm-Message-State: AOJu0YwGnwGy/Gv0Q4mvUwg1IfrUGdFRFfPvWXi6gCYCxMhWApDwE6TS ENDFqLgpd4NEuTg9HDbPAdc= X-Google-Smtp-Source: AGHT+IGf2yh/BOotGTgsXmYaaGQBLu0XGFOT2TeDIDab903lxvXLvovWRyYYSDCXy47puh+eVEdJ8g== X-Received: by 2002:a17:906:105d:b0:992:ab3a:f0d4 with SMTP id j29-20020a170906105d00b00992ab3af0d4mr5901843ejj.17.1692649734243; Mon, 21 Aug 2023 13:28:54 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id k26-20020a1709062a5a00b00997cce73cc7sm7084450eje.29.2023.08.21.13.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 13:28:53 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH 2/2] fork: group allocation of per-cpu counters for mm struct Date: Mon, 21 Aug 2023 22:28:29 +0200 Message-Id: <20230821202829.2163744-3-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821202829.2163744-1-mjguzik@gmail.com> References: <20230821202829.2163744-1-mjguzik@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: axtokqgwdi78efood86deaijjkm58too X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E2BA7120016 X-HE-Tag: 1692649735-332945 X-HE-Meta: U2FsdGVkX19ltgHxzqR2SVs7jYf7c8d1COHP9p6tUwvDJjXvP3udur0H2hugWTc2w+7BRCDArRImHIXlK1gpiAsmykpuTSgdeXhoYdzPTkfBoA66e4FMOsSLOK7m2fczurUlhwPgNFWVDYMpv3Yv5rqtvd2OUAfLwj1CD1uB93eh1ggCwY/aYO9rp6uYRF0eQ1gypcDt0jPnmx2C843ylGnDvgmF8Llb9xXkERpIyF7sKR0esF7WkrX3tZ30XWVkGJ4KkV6lnwTfeCmnzgJCz+4v/aYcgZUzfNDTIpZZDMkwad3cnbwDDMbdph2VAj+Nu62Eb6yeC7ODuXoEj5bJl9RxzVXWGCvF5hP352JgmzZMdYuOHHZ9MhUmUGuxETOQ46ET67lElVbSB7R0n/VUymjptMt1JlLjG4aA53phJAj0CNSB1qITt6D4ZSF03KOeWrjGo6R6psktt5umaxmsdaUKbz+Ao2tdMQrV4gYA/ImQtPziX5Pj3+LAGgV7tTJkvOTYckULnMgslo5XHY9U66DprUPJrs9OUxPO2FC8tjumabRxKjvi3CKgvQac0rNgmDbwlroKXQoJlY5muYes3+GL6JMVys7IvAnu4VSdotE79ccJZWajDIYCHHBPTzSgssIaHo05eQ7HKb+b+9LnnoQvuExQgLJmS9WtM/bBkCa748R0MwTPfWpVz2ZbqSpFErLBWbtz+ZbuplmiEwlza1xRENDYCIjMqo6T8EY6FGCdo8yQX6tzgMklbpOe9qHULJ4udGGf9RRHBhRVXJVRJZ70w47Smq2Mr7iHdL5Q9VjSeID2UmswEygOF4Q1rzW3gcBDbyYxjYMwC5rcj7tqd+SiALrMifrE62loxbkEmIps2NtFDlYVx+1Ws+lAoeu+UDPim5gzxXAANfTyEjkjELVwY3H5R2TWMjULp3t7z60SRcB5Y528ZEypyID576VyD8Sl4dmWvpSpqd9cq+v n9Ki3CTo TM4mcE+OOdEGoomXb0TcMnIqfi41I0HX3XijBVgm92agJwf020cDaCH78IIgODtZ/el8PT+M494LQ4aqvVUSEqmLjoukeSfPg6MTtwEccUdv82RDWrnd6L9RAYFLW9YC2zQk8kkE09vhU+edITzqr+EA8UVJqYC9DD0CDdv+QWVQHO+/wvpaGPO8pEwYpW+RjwU7Kg+nIA9FReqmJX0FnifYMUEkj9WdSoPUweUZSzffbqiFlqtg3y3oCYamfwV3GuuPwjz+R24Q9/vdu8RC9G2OT+6R0xbpASu6wZ1n5DBSwluhmf8iFcXNpqQZcPnXlu84I18Nay3J3mv0liKmC8153k75S33MWjMpVE/43O0blDInAAc8VuZPwb6yPhGlY6cnXHQDEFQRTdi+BeZi3tLlJstoRiEYtmcGi7tzqqwWeVl8x0WvPF1hOn9a2HMPGvIdqDssy6URo1pCC0VoZb4f9Gf++thIldykqlcNrcJsCY3Rbj+GZ2MjM+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A trivial execve scalability test which tries to be very friendly (statically linked binaries, all separate) is predominantly bottlenecked by back-to-back per-cpu counter allocations which serialize on global locks. Ease the pain by allocating and freeing them in one go. Bench can be found here: http://apollo.backplane.com/DFlyMisc/doexec.c $ cc -static -O2 -o static-doexec doexec.c $ ./static-doexec $(nproc) Even at a very modest scale of 26 cores (ops/s): before: 133543.63 after: 186061.81 (+39%) While with the patch these allocations remain a significant problem, the primary bottleneck shifts to: __pv_queued_spin_lock_slowpath+1 _raw_spin_lock_irqsave+57 folio_lruvec_lock_irqsave+91 release_pages+590 tlb_batch_pages_flush+61 tlb_finish_mmu+101 exit_mmap+327 __mmput+61 begin_new_exec+1245 load_elf_binary+712 bprm_execve+644 do_execveat_common.isra.0+429 __x64_sys_execve+50 do_syscall_64+46 entry_SYSCALL_64_after_hwframe+110 Signed-off-by: Mateusz Guzik --- kernel/fork.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index d2e12b6d2b18..86ff78e001c1 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -909,8 +909,6 @@ static void cleanup_lazy_tlbs(struct mm_struct *mm) */ void __mmdrop(struct mm_struct *mm) { - int i; - BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); @@ -925,9 +923,8 @@ void __mmdrop(struct mm_struct *mm) put_user_ns(mm->user_ns); mm_pasid_drop(mm); mm_destroy_cid(mm); + percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); - for (i = 0; i < NR_MM_COUNTERS; i++) - percpu_counter_destroy(&mm->rss_stat[i]); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); @@ -1252,7 +1249,6 @@ static void mm_init_uprobes_state(struct mm_struct *mm) static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { - int i; mt_init_flags(&mm->mm_mt, MM_MT_FLAGS); mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock); @@ -1301,17 +1297,14 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_cid(mm)) goto fail_cid; - for (i = 0; i < NR_MM_COUNTERS; i++) - if (percpu_counter_init(&mm->rss_stat[i], 0, GFP_KERNEL_ACCOUNT)) - goto fail_pcpu; + if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT, NR_MM_COUNTERS)) + goto fail_pcpu; mm->user_ns = get_user_ns(user_ns); lru_gen_init_mm(mm); return mm; fail_pcpu: - while (i > 0) - percpu_counter_destroy(&mm->rss_stat[--i]); mm_destroy_cid(mm); fail_cid: destroy_context(mm); -- 2.39.2