From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39F90C27C40 for ; Thu, 24 Aug 2023 06:27:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C81F8280094; Thu, 24 Aug 2023 02:27:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C32068E0011; Thu, 24 Aug 2023 02:27:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF8FB280094; Thu, 24 Aug 2023 02:27:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9F9A88E0011 for ; Thu, 24 Aug 2023 02:27:05 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 729E4801D4 for ; Thu, 24 Aug 2023 06:27:05 +0000 (UTC) X-FDA: 81158015610.30.2A25E60 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) by imf15.hostedemail.com (Postfix) with ESMTP id BB20AA000F for ; Thu, 24 Aug 2023 06:27:02 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf15.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.166.51 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692858422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zaYFol9VpMoa7wRnrH7aK64tMzJrKbUDz5W/yZnlsfE=; b=ZrSlGXHnfBU+QTSPqMm9FHKUcxiFrSneA6Odz/vPwXmnT/pBMPhJTczntOKjYehg0gpdgG 9YoBjOrxlLrHJhPRI159p8i7gLV2v3cPCQs+/T8n/D0ujvCicpMNePxQLM76lyIFLuaC4Q j3fL1S9epwxp9F3K9h6zz4NL1/f/kPA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf15.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.166.51 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692858422; a=rsa-sha256; cv=none; b=h2W1z93gx8d6M6wSWjg+4TuJ/X9FFUw04040QWJ5qPrTcfcQyfq2w0d25KHB0Qxms/Qi1b z867VVmQ/cEJMF4rzFH0qnoqnc/xhslO7fRwIGLWlDianhiMCkyZ7VBFsOkcdOZKIJ8QyQ c7a5KZDhNPOHVl/2Zx+YobfTEDIF/NI= Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-79231457b7fso137356239f.3 for ; Wed, 23 Aug 2023 23:27:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692858422; x=1693463222; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=zaYFol9VpMoa7wRnrH7aK64tMzJrKbUDz5W/yZnlsfE=; b=OxMgDnpDt/BBEUV4IZpscTcZUoHzUPUYy5a950jMT4YZ0noFpr+j2nIrjTc5LXQi8e 0YX4tEZcPQnO0/S0AniMp7LYTGfrRDwN+yIcg1YNACgMG3erqOBhxUAbubCQmjc8+wWR SqWz3F0GE39F78tM3s15blO7/X46FLrhmxaB11/KRHrNfYxU9xt7ZycMtCuEQP1XN3kU xpIc3PmT2xvAB1YWBEOoTd6sIGBOWN1lYpZwtIx+aLHb3gRgd1tMRiEpNbsJjlc36Gee 96OsBy/0NCYZ3UjbNmDsonBTgLJ92+GB3n1ItgU5d97JLmrZZ2+UyjQVtr4ACmwlOQ/5 4pRg== X-Gm-Message-State: AOJu0YxEQpTOKjbfgGkqyBP/A638M4iBReP+hgZWD/CqLNe8ZjBSNBrs LKVLv7clrMmuK5mB3XWOfN8= X-Google-Smtp-Source: AGHT+IFh2m58yqt3RSg/o96un1ggqGI1jq8zQZvoCW3XNg5TdHmiYRDBByWAUNMdKvoLsoI8mKKFwA== X-Received: by 2002:a5d:930c:0:b0:786:f352:e3d4 with SMTP id l12-20020a5d930c000000b00786f352e3d4mr5539601ion.7.1692858421804; Wed, 23 Aug 2023 23:27:01 -0700 (PDT) Received: from snowbird (c-73-228-235-230.hsd1.mn.comcast.net. [73.228.235.230]) by smtp.gmail.com with ESMTPSA id l17-20020a5e8811000000b007871aa2f144sm3559487ioj.16.2023.08.23.23.27.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 23:27:01 -0700 (PDT) Date: Wed, 23 Aug 2023 23:26:59 -0700 From: Dennis Zhou To: Mateusz Guzik Cc: linux-kernel@vger.kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, vegard.nossum@oracle.com, linux-mm@kvack.org Subject: Re: [PATCH v3 1/2] pcpcntr: add group allocation/free Message-ID: References: <20230823050609.2228718-1-mjguzik@gmail.com> <20230823050609.2228718-2-mjguzik@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230823050609.2228718-2-mjguzik@gmail.com> X-Rspamd-Queue-Id: BB20AA000F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 6nbz7o4h6d7m9gkrmzic3zsmza4fqd4a X-HE-Tag: 1692858422-636798 X-HE-Meta: U2FsdGVkX19nx3OufAK2mvIMjKtEaxIg8oXta7Y5BeiI0hm0ivuljSNzZMKrds+mmIvT1Cd1w+g/s7ZZxP3K6NqlFsM2wmOXfKkHDIuONMdALgMmuYl5XJT7eDCA9Bo+EQerh+EUgkaRInXCfESEzKmyodqe2oSs+ELGQ8ULsgqdVBFSCpKFIbj6NGVsIvKC9UKCKkV4RnpV06wLTD23k7g1qBlf7VrNbBhNMJYisXMWhEYHOSAtRjtVaZzCadx76deA3l7w1q4WBJtVzeeHCjnXAn56IfmPWn7Gxkp95NDu/izNjbiuu5pCdNMA9yq6z9E6hGJgsGEPNYd1y3CmpSia/PeNOGYu/vWbO3TkyM9zmHRRB3UJUJL78UEL8Gu4XtA4k/3yf6LFNyq7v/6HHuGQvKD6SAaYHrCDKAAQBaSoraavSsj/CGoxfeCAYgD6O3PtQ+wIVAJ6/8oWBqYKFFGwiVKzy2wd8F9YI+tUSSL6f4Y/8fMH1O3OB5uOrp6TZhsLd6KwKBADEkIin6SX5CQjuoC8fSaRPrm0vlk1MUf/QM/t7YgtnCGHlfFGQQ9f2GWcEAgnrx+AVS2B5i3RZgXXf+g62nPCZEuWQ0fPoAnT4rIxg9rmuNrq4arxk90CI4X5zCiAN7/BNYce/AAJuBOVMMul9mcqCy8XWXBO4/SVdGqpiVVkOehSJIXR3R+kcxd8WbIY83GgGhBJ6GwCrf+iNYd6RY63K8URv7jCRn9O+3wm11WwKvby3Lq6J0CRSeF5L7KnYm0NvvfsS9AjglGlrMyLm19/AU3S4ATlHv7mwzlvpnbo4BGXHcRFh96Pc4B9XoXEShfwBJceuhfa9/Qk+nNGPwwymkdTva/la73yp8x2PUgTt/rW3A3vln81xFL0O10Y9PkDKs2YBGSNRSRdsm+s7UtgPCcjLyXdOCCuGdUyQeUEUyrQB6kqZobd9Nej0LutlMJbJKVvKf7 HP7zs0wb x0sEYCRoZKAwRh+/jKkaVLDR+5b2fzmgGViVELzICRX2XnhOR8dbqGfThAm4p/ftG+prKc4602Tb/Uri4ea4j2gJEJvPFTmLwe8LUwybjjRDWdPyCkHB2mtNcPR6wI1eQaOwb/eOcLcfQ3fSWKIp7vSiIYQ1RYa/SVmc92jgH4+pvZnAt5BAfG5JgV/TRddpm7hGgzhLFN5AZwwamm10m/fozGRwXyL2nBd+Tb+9t/nD6d0jgcBjNNS0/ztbkHbo4g2Rt2qjoYvkSwJJMBgi9iS2gDs9smf3qas9X4OLFUDhsPnBvrtbxwWmSgKQ3xyI2aP0WEsVymW41/RnC3snpX09Er52AOPdbC0+we6Z+dm8H3wZ27QB83hXuda4IaPvGQOnfwyCPL11coWxZnAw4+7qoLtEj11B8aajdSIwIAxDK5oRHqFtiIQTWK4KfdEjKzClhGvnCB2Mep/q6EngenABIVqtoy7riM1JDls3n9YKh9Z0IO20cQV3Bz+oIQowfePE/wG4q3SYiYCv3GhRiObwVn39CtyvWaVfd+km+2x+SANE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Aug 23, 2023 at 07:06:08AM +0200, Mateusz Guzik wrote: > Allocations and frees are globally serialized on the pcpu lock (and the > CPU hotplug lock if enabled, which is the case on Debian). > > At least one frequent consumer allocates 4 back-to-back counters (and > frees them in the same manner), exacerbating the problem. > > While this does not fully remedy scalability issues, it is a step > towards that goal and provides immediate relief. > > Signed-off-by: Mateusz Guzik I'm happy with this. There are a few minor reflow of lines that I'd like to do but other than that nice. If there are no other comments and it's okay with Andrew I'll pick this up tomorrow for-6.6 and the corresponding changes to fork.c. Reviewed-by: Dennis Zhou Thanks, Dennis > --- > include/linux/percpu_counter.h | 39 ++++++++++++++++++---- > lib/percpu_counter.c | 61 +++++++++++++++++++++++----------- > 2 files changed, 74 insertions(+), 26 deletions(-) > > diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h > index 75b73c83bc9d..f1e7c987e3d3 100644 > --- a/include/linux/percpu_counter.h > +++ b/include/linux/percpu_counter.h > @@ -30,17 +30,27 @@ struct percpu_counter { > > extern int percpu_counter_batch; > > -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, > - struct lock_class_key *key); > +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, > + u32 nr_counters, struct lock_class_key *key); > > -#define percpu_counter_init(fbc, value, gfp) \ > +#define percpu_counter_init_many(fbc, value, gfp, nr_counters) \ > ({ \ > static struct lock_class_key __key; \ > \ > - __percpu_counter_init(fbc, value, gfp, &__key); \ > + __percpu_counter_init_many(fbc, value, gfp, nr_counters,\ > + &__key); \ > }) > > -void percpu_counter_destroy(struct percpu_counter *fbc); > + > +#define percpu_counter_init(fbc, value, gfp) \ > + percpu_counter_init_many(fbc, value, gfp, 1) > + > +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters); > +static inline void percpu_counter_destroy(struct percpu_counter *fbc) > +{ > + percpu_counter_destroy_many(fbc, 1); > +} > + > void percpu_counter_set(struct percpu_counter *fbc, s64 amount); > void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, > s32 batch); > @@ -116,11 +126,26 @@ struct percpu_counter { > s64 count; > }; > > +static inline int percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, > + gfp_t gfp, u32 nr_counters) > +{ > + u32 i; > + > + for (i = 0; i < nr_counters; i++) > + fbc[i].count = amount; > + > + return 0; > +} > + > static inline int percpu_counter_init(struct percpu_counter *fbc, s64 amount, > gfp_t gfp) > { > - fbc->count = amount; > - return 0; > + return percpu_counter_init_many(fbc, amount, gfp, 1); > +} > + > +static inline void percpu_counter_destroy_many(struct percpu_counter *fbc, > + u32 nr_counters) > +{ > } > > static inline void percpu_counter_destroy(struct percpu_counter *fbc) > diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c > index 5004463c4f9f..9338b27f1cdd 100644 > --- a/lib/percpu_counter.c > +++ b/lib/percpu_counter.c > @@ -151,48 +151,71 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) > } > EXPORT_SYMBOL(__percpu_counter_sum); > > -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, > - struct lock_class_key *key) > +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, > + u32 nr_counters, struct lock_class_key *key) > { > unsigned long flags __maybe_unused; > - > - raw_spin_lock_init(&fbc->lock); > - lockdep_set_class(&fbc->lock, key); > - fbc->count = amount; > - fbc->counters = alloc_percpu_gfp(s32, gfp); > - if (!fbc->counters) > + size_t counter_size; > + s32 __percpu *counters; > + u32 i; > + > + counter_size = ALIGN(sizeof(*counters), __alignof__(*counters)); > + counters = __alloc_percpu_gfp(nr_counters * counter_size, > + __alignof__(*counters), gfp); > + if (!counters) { > + fbc[0].counters = NULL; > return -ENOMEM; > + } > > - debug_percpu_counter_activate(fbc); > + for (i = 0; i < nr_counters; i++) { > + raw_spin_lock_init(&fbc[i].lock); > + lockdep_set_class(&fbc[i].lock, key); > +#ifdef CONFIG_HOTPLUG_CPU > + INIT_LIST_HEAD(&fbc[i].list); > +#endif > + fbc[i].count = amount; > + fbc[i].counters = (void *)counters + (i * counter_size); > + > + debug_percpu_counter_activate(&fbc[i]); > + } > > #ifdef CONFIG_HOTPLUG_CPU > - INIT_LIST_HEAD(&fbc->list); > spin_lock_irqsave(&percpu_counters_lock, flags); > - list_add(&fbc->list, &percpu_counters); > + for (i = 0; i < nr_counters; i++) > + list_add(&fbc[i].list, &percpu_counters); > spin_unlock_irqrestore(&percpu_counters_lock, flags); > #endif > return 0; > } > -EXPORT_SYMBOL(__percpu_counter_init); > +EXPORT_SYMBOL(__percpu_counter_init_many); > > -void percpu_counter_destroy(struct percpu_counter *fbc) > +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters) > { > unsigned long flags __maybe_unused; > + u32 i; > + > + if (WARN_ON_ONCE(!fbc)) > + return; > > - if (!fbc->counters) > + if (!fbc[0].counters) > return; > > - debug_percpu_counter_deactivate(fbc); > + for (i = 0; i < nr_counters; i++) > + debug_percpu_counter_deactivate(&fbc[i]); > > #ifdef CONFIG_HOTPLUG_CPU > spin_lock_irqsave(&percpu_counters_lock, flags); > - list_del(&fbc->list); > + for (i = 0; i < nr_counters; i++) > + list_del(&fbc[i].list); > spin_unlock_irqrestore(&percpu_counters_lock, flags); > #endif > - free_percpu(fbc->counters); > - fbc->counters = NULL; > + > + free_percpu(fbc[0].counters); > + > + for (i = 0; i < nr_counters; i++) > + fbc[i].counters = NULL; > } > -EXPORT_SYMBOL(percpu_counter_destroy); > +EXPORT_SYMBOL(percpu_counter_destroy_many); > > int percpu_counter_batch __read_mostly = 32; > EXPORT_SYMBOL(percpu_counter_batch); > -- > 2.41.0 >