From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF136EE49A5 for ; Wed, 23 Aug 2023 01:14:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 352C6280059; Tue, 22 Aug 2023 21:14:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 303BB940007; Tue, 22 Aug 2023 21:14:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CAB8280059; Tue, 22 Aug 2023 21:14:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0CB69940007 for ; Tue, 22 Aug 2023 21:14:39 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CEAD4160437 for ; Wed, 23 Aug 2023 01:14:38 +0000 (UTC) X-FDA: 81153599436.22.999FC7C Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) by imf18.hostedemail.com (Postfix) with ESMTP id 172351C0008 for ; Wed, 23 Aug 2023 01:14:36 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=P6GEHetF; spf=pass (imf18.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692753277; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=crWxE8XOEcCvVLdDxfeVx0+++iwdu7eHz9mdifeWTUk=; b=V7PX/pjnQ7Mh+dLngosPzAQzt2CBZTcFQBf4mt1fBxAlQmYzPsyBsaQiY1HztrSImplIHp pfZfx58vCN7EohDPrGcDav7LRwP5OBQnXaa12SKDYjVhdTkzuBMZatvUAKOA/zEyBJYv+O G2iWdWoDXQQnrMcirlNuPTQ1q9y2msY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692753277; a=rsa-sha256; cv=none; b=UspFWNBTed7qX1NFTJT14IcaNMJEdG4n7ok2loQQCksSWSzhdpbdsWuEfhyGXkxAdBkejr dvtNrbgFB8t68d4/2N/95LjxwruJywiHHrSD0yAx6i0FEBi4SnyrsgHTMo1cdREANrdQlh 4hMDlYPygtey8Ek0HTWI9t1IF2NVUUk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=P6GEHetF; spf=pass (imf18.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-56e16e4aba7so3218735eaf.2 for ; Tue, 22 Aug 2023 18:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692753276; x=1693358076; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=crWxE8XOEcCvVLdDxfeVx0+++iwdu7eHz9mdifeWTUk=; b=P6GEHetF/kKHFyRe0mUZSg7SdmhzPs46vBYIZhsIW/iOqsi81hjMj59eodVPDInTiS gcyP+ovHSs9riPzkL26OqY6gT8OBakvCb5jHCpOFEu9GNs0uo0SwFndMJX3NAUgQNfmp EtG0G/doO5uIT4CO68SLBTxeob29tWzlzkJrqDekXwUQru5heXYfJsxQSvIDDYH1n1dU oJ6IAbcDbv21/AUD5tduA79hm1TRBBiJ9bx126b2a7X668YqHaP5MN1y9ZC0A9Y92bul uzcNCpdG3+56TuLKQ6j+vAs3X4K/ZS7RrECDfzWsK3Kk6+emWMgtHUv7iNhU0g1JXJDR 5J7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692753276; x=1693358076; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=crWxE8XOEcCvVLdDxfeVx0+++iwdu7eHz9mdifeWTUk=; b=UzKPe4jL4mtBAa4bZR5pPXA0cH1a1kYEHPFQ/Eolxa+LvSKFmnNPQ4OhOskMNkgKh7 Phy5ckr37wfKV7zmCp2bgsDVfZNbGZE+xR1h0Rk8IgC79jOC5Y+61/MYYLtW41HWEfGs MsIyObbYKxXKTIUPo4w55Q8rkiYpKP9JxZhwTOwNW1NQ2GXngOPWvXXGrGeZfDePSK9O WEC8Qa4sBvMarw8uVxUkgL2ZNUZQq0nR89q8JYYcB8gc9bhasNyXSKsVKJr9QR0QA5FD bSNbfn+uBCpkbbQNb8lAj3YS28fju6Su662U3wyPNx8BLadUGh6vCetn26AN0rfnkPXH mm6A== X-Gm-Message-State: AOJu0Yy7NwNImJX2tmrwCECBidjyq5wd8DYjLcgkiVSzo2M2uVxazOty IKnjRWMYzyVOhKPB3SVCPvRyvVgCcXH6dr2UneQ= X-Google-Smtp-Source: AGHT+IEGOpp4iDxWPtESK4MPJRKZRNumCpbWWRQfzijm8v4Ib7IdbLo6M3zGWsWhkucbY4T61zR78xXloz+nt+nS1rA= X-Received: by 2002:a4a:624c:0:b0:566:f8ee:fa67 with SMTP id y12-20020a4a624c000000b00566f8eefa67mr11236070oog.0.1692753276042; Tue, 22 Aug 2023 18:14:36 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ac9:5797:0:b0:4f0:1250:dd51 with HTTP; Tue, 22 Aug 2023 18:14:35 -0700 (PDT) In-Reply-To: <20230822184152.2194558-2-mjguzik@gmail.com> References: <20230822184152.2194558-1-mjguzik@gmail.com> <20230822184152.2194558-2-mjguzik@gmail.com> From: Mateusz Guzik Date: Wed, 23 Aug 2023 03:14:35 +0200 Message-ID: Subject: Re: [PATCH v2 1/2] pcpcntr: add group allocation/free To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org, Mateusz Guzik Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 172351C0008 X-Rspam-User: X-Stat-Signature: hdpgqk4m8gnmmig8mz7bekca9ntywgyf X-Rspamd-Server: rspam03 X-HE-Tag: 1692753276-413096 X-HE-Meta: U2FsdGVkX19GDUU9YR1vU+eCHIyo54y9PEwVkvyjgVOHYYucrcKMjtWIcSNQLlnmMLXB+mR76606k5jhRTW7iJPThEs71ZtaUPt6PTF3v50E5KwnjxO6oOAOHb6UJ4RrBoDfJNOT5yg8XuYciHXPZxfCfarpP73h8KL/YVFXaM9WbGBNHjM/MUkgF6/+P4RmNJg8+TpYyfjqGUNvoStd9mIiGhrVPE/DR1ekRY0Uf3VQNRMGPBSiu0ja1DFNbZSWugnxKak+2H+XVtjl1ShCWAZCCVj32v3Xzev45Zt11nCoiblJ83rIfN5jqKHV9mf0vw+V62FVXNi7knlv+t2g1ZLCylkWohiF4JbGJgnDEvgL6JTErVG55KbFPApM6fm8ow0JOoKNuCRPlESKU4HK5cGMgZwvpYNQ14OniUaHLja3amc9HAMHtA2y9uVP1wQ0ln0Iiq9QhZdwT6YlgoisaQ3MqHPkJlnf2nM00cNDJ10clEOVjNZXi1pVHYGABWAsYn/maRzVCSLwJq/Eo8O3pOUMD4B3RUGbcxxby0L3oKUmUvLRrW7bKeKda8bLSzcloeN0ewNFiSfeYyCKlc/uRi3QUtvwJ3SncAurpjySw/+0Mjp0mlqjAu954PkIhhDILAVZWoh3BWXjs3s9N/t3KMbzmlNd8pkIHyjKeFklVZ07qtMTER8W73JnY2PqL5xIwwle/VitciNXOQ8JQaOwu6HbsQUpQ76d4WLQKxZhgnpJe24dZ21u7r2/JJycsBb6v1M0AwrkzkL2lf9FF4QhxSIk8h6a6YRRuPkGT/EXWbY/tjl8bbkKk1PxjlYMNh19L83c7kQSEoP2bZECOmBItdLnWPUZFdR70Rmbo/qpL3bBPIUCtXatjQ1iwTt8TnZO1HhBcrAPaMajk/QGYGumPhGqsglz9w+lJk0nJMVBR611fErs4eR4NLPk9LUZWTjRX0PPS+Y5OQ2fT7t/Pnj A6Y0Mp92 aE5pr7276a/DEb4djrLdqPEnPrJBoDwJvoskbDKqhfb5jXHvTdTu32TnzecmHf2KNZO7F+LTduip1YTeR3ONKct3R5Nii8I/dbgh89OjJPo1t7fZUZ6rOX4dvFBfkx7rfJO/R4I1pS2i/iAl9lLg3FOjJPQ63MohinL4zeO87OVoWoh/Trslf28E92Jp26VC4TeZpTVAZGLYOkNpy+gTnuCJ2G+3qt3179V18tcF9lrSbqKsrzzMzKiJWGMxmOY/JBWpUKOaL9PT4nfkPlQBSrFJlafV41X2YAkZunDLfyT0fIW8C2Ak3wk/8SdP9TejSe2YeCu422guWiePuCCiQRLiW1O+x9tje1C41XkQvEbWjgGlXppjaXN8azsMCirr0b4x43M8+uMcURT8QhAJPhmrQ7fAEPrUkvda/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 8/22/23, Mateusz Guzik wrote: > Allocations and frees are globally serialized on the pcpu lock (and the > CPU hotplug lock if enabled, which is the case on Debian). > > At least one frequent consumer allocates 4 back-to-back counters (and > frees them in the same manner), exacerbating the problem. > > While this does not fully remedy scalability issues, it is a step > towards that goal and provides immediate relief. > I just found I'm going to have to send a v3 to handle !CONFIG_SMP. Temporarily I can't even compile-test that, so for now I'm just asking if this v2 looks fine (modulo the !smp problem). Sorry for the spam. > Signed-off-by: Mateusz Guzik > --- > include/linux/percpu_counter.h | 20 ++++++++--- > lib/percpu_counter.c | 61 +++++++++++++++++++++++----------- > 2 files changed, 57 insertions(+), 24 deletions(-) > > diff --git a/include/linux/percpu_counter.h > b/include/linux/percpu_counter.h > index 75b73c83bc9d..518a4088b964 100644 > --- a/include/linux/percpu_counter.h > +++ b/include/linux/percpu_counter.h > @@ -30,17 +30,27 @@ struct percpu_counter { > > extern int percpu_counter_batch; > > -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t > gfp, > - struct lock_class_key *key); > +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, > gfp_t gfp, > + u32 nr_counters, struct lock_class_key *key); > > -#define percpu_counter_init(fbc, value, gfp) \ > +#define percpu_counter_init_many(fbc, value, gfp, nr_counters) \ > ({ \ > static struct lock_class_key __key; \ > \ > - __percpu_counter_init(fbc, value, gfp, &__key); \ > + __percpu_counter_init_many(fbc, value, gfp, nr_counters,\ > + &__key); \ > }) > > -void percpu_counter_destroy(struct percpu_counter *fbc); > + > +#define percpu_counter_init(fbc, value, gfp) \ > + percpu_counter_init_many(fbc, value, gfp, 1) > + > +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 > nr_counters); > +static inline void percpu_counter_destroy(struct percpu_counter *fbc) > +{ > + percpu_counter_destroy_many(fbc, 1); > +} > + > void percpu_counter_set(struct percpu_counter *fbc, s64 amount); > void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, > s32 batch); > diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c > index 5004463c4f9f..9338b27f1cdd 100644 > --- a/lib/percpu_counter.c > +++ b/lib/percpu_counter.c > @@ -151,48 +151,71 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) > } > EXPORT_SYMBOL(__percpu_counter_sum); > > -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t > gfp, > - struct lock_class_key *key) > +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, > gfp_t gfp, > + u32 nr_counters, struct lock_class_key *key) > { > unsigned long flags __maybe_unused; > - > - raw_spin_lock_init(&fbc->lock); > - lockdep_set_class(&fbc->lock, key); > - fbc->count = amount; > - fbc->counters = alloc_percpu_gfp(s32, gfp); > - if (!fbc->counters) > + size_t counter_size; > + s32 __percpu *counters; > + u32 i; > + > + counter_size = ALIGN(sizeof(*counters), __alignof__(*counters)); > + counters = __alloc_percpu_gfp(nr_counters * counter_size, > + __alignof__(*counters), gfp); > + if (!counters) { > + fbc[0].counters = NULL; > return -ENOMEM; > + } > > - debug_percpu_counter_activate(fbc); > + for (i = 0; i < nr_counters; i++) { > + raw_spin_lock_init(&fbc[i].lock); > + lockdep_set_class(&fbc[i].lock, key); > +#ifdef CONFIG_HOTPLUG_CPU > + INIT_LIST_HEAD(&fbc[i].list); > +#endif > + fbc[i].count = amount; > + fbc[i].counters = (void *)counters + (i * counter_size); > + > + debug_percpu_counter_activate(&fbc[i]); > + } > > #ifdef CONFIG_HOTPLUG_CPU > - INIT_LIST_HEAD(&fbc->list); > spin_lock_irqsave(&percpu_counters_lock, flags); > - list_add(&fbc->list, &percpu_counters); > + for (i = 0; i < nr_counters; i++) > + list_add(&fbc[i].list, &percpu_counters); > spin_unlock_irqrestore(&percpu_counters_lock, flags); > #endif > return 0; > } > -EXPORT_SYMBOL(__percpu_counter_init); > +EXPORT_SYMBOL(__percpu_counter_init_many); > > -void percpu_counter_destroy(struct percpu_counter *fbc) > +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 > nr_counters) > { > unsigned long flags __maybe_unused; > + u32 i; > + > + if (WARN_ON_ONCE(!fbc)) > + return; > > - if (!fbc->counters) > + if (!fbc[0].counters) > return; > > - debug_percpu_counter_deactivate(fbc); > + for (i = 0; i < nr_counters; i++) > + debug_percpu_counter_deactivate(&fbc[i]); > > #ifdef CONFIG_HOTPLUG_CPU > spin_lock_irqsave(&percpu_counters_lock, flags); > - list_del(&fbc->list); > + for (i = 0; i < nr_counters; i++) > + list_del(&fbc[i].list); > spin_unlock_irqrestore(&percpu_counters_lock, flags); > #endif > - free_percpu(fbc->counters); > - fbc->counters = NULL; > + > + free_percpu(fbc[0].counters); > + > + for (i = 0; i < nr_counters; i++) > + fbc[i].counters = NULL; > } > -EXPORT_SYMBOL(percpu_counter_destroy); > +EXPORT_SYMBOL(percpu_counter_destroy_many); > > int percpu_counter_batch __read_mostly = 32; > EXPORT_SYMBOL(percpu_counter_batch); > -- > 2.39.2 > > -- Mateusz Guzik