From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.7 required=3.0 tests=BAYES_00, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D236C07E95 for ; Wed, 7 Jul 2021 06:33:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 665E261CAA for ; Wed, 7 Jul 2021 06:33:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 665E261CAA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=sina.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CC33A6B0036; Wed, 7 Jul 2021 02:33:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C72E46B005D; Wed, 7 Jul 2021 02:33:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B151E6B006C; Wed, 7 Jul 2021 02:33:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id 7CA916B0036 for ; Wed, 7 Jul 2021 02:33:15 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C21571ADA8 for ; Wed, 7 Jul 2021 06:33:14 +0000 (UTC) X-FDA: 78334824708.07.1A92DAA Received: from mail3-164.sinamail.sina.com.cn (mail3-164.sinamail.sina.com.cn [202.108.3.164]) by imf08.hostedemail.com (Postfix) with SMTP id 9A6FF30000A5 for ; Wed, 7 Jul 2021 06:33:12 +0000 (UTC) Received: from unknown (HELO localhost.localdomain)([1.24.238.70]) by sina.com (172.16.97.35) with ESMTP id 60E54AA4000132A8; Wed, 7 Jul 2021 14:33:10 +0800 (CST) X-Sender: hdanton@sina.com X-Auth-ID: hdanton@sina.com X-SMAIL-MID: 31270815073901 From: Hillf Danton To: Vlastimil Babka Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sebastian Andrzej Siewior Subject: Re: [RFC v2 29/34] mm: slub: Move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context Date: Wed, 7 Jul 2021 14:33:04 +0800 Message-Id: <20210707063304.2074-1-hdanton@sina.com> In-Reply-To: <20210609113903.1421-30-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of hdanton@sina.com designates 202.108.3.164 as permitted sender) smtp.mailfrom=hdanton@sina.com; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9A6FF30000A5 X-Stat-Signature: mcry8fr5dedj7y9bo7kb9zi158wnczmq X-HE-Tag: 1625639592-304701 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 9 Jun 2021 13:38:58 +0200 >From: Sebastian Andrzej Siewior > >flush_all() flushes a specific SLAB cache on each CPU (where the cache >is present). The deactivate_slab()/__free_slab() invocation happens >within IPI handler and is problematic for PREEMPT_RT. > >The flush operation is not a frequent operation or a hot path. The >per-CPU flush operation can be moved to within a workqueue. > >[vbabka@suse.cz: adapt to new SLUB changes] >Signed-off-by: Sebastian Andrzej Siewior >Signed-off-by: Vlastimil Babka >--- > mm/slub.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 48 insertions(+), 8 deletions(-) > >diff --git a/mm/slub.c b/mm/slub.c >index 5c70fc17e9be..6e747e0d7dcd 100644 >--- a/mm/slub.c >+++ b/mm/slub.c >@@ -2474,33 +2474,73 @@ static inline void __flush_cpu_slab(struct kmem_= c=3D >ache *s, int cpu) > unfreeze_partials_cpu(s, c); > } > >+struct slub_flush_work { >+ struct work_struct work; >+ struct kmem_cache *s; >+ bool skip; >+}; >+ > /* > * Flush cpu slab. > * >- * Called from IPI handler with interrupts disabled. >+ * Called from CPU work handler with migration disabled. > */ The comment muddies a pint more than it could clear up - kworkers do thei= r works without an eye on cpuhp. It is fine just to cut it given the cpus_read_lock() in flush_all(). >-static void flush_cpu_slab(void *d) >+static void flush_cpu_slab(struct work_struct *w) > { >- struct kmem_cache *s =3D d; >- struct kmem_cache_cpu *c =3D this_cpu_ptr(s->cpu_slab); >+ struct kmem_cache *s; >+ struct kmem_cache_cpu *c; >+ struct slub_flush_work *sfw; >+ >+ sfw =3D container_of(w, struct slub_flush_work, work); >+ >+ s =3D sfw->s; >+ c =3D this_cpu_ptr(s->cpu_slab); > > if (c->page) >- flush_slab(s, c, false); >+ flush_slab(s, c, true); > > unfreeze_partials(s); > } > >-static bool has_cpu_slab(int cpu, void *info) >+static bool has_cpu_slab(int cpu, struct kmem_cache *s) > { >- struct kmem_cache *s =3D info; > struct kmem_cache_cpu *c =3D per_cpu_ptr(s->cpu_slab, cpu); > > return c->page || slub_percpu_partial(c); > } > >+static DEFINE_MUTEX(flush_lock); >+static DEFINE_PER_CPU(struct slub_flush_work, slub_flush); >+ > static void flush_all(struct kmem_cache *s) > { >- on_each_cpu_cond(has_cpu_slab, flush_cpu_slab, s, 1); >+ struct slub_flush_work *sfw; >+ unsigned int cpu; >+ >+ cpus_read_lock(); >+ mutex_lock(&flush_lock); >+ >+ for_each_online_cpu(cpu) { >+ sfw =3D &per_cpu(slub_flush, cpu); >+ if (!has_cpu_slab(cpu, s)) { >+ sfw->skip =3D true; >+ continue; >+ } >+ INIT_WORK(&sfw->work, flush_cpu_slab); >+ sfw->skip =3D false; >+ sfw->s =3D s; >+ schedule_work_on(cpu, &sfw->work); >+ } >+ >+ for_each_online_cpu(cpu) { >+ sfw =3D &per_cpu(slub_flush, cpu); >+ if (sfw->skip) >+ continue; >+ flush_work(&sfw->work); >+ } >+ >+ mutex_unlock(&flush_lock); >+ cpus_read_unlock(); > } > > /* >-- >2.31.1