From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AD07C28D13 for ; Thu, 25 Aug 2022 08:49:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B319F940008; Thu, 25 Aug 2022 04:49:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE161940007; Thu, 25 Aug 2022 04:49:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A872940008; Thu, 25 Aug 2022 04:49:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8D154940007 for ; Thu, 25 Aug 2022 04:49:13 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5E1431A1AED for ; Thu, 25 Aug 2022 08:49:13 +0000 (UTC) X-FDA: 79837490586.08.7AEC1FE Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf29.hostedemail.com (Postfix) with ESMTP id 93AB7120003 for ; Thu, 25 Aug 2022 08:49:11 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id n65-20020a17090a5ac700b001fbb4fad865so681035pji.1 for ; Thu, 25 Aug 2022 01:49:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc; bh=0l2wxGraUAFhaW858bMWtPfFHB499eyjQhyZRsajDTU=; b=jNgfPFJOn0Xk+nOmTr64WaubV7OTHFEOU0nIZ0+QCZaOkiiDwFDyJAjjd/12zuwj/t mIPi97uS6xKPp/79FQ9jxRBlRa0rJfPQ1RANwA2yPuv319uie8cedQHfUjXyenC79+IU QTppGg1jEpOFhWyPX/FYBCpoU0u8cqv18uJ8C+W83ONzYqI6hXafaufX/94xbCUlqrbn Cw3dSpCdyrM5Aw9PBoekT+g53uaLtl2I+21Yspvu3tLreFdYZB0Aiady2MSLm65nsbCe pH+HPWwVcZ7kbedu6KOJBnHpuC1e/7wbjVm9dpWo3Pm/GjRX5ai/HH5zcd33Nze+n4zr 9B7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc; bh=0l2wxGraUAFhaW858bMWtPfFHB499eyjQhyZRsajDTU=; b=0tAM1zCD2QJEoNAVhq0x9ec0YRpgThTe5u9EkKRO6k8HWHDEwz5XMhejZTtRsNKptE CEnlxpUZ2dX4qkX3VsgEuhh3YqPuWSq2wkQGwpCFYZGYsqnYaQeG+6xCg2i+87k7mgX8 lgxHiUyX/mMeyFSANAi/R8Y5BywBmnRIn/Kd2EfckaVUEJI8r5AybdBIBwSsIOLJv53B y2986LsQUhxSEOv3zqp9lRg2qOgi2N6FRh1QDyuVwnTY5nSgEqYwDgZ41MjcZ3iTGdod eXChhomxiqTseJUG6z6mjWcViKHHy5nAWIgsnj4kjX/TP4len0Fe8n8VF8zQHWSBCyP5 gEYg== X-Gm-Message-State: ACgBeo2ItfhoBC2Ecw7gloUvSYWTPRwd9sGKLMsUgzNZGrTvINk4RIqY ApTI+llJtAieCQQ8Nza51YA= X-Google-Smtp-Source: AA6agR5AnI76xfPqQNp6o26qphP9ZXPS0tXF/KHrqkJIBCQyo18HkMD+IGVFbDqfxwlEMzYMwn/+dw== X-Received: by 2002:a17:902:ea0d:b0:172:ce60:1d4f with SMTP id s13-20020a170902ea0d00b00172ce601d4fmr2811451plg.68.1661417350690; Thu, 25 Aug 2022 01:49:10 -0700 (PDT) Received: from hyeyoo ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id o19-20020aa79793000000b005364944e538sm11060749pfp.99.2022.08.25.01.49.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 01:49:09 -0700 (PDT) Date: Thu, 25 Aug 2022 17:49:03 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Sebastian Andrzej Siewior Cc: Vlastimil Babka , Rongwei Wang , Christoph Lameter , Joonsoo Kim , David Rientjes , Pekka Enberg , Roman Gushchin , linux-mm@kvack.org, Thomas Gleixner , Mike Galbraith , Andrew Morton Subject: Re: [PATCH 6/5] slub: Make PREEMPT_RT support less convoluted Message-ID: References: <20220823170400.26546-1-vbabka@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jNgfPFJO; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661417351; a=rsa-sha256; cv=none; b=qxbrZkqSp34lGNg1fee0QGV9nUvKi/q0r+xqcRda9L/o/Ny88Ir+wM5+szSiXdelmXxmf5 LyfbD6yKuGsTwXaCAky9ckoRBlku1jQ5trpaQDnrxVXvw+01axiJ+0wXdJZmCQdnn5Dsed oriHgCHBdgUij3h8Eu2w4TmIf6LWvTE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661417351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0l2wxGraUAFhaW858bMWtPfFHB499eyjQhyZRsajDTU=; b=VahiJUC3DtCQxM6zEON1io5OvJx8TVHnVIQ65jwqYyxnPtvhKlxdcOkrbNY6Bcmfn4JTYW TkblvNIXc5/p+7fCfM5cLkVHcwSRI2tnZS/GCWZ+TIilXYUx0pfglBbjPX1E3uZSTj8VVy rX1XlnqpCXNbts9BiPgHTDUjeJ1nw9I= X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 93AB7120003 X-Rspam-User: X-Stat-Signature: ps17wh3jyegk3opgihxwd1xrueegkwq6 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jNgfPFJO; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1661417351-78822 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 25, 2022 at 09:51:36AM +0200, Sebastian Andrzej Siewior wrote: > From: Thomas Gleixner > > The slub code already has a few helpers depending on PREEMPT_RT. Add a few > more and get rid of the CONFIG_PREEMPT_RT conditionals all over the place. > > No functional change. > > Signed-off-by: Thomas Gleixner > Cc: Andrew Morton > Cc: Christoph Lameter > Cc: David Rientjes > Cc: Joonsoo Kim > Cc: Pekka Enberg > Cc: Vlastimil Babka > Cc: linux-mm@kvack.org > Signed-off-by: Sebastian Andrzej Siewior > Acked-by: Peter Zijlstra (Intel) > --- > > Vlastimil, does it work for you to include this patch in your series? It > depends now on your series :) It has this USE_LOCKLESS_FAST_PATH() Linus > asked about so we should be good. > > mm/slub.c | 56 ++++++++++++++++++++++++-------------------------------- > 1 file changed, 24 insertions(+), 32 deletions(-) > > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -104,9 +104,11 @@ > * except the stat counters. This is a percpu structure manipulated only by > * the local cpu, so the lock protects against being preempted or interrupted > * by an irq. Fast path operations rely on lockless operations instead. > - * On PREEMPT_RT, the local lock does not actually disable irqs (and thus > - * prevent the lockless operations), so fastpath operations also need to take > - * the lock and are no longer lockless. > + * > + * On PREEMPT_RT, the local lock neither disables interrupts nor preemption > + * which means the lockless fastpath cannot be used as it might interfere with > + * an in-progress slow path operations. In this case the local lock is always > + * taken but it still utilizes the freelist for the common operations. Thank you for correction! > * > * lockless fastpaths > * > @@ -167,8 +169,9 @@ > * function call even on !PREEMPT_RT, use inline preempt_disable() there. > */ > #ifndef CONFIG_PREEMPT_RT > -#define slub_get_cpu_ptr(var) get_cpu_ptr(var) > -#define slub_put_cpu_ptr(var) put_cpu_ptr(var) > +#define slub_get_cpu_ptr(var) get_cpu_ptr(var) > +#define slub_put_cpu_ptr(var) put_cpu_ptr(var) > +#define USE_LOCKLESS_FAST_PATH() (true) > #else > #define slub_get_cpu_ptr(var) \ > ({ \ > @@ -180,6 +183,7 @@ do { \ > (void)(var); \ > migrate_enable(); \ > } while (0) > +#define USE_LOCKLESS_FAST_PATH() (false) > #endif > > #ifdef CONFIG_SLUB_DEBUG > @@ -474,7 +478,7 @@ static inline bool __cmpxchg_double_slab > void *freelist_new, unsigned long counters_new, > const char *n) > { > - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) > + if (USE_LOCKLESS_FAST_PATH()) > lockdep_assert_irqs_disabled(); > #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ > defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) > @@ -3287,14 +3291,8 @@ static __always_inline void *slab_alloc_ > > object = c->freelist; > slab = c->slab; > - /* > - * We cannot use the lockless fastpath on PREEMPT_RT because if a > - * slowpath has taken the local_lock_irqsave(), it is not protected > - * against a fast path operation in an irq handler. So we need to take > - * the slow path which uses local_lock. It is still relatively fast if > - * there is a suitable cpu freelist. > - */ > - if (IS_ENABLED(CONFIG_PREEMPT_RT) || > + > + if (!USE_LOCKLESS_FAST_PATH() || > unlikely(!object || !slab || !node_match(slab, node))) { > object = __slab_alloc(s, gfpflags, node, addr, c); > } else { > @@ -3554,6 +3552,7 @@ static __always_inline void do_slab_free > void *tail_obj = tail ? : head; > struct kmem_cache_cpu *c; > unsigned long tid; > + void **freelist; > > redo: > /* > @@ -3568,9 +3567,13 @@ static __always_inline void do_slab_free > /* Same with comment on barrier() in slab_alloc_node() */ > barrier(); > > - if (likely(slab == c->slab)) { > -#ifndef CONFIG_PREEMPT_RT > - void **freelist = READ_ONCE(c->freelist); > + if (unlikely(slab != c->slab)) { > + __slab_free(s, slab, head, tail_obj, cnt, addr); > + return; > + } > + > + if (USE_LOCKLESS_FAST_PATH()) { > + freelist = READ_ONCE(c->freelist); > > set_freepointer(s, tail_obj, freelist); > > @@ -3582,16 +3585,8 @@ static __always_inline void do_slab_free > note_cmpxchg_failure("slab_free", s, tid); > goto redo; > } > -#else /* CONFIG_PREEMPT_RT */ > - /* > - * We cannot use the lockless fastpath on PREEMPT_RT because if > - * a slowpath has taken the local_lock_irqsave(), it is not > - * protected against a fast path operation in an irq handler. So > - * we need to take the local_lock. We shouldn't simply defer to > - * __slab_free() as that wouldn't use the cpu freelist at all. > - */ > - void **freelist; > - > + } else { > + /* Update the free list under the local lock */ > local_lock(&s->cpu_slab->lock); > c = this_cpu_ptr(s->cpu_slab); > if (unlikely(slab != c->slab)) { > @@ -3606,11 +3601,8 @@ static __always_inline void do_slab_free > c->tid = next_tid(tid); > > local_unlock(&s->cpu_slab->lock); > -#endif > - stat(s, FREE_FASTPATH); > - } else > - __slab_free(s, slab, head, tail_obj, cnt, addr); > - > + } > + stat(s, FREE_FASTPATH); > } > > static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab, I have no strong opinion on its naming, but from view of correctness: Looks good to me. Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> -- Thanks, Hyeonggon