From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49086C7EE23 for ; Tue, 30 May 2023 19:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A022900002; Tue, 30 May 2023 15:33:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8289C6B0074; Tue, 30 May 2023 15:33:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A2E9900002; Tue, 30 May 2023 15:33:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5741F6B0072 for ; Tue, 30 May 2023 15:33:49 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 211E61602EC for ; Tue, 30 May 2023 19:33:49 +0000 (UTC) X-FDA: 80847921378.01.C97C24E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id B69B91A0022 for ; Tue, 30 May 2023 19:33:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JM7SiKdz; dmarc=none; spf=none (imf19.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685475226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4dc8sj3FHzzLhi63BMaL8ARsadXKK6PBMKS8wZ1D+bM=; b=NuXLU/YWGIy7Ma8G4KUz7/LYi17pXq4bCTiCfsAkLg2W83mQbUbv1Zj310kZxM/JbRgVa6 YIF1vhD3GrvcU57DH/JjJUMhgk+KtMlHd2VozANO1/4HLtA5UPq2pJu1z8XESxihoHifqa 0+NfV8igDko2hbcfqNPvgup3kYxakis= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JM7SiKdz; dmarc=none; spf=none (imf19.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685475226; a=rsa-sha256; cv=none; b=VnYsbncJRS/qqudUyaTMrRgOVTwAJdo+fvsN88uk0RKRlz88BdynWAs38SBb8rs54s7cga s8smcfvnnBH49AaA5TRmRJEJ7j3gK37mim34eQa0jIAfwXWhFetuLIc761/qfg7EkJGvks TKy9OVcUXfUe357aBr6sxHnusXFzwHw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=4dc8sj3FHzzLhi63BMaL8ARsadXKK6PBMKS8wZ1D+bM=; b=JM7SiKdzfwoBik0AKgxOsdxCpD sDgnOT4X96GVp8CNJPJb+VOQVteRUg7rUp37Z51bccNB5PtdZZdFWuUdw496KccyYIcrBvzDCr6Bv Y9Jko3S3agioN15y2YYHTWvPLYowFj33IOP2SMXTSAX3jmWiVTemAd+VQ5v3vP1z5MSBp1M6+QMNj Yl95WQRRrZI/coCZDx5VKzQpkvp2WzKa6i9icU6WPABaTXb+MamtUc/8gix4iHb9+PGyAKxTFw0YV PhcQmCHWi2/I64g8MCskmYk+R7GDeCcI2mbko//pOUA83Bdwaq1ULOdq+hdnfRe4a1X3c5leImvPG MPvGP5Ww==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1q455b-006ZKF-17; Tue, 30 May 2023 19:33:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 9784F300233; Tue, 30 May 2023 21:32:58 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id 45294243A9FD8; Tue, 30 May 2023 21:32:58 +0200 (CEST) Date: Tue, 30 May 2023 21:32:58 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org, sfr@canb.auug.org.au, mpe@ellerman.id.au, James.Bottomley@hansenpartnership.com, deller@gmx.de, linux-parisc@vger.kernel.org Subject: Re: [PATCH v3 08/11] slub: Replace cmpxchg_double() Message-ID: <20230530193258.GB211927@hirez.programming.kicks-ass.net> References: <20230515075659.118447996@infradead.org> <20230515080554.453785148@infradead.org> <20230524093246.GP83892@hirez.programming.kicks-ass.net> <20230530142232.GA200270@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230530142232.GA200270@hirez.programming.kicks-ass.net> X-Rspamd-Queue-Id: B69B91A0022 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hmhrr871i3c1et8u8iywuf1tez7yhbfi X-HE-Tag: 1685475225-793751 X-HE-Meta: U2FsdGVkX187qoSJ6U0kKyhGMDzunLCXy1jryt5P3H+BjyXl1EkAYRLCIpgOe6/b2k7cSRshq1/1pQLR80tdvlQ8onEnDDK3prgKND1Ig7RzFWZntmQzIyHONofHTBu/IrUC3WdusfuBSU+uj/lIO9q6tOWO2OxM5x4cBBWeAbvVLrsfYygKqloNCy2Klw3Ao8ESUr8429LPuB2xmfQloJkDYQKDPgmbwZyVLj9V/ICVHGJ8Yuv3e6/tZtYfS9CtadtBXLUQ4JNCQrevzdHNPpWuuZEVy793ET8UsCuXz7HvCP9hwRZocee+zwMspBkezTrOYkJQPMOQgUFoUTwB2R98tDZLQ0YNszOmB/imRQnJUMmz8hhlYdf9+NNEPdRQ9yvm3krjU7GKJ/7JHgceVJzcW5EYV8GdtTQVHt2I8OBF52knxs8d3zyzm7Gim6RdU+8OqK9Cqm+1TRoEAna1qv3DtE9tEqlZCy3zP64/ExRiOZBrJ1AAz9F9/Ic9pefACcMTudysuXSaUTu8s+msiNJBPTiS+gadqwpT1pUxUD6bQHZ6yX+G0vSzHL6Aypwm+TFLol/VuDo5jZ/nMQ/K7gwZWevHILmT0jAkkT2g4zLs2D4tF0zbLViSpo9W4dPlNXFtaDwV50Y0JXt1i8x2gf9fiQ8RenobDdlk9Q4XSqUo5B1pY8u9pWsZARGvEPzm2Jmf7MAkZDa8bl/hylks5w0mp7rBsFlJ2Vp94I9EhXe3Y3e8uk69m0hCVq9LTjoAK7G5FkjswAUncv3S+SlKHQpiot1tu1sBkuwo6h1qoNNLCHvym3S9erg5JzIh6/+UiYwE8gnpolUyihA7gb1Q3PHxaBWT/gWR3eLlnCW84f2qB6bDfOCG7cI3261hxJxGULJvvesP7rIAtz+aKbxLuRaITMEziDDYmo96mjMmNF2Zp6C+AjnI/po/Xx6Ib6yPlkarFD8nd0+3GceAcF2 5pIbQTMn y+kce/en5IGQ1np/2B8yieQ52yfVPr5PXiJMP/iF/W9mhfLsDIlr+RvNNcldjzTV82Uv1QgIhYsGNqig/hKAkA0l7MlrPGNNzpNuKBvyu0IOauRKyHDl3Oo5x3/iHO0biv96BJmoOtZ4V/PIDCiCja/kk00dFlsdix1/CRelf7VTHest3O+BMcPc7kFrj46RfOq49XbLFCDV/unNPtQv4iv14wPrEIc57jUYeBttpztn5ANGmC5mMuzJTsF+0IcjJscWP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 30, 2023 at 04:22:32PM +0200, Peter Zijlstra wrote: > Yet another alternative is using a struct type and an equality function, > just for this. The best I could come up with in the regard is the below. It builds on HPPA64 and x86_64, but I've not ran it yet. (also, the introduction of this_cpu_try_cmpxchg() should probably be split out into its own patch) --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -99,6 +99,15 @@ do { \ __ret; \ }) +#define raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \ +({ \ + typeof(pcp) __ret, __old = *(ovalp); \ + __ret = raw_cpu_cmpxchg(pcp, __old, nval); \ + if (!likely(__ret == __old)) \ + *(ovalp) = __ret; \ + likely(__ret == __old); \ +}) + #define __this_cpu_generic_read_nopreempt(pcp) \ ({ \ typeof(pcp) ___ret; \ @@ -167,6 +176,15 @@ do { \ __ret; \ }) +#define this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \ +({ \ + typeof(pcp) __ret, __old = *(ovalp); \ + __ret = this_cpu_cmpxchg(pcp, __old, nval); \ + if (!likely(__ret == __old)) \ + *(ovalp) = __ret; \ + likely(__ret == __old); \ +}) + #ifndef raw_cpu_read_1 #define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp) #endif @@ -258,6 +276,36 @@ do { \ #define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval) #endif +#ifndef __SIZEOF_INT128__ +#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \ +({ \ + typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \ + typeof(pcp) __ret, __old = *(ovalp); \ + bool __s; \ + __ret = *__p; \ + if (!__builtin_memcmp(&__ret, &__old, sizeof(pcp))) { \ + *__p = nval; \ + __s = true; \ + } else { \ + *(ovalp) = __ret; \ + __s = false; \ + } \ + __s; \ +}) + +#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \ +({ \ + typeof(pcp) __old = (oval); \ + raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \ + __old; \ +}) + +#define raw_cpu_cmpxchg128(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) +#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) +#endif + #ifndef raw_cpu_cmpxchg_1 #define raw_cpu_cmpxchg_1(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) @@ -283,6 +331,31 @@ do { \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_try_cmpxchg_1 +#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef raw_cpu_try_cmpxchg_2 +#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef raw_cpu_try_cmpxchg_4 +#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef raw_cpu_try_cmpxchg_8 +#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef raw_cpu_try_cmpxchg64 +#define raw_cpu_try_cmpxchg64(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef raw_cpu_try_cmpxchg128 +#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif + #ifndef this_cpu_read_1 #define this_cpu_read_1(pcp) this_cpu_generic_read(pcp) #endif @@ -374,6 +447,33 @@ do { \ #define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval) #endif +#ifndef __SIZEOF_INT128__ +#define this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \ +({ \ + bool __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_cmpxchg128(pcp, oval, nval) \ + this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) +#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) +#endif + #ifndef this_cpu_cmpxchg_1 #define this_cpu_cmpxchg_1(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) @@ -399,4 +499,29 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_try_cmpxchg_1 +#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef this_cpu_try_cmpxchg_2 +#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef this_cpu_try_cmpxchg_4 +#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef this_cpu_try_cmpxchg_8 +#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef this_cpu_try_cmpxchg64 +#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif +#ifndef this_cpu_try_cmpxchg128 +#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) +#endif + #endif /* _ASM_GENERIC_PERCPU_H_ */ --- a/include/linux/types.h +++ b/include/linux/types.h @@ -13,6 +13,13 @@ #ifdef __SIZEOF_INT128__ typedef __s128 s128; typedef __u128 u128; +#else +#ifdef CONFIG_64BIT +/* hack for this_cpu_cmpxchg128 */ +typedef struct { + u64 a, b; +} u128 __attribute__((aligned(16))); +#endif #endif typedef u32 __kernel_dev_t; --- a/mm/slab.h +++ b/mm/slab.h @@ -11,14 +11,14 @@ void __init kmem_cache_init(void); # define system_has_freelist_aba() system_has_cmpxchg128() # define try_cmpxchg_freelist try_cmpxchg128 # endif -#define this_cpu_cmpxchg_freelist this_cpu_cmpxchg128 +#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg128 typedef u128 freelist_full_t; #else /* CONFIG_64BIT */ # ifdef system_has_cmpxchg64 # define system_has_freelist_aba() system_has_cmpxchg64() # define try_cmpxchg_freelist try_cmpxchg64 # endif -#define this_cpu_cmpxchg_freelist this_cpu_cmpxchg64 +#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg64 typedef u64 freelist_full_t; #endif /* CONFIG_64BIT */ --- a/mm/slub.c +++ b/mm/slub.c @@ -3037,8 +3037,8 @@ __update_cpu_freelist_fast(struct kmem_c freelist_aba_t old = { .freelist = freelist_old, .counter = tid }; freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) }; - return this_cpu_cmpxchg_freelist(s->cpu_slab->freelist_tid.full, - old.full, new.full) == old.full; + return this_cpu_try_cmpxchg_freelist(s->cpu_slab->freelist_tid.full, + &old.full, new.full); } /*