From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F939C636D4 for ; Fri, 3 Feb 2023 16:55:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAABB6B0078; Fri, 3 Feb 2023 11:55:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B5BF26B007B; Fri, 3 Feb 2023 11:55:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A21F66B007D; Fri, 3 Feb 2023 11:55:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 91D4C6B0078 for ; Fri, 3 Feb 2023 11:55:32 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 60834120855 for ; Fri, 3 Feb 2023 16:55:32 +0000 (UTC) X-FDA: 80426581704.10.8515012 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 95CC2A0020 for ; Fri, 3 Feb 2023 16:55:30 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675443330; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5FF6kwP6rv2JifZc1sfBTCJyxTo9PjVPiO9vgV47ym4=; b=SgwdsCtCxWAvThoRIQwPrfElQZwnsElJPQVU1EQZ8srWJ4ZUgAiQMz9Eddchq4hQwLkXGw J+QPn4Az7HIoeYpiqwltM3J7JXwpLpjPM++JB+s0f7Ug0gFaB/fJdUXMCfIpHOsoo01Efg q32DhKsXC+/MSiUqbq12/DDFZf6zM40= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675443330; a=rsa-sha256; cv=none; b=K6bgsWWR+93jLcNmif6Im5z2JSKLcLELBp/adeKmmQ52KeKEY7Vocot/XTInf4FxrmsWGc f0gXdgIU6+Gq4zxjrMvvzA/JXVGHNpdOO6Nf22CLl4okL48Sc0JiNmdELdxAJtypAgzJGS B1PShPuRTCx9nsiWAKExs7nK+AnziIo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EFDDC14; Fri, 3 Feb 2023 08:56:11 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.90.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 921893F71E; Fri, 3 Feb 2023 08:55:23 -0800 (PST) Date: Fri, 3 Feb 2023 16:55:20 +0000 From: Mark Rutland To: Peter Zijlstra Cc: torvalds@linux-foundation.org, corbet@lwn.net, will@kernel.org, boqun.feng@gmail.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: Re: [PATCH v2 04/10] instrumentation: Wire up cmpxchg128() Message-ID: References: <20230202145030.223740842@infradead.org> <20230202152655.433050731@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230202152655.433050731@infradead.org> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 95CC2A0020 X-Stat-Signature: dspngr7qktoef3w76uzof4465nxza6kx X-Rspam-User: X-HE-Tag: 1675443330-343718 X-HE-Meta: U2FsdGVkX1/YfV+hMdjXDpOrZSGgWv634hxJO4zlG1+aC5R4EWYo7p2fjB+vSfVET+X4u0qMdL6sZycL/wHtJDYmYzWjk0T6eLkRPCMs18S7TSSvY5PbP2J02JFFdYtj0sqVAuUoN01jUIN+wTC6Emt15sEuqPtpWZX+YUzZBKI2l6xpk5Sdj7Q88eqMEyGUqByT2d6DMTr2giTehfKQzJ/8YLglpbK0GQmbsaW38zKocUMa5+oKgX+YmXFFqAJ1HPCzg7CMEJ4VXOiOsYgk8+u7IXa2CBliBhVJxwSFls5gr2osn95yjBgjYyJgWyjfiwSsQLu2QiM42+pJPwcQfK7sEgX3hqoRESOgXrmvQYKPXhJPnG2wZD9OWOJ+QoqVfAfkC7gPUHWkR8MZGz5bVR+irRTI1/PPlML8Q4E1b4BlYxnkTNWvhVfHAwVS136wkdGiBrPvdmRw1UKgBzegce99VWYtcag54YycR2gpmUKsmcgKDmOuZs7JuEhORS0wk2sda/+wNAdkEx6TuK4i9aRGdb6F6GhSfWXWo0WzrEGWNaKM9t1mS5ZxFE2CVrCjJ2YOke4epy2qoDRAzTIj18oIlPayCJANy0ZFVFaTk+6G7Hinb9PIIaD6Vo9KtvhS8d+Er5YtWrRYyrsNi2VVWSglbMINoH0SCn0uOBM0AqOs5CwNmHa6C3YjfgkDCtV4lNtqpVZS/pAXtT87EMLU2RNRzJgKfjFEHyOwwWqi7WMfuNeU1FQRW2MiDSV+fNTtAZsA+FfRBaNEOyOCJArTLWGRBdfl9WdZ/0JdIViRXjLge6RVN0P4/y5J0Eg7czcMuSDV51dcw/Jhbul75GpG/RQU6HyBlchX0/ERYbs9T9Jy4+4x+WNnMUTCIx68S0rIou3Lhh+8S9ZtWXPNhLkljbMnjiXFvTmZ/ykyEJIBtmvv/yAp2IZNSdphjVBzM8YJNsexqKjDRE3jfI0y6/K lM5mcJsT 9QHWeuWpibc4BiSMD+ggPBaA2i8H4OfmsG/oIKxPl1Q5nVpD+icPqY3qFHjn5L6ES7oBaRGpH5gvk9rp9EBaypdkPHG2w0d1yMXuygJaMXS/jt45nP0i2A/gAJLZhkMEd/Ehtuk1UIXeK23ahZX50P3LyyhtFY6eQoJK46/tSSWwidNGWVQUPGsujvy29TLMDdImCAjVcMp6G/7lR3Bt1wcYtvejox8RKNxFHylpbFElKX9YzEB3Jj5vfuwH6ia//v6KlKxgo5y/Fn9c7S6OTj/hPrwV7DY1WQTW3Re+PU2BOum1uYwG42XyhcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 02, 2023 at 03:50:34PM +0100, Peter Zijlstra wrote: > Wire up the cmpxchg128 familty in the atomic wrappery scripts. s/familty/family/ (and s/wrappery/wrapper/ ?) > > These provide the generic cmpxchg128 family of functions from the > arch_ prefixed version, adding explicit instrumentation where needed. > > Signed-off-by: Peter Zijlstra (Intel) Acked-by: Mark Rutland Mark. > --- > include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++++++++++++++++++++- > include/linux/atomic/atomic-instrumented.h | 77 ++++++++++++++++++++++ > scripts/atomic/gen-atomic-fallback.sh | 4 - > scripts/atomic/gen-atomic-instrumented.sh | 4 - > 4 files changed, 174 insertions(+), 6 deletions(-) > > --- a/include/linux/atomic/atomic-arch-fallback.h > +++ b/include/linux/atomic/atomic-arch-fallback.h > @@ -77,6 +77,29 @@ > > #endif /* arch_cmpxchg64_relaxed */ > > +#ifndef arch_cmpxchg128_relaxed > +#define arch_cmpxchg128_acquire arch_cmpxchg128 > +#define arch_cmpxchg128_release arch_cmpxchg128 > +#define arch_cmpxchg128_relaxed arch_cmpxchg128 > +#else /* arch_cmpxchg128_relaxed */ > + > +#ifndef arch_cmpxchg128_acquire > +#define arch_cmpxchg128_acquire(...) \ > + __atomic_op_acquire(arch_cmpxchg128, __VA_ARGS__) > +#endif > + > +#ifndef arch_cmpxchg128_release > +#define arch_cmpxchg128_release(...) \ > + __atomic_op_release(arch_cmpxchg128, __VA_ARGS__) > +#endif > + > +#ifndef arch_cmpxchg128 > +#define arch_cmpxchg128(...) \ > + __atomic_op_fence(arch_cmpxchg128, __VA_ARGS__) > +#endif > + > +#endif /* arch_cmpxchg128_relaxed */ > + > #ifndef arch_try_cmpxchg_relaxed > #ifdef arch_try_cmpxchg > #define arch_try_cmpxchg_acquire arch_try_cmpxchg > @@ -217,6 +240,76 @@ > > #endif /* arch_try_cmpxchg64_relaxed */ > > +#ifndef arch_try_cmpxchg128_relaxed > +#ifdef arch_try_cmpxchg128 > +#define arch_try_cmpxchg128_acquire arch_try_cmpxchg128 > +#define arch_try_cmpxchg128_release arch_try_cmpxchg128 > +#define arch_try_cmpxchg128_relaxed arch_try_cmpxchg128 > +#endif /* arch_try_cmpxchg128 */ > + > +#ifndef arch_try_cmpxchg128 > +#define arch_try_cmpxchg128(_ptr, _oldp, _new) \ > +({ \ > + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > + ___r = arch_cmpxchg128((_ptr), ___o, (_new)); \ > + if (unlikely(___r != ___o)) \ > + *___op = ___r; \ > + likely(___r == ___o); \ > +}) > +#endif /* arch_try_cmpxchg128 */ > + > +#ifndef arch_try_cmpxchg128_acquire > +#define arch_try_cmpxchg128_acquire(_ptr, _oldp, _new) \ > +({ \ > + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > + ___r = arch_cmpxchg128_acquire((_ptr), ___o, (_new)); \ > + if (unlikely(___r != ___o)) \ > + *___op = ___r; \ > + likely(___r == ___o); \ > +}) > +#endif /* arch_try_cmpxchg128_acquire */ > + > +#ifndef arch_try_cmpxchg128_release > +#define arch_try_cmpxchg128_release(_ptr, _oldp, _new) \ > +({ \ > + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > + ___r = arch_cmpxchg128_release((_ptr), ___o, (_new)); \ > + if (unlikely(___r != ___o)) \ > + *___op = ___r; \ > + likely(___r == ___o); \ > +}) > +#endif /* arch_try_cmpxchg128_release */ > + > +#ifndef arch_try_cmpxchg128_relaxed > +#define arch_try_cmpxchg128_relaxed(_ptr, _oldp, _new) \ > +({ \ > + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ > + ___r = arch_cmpxchg128_relaxed((_ptr), ___o, (_new)); \ > + if (unlikely(___r != ___o)) \ > + *___op = ___r; \ > + likely(___r == ___o); \ > +}) > +#endif /* arch_try_cmpxchg128_relaxed */ > + > +#else /* arch_try_cmpxchg128_relaxed */ > + > +#ifndef arch_try_cmpxchg128_acquire > +#define arch_try_cmpxchg128_acquire(...) \ > + __atomic_op_acquire(arch_try_cmpxchg128, __VA_ARGS__) > +#endif > + > +#ifndef arch_try_cmpxchg128_release > +#define arch_try_cmpxchg128_release(...) \ > + __atomic_op_release(arch_try_cmpxchg128, __VA_ARGS__) > +#endif > + > +#ifndef arch_try_cmpxchg128 > +#define arch_try_cmpxchg128(...) \ > + __atomic_op_fence(arch_try_cmpxchg128, __VA_ARGS__) > +#endif > + > +#endif /* arch_try_cmpxchg128_relaxed */ > + > #ifndef arch_atomic_read_acquire > static __always_inline int > arch_atomic_read_acquire(const atomic_t *v) > @@ -2456,4 +2549,4 @@ arch_atomic64_dec_if_positive(atomic64_t > #endif > > #endif /* _LINUX_ATOMIC_FALLBACK_H */ > -// b5e87bdd5ede61470c29f7a7e4de781af3770f09 > +// 46357a526de89c762d30fb238f35a7d5950a670b > --- a/include/linux/atomic/atomic-instrumented.h > +++ b/include/linux/atomic/atomic-instrumented.h > @@ -1968,6 +1968,36 @@ atomic_long_dec_if_positive(atomic_long_ > arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \ > }) > > +#define cmpxchg128(ptr, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + kcsan_mb(); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + arch_cmpxchg128(__ai_ptr, __VA_ARGS__); \ > +}) > + > +#define cmpxchg128_acquire(ptr, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + arch_cmpxchg128_acquire(__ai_ptr, __VA_ARGS__); \ > +}) > + > +#define cmpxchg128_release(ptr, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + kcsan_release(); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + arch_cmpxchg128_release(__ai_ptr, __VA_ARGS__); \ > +}) > + > +#define cmpxchg128_relaxed(ptr, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + arch_cmpxchg128_relaxed(__ai_ptr, __VA_ARGS__); \ > +}) > + > #define try_cmpxchg(ptr, oldp, ...) \ > ({ \ > typeof(ptr) __ai_ptr = (ptr); \ > @@ -2044,6 +2074,44 @@ atomic_long_dec_if_positive(atomic_long_ > arch_try_cmpxchg64_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > }) > > +#define try_cmpxchg128(ptr, oldp, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + typeof(oldp) __ai_oldp = (oldp); \ > + kcsan_mb(); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > + arch_try_cmpxchg128(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > +}) > + > +#define try_cmpxchg128_acquire(ptr, oldp, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + typeof(oldp) __ai_oldp = (oldp); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > + arch_try_cmpxchg128_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > +}) > + > +#define try_cmpxchg128_release(ptr, oldp, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + typeof(oldp) __ai_oldp = (oldp); \ > + kcsan_release(); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > + arch_try_cmpxchg128_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > +}) > + > +#define try_cmpxchg128_relaxed(ptr, oldp, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + typeof(oldp) __ai_oldp = (oldp); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ > + arch_try_cmpxchg128_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ > +}) > + > #define cmpxchg_local(ptr, ...) \ > ({ \ > typeof(ptr) __ai_ptr = (ptr); \ > @@ -2058,6 +2126,13 @@ atomic_long_dec_if_positive(atomic_long_ > arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \ > }) > > +#define cmpxchg128_local(ptr, ...) \ > +({ \ > + typeof(ptr) __ai_ptr = (ptr); \ > + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ > + arch_cmpxchg128_local(__ai_ptr, __VA_ARGS__); \ > +}) > + > #define sync_cmpxchg(ptr, ...) \ > ({ \ > typeof(ptr) __ai_ptr = (ptr); \ > @@ -2083,4 +2158,4 @@ atomic_long_dec_if_positive(atomic_long_ > }) > > #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ > -// 764f741eb77a7ad565dc8d99ce2837d5542e8aee > +// 27320c1ec2bf2878ecb9df3ea4816a7bc0c57a52 > --- a/scripts/atomic/gen-atomic-fallback.sh > +++ b/scripts/atomic/gen-atomic-fallback.sh > @@ -217,11 +217,11 @@ cat << EOF > > EOF > > -for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64"; do > +for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64" "arch_cmpxchg128"; do > gen_xchg_fallbacks "${xchg}" > done > > -for cmpxchg in "cmpxchg" "cmpxchg64"; do > +for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do > gen_try_cmpxchg_fallbacks "${cmpxchg}" > done > > --- a/scripts/atomic/gen-atomic-instrumented.sh > +++ b/scripts/atomic/gen-atomic-instrumented.sh > @@ -166,14 +166,14 @@ grep '^[a-z]' "$1" | while read name met > done > > > -for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do > +for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do > for order in "" "_acquire" "_release" "_relaxed"; do > gen_xchg "${xchg}" "${order}" "" > printf "\n" > done > done > > -for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg"; do > +for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg"; do > gen_xchg "${xchg}" "" "" > printf "\n" > done > >