From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2E2BC53210 for ; Tue, 3 Jan 2023 13:25:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D0C98E0002; Tue, 3 Jan 2023 08:25:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 081158E0001; Tue, 3 Jan 2023 08:25:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB1748E0002; Tue, 3 Jan 2023 08:25:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E00528E0001 for ; Tue, 3 Jan 2023 08:25:50 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B9D1FA0A51 for ; Tue, 3 Jan 2023 13:25:50 +0000 (UTC) X-FDA: 80313560460.06.8B28F80 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id ADD4D80012 for ; Tue, 3 Jan 2023 13:25:47 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672752348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZOcPhJQ5+BIO1TRm2b9Bv+0M39sSYglc7SRJvs0HwuE=; b=VvT4ZIqcq/GaTw7wbezVdzPf0UQBIMIkZdu/g1xI2uOJBFnxSXE9XwXSAysDeJqJTLdMII PStuqVM+kEy63ahXxhEEwZ/l7wRc0ax+VclDKebhLOmgWilIXc1vM9+94jdvPfdrcRv3QZ xeR1wpOJqeUIFBUtdh5pojJgjjERIEY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672752348; a=rsa-sha256; cv=none; b=tON3cUtYhxyU3G11ySzOJ1PQH4tHl6fJD1XhCsNP+iuoltUEjHxyou5txXe/+WhE3gomIq MKwQ62S+KWk7bkRZ2X7N28eu6lF7cgMkw/Jf2lAgh5/yy9Lz4QzAN86Pf6P7Xeiotqf70h tuK8Hmy93uiCeBGha9x6hatmuRXqLrM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4809B2F4; Tue, 3 Jan 2023 05:26:28 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.37.13]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CA5683F587; Tue, 3 Jan 2023 05:25:40 -0800 (PST) Date: Tue, 3 Jan 2023 13:25:35 +0000 From: Mark Rutland To: Peter Zijlstra Cc: Boqun Feng , torvalds@linux-foundation.org, corbet@lwn.net, will@kernel.org, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, Herbert Xu , davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org Subject: Re: [RFC][PATCH 05/12] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() Message-ID: References: <20221219153525.632521981@infradead.org> <20221219154119.154045458@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: ADD4D80012 X-Rspam-User: X-Stat-Signature: xtrewjwnfc3h89q5g7chs9tpr7mfcqme X-HE-Tag: 1672752347-862186 X-HE-Meta: U2FsdGVkX18Ksebpf3ZmMvTOT34FWDllL76JxasDRBnYtNm4i6ac279ZZBmjfePZxLHgil6u7MkMr+JiNV30gKYDZ0kV0OAYFyU7qVsbsG+XgAGmKwoES6JvUOReH29lKiexp+7XwBsV1ao0NtjalV2rWCj3R1+hf+EDrztTdUU8WX4TzWLSLWbsiT8wm/wedtxs9ILMeUyhzSlqlXzMfsEbQb/lmveP0Z+xZcBU7IhuNbMtD65qs30mabMd6IrOLhVjihuk6lLJgn8qlHHXru+Ryagq6tuCkTHVUNB9LjPJpSAAvBcvriPTnbaxAHMf62Nxm/MPbt2GPQ+IuH9imXhR5HrGJQ/FOxIUoCOdALiygzyGNa8YFPoKTfsFG47PlVHwNq35SSXITkTg5+adiK1o3VUHNhIPY3D5yvS1XsVXi/J8KGct+xIG93tLb0B+5nHMN/wmwrsvMA3TO7Uvr5JlxO8q7lA53ShpmkNKm4J9jXoWDQHBc38VH6WX99YQ9kLnRpJS8wKLHjidknGgNrAM7OIhIdwYmOzvuS8ACnti/u8FG9jYWTGeGd+rjn/1fnVSnVUbXEVcnwB/MnKBcZsjhEMz940sSG93YLJdGumscGt00i62HIW38u/g7Iha8MZbHbghCwQWkS7+CfV3C/IJckX/W14ERnB93BxNifA2cGcWTcttkeWjNivfHU0z+/ejA3+pCXJf9kmiBrEzpnQK8JAPrnRxywPiTJzroMXqJtQd9BsaTOEyca9GVKyO38jomK9h4+V1Azo0RQmDeVUJFJxeICpDaOJi2ldRARNRC78+2vMjcMdrbnSkVviUe/4rF7TMkusgD9sbinHhT2yyEK87Ln9sRAZRUDI5hqiHgoMADT5mH/C8q583nR+sRr68E5co6sflFsHpVe35mGkLeWLJSJPF2TObl6rBRyqWuwDAquL4I/eKvKjDJCWP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 20, 2022 at 12:08:16PM +0100, Peter Zijlstra wrote: > On Mon, Dec 19, 2022 at 12:07:25PM -0800, Boqun Feng wrote: > > On Mon, Dec 19, 2022 at 04:35:30PM +0100, Peter Zijlstra wrote: > > > For all architectures that currently support cmpxchg_double() > > > implement the cmpxchg128() family of functions that is basically the > > > same but with a saner interface. > > > > > > Signed-off-by: Peter Zijlstra (Intel) > > > --- > > > arch/arm64/include/asm/atomic_ll_sc.h | 38 +++++++++++++++++++++++ > > > arch/arm64/include/asm/atomic_lse.h | 33 +++++++++++++++++++- > > > arch/arm64/include/asm/cmpxchg.h | 26 ++++++++++++++++ > > > arch/s390/include/asm/cmpxchg.h | 33 ++++++++++++++++++++ > > > arch/x86/include/asm/cmpxchg_32.h | 3 + > > > arch/x86/include/asm/cmpxchg_64.h | 55 +++++++++++++++++++++++++++++++++- > > > 6 files changed, 185 insertions(+), 3 deletions(-) > > > > > > --- a/arch/arm64/include/asm/atomic_ll_sc.h > > > +++ b/arch/arm64/include/asm/atomic_ll_sc.h > > > @@ -326,6 +326,44 @@ __CMPXCHG_DBL( , , , ) > > > __CMPXCHG_DBL(_mb, dmb ish, l, "memory") > > > > > > #undef __CMPXCHG_DBL > > > + > > > +union __u128_halves { > > > + u128 full; > > > + struct { > > > + u64 low, high; > > > + }; > > > +}; > > > + > > > +#define __CMPXCHG128(name, mb, rel, cl) \ > > > +static __always_inline u128 \ > > > +__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ > > > +{ \ > > > + union __u128_halves r, o = { .full = (old) }, \ > > > + n = { .full = (new) }; \ > > > + \ > > > + asm volatile("// __cmpxchg128" #name "\n" \ > > > + " prfm pstl1strm, %2\n" \ > > > + "1: ldxp %0, %1, %2\n" \ > > > + " eor %3, %0, %3\n" \ > > > + " eor %4, %1, %4\n" \ > > > + " orr %3, %4, %3\n" \ > > > + " cbnz %3, 2f\n" \ > > > + " st" #rel "xp %w3, %5, %6, %2\n" \ > > > + " cbnz %w3, 1b\n" \ > > > + " " #mb "\n" \ > > > + "2:" \ > > > + : "=&r" (r.low), "=&r" (r.high), "+Q" (*(unsigned long *)ptr) \ > > > > I wonder whether we should use "(*(u128 *)ptr)" instead of "(*(unsigned > > long *) ptr)"? Because compilers may think only 64bit value pointed by > > "ptr" gets modified, and they are allowed to do "useful" optimization. > > In this I've copied the existing cmpxchg_double() code; I'll have to let > the arch folks speak here, I've no clue. We definitely need to ensure the compiler sees we poke the whole thing, or it can get this horribly wrong, so that is a latent bug. See commit: fee960bed5e857eb ("arm64: xchg: hazard against entire exchange variable") ... for examples of GCC being clever, where I overlooked the *_double() cases. I'll go spin a quick fix for that so that we can have something go to stable before we rejig the API. Mark.