From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2601ECF6491 for ; Sat, 28 Sep 2024 15:34:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADFD56B0248; Sat, 28 Sep 2024 11:34:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A68FF6B0249; Sat, 28 Sep 2024 11:34:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E2A56B024A; Sat, 28 Sep 2024 11:34:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6CC436B0248 for ; Sat, 28 Sep 2024 11:34:24 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EB394C1ABC for ; Sat, 28 Sep 2024 15:34:23 +0000 (UTC) X-FDA: 82614543606.17.E4E5849 Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf30.hostedemail.com (Postfix) with ESMTP id 462088000B for ; Sat, 28 Sep 2024 15:34:22 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b="NF/A0oUi"; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf30.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727537624; a=rsa-sha256; cv=none; b=EsZypNZqjKKrjuAWPopiTIRKsTN3rlVAivGEfBgkLf+W3451CDMzkr+cuLbqWgbWQZOVVe NrmJxQnARVXsRWbLvH6idLyZkyVx8RM/55dfH59/ksTgO/yjg7exV4RzlCL7TiIkHicMmU zJPFpYwqaHrCxUTAk8bfEN1w0vRPIhQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b="NF/A0oUi"; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf30.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727537624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ipmdv0ODROc2llCqaWdBCJcKeDaq+k+DnY3Bxneei7c=; b=kJZQsMgoyE+lUwb4WM83OVrLLZyWDYEto12sJw2l6vNvMXw15n87E5ThzvLtuN6elfE0U3 5frgbSdVOwHfgzQPmp7YisDY9kE9L3wVQ8jxFjquJvpdyz3oPSdpbVOe68ArcONWT2v86C 90crZsOKSThisoe3bmaGkdTpSNskWnI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1727537661; bh=NrUURg2qa5F4rEbHXvtVlrKaewXw7D0H1TL5+6bR/BU=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=NF/A0oUi2fivzhVq8qk1HW0xQWMmESJB+bE8UZVx7MDYOXUurFEERhAxRmPdq3pbz qbxLT/STRrsBJbc2g9m4IN6Mf9V2mbNbTELP/GL39LhGl2Nbp/82qvFO51ZfEsQzrZ X1Ik/cURKydSKptIhnF/tXpVSTazDJxL6r2rbdNypCGizof3w8VoQ7YtFqf7zF+3of sOYjkof8FBDRMFBEn54n4XRgcIvzMTFYK3CVMzsx4d9nLxx0+xkBiQRArCHOH5O4cq 9ykDHlawYDN7tiNYjefuumIXHAbBYbMWT/f5eYCvikR/M9IacdUARUCBP3ycR9rmra ZbsL+jcCGiIQA== Received: from [IPV6:2606:6d00:100:4000:cacb:9855:de1f:ded2] (unknown [IPv6:2606:6d00:100:4000:cacb:9855:de1f:ded2]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4XGBG50P8Tz1cN3; Sat, 28 Sep 2024 11:34:21 -0400 (EDT) Message-ID: <2091628c-2d96-4492-99d9-0f6a61b08d1d@efficios.com> Date: Sat, 28 Sep 2024 11:32:18 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/2] compiler.h: Introduce ptr_eq() to preserve address dependency To: Alan Stern Cc: Linus Torvalds , linux-kernel@vger.kernel.org, Greg Kroah-Hartman , Sebastian Andrzej Siewior , "Paul E. McKenney" , Will Deacon , Peter Zijlstra , Boqun Feng , John Stultz , Neeraj Upadhyay , Frederic Weisbecker , Joel Fernandes , Josh Triplett , Uladzislau Rezki , Steven Rostedt , Lai Jiangshan , Zqiang , Ingo Molnar , Waiman Long , Mark Rutland , Thomas Gleixner , Vlastimil Babka , maged.michael@gmail.com, Mateusz Guzik , Gary Guo , Jonas Oberhauser , rcu@vger.kernel.org, linux-mm@kvack.org, lkmm@lists.linux.dev References: <20240928135128.991110-1-mathieu.desnoyers@efficios.com> <20240928135128.991110-2-mathieu.desnoyers@efficios.com> <02c63e79-ec8c-4d6a-9fcf-75f0e67ea242@rowland.harvard.edu> From: Mathieu Desnoyers Content-Language: en-US In-Reply-To: <02c63e79-ec8c-4d6a-9fcf-75f0e67ea242@rowland.harvard.edu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 462088000B X-Rspamd-Server: rspam01 X-Stat-Signature: sck5pwkk76gizmhoiyfr3mkfs955b611 X-HE-Tag: 1727537662-354228 X-HE-Meta: U2FsdGVkX19CPT9E0Bh+OITBMACHfA2qg3kFWdCW3g/svT6JUkO4lHaJY8EpnrYb6IJKPaTsyiGInLzTt4Lc93pNgDTVwPjRkDODJkA9X8a3xfrFhO7ka3vndPI6NYDRoFyrAjYdswktRNkYAIw4n5B9G3v9rTKpuntMsVcJnuu/0RLbbbLWUCuza/P6W/IHcmrLcsBH0KREGqTnXcsA7rYCldcrtP4nwt0sUl9qljdyzKUmU8wjaqunSZLUqTsd+MGRo1VM/jH750LjthhzuueTy96ZNeMAeikHJXnqnBHdApw/3okrHDUA7B2L7KuaiE+fQ5oyPaRtSMqTCyx04gsQoj/8Hn9mVQIuTfJRvlatcHcn0LQMgYk3Jg4NO/60ELf+yN3nUqIvwue2EEE/d68olZFfK3yFG4CJ/kmy7DZxkP3aSC1qfv0DnXaPxcrJ+aDw5Jw/cFSH4n2AGIBZptbqYGtNFjtxyGicmDkhofM73q3tWvEVyVVVfD5m1762sPQudIWYu0yPLfq8BnKt54ZBOshJRgjhwmkigD7vFDcGizjuu4NeXKz3GAxzjWhV1VxpfKVRXeu5YkdcI5JeFrPHnUvaJlXwFyogICJlo8SCZjgEWEjXjXY2o6nzXM46opuUYAsuQVno1CHsFD39F/CoDfRvtYOgyKuYRSjzxrnuGLjmWpU3wdeA1k916/W+nOd6qe+qBvsruywkjYEFIwowZoUiPRC9XyzbwxUwOjdfBH1JOOFV8a5xLVDSdmncLjxRIFgypLr7sG2GA3oOTS73otPz1KzShpjq7r8iVqe2psw5fESVHxZPrbZvld+Uego/BLcJFXeEo+y/HlPhW2pNlQ7eWaXU/2YMhO+V3Mk0iTfxj70n5ilQuSMn8Dr0NNmgWxnYmH70FKUO1EzABa+ziJOIPfiC4JnDR9LgyoN4uQblxR1djnmsfCD6cGuBp7FJgvJtnWSMPTklbMh ULw5gdIH EtfcZJ773HMDcjTZe2iU9dS6pLx0GCaJCy2+LWL918ZDT6YLmFWjJkVw83A2qRauLl4OBcP3p1fd7E5Istr9YWmpAZhoskro4nomYReIBT2srMWS19Ttv8pzZ2GQrzDsiJioIaM95didrVpfKJ7/xUmxuEw4KspHCYloktgQQiOqa9JKunZfYM/cx17hW0E8X7cEPu1iegYHfD0hlszauo+jr0Mp2O/9iisxJCYVssbZUadrhQkM3mC8lLOp2P5116Wf6nTvvVjL28PfetVbLa8GleuBs+efED5KNaeLFOZ2Px7aeNbHTdJS7f8mb5Hg62nCPf/+L67BUMegcjN6lbkX5LMQuG+AAUCjuoiBS76owNOyLsXZCc5uZQuHUMzxsSNj2r31EQkXJOSAh22NUjU0o7jGCM/qcjzWA36xjdk2YSR9QzurOT854yIy2RQbzRmXmBlybXOP9OmyLHF3hkOod5Pl+FlLXrSYGCBEWUoJX74i8K723gIRjp3q746M4OwnptIk5YypyTDZzedAv6rUIco1CEKTT6gBd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-09-28 16:49, Alan Stern wrote: > On Sat, Sep 28, 2024 at 09:51:27AM -0400, Mathieu Desnoyers wrote: >> Compiler CSE and SSA GVN optimizations can cause the address dependency >> of addresses returned by rcu_dereference to be lost when comparing those >> pointers with either constants or previously loaded pointers. >> >> Introduce ptr_eq() to compare two addresses while preserving the address >> dependencies for later use of the address. It should be used when >> comparing an address returned by rcu_dereference(). >> >> This is needed to prevent the compiler CSE and SSA GVN optimizations >> from replacing the registers holding @a or @b based on their > > "Replacing" isn't the right word. What the compiler does is use one > rather than the other. Furthermore, the compiler can play these games > even with values that aren't in registers. > > You should just say: "... from using @a (or @b) in places where the > source refers to @b (or @a) (based on the fact that after the > comparison, the two are known to be equal), which does not ..." OK. > >> equality, which does not preserve address dependencies and allows the >> following misordering speculations: >> >> - If @b is a constant, the compiler can issue the loads which depend >> on @a before loading @a. >> - If @b is a register populated by a prior load, weakly-ordered >> CPUs can speculate loads which depend on @a before loading @a. > > It shouldn't matter whether @a and @b are constants, registers, or > anything else. All that matters is that the compiler uses the wrong > one, which allows weakly ordered CPUs to speculate loads you wouldn't > expect it to, based on the source code alone. I only partially agree here. On weakly-ordered architectures, indeed we don't care whether the issue is caused by the compiler reordering the code (constant) or the CPU speculating the load (registers). However, on strongly-ordered architectures, AFAIU, only the constant case is problematic (compiler reordering the dependent load), because CPU speculating the loads across the control dependency is not an issue. So am I tempted to keep examples that clearly state whether the issue is caused by compiler reordering instructions, or by CPU speculation. > >> The same logic applies with @a and @b swapped. >> >> The compiler barrier() is ineffective at fixing this issue. >> It does not prevent the compiler CSE from losing the address dependency: >> >> int fct_2_volatile_barriers(void) >> { >> int *a, *b; >> >> do { >> a = READ_ONCE(p); >> asm volatile ("" : : : "memory"); >> b = READ_ONCE(p); >> } while (a != b); >> asm volatile ("" : : : "memory"); <----- barrier() >> return *b; >> } >> >> With gcc 14.2 (arm64): >> >> fct_2_volatile_barriers: >> adrp x0, .LANCHOR0 >> add x0, x0, :lo12:.LANCHOR0 >> .L2: >> ldr x1, [x0] <------ x1 populated by first load. >> ldr x2, [x0] >> cmp x1, x2 >> bne .L2 >> ldr w0, [x1] <------ x1 is used for access which should depend on b. >> ret >> >> On weakly-ordered architectures, this lets CPU speculation use the >> result from the first load to speculate "ldr w0, [x1]" before >> "ldr x2, [x0]". >> Based on the RCU documentation, the control dependency does not prevent >> the CPU from speculating loads. >> [...] >> diff --git a/include/linux/compiler.h b/include/linux/compiler.h >> index 2df665fa2964..f26705c267e8 100644 >> --- a/include/linux/compiler.h >> +++ b/include/linux/compiler.h >> @@ -186,6 +186,68 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val, >> __asm__ ("" : "=r" (var) : "0" (var)) >> #endif >> >> +/* >> + * Compare two addresses while preserving the address dependencies for >> + * later use of the address. It should be used when comparing an address >> + * returned by rcu_dereference(). >> + * >> + * This is needed to prevent the compiler CSE and SSA GVN optimizations >> + * from replacing the registers holding @a or @b based on their >> + * equality, which does not preserve address dependencies and allows the Replacing with: * This is needed to prevent the compiler CSE and SSA GVN optimizations * from using @a (or @b) in places where the source refers to @b (or @a) * based on the fact that after the comparison, the two are known to be * equal, which does not preserve address dependencies and allows the * following misordering speculations: >> + * following misordering speculations: >> + * >> + * - If @b is a constant, the compiler can issue the loads which depend >> + * on @a before loading @a. >> + * - If @b is a register populated by a prior load, weakly-ordered >> + * CPUs can speculate loads which depend on @a before loading @a. >> + * >> + * The same logic applies with @a and @b swapped. > > This could be more concise, and it should be more general (along the > same lines as the description above). As per my earlier comment, I would prefer to keep the examples specific rather than general so it is clear which scenarios are problematic on weakly vs strongly ordered architectures. [...] Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com