From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEB3FC3DA78 for ; Sat, 14 Jan 2023 04:15:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 309788E0002; Fri, 13 Jan 2023 23:15:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 292068E0001; Fri, 13 Jan 2023 23:15:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10C7E8E0002; Fri, 13 Jan 2023 23:15:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EE6CD8E0001 for ; Fri, 13 Jan 2023 23:15:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B3CAE160170 for ; Sat, 14 Jan 2023 04:15:37 +0000 (UTC) X-FDA: 80352090714.30.AFA582C Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 1E21A100007 for ; Sat, 14 Jan 2023 04:15:35 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TIiL48S1; spf=pass (imf05.hostedemail.com: domain of "SRS0=Esch=5L=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=Esch=5L=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673669736; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P+QDaDpCfafXb/iF/0d+NS5AkcB6KUT2GRPdZV6ZCh0=; b=6qDTMjz9fUibqF0/Q1jehed7NPJkxYn2EvN4lcIiok68NiFJrFw9yNyZFHDTQPtr+zaems m0jOjnm3h3ruy2kNVIPJHsK0izzUlcuG9dFFFFeMp1Xihczhxgy27v/vLo6qACNf2pZTAz ZFHbxqVJSEZtm5ZKhG/D1yhKbedNTaE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TIiL48S1; spf=pass (imf05.hostedemail.com: domain of "SRS0=Esch=5L=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=Esch=5L=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673669736; a=rsa-sha256; cv=none; b=aUWZMpgoa7h4dh7OsrM1z2TF88FBlB2q0J+VK0XJBS+p5UgiBh73Zd8unR14qOluyD8vwV Z1fSOKSr4F9WcfsBAWaATnj5sCX8ptcngPQUZUnu7EPewyjexzGPiWIeo+K2wGy3bKUSlH LzqhHAoay1gyHg4n3fTXYcZ1ONTH/KU= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 65DB060244; Sat, 14 Jan 2023 04:15:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA4A6C433EF; Sat, 14 Jan 2023 04:15:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673669732; bh=t1OCBELEvB8uOiGS/ChlhgB36fyFMUsf1DNlkXrzI8g=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=TIiL48S1B1MciV7Fd4dOLguyoGqfS34IQp80QKIdyjyksD2wWuETou+6zTD+oghA6 GSLpzqIn4jFHdjA5pOJenWZW7ejx71rG4u5i0aEQDs4DyecCOE5TSa20GKtIi4Ya32 yDhxFHJ5V9NTW2ArBwHnS8tJQuBcxrKl3Pt72Kg7lgE/XfROArUgbC7kTd5dCMmKWz ubBMMsd9AZ5gjcCVxRVXV6UyZmOMFYoseTdX92cPPOLkItIEbqQUXcajlFRXIrAjwy aBfMiVbN8fh64fzogVJworlKoHloDAOsVjFGZQkgnvO9ow67FsFjDS/9lcZNNhPWlD qn2AH9kOQ80jw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 70ED45C0920; Fri, 13 Jan 2023 20:15:32 -0800 (PST) Date: Fri, 13 Jan 2023 20:15:32 -0800 From: "Paul E. McKenney" To: Yang Shi Cc: linux-mm@kvack.org Subject: Re: [QUESTION] Linux memory model: control dependency with bitfield Message-ID: <20230114041532.GK4028633@paulmck-ThinkPad-P17-Gen-1> Reply-To: paulmck@kernel.org References: <20230109173155.GS4028633@paulmck-ThinkPad-P17-Gen-1> <20230109231106.GZ4028633@paulmck-ThinkPad-P17-Gen-1> <20230110000418.GC4028633@paulmck-ThinkPad-P17-Gen-1> <20230112001511.GD4028633@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1E21A100007 X-Stat-Signature: qiuir8cy8szqmhokhrx838gzrycj4h1q X-Rspam-User: X-HE-Tag: 1673669735-615387 X-HE-Meta: U2FsdGVkX1/h3wPLoezr9Vini/BXoWdv23oBMD1GhqWlXL+W7bxLEtK30iz3+9sUj5AYLbk8nb3m95NbEXbb5NuEoj9cVaVuizolP6iTRzjBnxFxzcRVANJZKotQEggvDt7S/8bYBE/O7wtQrN+JdTQ4WqAoYIlIURI7lRQLu3NEIFcsDN1UdchCh74nK5VQ9MoZpYiZZfnf6S9iDCAFBpxuOqPqn5KyORYfSIcpkRAw5R2nlPKLCK7U8Llm4K05TWDD9gEz8NcfjkgMD4VbPK0EgyffeAlC6W6Awss47xPuNVSzB6oQBTmVfZMynDKZckf82B/zXtZyAflwOHxkg5nykC8ya0V35oBmfyRlkkgEWU1JonQxUze6uCPLiCCu0jCx6ebhP93J8YyopNU0Vfgx6SF2aAcgR44sioW9YHWaUDIejMHOO5mqq6fR9ZNK9evlnRE4XGc66ZGLKO2n+aFz8oWflDGkqT4la+7o8I8Aal1ZZvbLWnEsYkIWyjsD0A0sIRAz0VJ9/aaajalLHZPtp4BYKCZvBmtuXByfZQqcJtjzu4oxgxJ2oSHMHX1ypFB0OrCGqwP8vL/gjurVV3m8X1RAU4XkHleTUXxzTondfXADhII6NARIzLUAeIJN01szT+sQilDPyM8EjVpiXjoFtf1W/YSB/dY9YnjWRFecD5A1xivrRAaT19tZjJPRmZ8caoLbWzWAmxOhwPCo+RXMdeGexXqDFfMokNK0TP8GOOXm4y9a6Cl6I0V4BZKC1QodS9iOj1oenPsOxfYM2PHiXOiIc3l75gfHJ169AFZJ6nlA5mEndMRPzC2UqtVYoaIUEdzOiypi4SqZHJmpAN1Occi2j8gF03FN00aG2K6ZGBBPyW/edYfapFCl03MZTnn4s35yclPjZYETuESWNg+HCI1HVWJcms/iEwXIsTEWvZzQ5SBi6zyleCXTLF5KOGxuyRGg0Dz5vR3Zsoh rmjsqPC9 49yyJ4mQbkw4nQWhbM5KsfmxOnkJ8u0tWe7ckWtZfbj/bN7iVH9OWa65Yep2XlhyBYWkfmJbfqIRePhwmH8u/nl+PsbPCzWNpJe7tke9z71S0TVUvnEtS4ZiDRFhv9YUc656CeVlsSzV6NnK/p+zHAj5WgPsJ0/yxk6I7tVrER5uS6eI3hdBNSRQTll0Dwamhs+QiEsbh+zbggkvz6cCjyiDHBprC1kia7zwUD9XJ9sfveNiCZqcsuBVghfgoptswupkuEy25psyZulEZp+hZRRIgt8LgXUgs/p6O2D4x0IOeIgDEa8ZSjsPXdm4xmSypX3HaHdSZ6lHtVTQtAimgYstcOiHdQJn+QrymYasHuC6bGmOgVn7QCETMTC4CcoCn8k/QwfrWptOKWf26JAVCsQccnqUqkcWJO6x6Nd/JuuUf8Zg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 13, 2023 at 06:37:42PM -0800, Yang Shi wrote: > On Wed, Jan 11, 2023 at 4:15 PM Paul E. McKenney wrote: > > > > On Wed, Jan 11, 2023 at 04:01:34PM -0800, Yang Shi wrote: > > > On Mon, Jan 9, 2023 at 4:04 PM Paul E. McKenney wrote: > > > > > > > > On Mon, Jan 09, 2023 at 03:44:54PM -0800, Yang Shi wrote: > > > > > On Mon, Jan 9, 2023 at 3:11 PM Paul E. McKenney wrote: > > > > > > On Mon, Jan 09, 2023 at 02:08:35PM -0800, Yang Shi wrote: > > > > > > > On Mon, Jan 9, 2023 at 9:31 AM Paul E. McKenney wrote: > > > > > > > > On Mon, Jan 09, 2023 at 09:14:19AM -0800, Yang Shi wrote: > > > > > > > > > Hi Paul, > > > > > > > > > > > > > > > > > > Hope this email finds you are doing well. I recently ran into a > > > > > > > > > problem which might be related to control dependency of the memory > > > > > > > > > model. Conceptually, the code does (from copy_present_pte()): > > > > > > > > > > > > > > > > > > acquire mmap_lock > > > > > > > > > spin_lock > > > > > > > > > ... > > > > > > > > > clear bit (a bit in page flags) > > > > > > > > > ... > > > > > > > > > VM_BUG_ON(test bit) > > > > > > > > > ... > > > > > > > > > spin_unlock > > > > > > > > > release mmap_lock > > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC there is control dependency between the "clear bit" and > > > > > > > > > "VM_BUG_ON" since VM_BUG_ON simply tests the bit then raises the BUG. > > > > > > > > > They do touch the overlapping address (the page flags from the same > > > > > > > > > struct page), but they are bit field operations. Per the memory model > > > > > > > > > documentation, the order is not guaranteed for bit field operations > > > > > > > > > IIRC. > > > > > > > > > > > > > > > > > > And there are not any implicit barriers between clear bit and test > > > > > > > > > bit, so the question is whether an explicit barrier, for example, > > > > > > > > > smp_mb__after_atomic() is required after clear bit to guarantee it > > > > > > > > > works as expected? > > > > > > > > > > > > > > > > I am not familiar with this code, so I will stick with LKMM > > > > > > > > clarifications. > > > > > > > > > > > > > > Yeah, sure. This is why I tried to generalize the code. > > > > > > > > > > > > > > > First, please don't forget any protection and ordering that might be > > > > > > > > provided by the two locks held across this code. > > > > > > > > > > > > > > Yes, but for this case I just care about the code between clear bit > > > > > > > and VM_BUG_ON. > > > > > > > > > > > > Fair enough! > > > > > > > > > > > > > > Second, a control dependency extends from a READ_ONCE() or stronger > > > > > > > > (clear_bit() included) to a later store. Please note "store", not > > > > > > > > "load". If you need to order an earlier READ_ONCE() or clear_bit() > > > > > > > > > > > > > > So you mean: > > > > > > > > > > > > > > clear bit > > > > > > > ... > > > > > > > if (test bit) { > > > > > > > load_1 > > > > > > > store_1 > > > > > > > load_2 > > > > > > > store_2 > > > > > > > } > > > > > > > > > > > > > > The dependency reaches to the first store? > > > > > > > > > > > > It reaches both stores, but neither load. > > > > > > > > > > > > That means that your example might well execute as if it had instead > > > > > > been written as follows: > > > > > > > > > > > > load_1 > > > > > > load_2 > > > > > > if (test bit) { > > > > > > store_1 > > > > > > store_2 > > > > > > } > > > > > > > > > > > > Assuming that you mean the test_bit() function. If you instead mean > > > > > > a C-language statement that tests a bit, then the compiler can do all > > > > > > sorts of things to you. The compiler can also do interesting things > > > > > > to you if the stores are plain C-language stores instead of something > > > > > > like WRITE_ONCE(). > > > > > > > > > > It is a test_bit() function. Is it possible clear_bit() is reordered > > > > > with test_bit(), or test_bit() doesn't see the result from > > > > > clear_bit()? > > > > > > > > If the various calls to test_bit() and clear_bit() are to the same > > > > location, then they will not be reordered with each other. > > > > > > > > If they are to different locations, they can be reordered. But in that > > > > case, they would not see each others' results anyway. > > > > > > Yeah, make sense. > > > > > > > > > > > > > > > with a later load, you will need acquire semantics (smp_load_acquire(), > > > > > > > > for example) or an explicit barrier such as smp_rmb(). Use of acquire > > > > > > > > semantics almost always gets you code that is more readable. > > > > > > > > > > > > > > Does the load acquire have to pair with a smp_store_release()? > > > > > > > smp_mb__after_stomic() is not needed because it is too strong and the > > > > > > > weaker barrier is good enough, right? > > > > > > > > > > > > It needs to pair with some type of applicable ordering, but yes, > > > > > > smp_store_release() is a good one. > > > > > > > > > > So, it should look like IIUC: > > > > > > > > > > clear_bit() > > > > > smp_load_acquire() > > > > > ... > > > > > if (test_bit()) { > > > > > smp_store_release() > > > > > load_1 > > > > > store_1 > > > > > load_2 > > > > > store_2 > > > > > } > > > > > > > > Just so you know, smp_load_acquire() does a load and smp_store_release() > > > > does a store. > > > > > > > > Also, is this code executed by a single CPU/task? If so, you need to > > > > also consider the corresponding code executed by some other CPU/task. > > > > > > The code could be executed by multiple tasks in parallel. > > > > It is hard for me to tell you what to write without more information > > about what you do and do not want to happen, but here is one possibility: > > > > clear_bit(5, &my_bits); > > if (test_bit_acquire(5, &my_bits)) { > > r1 = READ_ONCE(a); > > WRITE_ONCE(b, 1729); > > r2 = READ_ONCE(c); > > WRITE_ONCE(d, 65535); > > } > > > > This is of course a bit nonsensical because something somewhere would > > need to set bit 5 in my_bits for the body of the "if" statement to ever > > be executed. I am assuming that this happens somewhere else. > > > > The clear_bit() would be ordered before the test_bit_acquire() due to > > their both accessing the same location. The test_bit_acquire() would > > be orderd before the body of that "if" statement due to the _acquire() > > suffix. > > > > Is that what you are looking for? If not, what are you looking for? > > Sorry for the late reply. I think the _acquire() should be something > I'm looking for. I just figured out a stable reproducer for my > problem, so I will collect more debug information and try some debug > patch with _acquire(). Hopefully I could get more clear picture next > week. Glad it helped, but are you sure that it is just a memory-ordering problem as opposed to an algorithmic bug? I always have to ask... Thanx, Paul