From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96D11C54E76 for ; Tue, 17 Jan 2023 21:28:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D7F36B0075; Tue, 17 Jan 2023 16:28:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 188416B0078; Tue, 17 Jan 2023 16:28:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 028E16B007B; Tue, 17 Jan 2023 16:28:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E81F66B0075 for ; Tue, 17 Jan 2023 16:28:33 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C38EE14096D for ; Tue, 17 Jan 2023 21:28:33 +0000 (UTC) X-FDA: 80365580106.04.2165A04 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf19.hostedemail.com (Postfix) with ESMTP id 3A2331A0009 for ; Tue, 17 Jan 2023 21:28:32 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D6NPijsb; spf=pass (imf19.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673990912; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WyGAUsCm7KCJHqqi4lBwANtoZ0ojOXiCOR5dPV9xvO4=; b=fut6BJAvsNeV3KrtUHrEP2XwQlEfy5O5BAuJ8TwgJDIZVupBS8LUniVN84ghyX9Jpmpq5e RnN7iDdhISWMEeZro1IM4s65JQkj+mnhWQPQiWSidzxt48qnPwOZNKcs+CmGoSg6fYBakg CjQSuwbN8/0ROrwQxv01ZlYcvlNKgFQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D6NPijsb; spf=pass (imf19.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673990912; a=rsa-sha256; cv=none; b=ov1Eed69uSJUkBFhaZrjfVHoSIvO/Ms9LMrn0iZDTOkUiKs8VhZYQvvhITFF/qa2EalZjm ZmoF2828ZYfUFThmDUUnI2yawtHAAhSXa93L286INnCRuSVJangeJ2IP3TMwwXXneDojJh +x0IcIA2998Vukv+scQbt5v8Nahyd7A= Received: by mail-pf1-f179.google.com with SMTP id i1so9573840pfk.3 for ; Tue, 17 Jan 2023 13:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WyGAUsCm7KCJHqqi4lBwANtoZ0ojOXiCOR5dPV9xvO4=; b=D6NPijsbvf9SJHK6h4cNPzxO+PmDG+oLsAJyryhXX9PlWX4i9J9KH+ZHpbCKmaFPa9 RJez8/8wH//t7prxg4WwRYX+c9wmfilUebQCJcEzUwamNJe/cWuNAuCd4RCrXozXiPhX 5tcHGXH0JzoBcRcXIv2YOCKq/AyHULmqO/nNt2savuguQpLSskllk/gDmovtQdPVL0CY 6KORg5YQCT+OgjRSp17NQOYzm94swZCG/UjNe6XnPUV+0MxrosMe1uW3Vn2VSnsZ5Y8A 2tvieYNmzABFuZI1mnPPNWXSjsbG9IQ59dSJrtZvdWOq/9kJ3NAfmxDBcPAwcb6MkE24 s7xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WyGAUsCm7KCJHqqi4lBwANtoZ0ojOXiCOR5dPV9xvO4=; b=eFRXvD+GQHIbxvh6GNWhucmHASq68hiu9zcRs9AZ3i+12puWHRKi6+rURYGxGpwf26 D4JvwsJ+e+FkM94YapXeuJLdF3P5aDeitrF6OjS9trBLdwe0r/n5nfb4NErCG1OU6Lv/ zYvHfmU1h1U3LFTN2GDjKJZ+YkbhHSfbsVW9jPuVD81Wo/3Adq+dxjzzsjLQeN9myJI5 LEzF4fscOV4I/x69re6dWvq5DgpR0CIe3irsHgJCMgn027sPWLIjGONB9pKj+3uKMCi2 S6t6kHA3Y0EggqGez9o8htj0q4+QFFSMAUqvZUtTfYPQYOGx/FBVPMHjufD/XwMQ/c1p Ss4g== X-Gm-Message-State: AFqh2kotz0upJyp6pPcs5ZzO5SjutLPJoJ7yUpz+G/ZlNStT27tSN2Lh KKP4wcUwTNd4qkCF+X3U2YzERZ6boJLwgSCc9Ejdjtxc X-Google-Smtp-Source: AMrXdXvlga4JfS2c6j998PTkBrTKR2YK62XOcP8wMb27J7S+ZkwJzTbkaZW3vcrSFRYdXkFL+3bF0TEzT/dzD0gu3dE= X-Received: by 2002:a63:1310:0:b0:483:cd24:1cb7 with SMTP id i16-20020a631310000000b00483cd241cb7mr308923pgl.190.1673990911042; Tue, 17 Jan 2023 13:28:31 -0800 (PST) MIME-Version: 1.0 References: <20230109173155.GS4028633@paulmck-ThinkPad-P17-Gen-1> <20230109231106.GZ4028633@paulmck-ThinkPad-P17-Gen-1> <20230110000418.GC4028633@paulmck-ThinkPad-P17-Gen-1> <20230112001511.GD4028633@paulmck-ThinkPad-P17-Gen-1> <20230114041532.GK4028633@paulmck-ThinkPad-P17-Gen-1> In-Reply-To: <20230114041532.GK4028633@paulmck-ThinkPad-P17-Gen-1> From: Yang Shi Date: Tue, 17 Jan 2023 13:28:19 -0800 Message-ID: Subject: Re: [QUESTION] Linux memory model: control dependency with bitfield To: paulmck@kernel.org Cc: linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 3A2331A0009 X-Stat-Signature: bftimkcp3id3km7pjrcf6ccwcrqa9yyc X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673990911-669323 X-HE-Meta: U2FsdGVkX1/BNhC1eAwm0hXAKjTUbKZMgkgUktnUsqZzZvGOFInEwk5WhlRRQPups3FpFYJHNOoJPJBnmXCyVKfF741Mz/VVPBSLqyX7J7twL5IZikNkCzBXg4KwV+yI+9M9qjgYaaMbzFN0H5YKlZsxLidX2s8y45cnxhy3IZ3W5VP10bYXskRzax0gKAipEZ/UJdjS7zBJ9e+MvtHzfVioCiVkputa9cXJ12DtzZS948Sb/42tPnJKC/6FHDZ8HrW3MpGRF8mCfIF7e8Dw4XFI5EOlxWonMfWeLxJOuLMYLoCjksKroJRLwpICvfMHULI6e8yVpCqemLoN6Ffpd5yh5Ap1soCP6j2qB1ghcTdsEi+VYU7pr1XFbDdIyybrUTi339TAQElJp1HJKM1bBDsSqqAcQeGKn4aWhX/TcnIWJXATeYbAcO20eZVa/MUob3hJiRnsJi3hGuNID4sA+o5eiJHXTvvmlVJFp9VnI0Ea8Qbz6D6g3sudX1SU34RSmJTi/cjUnLrCE8iEyQKuVHE+5aDHFnTXkoPVf3rn/eFY9zBz4seMsf+U4MkS6+rvpYJcrzYAsW/FyIEWLTSlsBluwVM9ThPGLyROjjgr1+hxj0JOCnZ0JO8IedjuJ4Ot/T7JyFKFSBQ/qAhdLg/7OSC3tQagQpP49JWDkoh+BPK3kE2w4iUrmBsxoy09ySdGv10ivwc4ufnVkKsmRs3DxfLx7/9oaXG6+KWh+xqrix+6ngclu1bq85rfHQaBguTfFqPxlOK1NvwhKd2xpFZj1Lyts51VKEX/l/LYAQBQygGvuprqZn6sE/vSYlc9WfJD/M5Xm77Z0rcevkNM++FXMtRCFlGs7eumrr0oIATp2pEHjI2j1syS/wOafOrXcfpQxMzk7PCsTYyBGjV5w/YQyF5f9NIY4+m00Z1kEV08+TVItdTB63pCnZjAD/M8o6Mrg/Ar2+FOeGwIOtDoc64 vlVUEw7b gMCpbMmoDM2K/ZsfF+m7okMla34c/vvNH3awS6A39dBNoFKv2824QFRe4CQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 13, 2023 at 8:15 PM Paul E. McKenney wrote: > > On Fri, Jan 13, 2023 at 06:37:42PM -0800, Yang Shi wrote: > > On Wed, Jan 11, 2023 at 4:15 PM Paul E. McKenney wrote: > > > > > > On Wed, Jan 11, 2023 at 04:01:34PM -0800, Yang Shi wrote: > > > > On Mon, Jan 9, 2023 at 4:04 PM Paul E. McKenney wrote: > > > > > > > > > > On Mon, Jan 09, 2023 at 03:44:54PM -0800, Yang Shi wrote: > > > > > > On Mon, Jan 9, 2023 at 3:11 PM Paul E. McKenney wrote: > > > > > > > On Mon, Jan 09, 2023 at 02:08:35PM -0800, Yang Shi wrote: > > > > > > > > On Mon, Jan 9, 2023 at 9:31 AM Paul E. McKenney wrote: > > > > > > > > > On Mon, Jan 09, 2023 at 09:14:19AM -0800, Yang Shi wrote: > > > > > > > > > > Hi Paul, > > > > > > > > > > > > > > > > > > > > Hope this email finds you are doing well. I recently ran into a > > > > > > > > > > problem which might be related to control dependency of the memory > > > > > > > > > > model. Conceptually, the code does (from copy_present_pte()): > > > > > > > > > > > > > > > > > > > > acquire mmap_lock > > > > > > > > > > spin_lock > > > > > > > > > > ... > > > > > > > > > > clear bit (a bit in page flags) > > > > > > > > > > ... > > > > > > > > > > VM_BUG_ON(test bit) > > > > > > > > > > ... > > > > > > > > > > spin_unlock > > > > > > > > > > release mmap_lock > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC there is control dependency between the "clear bit" and > > > > > > > > > > "VM_BUG_ON" since VM_BUG_ON simply tests the bit then raises the BUG. > > > > > > > > > > They do touch the overlapping address (the page flags from the same > > > > > > > > > > struct page), but they are bit field operations. Per the memory model > > > > > > > > > > documentation, the order is not guaranteed for bit field operations > > > > > > > > > > IIRC. > > > > > > > > > > > > > > > > > > > > And there are not any implicit barriers between clear bit and test > > > > > > > > > > bit, so the question is whether an explicit barrier, for example, > > > > > > > > > > smp_mb__after_atomic() is required after clear bit to guarantee it > > > > > > > > > > works as expected? > > > > > > > > > > > > > > > > > > I am not familiar with this code, so I will stick with LKMM > > > > > > > > > clarifications. > > > > > > > > > > > > > > > > Yeah, sure. This is why I tried to generalize the code. > > > > > > > > > > > > > > > > > First, please don't forget any protection and ordering that might be > > > > > > > > > provided by the two locks held across this code. > > > > > > > > > > > > > > > > Yes, but for this case I just care about the code between clear bit > > > > > > > > and VM_BUG_ON. > > > > > > > > > > > > > > Fair enough! > > > > > > > > > > > > > > > > Second, a control dependency extends from a READ_ONCE() or stronger > > > > > > > > > (clear_bit() included) to a later store. Please note "store", not > > > > > > > > > "load". If you need to order an earlier READ_ONCE() or clear_bit() > > > > > > > > > > > > > > > > So you mean: > > > > > > > > > > > > > > > > clear bit > > > > > > > > ... > > > > > > > > if (test bit) { > > > > > > > > load_1 > > > > > > > > store_1 > > > > > > > > load_2 > > > > > > > > store_2 > > > > > > > > } > > > > > > > > > > > > > > > > The dependency reaches to the first store? > > > > > > > > > > > > > > It reaches both stores, but neither load. > > > > > > > > > > > > > > That means that your example might well execute as if it had instead > > > > > > > been written as follows: > > > > > > > > > > > > > > load_1 > > > > > > > load_2 > > > > > > > if (test bit) { > > > > > > > store_1 > > > > > > > store_2 > > > > > > > } > > > > > > > > > > > > > > Assuming that you mean the test_bit() function. If you instead mean > > > > > > > a C-language statement that tests a bit, then the compiler can do all > > > > > > > sorts of things to you. The compiler can also do interesting things > > > > > > > to you if the stores are plain C-language stores instead of something > > > > > > > like WRITE_ONCE(). > > > > > > > > > > > > It is a test_bit() function. Is it possible clear_bit() is reordered > > > > > > with test_bit(), or test_bit() doesn't see the result from > > > > > > clear_bit()? > > > > > > > > > > If the various calls to test_bit() and clear_bit() are to the same > > > > > location, then they will not be reordered with each other. > > > > > > > > > > If they are to different locations, they can be reordered. But in that > > > > > case, they would not see each others' results anyway. > > > > > > > > Yeah, make sense. > > > > > > > > > > > > > > > > > > with a later load, you will need acquire semantics (smp_load_acquire(), > > > > > > > > > for example) or an explicit barrier such as smp_rmb(). Use of acquire > > > > > > > > > semantics almost always gets you code that is more readable. > > > > > > > > > > > > > > > > Does the load acquire have to pair with a smp_store_release()? > > > > > > > > smp_mb__after_stomic() is not needed because it is too strong and the > > > > > > > > weaker barrier is good enough, right? > > > > > > > > > > > > > > It needs to pair with some type of applicable ordering, but yes, > > > > > > > smp_store_release() is a good one. > > > > > > > > > > > > So, it should look like IIUC: > > > > > > > > > > > > clear_bit() > > > > > > smp_load_acquire() > > > > > > ... > > > > > > if (test_bit()) { > > > > > > smp_store_release() > > > > > > load_1 > > > > > > store_1 > > > > > > load_2 > > > > > > store_2 > > > > > > } > > > > > > > > > > Just so you know, smp_load_acquire() does a load and smp_store_release() > > > > > does a store. > > > > > > > > > > Also, is this code executed by a single CPU/task? If so, you need to > > > > > also consider the corresponding code executed by some other CPU/task. > > > > > > > > The code could be executed by multiple tasks in parallel. > > > > > > It is hard for me to tell you what to write without more information > > > about what you do and do not want to happen, but here is one possibility: > > > > > > clear_bit(5, &my_bits); > > > if (test_bit_acquire(5, &my_bits)) { > > > r1 = READ_ONCE(a); > > > WRITE_ONCE(b, 1729); > > > r2 = READ_ONCE(c); > > > WRITE_ONCE(d, 65535); > > > } > > > > > > This is of course a bit nonsensical because something somewhere would > > > need to set bit 5 in my_bits for the body of the "if" statement to ever > > > be executed. I am assuming that this happens somewhere else. > > > > > > The clear_bit() would be ordered before the test_bit_acquire() due to > > > their both accessing the same location. The test_bit_acquire() would > > > be orderd before the body of that "if" statement due to the _acquire() > > > suffix. > > > > > > Is that what you are looking for? If not, what are you looking for? > > > > Sorry for the late reply. I think the _acquire() should be something > > I'm looking for. I just figured out a stable reproducer for my > > problem, so I will collect more debug information and try some debug > > patch with _acquire(). Hopefully I could get more clear picture next > > week. > > Glad it helped, but are you sure that it is just a memory-ordering > problem as opposed to an algorithmic bug? I always have to ask... I did ask myself the same question. But AFAICT, all modifications to the flag (set and clear) are protected by page table lock. > > Thanx, Paul