From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63FC2C4708D for ; Mon, 5 Dec 2022 01:57:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD96B8E0002; Sun, 4 Dec 2022 20:57:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A89918E0001; Sun, 4 Dec 2022 20:57:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92A7F8E0002; Sun, 4 Dec 2022 20:57:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 810348E0001 for ; Sun, 4 Dec 2022 20:57:11 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 560A3802EF for ; Mon, 5 Dec 2022 01:57:11 +0000 (UTC) X-FDA: 80206589862.28.630C6F0 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf10.hostedemail.com (Postfix) with ESMTP id 0C23CC0006 for ; Mon, 5 Dec 2022 01:57:10 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oBiXLQ2q; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670205431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zm8T1Im5lQZlS8KmDG8ug3k7vKs3MbAFvEQ/V/+qGMs=; b=NjBG1qQSS/XvA0177QTzGVbdLwHHqyWCAFU2X1f7zo8R3Y2uKyv1ujXZGoTnp/N+x527Ag 2gFewXW62EM4Yp1KZrwViNlR9csdfOXslQnGjt9iMyNomT/HPcq1MyoX1KCZ14dsxyaJi0 rEZFfiVvGswolhBBiKPATbIS5WvXP/A= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oBiXLQ2q; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670205431; a=rsa-sha256; cv=none; b=L9LC6bCJcQJ/YaLC1MtuzS+/jvQ9igTI1CGXrezziHexODCxRtfTivGWgOW1vc0hbLMoU0 FcyY7DvGaAfBdOXHQBVje8q3WOgAflI4PsN6PBVo0UUwF+qzG/s/hHCT6zVs0YbJaJOdxQ +K4CfXjpEUkx+i+c80U6Xmo7a1QOfxw= Received: by mail-qt1-f175.google.com with SMTP id r19so10527657qtx.6 for ; Sun, 04 Dec 2022 17:57:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=zm8T1Im5lQZlS8KmDG8ug3k7vKs3MbAFvEQ/V/+qGMs=; b=oBiXLQ2qH0u4bTX6QnOElzutUIbpUCqJcRYVD0Rc8+cYgu8GHeNCMt2Nsg8Q6JTZpA gO8BvvOiANjRddJMlUsTl/CYkfqW05Xf/zq0LDB7ZBUOwynxQCkdcLQSAoqvBFGgjAyx Vg9LmyDDCwiY9X38XiMMewAgZC1/rw9AcayJMnNee7NuGUhSkZ5gHOC5Rt1HCcCYCSZx kRx1SL7HeRRvVvy37bTAPdiCx1iWbBp3n8yh1HXpJpzGNIe8Pbat5KzP7R6HP9//Vqpg rpFs8UzLiB9lE0IkEmRShOyrYFXHcjf3nY4pDhTkW2vO54ZYzH8YupuPBCXP8XxbwJHR LUiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zm8T1Im5lQZlS8KmDG8ug3k7vKs3MbAFvEQ/V/+qGMs=; b=6Bc4OYNAbZyGn+GlR5LS8S0b+udsuXNTwgBekjLIvp6PfdfSd8pLq2OfTXo5Zsv5WA mksRGBAsKwT5hv1wC0QeKlIVPoOWsYVahAm8Dz1Y1BfROwhnM94kMZRf5a8533EUxiNr S7l27xCq/nBp6TqF8SV40iVB+Nx5QonJ6gbPT3unhM6eBj2Dg558BkGPPq0hxarXtAT4 TWDkgCATLipDgPl9gws4tvce5wVHiYcpDF/4MQNM/qw4LuX7YZCM5pokOgTG+uNys5E6 Uz6APf7SLbC/ZyqvaB53IYbPXYm61jFtbz5nijUoDrEjpDTNSupRL1Zcs/UJ79GxZ/sS ja5w== X-Gm-Message-State: ANoB5plIRinbBmiSYUpaMVzAfS0l6ja+a9LEZii9vffw6IZgH/j0X9gg M1EqO8pV3FSd4q1NemgULixrow== X-Google-Smtp-Source: AA0mqf4FF+tKOxOy5vd4ImmVGJI9e9n6QvIzATrmRSsYm/4BUA9QXWDnuf5DBHmxRxo0MJsg33bcRw== X-Received: by 2002:a05:622a:40ca:b0:3a6:328d:ee8e with SMTP id ch10-20020a05622a40ca00b003a6328dee8emr59163925qtb.421.1670205430201; Sun, 04 Dec 2022 17:57:10 -0800 (PST) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id ck15-20020a05622a230f00b0039953dcc480sm9094377qtb.88.2022.12.04.17.57.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 Dec 2022 17:57:09 -0800 (PST) Date: Sun, 4 Dec 2022 17:57:07 -0800 (PST) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Linus Torvalds , Johannes Weiner , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Vlastimil Babka , Peter Xu , Yang Shi , John Hubbard , Mike Kravetz , Sidhartha Kumar , Muchun Song , Miaohe Lin , Naoya Horiguchi , Mina Almasry , James Houghton , Zach O'Keefe , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH] mm,thp,rmap: fix races between updates of subpages_mapcount Message-ID: <4388158-3092-a960-ff2d-55f2b0fe4ef8@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spamd-Result: default: False [-1.90 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.160.175:from]; SUBJECT_HAS_UNDERSCORES(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; BAD_REP_POLICIES(0.10)[]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[google.com:+]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWELVE(0.00)[20]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 0C23CC0006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xdhq57nu9xii3agoohtjphyr3dw5izo5 X-HE-Tag: 1670205430-942826 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 4b51634cd16a, introducing the COMPOUND_MAPPED bit, paid attention to the impossibility of subpages_mapcount ever appearing negative; but did not attend to those races in which it can momentarily appear larger than thought possible. These arise from how page_remove_rmap() first decrements page->_mapcount or compound_mapcount, then, if that transition goes negative (logical 0), decrements subpages_mapcount. The initial decrement lets a racing page_add_*_rmap() reincrement _mapcount or compound_mapcount immediately, and then in rare cases its corresponding increment of subpages_mapcount may be completed before page_remove_rmap()'s decrement. There could even (with increasing unlikelihood) be a series of increments intermixed with the decrements. In practice, checking subpages_mapcount with a temporary WARN on range, has caught values of 0x1000000 (2*COMPOUND_MAPPED, when move_pages() was using remove_migration_pmd()) and 0x800201 (do_huge_pmd_wp_page() using __split_huge_pmd()): page_add_anon_rmap() racing page_remove_rmap(), as predicted. I certainly found it harder to reason about than when bit_spin_locked, but the easy case gives a clue to how to handle the harder case. The easy case being the three !(nr & COMPOUND_MAPPED) checks, which should obviously be replaced by (nr < COMPOUND_MAPPED) checks - to count a page as compound mapped, even while the bit in that position is 0. The harder case is when trying to decide how many subpages are newly covered or uncovered, when compound map is first added or last removed: not knowing all that racily happened between first and second atomic ops. But the easy way to handle that, is again to count the page as compound mapped all the while that its subpages_mapcount indicates so - ignoring the _mapcount or compound_mapcount transition while it is on the way to being reversed. Fixes: 4b51634cd16a ("mm,thp,rmap: subpages_mapcount COMPOUND_MAPPED if PMD-mapped") Signed-off-by: Hugh Dickins --- mm/rmap.c | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 459dc1c44d8a..b616870a09be 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1232,7 +1232,7 @@ void page_add_anon_rmap(struct page *page, if (first && PageCompound(page)) { mapped = subpages_mapcount_ptr(compound_head(page)); nr = atomic_inc_return_relaxed(mapped); - nr = !(nr & COMPOUND_MAPPED); + nr = (nr < COMPOUND_MAPPED); } } else if (PageTransHuge(page)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1241,8 +1241,16 @@ void page_add_anon_rmap(struct page *page, if (first) { mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr_pmdmapped = thp_nr_pages(page); + nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } } } @@ -1330,7 +1338,7 @@ void page_add_file_rmap(struct page *page, if (first && PageCompound(page)) { mapped = subpages_mapcount_ptr(compound_head(page)); nr = atomic_inc_return_relaxed(mapped); - nr = !(nr & COMPOUND_MAPPED); + nr = (nr < COMPOUND_MAPPED); } } else if (PageTransHuge(page)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1339,8 +1347,16 @@ void page_add_file_rmap(struct page *page, if (first) { mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr_pmdmapped = thp_nr_pages(page); + nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } } } @@ -1387,7 +1403,7 @@ void page_remove_rmap(struct page *page, if (last && PageCompound(page)) { mapped = subpages_mapcount_ptr(compound_head(page)); nr = atomic_dec_return_relaxed(mapped); - nr = !(nr & COMPOUND_MAPPED); + nr = (nr < COMPOUND_MAPPED); } } else if (PageTransHuge(page)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1396,8 +1412,16 @@ void page_remove_rmap(struct page *page, if (last) { mapped = subpages_mapcount_ptr(page); nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + if (likely(nr < COMPOUND_MAPPED)) { + nr_pmdmapped = thp_nr_pages(page); + nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + /* Raced ahead of another remove and an add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* An add of COMPOUND_MAPPED raced ahead */ + nr = 0; + } } } -- 2.35.3