From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0757C433FE for ; Sat, 5 Nov 2022 20:06:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F61D6B0072; Sat, 5 Nov 2022 16:06:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A5D46B0073; Sat, 5 Nov 2022 16:06:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26D948E0001; Sat, 5 Nov 2022 16:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 16CE06B0072 for ; Sat, 5 Nov 2022 16:06:52 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D975EA038F for ; Sat, 5 Nov 2022 20:06:51 +0000 (UTC) X-FDA: 80100471822.01.A3E6D22 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by imf29.hostedemail.com (Postfix) with ESMTP id 875D412000B for ; Sat, 5 Nov 2022 20:06:51 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 7D1D95C00D0; Sat, 5 Nov 2022 16:06:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Sat, 05 Nov 2022 16:06:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1667678809; x=1667765209; bh=Bs Ceaf14jLHWb+fLkSxCTF9UB6AbPje9aPDJaF0NlXs=; b=mFdJR4hMQbCC0fNwm9 5D2Ysku7jguxJH5YJQqzSbKuQxQhcKfu/gExHUza1ST+aUe0WRdQm6G9CE3o6emB YsjU/PGP6qgx9u+EBR+DFM1agFPLBasBEm5Ce9ypwy072Dux55D5SWItF8Hjhq7w j2/p+5FHijEKuDQwB3xIAsdr/RJyV5C1IE8mk85RPXcB1mV9+8PsirJ/Ft7IKiEC zYRovfRbis4V3H4Whon1V8fhVckSdM3yG4w0Tm983z6rDUNHJirwMLz63pBzA+1/ aTV4FTgrpfEMmmSrid9mX67F3rLsXMQKzJbPVlNrhwHpeCtFXTAdfORVQmUKgowl bMAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1667678809; x=1667765209; bh=BsCeaf14jLHWb+fLkSxCTF9UB6Ab Pje9aPDJaF0NlXs=; b=WzvdEjYiIUdCqzI8fZqymDcQ2kIGnEfpVR13xVgU3ky1 wntOBLbC68yGVso9yrkAs45GzXgiIBLI1J6ThCiSHgb0onu6cbBektEbCdmADIwr RYrYcc7CynC0uHP28Bhzhx0lWNQU9vhsmMv/5UHKgrC2c0AXxXgfQ8eJlCo50YD3 nWhMkG5RWnbMGVAvarAjbXP0k+iIx1YglZrP22bc/xMTKl9Ww+Bz+0+KmXp5Z3eC puYArEH7PpW6eHURd4AYsWn+uA4HcgMUhGjxNCdD4XRmGydvOzwl+lyrCmOLM+lE yecJV+sJ2h4dWCFe7Pssow/2J4xB6nkrs75VF6LGIQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvgedrvdeggddujecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehttddttddttddvnecuhfhrohhmpedfmfhirhhi lhhlucetrdcuufhhuhhtvghmohhvfdcuoehkihhrihhllhesshhhuhhtvghmohhvrdhnrg hmvgeqnecuggftrfgrthhtvghrnhephfeigefhtdefhedtfedthefghedutddvueehtedt tdehjeeukeejgeeuiedvkedtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepkhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgv X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 5 Nov 2022 16:06:47 -0400 (EDT) Received: by box.shutemov.name (Postfix, from userid 1000) id 448F9104449; Sat, 5 Nov 2022 23:06:46 +0300 (+03) Date: Sat, 5 Nov 2022 23:06:46 +0300 From: "Kirill A. Shutemov" To: Hugh Dickins Cc: Andrew Morton , Matthew Wilcox , David Hildenbrand , Vlastimil Babka , Peter Xu , Yang Shi , John Hubbard , Mike Kravetz , Sidhartha Kumar , Muchun Song , Miaohe Lin , Naoya Horiguchi , Mina Almasry , James Houghton , Zach O'Keefe , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 3/3] mm,thp,rmap: lock_compound_mapcounts() on THP mapcounts Message-ID: <20221105200646.wmfilka6prusrb56@box.shutemov.name> References: <5f52de70-975-e94f-f141-543765736181@google.com> <1b42bd1a-8223-e827-602f-d466c2db7d3c@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1b42bd1a-8223-e827-602f-d466c2db7d3c@google.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667678811; a=rsa-sha256; cv=none; b=yviS9gizXEtKBSiMCB7MWalw2e7lHaNhJaiejQCyYqIUX1rzZzmnqAqvj+03RMOyXarCqb VGMm9NS/vW7gPE4Q9LkxipKbGSN34dSY9tgoNiqGXsuwHkAGY1UVOaNVfFGbEMyvEFtY/S jdMcpKcTY5mQVgXl4ZsiNzicpSqlX9o= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=shutemov.name header.s=fm2 header.b=mFdJR4hM; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=WzvdEjYi; dmarc=none; spf=pass (imf29.hostedemail.com: domain of kirill@shutemov.name designates 66.111.4.28 as permitted sender) smtp.mailfrom=kirill@shutemov.name ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667678811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BsCeaf14jLHWb+fLkSxCTF9UB6AbPje9aPDJaF0NlXs=; b=ImzjbkSY6+RQe2kEmLEwYIvJ9i44Plq23N+HDXC+ePobUuJnp9PgJkj9zvcM9ITdmXAbhe ZUVC3VlXR1aNyJ+iLFzm/QbdjP4EH91XYeQHdSaa1PeNGFNd3jLZumer7Mbz+8bLjFKsq/ +YsM4Ps8oTB4GwZdYvoiHFGJFbbXLsU= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 875D412000B X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=shutemov.name header.s=fm2 header.b=mFdJR4hM; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=WzvdEjYi; dmarc=none; spf=pass (imf29.hostedemail.com: domain of kirill@shutemov.name designates 66.111.4.28 as permitted sender) smtp.mailfrom=kirill@shutemov.name X-Stat-Signature: fntgx37m1baf5werpqja1rkzmxhbb1th X-HE-Tag: 1667678811-15570 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Nov 02, 2022 at 06:53:45PM -0700, Hugh Dickins wrote: > Fix the races in maintaining compound_mapcount, subpages_mapcount and > subpage _mapcount by using PG_locked in the first tail of any compound > page for a bit_spin_lock() on such modifications; skipping the usual > atomic operations on those fields in this case. > > Bring page_remove_file_rmap() and page_remove_anon_compound_rmap() > back into page_remove_rmap() itself. Rearrange page_add_anon_rmap() > and page_add_file_rmap() and page_remove_rmap() to follow the same > "if (compound) {lock} else if (PageCompound) {lock} else {atomic}" > pattern (with a PageTransHuge in the compound test, like before, to > avoid BUG_ONs and optimize away that block when THP is not configured). > Move all the stats updates outside, after the bit_spin_locked section, > so that it is sure to be a leaf lock. > > Add page_dup_compound_rmap() to manage compound locking versus atomics > in sync with the rest. In particular, hugetlb pages are still using > the atomics: to avoid unnecessary interference there, and because they > never have subpage mappings; but this exception can easily be changed. > Conveniently, page_dup_compound_rmap() turns out to suit an anon THP's > __split_huge_pmd_locked() too. > > bit_spin_lock() is not popular with PREEMPT_RT folks: but PREEMPT_RT > sensibly excludes TRANSPARENT_HUGEPAGE already, so its only exposure > is to the non-hugetlb non-THP pte-mapped compound pages (with large > folios being currently dependent on TRANSPARENT_HUGEPAGE). There is > never any scan of subpages in this case; but we have chosen to use > PageCompound tests rather than PageTransCompound tests to gate the > use of lock_compound_mapcounts(), so that page_mapped() is correct on > all compound pages, whether or not TRANSPARENT_HUGEPAGE is enabled: > could that be a problem for PREEMPT_RT, when there is contention on > the lock - under heavy concurrent forking for example? If so, then it > can be turned into a sleeping lock (like folio_lock()) when PREEMPT_RT. > > A simple 100 X munmap(mmap(2GB, MAP_SHARED|MAP_POPULATE, tmpfs), 2GB) > took 18 seconds on small pages, and used to take 1 second on huge pages, > but now takes 115 milliseconds on huge pages. Mapping by pmds a second > time used to take 860ms and now takes 86ms; mapping by pmds after mapping > by ptes (when the scan is needed) used to take 870ms and now takes 495ms. > Mapping huge pages by ptes is largely unaffected but variable: between 5% > faster and 5% slower in what I've recorded. Contention on the lock is > likely to behave worse than contention on the atomics behaved. > > Signed-off-by: Hugh Dickins Acked-by: Kirill A. Shutemov -- Kiryl Shutsemau / Kirill A. Shutemov