From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A180C48260 for ; Thu, 8 Feb 2024 15:49:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5D3B6B008A; Thu, 8 Feb 2024 10:49:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE6E56B008C; Thu, 8 Feb 2024 10:49:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B60996B0092; Thu, 8 Feb 2024 10:49:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9FD2E6B008A for ; Thu, 8 Feb 2024 10:49:57 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4687A160E8E for ; Thu, 8 Feb 2024 15:49:57 +0000 (UTC) X-FDA: 81769072434.17.C24978E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 1502F1C001E for ; Thu, 8 Feb 2024 15:49:53 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="I1n35/Xb"; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707407395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8InSULzEnAUdTmpoS9ssOSUDssOveuEGasLQle1XQDo=; b=Q//ma8Ld4WQjvRNcII/mxfOuZp0IHjdo3s1x8Ux2U7GICtyhpV1H/QKWZPszHZ+ZuSSSjp pZCzb00Nb15InVEdijFu31z/ckp+HH0R2gYRXH3v5GqoKf3ctktIJ8qcjIbGoMQKuWw2lU sH0EyxgOVZLXJEj9fN4kieEWD7NWETg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707407395; a=rsa-sha256; cv=none; b=bhqKU+zC2X1Qbz6KZ2f+WJ0gRl3CKVF7AaJVmF7xAKOuH2Q3lc02ottb/7CNdKBiHknCw1 bTiYHpQzhClgjoDRxR5m21Fzh6qGhFoJrsaJaE65HLmmxzn0Xj/xXXBLYDEpeOEtD1pM/H Nn0xnBuXv7QVkmjxruYXDQGC6rAlAwA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="I1n35/Xb"; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Sender:Reply-To:Content-ID:Content-Description; bh=8InSULzEnAUdTmpoS9ssOSUDssOveuEGasLQle1XQDo=; b=I1n35/Xb45Ef2i3G6zJeGZvJLL 2p56uybAIe4VSFtYe86EqxUETFwtdvR1K/Fs0zdFCxx57C3ijm9mYLYmtFLKdrVv9STMmy5qhXzCM AytuRSdktCqe0Kodheq6f+vLkrfdZy0zsszB6Fk+3d1yPXzYDzXUmTu3jL6e7ieytvZSBzlWh5c9w XGjbLu/1GSMLVVS4iPoMpoB7sRU+C3nVLCLsisFFZXqBIg2n+QZKnS4bCNuQjpZe4A9gIZhTeblTs 3kBr0Zcd+5Ocm5lz5SpvtggG3Ipaz3A4b9v+ZeZjAiSvhdrzQ6v/05JApQCmBSqqDwc0X4ZksIDe4 4/Jb95kg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rY6ea-00000000b9S-16ZY; Thu, 08 Feb 2024 15:49:32 +0000 Date: Thu, 8 Feb 2024 15:49:32 +0000 From: Matthew Wilcox To: Jane Chu Cc: Will Deacon , Nanyong Sun , Catalin Marinas , muchun.song@linux.dev, akpm@linux-foundation.org, anshuman.khandual@arm.com, wangkefeng.wang@huawei.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3 0/3] A Solution to Re-enable hugetlb vmemmap optimize Message-ID: References: <20240113094436.2506396-1-sunnanyong@huawei.com> <20240207111252.GA22167@willie-the-truck> <20240207121125.GA22234@willie-the-truck> <908066c7-b749-4f95-b006-ce9b5bd1a909@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <908066c7-b749-4f95-b006-ce9b5bd1a909@oracle.com> X-Stat-Signature: xw5mw398mqued3cx9zpub1w4juaaegdu X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1502F1C001E X-Rspam-User: X-HE-Tag: 1707407393-706918 X-HE-Meta: U2FsdGVkX1+fFRuaqOpVVyUinCkbKnLaoczz6u6WYFTqckcNPsdh0Kc0nyPN7qsgmTNxKZ3fuDOCZN5eiRJLVdzvq3UL+zmwmiApsZcoyTQtj9w+1Ytn/F0pycCsz83ym3+ck5N/Usff9hzUsx1s+QOBSmigC8FjTMNn9/7LZwrYUmiou5A2R2gr1Mq74/QU04OZYFVF6wRMcpK3zPEle/fRm0u5VhTHUoJwuWvIzvQSZZ1q7WiLi5exCRIZlb/xfTVJ9p2nQuglYxD4cbnUvEpZ8JbUekYcTYczO8129cM6und1GZAUSBRgQqADhD3ibT92Kj2MwlHH5N7leFBq3WSrrxdM4vC2DBe2ao3hrY4wHARhWLPIjP+vV9+atoztStwQNEuh9t66yxSU38CkQaLdkNs4ok7FK4+3j0gwB7XI7meQNDTvAtgm8HXzQ4X+2uN77i/Rb84Jo70vin7cml/GNeGCrjFN2GOcVsuIQKBBydWXSsSWTzvGpsKQcstoSj62q+gUsr8lpG+UH+JzC0wbk0UNXX8ijh/OBxIdvdCj0UbW35oWdcIHr/CMAOUT3DbPJYBFgj1gStrMyO5eSm16Xr8v3qb0Syx8+tzdPq6j9Dbugy3O/VkLpwGNwEcQD0Qk+jadwHe80mW2IoMQmJrbjcqOFNs301GOlAwybx4ZuiG3W68+77SMytLI6p0VlaAhgzNRCqOS5Q6bl3sSKC8dxtpck1J/uh2epnrUVmLsf/+TWX9qyW22UUJLbIXoltRSRsS97gW79jUBynQV3V1+plHSULz3tzUY23M08SEd9KzmSYienNzDz9/+eAIy8D2ITSbp/EtRtFF1R91OrIC4rAIV2DuFTXfXA400h5vpiY0niKnbBS2wgLI6arwwy3x84g4vCvH9kX2/p1mA9Ko0mxszXBH3hylmfkbMNDaGXWX15bUKlmu4XrT6x2PaEoYuLDI9qct3M8pkv5i zl9ge0iN OHurrAWVwZzkMBDizuMH4j6ckDEyM2NJR8D2Ydy+mFk5keSkYULC6Efjbrasg+8bFedicWhKT1cu9K3rumvk5lcQMiFp3PFKxeVooad5omFgp3glv8gmRS+Ofc2lWjldzzWQVBWd+ETTPuIrUr2JD4DpidmNi6xAXqrAo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Feb 07, 2024 at 06:24:52PM -0800, Jane Chu wrote: > On 2/7/2024 6:17 AM, Matthew Wilcox wrote: > > While this array of ~512 pages have been allocated to hugetlbfs, and one > > would think that there would be no way that there could still be > > references to them, another CPU can have a pointer to this struct page > > (eg attempting a speculative page cache reference or > > get_user_pages_fast()). That means it will try to call > > atomic_add_unless(&page->_refcount, 1, 0); > > > > Actually, I wonder if this isn't a problem on x86 too? Do we need to > > explicitly go through an RCU grace period before freeing the pages > > for use by somebody else? > > > Sorry, not sure what I'm missing, please help. Having written out the analysis, I now think it can't happen on x86, but let's walk through it because it's non-obvious (and I think it illustrates what people are afraid of on Arm). CPU A calls either get_user_pages_fast() or __filemap_get_folio(). Let's do the latter this time. folio = filemap_get_entry(mapping, index); filemap_get_entry: rcu_read_lock(); folio = xas_load(&xas); if (!folio_try_get_rcu(folio)) goto repeat; if (unlikely(folio != xas_reload(&xas))) { folio_put(folio); goto repeat; } folio_try_get_rcu: folio_ref_try_add_rcu(folio, 1); folio_ref_try_add_rcu: if (unlikely(!folio_ref_add_unless(folio, count, 0))) { /* Either the folio has been freed, or will be freed. */ return false; folio_ref_add_unless: return page_ref_add_unless(&folio->page, nr, u); page_ref_add_unless: atomic_add_unless(&page->_refcount, nr, u); A rather deep callchain there, but for our purposes the important part is: we take the RCU read lock, we look up a folio, we increment its refcount if it's not zero, then check that looking up this index gets the same folio; if it doesn't, we decrement the refcount again and retry the lookup. For this analysis, we can be preempted at any point after we've got the folio pointer from xa_load(). > From hugetlb allocation perspective,  one of the scenarios is run time > hugetlb page allocation (say 2M pages), starting from the buddy allocator > returns compound pages, then the head page is set to frozen, then the > folio(compound pages) is put thru the HVO process, one of which is > vmemmap_split_pmd() in case a vmemmap page is a PMD page. > > Until the HVO process completes, none of the vmemmap represented pages are > available to any threads, so what are the causes for IRQ threads to access > their vmemmap pages? Yup, this sounds like enough, but it's not. The problem is the person who's looking up the folio in the pagecache under RCU. They've got the folio pointer and have been preempted. So now what happens to our victim folio? Something happens to remove it from the page cache. Maybe the file is truncated, perhaps vmscan comes along and kicks it out. Either way, it's removed from the xarray and gets its refcount set to 0. If the lookup were to continue at this time, everything would be fine because it would see a refcount of 0 and not increment it (in page_ref_add_unless()). And this is where my analysis of RCU tends to go wrong, because I only think of interleaving event A and B. I don't think about B and then C happening before A resumes. But it can! Let's follow the journey of this struct page. Now that it's been removed from the page cache, it's allocated by hugetlb, as you describe. And it's one of the tail pages towards the end of the 512 contiguous struct pages. That means that we alter vmemmap so that the pointer to struct page now points to a different struct page (one of the earlier ones). Then the original page of vmemmap containing our lucky struct page is returned to the page allocator. At this point, it no longer contains struct pages; it can contain literally anything. Where my analysis went wrong was that CPU A _no longer has a pointer to it_. CPU A has a pointer into vmemmap. So it will access the replacement struct page (which definitely has a refcount 0) instead of the one which has been freed. I had thought that CPU A would access the original memory which has now been allocated to someone else. But no, it can't because its pointer is virtual, not physical. --- Now I'm thinking more about this and there's another scenario which I thought might go wrong, and doesn't. For 7 of the 512 pages which are freed, the struct page pointer gathered by CPU A will not point to a page with a refcount of 0. Instead it will point to an alias of the head page with a positive refcount. For those pages, CPU A will see folio_try_get_rcu() succeed. Then it will call xas_reload() and see the folio isn't there any more, so it will call folio_put() on something which used to be a folio, and isn't any more. But folio_put() calls folio_put_testzero() which calls put_page_testzero() without asserting that the pointer is actually to a folio. So everything's fine, but really only by coincidence; I don't think anybody's thought about this scenario before (maybe Muchun has, but I don't remember it being discussed).