From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B697C5475B for ; Wed, 6 Mar 2024 17:42:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8E586B0074; Wed, 6 Mar 2024 12:42:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3E576B0075; Wed, 6 Mar 2024 12:42:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9064D6B0078; Wed, 6 Mar 2024 12:42:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7BE446B0074 for ; Wed, 6 Mar 2024 12:42:01 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2C31140D45 for ; Wed, 6 Mar 2024 17:42:01 +0000 (UTC) X-FDA: 81867332442.27.5E7CF36 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 12005120009 for ; Wed, 6 Mar 2024 17:41:58 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709746919; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yq8/Rf1vraXBZR7qYWb0R55SgJEqzM7V+PvawhQ7blc=; b=3dXI82rOi9x7FajArFcm/uTqWlz+CHxaKLqDd+slJGLlTCh0FMWN5WGRoDz8q4oBkuqgjV 4jnMVvQUeVKgIU69a4QCqoViVFDUWkv/YNAvtvO906yxbSA1U/VoeFvw4gscAuPeSR0A6F qouVH96eBb1cw9dB5rHk7KJtPrfYWFA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709746919; a=rsa-sha256; cv=none; b=ZivNvMVTumWGXD6mUO6rgIq7aJ8+MWM6QyVfWCMffb6dGGfAHbFM1LbG66yYoGFwCoL1Z5 J372cj0DGjj4mTJub7bs9aI4hxvLgrI9+ujuLnpvrvTV/2FTKqqtJKfABvmoQ5MsplI44Y UWeeu7e8ZnTSOCZmBuQ52yFbCO2maCc= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C55191FB; Wed, 6 Mar 2024 09:42:34 -0800 (PST) Received: from [10.57.68.241] (unknown [10.57.68.241]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D78FA3F738; Wed, 6 Mar 2024 09:41:56 -0800 (PST) Message-ID: <36bdda72-2731-440e-ad15-39b845401f50@arm.com> Date: Wed, 6 Mar 2024 17:41:55 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 10/18] mm: Allow non-hugetlb large folios to be batch processed Content-Language: en-GB From: Ryan Roberts To: Matthew Wilcox Cc: Andrew Morton , linux-mm@kvack.org References: <20240227174254.710559-1-willy@infradead.org> <20240227174254.710559-11-willy@infradead.org> <367a14f7-340e-4b29-90ae-bc3fcefdd5f4@arm.com> <85cc26ed-6386-4d6b-b680-1e5fba07843f@arm.com> In-Reply-To: <85cc26ed-6386-4d6b-b680-1e5fba07843f@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 12005120009 X-Stat-Signature: nwryb7wewoxrq1ke377x3mfcw8jpgyi5 X-Rspam-User: X-HE-Tag: 1709746918-394321 X-HE-Meta: U2FsdGVkX18CQiDbJtQKSx0r2fxLaW2tftwRPAunsbyJ0yeahPj+plaeWEJO0WIFQg+UY9UANdsrBsT0RYC2A+3vo4b+eUKY1Z1d7QCtgNI1k7ar1JshcjgAmZeoeMUCqjKA3VP2MV71s0nCIVofkLqk1nE3RDHka6Gw6KY96ulO4+BDrP73EENPUsSGNuQT0e5KlS3WDgLJjv6HKsJ4eWcXE5WXlYJSptiiTjAiGoCGJ2uj/4jVUyGgTdLQD+Afp9wj36xQN0VQiAcW0HWcdDNV0gC56CzoMk3MsqdmT2oeuZrYGBn8ARB1Kfnhy6JKSvUri8lWscGo525ceSkO+tE5HCsMw21wLtiWq93QAA7+xlDz3VYYc1TeUoAh1YWF9+fAfEBOZQdC1NrpfGByGeHmwfun4VgIAeNjfksXwv9A7zqhSA3tgqYAooYFwEyXBuX808vWM3+IbCmOFfslHC+WdF1rwf7a8QgVlbD7er1soAKGalL6g07QCnEbGL6eqYPf1W/MNzswr67mcuHbyBNbNMvF0ARfLdYGKWmirQ4IcPLjgcdmMqWaW/ox6M2UXzg/AYqMOjfCUDYhjn3JNcFwdi+MMRsN2xN4rHfSi6ztINRKFEVlI+1xdRjeVj6FllbcVCnRKNo5IrMKUZWTKZL03/bqQEfkoJ2S7FPvc5+LArVSOU2l/1dCcsJAFGkVnvguLFb+TyC0v1eB0vQWOpILCaq8ZCvcYDDAyXVKwEFTwQLyb6AAWzEgnJKJCHw6Wn4YsZ0ERs780lNU6X7f7M4P0UrWlpZNSrD5cyLv6OKzoOKtcgBj2z+4r12kTcLpD8v9mzdHl2Itj8Bb0tZL7RB75Oy5Inc1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 06/03/2024 16:19, Ryan Roberts wrote: > On 06/03/2024 16:09, Matthew Wilcox wrote: >> On Wed, Mar 06, 2024 at 01:42:06PM +0000, Ryan Roberts wrote: >>> When running some swap tests with this change (which is in mm-stable) >>> present, I see BadThings(TM). Usually I see a "bad page state" >>> followed by a delay of a few seconds, followed by an oops or NULL >>> pointer deref. Bisect points to this change, and if I revert it, >>> the problem goes away. >> >> That oops is really messed up ;-( We're clearly got two CPUs oopsing at >> the same time and it's all interleaved. That said, I can pick some >> nuggets out of it. >> >>> [ 76.239466] BUG: Bad page state in process usemem pfn:2554a0 >>> [ 76.240196] kernel BUG at include/linux/mm.h:1120! >> >> These are the two different BUGs being called simultaneously ... >> >> The first one is bad_page() in page_alloc.c and the second is >> put_page_testzero() >> VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); >> >> I'm sure it's significant that both of these are the same page (pfn >> 2554a0). Feels like we have two CPUs calling put_folio() at the same >> time, and one of them underflows. It probably doesn't matter which call >> trace ends up in bad_page() and which in put_page_testzero(). >> >> One of them is coming from deferred_split_scan(), which is weird because >> we can see the folio_try_get() earlier in the function. So whatever >> this folio was, we found it on the deferred split list, got its refcount, >> moved it to the local list, either failed to get the lock, or >> successfully got the lock, split it, unlocked it and put it. >> >> (I can see this was invoked from page fault -> memcg shrinking. That's >> probably irrelevant but explains some of the functions in the backtrace) >> >> The other call trace comes from migrate_folio_done() where we're putting >> the _source_ folio. That was called from migrate_pages_batch() which >> was called from kcompactd. >> >> Um. Where do we handle the deferred list in the migration code? >> >> >> I've also tried looking at this from a different angle -- what is it >> about this commit that produces this problem? It's a fairly small >> commit: >> >> - if (folio_test_large(folio)) { >> + /* hugetlb has its own memcg */ >> + if (folio_test_hugetlb(folio)) { >> if (lruvec) { >> unlock_page_lruvec_irqrestore(lruvec, flags); >> lruvec = NULL; >> } >> - __folio_put_large(folio); >> + free_huge_folio(folio); >> >> So all that's changed is that large non-hugetlb folios do not call >> __folio_put_large(). As a reminder, that function does: >> >> if (!folio_test_hugetlb(folio)) >> page_cache_release(folio); >> destroy_large_folio(folio); >> >> and destroy_large_folio() does: >> if (folio_test_large_rmappable(folio)) >> folio_undo_large_rmappable(folio); >> >> mem_cgroup_uncharge(folio); >> free_the_page(&folio->page, folio_order(folio)); >> >> So after my patch, instead of calling (in order): >> >> page_cache_release(folio); >> folio_undo_large_rmappable(folio); >> mem_cgroup_uncharge(folio); >> free_unref_page() >> >> it calls: >> >> __page_cache_release(folio, &lruvec, &flags); >> mem_cgroup_uncharge_folios() >> folio_undo_large_rmappable(folio); >> >> So have I simply widened the window for this race > > Yes that's the conclusion I'm coming to. I have reverted this patch and am still > seeing what looks like the same problem very occasionally. (I was just about to > let you know when I saw this reply). It's much harder to reproduce now... great. > > The original oops I reported against your RFC is here: > https://lore.kernel.org/linux-mm/eeaf36cf-8e29-4de2-9e5a-9ec2a5e30c61@arm.com/ > > Looks like I had UBSAN enabled for that run. Let me turn on all the bells and > whistles and see if I can get it to repro more reliably to bisect. > > Assuming the original oops and this are related, that implies that the problem > is lurking somewhere in this series, if not this patch. > > I'll come back to you shortly... Just a bunch of circumstantial observations, I'm afraid. No conclusions yet... With this patch reverted: - Haven't triggered with any of the sanitizers compiled in - Have only triggered when my code is on top (swap-out mTHP) - Have only triggered when compiled using GCC 12.2 (can't trigger with 11.4) So perhaps I'm looking at 2 different things, with this new intermittent problem caused by my changes. Or perhaps my changes increase the window significantly. I have to go pick up my daughter now. Can look at this some more tomorrow, but struggling for ideas - need a way to more reliably reproduce. > >> , whatever it is >> exactly? Something involving mis-handling of the deferred list? >> >