From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADB17C432BE for ; Tue, 31 Aug 2021 12:44:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 36CA960240 for ; Tue, 31 Aug 2021 12:44:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 36CA960240 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B0FCE6B0071; Tue, 31 Aug 2021 08:44:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABF018D0001; Tue, 31 Aug 2021 08:44:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D5DB6B0073; Tue, 31 Aug 2021 08:44:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 8A6396B0071 for ; Tue, 31 Aug 2021 08:44:53 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4C4472288B for ; Tue, 31 Aug 2021 12:44:53 +0000 (UTC) X-FDA: 78535345266.13.B6D1BAF Received: from outbound-smtp50.blacknight.com (outbound-smtp50.blacknight.com [46.22.136.234]) by imf10.hostedemail.com (Postfix) with ESMTP id A07546001982 for ; Tue, 31 Aug 2021 12:44:52 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp50.blacknight.com (Postfix) with ESMTPS id CAB82FA904 for ; Tue, 31 Aug 2021 13:44:50 +0100 (IST) Received: (qmail 23353 invoked from network); 31 Aug 2021 12:44:50 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.17.29]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 31 Aug 2021 12:44:50 -0000 Date: Tue, 31 Aug 2021 13:44:49 +0100 From: Mel Gorman To: Sultan Alsawaf Cc: linux-mm@kvack.org, mhocko@suse.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Subject: Re: Stuck looping on list_empty(list) in free_pcppages_bulk() Message-ID: <20210831124449.GB4128@techsingularity.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.136.234 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A07546001982 X-Stat-Signature: 6pbw3gyo1u5eynf1nw1e7zr31gep4r58 X-HE-Tag: 1630413892-587004 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Aug 30, 2021 at 04:12:51PM -0700, Sultan Alsawaf wrote: > I apologize in advance for reporting a bug on an EOL kernel. I don't see any > changes as of 5.14 that could address something like this, so I'm emailing in > case whatever happened here may be a bug affecting newer kernels. > > With gdb, it appears that the CPU got stuck in the list_empty(list) loop inside > free_pcppages_bulk(): > ----------------8<---------------- > do { > batch_free++; > if (++migratetype == MIGRATE_PCPTYPES) > migratetype = 0; > list = &pcp->lists[migratetype]; > } while (list_empty(list)); > ---------------->8---------------- > > Although this code snippet is slightly different in 5.14, it's still ultimately > the same. Side note: I noticed that the way `migratetype` is incremented causes > `&pcp->lists[1]` to get looked at first rather than `&pcp->lists[0]`, since > `migratetype` will start out at 1. This quirk is still present in 5.14, though > the variable in question is now called `pindex`. > > With some more gdb digging, I found that the `count` variable was stored in %ESI > at the time of the stall. According to register dump in the splat, %ESI was 7. > > It looks like, for some reason, the pcp count was 7 higher than the number of > pages actually present in the pcp lists. > That's your answer -- the PCP count has been corrupted or misaccounted. Given this is a Fedora kernel, check for any patches affecting mm/page_alloc.c that could be accounting related or that would affect the IRQ disabling or zone lock acquisition for problems. Another possibility is memory corruption -- either kernel or the hardware itself. > I tried to find some way that this could happen, but the only thing I could > think of was that maybe an allocation had both __GFP_RECLAIMABLE and > __GFP_MOVABLE set in its gfp mask, in which case the rmqueue() call in > get_page_from_freelist() would pass in a migratetype equal to MIGRATE_PCPTYPES > and then pages could be added to an out-of-bounds pcp list while still > incrementing the overall pcp count. This seems pretty unlikely though. It's unlikely because it would be an outright bug to specify both flags. > As > another side note, it looks like there's nothing stopping this from occurring; > there's only a VM_WARN_ON() in gfp_migratetype() that checks if both bits are > set. > There is no explicit check for it because they should not be both set. I don't think this happens in kernel but if an out-of-tree module did it, it might corrupt adjacent PCPs. -- Mel Gorman SUSE Labs