From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49107C432BE for ; Mon, 30 Aug 2021 23:12:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA39F60FD8 for ; Mon, 30 Aug 2021 23:12:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AA39F60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kerneltoast.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 9BFB08D0001; Mon, 30 Aug 2021 19:12:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96E9A6B0072; Mon, 30 Aug 2021 19:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85FCF8D0001; Mon, 30 Aug 2021 19:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id 75B426B0071 for ; Mon, 30 Aug 2021 19:12:55 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 077D3276EF for ; Mon, 30 Aug 2021 23:12:55 +0000 (UTC) X-FDA: 78533299110.04.76B649F Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf09.hostedemail.com (Postfix) with ESMTP id A97DD3000103 for ; Mon, 30 Aug 2021 23:12:54 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id d5so4756445pjx.2 for ; Mon, 30 Aug 2021 16:12:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition; bh=v3IzYk5Cz+OUtPw+/T83szt+1CdCHRLLUCDNc9RQVcs=; b=MdFenmp6UPpTaROeFRNRPmFtXeSZ7FyToMsD3K0lKV/w9Ne+nbXGK5+uqChi38nNRJ 1YhJvv0TFcJnxekSA3hv69OmITakkAxdFgYL94782+Ti+AvD1BIfld4+XMNdgWqCzuuB A8yrRdZ3+TxTAgQ+UJerCxILToGJaaWJNPL07vHTklYBT9opUh4mCyQaYWhHXDR0J/E4 afrOK/P0nWVLdoa5rq+KwjafScM7peFHRPCi4mD/Ak7bvVLtCB50sivm1lYtf6G2H5EK iZyDM122FlVWNCJ2/N/Q81BwGzecF3NDGh6nkmSXnwudT7FiXE8RSQmhp020TLK7GmPO sx4A== X-Gm-Message-State: AOAM532P2OQDty92GndjznlwmtR7ERmW17urE44/+l738veveGqnwAKL oWSUDDlSYODxrWKRswD+/Z/YE6hrqNbZrg== X-Google-Smtp-Source: ABdhPJyM+xavDMy2nU5nHoZXItH9Z32dpIp3B6TEu9y18rZ8vFM4JFAMkb8gBzWR8C+1D3lMK1Vm+Q== X-Received: by 2002:a17:90a:3cc6:: with SMTP id k6mr1630328pjd.134.1630365173468; Mon, 30 Aug 2021 16:12:53 -0700 (PDT) Received: from sultan-box.localdomain (static-198-54-131-119.cust.tzulo.com. [198.54.131.119]) by smtp.gmail.com with ESMTPSA id w11sm8448566pfj.65.2021.08.30.16.12.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Aug 2021 16:12:53 -0700 (PDT) Date: Mon, 30 Aug 2021 16:12:51 -0700 From: Sultan Alsawaf To: linux-mm@kvack.org Cc: mhocko@suse.com, mgorman@techsingularity.net, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Subject: Stuck looping on list_empty(list) in free_pcppages_bulk() Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf09.hostedemail.com: domain of sultankerneltoast@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=sultankerneltoast@gmail.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: A97DD3000103 X-Stat-Signature: ijmo75id9xpxmuyjyit61w9rart9eciu X-HE-Tag: 1630365174-59035 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, I was recently given the following CPU stall splat and asked to look into it: ----------------8<---------------- rcu: INFO: rcu_sched detected stalls on CPUs/tasks: rcu: 44-...0: (1 GPs behind) idle=77e/1/0x4000000000000000 softirq=28480656/28480657 fqs=279939 (detected by 62, t=1140032 jiffies, g=224165577, q=8883218) Sending NMI from CPU 62 to CPUs 44: NMI backtrace for cpu 44 CPU: 44 PID: 83957 Comm: perl Tainted: G L 5.8.18-100.fc31.x86_64 #1 RIP: 0010:free_pcppages_bulk+0x63/0x2a0 RSP: 0018:ffffb3078698fb60 EFLAGS: 00000086 RAX: ffff8b647db30390 RBX: ffffee5fcab67f48 RCX: ffffee5f30c79980 RDX: 0000000000c31e66 RSI: 0000000000000007 RDI: 0000000000000007 RBP: 0000000000000000 R08: ffffb3078698fb80 R09: ffffb3078698fb80 R10: 00000000002ffa93 R11: 0000000000000000 R12: ffff8b647db30390 R13: 0000000000000000 R14: ffff8b647db30380 R15: 000000006a340084 FS: 0000000000000000(0000) GS:ffff8b647db00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff2b5200fa8 CR3: 00000034ada0a000 CR4: 0000000000340ee0 Call Trace: free_unref_page_list+0x113/0x1a0 release_pages+0x3ad/0x450 tlb_flush_mmu+0x36/0x160 unmap_page_range+0xab6/0xee0 unmap_vmas+0x6a/0xd0 exit_mmap+0x97/0x170 mmput+0x61/0x140 do_exit+0x306/0xb80 ? syscall_trace_enter+0x160/0x290 do_group_exit+0x3a/0xa0 __x64_sys_exit_group+0x14/0x20 do_syscall_64+0x4d/0x90 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---------------->8---------------- I apologize in advance for reporting a bug on an EOL kernel. I don't see any changes as of 5.14 that could address something like this, so I'm emailing in case whatever happened here may be a bug affecting newer kernels. With gdb, it appears that the CPU got stuck in the list_empty(list) loop inside free_pcppages_bulk(): ----------------8<---------------- do { batch_free++; if (++migratetype == MIGRATE_PCPTYPES) migratetype = 0; list = &pcp->lists[migratetype]; } while (list_empty(list)); ---------------->8---------------- Although this code snippet is slightly different in 5.14, it's still ultimately the same. Side note: I noticed that the way `migratetype` is incremented causes `&pcp->lists[1]` to get looked at first rather than `&pcp->lists[0]`, since `migratetype` will start out at 1. This quirk is still present in 5.14, though the variable in question is now called `pindex`. With some more gdb digging, I found that the `count` variable was stored in %ESI at the time of the stall. According to register dump in the splat, %ESI was 7. It looks like, for some reason, the pcp count was 7 higher than the number of pages actually present in the pcp lists. I tried to find some way that this could happen, but the only thing I could think of was that maybe an allocation had both __GFP_RECLAIMABLE and __GFP_MOVABLE set in its gfp mask, in which case the rmqueue() call in get_page_from_freelist() would pass in a migratetype equal to MIGRATE_PCPTYPES and then pages could be added to an out-of-bounds pcp list while still incrementing the overall pcp count. This seems pretty unlikely though. As another side note, it looks like there's nothing stopping this from occurring; there's only a VM_WARN_ON() in gfp_migratetype() that checks if both bits are set. Any ideas on what may have happened here, or a link to a commit that may have fixed this issue in newer kernels, would be much appreciated. Thanks, Sultan