From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2653C636CD for ; Fri, 10 Feb 2023 15:47:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24D206B014E; Fri, 10 Feb 2023 10:47:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FD2D6B014F; Fri, 10 Feb 2023 10:47:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C5BA6B0150; Fri, 10 Feb 2023 10:47:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F30636B014E for ; Fri, 10 Feb 2023 10:47:07 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6F302C105F for ; Fri, 10 Feb 2023 15:47:07 +0000 (UTC) X-FDA: 80451810894.17.A275E92 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf08.hostedemail.com (Postfix) with ESMTP id 3AD1816001A for ; Fri, 10 Feb 2023 15:47:03 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UZw8qv+K; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=OegnDL9G; spf=pass (imf08.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676044024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zQiRrtYqGHTqMan1QOBtU2HAPA2lYDV21aWW21XE6PQ=; b=fbw5BOt32ocR8Bdqy42UpbVuOvmK2AdsVD9ci9ujjRloVuTUU7RgHTUomYz1TsLs6Bmh2r TXGyS98HaPMXeArKE0o5bU+Zoii3fxto1M4o8hWHk+d/IClUcaMwv/FKWrgDEtsz1AsG6B i2eTkNqFruigqoQoKWU+RFO5U028fBM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UZw8qv+K; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=OegnDL9G; spf=pass (imf08.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676044024; a=rsa-sha256; cv=none; b=kABkNGvXzkilkzK0nij2yAMQV7laisuGCryXa0Q8J/OfoQPnGjGfV7gm2RiHnHQCejr2IE J/MMsiy7E/asuQ8W0mCboJLOAKZBi8uvnYeZRFeB6vvTk/GJjJTaDIo2irKA+dLtr9qUFX YcEx2upjlm3Q4BbhdrdiHBP7qnjzUw4= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9457F33D3E; Fri, 10 Feb 2023 15:47:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1676044022; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zQiRrtYqGHTqMan1QOBtU2HAPA2lYDV21aWW21XE6PQ=; b=UZw8qv+Kd4zClQd8KO3TPveGTkmVKY4aIYXMRqxzVPlszQUsphTQTlyOvUv/HfLO/jjJXw P7LAU5hjiOt/vePjJASa7uLHgbt4YPAZ60U8vRieT8FYTOcqmjS+crLEeIaBkBnxVM+nxb xPcLU+abjJRLH06UNwXi7lUrqmjsrlw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1676044022; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zQiRrtYqGHTqMan1QOBtU2HAPA2lYDV21aWW21XE6PQ=; b=OegnDL9GaFqyLqe7iav24qdt1DV9FmsQH2HgPxL0OQar/5DSXZcSIeWbwrtUBA/mIguufx TbrkJqF0u2wDSiDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 76B481325E; Fri, 10 Feb 2023 15:47:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id Mk1BHPZm5mNKbwAAMHmgww (envelope-from ); Fri, 10 Feb 2023 15:47:02 +0000 Message-ID: <7c72fefb-a3a2-0996-e800-7b56288002ab@suse.cz> Date: Fri, 10 Feb 2023 16:47:02 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [PATCH] Fix page corruption caused by racy check in __free_pages Content-Language: en-US To: David Chen , "linux-kernel@vger.kernel.org" Cc: Andrew Morton , "Matthew Wilcox (Oracle)" , "linux-mm@kvack.org" , "stable@vger.kernel.org" References: From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 3AD1816001A X-Stat-Signature: dhzzzm7tywqynunq6xb7pyzyt5b5ri66 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676044023-862697 X-HE-Meta: U2FsdGVkX1+Ucn151e4aq01y6dZzWP9i41W9h7BQepaRtVDpCYQ8EBzq9bsxYdKXuPDftw3W+qvg68HhPQtJUlHFfbiiljNfWjoup4P1amCpMNToUUIyV6Jmvg1oicF1P0Rx0ltX0M6GgbaxO8VB70GVKhJTEdQcfYZST1md9tpidHhuvwbw2Q2oyAfa7MvGd+5/M7YHvnN1OCWvbrPssSXHymBYPo/VHlPcBtMKT2LNYOveqpSmdHUXWqkYl3Kyxq6gP4wuwjzO99WGDRw5FgYQXIlHUKIj3nVLMrfPg5UMCCQdK428JE9KMeP1Wts69HSyqxqY5Uw05al87a6Na5tKPeCbk9GQQHPkl4EGhBAG3pzxl2jimR6E0uUkS09zm/rvMzxRd7OXY/dykQ5+JpdDkx570jtmHBARQr5h/ePBIr27l2CUk61WPoewP2+/ZScABRggFN7Oxhs83p1YMewkFxSuplobUnA2T43inGu9L/wU8JLtLMryJZ08ZHfSp2N4ngoIyzIMp9hsxpXPBEbp3DhEkCUesiZXEAw3pTL1hFd+qzafam/G6fKC/KMKf6RYR3NWiLJ7YQ/xzUHlMsBk0Ef356L25ftgdzsMXLS5KDdYhQWk1EdA9ihVdIaiGRUcpETDcgVEOONe4UrCXRTVxh0wehJSYk0KRuQ+xurYs+1KymR2NBbd+EDaWhpekqu1OrMfoSFLSerB9n52M2AUPn6jTWLjDKIyWG0CHBP13XkGE3CkNlfQgywYMf0ITFijbZD1V6+yiiI/ZuAhixd7nl9zRMghdfCKr4h1OQca+Tudk7i+vUy4dAjSxFb2peqhcxiTg2SRQ2MIrhZ+QISRhBdnDOXP9gPAXgIhr8Pyfywb2DOz1OUBhwmI4ApXCCL4zFsL3jGw44+4uq1Y39MFQPY/pi2ddBhIBDY/1f+J4Zittp40N7LQnOemG/CErFNqGSUuum5XHYRnBEv 6f+8n5nd RaiO4HEIo4cvIJ+PiV3KtupWRdfGiExEMZGuyg8zTPuDdm4vgS8G9FecEd8FDy2pxpVLFdaZYxVHkH6fe3t5e/ZqXZvRLpj75RsqUcby2DTQfM8et647iZhMYZFRSdvXS4k6fKOpnsBXY4BItJIidcQHEStaPW+Rq9lN3qb6cLgasR1ht29jPJ2PaNWzYmzDWXFoYtUH9RRX+lTCNUi1kQvEbky/w3BU+7GC9/9tm6aSbd5u0BB9gnywUyfYL9Mrfuflp1YdCZsQVr9DKZnE395Wbg5qzDNkU2b17kMAfGW8D5irvbwsXzNQp05d9FVSSWlbjpz1PhNL0YFAhg5PDawdpTmEbhFZ1vJpLl1Va9y+cPoTDMrZRgIFwfAQm7HqBh02JPt+xofFl85sIgw7yahe+A531SCLtbwH7mFigK1418B91KsQ93G/NzpyfCZ1Jc2G6qy38xiicFmVs10JX1/1CCQlkcSvpiLC9V+Yy0f0STuYz09GUlgsEOz5zriVBh7HSaZrP4SuD9xSZu8hc68OVAXg+aF34hKb3Gh0yJJGVi7H1YbsKEOK3crC7HZ3EBH4MMcBII6ZwmZ+xSKWQi6RtcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/9/23 18:48, David Chen wrote: > When we upgraded our kernel, we started seeing some page corruption like > the following consistently: > > BUG: Bad page state in process ganesha.nfsd pfn:1304ca > page:0000000022261c55 refcount:0 mapcount:-128 mapping:0000000000000000 index:0x0 pfn:0x1304ca > flags: 0x17ffffc0000000() > raw: 0017ffffc0000000 ffff8a513ffd4c98 ffffeee24b35ec08 0000000000000000 > raw: 0000000000000000 0000000000000001 00000000ffffff7f 0000000000000000 > page dumped because: nonzero mapcount > CPU: 0 PID: 15567 Comm: ganesha.nfsd Kdump: loaded Tainted: P B O 5.10.158-1.nutanix.20221209.el7.x86_64 #1 > Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 > Call Trace: > dump_stack+0x74/0x96 > bad_page.cold+0x63/0x94 > check_new_page_bad+0x6d/0x80 > rmqueue+0x46e/0x970 > get_page_from_freelist+0xcb/0x3f0 > ? _cond_resched+0x19/0x40 > __alloc_pages_nodemask+0x164/0x300 > alloc_pages_current+0x87/0xf0 > skb_page_frag_refill+0x84/0x110 > ... > > Sometimes, it would also show up as corruption in the free list pointer and > cause crashes. > > After bisecting the issue, we found the issue started from e320d3012d25: > > if (put_page_testzero(page)) > free_the_page(page, order); > else if (!PageHead(page)) > while (order-- > 0) > free_the_page(page + (1 << order), order); > > So the problem is the check PageHead is racy because at this point we > already dropped our reference to the page. So even if we came in with > compound page, the page can already be freed and PageHead can return > false and we will end up freeing all the tail pages causing double free. > > Fixes: e320d3012d25 ("mm/page_alloc.c: fix freeing non-compound pages") > Cc: Andrew Morton > Cc: Matthew Wilcox (Oracle) > Cc: linux-mm@kvack.org > Cc: stable@vger.kernel.org > Signed-off-by: Chunwei Chen Reviewed-by: Vlastimil Babka That's nasty enough to go into 6.2, IMHO. > --- > mm/page_alloc.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 0745aedebb37..3bb3484563ed 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -5631,9 +5631,12 @@ EXPORT_SYMBOL(get_zeroed_page); > */ > void __free_pages(struct page *page, unsigned int order) > { > + /* get PageHead before we drop reference */ > + int head = PageHead(page); > + > if (put_page_testzero(page)) > free_the_page(page, order); > - else if (!PageHead(page)) > + else if (!head) > while (order-- > 0) > free_the_page(page + (1 << order), order); > }