From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41FF0C3A59D for ; Sun, 7 Apr 2024 04:52:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C88576B0088; Sun, 7 Apr 2024 00:52:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C38A76B0089; Sun, 7 Apr 2024 00:52:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFFAD6B008A; Sun, 7 Apr 2024 00:52:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 92C8A6B0088 for ; Sun, 7 Apr 2024 00:52:03 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1224E1204A1 for ; Sun, 7 Apr 2024 04:52:03 +0000 (UTC) X-FDA: 81981513726.22.E2CA8D4 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf25.hostedemail.com (Postfix) with ESMTP id B3830A0002 for ; Sun, 7 Apr 2024 04:52:00 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=xfgOrlJI; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=Q6YUqoR1; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=SxSc0Nzi; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=MDTGqn+l; spf=pass (imf25.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712465521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TOC31N0dUOY57cqlivIP5Wt1ycysMDUSGgir8PPvt98=; b=hdrqsVL85AeDFXxU8allCS7OngttlJw3RBlEomLYPc6OCp6lcGfcu7htWOunr+HpzKcGgb 6ENs8KMPraJrg72daxF7E3YgHDN4ZK2XiMcgCbNqM2JMLOiWcZN7IYLjWiuVIea+1zSedD PsTzGgTyr1Prn/kKeRwuctCCOxomRrI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712465521; a=rsa-sha256; cv=none; b=UTeKza+BQFBxgONT+Jf0tyuUsk5vFhX5WbfP3JTnygtBtSFLKLHYQZV+AVZj4jyH6HaiUr vp3LCQhoFBvNdnwECZot7IVzw7+d6Wb+L6QsEJ71r9U6J/yhsLzBy0hxbhe7qDicep4MNY ET7kEvI0G3FVSj4FZKRsomeJGkfSKjc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=xfgOrlJI; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=Q6YUqoR1; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=SxSc0Nzi; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=MDTGqn+l; spf=pass (imf25.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8D89221B12; Sun, 7 Apr 2024 04:51:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1712465518; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TOC31N0dUOY57cqlivIP5Wt1ycysMDUSGgir8PPvt98=; b=xfgOrlJID0g/nHPtrmPZ96OM3qgpzum4nHDfrosR5/jtNkGb/mxdXBuH0I9R+4DByJaBUy 8O9Fj+uJicQ2TWz8Te7vpumnJH5tYH+z5CW0FSs3ChWnF4ih5jDi6e9p4wSmcQKp1V23kU rUv8hP8JeH45W6+PH5ZWFsU2YMKaZt0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1712465518; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TOC31N0dUOY57cqlivIP5Wt1ycysMDUSGgir8PPvt98=; b=Q6YUqoR1A+5sAH+FvWzsn+xbX7vgZLrrqCDZP9PQIud9UudaZxlhaoxtjioyLj8n62rM/Q qiI9UeApFZ8jZcBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1712465517; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TOC31N0dUOY57cqlivIP5Wt1ycysMDUSGgir8PPvt98=; b=SxSc0NzixFvARdW/eNzWfg7k1mih2NPOl6Idm0efJNYYxbVFp2JFkIWyfWFdbaDFLdYo9p kg8hBxb+CtQLBmS4okSnmTo1+jY9fEFRH74sXd1456RAIkdN9cjyh0XQixDBScM8dVMtLo n8Y+s+8luDySDrc18z4UrQr3VQOoSRA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1712465517; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TOC31N0dUOY57cqlivIP5Wt1ycysMDUSGgir8PPvt98=; b=MDTGqn+lES5vnX7xHbDoxgQJu5tf68v7a4lx0SZvNUkNRv1FB/DbSUOI2sOoaiwyl/C8a+ jz4v2FzGfetDmHDw== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 06970132FF; Sun, 7 Apr 2024 04:51:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id UNIeOmwmEmZhDwAAn2gu4w (envelope-from ); Sun, 07 Apr 2024 04:51:56 +0000 Date: Sun, 7 Apr 2024 06:51:51 +0200 From: Oscar Salvador To: "Luck, Tony" Cc: David Hildenbrand , Borislav Petkov , Yazen Ghannam , Miaohe Lin , Naoya Horiguchi , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" Subject: Re: Machine check recovery broken in v6.9-rc1 Message-ID: References: <1e943439-6044-4aa4-8c41-747e9e4dca27@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Action: no action X-Stat-Signature: i56dsjrr5nexk4h1twa5om5kza6j7jzd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B3830A0002 X-Rspam-User: X-HE-Tag: 1712465520-372812 X-HE-Meta: U2FsdGVkX194ceHtYDJycugEYADxSBeHh+ID5jEZ6BKjCtIbBwbsYxZDSPmMG5+UyfAEncQcUpp5tkcpsW0JPXDEmszp69IjokeyZ4MOCTrFD4q3qdiUZf1FM9C1DZP0JA8KUv44IN6Uf9RVZ/IcrcnOZyG99mrXVlmWO4y9df+4uQQJsWC/WUFodga1Mekn3lx7sfe5rNe+7eBW7JXWhPEWpxdfMLxl0f4XAAqlYrrGCd7CfnKFIKHyf397QJqPQXt00yxXUWm8J60EZ06BcrkrzTPSRWKtxzC0WwbFNbbokBQetjILF/U4F/9lqEztS4v+3nWPBMNgMALP4rK6oTnZPgEGRrbrawW2890J2rZlC9Ea9kSfp80jxocvM9+/ycJAXt+gRPf7ZcIjEn9qowlCvmFOaPrvjpqwPmxkOXim9hQu+0kTZvAViCDFy4uBcGuPXI8+9Z13B6vSJg8DgWnqbiRFUwOwuvJbHlQl6tf9oN0dz3V+WNSZ9X+iFOdNL9RAkj52D7ObkRpBLVy9GVVO2pW5zYys7go5GZ2xNRuCiEiwwUGnkn5XH9MYosB00wMChZ7bOKGMCLXfw5Ym9m2fRCnqNhwM98MtwQP9d/PRaJvVkDF+M65yrBtYuDqFTbk+wD63bVpoYhmA5+XTVbiIDjQRTKlurvkTW7HFGuOnq08y4JF07bVFj1sCbM0Rrr6FisGAUvTKNPaUY09gSCPkiN0NTw7F1itjYYfSwk3a2jkeMRUCEkEdpScxiBsEvgfw/Zj8JmIYdrgnm5biHw06NN+fNPcUTSk4R2mzBeI0tR/P1wyFtI+S581zT1aNbEpQFvQih+Zx5DALBGwiQpb0zsd2fhGYQvjtu1BVHyaaLVpSeyr/1vdzlzwGa8HtsqOqv0e2zeB7SlX+/GWR3iqGxbkIv0N37te+bH/hO7ohP3Onhm1hWHfmk65JpS4FTBK8P1khMOdigiyzACk /Qu1RupM weZ7U5uIyOq8PjfV1q82HrLyhVE2o77iewPrOWDBw9FhREUhT5Mnw0IplhLN9R6AaJH9FHzlMdO3Wroa/QVFveb7J/UwKPftF6YSLtLYW0DlHydbbxuOxOmLHSjHTT3StYBEbteJHKYuuqBcANYHu6hn1GmmC9Etc+wLPHfrF6+XxQkpBzsfb1cVq1/CWNaQdC70uxgdL5Bw4cxreDyQLbzw3Kc48JifaW8THkQBfxoEBZgL8Wl6yJtQLXGGXNWv3BIe/D59iz6rpx4s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Apr 07, 2024 at 12:08:30AM +0000, Luck, Tony wrote: > Oscar. > > Both the 6.1 and 6.9-rc2 patches make the BUG (and subsequent issues) go away. Thanks for the switf test Tony! > Here's what's happening. > > When the machine check occurs there's a scramble from various subsystems > to report the memory error. > > ghes_do_memory_failure() calls memory_failure_queue() which later > calls memory_failure() from a kernel thread. Side note: this happens TWICE > for each error. Not sure yet if this is a BIOS issue logging more than once. > or some Linux issues in acpi/apei/ghes.c code. > > uc_decode_notifier() [called from a different kernel thread] also calls > do_memory_failure() > > Finally kill_me_maybe() [called from task_work on return to the application > when returning from the machine check handler] also calls memory_failure() > > do_memory_failure() is somewhat prepared for multiple reports of the same > error. It uses an atomic test and set operation to mark the page as poisoned. > > First called to report the error does all the real work. Late arrivals take a > shorter path, but may still take some action(s) depending on the "flags" > passed in: > > if (TestSetPageHWPoison(p)) { > pr_err("%#lx: already hardware poisoned\n", pfn); > res = -EHWPOISON; > if (flags & MF_ACTION_REQUIRED) > res = kill_accessing_process(current, pfn, flags); > if (flags & MF_COUNT_INCREASED) > put_page(p); > goto unlock_mutex; > } Thanks for the detailed explanation. > In this case the last to arrive has MF_ACTION_REQUIRED set, so calls > kill_accessing_process() ... which is in the stack trace that led to the: > > kernel BUG at include/linux/swapops.h:88! > > I'm not sure that I fully understand your patch. I guess that it is making sure to > handle the case that the page has already been marked as poisoned? Basically what is happening is: 1) We mark the page as HWPoison 2) We see that the page is mapped by someone 3) We try to unmap it, and in the process we create a hwpoison swap entry. See the following chunk from try_to_unmap_one(): " if (PageHWPoison(subpage) && (flags & TTU_HWPOISON)) { pteval = swp_entry_to_pte(make_hwpoison_entry(subpage)); if (folio_test_hugetlb(folio)) { hugetlb_count_sub(folio_nr_pages(folio), mm); set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); } else { dec_mm_counter(mm, mm_counter(folio)); set_pte_at(mm, address, pvmw.pte, pteval); } ... } " 4) Now there is a second memory event (maybe the previous one has already finished, I do not think it matters for the sake of this problem) 5) The second event sees that the page has already been marked as HWPoison but since it has MF_ACTION_REQUIRED specified, it goes to kill_accessing_process() to do what its name says. 6) We walk the page tables of the accessing process to see if it has the poisoned pfn. 7) check_hwpoisoned_entry() (which is called from walk_page_range()->walk_{pgd,p4d,pud,pmd}_range()->ops->pmd_entry()) checks whether any of the ptes is poisoned. 8) Since the previous MCE event unmapped the page, pte_present() == 0, so we want to get the swap entry, and this is where it falls off the cliff. See check_hwpoisoned_entry() static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift, unsigned long poisoned_pfn, struct to_kill *tk) { unsigned long pfn = 0; if (pte_present(pte)) { pfn = pte_pfn(pte); } else { swp_entry_t swp = pte_to_swp_entry(pte); if (is_hwpoison_entry(swp)) pfn = swp_offset_pfn(swp); } ... } is_hwpoison_entry() returns true (remember the make_hwpoison_entry() call we did?) But when we try to get the pfn from the swap entry, we stumble upon the VM_BUG_ON(), because is_pfn_swap_entry() only checks for: is_migration_entry() is_device_private_entry() is_device_exclusive_entry() but it should also check for is_hwpoison_entry(). Since it does not, is_pfn_swap_entry() returns false in our case, leading to the VM_BUG_ON. Note that this should only matter in environments where CONFIG_DEBUG_VM is set. I hope I shed some light in here. > Anyway ... thanks for the quick fix. I hope the above helps write a good > commit message to get this applied and backported to stable. > > Tested-by: Tony Luck Thanks again Tony, much appreciated. I will write the patch and most likely send it out either today in the afternoon or tomorrow early in the morning. -- Oscar Salvador SUSE Labs