From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09CFBCAC5B2 for ; Wed, 18 Sep 2024 02:16:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D85A96B0082; Tue, 17 Sep 2024 22:16:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D35076B0083; Tue, 17 Sep 2024 22:16:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFC7B6B0085; Tue, 17 Sep 2024 22:16:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A60A46B0082 for ; Tue, 17 Sep 2024 22:16:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 150DA16023D for ; Wed, 18 Sep 2024 02:16:31 +0000 (UTC) X-FDA: 82576244982.29.44B2234 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) by imf13.hostedemail.com (Postfix) with ESMTP id 50D8520008 for ; Wed, 18 Sep 2024 02:16:24 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="VbwWFf8/"; spf=pass (imf13.hostedemail.com: domain of xueshuai@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=xueshuai@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726625675; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iQVRTT3zRn6y1FZI9IrSAdswUSfREPmbfPT4pr4LFsc=; b=WtGY6nCkyiqSMVpu8uKn8W9FQqknhUqbmQado9ZtGcwFoWkDZDkMEccvjQ6ka4yXw7D2mT nMFQzHetd32vCylpex0ebCQA/49U92+sWhG+a+sJmrW5SUf64e/c2dp8KuZoiRCVVH1QPi mIbmtTcmcfIQYjVX1NoKDmtQBCuhNAI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726625675; a=rsa-sha256; cv=none; b=e0YfHOLefh+Stlez5dEEwLKhTjc69Mm2uDPnllzYoYGi6YzsVXIonlmIDYXJm5UsGTvRqH 2noYnoG/wLFCDLwz3Hcm3xWm7/eBqxlIp6fSpBZ/3cYcyylcj6lTZNVmuUjavTgRMGTq4b eCL1HjMn3bGJooRqwoejvnAihUfx/LI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="VbwWFf8/"; spf=pass (imf13.hostedemail.com: domain of xueshuai@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=xueshuai@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1726625775; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=iQVRTT3zRn6y1FZI9IrSAdswUSfREPmbfPT4pr4LFsc=; b=VbwWFf8/t2FFJSfIFR77ir/Le0OYmsSrOiN2JV1SI6uvGgewy64j/+reoNsAws0w2BvrChFZRp+rTShy2uYRVKE/b/E+qChfHUwLxbU9YAiarwdJz6KdX/GVbTGYzRYR3fhEHXj2ZtduybxpLFxubjI4NwMdADlSXSx1Eb+BQBc= Received: from 30.246.161.141(mailfrom:xueshuai@linux.alibaba.com fp:SMTPD_---0WFC7uFG_1726625771) by smtp.aliyun-inc.com; Wed, 18 Sep 2024 10:16:13 +0800 Message-ID: <3e1be2ab-3fad-4025-8b87-8daf1305e0c3@linux.alibaba.com> Date: Wed, 18 Sep 2024 10:16:10 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v12 0/3] ACPI: APEI: handle synchronous errors in task work To: bp@alien8.de, rafael@kernel.org, wangkefeng.wang@huawei.com, tanxiaofei@huawei.com, mawupeng1@huawei.com, tony.luck@intel.com, linmiaohe@huawei.com, naoya.horiguchi@nec.com, james.morse@arm.com, tongtiangen@huawei.com, gregkh@linuxfoundation.org, will@kernel.org, jarkko@kernel.org Cc: linux-acpi@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, linux-edac@vger.kernel.org, x86@kernel.org, justin.he@arm.com, ardb@kernel.org, ying.huang@intel.com, ashish.kalra@amd.com, baolin.wang@linux.alibaba.com, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, lenb@kernel.org, hpa@zytor.com, robert.moore@intel.com, lvying6@huawei.com, xiexiuqi@huawei.com, zhuo.song@linux.alibaba.com References: <20221027042445.60108-1-xueshuai@linux.alibaba.com> <20240902030034.67152-1-xueshuai@linux.alibaba.com> From: Shuai Xue In-Reply-To: <20240902030034.67152-1-xueshuai@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 50D8520008 X-Stat-Signature: s7erc4num1n9zj15nes1abrsw3xcb54g X-HE-Tag: 1726625784-533187 X-HE-Meta: U2FsdGVkX1802524Lf3T3pxPayZxZbmL5jiX8Xf1AX+gjbHAzToSBciQJprhnndw2P+X8TtfekOWDeVS5iNdRWubyABs/MRfpOM+rr9eoSUIUMw9IpsOomUY5O3+2wfylaayVNnXwXJEfKSaZa6rD4yFjOP3Q7wtwRgIdupzWXjqy1+2t4N2blEBjCIQu0nGY8Y12fTCStzdvHPMsxOuDKzcZtkD6xgXM5TVdRVzO0Bn5afVKfz3sZLZLqKoUas2OgEd9IPI/P0yBPuqsIEMQQ/bRswwAb5ov2m093ows+n62YxN8E7aUduu1Za69uqtRv0jawCk0gBcsC5vCoocxgP5rthMqFFCoVyggvIrBwWydaDeHBR/AroT7MDAVzftK4Ta/6m+KpbMlErWE+NWKBRKAyB4Dbi99GNyol8Ki0t4Bjp9TGQ2v1pWcHDpGQRQuHrQfJbyiwKjY5EsEvKGNNsk4mmWFiJeP8prVkiDL4gIA+qWt/wktBx24GdzTKdkDaCqm9hujIfLj3tif326e+kuUMfvAOki2VKlheiDAxsin0RwFyKeDCqrGUX19jVej6TP6eWa1En9VvxjFOji4E81RrJpMHqfuHJAFv7fWttw64GwApEKBRw9Boil46aQmXHAWacJ1lEWlGjAI0FmETmcFsAsqWrMYEI5yV8HLaLhA+zwhg1F+piU35o56K8M4LnDf9olafii6pCiXnifSJWOsInR/piA37vA9Xgtk0rADIJJUBVKW0uY/ab2m9vZ+SgcbZISNyGAsc3IicHN4Q1VJSQ4EnPvlV4rLng+10tqIGJPycC6fF8C2tg695TS9d8bBV3HmCxMSK7rQMvANoBIJsqdFTMFzn28xsElHBLYJ6SEucJbQMGvdrpEj4ZrfDAoayi24mNWlezvr7qU3wzx0ff+PVzxGIip697dUx2czwcm3ZaCjIG9fVQPV1z+B/P1uBLQZDRA3lFNlOa 3VuyljKA TTe51OpdmXRFYDXI3P5v9CnmGbSZIYjj+U85HbRpkfY302MCZcGZJ6vgrZvYwFFE1V6ZvNO2bsqVZ2ypOSZaYRCFWm+q5ba9rqb3BKgwAjG2kV7/cgm0UfTVYVVdh2/66XzXK1LmWmZeLXfVZyINQ5q1QSSfqTTuKOsOXeKQow4yqKK5qqSlxt/lIuRBEbzB5sg/M7avGRCqjaOa1+eV7dXPJ095tN058i8f+h/W0kBTyTD2pz61tlpSXI8xDePLAw+Tr0x8olVk+texoJ5LH8D7TXLs01Bk5OotKWERDOwRoiPH8qC3CgoyM70AwPqTEK4QKQC1aYSZGKum7PjdPhKR4dyU1ROYg0IqLshIT/7ZEG6W3iYlG+BFYBefHd3bnXCqM3zmYmTxJwmuX0NubUmvSB0VFMPOfAaPKEqqilGCgNz6IRW+mgs4/F6umhxUwU4JEBnzSCMzw9t1PDv1cQGS/d/dQizPDFSrRyXh8y5ujp6k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, all, Gentle ping. Best Regards, Shuai 在 2024/9/2 11:00, Shuai Xue 写道: > ## Changes Log > > changes since v11: > - rebase to Linux 6.11-rc6 > - fix grammer and typo in commit log (per Borislav) > - remove `sync_` perfix of `sync_task_work` (per Borislav) > - comments flags and description of `task_work` (per Borislav) > > changes since v10: > - rebase to v6.8-rc2 > > changes since v9: > - split patch 2 to address exactly one issue in one patch (per Borislav) > - rewrite commit log according to template (per Borislav) > - pickup reviewed-by tag of patch 1 from James Morse > - alloc and free twcb through gen_pool_{alloc, free) (Per James) > - rewrite cover letter > > changes since v8: > - remove the bug fix tag of patch 2 (per Jarkko Sakkinen) > - remove the declaration of memory_failure_queue_kick (per Naoya Horiguchi) > - rewrite the return value comments of memory_failure (per Naoya Horiguchi) > > changes since v7: > - rebase to Linux v6.6-rc2 (no code changed) > - rewritten the cover letter to explain the motivation of this patchset > > changes since v6: > - add more explicty error message suggested by Xiaofei > - pick up reviewed-by tag from Xiaofei > - pick up internal reviewed-by tag from Baolin > > changes since v5 by addressing comments from Kefeng: > - document return value of memory_failure() > - drop redundant comments in call site of memory_failure() > - make ghes_do_proc void and handle abnormal case within it > - pick up reviewed-by tag from Kefeng Wang > > changes since v4 by addressing comments from Xiaofei: > - do a force kill only for abnormal sync errors > > changes since v3 by addressing comments from Xiaofei: > - do a force kill for abnormal memory failure error such as invalid PA, > unexpected severity, OOM, etc > - pcik up tested-by tag from Ma Wupeng > > changes since v2 by addressing comments from Naoya: > - rename mce_task_work to sync_task_work > - drop ACPI_HEST_NOTIFY_MCE case in is_hest_sync_notify() > - add steps to reproduce this problem in cover letter > > changes since v1: > - synchronous events by notify type > - Link: https://lore.kernel.org/lkml/20221206153354.92394-3-xueshuai@linux.alibaba.com/ > > ## Cover Letter > > There are two major types of uncorrected recoverable (UCR) errors : > > - Synchronous error: The error is detected and raised at the point of the > consumption in the execution flow, e.g. when a CPU tries to access > a poisoned cache line. The CPU will take a synchronous error exception > such as Synchronous External Abort (SEA) on Arm64 and Machine Check > Exception (MCE) on X86. OS requires to take action (for example, offline > failure page/kill failure thread) to recover this uncorrectable error. > > - Asynchronous error: The error is detected out of processor execution > context, e.g. when an error is detected by a background scrubber. Some data > in the memory are corrupted. But the data have not been consumed. OS is > optional to take action to recover this uncorrectable error. > > Currently, both synchronous and asynchronous error use > memory_failure_queue() to schedule memory_failure() exectute in kworker > context. As a result, when a user-space process is accessing a poisoned > data, a data abort is taken and the memory_failure() is executed in the > kworker context: > > - will send wrong si_code by SIGBUS signal in early_kill mode, and > - can not kill the user-space in some cases resulting a synchronous > error infinite loop > > Issue 1: send wrong si_code in early_kill mode > > Since commit a70297d22132 ("ACPI: APEI: set memory failure flags as > MF_ACTION_REQUIRED on synchronous events")', the flag MF_ACTION_REQUIRED > could be used to determine whether a synchronous exception occurs on > ARM64 platform. When a synchronous exception is detected, the kernel is > expected to terminate the current process which has accessed poisoned > page. This is done by sending a SIGBUS signal with an error code > BUS_MCEERR_AR, indicating an action-required machine check error on > read. > > However, when kill_proc() is called to terminate the processes who have > the poisoned page mapped, it sends the incorrect SIGBUS error code > BUS_MCEERR_AO because the context in which it operates is not the one > where the error was triggered. > > To reproduce this problem: > > # STEP1: enable early kill mode > #sysctl -w vm.memory_failure_early_kill=1 > vm.memory_failure_early_kill = 1 > > # STEP2: inject an UCE error and consume it to trigger a synchronous error > #einj_mem_uc single > 0: single vaddr = 0xffffb0d75400 paddr = 4092d55b400 > injecting ... > triggering ... > signal 7 code 5 addr 0xffffb0d75000 > page not present > Test passed > > The si_code (code 5) from einj_mem_uc indicates that it is BUS_MCEERR_AO > error and it is not fact. > > To fix it, queue memory_failure() as a task_work so that it runs in > the context of the process that is actually consuming the poisoned data. > > After this patch set: > > # STEP1: enable early kill mode > #sysctl -w vm.memory_failure_early_kill=1 > vm.memory_failure_early_kill = 1 > > # STEP2: inject an UCE error and consume it to trigger a synchronous error > #einj_mem_uc single > 0: single vaddr = 0xffffb0d75400 paddr = 4092d55b400 > injecting ... > triggering ... > signal 7 code 4 addr 0xffffb0d75000 > page not present > Test passed > > The si_code (code 4) from einj_mem_uc indicates that it is BUS_MCEERR_AR > error as we expected. > > Issue 2: a synchronous error infinite loop due to memory_failure() failed > > If a user-space process, e.g. devmem, a poisoned page which has been set > HWPosion flag, kill_accessing_process() is called to send SIGBUS to the > current processs with error info. Because the memory_failure() is > executed in the kworker contex, it will just do nothing but return > EFAULT. So, devmem will access the posioned page and trigger an > excepction again, resulting in a synchronous error infinite loop. Such > loop may cause platform firmware to exceed some threshold and reboot > when Linux could have recovered from this error. > > To reproduce this problem: > > # STEP 1: inject an UCE error, and kernel will set HWPosion flag for related page > #einj_mem_uc single > 0: single vaddr = 0xffffb0d75400 paddr = 4092d55b400 > injecting ... > triggering ... > signal 7 code 4 addr 0xffffb0d75000 > page not present > Test passed > > # STEP 2: access the same page and it will trigger a synchronous error infinite loop > devmem 0x4092d55b400 > > To fix it, if memory_failure() failed, perform a force kill to current process. > > Issue 3: a synchronous error infinite loop due to no memory_failure() queued > > No memory_failure() work is queued unless all bellow preconditions check passed: > > - `if (!(mem_err->validation_bits & CPER_MEM_VALID_PA))` in ghes_handle_memory_failure() > - `if (flags == -1)` in ghes_handle_memory_failure() > - `if (!IS_ENABLED(CONFIG_ACPI_APEI_MEMORY_FAILURE))` in ghes_do_memory_failure() > - `if (!pfn_valid(pfn) && !arch_is_platform_page(physical_addr)) ` in ghes_do_memory_failure() > > If the preconditions are not passed, the user-space process will trigger SEA again. > This loop can potentially exceed the platform firmware threshold or even > trigger a kernel hard lockup, leading to a system reboot. > > To fix it, if no memory_failure() queued, perform a force kill to current process. > > And the the memory errors triggered in kernel-mode[5], also relies on this > patchset to kill the failure thread. > > Lv Ying and XiuQi from Huawei also proposed to address similar problem[2][4]. > Acknowledge to discussion with them. > > [1] Add ARMv8 RAS virtualization support in QEMU https://patchew.org/QEMU/20200512030609.19593-1-gengdongjiu@huawei.com/ > [2] https://lore.kernel.org/lkml/20221205115111.131568-3-lvying6@huawei.com/ > [3] https://lkml.kernel.org/r/20220914064935.7851-1-xueshuai@linux.alibaba.com > [4] https://lore.kernel.org/lkml/20221209095407.383211-1-lvying6@huawei.com/ > [5] https://patchwork.kernel.org/project/linux-arm-kernel/cover/20240528085915.1955987-1-tongtiangen@huawei.com/ > > Shuai Xue (3): > ACPI: APEI: send SIGBUS to current task if synchronous memory error > not recovered > mm: memory-failure: move return value documentation to function > declaration > ACPI: APEI: handle synchronous exceptions in task work > > arch/x86/kernel/cpu/mce/core.c | 7 --- > drivers/acpi/apei/ghes.c | 86 +++++++++++++++++++++------------- > include/acpi/ghes.h | 3 -- > include/linux/mm.h | 1 - > mm/memory-failure.c | 22 +++------ > 5 files changed, 60 insertions(+), 59 deletions(-) >