From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87566C25B7E for ; Tue, 4 Jun 2024 07:19:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B1646B009A; Tue, 4 Jun 2024 03:19:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 039226B009C; Tue, 4 Jun 2024 03:19:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1BC76B009D; Tue, 4 Jun 2024 03:19:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C14C96B009A for ; Tue, 4 Jun 2024 03:19:44 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 33C3112069E for ; Tue, 4 Jun 2024 07:19:44 +0000 (UTC) X-FDA: 82192356288.01.1D9A14B Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf10.hostedemail.com (Postfix) with ESMTP id CD7F0C0004 for ; Tue, 4 Jun 2024 07:19:40 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717485582; a=rsa-sha256; cv=none; b=t/GEdatuoDIzQRwW9K+sTM4/8mAowzCVOeB7xYxvEG2vZJkF5STzYFKSRGNsTSpr6s2Mrs mYWFKO3ikxMWH1v1GBTJuZWFkD4TgTxRUByykSqvzIonYSLHBXwfnDiR0YlWal5oWAgPMl v/rVK+IQTI2bNsc6J+9RC19nADVXpT0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717485582; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IAcbsucx9H1RyrD8TAVnojEgfkhsjtsVuNpR3UPNEAs=; b=DRQ+Vjqiuwaofo+zciSYhYSTkZi5qsaLevYZ26vNqlEHhnYoIKK+1BixqAB9b0r1PmR5Sm Nvrkd5NRx8X+dr0zccccqcEWZzXHIhGPHafDETh10tqXcjmN5tyqtmlVypihOsMaPttd7P 82PUlv28lfdhMM3khbRn+bKJEM2L5yE= Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4Vthj02kJWz1ysV7; Tue, 4 Jun 2024 15:16:20 +0800 (CST) Received: from canpemm500002.china.huawei.com (unknown [7.192.104.244]) by mail.maildlp.com (Postfix) with ESMTPS id 1CF0E1A016C; Tue, 4 Jun 2024 15:19:36 +0800 (CST) Received: from [10.173.135.154] (10.173.135.154) by canpemm500002.china.huawei.com (7.192.104.244) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 4 Jun 2024 15:19:35 +0800 Subject: Re: [PATCH v1 0/3] Userspace controls soft-offline HugeTLB pages To: Jiaqi Yan , CC: , , , , , , , , , , , References: <20240531213439.2958891-1-jiaqiyan@google.com> From: Miaohe Lin Message-ID: <21841726-6427-5a92-a3d1-8aac2687c01a@huawei.com> Date: Tue, 4 Jun 2024 15:19:35 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20240531213439.2958891-1-jiaqiyan@google.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.173.135.154] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To canpemm500002.china.huawei.com (7.192.104.244) X-Rspamd-Queue-Id: CD7F0C0004 X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: yot34dfuux5znfzwxkyr8hj1gjpjr5xd X-HE-Tag: 1717485580-544602 X-HE-Meta: U2FsdGVkX18tqSOXuXtkaf3y0kCtJqWdMFjFERiBqx4HuLHym0Wjh5p/pMwCBtbf0gVVWxKNWLR7GMI5xHA2cjfBOexhnIPa276+p1JyEixnEXU3xb2pPHL/le5am1qlIyRAlEqlZxQ02OimPN4rXfJqSOybMO47EHl4NVXjzek+upClvZumcJgdwC37TlTynC37TbmRWs6jyaoklP0hxR+XbM9NgFGN0HnSfPPZ0ARn9YoEbFx6ivxgF9EsrmRZi45vDyGuslZWyFsgu5CPP19iZLsVivPmuJdNmy8V1CN7nXrRt+jqiSQI71rJOXdOQRxGskyzKz7ngXx/bbJetSHl06VOX9Mhcv3a9/pPOiOInb5FHKuie7Q9yRloJ6QSuNP0ZULqGEiIKnXH58P/CV/LeJSaI0DvfhON+WJw83G8DW+egG5M/2rQFWqPMeWZP4rhFJPaD58K+5Eu+PZLYWyBNb00kCw6OznSm1TBRA8LNeYOPTo7I/3rBLk9+gHQRp0A5zU49bDjWIog1CVKd2B8S0OXzpwCx1uEmvYRDEpko2PPfOBW12WzgyFpi6nk5pW/3XQy/jBkYLEOKJX0CCgYPWkPjpJOD91bSeYfIu8Y0/7gHh4dfoWFZxsFsoEXodlEpg2ffCiQQCM8TPHRkW+fw731gBuFUH7QTo5h4VBiHnc/rYBVDmYfcg204ToHxCF4cqCFvUaS5vDsOsSynyyGgNKSgqMtf1um5cZATPyjRvECjy6GcnNNrFD7yQdm7ItbX3W372sdq237559h4n1M7zo2FtIAXmibRcE85XPaWOlkwXkr9g6e0pAjZdDA32fqNmORo2CftkNYL7AOTq5V1M7pwxicHXPyGdyYp36UmxIquB1NOHQBksBQQ0c5D5WQ65kBvby50T++3B1GZulIrgFJoNeEUNTSx1tNJwWWRi/peIzYy9nn6JKJIITN6FxQgE8XGeI2XtBRWOh MjSk+4nC xlJW1QfCbwdb34/pN1NFoTmSRGyVU8CWpf5Y8gJQZGWzNUki4i3OgTnh0hSsWQIGe+eij0Bbj30ufOcRBux3hRLhvgSoIVZ9ntE13q/MCFANbzttfCaV/J6Z+bXzmlSPisyBltdbX5QMudVI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/6/1 5:34, Jiaqi Yan wrote: > Correctable memory errors are very common on servers with large > amount of memory, and are corrected by ECC, but with two > pain points to users: > 1. Correction usually happens on the fly and adds latency overhead > 2. Not-fully-proved theory states excessive correctable memory > errors can develop into uncorrectable memory error. Thanks for your patch. > > Soft offline is kernel's additional solution for memory pages > having (excessive) corrected memory errors. Impacted page is migrated > to healthy page if it is in use, then the original page is discarded > for any future use. > > The actual policy on whether (and when) to soft offline should be > maintained by userspace, especially in case of HugeTLB hugepages. > Soft-offline dissolves a hugepage, either in-use or free, into > chunks of 4K pages, reducing HugeTLB pool capacity by 1 hugepage. > If userspace has not acknowledged such behavior, it may be surprised > when later mmap hugepages MAP_FAILED due to lack of hugepages. For in use hugetlb folio case, migrate_pages() is called. The hugetlb pool capacity won't be modified in that case. So I assume you're referring to the free hugetlb folio case? The Hugetlb pool capacity is reduced in that case. But if we don't do that, we might encounter uncorrectable memory error later which will be more severe? Will it be better to add a way to compensate the capacity? > In addition, discarding the entire 1G memory page only because of > corrected memory errors sounds very costly and kernel better not > doing under the hood. But today there are at least 2 such cases: > 1. GHES driver sees both GHES_SEV_CORRECTED and > CPER_SEC_ERROR_THRESHOLD_EXCEEDED after parsing CPER. > 2. RAS Correctable Errors Collector counts correctable errors per > PFN and when the counter for a PFN reaches threshold > In both cases, userspace has no control of the soft offline performed > by kernel's memory failure recovery. Userspace can figure out the hugetlb folio pfn range by using `page-types -b huge -rlN` and then decide whether to soft offline the page according to it. But for GHES driver, I think it has to be done in the kernel. So add a control in /sys/ seems like a good idea. > > This patch series give userspace the control of soft-offlining > HugeTLB pages: kernel only soft offlines hugepage if userspace has > opt-ed in for that specific hugepage size, and exposed to userspace > by a new sysfs entry called softoffline_corrected_errors under > /sys/kernel/mm/hugepages/hugepages-${size}kB directory: > * When softoffline_corrected_errors=0, skip soft offlining for all > hugepages of size ${size}kB. > * When softoffline_corrected_errors=1, soft offline as before this Will it be better to be called as "soft_offline_corrected_errors" or simplify "soft_offline_enabled"? Thanks. .