From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94210C27C53 for ; Fri, 7 Jun 2024 22:23:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01D0F6B0093; Fri, 7 Jun 2024 18:23:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE8856B0095; Fri, 7 Jun 2024 18:22:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D89646B0098; Fri, 7 Jun 2024 18:22:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B43E06B0093 for ; Fri, 7 Jun 2024 18:22:59 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 13E181419AB for ; Fri, 7 Jun 2024 22:22:59 +0000 (UTC) X-FDA: 82205518878.30.2263667 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by imf08.hostedemail.com (Postfix) with ESMTP id 3A6DA160006 for ; Fri, 7 Jun 2024 22:22:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0yyO5TGy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=jiaqiyan@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717798977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ouTQPU5YyVEsaf9+24FgrOmjgrgA/NCQDAtyFoUqx6w=; b=bpiLh5RNJw4DMTe5nzA8uo48klZtDOGD3NX45AZuq0Xtqyodh3DwKf1hcIYOzOfMbunoNY N3VoXDekk7qXbEZbWWHt1UCT3F9IR9Lz8ikbvgg2pFUdIqafjY4d38V0BUsSfdQcRTLYIZ 3UX6BhIEJM4I7t4QXrrw2XEd/PNJimM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717798977; a=rsa-sha256; cv=none; b=c3puXC2rIEv8bDntqEh2KVRoXx3WS93q4kaj3X5JQGtpj6Zhq7x7L5MQ1s42KlfngYRWo4 QVxY4TO6Q2anpuaakYMXmDX7itQdbe5c5mC9I/h6/wa/j5yzfwHPSs7XckHCLxn5SXEquA 3nVl7mAfRy8HfCGXK00GcjntpPl96ho= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0yyO5TGy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=jiaqiyan@google.com Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-42163fa630aso15493695e9.3 for ; Fri, 07 Jun 2024 15:22:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717798975; x=1718403775; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ouTQPU5YyVEsaf9+24FgrOmjgrgA/NCQDAtyFoUqx6w=; b=0yyO5TGyZ0Fxk9Z3kPf+ZFRy6UAVo5M1sDRbwQyqz+XTlR12/0kR1VE4am53bEenJO oxolYH9Z7z6tGWDUyPOxm7sEeEWfotF2tX0Zp5DlW2UaYtg/c1V6RAyT+GSvtILJJm0z 3fNRkcqzPfw8fPpfY2UtRyUe2l6bSSnuTkRmmVoWz29IGA8YX0xkZ/PBzGG09/QP8+QI ndDl7dYIbXDNAZ4b5iDMGnKKN6Xelp/XJ4zcS2zxRKT0nD7BkJU2a0040zq09s+G7VMs uJ+hkIh/zIDzhsCdvaMR9PRTu9YwcuC7wqpg9fj1lV1JCi13dnVhLEc/oBq+/du2QNgd 8DOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717798975; x=1718403775; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ouTQPU5YyVEsaf9+24FgrOmjgrgA/NCQDAtyFoUqx6w=; b=S59a2Reqtbcwhg/ujvZk3NiCJmagiU9kRpD6bwNLXeujwCGLw0xHzySCFHO+x0ZN2b gSyXb7dK8f4k8oGA0LlBjBUxeU86Th3lAZRa6nPqeJ9Ua0iWLTiAJK80s3SnfO78ACVa c07Tic52HHpgO78N1bzzFTnKsaGHSzs+ROROYqVrcYPMoPsfpvCiYjZr+KR8C5uP4Gf1 GNXq64SJaWhhEEaC3iC66iYJYnJ/6MpCWipIQsQHg4X5yoexO+1nUhbhvashzaZpU0LQ OJYkYr7SGUB4LbSAFH52z+7MkkR0awguaxk+gQDD42qvBixeepwJ6jrWj08PJlNNvh2U 4Hgw== X-Forwarded-Encrypted: i=1; AJvYcCVLdWn1HEKq0z5/AEeLgU1T2E+vMPcoG1/StxHoYESiNjTf07JSixHRGPRHLVK5Aukt8MThOPtOTh3WCUG8CWdWplo= X-Gm-Message-State: AOJu0YxAsVuiNRlEDpkIKH1vVzEs19CsoGMZQDF6oPOqXiWal41HH6yk VVCioOAlyuktnemWq641MPD81OUgW5Li6rCnuqNwdGaMVS0mc5asdAr6Bjq+YYwy/tvJnw3Bn2k 2GLgrcB3GslA3avDP/eWJLtvV64a4talK9Ug7 X-Google-Smtp-Source: AGHT+IFPN9H9WFajQMarkR9dF71kl+7xMtyZUOalUgT1A2vw8tRER0AD6uQpeQtDgn3zXVOr+fDaQAS4f/qC9XF5hOE= X-Received: by 2002:adf:e543:0:b0:35e:51cf:690d with SMTP id ffacd0b85a97d-35efea823afmr2588117f8f.0.1717798975238; Fri, 07 Jun 2024 15:22:55 -0700 (PDT) MIME-Version: 1.0 References: <20240531213439.2958891-1-jiaqiyan@google.com> <21841726-6427-5a92-a3d1-8aac2687c01a@huawei.com> In-Reply-To: <21841726-6427-5a92-a3d1-8aac2687c01a@huawei.com> From: Jiaqi Yan Date: Fri, 7 Jun 2024 15:22:43 -0700 Message-ID: Subject: Re: [PATCH v1 0/3] Userspace controls soft-offline HugeTLB pages To: Miaohe Lin Cc: naoya.horiguchi@nec.com, akpm@linux-foundation.org, shuah@kernel.org, corbet@lwn.net, osalvador@suse.de, rientjes@google.com, duenwen@google.com, fvdl@google.com, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, muchun.song@linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: b4fa3nsrn61mcuzaxzyia5a8gdmsqepu X-Rspamd-Queue-Id: 3A6DA160006 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717798977-578873 X-HE-Meta: U2FsdGVkX188xHhSbLxbeSio14sDeNBBH4upQjXSHtcWVFSXLGiVOFAEbmVCZKCHcXQC7oFfccLFx+crIcakNKUmwtx3s6MozU5KuzJwUTwkWxa2RYzPNP7mdt1OwIvRtpQ8l1cw3DtKGWmNszMaPwaxAYSjO/qu8k6jR9uJdLAlkxssAXcGM2nrS0AwSmcVdOaXnMA4j8NTh7en4rNym1vIOOX0ELaS6GtY4PY3OVMJULDJhqAxRVUmnvQ4CghIxIcXtmt/mq22hntYiUDOwy//p6jtZvlyX3yrtzr+ouTb/4P+Ue/RyVcnNP29P3C3Jp2Ct83zuLIEmbDT0KuhC3ord0yJyfkKwFunthFt+zXA9HTPV5b3oZanqiYcQp7cJbptVFkJ0eT5U5bTzhxV7f5ub8NmbC6UnUTwkAlDh3d8aS0gRKLDG/G8J28EtemI0yo1WPix/qR8gwdV/hXYjpsm3ARcdJXJJ8NMi0pS42sy/QGFIY1u9PM2iaa8/H4K4rZp+lmVMi1hOg7gxlQHFc4tQP8kkuu41rlIRudfASvwhUXlKWsmL0PSScfcXLC/vl0vlZSbcTf5vH5O/EY2rotxWpAccjH9zdXiS8xNDyxKJZNfxbQbdWoBOSDQMxmRL0HKhsVXk69J9OEB59sKSAwj+TBbGN4EnHCPmUXktXewSsL26jwfPYO2643XLZ1PEoNdETPhLmqnQ7uh58IWshQLDkgCeXpOKbu7Hqw8NVdcPLc2MU8YyRtRQDF+zvJz3OU29Sdogbhxm4cOUnCLQ27ofXUN46k9i7abVNSFODZdbqBo6h/36hSFXTcaAMa74rdWwUjXekAFqWOtYlLGkH3APMKX7Q7o/w8xTwXOg3XBmshobcpz9SjeBDvrZ25mSvnEZ5vs0nzCoxnBI9nsSfwxn4LuFKflvfG+KVrQuV18IL9eUzpTQEnquUwrWCAhaIss2dp/YJbYGm0CgLB 2hKtPLea C+V72hSnChPHV8UnD18kGfJm0C/U7MCvIYhqCgIlJlablQAgv5qJk7d4djxy8zZglyRVHwtf+XB8JO8GcyjEXJqOYKUw1hgds/NM+3YCKzLjneMGAp/OWbOqU4zkQn8PTD4/6x3whgrOlPPporSZ7ASKj2wRLgkLqAGZK9hmFieD4L28i2hfxIZre5V70svZlKwOZ1cajCgQgHUdDtd2fu0RZwuTon/f8HfXsH8cBOLg0Tu0YiPW8WQFI0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 4, 2024 at 12:19=E2=80=AFAM Miaohe Lin w= rote: > > On 2024/6/1 5:34, Jiaqi Yan wrote: > > Correctable memory errors are very common on servers with large > > amount of memory, and are corrected by ECC, but with two > > pain points to users: > > 1. Correction usually happens on the fly and adds latency overhead > > 2. Not-fully-proved theory states excessive correctable memory > > errors can develop into uncorrectable memory error. > > Thanks for your patch. Thanks Miaohe, sorry I missed your message (Gmail mistakenly put it in my spam folder). > > > > > Soft offline is kernel's additional solution for memory pages > > having (excessive) corrected memory errors. Impacted page is migrated > > to healthy page if it is in use, then the original page is discarded > > for any future use. > > > > The actual policy on whether (and when) to soft offline should be > > maintained by userspace, especially in case of HugeTLB hugepages. > > Soft-offline dissolves a hugepage, either in-use or free, into > > chunks of 4K pages, reducing HugeTLB pool capacity by 1 hugepage. > > If userspace has not acknowledged such behavior, it may be surprised > > when later mmap hugepages MAP_FAILED due to lack of hugepages. > > For in use hugetlb folio case, migrate_pages() is called. The hugetlb poo= l > capacity won't be modified in that case. So I assume you're referring to = the I don't think so. For in-use hugetlb folio case, after migrate_pages, kernel will dissolve_free_hugetlb_folio the src hugetlb folio. At this point refcount of src hugetlb folio should be zero already, and remove_hugetlb_folio will reduce the hugetlb pool capacity (both nr_hugepages and free_hugepages) accordingly. For the free hugetlb folio case, dissolving also happens. But CE on free pages should be very rare (since no one is accessing except patrol scrubber). One of my test cases in patch 2/3 validates my point: the test case MADV_SOFT_OFFLINE a mapped page and at the point soft offline succeeds, both nr_hugepages and nr_freepages are reduced by 1. > free hugetlb folio case? The Hugetlb pool capacity is reduced in that cas= e. > But if we don't do that, we might encounter uncorrectable memory error la= ter If your concern is more correctable error will develop into more severe uncorrectable, your concern is absolutely valid. There is a tradeoff between reliability vs performance (availability of hugetlb pages), but IMO should be decided by userspace. > which will be more severe? Will it be better to add a way to compensate t= he > capacity? Corner cases: What if finding physically contiguous memory takes too long? What if we can't find any physically contiguous memory to compensate? (then hugetlb pool will still need to be reduced). If we treat "compensate" as an improvement to the overall soft offline process, it is something we can do in future and it is something orthogonal to this control API, right? I think if userspace explicitly tells kernel to soft offline, then they are also well-prepared for the corner cases above. > > > In addition, discarding the entire 1G memory page only because of > > corrected memory errors sounds very costly and kernel better not > > doing under the hood. But today there are at least 2 such cases: > > 1. GHES driver sees both GHES_SEV_CORRECTED and > > CPER_SEC_ERROR_THRESHOLD_EXCEEDED after parsing CPER. > > 2. RAS Correctable Errors Collector counts correctable errors per > > PFN and when the counter for a PFN reaches threshold > > In both cases, userspace has no control of the soft offline performed > > by kernel's memory failure recovery. > > Userspace can figure out the hugetlb folio pfn range by using `page-types= -b huge > -rlN` and then decide whether to soft offline the page according to it. B= ut for > GHES driver, I think it has to be done in the kernel. So add a control in= /sys/ > seems like a good idea. Thanks. > > > > > This patch series give userspace the control of soft-offlining > > HugeTLB pages: kernel only soft offlines hugepage if userspace has > > opt-ed in for that specific hugepage size, and exposed to userspace > > by a new sysfs entry called softoffline_corrected_errors under > > /sys/kernel/mm/hugepages/hugepages-${size}kB directory: > > * When softoffline_corrected_errors=3D0, skip soft offlining for all > > hugepages of size ${size}kB. > > * When softoffline_corrected_errors=3D1, soft offline as before this > > Will it be better to be called as "soft_offline_corrected_errors" or simp= lify "soft_offline_enabled"? "soft_offline_enabled" is less optimal as it can't be extended to support something like "soft offline this PFN if something repeatedly requested soft offline this exact PFN x times". (although I don't think we need it). softoffline_corrected_errors is one char less, but if you insist, soft_offline_corrected_errors also works for me. > > Thanks. > . >