From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 606E6C27C65 for ; Tue, 11 Jun 2024 18:13:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD26A6B00BC; Tue, 11 Jun 2024 14:13:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C81B16B00BD; Tue, 11 Jun 2024 14:13:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B49936B00BE; Tue, 11 Jun 2024 14:13:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 987336B00BC for ; Tue, 11 Jun 2024 14:13:09 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A88AC14EE for ; Tue, 11 Jun 2024 18:13:09 +0000 (UTC) X-FDA: 82219404498.24.75DE810 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by imf12.hostedemail.com (Postfix) with ESMTP id 5213940016 for ; Tue, 11 Jun 2024 18:13:07 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=db7ep5rl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=jiaqiyan@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718129587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YhMpc/bkyf85hQ35VDvGcznVMmv9iMFVbkxrvyl/q1o=; b=dy7DJ7hWpTIUyeAo3oPR+TQgo1gUA1iV05BwzlaHlTOxwU2t2fBTIbrWM+rFfih+KfoZHQ h8kDTgJ4p7LnxvmuNIvkXeqNyH3jp8VCMDIr35scO/584rEqJqABXYrNPth939vrpADNuA JU5kTzUXVJT6L9IA6o8kN0e7mKvqo3U= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=db7ep5rl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=jiaqiyan@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718129587; a=rsa-sha256; cv=none; b=U45n+I5ZK8prkq3rdwS3IG9cdoag9AfdZ6eIbn7MYBKLTfwekP57fH5osT2IbOD564+P5c Eg5CzydCb3bcrFWMK7aUjBhGC28CUbFMbV9AcUQ2nlA4qvllvaYqmGUNyHOi9KiBF7AtWU aw2dNMpfIzuz2EosqT3rAfRyErp7tag= Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-35f1bc2ab37so2596696f8f.1 for ; Tue, 11 Jun 2024 11:13:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718129586; x=1718734386; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=YhMpc/bkyf85hQ35VDvGcznVMmv9iMFVbkxrvyl/q1o=; b=db7ep5rlExD+37wejDwMLXTqjd1wJACTuqfBLUUxo1sVZ8qqQcgKD29DK198sxVn5l aqhe4epHEmZDkFKVr9El7SKgNJm8N9bISFWf5LQYknhbinZDA2rg7MowmeBNBVYd9XC8 LUS2uhhwgi4MdgFZrgSxsllx60PBaud4t/UTxAYRt6s9c1lJb/kZZ4lGBd2fwQ/ZWs5n +f5i0Qu4nHWJLAUZYOpihgU8b+LcR8v0mfrhOF04iZI5JWGmeAn/cG5bxnaWAbg+UnR7 bZ42EWX1AbW4ragfMtWUbaZz6wvuM46DEm4B0tWKC6ahAokLINkGL7gGLtFR00vM7OPL 3Gtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718129586; x=1718734386; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YhMpc/bkyf85hQ35VDvGcznVMmv9iMFVbkxrvyl/q1o=; b=gDZHH+qpbEXIQEfZHpfVtna+bmlbPddNrIEuxmcV/28yi3kyXQ7n/nwYFi5kOUkQn/ AJd+cyt8osJAVVoAYWUWEHFCOeSLQMxzCSYkUIqPZz4QhmxPOOAgufyLIsoCSTyQhEtg 0HSzWVhpJX/3HG0zU0dp+UNuV5MoC1XzH7oVlOVqxvKDtte0EoJqKM0rHxS2zgFzcAQr CboGVAFTa1My60wyKLns6g7VeCUJZs5RdDhjRlfF5tzMbHnrmGW8tSXcLE2EBoYJO4pJ ih6QK86Ns+DKEjwbSAslcOOCyZBrCJYDkVBGuTzxfpkIS0/Vlb0+VQQJ1oTAMB4F3Cvd 7kxA== X-Forwarded-Encrypted: i=1; AJvYcCVI6rW26A40xIGFA0W/uInV743W1CgS8lhzfqn9OWlQwOdkays2uYVeSOZTOkhR8k3QWiw+CWK7SZzufLvgEuHpSV0= X-Gm-Message-State: AOJu0Yxp5yM5sG8UqYEDjSD+rgn7SBRZadicpnYI1SE9az0zLsoJr/sS OynBhuF74hMGGsd4TZPQOWpoDC6epidGRaCAVrRxBn3xnB884fvHS1T6fmo5JZcwMHEBvZWok+m Mzi4oJRS0ogtfvE7tDIp+j0teQ/FJACqI14T7 X-Google-Smtp-Source: AGHT+IFmvmy9iswgqOLOcDOP/y32JIWT7KzVxafuZ88HvWc86Q6Y1lF5Pfz2SbEHPCwQueonamIu49Xyo62slcy4wk8= X-Received: by 2002:a05:6000:e81:b0:35f:1c34:adfc with SMTP id ffacd0b85a97d-35f1c34bddamr5036095f8f.67.1718129585417; Tue, 11 Jun 2024 11:13:05 -0700 (PDT) MIME-Version: 1.0 References: <20240531213439.2958891-1-jiaqiyan@google.com> <21841726-6427-5a92-a3d1-8aac2687c01a@huawei.com> <2738aa0e-99d8-44d7-ac81-e38fd64591b7@oracle.com> In-Reply-To: From: Jiaqi Yan Date: Tue, 11 Jun 2024 11:12:51 -0700 Message-ID: Subject: Re: [PATCH v1 0/3] Userspace controls soft-offline HugeTLB pages To: Jane Chu Cc: Miaohe Lin , naoya.horiguchi@nec.com, akpm@linux-foundation.org, shuah@kernel.org, corbet@lwn.net, osalvador@suse.de, rientjes@google.com, duenwen@google.com, fvdl@google.com, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, muchun.song@linux.dev, Kefeng Wang Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 5213940016 X-Stat-Signature: 1gqrwd3bn7g7ke7ubjsihih6ak94bnoh X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1718129587-268658 X-HE-Meta: U2FsdGVkX1+zBVJET1jXpGmgux+uACb5gRT00OA2wzswDvbH4wtz/+1x45CYE1vp4Ohea9RhMZg7lkJRs32PzWy0sIHm4ydQrbYoaRAkWJgCPukiiIXmv3xh4XBMm/Rdc1RlzI8/aim6WZGhgo1UYdSW4q6n5Ry7I0NtT1hp2X81cyz5ykbiDyJmBFrSF52MprkiV64AiDHYeFIZzzxoQiXKcCaiNVa1ZbNIuSb54y/avtVvPeg6Cshaf3btJ00n8uGgq/klOvqbkMgtvFWVAPL1heVusPC3iptdVUMyvejuE7UjI+yLHAAS4rLFhDwah3DPK6I8mmgyuXArLFDiLm3zIEZmleJS6sRy+Zn6++8Ahb23ruF7ltb8NkrfxxQysoPA7mr7yQ+pUaG05d6h6pBVn07LvkyWscaBTDXdwfWU+Nb4JMUX6a26mAp/WDXZvtiVYPwb1S247BMrOodm6k3zBGEpZV5bWBW4hKZfwMt0NqhdzL58s3afmtTOs5WoOk9UiblYgE+Domlj+ncB4MIMwmtipdxQZKA8Vvz3QpxBewhBN24CPLt9Lnfq7tPjQU57RR0wMXpC/wt5VZdeNetU2J0NJQZJSgvk+Xm8Skr6z+lkLPSONYuBejDUggZhyR8kmWoFIozxJgf7szmfXn7yYE9ZSDoMc286OCRqQD9uBahBr9DAV7jq8fbFtRcbecdMb04gLDT+n1a4S9BnjliNZ98xezVsgPL1faxLd8555YQPvc+q/uQ973AkWH5oivPjA2xbWeezyMogtH0ZKheFj/HWC+B/S3SqxxE81DTrSmOYqHxowboEGMxzBvP+UDTPBo5MbC15pbGHyj3uFrQUX98UtfzesHef6aydeatnmKT2GT3pWirBTn/Q2WkdVZcOh/fxVWY3Ac0OCv2Ot32ulX90wgD1i/M6GFAVrjGV+b5kYNioe3jKcC9ol0ZwH414YWZw6eKPGNf+exC rcKbEYtM 2lFp4fD5uoHg7YipWDyuMGhbAqIBj0JfFOqqe/FR9naO/B3Td2JfLdWlb18/+Eygp8Nnov+tm1VUDsQDlUtd/VE7DPSuHn2gv2Xmh/4Z4W+pW+v1zIZ5qnQ2P6GRak4D5M8svTZN+PPxwKBMsM1ms0kQasHrYAYBewvbA8a8WO/+ex7zaDIiMicrulF5efQBxVSGk0YJ+J2pewx6Z+IbQMvWZRugaidM/GG3duShfls8qQYup7UYyJuMZ3DnMc/iRT+D08SB9dbrbBJQlqAFrSG/Eaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 11, 2024 at 10:55=E2=80=AFAM Jane Chu wro= te: > > On 6/10/2024 3:55 PM, Jiaqi Yan wrote: > > > Thanks for your feedback, Jane! > > > > On Mon, Jun 10, 2024 at 12:41=E2=80=AFPM Jane Chu = wrote: > >> On 6/7/2024 3:22 PM, Jiaqi Yan wrote: > >> > >>> On Tue, Jun 4, 2024 at 12:19=E2=80=AFAM Miaohe Lin wrote: > >>>> On 2024/6/1 5:34, Jiaqi Yan wrote: > >>>>> Correctable memory errors are very common on servers with large > >>>>> amount of memory, and are corrected by ECC, but with two > >>>>> pain points to users: > >>>>> 1. Correction usually happens on the fly and adds latency overhead > >>>>> 2. Not-fully-proved theory states excessive correctable memory > >>>>> errors can develop into uncorrectable memory error. > >>>> Thanks for your patch. > >>> Thanks Miaohe, sorry I missed your message (Gmail mistakenly put it i= n > >>> my spam folder). > >>> > >>>>> Soft offline is kernel's additional solution for memory pages > >>>>> having (excessive) corrected memory errors. Impacted page is migrat= ed > >>>>> to healthy page if it is in use, then the original page is discarde= d > >>>>> for any future use. > >>>>> > >>>>> The actual policy on whether (and when) to soft offline should be > >>>>> maintained by userspace, especially in case of HugeTLB hugepages. > >>>>> Soft-offline dissolves a hugepage, either in-use or free, into > >>>>> chunks of 4K pages, reducing HugeTLB pool capacity by 1 hugepage. > >>>>> If userspace has not acknowledged such behavior, it may be surprise= d > >>>>> when later mmap hugepages MAP_FAILED due to lack of hugepages. > >>>> For in use hugetlb folio case, migrate_pages() is called. The hugetl= b pool > >>>> capacity won't be modified in that case. So I assume you're referrin= g to the > >>> I don't think so. > >>> > >>> For in-use hugetlb folio case, after migrate_pages, kernel will > >>> dissolve_free_hugetlb_folio the src hugetlb folio. At this point > >>> refcount of src hugetlb folio should be zero already, and > >>> remove_hugetlb_folio will reduce the hugetlb pool capacity (both > >>> nr_hugepages and free_hugepages) accordingly. > >>> > >>> For the free hugetlb folio case, dissolving also happens. But CE on > >>> free pages should be very rare (since no one is accessing except > >>> patrol scrubber). > >>> > >>> One of my test cases in patch 2/3 validates my point: the test case > >>> MADV_SOFT_OFFLINE a mapped page and at the point soft offline > >>> succeeds, both nr_hugepages and nr_freepages are reduced by 1. > >>> > >>>> free hugetlb folio case? The Hugetlb pool capacity is reduced in tha= t case. > >>>> But if we don't do that, we might encounter uncorrectable memory err= or later > >>> If your concern is more correctable error will develop into more > >>> severe uncorrectable, your concern is absolutely valid. There is a > >>> tradeoff between reliability vs performance (availability of hugetlb > >>> pages), but IMO should be decided by userspace. > >>> > >>>> which will be more severe? Will it be better to add a way to compens= ate the > >>>> capacity? > >>> Corner cases: What if finding physically contiguous memory takes too > >>> long? What if we can't find any physically contiguous memory to > >>> compensate? (then hugetlb pool will still need to be reduced). > >>> > >>> If we treat "compensate" as an improvement to the overall soft offlin= e > >>> process, it is something we can do in future and it is something > >>> orthogonal to this control API, right? I think if userspace explicitl= y > >>> tells kernel to soft offline, then they are also well-prepared for th= e > >>> corner cases above. > >>> > >>>>> In addition, discarding the entire 1G memory page only because of > >>>>> corrected memory errors sounds very costly and kernel better not > >>>>> doing under the hood. But today there are at least 2 such cases: > >>>>> 1. GHES driver sees both GHES_SEV_CORRECTED and > >>>>> CPER_SEC_ERROR_THRESHOLD_EXCEEDED after parsing CPER. > >>>>> 2. RAS Correctable Errors Collector counts correctable errors per > >>>>> PFN and when the counter for a PFN reaches threshold > >>>>> In both cases, userspace has no control of the soft offline perform= ed > >>>>> by kernel's memory failure recovery. > >>>> Userspace can figure out the hugetlb folio pfn range by using `page-= types -b huge > >>>> -rlN` and then decide whether to soft offline the page according to = it. But for > >>>> GHES driver, I think it has to be done in the kernel. So add a contr= ol in /sys/ > >>>> seems like a good idea. > >>> Thanks. > >>> > >>>>> This patch series give userspace the control of soft-offlining > >>>>> HugeTLB pages: kernel only soft offlines hugepage if userspace has > >>>>> opt-ed in for that specific hugepage size, and exposed to userspace > >>>>> by a new sysfs entry called softoffline_corrected_errors under > >>>>> /sys/kernel/mm/hugepages/hugepages-${size}kB directory: > >>>>> * When softoffline_corrected_errors=3D0, skip soft offlining for al= l > >>>>> hugepages of size ${size}kB. > >>>>> * When softoffline_corrected_errors=3D1, soft offline as before thi= s > >>>> Will it be better to be called as "soft_offline_corrected_errors" or= simplify "soft_offline_enabled"? > >>> "soft_offline_enabled" is less optimal as it can't be extended to > >>> support something like "soft offline this PFN if something repeatedly > >>> requested soft offline this exact PFN x times". (although I don't > >>> think we need it). > >> The "x time" thing is a threshold thing, and if your typical applicati= on > >> needs to have a say about performance(and maintaining physically > >> contiguous memory) over RAS, shouldn't that be baked into the driver > >> rather than hugetlbfs ? > > I mostly agree, only that I want to point out the threshold has > > already been maintained by some firmware. For example, CPER has > > something like the following defined in UEFI Spec Table N.5: Section > > Descriptor: > > > > Bit 3 - Error threshold exceeded: If set, OS may choose to discontin= ue > > use of this resource. > > > > In this case, I think "enable_soft_offline" is a better name for "OS > > choose to discontinue use of this page" (enable_soft_offline=3D1) or no= t > > (enable_soft_offline=3D0). WDYT? > > Yes, as long as enable_soft_offline=3D1 is the default. Out of thought, I For sure, like this patcheset, I will ensure enable_soft_offline keeps the "default on" behavior. > suppose the CE count and threshold can be retrieved by the GHES driver? Unfortunately GHES doesn't have visibility to CE count and threshold value (RAS Correctable Errors Collector does by itself) . GHES driver only knows it from the CPER_SEC_ERROR_THRESHOLD_EXCEEDED bit in the CPER reported by some firmware. > I haven't checked. If so, maybe another way is to implement a per task > CE threshold: add a new field.ce_thresholdto the tsak struct, add a > function to prctl(2) for a user thread to specify a CE threshold, also a > function to retrieve the firmware defined default CE threshold, and let > soft_offline_page() check against the task->ce_threshold to decide > whether to offline the page. If you want to apply the CE threshold to > patrol scrub triggered soft offline, than you could define a > global/system wide CE threshold. That said, this might be an overblown > to what you need, I'm just letting it out there for the sake of brain > storming. Thanks for your great idea! But yeah, it sounds like an overkill for the current problem. I think starting with OS-wide control of whether to soft offline any page already gives much better flexibility to userspace. > > > > >> Also, I am not comfortable with this being hugetlbfs specific. What is > >> the objection to creating a "soft_offline_enabled" switch that is > >> applicable to any user page size? > > I have no objection to making the "soft_offline_enabled" switch to > > apply to anything (hugetlb, transparent hugepage, raw page, etc). The > > only reason my current patch is hugetlb specific is because > > softoffline behavior is very disruptive in the hugetlb 1G page case, > > and I want to start with a limited scope in my first attempt. > > > > If Miaohe, you, and other people are fine with making it applicable to > > any user pages, maybe a better interface for this could be at > > something like /sys/devices/system/memory/enable_soft_offline > > (location-wise close to /sys/devices/system/memory/soft_offline_page)? > > Or, you could use /proc/sys/vm/enable_soft_offline, side by side with > the existing 'memory_failure_early_kill' and 'memory_failure_recovery' > switches. I was actually looking into this better option, but your reply beats me ;) > > You could also make 'enable_soft_offline' a per process option, similar > to 'PR_MCE_KILL_EARLY' in prctl(2).* > * > > thanks, > > -jane > > > > >> thanks, > >> > >> -jane > >> > >>> softoffline_corrected_errors is one char less, but if you insist, > >>> soft_offline_corrected_errors also works for me. > >>> > >>>> Thanks. > >>>> . > >>>>