From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4E22C27C53 for ; Wed, 19 Jun 2024 06:36:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 542B58D0073; Wed, 19 Jun 2024 02:36:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F2028D0066; Wed, 19 Jun 2024 02:36:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BA138D0073; Wed, 19 Jun 2024 02:36:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1D4BD8D0066 for ; Wed, 19 Jun 2024 02:36:09 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 838B6404E2 for ; Wed, 19 Jun 2024 06:36:08 +0000 (UTC) X-FDA: 82246678416.01.5D226FA Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf20.hostedemail.com (Postfix) with ESMTP id B4C9A1C0011 for ; Wed, 19 Jun 2024 06:36:06 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FEHfTqer; spf=pass (imf20.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=jiaqiyan@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718778959; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1hlGoj1WJJZ3FnqOdHCOQOom5O1x4l3tplNmOT+ncro=; b=dHai3jW9JyK/EijMMVjLRSsG81l1k42feJ3x+P2TFoaCQUnPKvd1HdMxHeH6A/7NJm9sW2 XVRr3IOYyer0KdNtMlMsKeJlq3OaHM3Yr5jtzOBpoVI1ZvbhN/IoGij0PM2x4kcIwaRuPP EVzYa6NMtWoTqh2x04FaGUJKQXahCwU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718778959; a=rsa-sha256; cv=none; b=OAJ9ti7dIqbtVBcaflPCDhMn6OYx+lAZAAEkIfmnebPBkChATm/CBUQVA+KFvPXmT8mNrZ 1SRFw4eavkKIac9yzJyYbHSiiJHHKYARnmeRALcTXJAIWQ6PRfbrhRQKWc9tgC0BTqSG5Y MZuN6aSS8O6f7V+fKUsQoQ9BsFsjIhM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FEHfTqer; spf=pass (imf20.hostedemail.com: domain of jiaqiyan@google.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=jiaqiyan@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-354b722fe81so5495646f8f.3 for ; Tue, 18 Jun 2024 23:36:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718778965; x=1719383765; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1hlGoj1WJJZ3FnqOdHCOQOom5O1x4l3tplNmOT+ncro=; b=FEHfTqerQjVuIYAGFuzOc7YtRKjBNEv6moSzJ+qYNiC/MgH0nQnSR2CtkG8BBv3/rg KsReTGNl507BhSdYbpAcZmhotvzGMvSVHTyo95cFwjVMjVejI1RS7y6mKqS8hrWjhafW CSQCmpWriVIFl95o+juMQ7tIJntYbryA+GX0qcUVdNwxyUfhQFJTD9b6PEppdSHj1RL4 +H4aRuVL6Uegwiu24v8hbQP5zaxLF73ksBHqJduUMbswsnYS0rtqtwVizHR6jyFIpUUS t77xgJjj67rrGOAHHFsENVPb5jMlHUpVvOqnHaqDqWh3xNWRNKXPamcySOOotYKwCrai 41gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718778965; x=1719383765; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1hlGoj1WJJZ3FnqOdHCOQOom5O1x4l3tplNmOT+ncro=; b=V/BFu8FmcWLzp4+IJ2XxDqRua/aBhvW7qNjajf1cG2C7r9UabK0IsCvEmMd4zOsqDN zW+GvZal9X+zhThmuATFPCsM2yUeIkPca56Rw0H6bCPQ9PYwq3Vur7Fi9cR50XCqhs5/ ewv3+AAWctaCTldsUHZwmphXoRZcTgMwsFU8nUSQqya7S73sVHlrBRODH/vIITgcv5gQ G4EOIQV5MHNaAiP55OsrSAq/oEbqIkfa0qY9UeXjzJ5hlNWlU5zLbfuCCyf2vbxac+1Z lxgdLy1WAMzwvnKCFgjipQqM+I4MDI1UAK1141aHC7Xd13+3bJ7Wn7bb87r2uTJWJ9Xg rETQ== X-Forwarded-Encrypted: i=1; AJvYcCXTgCvl9lIs8y48vzfQy6PwbV8CJ8Jak2a+cmGiFAABKCRb5/DoUB0grBKO6H4xu4tY/iPJuZ2ABT8htlgxyO894rA= X-Gm-Message-State: AOJu0YxiYfNP3YgX8yxbMmlroZjt6jivzF5IsaTcEqgvpCWpEzXSFEKW CExDK5FM3UlgFI199W/NsqiNau1wH0476DKWkYwYryt4CpTwnTTz4tMQGB/A9pUdRNDo1rNPWcb ukEKoywSHyQh2pXvFs5RKzWugnEYPAdeoAxOt X-Google-Smtp-Source: AGHT+IFmJwrN1XtzxVor8Ksc5pIRoSEYDV4HvFEp9czSMYJ+DnMAA3sRjTwtOF/ttR1vYn960PnYiUWt7/NrxEw/MPE= X-Received: by 2002:a5d:4d4e:0:b0:35f:1e09:ac7a with SMTP id ffacd0b85a97d-363177a3931mr1215725f8f.15.1718778964729; Tue, 18 Jun 2024 23:36:04 -0700 (PDT) MIME-Version: 1.0 References: <20240617170545.3820912-1-jiaqiyan@google.com> <20240617170545.3820912-2-jiaqiyan@google.com> <20240617121348.4ee672a1d6bc9202b3ad0e08@linux-foundation.org> <1a40217a-240c-4efb-5c2a-fe885c0109ea@huawei.com> In-Reply-To: <1a40217a-240c-4efb-5c2a-fe885c0109ea@huawei.com> From: Jiaqi Yan Date: Tue, 18 Jun 2024 23:35:52 -0700 Message-ID: Subject: Re: [PATCH v3 1/3] mm/memory-failure: userspace controls soft-offlining pages To: Miaohe Lin Cc: Andrew Morton , nao.horiguchi@gmail.com, jane.chu@oracle.com, ioworker0@gmail.com, muchun.song@linux.dev, shuah@kernel.org, corbet@lwn.net, osalvador@suse.de, rientjes@google.com, duenwen@google.com, fvdl@google.com, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B4C9A1C0011 X-Stat-Signature: ftiru41fhz6a4wocj9orrzuf17qehcmb X-HE-Tag: 1718778966-252933 X-HE-Meta: U2FsdGVkX1/Spv66JI2rjXLmJgNTRL0vNCWg5tfjFUmTdEr0hrR5tvv8eSrcuS8BKLlDIapwsxTMXAnx1pJyH2uQIQEqxCmId6eMynvYo+lAdz5C3ktKIUNfSHZqBQitYL2jnhDtgKbmNtUWb/NXx2Wsp/2QBBDXXNGkQLmDeod3ciJ+CPQypqG96kYMTswmyQPPTWvURUqQaH+lUyvbJghGU+Ib4MC+1pXKzAk/uI2X+KEYeS/EOaMeq3MOuY2CQ1eMgplhLMbGElaVCt/8uxgozZral+D+lY1DQ9dhKK/I5pZ1jNLaR+V1grI3VLWYiBaat8eecWfMmfnDalq/Sz6fF4NgDj5MaHM5TOYqWAEFEkNK7RXpq/P2w6OQ2ZWK3ZwYnqVfQRl3CEtVE8bH9x1hPnh0ptZ3JUi2Q0V6EqD49FIDrDmxUbdjkeX73xka0vT9ZzG6+1uh4vzPFhk8tZe+f1rFDLpkQcrkNqgnStHft9h1oE6WxSuZVa6J6sb/uF3JAL+WldpWM+IMZGmKZ7zqG++zkzBB5IEzaVTet423gEHErgxlSCoyh3B9uIdhe+YWPWdqkEZP4b8upMjdsYdLJuYoO5bpBocrmmdFUgRG/KHjlb15uBVxszCfRzxYNjUu7hLET1rGvvGlMlKbvaunTEeyljaNXc9yCCsnaGJacYhziCJb5TM+ljEkT19Bxf+7n1Gys5n4WzdvE6tty6YtU16VVkXbyLIK1SgoZVNxq2SLt2f5nrQ508lgkcDgwZp6yuLbYH52uFP3OQKsIRajv2bzY3G64PjGsiCvWBjYoQTzi3hD6qdkV9bJbidQKV/Pwbyk+R5G+oEgqfJO2JRS9xbHh3FgUgSPl3NKld63De73JAsAQFTI+/vceq98JlfieCDUbzV24nniMVL7uZf6OQ/GyDQl+ZTvjR8rUf239fwRf22MNBGMHKAUb4RLdgfi+u0LdZobw9ih75K Udc0gvwX BkCwLeghYvn5VRNs7zTSE4+D4C3Y0HZfnd6CK1VOL/QI7GU0INQPzS7FG8Sxj/b2hGdgz0mOl1GqThMnE+PHO0hCrYqiPtgrbEQpoC6qFWWf0tnVL1k0TDE/xGgcxqX2va5aXHoNla5yZQDDYLDDXhIPETReVD9OQgSZC1Obdd50ByeiwA2b+MjAkFuvF/iQNyarVFukHIpT9APhz1fxocfqa8Bk3oDFGJ01N9AHDFmEdULZNIbMtgGBQeIxKdXCOW0ryE6sruH9nOjvJ7nRo7SfO8zyxKLg4e1ejDDDWoPmOOt9NYRpXUY1lVh3HKg2r8LqotAeL5SrnR+2v0ovIlz6/+EZa10NBD7dA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jun 17, 2024 at 8:01=E2=80=AFPM Miaohe Lin w= rote: > > On 2024/6/18 7:17, Jiaqi Yan wrote: > > On Mon, Jun 17, 2024 at 12:13=E2=80=AFPM Andrew Morton > > wrote: > >> > >> On Mon, 17 Jun 2024 17:05:43 +0000 Jiaqi Yan wro= te: > >> > >>> Correctable memory errors are very common on servers with large > >>> amount of memory, and are corrected by ECC. Soft offline is kernel's > >>> additional recovery handling for memory pages having (excessive) > >>> corrected memory errors. Impacted page is migrated to a healthy page > >>> if it is in-use; the original page is discarded for any future use. > >>> > >>> The actual policy on whether (and when) to soft offline should be > >>> maintained by userspace, especially in case of an 1G HugeTLB page. > >>> Soft-offline dissolves the HugeTLB page, either in-use or free, into > >>> chunks of 4K pages, reducing HugeTLB pool capacity by 1 hugepage. > >>> If userspace has not acknowledged such behavior, it may be surprised > >>> when later failed to mmap hugepages due to lack of hugepages. > >>> In case of a transparent hugepage, it will be split into 4K pages > >>> as well; userspace will stop enjoying the transparent performance. > >>> > >>> In addition, discarding the entire 1G HugeTLB page only because of > >>> corrected memory errors sounds very costly and kernel better not > >>> doing under the hood. But today there are at least 2 such cases > >>> doing so: > >>> 1. GHES driver sees both GHES_SEV_CORRECTED and > >>> CPER_SEC_ERROR_THRESHOLD_EXCEEDED after parsing CPER. > >>> 2. RAS Correctable Errors Collector counts correctable errors per > >>> PFN and when the counter for a PFN reaches threshold > >>> In both cases, userspace has no control of the soft offline performed > >>> by kernel's memory failure recovery. > >>> > >>> This commit gives userspace the control of softofflining any page: > >>> kernel only soft offlines raw page / transparent hugepage / HugeTLB > >>> hugepage if userspace has agreed to. The interface to userspace is a > >>> new sysctl at /proc/sys/vm/enable_soft_offline. By default its value > >>> is set to 1 to preserve existing behavior in kernel. When set to 0, > >>> soft-offline (e.g. MADV_SOFT_OFFLINE) will fail with EOPNOTSUPP. > >>> > >> > >> Seems reasonable. A very simple patch. > > > > Thanks for taking a look, Andrew! > > > >> > >> Is there sufficient instrumentation in place for userspace to be able > >> to know that these errors are occurring? To be able to generally > >> monitor the machine's health? > > > > For corrected memory errors, in general they are available in kernel > > logs. On X86 Machine Check handling will log unparsed MCs (one needs > > to read mci_status to know what exactly the error is). On ARM, GHES > > logs parsed CPER (already containing error type and error severity). > > The shortcoming is logs are rate limited. So in a burst of corrected > > memory errors the user may not be able to figure out exactly how many > > there were. > > > > For uncorrectable memory errors, num_poisoned_pages is a reliable count= er. > > > >> > >>> @@ -2783,6 +2795,12 @@ int soft_offline_page(unsigned long pfn, int f= lags) > >>> return -EIO; > >>> } > >>> > >>> + if (!sysctl_enable_soft_offline) { > >>> + pr_info("%#lx: OS-wide disabled\n", pfn); > >> > >> This doesn't seem a very good message. There's no indication that it > >> comes from the memory failure code at all. If the sysadmin sees this > >> come out in the kernels logs, he/she will have to grep the kernel > >> sources just to figure out where the message came from. Perhaps we ca= n > >> be more helpful here.. > > > > For sure. I took it for granted that any pr_info will have the "Memory > > failure: " prefix, but now realize there is a `#undef pr_fmt` + > > `#define pr_fmt(fmt) "" fmt` just above unpoison_memory. > > > > I propose to do `#define pr_fmt(fmt) "Soft offline: " fmt` above > > mf_isolate_folio, so that any soft-offline related code generates logs > > with the same following format: > > > > "Soft offline: 0x${pfn}: ${detailed_message}" > > > > If everyone thinks this is reasonable, in v4 I can insert a new commit > > to make the log formats unified. > > This sounds fine to me. And even better, `#define pr_fmt(fmt) "Unpoison: = " fmt` can > also be done just above unpoison_memory. Of course. I just sent out a standalone patch for unpoison_memory to you. > > Thanks. > . >