From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1318E77170 for ; Wed, 4 Dec 2024 17:01:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E86B56B0085; Wed, 4 Dec 2024 12:01:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E36DF6B0088; Wed, 4 Dec 2024 12:01:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFE756B0089; Wed, 4 Dec 2024 12:01:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AD93A6B0085 for ; Wed, 4 Dec 2024 12:01:44 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 62E1A141107 for ; Wed, 4 Dec 2024 17:01:44 +0000 (UTC) X-FDA: 82857892446.09.8AA709B Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by imf09.hostedemail.com (Postfix) with ESMTP id 1393B14002E for ; Wed, 4 Dec 2024 17:01:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0Tqss5AZ; spf=pass (imf09.hostedemail.com: domain of fvdl@google.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=fvdl@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733331691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZrlOkFUQYhIWJOetiFuAK2vLC+JxOCjuRnQUbkG0IO4=; b=DblZraD1LxAM9rkS1NEwyz0CdfLzqondO7rCnoX4OGbdCgDPKBXDJKy6CuKvrLn/+Ts/qO 7/zft/TkmbVGcSiGnDUbyh3T5otIZtndI6ZHieQPgX/vJkySGpyVr4zxS87zSq69ZY51fU rzOEEX75nQC36CTO7oFLl7Chy2U5pXs= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0Tqss5AZ; spf=pass (imf09.hostedemail.com: domain of fvdl@google.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=fvdl@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733331691; a=rsa-sha256; cv=none; b=Bu4XoF2y+SrhaHeiMpMtVpkuxIFZ53SsDXuVPuEavGMGJK6f/qBPyqBBbMMTM9RcSokIGy OfXAqMpAl4fXN5OWJuAYFsR6s1FNZBPWjQ3eRMajn81f4TK6Myb2VOOR4ViwqAQzFdoYtc ZwR9tj+7S3/y6bsDRNO5SBGMGV84H7Y= Received: by mail-lf1-f44.google.com with SMTP id 2adb3069b0e04-53e2091c383so2854e87.0 for ; Wed, 04 Dec 2024 09:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733331699; x=1733936499; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ZrlOkFUQYhIWJOetiFuAK2vLC+JxOCjuRnQUbkG0IO4=; b=0Tqss5AZTwdVx1nu8FaNGME8gpbwTmmY+BsXfuEHSfKRlgxca0jUzH/92wwzwJ8amN O5o4ygqGg1XmlHLzK7Az40aWdWw72i3j2tXZl6YEN4KPVTPNTZoRelF19As0DBrNQXam p5S0UlMLpTgXZZuri+g4rEW4p3s1KsF2xyl+D0fk8qinDw7HjXc7Nw8FgDLcPHq6/W8x s+S462u1Id2VcBN2tnikYBgcBYayqGHpsmSCe0hezcrfjWpATOuLQgumPuDhx3vaZV80 wKQnjIRAnHhmYr2lNfZOhGzly5AGINW1n1c2kpqosCulGiDFgGga88+zn3qhAUMCg8lc B7sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733331699; x=1733936499; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZrlOkFUQYhIWJOetiFuAK2vLC+JxOCjuRnQUbkG0IO4=; b=thEvmNKRlaMJpOkFJXICEHXb4hQ0D6b0R/eqcNmsQPtNafNXGShBNLSMT/M45eXtYs gXFvT42bR2c4Sfp7WHZ+6E5eEyBdcOtpa6MHL31Ty1oom3TT+Ubf4IBMnkj/X1c20aps J+T9dGO/5jhnK+hidPgl0hry9tOHrc4OYjg9DP8QYorcxZk5zXabtQd1NfQJ9fliOPoe 0lbM4N55AwAWy/qxrCA0aniV7XWPUB7IiEfN6z8Yz6xL2wYHw5wr01nKcrcVBHxzSnwk 3gpWOKPsacM/R6L6MfqvrORqLkQHye8dQhva//NUhSVc4+1wh2PFRGw2Ku03ZOCnPVTg I9+Q== X-Forwarded-Encrypted: i=1; AJvYcCXVMnbtd0+DvT5AuIpAhw17pRs00ZJfCScQIx47XooklqnpYwhAmFhweGF6ODTWxcOKGGuEfc2o8Q==@kvack.org X-Gm-Message-State: AOJu0Yx0W3dIfuTroyEVKfUwTFEMEpW9jftda9qVLEOxAQ9QxQMW5lkA kTNqiNfISYtp3n0qwwNb1JNsnIaRIBW0xLpObZrnVm/ibKsbJptUVoMhTBXJU8hrkWro0L9YlrK Reer9eRb79M9nDhh6ptNUNeMuhe7tPjxzi8ce X-Gm-Gg: ASbGncuKsV1jyGCJW3zOoePZ5sGyNtBDHKUhNuQ0umV6/h+zxzaEJnpER4RKCMZoIC4 0sA8ciRa0mF3sj8LtHCpJ2oPwBjOV X-Google-Smtp-Source: AGHT+IFBqtc+TZMdEO3TjpxzkaycKSYs3PtWeWBfzgvpNYIDrcYDBXM9ysAGGTHByxdmCC7gKVzE61goA7/NjFDYNyM= X-Received: by 2002:ac2:4bd1:0:b0:53d:e75c:9f9b with SMTP id 2adb3069b0e04-53e1cab0f00mr201292e87.1.1733331697396; Wed, 04 Dec 2024 09:01:37 -0800 (PST) MIME-Version: 1.0 References: <20241202202058.3249628-1-fvdl@google.com> <3tqmyo3qqaykszxmrmkaa3fo5hndc4ok6xrxozjvlmq5qjv4cs@2geqqedyfzcf> <87mshccnxr.fsf@oracle.com> In-Reply-To: <87mshccnxr.fsf@oracle.com> From: Frank van der Linden Date: Wed, 4 Dec 2024 09:01:25 -0800 Message-ID: Subject: Re: [PATCH] mm/hugetlb: optionally pre-zero hugetlb pages To: Ankur Arora Cc: Mateusz Guzik , linux-mm@kvack.org, akpm@linux-foundation.org, Muchun Song , Miaohe Lin , Oscar Salvador , David Hildenbrand , Peter Xu , linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam05 X-Stat-Signature: sfnc7u6jmu3juwg8xu5xt1bg1b7ga3zc X-Rspamd-Queue-Id: 1393B14002E X-Rspam-User: X-HE-Tag: 1733331690-508573 X-HE-Meta: U2FsdGVkX19H84Tkw9odVs5U+YEI3L3B7bu0s8yIA/g0ev8Rm5/lI2M/ilNYPVdwDNvXdrLsd6TxJ2fzhx1CABk0a6Mhztri0HVSoTFFfm5J4k4cjjILayAsJXhaxYBniHxmViUpGXeP5E0VNrKec4tAT4PFFi1L9cQiTkWuUZ2t4AxYxSZm3JPKFp51XxPWp6GrF5OgL/a1zpMYXJe/gcL7ZW6IbhMyuxPAc7eW0wnizGFPbBKp1Ex8Ph+eiThKi5uPKcdp2OsNh88NMGYY/RjGPKJDhzzbacl1jV/05tKbwZfxjXv4nhv0TNIyNa/pOEqfYliShRdJw+oen5gkOa5DVEaWD9MDtTkYleSkPxxIbVYR7baHqF24UOj4r4xJ1vFdaC7FSC519PNyaV1P2pG6sEhyBu9IH4L/iUpVEWdnFBHZRGm0y6Fj2LOA+QO0Bi9cjWLroGXOrLRbJZ95ISRtaCGFbgBbkjOKwC2CHBbRZFKyenRbqhjfNqs8DLE+DPnt2Pj1Ek3NHsBPf75OVdyHTK4KM7NuXPe6m8gHkfe+3tU0krfCgI6fsfV8hFWr4C25r5En45l0N9OZd9GfDjcpYnCBjZBVOwn8i7FjH0xCPIj3SLq4KBP3WS3Ay6V9uAmm6CD8Ol0rLgNYtv+WDUA5ZHFJWkWIIXS5koqGA8Gx9uZAwVWVMhKKZ08Mycx0wwqOucaSChzqKb2lseO53b2fvzmhsTQxHPTTQ8vIWhZ65nNloOuMNZsI+nZd3Hct4EFhMeca1xjNXCRhsYny3wWS3gKnb+QS2Mqa0K7msttS/hKp5bN6r3x7OjQKPvIxzX7sDFz3znXYrl/TA+QpCt3N+JB+tHUyEUB9LqudUtRViI77Wjt/ICAA+ZffCKL0EHsv9LxswBrfj8fNl/YgnMxloRlv6SJ+zHmQjdyiyO1f7Hi9FXFKTKrspItWg7Yhk7TftUXaEDFIpIQ9D49 Snqxqc7G gIW2OGhU3qYm06mnWp9UrKz9QXvLC/79noM9RSJWjCykjmLuI7iUpO+CMycmYhXeo5yl39fwi8gDH1aic7yjzc+crtwoYPy6QX8m1j7iyylRrM1KkBMHOJPe9Y3iDzQZlLZUKI9BFn7rEG7KLrEHsTUgu3zjz/0KTDunGh+ZY3+u//vZREz7+BDFdmFFy0RI5z5J8cJb8tIuO+IKQliOSywtkKBY8GsaR/SYtWpnsEg9EkvOJQK554xBc5atgBM2WvWU53CFkkodVc84B675JdEDPSK02ZMlheaiw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Dec 3, 2024 at 4:05=E2=80=AFPM Ankur Arora wrote: > > > Mateusz Guzik writes: > > > On Mon, Dec 02, 2024 at 08:20:58PM +0000, Frank van der Linden wrote: > >> Fresh hugetlb pages are zeroed out when they are faulted in, > >> just like with all other page types. This can take up a good > >> amount of time for larger page sizes (e.g. around 40 > >> milliseconds for a 1G page on a recent AMD-based system). > >> > >> This normally isn't a problem, since hugetlb pages are typically > >> mapped by the application for a long time, and the initial > >> delay when touching them isn't much of an issue. > >> > >> However, there are some use cases where a large number of hugetlb > >> pages are touched when an application (such as a VM backed by these > >> pages) starts. For 256 1G pages and 40ms per page, this would take > >> 10 seconds, a noticeable delay. > > > > The current huge page zeroing code is not that great to begin with. > > Yeah definitely suboptimal. The current huge page zeroing code is > both slow and it trashes the cache while zeroing. > > > There was a patchset posted some time ago to remedy at least some of it= : > > https://lore.kernel.org/all/20230830184958.2333078-1-ankur.a.arora@orac= le.com/ > > > > but it apparently fell through the cracks. > > As Joao mentioned that got side tracked due to the preempt-lazy stuff. > Now that lazy is in, I plan to follow up on the zeroing work. > > > Any games with "background zeroing" are notoriously crappy and I would > > argue one should exhaust other avenues before going there -- at the end > > of the day the cost of zeroing will have to get paid. > > Yeah and the background zeroing has dual cost: the cost in CPU time plus > the indirect cost to other processes due to the trashing of L3 etc. I'm not sure what you mean here - any caching side effects of zeroing happen regardless of who does it, right? It doesn't matter if it's a kthread or the calling thread. If you're concerned about the caching side effects in general, using non-temporal instructions helps (e.g. movnti on x86). See the link I mentioned for a patch that was sent years ago ( https://lore.kernel.org/all/20180725023728.44630-1-cannonmatthews@google.co= m/ ). Using movnti on x86 definitely helps performance (up to 50% in my experiments). Which is great, but it still leaves considerable delay for the use case I mentioned. - Frank