From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AD87C369AB for ; Tue, 15 Apr 2025 10:16:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7758280076; Tue, 15 Apr 2025 06:16:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D21B2280073; Tue, 15 Apr 2025 06:16:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC15E280076; Tue, 15 Apr 2025 06:16:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9D381280073 for ; Tue, 15 Apr 2025 06:16:41 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4D201BB4EF for ; Tue, 15 Apr 2025 10:16:43 +0000 (UTC) X-FDA: 83335874286.04.53F0C8C Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf22.hostedemail.com (Postfix) with ESMTP id 5BC22C000D for ; Tue, 15 Apr 2025 10:16:41 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bh56MUQv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744712201; a=rsa-sha256; cv=none; b=3+DEFQzQd3ZaOgL5XGhpbWFUdy/JAzFTjVTrt0yG8aiLw94AYdfXjRvwDiT/Ezj5EAr06n 0nGowlvRm+bN5RnbQoOhRUIHU1emJEjAK6LG/Uqq2Kq+H4DAiqR0ChRBnyGsNBz7gCfAYA SokN80pNqcIKOkGNPlW0xfpP33dx8Is= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bh56MUQv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744712201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3wPLWpzg0uJc6DKNe0j57Rzmezfms/1F6deuRd/kaWA=; b=LTHcYiK/sGX28Zo10mfStzlsOygI+Osi8EIO1kY07rkvXIGhQjs5sY0zqQhLHD2ycYLYEv UcoQ4da6e7Nl1l1lQDB3oNIfbGNqPZnMyI1D7LWeSNPgbKByOFJt4Fpw9RDDNrunBy2TQX d1jnJrT7Evqc4gh+TBBk5NoYeE+IE+U= Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-43ed8d32a95so45108415e9.3 for ; Tue, 15 Apr 2025 03:16:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744712200; x=1745317000; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3wPLWpzg0uJc6DKNe0j57Rzmezfms/1F6deuRd/kaWA=; b=bh56MUQvZi80uzEakgDKgA6oVzxxvBGL0rJJBjlDPppCEM8eCbU3nRLymiwBeEGccg d/EhwT2X3OVYv0veSkQnxFGJdbd8vdQHZe9T9BHKf5EN/R0u2PzG1QwgySK83JCeU8QD S8tKWxlMrkEpw2Ed+6y901FQBP1Cq9uklNlupZSn+ewkuZgBE/sHejh1KsjBjKr5QIKn rfNrU1fTCDojpnCpY/tiPNqny0ohudo49EW1ULgw1QalWTcjuUfTeG3gIM71UZ3xklCq KtbKpnfU6Zt6iyk2VGPKiC1O+4VourkBW+U3l608MB3poWcBw9xlzJLavZ1iQz38U1vL IqKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744712200; x=1745317000; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3wPLWpzg0uJc6DKNe0j57Rzmezfms/1F6deuRd/kaWA=; b=U8VPOwSeB59RqJwW5aO6DAsuCdyFKRlmspviupEUk6phI3w+5jhjVrwgiSi1J/U1Dv GBPvMMeAZMlBIqCYsKfDbiXXM6iDox3wj3KmtCMxdLgJh13Iw4La557zdre5d3TYyhsP PGH/wRNYek/JbFj8ETA1Ld43uKkwCMkfLLTgLCqVgeD4l+NIn0jKlrMxboCHhi0NvF+k O/75uX6HAOsjZmNPI4EZv/xCYYsEvoTZiBcc2w3CTPHLf98NaR6LACapnbG1gZ/1OX3X hveGrsKDGxauCrJ7thMF7dxZIkhjH9TUpveu1pdhPJ4k+kesgR9U776zXtvNy2DgFxDI qo+A== X-Forwarded-Encrypted: i=1; AJvYcCXLI7QZ9G9hFRyB1ytaZCJEQG/+nZMJ/GfIICbEEv1ona34UaS4WskWDVhxq3M0rJ0ZqeWWECniUA==@kvack.org X-Gm-Message-State: AOJu0YxhbX7XqQXADBUZepNloXl0PaVUdR4D8l2UfUDai6M398PPs3Ja QXO0jylqhgvUzfLYkVMW1lJAAbkxAR8kBDMuoTUgr8FUhdk1QrHW X-Gm-Gg: ASbGncteyaHw/TpiVAJl1MJ1VzkvdBTg5iFb6ovV0oRuldB6b2aEAA0Ns5Aqlo/lyr5 4F4LOdJOVV6iBcGwzkIoZxhBSioph1OZ/Pj8jXUx4WN4rFMSnmfmtc60v6TAM2OdosSA8hW5m2U yvci9msSKO20lts7/ffcBHEeHNwiXP2DNkeSr1tKnRnA2n7eQddX9AjylooZ9b7YePhsrt2r9jP hoc0n1BeHI1kX8CwX4BvXvGEXbiTGusjSrPKF5S6oxT6aPxO/niwlxqiYFLIV/wLjjC4sOVCUG4 fT5yatZOe1TMipq5Wapbsj727UH1nMYmU6RNat5DpW6imNixUWOJUQ== X-Google-Smtp-Source: AGHT+IFrXaLuRdHFBlBDP1BMt/Hh83oA+N35XyDE64nV8bTXaAqlcbovRdOe56UhpE4z32chMbZfiw== X-Received: by 2002:a05:600c:3501:b0:43c:fe15:41e1 with SMTP id 5b1f17b1804b1-43f3a925a32mr157186225e9.4.1744712199310; Tue, 15 Apr 2025 03:16:39 -0700 (PDT) Received: from f (cst-prg-79-34.cust.vodafone.cz. [46.135.79.34]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43f205ecac8sm207375175e9.4.2025.04.15.03.16.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Apr 2025 03:16:38 -0700 (PDT) Date: Tue, 15 Apr 2025 12:16:26 +0200 From: Mateusz Guzik To: Ankur Arora Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org, torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com Subject: Re: [PATCH v3 4/4] x86/folio_zero_user: multi-page clearing Message-ID: References: <20250414034607.762653-1-ankur.a.arora@oracle.com> <20250414034607.762653-5-ankur.a.arora@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20250414034607.762653-5-ankur.a.arora@oracle.com> X-Rspam-User: X-Rspamd-Queue-Id: 5BC22C000D X-Rspamd-Server: rspam04 X-Stat-Signature: km3fbcsq5gq8t3z4u3uou6eac8x7i3gf X-HE-Tag: 1744712201-962562 X-HE-Meta: U2FsdGVkX18yLE4Xl7QwlL0apNQXXUO/gClCmlOoCPy8of/96mKt3zLfHg2DABrhA0BL6M/nvV4jOxrSwVg/t4Wjs7WOOjdIglcUnSiZ6tDIapP/tOa7JXiwrJE3mrWUEr4RQ8sxFR4aJbp9OoINlm4HrDmBg3pOg+nQLJISqIJfbnb/UzXSC0mM5t15vVCelVP6c/QdF3t5GzSHLxRBbJgCXMXoC77c7O1aMIZjApSLAzqwKH0dIzqXDuJXS4brh7yb6HgoL37a5QymVLD3MpcvVqrgu498UvNSZe0f0Pjqp8ao0u53Y7UGoQ7dHtmatNRsM85zMq0jWe4fR3fwnjgGHUzWHaFeTwa/+lTqXJJPrZEbJZxo98a4OjmYgDJVF00qQvso2Ozw0T4siLJG4e0wI9jUc560mTX2MmO+UDsXQXwg9oOi/DF9zMo+HHBhuvs7+rcsTfVYXgpD07ek/sLDKl5tvwkz5egRrGx7FW/7qy2gJUzOLWVqVkxiKcmFapu629Q/GoYGWZ26hJnM++zUXYqtwYn4gq/HJIhctXu1NahQ3NqeNI8IyNNYAkBgoZSbNf840rq8Kfwts/QS88EC5sRk7OuTFDpoB7Vek87VIlJ8oSlDaD+iNEzkyd5r2wrZnx4f6qDFTqn67rpC6mdhxEvJ+otsGTfBtWpbMA5YanudOjPEgEGV8J2p8A0TZKOB2jA1j5+sncQ1eKsGpSgcaH3anwQSIz3J0fmEr9amMLFpqZExTWWly9vPZZI/G+EgO9WK8zvJx16SFtPqyGbD9L3C2E9oNTwtRK7fQiTyKcK5qSlAG2Nm+ALVsWuwfNTp27tL0wHJhCHyDf4o7t2SdlaOSMTU/Swurcs0PPxv0HUfFJWeG/HYGdaSiK3qrmLcqMr7Asxmtrj44ywhYqPlk1AYJL+EfBLXhwAx+VOYhjpTnmG61xuasC+KX9UwbECACUuln5lDq6uF8sc jQ7wPmAp ebpmXyzECdTDhhhQcCXbjlAK7wHUv1r9buUe2xJEWlwezWM9bQIlLdrtOruy75P5hIKr/aSDKmoQIOE0+fks5flH9FJaFL5P80/43x+SGD4P0d3nYuW+D/7yJd0vqSupmM/zaoNaGpkTeFnc5VCoMruT5fkPK7H6Nw5HyW5QwDkQCmxd4PH+cFhlMMWGxtCQZ3FeTowx7bQrY9W1r6nuzk0+PGz+naadkG1u8hm1OslGKGXaSS+2hHSgafeiOE4EAlKEDJ2wYzgxdWFKgNJ0O7Psbhmd0lXf/qbLs/w2Wfw3gEjA7NAkRImGOGHxPuT+iFn17jPszCIqcS02dK+Tvvq6EcBhtLe541e8ukWubXz1Orc/6LZ75Kl/PpYqHPk/Q8FZcC2KdLRy7nJyPmCjhZqTUFhiTpLQsvqa2GexvT1U8bTVDLNyZqG9NPMLGdRk8BYOKClkYigPZfJoAspaNMe+iJw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Apr 13, 2025 at 08:46:07PM -0700, Ankur Arora wrote: > clear_pages_rep(), clear_pages_erms() use string instructions to zero > memory. When operating on more than a single page, we can use these > more effectively by explicitly advertising the region-size to the > processor, which can use that as a hint to optimize the clearing > (ex. by eliding cacheline allocation.) > > As a secondary benefit, string instructions are typically microcoded, > and working with larger regions helps amortize the cost of the decode. > > When zeroing the 2MB page, maximize spatial locality by clearing in > three sections: the faulting page and its immediate neighbourhood, the > left and the right regions, with the local neighbourhood cleared last. > > Performance > == > > Use mmap(MAP_HUGETLB) to demand fault a 64GB region on the local > NUMA node. > > Milan (EPYC 7J13, boost=0, preempt=full|lazy): > > mm/folio_zero_user x86/folio_zero_user change > (GB/s +- stddev) (GB/s +- stddev) > > pg-sz=2MB 11.89 +- 0.78% 16.12 +- 0.12% + 35.5% > pg-sz=1GB 16.51 +- 0.54% 42.80 +- 3.48% + 159.2% > > Milan uses a threshold of LLC-size (~32MB) for eliding cacheline > allocation, so we see a dropoff in cacheline-allocations for pg-sz=1GB. > > pg-sz=1GB: > - 9,250,034,512 cycles # 2.418 GHz ( +- 0.43% ) (46.16%) > - 544,878,976 instructions # 0.06 insn per cycle > - 2,331,332,516 L1-dcache-loads # 609.471 M/sec ( +- 0.03% ) (46.16%) > - 1,075,122,960 L1-dcache-load-misses # 46.12% of all L1-dcache accesses ( +- 0.01% ) (46.15%) > > + 3,688,681,006 cycles # 2.420 GHz ( +- 3.48% ) (46.01%) > + 10,979,121 instructions # 0.00 insn per cycle > + 31,829,258 L1-dcache-loads # 20.881 M/sec ( +- 4.92% ) (46.34%) > + 13,677,295 L1-dcache-load-misses # 42.97% of all L1-dcache accesses ( +- 6.15% ) (46.32%) > > That's not the case with pg-sz=2MB, where we also perform better but > the number of cacheline allocations remain the same. > > It's not entirely clear why the performance for pg-sz=2MB improves. We > decode fewer instructions and the hardware prefetcher can do a better > job, but the perf stats for both of those aren't convincing enough to > the extent of ~30%. > > pg-sz=2MB: > - 13,110,306,584 cycles # 2.418 GHz ( +- 0.48% ) (46.13%) > - 607,589,360 instructions # 0.05 insn per cycle > - 2,416,130,434 L1-dcache-loads # 445.682 M/sec ( +- 0.08% ) (46.19%) > - 1,080,187,594 L1-dcache-load-misses # 44.71% of all L1-dcache accesses ( +- 0.01% ) (46.18%) > > + 9,624,624,178 cycles # 2.418 GHz ( +- 0.01% ) (46.13%) > + 277,336,691 instructions # 0.03 insn per cycle > + 2,251,220,599 L1-dcache-loads # 565.624 M/sec ( +- 0.01% ) (46.20%) > + 1,092,386,130 L1-dcache-load-misses # 48.52% of all L1-dcache accesses ( +- 0.02% ) (46.19%) > > Icelakex (Platinum 8358, no_turbo=1, preempt=full|lazy): > > mm/folio_zero_user x86/folio_zero_user change > (GB/s +- stddev) (GB/s +- stddev) > > pg-sz=2MB 7.95 +- 0.30% 10.90 +- 0.26% + 37.10% > pg-sz=1GB 8.01 +- 0.24% 11.26 +- 0.48% + 40.57% > > For both page-sizes, Icelakex, behaves similarly to Milan pg-sz=2MB: we > see a drop in cycles but there's no drop in cacheline allocation. > Back when I was young and handsome and 32-bit x86 was king, people assumed 4K pages needed to be cleared with non-temporal stores to avoid evicting stuff from caches. I had never seen measurements showing this has the intended effect. Some time after this became a thing I did see measurements showing that this in fact *increases* cache misses. I am not saying this was necessarily the case for all x86 uarchs, merely that the sensibly sounding assumption turned bogus at some point (if it was ever legit). This brings me to the multi-stage clearing employed here for locality. While it sounds great on paper, for all I know it does not provide any advantage. It very well may be it is harmful by preventing the CPU from knowing what you are trying to do. I think doing this warrants obtaining stats from some real workloads, but given how time consuming this can be I think it would be tolerable to skip it for now. > Performance for preempt=none|voluntary remains unchanged. > So I was under the impression the benefit would be realized for all kernels. I don't know how preemption support is implemented on Linux. Do you always get an IPI? I was thinking something like this: a per-cpu var akin to preemption count, but indicating the particular code section is fully preemptible Then: preemptible_enter(); clear_pages(); preemptible_exit(); for simpler handling of the var it could prevent migration to other CPUs. then the IPI handler for preemption would check if ->preemptible is set + preemption disablement is zero, in which case it would take you off cpu. If this is a problem, then a better granularity would help (say 8 pages between cond_rescheds?) > Signed-off-by: Ankur Arora > --- > arch/x86/mm/Makefile | 1 + > arch/x86/mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++++ > include/linux/mm.h | 1 + > 3 files changed, 62 insertions(+) > create mode 100644 arch/x86/mm/memory.c > > diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile > index 32035d5be5a0..e61b4d331cdf 100644 > --- a/arch/x86/mm/Makefile > +++ b/arch/x86/mm/Makefile > @@ -55,6 +55,7 @@ obj-$(CONFIG_MMIOTRACE_TEST) += testmmiotrace.o > obj-$(CONFIG_NUMA) += numa.o numa_$(BITS).o > obj-$(CONFIG_AMD_NUMA) += amdtopology.o > obj-$(CONFIG_ACPI_NUMA) += srat.o > +obj-$(CONFIG_PREEMPTION) += memory.o > > obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o > obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o > diff --git a/arch/x86/mm/memory.c b/arch/x86/mm/memory.c > new file mode 100644 > index 000000000000..99851c246fcc > --- /dev/null > +++ b/arch/x86/mm/memory.c > @@ -0,0 +1,60 @@ > +// SPDX-License-Identifier: GPL-2.0-or-later > +#include > +#include > +#include > + > +#ifndef CONFIG_HIGHMEM > +/* > + * folio_zero_user_preemptible(): multi-page clearing variant of folio_zero_user(). > + * > + * Taking inspiration from the common code variant, we split the zeroing in > + * three parts: left of the fault, right of the fault, and up to 5 pages > + * in the immediate neighbourhood of the target page. > + * > + * Cleared in that order to keep cache lines of the target region hot. > + * > + * For gigantic pages, there is no expectation of cache locality so just do a > + * straight zero. > + */ > +void folio_zero_user_preemptible(struct folio *folio, unsigned long addr_hint) > +{ > + unsigned long base_addr = ALIGN_DOWN(addr_hint, folio_size(folio)); > + const long fault_idx = (addr_hint - base_addr) / PAGE_SIZE; > + const struct range pg = DEFINE_RANGE(0, folio_nr_pages(folio) - 1); > + int width = 2; /* pages cleared last on either side */ > + struct range r[3]; > + int i; > + > + if (folio_nr_pages(folio) > MAX_ORDER_NR_PAGES) { > + clear_pages(page_address(folio_page(folio, 0)), folio_nr_pages(folio)); > + goto out; > + } > + > + /* > + * Faulting page and its immediate neighbourhood. Cleared at the end to > + * ensure it sticks around in the cache. > + */ > + r[2] = DEFINE_RANGE(clamp_t(s64, fault_idx - width, pg.start, pg.end), > + clamp_t(s64, fault_idx + width, pg.start, pg.end)); > + > + /* Region to the left of the fault */ > + r[1] = DEFINE_RANGE(pg.start, > + clamp_t(s64, r[2].start-1, pg.start-1, r[2].start)); > + > + /* Region to the right of the fault: always valid for the common fault_idx=0 case. */ > + r[0] = DEFINE_RANGE(clamp_t(s64, r[2].end+1, r[2].end, pg.end+1), > + pg.end); > + > + for (i = 0; i <= 2; i++) { > + int len = range_len(&r[i]); > + > + if (len > 0) > + clear_pages(page_address(folio_page(folio, r[i].start)), len); > + } > + > +out: > + /* Explicitly invoke cond_resched() to handle any live patching necessary. */ > + cond_resched(); > +} > + > +#endif /* CONFIG_HIGHMEM */ > diff --git a/include/linux/mm.h b/include/linux/mm.h > index b7f13f087954..b57512da8173 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -4114,6 +4114,7 @@ enum mf_action_page_type { > }; > > #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) > +void folio_zero_user_preemptible(struct folio *fio, unsigned long addr_hint); > void folio_zero_user(struct folio *folio, unsigned long addr_hint); > int copy_user_large_folio(struct folio *dst, struct folio *src, > unsigned long addr_hint, > -- > 2.31.1 > >