From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C48FC8302D for ; Mon, 30 Jun 2025 11:48:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D5F86B00AC; Mon, 30 Jun 2025 07:48:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1868B6B00AD; Mon, 30 Jun 2025 07:48:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C3856B00AE; Mon, 30 Jun 2025 07:48:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EB64B6B00AC for ; Mon, 30 Jun 2025 07:48:08 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 68D1EC04F6 for ; Mon, 30 Jun 2025 11:48:08 +0000 (UTC) X-FDA: 83611893456.23.943638F Received: from techbitestudio.com (techbitestudio.com [75.119.147.106]) by imf21.hostedemail.com (Postfix) with ESMTP id 42FAB1C0002 for ; Mon, 30 Jun 2025 11:48:06 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kenip.in header.s=mail header.b=lDH+be8k; spf=pass (imf21.hostedemail.com: domain of siddhartha@kenip.in designates 75.119.147.106 as permitted sender) smtp.mailfrom=siddhartha@kenip.in; dmarc=pass (policy=none) header.from=kenip.in ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751284086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OSWJYTVlCnXdQQsB1ImV3/XiHEQb6cpDYP+MEyTQhY0=; b=l5gymJBHIegoHwj2jhTHYKydJM2cI8HLQGuDLN5XHADuSTsdIbsCC2o8VOE19gd7iScHlx OB7Y6ZwcyP8rldz+bItxpkMDSJQmeO18coIXADHGDynXS4FxUYadml17PmCXXF8qTTsE/L oomkJCjfulIqM4qeLIiAMzceMwQslSI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kenip.in header.s=mail header.b=lDH+be8k; spf=pass (imf21.hostedemail.com: domain of siddhartha@kenip.in designates 75.119.147.106 as permitted sender) smtp.mailfrom=siddhartha@kenip.in; dmarc=pass (policy=none) header.from=kenip.in ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751284086; a=rsa-sha256; cv=none; b=OYCGQ1zqwyeZm2xF6b/9pY1PSj0jFiOlQwTQTgdNFdUXMMJfq/kXTb9ZaF12o89EBByo41 lESt7hfUwF8jgH2r8FlBOR1lBPf61Sx6skGWhbZtE1nVqXY8qI2FcmqEmsOJmuYUOXKYSg SxiJy2sFsSdGqbqdlofXI9jTr/BS1MI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kenip.in; s=mail; h=Content-Transfer-Encoding:Content-Type:Message-ID:References: In-Reply-To:Subject:Cc:To:From:Date:MIME-Version:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=OSWJYTVlCnXdQQsB1ImV3/XiHEQb6cpDYP+MEyTQhY0=; b=lDH+be8kx1p0FQFteLHIqR7HUf +Pww22Huaurj7WuCMk7S/yjPTCsQXRY29jk2KSCL2E80+dHl+IePh9UYPZ7pzQrkhjj9p8HtjHwUs GjplaubOO/lyDGhGKViiOr+FuC+1Q/L0jguj6Yvj9LTX6pOAuF0SJ4bA2jVt/HBk08qc=; Received: from localhost ([127.0.0.1] helo=kenip.in) by techbitestudio.com with esmtpa (Exim 4.93) (envelope-from ) id 1uWCzS-0003ha-3K; Mon, 30 Jun 2025 17:18:02 +0530 MIME-Version: 1.0 Date: Mon, 30 Jun 2025 17:18:02 +0530 From: siddhartha@kenip.in To: Lorenzo Stoakes Cc: Dev Jain , linux-mm@kvack.org, linux-kernel@vger.kernel.org, mgorman@suse.de Subject: =?UTF-8?Q?Re=3A_=5BPATCH=5D_mm=3A_limit_THP_alignment_=E2=80=93_?= =?UTF-8?Q?performance_gain_observed_in_AI_inference_workloads?= In-Reply-To: References: <28338f055b3c9afa8b69ff6f05ea20ed@kenip.in> <4990838b-660d-46a2-b21c-67adcba61ff9@lucifer.local> <19714cae-6b73-43ec-af7a-1455196561d1@arm.com> <3ee2e7fea6f263aa884e3e715632b09f@kenip.in> Message-ID: <8128c0338e5df5476ec9fd6eb3079964@kenip.in> X-Sender: siddhartha@kenip.in X-Priority: 1 (Highest) Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Stat-Signature: bknrzheie7ebndy6nwotnb3qegsx8gz9 X-Rspamd-Queue-Id: 42FAB1C0002 X-Rspamd-Server: rspam11 X-Rspam-User: X-HE-Tag: 1751284086-188549 X-HE-Meta: U2FsdGVkX1+RdaEAKXePjCVHT+ncCVCmE4VZmBNL3Vyjg3jCT7qg6Ir7hdExFMgmIEGawhhtHyaqqR0yHkB1ZD5Vcv0N469SsbwBFOYf2amdTSeGuh27aVkG4f47KYVbrtA3UZrBdO7lRrf6x/KTSlfKC2abHiZXnTpTEpo6GtJY7sSAPH0pix8Zq7XdE6Lr05P5dwAQrWgG8/duKGsjoWG0yVsL7oAl/Ag5xz+rtHgg+T7yX+GI44nlXxYpybhGTfl3awtlFNZ0ajvuI3/kGRKPFCvazzDDQ0NKk08Spc1bDe6C/3zp8I3sjzz/cF97KMAdHCIHMXNgZ1V3R7tPnTC8zUBCdWWtp1fDEEUNQettLC9o5KS490oV590gHlbo3osMuPgADihB72TzpLXg0XyDEVmjkP+PXKTxSN6kWQwmrIpHBRPN23gJSVyMxSsencNp7BMmRYMq9oGneL7BH/1mLJBkL17pDS+4RaZAzDLmOXt5VayCOliS+lWWrISEUTIoH/NDBBggx1aOf3+3hoycyyXzIqoRVD3r6u7XobcRfcHZ0yjsYI+dS0Iz1+wuqBk+lN4KlqOphZRMvuN6oq67yigB9UfNBdUGb7OzEYlzvvcGsrRo/QQY0X5gPhunZNLFGAlgH4xoUxJUn/iJz6w04k7qYc6DoyQ1t8Azkj2oj8c7i+I2qdymN9b5Mae1eG00QwdvaCYQTcK8tTJ+cY02qTH9QJPT9UHiOKcGYmKZ0TY3fhQx1W65FzspEmAz/C0J8GY+jWt/EBPnU731UBpCbjT32pgGxwFDcvj40Z27qkyej+Bfyjcpv1uz07uQQFeimcc7AB8u/XklyQQl1S93GBLv8H68bWZy6JWh0T2K4WzRPL7vLJk9dVBudes+e4tRiGQ/D4fgFSOubBsIibz2Os5K0ue8bZUex6LrQJf1O8S9Nqb6PxfU6egKG2JhXeglUN+MURKPolLhO0X VgHfYYNS 12jTrzS8WvwDY7nPuOnlSz3u6Wu8L3yVzFWMzNKY0eQVkesdPYpis6pLTK4wuvEP+FOCL1KoUJAE+fwVLKQbLx7IEfDtQ8Kp7p5Ivq5dSKo44ttYDOxhgYNgDp5wChTrjf2kaJjxQ1/+GBIseMg/lA2lUz7qK3k3xMs4uWzuuEFZLi36KRC+UdjxoUBknXogTyCGY91jZ1cyGYQmasndKsMM+UXvqtWKax4YVhi5K1xEN3jWkmnnA1/nEakYwwLuSW34zRcieLThHS52xffiRlD5Sdq0Yib9RNDZ687cmyHYvdVnQsqTaNGoKYttTjpC4pT1zXrBWzGRWv7hqnqpDvZkEHKTmuQLzW0aKBD7qOZ6iKcQCAfuu6wbH5bDR/nB3B/bZmvdIkj0c9uQ3uIpsST6bA6ltVAKWekZIBVl8e1TFb8AGmTbaDSxR2s+CBQ9ca71Y5vi9TX0Kk+SfMLXBil2k7UayADji0mF4babjYo2P9A0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2025-06-30 16:24, Lorenzo Stoakes wrote: > +cc Vlastimil, please keep him cc'd on discussions here as the author > of this > fix in the conversation. > > On Mon, Jun 30, 2025 at 10:55:52AM +0530, Dev Jain wrote: >> >> >> For this workload, do you enable mTHPs on your system? My plan is to >> make a >> similar patch for >> >> the mTHP case and I'd be grateful if you can get me some results : ) > > I'd urge caution here. > > The reason there was a big perf improvement is that, for certain > workloads, the > original patch by Rik caused issues with VMA fragmentation. So rather > than > getting adjacent VMAs that might later be khugepage'd, you'd get a > bunch of VMAs > that were auto-aligned and thus fragmented from one another. > > So while you got speed ups on some workloads, you got really bad perf > impact on > some that were subject to this. > > The observed speed up was on a very specific benchmark also. While it's > a great > improvement, it's important to understand the context (see the original > patch > for details [0]). > > I do think it's worth considering changing > thp_get_unmapped_area_vmflags() for > mTHP, as it's currently very limited (just PMD alignment) and it'd > possibly be > sensible to change this to checking against allowed THP alignments, but > I'd not > assume this is going to get some crazy speed up as observed here. > > Note that any such change would probably require some refactoring in > THP first > to make it not quite so awful. > > I also think for Siddharta's usecase mTHP isn't really relevant is it, > as intel > do not support mTHP currently do they? > > Regards, Lorenzo > > [0]: > https://lore.kernel.org/all/20241024151228.101841-2-vbabka@suse.cz/T/#u Hi Lorenzo, Dev, All, Thank you for the thoughtful responses and for engaging with the performance implications of the patch. You're absolutely right that the observed speedup came from a specific class of workloads — in this case, token-length-variable AI inference pipelines based on Hugging Face Transformers and ONNX Runtime. These workloads trigger highly dynamic, anonymous memory allocation patterns, often in bursts aligned with model shard loading and attention map resizing. In such cases, VMA fragmentation due to PMD-aligned, non-PMD-sized mappings led to near-complete loss of THP utilization. Once the alignment restriction was lifted (via Rik’s patch), we observed substantial restoration of THP behavior, which is where the performance gains came from. That said, I completely agree that: Not all workloads benefit from this Some may even regress if the underlying VMAs aren't THP-coalescible for other reasons Still, for high-throughput inference workloads on modern Intel CPUs, this behavior isn’t a corner case. The shift toward multi-model concurrent serving (e.g., LLM-as-a-Service) means this dynamic allocation pattern is becoming common, especially in edge/latency-sensitive deployments. 🧠 On mTHP: Intel Does Support It Regarding mTHP — yes, Intel platforms (especially server-grade Xeon processors from Cascade Lake onward) do support mapped transparent huge pages, including via: tmpfs-backed files madvise(MADV_HUGEPAGE) on file mappings shmem usage with shmem_enabled in the kernel So I’d say mTHP is certainly relevant for workloads where model weights or tensors are pre-cached or memory-mapped — a pattern we’re also seeing as Hugging Face, ONNX, and PyTorch ecosystems move toward zero-copy tensor sharing. Given that, I'd absolutely be interested in testing any mTHP-targeted patch — and I’d be happy to help validate it, especially if it avoids the VMA fragmentation pitfall you rightly pointed out. Thanks again for the detailed feedback, and I’ll try to replicate and share further traces (from my local testbed) since I currently don’t have access to the original Intel Developer Cloud logs. Best regards, Siddhartha Sharma