From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CCB3C02181 for ; Fri, 24 Jan 2025 05:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28E056B009F; Fri, 24 Jan 2025 00:58:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 23D8828003F; Fri, 24 Jan 2025 00:58:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12C8F6B00A3; Fri, 24 Jan 2025 00:58:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E85476B009F for ; Fri, 24 Jan 2025 00:58:32 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5A508120C9E for ; Fri, 24 Jan 2025 05:58:32 +0000 (UTC) X-FDA: 83041290864.12.04D1E6D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 14C04120006 for ; Fri, 24 Jan 2025 05:58:29 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eF8xY1e9; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737698310; a=rsa-sha256; cv=none; b=KdUKVtgKyd5hUru3yg0wWbZDIZ3gewLp+DXoz3yG5M53bGd9A8hQuhJzd85G93PdeUUNIL rZ2T+YXO8VB2WAzsGi8wgseWVugoaxfvrpjpCeCm7bguMoRvXK7XS223vtVj8tIzjVoaTs ChmDJPOxrXyjXiykQmWlw+fPxpiMt5g= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eF8xY1e9; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737698310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wxNlNLDu+fIZSQcO3Iicv//qqn4spfDKGdfi3EWd53Q=; b=rpwTqT7iNwnUaJUycvGhbU/ADE7yymxhTjSH4BgHn/tfHdeZRbh1D5QyCNxvACMPTyjLH7 qk7TImrYx8c72aHqI6TDiXezJ6RPnJXK/eeIpWp3OvU0yMX8AGGZEml3Chtu7yGYSYbwGJ pVEld09NhNCIOBRQjApRbpzTt14Onuc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=wxNlNLDu+fIZSQcO3Iicv//qqn4spfDKGdfi3EWd53Q=; b=eF8xY1e9uuwkHz3VVz56Evp0qC SIQoqiI1q46U5oh/ECKxBpzx4P//huxRF1BV1o06Wc0j3od0aGrYIvEEReWR8KH5EN+iwW/I/TpJu rV6Yw5HgAhf9gqwfUake9tuk/VQtym3wSxxUbkC4hCNK4OIp3vJSnXAgASIwWVBYRyWRcicbTRRBR 4AberFy1PKbz992G/bQRyHtWpGO3s5IplPAwsfCc5Ff6+Z/jZcAhKxs4Go9q6cTr8deWOhvR7dL2E YLvz3QNBTL4PN1R8ZwNKk8k6x2QMZ1RDWfSV0iOO7Bi62+aqklZbgiC+09NQsPSEhV6+hovSdwfvo awNWBcAw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tbChl-0000000Di6h-1UWa; Fri, 24 Jan 2025 05:58:09 +0000 Date: Fri, 24 Jan 2025 05:58:09 +0000 From: Matthew Wilcox To: Joshua Hahn Cc: gourry@gourry.net, hyeonggon.yoo@sk.com, ying.huang@linux.alibaba.com, rafael@kernel.org, lenb@kernel.org, gregkh@linuxfoundation.org, akpm@linux-foundation.org, honggyu.kim@sk.com, rakie.kim@sk.com, dan.j.williams@intel.com, Jonathan.Cameron@huawei.com, dave.jiang@intel.com, horen.chuang@linux.dev, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com Subject: Re: [PATCH v3] Weighted interleave auto-tuning Message-ID: References: <20250115185854.1991771-1-joshua.hahnjy@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250115185854.1991771-1-joshua.hahnjy@gmail.com> X-Rspam-User: X-Rspamd-Queue-Id: 14C04120006 X-Rspamd-Server: rspam10 X-Stat-Signature: nrd1m8b5tzmcwuoocyuttc7ieahc3nha X-HE-Tag: 1737698309-551944 X-HE-Meta: U2FsdGVkX19BqABEHk8rDWgIKSD6h6eFF/Y3kQIG3h/4iVQvHu6Ms0ZpVskEny0xUTZyCDRO4CVRlNurTS+9F0u4YW65A/+t6IZWoY0wyhnPkCR93Oum7MfuzV4o7ia8KEg6IlCoOs1XOunUbo+TGJp0vrXJ7vUVA0ykAQAom+bSJg2fKbLOaNjXROW8NBlq/FCfqbtRAvgnDzyesihTNWWwFzvPrmG7tiLzmuBMSSFjp9RCH3ZC7m2kyFp74hePjoSVE3zAkIxh7GQ3aUtikcWHpeDBq+GFy6QH/gCy/cErzknHnan8dOr529FutboqtYxFIKGKuq5Riqb2HO3hhXMUtgzH0kAM1LVU1nwuReIJD/5DFc0VURiADZvyo7VJ64au/F2e5N8XsAUQZHm4IlXT1/NEcnwm9/egqlP+InEALUyzx6VLhjLQV66PFbZjtLR8Dc5WcuRhaCg+jvAsaDposQJXrIkuq1Zrr2G+PtdsAvyhJXorZuUyxSQJkcDuNSAwBqnVEpV0SvfXB2hQ6rWVaGAMdCogEGNODTSWKNJC8K6p3fFHeCIwRhTi9UkpTH+wzncoJwn5627/Tm61LYWsU754RBOkM9i9M33/aN4AL4hKMaz8H4WfMGs3kH5l27qjw+aIhVMm+S9p5DO46Fm4WP/tx1+8rwLhx0m6qk8Wzs4wcbPOmxs45KUqSbtF0S592VLQghdwgBFmf8kEZfi7DWPla3+0J6rrSyB7QCmAQxQG9joLvd3NQSGPalxGc25wcUIHYHMzY+cr26nAXNELZ+S4kZmQCZIVRKubNAeq+8/SZSKsSCzFUBBiR8Mldu+gxEa6titx2J7SQ0Q/D/r7lvpznJ42FcCOzuRQMCKldGyxC68SyCGdUxQkWCHF81lFEm77DMQtctfLrR8hiDfdYonPcdKdukSlMdPpPSjDvKTszkyHoyTFdtNtMhQqLSf+hX7XOFIM0ETtWZo Cu6Szbs9 pwReZymdowDg6n10J+TzC32Rq0AKHZwp6A+LP727imQLbyuWiWjphNe16bJy3GbR9HFUmhr3I13SUmNIzLWOf0kRh8qPcP7aSWPz80O1/78o/983ggyYx43qJp3Zlr0KtZWYZm3THtmtw0C6gHmDZYX348FFaznk1ATY8NXCaYS/1WRdB6HdWU+yJ8qs1K/ehrSWX92Jc1GhLIHlrKpxCW2JBJ2wuafrYW7zb4dwZaiHr8ojAHfjD3k3b4dv2OwM6Io/6d40d5Ez6EKsZoUFl8jX/t65viVPJueDJ6MuOsQF/rI8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 15, 2025 at 10:58:54AM -0800, Joshua Hahn wrote: > On machines with multiple memory nodes, interleaving page allocations > across nodes allows for better utilization of each node's bandwidth. > Previous work by Gregory Price [1] introduced weighted interleave, which > allowed for pages to be allocated across NUMA nodes according to > user-set ratios. I still don't get it. You always want memory to be on the local node or the fabric gets horribly congested and slows you right down. But you're not really talking about NUMA, are you? You're talking about CXL. And CXL is terrible for bandwidth. I just ran the numbers. On a current Intel top-end CPU, we're looking at 8x DDR5-4800 DIMMs, each with a bandwidth of 38.4GB/s for a total of 300GB/s. For each CXL lane, you take a lane of PCIe gen5 away. So that's notionally 32Gbit/s, or 4GB/s per lane. But CXL is crap, and you'll be lucky to get 3 cachelines per 256 byte packet, dropping you down to 3GB/s. You're not going to use all 80 lanes for CXL (presumably these CPUs are going to want to do I/O somehow), so maybe allocate 20 of them to CXL. That's 60GB/s, or a 20% improvement in bandwidth. On top of that, it's slow, with a minimum of 10ns latency penalty just from the CXL encode/decode penalty. Putting page cache in the CXL seems like nonsense to me. I can see it making sense to swap to CXL, or allocating anonymous memory for tasks with low priority on it. But I just can't see the point of putting pagecache on CXL.