From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39341C35274 for ; Mon, 18 Dec 2023 07:07:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEFAE6B0074; Mon, 18 Dec 2023 02:07:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9ECE6B0088; Mon, 18 Dec 2023 02:07:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3F8B6B008C; Mon, 18 Dec 2023 02:07:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A39116B0074 for ; Mon, 18 Dec 2023 02:07:57 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 81F5CA02BE for ; Mon, 18 Dec 2023 07:07:57 +0000 (UTC) X-FDA: 81579059394.27.E92DECE Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf23.hostedemail.com (Postfix) with ESMTP id D3608140010 for ; Mon, 18 Dec 2023 07:07:54 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf23.hostedemail.com: domain of hyeongtak.ji@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=hyeongtak.ji@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702883275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=krSjKk+rF7jMIy1s3oY5+XS2mHWBxCM4C7SWuFzEcHQ=; b=YIzl9At8RwEW0jyRC+zsYw8pqh774+LOpldullAd8UGTZZnI6GxE/uksI1mENZBLkRYh77 BEMgnpYCTnHQuw06Aa/BQ/K7tdTw40bvt6otC6HujkizQMVOm81149RHHHxrH/mtFxPERN T3RL9+rEzl2D6tKPO2RIKcmYDHpAIyw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf23.hostedemail.com: domain of hyeongtak.ji@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=hyeongtak.ji@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702883275; a=rsa-sha256; cv=none; b=aFI8zCBx5Z9IQTtj+RPbHobXAslqwrkL+7aFamtf71z0d6Do070aqTLqBIpIVybHBQLygK 8WZZzv7kHgEZ+NTjD7k337t6sGcY0unMJEzd/f5alIkujiilaVh2XFVonv6XlzO37y3g4G ypgc1GaCzNoNeICjAPFM7T6CvkyQ7+g= X-AuditID: a67dfc5b-d85ff70000001748-ef-657fefc8c14b From: Hyeongtak Ji To: gourry.memverge@gmail.com Cc: Hasan.Maruf@amd.com, Jonathan.Cameron@Huawei.com, akpm@linux-foundation.org, arnd@arndb.de, bp@alien8.de, corbet@lwn.net, dan.j.williams@intel.com, dave.hansen@linux.intel.com, emirakhur@micron.com, fvdl@google.com, gregory.price@memverge.com, hannes@cmpxchg.org, haowang3@fb.com, hasanalmaruf@fb.com, hezhongkun.hzk@bytedance.com, honggyu.kim@sk.com, hpa@zytor.com, hyeongtak.ji@sk.com, jgroves@micron.com, john@jagalactic.com, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, luto@kernel.org, mhocko@kernel.org, mhocko@suse.com, mingo@redhat.com, peterz@infradead.org, rakie.kim@sk.com, ravis.opensrc@micron.com, seungjun.ha@samsung.com, sthanneeru@micron.com, tglx@linutronix.de, tj@kernel.org, vtavarespetr@micron.com, x86@kernel.org, ying.huang@intel.com, kernel_team@skhynix.com Subject: RE: [PATCH v3 00/11] mempolicy2, mbind2, and weighted interleave Date: Mon, 18 Dec 2023 16:07:48 +0900 Message-Id: <20231218070750.2123-1-hyeongtak.ji@sk.com> X-Mailer: git-send-email 2.37.3.windows.1 In-Reply-To: <20231213224118.1949-1-gregory.price@memverge.com> References: <20231213224118.1949-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Brightmail-Tracker: H4sIAAAAAAAAA02Sf0yMcRzH973ne8/zdByPq803Zc1NozaRtfnYLP3RH4+JWdYfwnLqoZu6 cilloyOXOhOi6CruqtGS7jxl0oq78+PKUFKtzWSUyVIpivVDepam/157f97v9+efN0upSuUr WK3umKDXaRLUtAIrBhdb1zUPZwobjJW+UGKrpmE6/zkDP+x/aOhznENwraANQb99lq5/b2fg qcVCQ//UFxkYznzCcEfcAeXPPyIwFtlpeJV9hoZ3Xe8QFN5bDu7xCRqctj1QZe2iofZXDg3W 7AoMjU0tGN42lNDQUz0jh9aGu3IYGL9FwYe8MGh3WGTgvuCQwefOBgb6c90yGK75jqBWLKBg 4vYzBO6efAZ+23vl0GK+gsP8+V/GPMwb26dpfnIiH/F/mqtp/sm3YYq/nDXE8A/N7xneIqby Z58OyvnaykC+vPGrjK8zjWFerMqleXE0n+GHXr9m+Obrk3iXd7RiS5yQoE0T9OtDDyjis16c xsmdS9NraispAypbZEIeLOFCiKvjKp7nlxeHGIlpbi0pNJRTJsSyXtxK4nq0yoQULMXVM6TA +YaSPJ7cNlI/2MhIHsz5kzFnqiQruU3ElNPEzFUGkEt1JTKJPbitROw6hyRWcaHENvmEnvMv Iy1FfViqobg1xHZDJckU50ey7hdT0lvCtbGky1r8r9ObOCu78SXEmRfEzf/j5gVxC6KqkEqr S0vUaBNCguIzdNr0oNikRBHNjubWyam99Wi0bbcLcSxSL1ZyAZmCSq5JS8lIdCHCUmovZXjZ rKSM02ScEPRJMfrUBCHFhXxYrF6u3Dh+PE7FHdYcE44IQrKgn7/KWI8VBuRzvCJoe++MotU7 esThLd8cORmY1BY5nff5ZlxM588Oa0zqwY0WvLqqf4nxgKqwdefjU37KgZng0i0DUbFHp1Rf PGP3Tal7MnI8zWsfiDpSdNthH0wOsbkNi5aER0WMdMc2HdKNJp+//NK/Z31vnehcd74A9uPw dt/sCNkjO12txinxmuBASp+i+QvQzCBrMAMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTYRiGe/cdNx19TbEPTaJFmUmZUfgUYQZFb0eCiDCKnPmVQzdlS2tR aB5KB5kHTF1TpobJUmdbkYmJTkuNTmquIDqpleaR1Ix5yhFR/y7u5+K+/zwsIcukvFml+qyg USti5LSElJj9gte1jSYKG75XEGC0VNIwm/OEgfGaORr6Gq8iyM97haC/ZoEKxjoZaDGZaOif +SaCpOQeEu5YD0DZk88I0gpraHh+JZmGd453CMa/DhJw4+5SyOibQtD600lDkyUMzCUOGpqL 2imwTaXTUHLlFgn1j9pJ6Koz0vChcp6Cl3VVFAz+LCfgY2YodDaaRNB6rVEEdsc3Cr501zHQ n9EqgtHqMQQ2ax4BztuPFxY+5DDwq6aXgnZDLhnqj6fSMkmc1jlL42lnDsJzbZU0bh4aJXB2 ygiDHxreM9hkjcepLcMUtlWsxWX1AyJ8Tz9JYqs5g8bWHzkMHnnxgsFtBdPkIZ9jkm2RQowy QdAEhoRLolKeXibjuhefr7ZVEEmo1E2PxCzPbeKfXR9hXExza/gbSWWEHrGsJ+fL2xtW6JGE Jbhahs9r6iBcjge3h68drmdcDsmt4ieb4l2xlAvm9emPmD+V/nzWPaPIxWJuO291XEUulnEh vGW6mf7jL+HbC/tIVw3B+fGWYpkrJrjlfMr9m0QWkhr+swz/LMN/lgkRZuSpVCeoFMqYzeu1 0VE6tfL8+lOxKita+JLySzPZtWiia7cdcSySu0s5/0RBRikStDqVHfEsIfeU7ixdiKSRCt0F QRN7UhMfI2jtyIcl5Uule48K4TLujOKsEC0IcYLm71XEir2TUNW5/D065wnJSY+I46UNy/yK vCZkEe76kfQ4SVbq/lOVZHPP3YzDR0aHNr7VzXtJLSqH24nnA7at4tUXNwRt//jJY3pzsep0 QENhdODAIh9jb5Vvx7ZU5wqTL3WurPygeMtkeOiDHcJ4dV9uV/DK4/vcM7vDdr0OwOaSN4Gl 86fV/nJSG6UIWktotIrfF9TRyCEDAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: D3608140010 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: beh743c3cfgeohgc9dikq34npnp34cdx X-HE-Tag: 1702883274-157769 X-HE-Meta: U2FsdGVkX18rQ5L0oi4M5bpITdeBEa/YEirEbuEOWszKtpVSt6/2WcjFO2ewCoTxNMdS8c57CAFsuIOxYydK2vl2CHy/Ip/KMCWn6Ql0Ua1mdvFI7vofokiaorHrWwYIpUPPEm6lgtfBfiCLIqCFSptxNkPcKEHKlkaaldV0BdSs9OuPEb8AcvX8dWWbeK5KQXpHFBLSyydjgo7vFmFvX87WltirkZD1CrHZ/eGt3WwYZrJoYw8H92uBSVHEOI+/HTfO8l1hzNRCR1IX7SakJyzfanqMprRYvE4WPZEBDyLlme9PfBRGnik1UgDOXrknvcExx7TuqY+AE2QjPOcP/QY0PQIMQnYb9zcntZpog+4XbWafyoXNC+vMVrHh1Tjq97zT0c/RODO0Z4aucCIZrexpIEBTcXIxYyx//4VSM7jd0vtVOgy1zq7OimJ1Nmp5gM5DrS/7pfknMxH6+eWZIsatB4dmdptpCL/2IJQ3Ppdo3x2RTBpkBndfg32wuvrePbqlat5Unf23zmBTlk1oVE2j8nIoVouY+54jjFm1U+3G8ot4Wb8IupRmahveCofIHDxk306eAin/HT5H+himvjgbxekPaCpj5GTO+R3XAQIk6IsmexKp820mlIFa3AawO+PzcGdbtZ+6R9VHotu0hBdX6CJVZp4umOQK7Ijjiw98dsE0EaW+vI/cbrzNGOETM5bjZOV215MeOfVplAQj3542AEyk4HK8PA5fEj/gKUoaf6VtoLLfyRqGz0ObuDBxrVxbv+7l2g+5FmIsbaLTsedST9UF0WPSoK+wIO8I94TUgxXEvU8XmTsIighzzjsdiROH0K4YWKqP0RcidQZbxhsQ+25XpFMwT3p4XQJmR57goIW0e6wUDy4eagJlR3+OJSFWoVTfy3UO9Swc4oFEamOp99sJXUA9apYzICkTI3zbA/XcHnYwuR630E/2DTMpl83MnbAU/A4i9KEdqpf OqEXcXjw h9t1vSDSe1yMlmYcV9vG+NQe+GAc2QraQlHo5kDK3z22kU/Lrb6/aSsnGU5obNIPkELrrW5zic4d9luNWagbYzhcYYUeeY08t6Sw5BUG1froJk5YKkt06+VcQew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Gregory, Thank you for the v3 patch. Gregory Price write: [snip] > ===================================================================== > Performance tests - MLC > From Ravi Jonnalagadda > > Workload: W2 > Data Signature: 2:1 read:write > DRAM only bandwidth (GBps): 298.8 > DRAM + CXL (default interleave) (GBps): 113.04 > DRAM + CXL (weighted interleave)(GBps): 412.5 > Gain over DRAM only: 1.38x > > Workload: W5 > Data Signature: 1:1 read:write > DRAM only bandwidth (GBps): 273.2 > DRAM + CXL (default interleave) (GBps): 117.23 > DRAM + CXL (weighted interleave)(GBps): 382.7 > Gain over DRAM only: 1.4x I've run XSBench based on the v3 patch and got numbers below. I used your sample numactl extension from here: Link: https://github.com/gmprice/numactl/tree/weighted_interleave_master Performance tests – XSBench NUMA node 0: 56 logical cores, 128 GB memory NUMA node 2: 96 GB CXL memory 1. dram only $ numactl -membind 0 ./XSBench -s XL –p 5000000 Threads: 56 Runtime: 36.235 seconds Lookups: 170,000,000 Lookups/s: 4,691,618 2. default interleave $ numactl –-interleave 0,2 ./XSBench –s XL –p 5000000 Threads: 56 Runtime: 55.243 seconds Lookups: 170,000,000 Lookups/s: 3,077,293 3. weighted interleave $ numactl --weighted --interleave 0,2 ./XSBench –s XL –p 5000000 Threads: 56 Runtime: 29.262 seconds Lookups: 170,000,000 Lookups/s: 5,809,513 In terms of runtime, weighted-interleaving shows 1.19x improvement compared to dram only, and 1.47x compared to default interleave. I’ve repeatedly run XSBench and have not observed any significant variations across the runs. Kind regards, Hyeongtak