From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7ADEC54E64 for ; Thu, 28 Mar 2024 05:49:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7988F6B009A; Thu, 28 Mar 2024 01:49:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 747876B009B; Thu, 28 Mar 2024 01:49:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EA406B009C; Thu, 28 Mar 2024 01:49:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3CD566B009A for ; Thu, 28 Mar 2024 01:49:56 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 028A0A0F48 for ; Thu, 28 Mar 2024 05:49:55 +0000 (UTC) X-FDA: 81945371592.10.2EEF7C5 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2069.outbound.protection.outlook.com [40.107.223.69]) by imf08.hostedemail.com (Postfix) with ESMTP id 0BCF9160004 for ; Thu, 28 Mar 2024 05:49:52 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=GsdEcJrL; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf08.hostedemail.com: domain of bharata@amd.com designates 40.107.223.69 as permitted sender) smtp.mailfrom=bharata@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711604993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JpfeUPr0I/YaETe9n5IlM0+Ca4a0g9N0BHTw8A9MreM=; b=gH/uk5DMGlC55mLCGVEBV/bzN+dt5ODQcpmTckIwbck7Rk6Tkv4REjwtont33S12JKmRrn e1UfUIuDx2ilevuXFrLeZXjnTPaTiur02eHNR0q4ik1YbjlyqMTKtOXNDMS6zzFD5elrOc T/K3NyP7msoHTnnUXPphzdAqok9g3HQ= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=GsdEcJrL; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf08.hostedemail.com: domain of bharata@amd.com designates 40.107.223.69 as permitted sender) smtp.mailfrom=bharata@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1711604993; a=rsa-sha256; cv=pass; b=KFIE/LMwzmj0Ode9yitmTIlbxR9zYcoLMnaxOPnvXhFY0SfOr0iilpTI9ki83S6Ql2IsSJ pmyC1PYpzQzKMQFubk5woPSn68w4McKkv7M9eofsxgVdDUtBWUxPJ3oce4KELEfQL477pW J/zkKjW5idUR9j2KnODOdE0YW9cW/gQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z6ja0cUROkWw+YfBc1GtL8aI5jeMgfzYcCOQ7SbqWK8bawXv7BBNLSr5e2D0PAg9MTwxtLCY4dnIkoqqjI1Prp+tSv/oJGJjyxVrMasuw14W+x3/XSSwQTXyUcSZBTi3xXL44PCX+StKiqrM4lNZsgCf0WXSNg7FmZRvvPrQX0n662BYfTgsHAVPfFQJzIcTw2F/4rLfhA3p+8iz6UUEMGKdbo2H/fXNfCAmZ0b+0vfwIEKjFmnzV5EAZV007pzinnyIZoVXMGjh1xT5XteUE1/eHrAYLlYkwnU5hiJsIRagygwh9zDEHw9Qvv/aIUkg+yDvW528DPO3L2862ft58g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JpfeUPr0I/YaETe9n5IlM0+Ca4a0g9N0BHTw8A9MreM=; b=Szt8E6IZbdLHFa/6fUDKDZMRIG6ttuz35MyOoI1v4cVcYL2EqoBIdmZirWU+Aw2KqgsnwTvZiZriXc2W+4vku8YXRlTKiIpKRUtvDhy62EXf4rTNQfNTF08jpv2ySgsCI0RG4e8Efs1IXkc/0wlS+/JQH4a8tvMlB5YEtOAZFHmd3U1jzw0cvMDiIYom9rX+3+dfI4rwTzIPihTFITQYzeZxz9RVX4sSyuTe2sJXNCLR4zyuZ6X8CwdBKeQhA5ibiDRtYlP2z0n8t1emZIcRxNOkxcqnfwPwcPpJaXTQwccgvaWrPfSrHby7vU7SPfzATQ6PImRzPQ/mUsmUXeMIKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JpfeUPr0I/YaETe9n5IlM0+Ca4a0g9N0BHTw8A9MreM=; b=GsdEcJrLOA7DFBEAqUabmMp94Esml013pg1tIEvVFvVHe4KuJWy8XZ4iqydEUxtAdUbWS6nODi60VqIO7HMoP3J8lbwvs5HdJbrbNkAKVzILeHeg3zJuvmHbyROlDXA0fRsLAIFl8Nzl3R/dujvm+WwXRaGUoiqL5dC2G8MZTtg= Received: from IA1PR12MB6434.namprd12.prod.outlook.com (2603:10b6:208:3ae::10) by SJ2PR12MB8943.namprd12.prod.outlook.com (2603:10b6:a03:547::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.33; Thu, 28 Mar 2024 05:49:50 +0000 Received: from IA1PR12MB6434.namprd12.prod.outlook.com ([fe80::a628:d2dc:a0e9:67e2]) by IA1PR12MB6434.namprd12.prod.outlook.com ([fe80::a628:d2dc:a0e9:67e2%5]) with mapi id 15.20.7409.031; Thu, 28 Mar 2024 05:49:50 +0000 Message-ID: Date: Thu, 28 Mar 2024 11:19:40 +0530 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address space Content-Language: en-US To: "Huang, Ying" Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, mgorman@techsingularity.net, raghavendra.kt@amd.com, dave.hansen@linux.intel.com, hannes@cmpxchg.org References: <20240327160237.2355-1-bharata@amd.com> <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Bharata B Rao In-Reply-To: <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN3PEPF0000017F.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c04::4f) To IA1PR12MB6434.namprd12.prod.outlook.com (2603:10b6:208:3ae::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA1PR12MB6434:EE_|SJ2PR12MB8943:EE_ X-MS-Office365-Filtering-Correlation-Id: b0063099-d9ae-409f-4ccf-08dc4eeae201 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fy/EQ5o17EoeMq2lGvJ/nsK7c16DxpM5Y4jRVB2s74ihi9nPx+JLwFlRGHLB5DJ0NDT/zMAvvpmlzZD01gMHtHmMfcwW/l53FqXcJd5tP/IinXcFtsomeu1Y0hYj5Q9lvSuuU1+z11UA8we0y6xKHtpL4gVEjEmxVqLIzUWVXyH3qEsCn9eUaUYxgdzCaVLjSKNcRjMfecshThhQMTiJF4nqj4aeephX/C5ejzieohw01CrQhyispY35oYdzo4XnwR1mL7/Fh3LsxYyLRl0TYgYMMFKOSrC0bxSRrj30YRvsyjGyhdD2vP6zQuJfI0iy0uo5LYFSey1hUH5D46NXFFKLb9Hfw06TvhkfSkjd9NGYOCeSqPbPE57L51FIaeic9/Ua/5ltKZS5MlD5Z3qdeNQ3kvEt0XYuIxlR6VKEiEORzK7516sy4V/ZILRZsZt88hpkxYTUf6E2gP1OQG9bw1E5idOua5G9onP/GUfmXTj1vpqf/ggEIxaGwrsSmJQctrLPI4UCaMMgQ2bMMvr4Hfz04HkNeaOWR1P/IUBaR/GEda+pCIZqizoZvcks/oWLpHf1s5NF5bjIY0WMFcvWgND0a1B0IjlaayO4tDmC9amCRwWMchZVSCI58/ooDkrQgW9HkOEUG3iIja82JrauiQLuW9+ilpWwBo333Hz3ago= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA1PR12MB6434.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(376005)(1800799015);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T0VIWXdVUktobXVBb0JjYUN2R0hmRWJZVEJIRHhLOTRQQlNKTmk4SlpMWG9j?= =?utf-8?B?cU9QaHdHUmNxWHBndkdYNVh0ZVQ2Tk96VzZRVkZ4L2o3V1VxYStUeXBoaEo0?= =?utf-8?B?Nnd3dytQOUx2NkxpSDlLM3ViK0lXRGluREZseU9yVEluaGh0OU00M1h2Z1hU?= =?utf-8?B?RkxJYmtaN0VPbmNSOE1oNXVVZU9oSGRyMngzK1grd014VVpNNysxd21JZjJO?= =?utf-8?B?bGNoYzV6YlVna3BTQ056elBIdmJoV0tkVTlFeFN6enZyblh0UGZIaGV1dEc5?= =?utf-8?B?MWRqejlMeGZsaFZWdDA0aTNicXJHNEo2UkEweGZoR1NreENCL0pZTGkyOGkw?= =?utf-8?B?MG9DVHY3eC9ZVnNwRGc3RUpuajRaU1FFVXJwMlBXY1h1clNXM3RwT095SnNL?= =?utf-8?B?OS9PMmRqUzRRZXp5eVByQ05yeDY4aHZQUldlQUVNajZWZmJzVklZSjdsYnVH?= =?utf-8?B?Wkp6SWJUdVY5Q05oZUZ0SkxnblZmOXkzOTArcjVtL1J6SDdyUzNBWkRhQ094?= =?utf-8?B?alF4ZjFWZWZqdDBKSWZHbEEzUURFZkNWUWhXZVN3dTRhVUhDVWxnN0xteGRv?= =?utf-8?B?RFU0emZvM3BVblU5REloWlRTc1NHZFN4R2txNUJ3aUZLd1BLUENsekVZN3ZZ?= =?utf-8?B?WGxQZEpKR0xFZnFRWnh2YlZlaTVIUFlHMFpLcmJRS2h4a0YwaW1VZ093eE84?= =?utf-8?B?cFlrN3lwODRFSzBzUXJrUHlzREUwU3FXREU2emhaTThWZk9rTHB3emVid3Zq?= =?utf-8?B?T1pPcTVTSmVJWUpYcUlncW9GNlA3UGJpQzdTbnlPaUNxTTlNbGpuRC9ORE55?= =?utf-8?B?YThQRC9qSkZHYVpQSElraklhbkUyYkJLZ1hVM3c3aXV6enlja0VQY2RIUity?= =?utf-8?B?RVpzRlJ6aVJRdlRnYUhPRE1ONFdiZWhHanJRSFVzMVFSaHlpUnJEVExENVJy?= =?utf-8?B?Wjh1QUdpUFJjZDE0d1VURkpwTm9IN1lGSXhDQ3lwd3Y4T0JsSGlCTEs5VzAr?= =?utf-8?B?czlYcHlTV0JEaEoweW5raGhMQW95Zng1YTA4Ym9MQTR1ZG1reTlhZW0wWTVZ?= =?utf-8?B?ZkJQb04wMHI3L0ZjT2J0NEsrUGN4cVp0eXVNTGprYmswY3ZUN1JEenMyd1Rz?= =?utf-8?B?TDR0OU4rUTJDb1JVanhXUDdxWjA0U0QzZE5lQk5YYTdnU0MvSFlrNDduald5?= =?utf-8?B?YUVzbFdKd3N4QmZOQUZIT0ZqTmVWTkVLNkk1L2MvN2lZbWtaRXVuYnBNSmlx?= =?utf-8?B?ZGNwcGsvYTVQWmZLeUNBdlNCTC9UaXFjamFFelpEMXpkdU9NcGNEL29JcUt6?= =?utf-8?B?UE42N284UVlLQzAyeGZ2SWhvamZRUEp3WGtwVmtpUDlmZy9pTTFaa0t4RjhW?= =?utf-8?B?WCt3L0dyWVEvTFJyM2hnVmZuREljbnUvR05KOFB6cERlWkcrQ0pjMnBYVTRS?= =?utf-8?B?WWgvdmYyeTNVTDVDNVlTT1M0aTQ0dHlUejY3bnFEbThzeUxrUlFVSERxL29F?= =?utf-8?B?MDFmVzEyL0lxUHVXekNIQUdQbXVIOSs5WFF3MUVvajVRYXl2N0NkREZvZE9I?= =?utf-8?B?ZlRwc3pGT3htdmV1OUxEc1hFUXZSWndHQis4NFdDRGdqQThVM2JsTkhxUnRH?= =?utf-8?B?c2RScXkyS00rRXBzUkdJWDcyRVYrVTBlMnFCdXhQUWZ5S3M2azNEK0xzZTUv?= =?utf-8?B?RFY3ZkZmWTZJbENjWUdoWWhFNmdzaHBONWprQlpVZWhValZYZ2Nnd1dDeWxF?= =?utf-8?B?TnZFSkV2VEhQLzZ5aEtnTXpPL0JzL0d5YjdMK3VPaURuK2FKTU5pRXdiWjcy?= =?utf-8?B?a05BZEViRXh4R09oMHpxdWljUzQyQktkYkN6YlFxOFpKOTR4UXZxL05EclZq?= =?utf-8?B?RGdGWlJmdHpxcmpxN0REaXduTDdRVFZqem1hTWJtdHhucGlxakpLTHlKMG90?= =?utf-8?B?WHpWN0FWaHJ6U1E4UmtsUk03L0NOUGFuSlRQZUE3bFAzM2N2TGtuVTlxb0VH?= =?utf-8?B?Q1VZSGZpTnRHZkVnaS84cG13TGlURGNXYytGSG9wb2lIRVQ4bW9Da2dlVDBj?= =?utf-8?B?R2tGUzlQb2NSZUJuZHU3SFByd2p6MVVsL0Q1d2lmckYvMTQzYzJhVWNVcytX?= =?utf-8?Q?9IwBEv8lGDu/Fj9imHqtWd8KX?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b0063099-d9ae-409f-4ccf-08dc4eeae201 X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2024 05:49:50.3761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YNvpWwdZpEX92/cAaeuqQYDNo2NnbM8+FDwYhMfZhJHzjTNMVCWK6V+JLQWl54ruVBkE/bOjbOy7mrHVrlk7zg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8943 X-Rspamd-Queue-Id: 0BCF9160004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: uzje6eaina5beyfx6m747nnd8qd99c58 X-HE-Tag: 1711604992-1006 X-HE-Meta: U2FsdGVkX1+rwe4Q828ATmWL6YT3Z1NjPQ852zxbNRFGyLpdtqNuuRDSPLWqt4kx+fX5L+7QKEk+jwLEVUMy1/gZO6vVgP7ertfx6jPaaq92AT8/mAIFP63iMOcI3BbX7UZ7xBl5+b7uTDPlJV2PVhB4G1RiRKoqUetQmRnlmF583FKydbqfFXuMXpRiDGqVRqVpIadMWgi2soCQEiO06vmYPV5dczie0TRsE+/KMNVXahgxsGOqGcLpT9BdE0pmYhcpHW3AIDBh1+DWNR2RX/PUCMT5pPIWcSz3B/RQWGgzR/P15zwVHXpLuQq+5blyZpd9QY0o1+mpFMH5dkXcC5MwLsnHEOF8JV0gScbi0gIkRsfJgQs609TuapJdaS9N1rhwmDmHnPojSE6FHyflAxYrH90pKOufSy1wZBD4RxBQPMsJndFjfoEWtqFexqrskOZZdb0I8kefPhGWz/+wmd1XhyhHG/swOxL3KVtGUfRlQ5852jBGYzQZFl78nWxiDu3JYBl7vnUQqYV5IpRbp0JsIgtGdWyq0URK5k4ImJkalg7EA4fKx5xGuWjkG1lvIcuJk6qDMKsRT6niHvpP6ShaNVLh+WQ7qFV0TFT3i9Mh2ZAtcrYEnH+Lv9bLnykdhLvBnNg0se3eI62d+8F/uEcX/7OGvVB0xOxnfBK2WZaq72iX/GwBXVA0JF8jlQjI9QDyozLrMo53orZ17DCuBaoFxokx+RsXRLAd+OP1VFSacTBUa07+h3NTUNpfIRf5jxMrPzLo2x+DfXh9eJiBtAo3LkvZQS0tKt2NRDSFxVwUcQhMrhAOpVzr3Doxk7hvtKKiLPPWbPDyMyO3c0fKsWrG+8b6hUGaTXoB8+59t2NMcMB775uSV6C7Ul5Q78MedVHtlnvUhHRmS4HKhoS3VDlqQIygD8OzisYPF71FC3EctlSkdcN3DN71FTZH7AUOFqcWrqOxpbY7Folovg0 uXeJNjOQ wINT/NfuNjwsaO9UYukOKmeDVqqDICsDajL1wneiSmNgJZfDV/jrWeTn0F1TeTUqsnorAih9kaM/0+Hw4OnCEBlcdBk4TWC96BEryfNfOmgTSdXu+H2eseXI9qc7MiZJ8xrgzHWoEf6hE0Q9yRpTllF8W+GxezcJHnSbMLhyNyPZXxgQbX6lvgfbOcp53zU9v8MZAYN/QHq0LfTR+OjR1G+dGZl7yIaEtUw7ZVwrVE8+0ighJwfArg+1barTULnq5A42qyj6trg1yiGw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 28-Mar-24 11:05 AM, Huang, Ying wrote: > Bharata B Rao writes: > >> In order to check how efficiently the existing NUMA balancing >> based hot page promotion mechanism can detect hot regions and >> promote pages for workloads with large memory footprints, I >> wrote and tested a program that allocates huge amount of >> memory but routinely touches only small parts of it. >> >> This microbenchmark provisions memory both on DRAM node and CXL node. >> It then divides the entire allocated memory into chunks of smaller >> size and randomly choses a chunk for generating memory accesses. >> Each chunk is then accessed for a fixed number of iterations to >> create the notion of hotness. Within each chunk, the individual >> pages at 4K granularity are again accessed in random fashion. >> >> When a chunk is taken up for access in this manner, its pages >> can either be residing on DRAM or CXL. In the latter case, the NUMA >> balancing driven hot page promotion logic is expected to detect and >> promote the hot pages that reside on CXL. >> >> The experiment was conducted on a 2P AMD Bergamo system that has >> CXL as the 3rd node. >> >> $ numactl -H >> available: 3 nodes (0-2) >> node 0 cpus: 0-127,256-383 >> node 0 size: 128054 MB >> node 1 cpus: 128-255,384-511 >> node 1 size: 128880 MB >> node 2 cpus: >> node 2 size: 129024 MB >> node distances: >> node 0 1 2 >> 0: 10 32 60 >> 1: 32 10 50 >> 2: 255 255 10 >> >> It is seen that number of pages that get promoted is really low and >> the reason for it happens to be that the NUMA hint fault latency turns >> out to be much higher than the hot threshold most of the times. Here >> are a few latency and threshold sample values captured from >> should_numa_migrate_memory() routine when the benchmark was run: >> >> latency threshold (in ms) >> 20620 1125 >> 56185 1125 >> 98710 1250 >> 148871 1375 >> 182891 1625 >> 369415 1875 >> 630745 2000 > > The access latency of your workload is 20s to 630s, which appears too > long. Can you try to increase the range of threshold to deal with that? > For example, > > echo 100000 > /sys/kernel/debug/sched/numa_balancing/hot_threshold_ms That of course should help. But I was exploring alternatives where the notion of hotness can be de-linked from the absolute scanning time to the extent possible. For large memory workloads where only parts of memory get accessed at once, the scanning time can lag from the actual access time significantly as the data above shows. Wondering if such cases can be addressed without having to be workload-specific. Regards, Bharata.