From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5580C05027 for ; Wed, 8 Feb 2023 18:13:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 129C86B0074; Wed, 8 Feb 2023 13:13:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DA296B0075; Wed, 8 Feb 2023 13:13:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0A476B0078; Wed, 8 Feb 2023 13:13:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E167E6B0074 for ; Wed, 8 Feb 2023 13:13:07 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A34F51413E8 for ; Wed, 8 Feb 2023 18:13:07 +0000 (UTC) X-FDA: 80444921214.11.96D9CD6 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf27.hostedemail.com (Postfix) with ESMTP id 616C240009 for ; Wed, 8 Feb 2023 18:13:04 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z6dbaJ0v; spf=pass (imf27.hostedemail.com: domain of dave.hansen@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675879985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kK4FDS3dJUMHyJSpdY8XPFwYyJWAdklmSNrHVszbAP8=; b=yRU4mJFvdCGhGCXGrbvr6tdNsrthzEOPx4pRgQ/oVoLVwdzTjBqzuO8uZ5jHj0RtUjoTdS IRUubzWJ9hcPNVFPcBFrfKMAF6yr3jErqsfiw4Q6kv42Cbxuc/AO/eH1SkH/4JFZ4WOdal Cnj4V5cU3nF9RfUsLrFhvIAbYO+8oq8= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z6dbaJ0v; spf=pass (imf27.hostedemail.com: domain of dave.hansen@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675879985; a=rsa-sha256; cv=none; b=idHWvlZniloskzbO7c1kDMrFILyWernromhPPQn94JhDiwTIYBUAhB1n7uvT/iguW1tDiT muclQE1ut08WSiAF6iJjITguFKaN0jEneKt/movLhX47HqOpY8e9144pZnFlSbARutcyMZ UXCkTxzPfCX8qfUoIpZHqn5o159YpOk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675879984; x=1707415984; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=jbli7xfR68iVXzQ0pgj5/0O25sBe9dbuYE36QYk6OHk=; b=Z6dbaJ0v04hmLDKghBGDiuAcrCgUrX1DGDFsHkA4MFID5o8168b7/WpN wAnQ/cmQc4CqyoowjWcG/sqDa38fiNUGVwUbIjTygFkxSP4Qgg/mLgFE6 VdIutXtt+iypFzD/ayoOf3uHxH5b0uV2Memm1SegzbZhJo0hsKbE/ezrb v7kb+QdLDFNjdTd0WD9k4jm+dv5Z1Gxhx14ECXwB7DvkUYJpqZ+Q7JSHc BAEm3WyDD+6fB/PldTTV5R8e/FQYmiMaBCcTmcXULilqTG+qNgH4a/5XU Zk61V1KbF5amW/f7JbVcmp2zLnKgISxnK3GntuknlrP+SgWB5ayGLru8b g==; X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="331165819" X-IronPort-AV: E=Sophos;i="5.97,281,1669104000"; d="scan'208";a="331165819" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2023 10:12:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="644958934" X-IronPort-AV: E=Sophos;i="5.97,281,1669104000"; d="scan'208";a="644958934" Received: from tbacklun-mobl.amr.corp.intel.com (HELO [10.209.14.225]) ([10.209.14.225]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2023 10:12:46 -0800 Message-ID: <4808d3fa-bb68-d4c8-681f-0b2770d78041@intel.com> Date: Wed, 8 Feb 2023 10:12:45 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing Content-Language: en-US To: Peter Zijlstra , Bharata B Rao Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, akpm@linux-foundation.org, luto@kernel.org, tglx@linutronix.de, yue.li@memverge.com, Ravikumar.Bangoria@amd.com, ying.huang@intel.com References: <20230208073533.715-1-bharata@amd.com> From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: bawkry5gnxsetqa4xr7gswaipa5jh3zg X-Rspamd-Queue-Id: 616C240009 X-HE-Tag: 1675879984-74263 X-HE-Meta: U2FsdGVkX1+QVIR2p+/gB/dzqtjb/XC6dL7I9nPjnF2Tg/LyTCTa67m3CjfndR+SXZRdQ/O4G8cF1FqrOCQC9+g+GOwP9KmlKHms++uqIw4Tt0ceoDCcHk2wwT9DPBl8uXCV3l262FK6XGtFFqzf/tGqWnWhpla7u9yoiBZLmoZe7HYcwmfe3YJS9I7tdaNfBB17MiTkLAInSP9ffEcrry/TQuV7dFuJ4eAhUqMz0rKQ7o2CZlPY3h+RWFIQxW0DtHNxxLSlrjdfV9XWHpXmXbU9IylFRnc51nKx9M4KbHbld3/tgs86Ah97ajGNL3MdEiEMBDffiEse3SWyz4+6vRofxwp6lASPB98W7Syd3ZcKYypvfFwE5zQitaobClFdnq0zPBbCo4G6mLCpXwgDEO/JUF9MbvujcAr2Kyk10cZgrltfSCMFiD/qLbO6qM6xlleGVyUAGVDOl06MvfsX5LL+kmsKiJ16swhaZjT31qhKN+s9r6W716fT6s8JoxqNyX4EZWP+QDvUuP+tZUjfdM6lPnHb4CWY/8EN5BJ3aJ9n4ZLfUX1fapZBtWD/ufVHmImMqI98G40JUdaJKaeej5ezSlfiJMVD+tQKhQKvnknJGo4kbmQinwUx48UYijl+eQ30L2qlsa8EjCn7y5xujHIKraHdY65q7eXdtz7YaH9X0rzadkC68nWwx/uY1vgOnZBNtvQb5j0/9Ex21ttYxLXwoTFQ81v89bWVU/rrMqEKKB91GZ88v3g3KnMOQMavQS7btBNPX/KjOcxvi3+FDIoTaUllQASXR/B/ecUqOzaWa/czvMfKvptF9S7Ijty/3UihtdpHZ2cxeWYNBEZ0q+irR0v7oPZiv60VcZQ2X3Vd/hXrBC+o3YI38kJqdRXrknoj/BLOXU5d1UrDywgHPrxlUxxO0ERWOFQxnHKhihMUBjkl99/OvI0BmtRTbhiKqlxQYWTyqhDhx4pJ0E+ ewWIQDwH erubeFlCS6PE5vOhBCCRI04VFTmEyYFTPpb9BPzpH5E7vu82nPpOBJqG4HXMIqleEEgHs7WG/CiynzEYOj9u+XPsmkzEKWy9/4GOJg0FcKaHtAJK+hXG9VBSD7nzyVbd7W7aeIqdLkT0SWEyiiTWYvVkk5hHQmBYM8p5T X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/8/23 10:03, Peter Zijlstra wrote: >> - Hardware provided access information could be very useful for driving >> hot page promotion in tiered memory systems. Need to check if this >> requires different tuning/heuristics apart from what NUMA balancing >> already does. > I think Huang Ying looked at that from the Intel POV and I think the > conclusion was that it doesn't really work out. What you need is > frequency information, but the PMU doesn't really give you that. You > need to process a *ton* of PMU data in-kernel. Yeah, there were two big problems. First, IIRC, Intel PEBS at the time only gave guest virtual addresses in the PEBS records. They had to be translated back to host addresses to be usable. That was extra expensive. Second, it *did* take a lot of processing to turn raw memory accesses into actionable frequency data. That meant that we started in a hole performance-wise and had to make *REALLY* good decisions about page migration to make up for it. The performance data here don't look awful, but they don't seem to add up to a clear win either. I'm having a hard time imagining who would turn this on and how widely it would get used in practice.