From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF76C61DA4 for ; Sun, 19 Feb 2023 14:43:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 124FA6B0072; Sun, 19 Feb 2023 09:43:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D4C56B0073; Sun, 19 Feb 2023 09:43:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB7166B0074; Sun, 19 Feb 2023 09:43:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DBE9B6B0072 for ; Sun, 19 Feb 2023 09:43:56 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AF0F8A9271 for ; Sun, 19 Feb 2023 14:43:56 +0000 (UTC) X-FDA: 80484310872.01.48B9259 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf11.hostedemail.com (Postfix) with ESMTP id 64B6C40004 for ; Sun, 19 Feb 2023 14:43:52 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=r6u67pNw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf11.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676817832; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eewbBuXXysSalBUW1l5/hpYuj0ZRgCIPfsivwLH37/A=; b=01TQBFpTQm7XYDAIQ31JpoVMyiGcIwtcw5eqBny81RIZHJikBNgSXJaJGxJgsIq6awBO/V Nq6bDRKX++oayjKdFesTuvn10yLZKmV2n2eCgEiBU3sYG7ZokpY02CC79txbjOSDJpZaQx kml+Vpxg7EgNY1kazSkVU0yukr7Hd1w= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=r6u67pNw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf11.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676817832; a=rsa-sha256; cv=none; b=KA6j3Deq1IP3CP6dnTGbaQgSVw5it1xGDEKzVe3L3Rx+IAWrHK3RkbbX7c6GB7A+MeHXaT ZzzsOpDugkyerNN3vG/kte6ufLQiE0chrlxV3WHQHmFqDcEXpNM32idMiIuSlTSCyjZ+EE aueSsT6jWKw/5Vaky6ghmAXFRA9IpWE= Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31JEge5N024620; Sun, 19 Feb 2023 14:43:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : mime-version : content-type; s=pp1; bh=eewbBuXXysSalBUW1l5/hpYuj0ZRgCIPfsivwLH37/A=; b=r6u67pNwmVSPGaGYsVvWLbuz+b2oY5moeWKhN61rAcsFtVq4N2sgHbQmvk0R6GmA87qo jvz6HnY7m/1Q9CMsSwt+uvX0vYUlvKcal0hzXLTWqmqdu/VjLUrYNrCMGFy29tfSMKQJ yNGRQUNild1VQ7G041ppPcely0rra0RUUcwML6PpVZHCP186rPqZGMy5rTkm5+doj1WN vUWDv7pEg4hbhwLaz/MnNLy78PLPsXUteAwjilcFkxt6jRqHMos/etdIcs3Rc1SSsim6 5q49H/bcHI8oM8eTl2dsUqVgsPgLTaxJM79B4Jfz4g4fYJgHXp/IXApzCIR5eBuJw92c rg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3nunrqg0f7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:43:43 +0000 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 31JEhgQm029557; Sun, 19 Feb 2023 14:43:42 GMT Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3nunrqg0ew-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:43:42 +0000 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 31JEJCTv013027; Sun, 19 Feb 2023 14:43:41 GMT Received: from smtprelay04.dal12v.mail.ibm.com ([9.208.130.102]) by ppma03wdc.us.ibm.com (PPS) with ESMTPS id 3ntpa6g1dn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:43:41 +0000 Received: from smtpav02.wdc07v.mail.ibm.com (smtpav02.wdc07v.mail.ibm.com [10.39.53.229]) by smtprelay04.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 31JEhex77799316 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Feb 2023 14:43:40 GMT Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 41D9058059; Sun, 19 Feb 2023 14:43:40 +0000 (GMT) Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C2B115805B; Sun, 19 Feb 2023 14:43:37 +0000 (GMT) Received: from skywalker.linux.ibm.com (unknown [9.43.112.244]) by smtpav02.wdc07v.mail.ibm.com (Postfix) with ESMTP; Sun, 19 Feb 2023 14:43:37 +0000 (GMT) X-Mailer: emacs 29.0.60 (via feedmail 11-beta-1 I) From: "Aneesh Kumar K.V" To: Matthew Wilcox Cc: lsf-pc@lists.linux-foundation.org, Linux MM , Yu Zhao , Dave Hansen , Johannes Weiner Subject: Re: [LSF/MM/BPF TOPIC] Using hardware counters to determine hot/cold pages In-Reply-To: References: <6bbf2c47-05ab-b78c-3165-2eff18962d6d@linux.ibm.com> Date: Sun, 19 Feb 2023 20:13:35 +0530 Message-ID: <87cz65g1go.fsf@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-GCONF: 00 X-Proofpoint-GUID: wo4xNqtYXCQQ_XPo8wXh_p6Qo37Vp3Uq X-Proofpoint-ORIG-GUID: z6N45lFKzbVUxcI1QzJqCjqwgs3s6-kG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-19_08,2023-02-17_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 bulkscore=0 mlxscore=0 mlxlogscore=999 clxscore=1011 phishscore=0 spamscore=0 lowpriorityscore=0 adultscore=0 suspectscore=0 priorityscore=1501 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302190137 X-Rspamd-Queue-Id: 64B6C40004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: cxyzx9mmyhuwoie7ckim5xgphssua59i X-HE-Tag: 1676817832-962930 X-HE-Meta: U2FsdGVkX19zhqwLEH9wCedmXgArJVLENJ2KTWuoURW9QkfG8TQcQpz+BTNdDVEi8j+v4BKpMchhVCJpyJ/w7dq8NhZLFTDLXYv44GwURNgFvNjRYuxj3vUgDFbiNErGVkAlISksjVxTEbdy9s5CcXDF2cIQtxLC5u8Ve/KWMluTXj/ju1FK225RpJJp88umfEb+sJAuofYSrx4zTxMOC3oV/ncSwcz1/D9X6P3RR37e514wq3BEzbFysCdokadhc+oVd/9hdcbfVXjjslZa9VmpY3wKwL/Nj0xXs/5C/nb3fPdoIGD2LQxeQbhGMoB7/FyIPCUtoJuoHtoA25+VYghe2Y9th3nVRwSOSsIx5oDlnmMhFaOGQ/YDlUFifi9Fz5qW3aT40VGoXjouaQvuOuLrFnKb9JRgNHefjT3pYuvmBqAuOo6+igUzs5AmwZAH3XynapgDNevLJ4gLaWuzrHgDr4sxhm5dBHe2JAuSXcgFJiRwlE6r3f1bDSd4kBWgIXAeTddySs9bf0Z/MiNegDzZe47xLRM3FE6eZksrqimKQxBO4x+lyN9fS17Z0Ndotqznp3mJnJCoiOT/AxIvmX41PGtUgnAtJCWJDupQJyeNUyc5xy6XPjoL9Mdp2Tu6j9ESSz8WkfQ9s06ehRnPB8xfHUBxYuJw9jp809S79aH5LRrDDlcF2bV9kF2JNc5ILmEt7bGhAvjdGesOfBGdBwxVF1v6aLrt2Db+Pu0Nf9GC1aZ6COO/JZSx2pFOCg+Q9jgyVcTHCUYAW4/znLdg5xCZmDaL3QHG/orqsPKg7gyzSuzeDGGpP2BQnYPgKpJOd6nBrRI1E8z3XJTvBPrkakk2N6NuO2F9G7lMU/q5u9TBuVekgumq6B8M/rFfmoK8HOrXahU+j+Xw4/4wt+ZAv0JVw9f/mOMZt0fSGr2AU3qC59duyaqKfrCWNRX1r4xZjyidpxER1Nk7MsIonOn QRHB33id KNyoTOTlQs8dj6FWfyzXq+FDOQmq+OuvIPf5Dsr+/KIXwoFVr3IV64qAywos/aUINOjHooWOw5Bl8cKamkSN5k8l2jG+IwFqjXX7kvVvAdaPVd8oZj2SlACogV4vVAo2zdy1ci7EyFp4soPEhtn7bAY2I/Dn1LRUJisIP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Matthew Wilcox writes: > On Fri, Feb 17, 2023 at 05:28:09PM +0530, Aneesh Kumar K V wrote: >> PowerPC architecture (POWER10) supports a Hot/Cold page tracking >> facility that provides access counter and access affinity details at >> configurable page size granularity [1]. I have been looking at using > > Does that advert contain any more information about this feature than: > > Hot/Cold page tracking | Recording for memory management > I will work with the hardware team to see if I can get a writeup done for use before the conference. But I am also interested in discussing things like who bears the cost of action based on hotness. Since a facility like this operates at the physical address range we may mostly be doing this outside the process context. For example, I could see the possibility of kpromoted which looks at the youngest generation in MGLRU and based on relative hotness move hot pages to the NUMA node from which there is frequent access. Should kpromoted do the migration? Or should it mark the pages migration ready (something like prot NUMA) and task on next access migrate the page? One of the other challenges I run into is determining the relative hotness. In most cases what we need is relative hotness not the absolute access count of a page. I also noticed that with the mongodb test, the performance varies a lot based on how we determine the relative hotness. > because I'd like to understand what its limitations are -- can > it be a per-VMA option, for example? Or is it set at bootup like > CONFIG_PAGE_SIZE? The hardware counters that are supported in the case of POWER10 are based on physical addresses. The hardware facility will count the access across a physical address range and there is a counter for each page that gives the access count and also information about which node did access the page. The page size is configurable and in POC I did use CONFIG_PAGE_SIZE. There is overhead in enabling/disabling the facility and I haven't looked at doing things like that in something like context switch granularity. Also, it monitors a physical address range and I am not sure how we can make that work for a VMA range or a task address space. > > For file-backed memory, the page cache will use variable sized > folios, depending on what it determines to be a useful granularity. > I'm _expecting_ something of the same sort for anonymous memory, although > maybe we'll make that determination on a per-VMA basis and make all > folios within a VMA the same size. -aneesh