From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39625C433EF for ; Wed, 8 Jun 2022 14:20:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F4B46B0072; Wed, 8 Jun 2022 10:20:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A44E6B0074; Wed, 8 Jun 2022 10:20:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 845226B0075; Wed, 8 Jun 2022 10:20:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 733FD6B0072 for ; Wed, 8 Jun 2022 10:20:41 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4A9CC34AF8 for ; Wed, 8 Jun 2022 14:20:41 +0000 (UTC) X-FDA: 79555279482.04.E691766 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf23.hostedemail.com (Postfix) with ESMTP id D53C0140065 for ; Wed, 8 Jun 2022 14:20:36 +0000 (UTC) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 258DrhDx003565; Wed, 8 Jun 2022 14:20:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=Wlpmhk8Sy94uyD2Oe92OGuqVGOTTtGLcVj9ZLjDI6hM=; b=gEkG9r7NyG3t6AqJ/hfQcYfZKnnLB+bjoNaKzbcwCQO2C8AfvfT+3tlOfctfoOxyAgWb v7pil6QQWK3krENrQfLzO3GY68Ll9eTSN8JHsdI3/2TC9Uyi3FQlanT2I+idrqfUjqHl alYLFtbiana8o6Si+2iwhVwhDuCOlL1izwkJWUC8UiboLUZGNOcUCoiIbg9F3NNPwz8e SeoQceB1cQPcSpMWKrOscJqzgB8Amt/K+CgnIemvVTkRTavRWIl4ouGs2ysX8lJX4PMd ZCHQt1j44jezBlL6WQcnGrSQk8orRYMX7FgqnSEbFfcR8dl+Rv29vwiH+yrind4YTQVs xQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gjw298kms-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 08 Jun 2022 14:20:22 +0000 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 258Du5Pj022410; Wed, 8 Jun 2022 14:20:22 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gjw298km1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 08 Jun 2022 14:20:21 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 258EHitp007459; Wed, 8 Jun 2022 14:20:19 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma06ams.nl.ibm.com with ESMTP id 3gfxnhwepe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 08 Jun 2022 14:20:19 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 258EKHul48824722 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 8 Jun 2022 14:20:17 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2D0A052054; Wed, 8 Jun 2022 14:20:17 +0000 (GMT) Received: from [9.43.53.124] (unknown [9.43.53.124]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 93CEB52050; Wed, 8 Jun 2022 14:20:12 +0000 (GMT) Message-ID: <8516237f-c1a0-aefa-274a-9f8794ae0ccd@linux.ibm.com> Date: Wed, 8 Jun 2022 19:50:11 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: [PATCH v5 0/9] mm/demotion: Memory tiers and demotion Content-Language: en-US To: Johannes Weiner Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> From: Aneesh Kumar K V In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: rEgwQMchZymyHtUVMDMzm0rKQtrcub_E X-Proofpoint-ORIG-GUID: xiWBMacox6yjAtuobay-DjL9M-QgzKmY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-08_04,2022-06-07_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 bulkscore=0 lowpriorityscore=0 spamscore=0 mlxlogscore=778 suspectscore=0 phishscore=0 adultscore=0 priorityscore=1501 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206080061 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D53C0140065 X-Stat-Signature: mdnjbw3xh3kw6gof444at3nhp4ksnj18 X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=gEkG9r7N; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com X-HE-Tag: 1654698036-281306 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 6/8/22 7:27 PM, Johannes Weiner wrote: > Hi Aneesh, > > On Fri, Jun 03, 2022 at 07:12:28PM +0530, Aneesh Kumar K.V wrote: >> * The current tier initialization code always initializes >> each memory-only NUMA node into a lower tier. But a memory-only >> NUMA node may have a high performance memory device (e.g. a DRAM >> device attached via CXL.mem or a DRAM-backed memory-only node on >> a virtual machine) and should be put into a higher tier. > > I have to disagree with this premise. The CXL.mem bus has different > latency and bandwidth characteristics. It's also conceivable that > cheaper and slower DRAM is connected to the CXL bus (think recycling > DDR4 DIMMS after switching to DDR5). DRAM != DRAM. > > Our experiments with production workloads show regressions between > 15-30% in serviced requests when you don't distinguish toptier DRAM > from lower tier DRAM. While it's fixable with manual tuning, your > patches would bring reintroduce this regression it seems. > > Making tiers explicit is a good idea, but can we keep the current > default that CPU-less nodes are of a lower tier than ones with CPU? > I'm having a hard time imagining where this wouldn't be true... Or why > it shouldn't be those esoteric cases that need the manual tuning. This was mostly driven by virtual machine configs where we can find memory only NUMA nodes depending on the resource availability in the hypervisor. Will these CXL devices be initialized by a driver? For example, if they are going to be initialized via dax kmem, we already consider them lower memory tier as with this patch series. -aneesh