From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F9E7C43334 for ; Thu, 9 Jun 2022 08:53:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E07D96B008C; Thu, 9 Jun 2022 04:53:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB72A6B0093; Thu, 9 Jun 2022 04:53:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7F876B0095; Thu, 9 Jun 2022 04:53:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BA6AE6B008C for ; Thu, 9 Jun 2022 04:53:20 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8B3D161363 for ; Thu, 9 Jun 2022 08:53:20 +0000 (UTC) X-FDA: 79558083360.04.D8F9428 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf10.hostedemail.com (Postfix) with ESMTP id 9283DC0034 for ; Thu, 9 Jun 2022 08:53:19 +0000 (UTC) Received: from fraeml736-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4LJd6P0x48z67vYh; Thu, 9 Jun 2022 16:48:29 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml736-chm.china.huawei.com (10.206.15.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 9 Jun 2022 10:53:16 +0200 Received: from localhost (10.81.202.195) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 9 Jun 2022 09:53:15 +0100 Date: Thu, 9 Jun 2022 09:53:13 +0100 From: Jonathan Cameron To: Aneesh Kumar K V CC: Johannes Weiner , , , Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 0/9] mm/demotion: Memory tiers and demotion Message-ID: <20220609095313.000003f7@Huawei.com> In-Reply-To: <8516237f-c1a0-aefa-274a-9f8794ae0ccd@linux.ibm.com> References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <8516237f-c1a0-aefa-274a-9f8794ae0ccd@linux.ibm.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.29; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.81.202.195] X-ClientProxiedBy: lhreml731-chm.china.huawei.com (10.201.108.82) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654764800; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KUGhvkeqyPja/l8hzg75zUwzBIH7t9yLlZAIiulUbq0=; b=c+zOYtjBF5p/Ko0xCgTSpCWjG9BzBQSpiSMFTflZbDJBmLmEdn414fkrt7XN40A7AINLls tCBO3/dauYMoq9Crp8fXYiR2yoDbCv397f1hZj/gxobMJXzXHcwg1cInX357LliktZZ+qC etiqB3jIpMHzWfv31ezDZrto73HTDUc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654764800; a=rsa-sha256; cv=none; b=eE3gOjLDAnEICgZntKyTJcXJUnGljOrY3bCE2m+8IFt+AHAyhWm5TpNvhu8D4rS0n5Akjw 54rQec+xjxKVt6GpVpKn16HmqhDMpF3cawBDBJhPtXrefr7ZRTdmyATL8AmoEB3RyXnQkN yJ8o5QEWc0Fux7H/H+vLdqpY7JdqWpI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf10.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf10.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com X-Stat-Signature: 5nd5cm7y4x3pfd6p8nbbpt9dofo18m16 X-Rspamd-Queue-Id: 9283DC0034 X-HE-Tag: 1654764799-525799 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 8 Jun 2022 19:50:11 +0530 Aneesh Kumar K V wrote: > On 6/8/22 7:27 PM, Johannes Weiner wrote: > > Hi Aneesh, > > > > On Fri, Jun 03, 2022 at 07:12:28PM +0530, Aneesh Kumar K.V wrote: > >> * The current tier initialization code always initializes > >> each memory-only NUMA node into a lower tier. But a memory-only > >> NUMA node may have a high performance memory device (e.g. a DRAM > >> device attached via CXL.mem or a DRAM-backed memory-only node on > >> a virtual machine) and should be put into a higher tier. > > > > I have to disagree with this premise. The CXL.mem bus has different > > latency and bandwidth characteristics. It's also conceivable that > > cheaper and slower DRAM is connected to the CXL bus (think recycling > > DDR4 DIMMS after switching to DDR5). DRAM != DRAM. > > > > Our experiments with production workloads show regressions between > > 15-30% in serviced requests when you don't distinguish toptier DRAM > > from lower tier DRAM. While it's fixable with manual tuning, your > > patches would bring reintroduce this regression it seems. > > > > Making tiers explicit is a good idea, but can we keep the current > > default that CPU-less nodes are of a lower tier than ones with CPU? > > I'm having a hard time imagining where this wouldn't be true... Or why > > it shouldn't be those esoteric cases that need the manual tuning. > > This was mostly driven by virtual machine configs where we can find > memory only NUMA nodes depending on the resource availability in the > hypervisor. > > Will these CXL devices be initialized by a driver? In many cases no (almost all cases pre CXL 2.0) - they are setup by the BIOS / firmware and presented just like normal memory (except pmem in which case kmem will be relevant as you suggest). Hopefully everyone will follow the guidance and provide appropriate HMAT as well as SLIT. If we want to do a better job of the default policy, we should take the actual distances into account (ideally including the detail HMAT provides). Jonathan > For example, if they > are going to be initialized via dax kmem, we already consider them lower > memory tier as with this patch series. > > -aneesh