From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 771AEC433EF for ; Wed, 8 Jun 2022 13:57:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B99236B0071; Wed, 8 Jun 2022 09:57:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B487A6B0072; Wed, 8 Jun 2022 09:57:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F2C96B0073; Wed, 8 Jun 2022 09:57:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8CC246B0071 for ; Wed, 8 Jun 2022 09:57:40 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id CFF526129C for ; Wed, 8 Jun 2022 13:57:39 +0000 (UTC) X-FDA: 79555221438.15.0D81B9A Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf08.hostedemail.com (Postfix) with ESMTP id 07E4116005D for ; Wed, 8 Jun 2022 13:57:38 +0000 (UTC) Received: by mail-qt1-f175.google.com with SMTP id x18so12052867qtj.3 for ; Wed, 08 Jun 2022 06:57:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=yKWpjNlFyucWAt9mqVc0aRQ08bT80cWgGBrBg1EuJ2E=; b=TxekB4r16dYUeS0Pbv5Yfw5HVjM/rXjUXlNEKauPmriwFZPOjIcmHkpO1e9zBAPiaE dEwLwCxUyTuSCyFSoAhwYHRD0rmBLDC3arxOlbSNTIC/GkCHLIUzbWsAJgxQz39UZUv0 3PZvIh3yI4WZyQMxbMwOwkqS+EkJuNaYCHhKA/rijb+V9T/6edPd5/9J9BTsRiM22kG+ 9CRe3TZA15IlAQvELgI/Ge9G880M3T/Y83+mN+8D1TNi7+TwrJQdgqesimVIST1inYGw Gfou9q3vcw+AlOPH4y4HMR65nrueYHlBwLHdGUNFxpVaG9SnkmC3Czpp6qYrBlu1dDKb Npvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=yKWpjNlFyucWAt9mqVc0aRQ08bT80cWgGBrBg1EuJ2E=; b=A1+qnP7LLg7+Qbl4YQ0p0I8n2MAAcJ4LzPwAPcbXeJHFFi6CBcVl8Ca9NlPCA45qnQ T1H6tCFdUL2mObEP4qEkjjaEs19uWQ3CzvbmpYhL1k7bzIpC9KVBL+Gshgumvg2j+LDz cMIkC043iruZ2Ie8b0S3pZGjZ8bgvvOSiA9+JJA8ZDBG5MX+YXcqdfou4t+HG3+2xFbV A1gJW/y56e+y8RKY9rAPJCRaGUmbCQUFPSV7QteMT+CGrG/7Qu9vaWTqoWLWQv0OqvTd mzg1LX2985n883oarE94pQgIzAknM5krkyd9DxJLGl2w7JPXVXtX3Vf3GPwZ2L3Zj3XK MFOA== X-Gm-Message-State: AOAM530goxouo+uZEHK6YkIO2JrMu3Kvaz1rkm4wCwMHIQ5cyWBcmjDZ mGyYXrHcgB/X4Lbs59Liepb3nw== X-Google-Smtp-Source: ABdhPJwgRvq7YLEVg9fZVuEUgucmnKBbV/WowGfsVbwQxXnAPSTBkNwab88LnBa9+ttaSV8AJmMADg== X-Received: by 2002:ac8:7dcb:0:b0:2f3:eb31:1e33 with SMTP id c11-20020ac87dcb000000b002f3eb311e33mr27199280qte.598.1654696658166; Wed, 08 Jun 2022 06:57:38 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:4759]) by smtp.gmail.com with ESMTPSA id k66-20020a37ba45000000b006a37eb728cfsm16158987qkf.1.2022.06.08.06.57.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 06:57:37 -0700 (PDT) Date: Wed, 8 Jun 2022 09:57:37 -0400 From: Johannes Weiner To: "Aneesh Kumar K.V" Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 0/9] mm/demotion: Memory tiers and demotion Message-ID: References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=TxekB4r1; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 07E4116005D X-Rspam-User: X-Stat-Signature: z13x81hop9b4jig5njujdh15rhi5ywgc X-HE-Tag: 1654696658-164092 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Aneesh, On Fri, Jun 03, 2022 at 07:12:28PM +0530, Aneesh Kumar K.V wrote: > * The current tier initialization code always initializes > each memory-only NUMA node into a lower tier. But a memory-only > NUMA node may have a high performance memory device (e.g. a DRAM > device attached via CXL.mem or a DRAM-backed memory-only node on > a virtual machine) and should be put into a higher tier. I have to disagree with this premise. The CXL.mem bus has different latency and bandwidth characteristics. It's also conceivable that cheaper and slower DRAM is connected to the CXL bus (think recycling DDR4 DIMMS after switching to DDR5). DRAM != DRAM. Our experiments with production workloads show regressions between 15-30% in serviced requests when you don't distinguish toptier DRAM from lower tier DRAM. While it's fixable with manual tuning, your patches would bring reintroduce this regression it seems. Making tiers explicit is a good idea, but can we keep the current default that CPU-less nodes are of a lower tier than ones with CPU? I'm having a hard time imagining where this wouldn't be true... Or why it shouldn't be those esoteric cases that need the manual tuning.