From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1F1CCCA47F for ; Wed, 8 Jun 2022 08:32:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 309B68D001D; Wed, 8 Jun 2022 04:32:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B9A98D0009; Wed, 8 Jun 2022 04:32:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 180FF8D001D; Wed, 8 Jun 2022 04:32:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 07F9C8D0009 for ; Wed, 8 Jun 2022 04:32:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DB29720F92 for ; Wed, 8 Jun 2022 08:32:58 +0000 (UTC) X-FDA: 79554403236.18.9C71EE6 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf18.hostedemail.com (Postfix) with ESMTP id 2AA971C0003 for ; Wed, 8 Jun 2022 08:32:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654677176; x=1686213176; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=G52/nLgCDh+Ek240z5/x7QVUAuqh4G1y5UEKf+oYdBg=; b=dn1Csmm9PwV6kHfw5lt3aQcQ5ouSGTuQUsW1GzHrbsXr7vK+CT51welf gLlve5HQQ6KkQZ7pEaGbZQLQ677+zRFNU/IWMCy9kizyc0cFNZNMXlnr+ q67/mgaPUPMvoLNR74q7rb1pouGFXgEezEu+VMMhg1t48WTv153tkzPGv kUIFSqEejKXU9T12p6mucsQzeofSgXuLjOQq5XRvkmv8T+jJJsXlzTfOJ lRJE+knLJ+kTVJl3fG3hhfYC6ZPr4gd4uYTtPlMWGiM0MuMWc+tDqjEpN FFC3LYhdMxioqkPuAd1qctxTDwTndW+W3oQnkX9V0bQkd+n4prwli0zSz A==; X-IronPort-AV: E=McAfee;i="6400,9594,10371"; a="338613081" X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="338613081" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 01:32:53 -0700 X-IronPort-AV: E=Sophos;i="5.91,285,1647327600"; d="scan'208";a="636676678" Received: from xding11-mobl.ccr.corp.intel.com ([10.254.214.239]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2022 01:32:48 -0700 Message-ID: Subject: Re: [PATCH v5 9/9] mm/demotion: Update node_is_toptier to work with memory tiers From: Ying Huang To: Aneesh Kumar K V , linux-mm@kvack.org, akpm@linux-foundation.org Cc: Wei Xu , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Date: Wed, 08 Jun 2022 16:32:46 +0800 In-Reply-To: <232817e0-24fd-e022-6c92-c260f7f01f8a@linux.ibm.com> References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> <20220603134237.131362-10-aneesh.kumar@linux.ibm.com> <6e94b7e2a6192e4cacba1db3676b5b5cf9b98eac.camel@intel.com> <11f94e0c50f17f4a6a2f974cb69a1ae72853e2be.camel@intel.com> <232817e0-24fd-e022-6c92-c260f7f01f8a@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2AA971C0003 X-Stat-Signature: 5xbio4y6694h7dytr65bctpb7tbzebkf X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dn1Csmm9; spf=none (imf18.hostedemail.com: domain of ying.huang@intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-HE-Tag: 1654677175-88816 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2022-06-08 at 13:58 +0530, Aneesh Kumar K V wrote: > On 6/8/22 12:56 PM, Ying Huang wrote: > > On Mon, 2022-06-06 at 14:03 +0530, Aneesh Kumar K V wrote: > > > On 6/6/22 12:54 PM, Ying Huang wrote: > > > > On Mon, 2022-06-06 at 09:22 +0530, Aneesh Kumar K V wrote: > > > > > On 6/6/22 8:41 AM, Ying Huang wrote: > > > > > > On Fri, 2022-06-03 at 19:12 +0530, Aneesh Kumar K.V wrote: > > > > > > > With memory tiers support we can have memory on NUMA nodes > > > > > > > in the top tier from which we want to avoid promotion tracking NUMA > > > > > > > faults. Update node_is_toptier to work with memory tiers. To > > > > > > > avoid taking locks, a nodemask is maintained for all demotion > > > > > > > targets. All NUMA nodes are by default top tier nodes and as > > > > > > > we add new lower memory tiers NUMA nodes get added to the > > > > > > > demotion targets thereby moving them out of the top tier. > > > > > > > > > > > > Check the usage of node_is_toptier(), > > > > > > > > > > > > - migrate_misplaced_page() > > > > > >      node_is_toptier() is used to check whether migration is a promotion. > > > > > > We can avoid to use it. Just compare the rank of the nodes. > > > > > > > > > > > > - change_pte_range() and change_huge_pmd() > > > > > >      node_is_toptier() is used to avoid scanning fast memory (DRAM) pages > > > > > > for promotion. So I think we should change the name to node_is_fast() > > > > > > as follows, > > > > > > > > > > > > static inline bool node_is_fast(int node) > > > > > > { > > > > > > return NODE_DATA(node)->mt_rank >= MEMORY_RANK_DRAM; > > > > > > } > > > > > > > > > > > > > > > > But that gives special meaning to MEMORY_RANK_DRAM. As detailed in other > > > > > patches, absolute value of rank doesn't carry any meaning. It is only > > > > > the relative value w.r.t other memory tiers that decide whether it is > > > > > fast or not. Agreed by default memory tiers get built with > > > > > MEMORY_RANK_DRAM. But userspace can change the rank value of 'memtier1' > > > > > Hence to determine a node is consisting of fast memory is essentially > > > > > figuring out whether node is the top most tier in memory hierarchy and > > > > > not just the memory tier rank value is >= MEMORY_RANK_DRAM? > > > > > > > > In a system with 3 tiers, > > > > > > > > HBM 0 > > > > DRAM 1 > > > > PMEM 2 > > > > > > > > In your implementation, only HBM will be considered fast. But what we > > > > need is to consider both HBM and DRAM fast. Because we use NUMA > > > > balancing to promote PMEM pages to DRAM. It's unnecessary to scan HBM > > > > and DRAM pages for that. And there're no requirements to promote DRAM > > > > pages to HBM with NUMA balancing. > > > > > > > > I can understand that the memory tiers are more dynamic now. For > > > > requirements of NUMA balancing, we need the lowest memory tier (rank) > > > > where there's at least one node with CPU. The nodes in it and the > > > > higher tiers will be considered fast. > > > > > > > > > > is this good (not tested)? > > > /* > > >    * build the allowed promotion mask. Promotion is allowed > > >    * from higher memory tier to lower memory tier only if > > >    * lower memory tier doesn't include compute. We want to > > >    * skip promotion from a memory tier, if any node which is > > >    * part of that memory tier have CPUs. Once we detect such > > >    * a memory tier, we consider that tier as top tier from > > >    * which promotion is not allowed. > > >    */ > > > list_for_each_entry_reverse(memtier, &memory_tiers, list) { > > > nodes_and(allowed, node_state[N_CPU], memtier->nodelist); > > > if (nodes_empty(allowed)) > > > nodes_or(promotion_mask, promotion_mask, allowed); > > > else > > > break; > > > } > > > > > > and then > > > > > > static inline bool node_is_toptier(int node) > > > { > > > > > > return !node_isset(node, promotion_mask); > > > } > > > > > > > This should work. But it appears unnatural. So, I don't think we > > should avoid to add more and more node masks to mitigate the design > > decision that we cannot access memory tier information directly. All > > these becomes simple and natural, if we can access memory tier > > information directly. > > > > how do you derive whether node is toptier details if we have memtier > details in pgdat? pgdat -> memory tier -> rank Then we can compare this rank with the fast memory rank. The fast memory rank can be calculated dynamically at appropriate places. Best Regards, Huang, Ying