From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A13F2CDB482 for ; Mon, 16 Oct 2023 08:00:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D4AC8D0014; Mon, 16 Oct 2023 04:00:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 184A38D0001; Mon, 16 Oct 2023 04:00:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0744A8D0014; Mon, 16 Oct 2023 04:00:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E9BDF8D0001 for ; Mon, 16 Oct 2023 04:00:03 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9B34E160B29 for ; Mon, 16 Oct 2023 08:00:03 +0000 (UTC) X-FDA: 81350576286.24.C884E1B Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by imf26.hostedemail.com (Postfix) with ESMTP id 546C7140008 for ; Mon, 16 Oct 2023 08:00:00 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Y6FGe4e7; spf=pass (imf26.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697443201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y4ikgIb4e0yh52GxTUxC2HW+WSNCLCNZXHVb8cDfLuE=; b=GHbdBPi6PalR5wzfDfp9pdwtWCYrezRrHdOg5DMK4xFmXFxJvE4DhOkU6yn5zeSPgCILKJ ZyTd1thC3/pN/F6vqIRYIzyfKguJhHknZOGmqyXf6IZ5CaWARdpnsd9Vic79Ym5ogeySca AXQ/JkTlpQIxUbo6JkDVmDQS5jOmZxk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697443201; a=rsa-sha256; cv=none; b=X1qoheBAalKY74DgCVC58fTmCLjMxLNc1efveLenRJLRYTdgiv9MoM8sT3c3AYKwa7thhN 9m35wyU86QX5ShX2WWowRRtpGnWa1kS1vYllv8FJAEZB7r5mfGFibfFE/NpLaK5U3DfslL BKa7LFx7kbCE683a1sTD6XnQ2JkLqtY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Y6FGe4e7; spf=pass (imf26.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697443200; x=1728979200; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=XYXbGQsgnjt7nQUqln8pZSvBdMQSiZ+wMR4h/e3odww=; b=Y6FGe4e7HEd51xx8WAWYulORpkrV9WmwbzVkM2+wDgGNJa8LmIP2nQkO Ka4J8d5ctuLEnFbqXOyp0F4om1EHVk6DLnMrohJCHeRTHpHZidFmjqh7b 66xrU4fOHzTFfOJPNjfWtMyoqKiLAQwcZeGP8RIqCwDuzpayAwWKE2LwC 3PyPSFv7KVaI+nAR3A+k/fqg9Cycn/HrmZguN4KIOF/hYR54s23Jg8tl3 0RePmQWXb50yQeLL4TatDVcLnelpijZL/ZGAD1muGdBpQgVn0FLKKeXD+ 8OqMuIN7jr1/bHVsMEF0nzCPVm9nu0pQ3qduwpBRPLQaG/gGX77PmMOVQ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="449677956" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="449677956" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Oct 2023 00:59:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10863"; a="790721641" X-IronPort-AV: E=Sophos;i="6.03,228,1694761200"; d="scan'208";a="790721641" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Oct 2023 00:59:56 -0700 From: "Huang, Ying" To: Gregory Price Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, akpm@linux-foundation.org, sthanneeru@micron.com, gregory.price@memverge.com Subject: Re: [RFC PATCH v2 0/3] mm: mempolicy: Multi-tier weighted interleaving References: <20231009204259.875232-1-gregory.price@memverge.com> Date: Mon, 16 Oct 2023 15:57:52 +0800 In-Reply-To: <20231009204259.875232-1-gregory.price@memverge.com> (Gregory Price's message of "Mon, 9 Oct 2023 16:42:56 -0400") Message-ID: <87o7gzm22n.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 546C7140008 X-Rspam-User: X-Stat-Signature: tuhgjcrgbsgi9y1sudaqqx6snszd46th X-Rspamd-Server: rspam03 X-HE-Tag: 1697443200-314145 X-HE-Meta: U2FsdGVkX193l2OBqkGBOA6ZxkX8l4a9azbrT8lN9kiPz4r6rP+tVMycJWNuear7fOA77yA93NNMaox2/rPH6jZ8yMwdlDcdVsgLMKKXWi144Y9ONxwXP1ZSjeG+cSQLvjrBXLuKXULOCcRwp24Y5guUtRUiLEliGB2+bT7Rq8Npd3AQHYYeDqcn/lbf65OBJDY/K9+9fqqr4IxT/mxk2J0iP0sb4MADtQ9G04oJdq29rTbuXx/sZBihIev8CenXb9Jjk8BM5BgiIRf0GSubk0gQ0P9D3V/DPcQSwI2CcfboTATB+sJ3mvg4agy432PSfGtlJ68lZ6ifOI1EeBz+RhSU/XWw3K72i8kOiZHUQlThRWTMMfC7J5AGN6LK5yW2V+b1p/wxKW0CRzV8tcm+7OZE7JAu1sz75S0XVzaLJAgl5rxY0kD2BlEIWqTWgaxJnb1k8rOUD4DKWGfMhPmItDS0N6vNeVsORRgQbOi8iRLDUDwtJgwHQ9PQEgGmVK1VgvhZtUEDsBOCMelMJDrR7gn6ol6MSxdtwM6PujXX4xyBjo4LVcoWAWAaaoTkybyHw4rrEgUnxiAZaEXeLZ+/dzfLvQWpvQXsaaLUdiFUviFbR0LJdCsPVNhkt50ccdvJfUWqFol7AaIioeYCPE6efeU5Zx5YzQSRhtDsf7SiDWq3AviZndZGtUhxPgcMgUJ6g/VO496zIo6cn3sfBlLZT42wv9Jnr8wJD6fI/S/kw8Cn7Y5iBI8750sTCFsbIotF3d5STi125NAgODfHyCtVJ8E6GcJHl6SMGAt1XcP0zGH1HAgnIsDl2DYZJPYlRODMPVYpiC/+KIwn+/aqSU71CbPwYxPK6jp7mfRtQLig2u3geXK7Uqw1n1D378rhbmPcOIQziLgrn/5NrEMbUEyW3WlQuSqqWK+BarMRtm9A9bzVCQWt1XjBSmJPh7xso4Gw6OB4wBeuynlq8UP8wqV OeoVG3rm ojYnkFBywZQb/6nU8SldX11qwbSlBRmjX7Yad1iV9k9S0LETowfqs9MdmOgE7l1KfuN/jHfU+iRFC1LHSdz5rH+tUfvvCq7yXB2hPFDmpygDu/tgd0eKvVXQ5OmQjTo66SqubT8WPE5RkocwonXYuXOW8qbu2UH+D5QaKZ1wSnN6vk3yygYwSOSkY+/0zvHFokCQwQ7YM0mykaUz/+O/B1b/3zvXv42grLTF7y7C+qJxrFgXnrxaRKuQoYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Gregory Price writes: > v2: change memtier mutex to semaphore > add source-node relative weighting > add remaining mempolicy integration code > > = v2 Notes > > Developed in colaboration with original authors to deconflict > similar efforts to extend mempolicy to take weights directly. > > == Mutex to Semaphore change: > > The memory tiering subsystem is extended in this patch set to have > externally available information (weights), and therefore additional > controls need to be added to ensure values are not changed (or tiers > changed/added/removed) during various calculations. > > Since it is expected that many threads will be accessing this data > during allocations, a mutex is not appropriate. IIUC, this is a change for performance. If so, please show some performance data. > Since write-updates (weight changes, hotplug events) are rare events, > a simple rw semaphore is sufficient. > > == Source-node relative weighting: > > Tiers can now be weighted differently based on the node requesting > the weight. For example CPU-Nodes 0 and 1 may have different weights > for the same CXL memory tier, because topologically the number of > NUMA hops is greater (or any other physical topological difference > resulting in different effective latency or bandwidth values) > > 1. Set weights for DDR (tier4) and CXL(teir22) tiers. > echo source_node:weight > /path/to/interleave_weight If source_node is considered, why not consider target_node too? On a system with only 1 tier (DRAM), do you want weighted interleaving among NUMA nodes? If so, why tie weighted interleaving with memory tiers? Why not just introduce weighted interleaving for NUMA nodes? > # Set tier4 weight from node 0 to 85 > echo 0:85 > /sys/devices/virtual/memory_tiering/memory_tier4/interleave_weight > # Set tier4 weight from node 1 to 65 > echo 1:65 > /sys/devices/virtual/memory_tiering/memory_tier4/interleave_weight > # Set tier22 weight from node 0 to 15 > echo 0:15 > /sys/devices/virtual/memory_tiering/memory_tier22/interleave_weight > # Set tier22 weight from node 1 to 10 > echo 1:10 > /sys/devices/virtual/memory_tiering/memory_tier22/interleave_weight -- Best Regards, Huang, Ying