From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ED95C4167B for ; Thu, 2 Nov 2023 09:36:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15B788001C; Thu, 2 Nov 2023 05:36:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10C898D0026; Thu, 2 Nov 2023 05:36:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEF9F8001C; Thu, 2 Nov 2023 05:36:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DCE618D0026 for ; Thu, 2 Nov 2023 05:36:00 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B7E5BB61F8 for ; Thu, 2 Nov 2023 09:36:00 +0000 (UTC) X-FDA: 81412507680.28.9EF6825 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2051.outbound.protection.outlook.com [40.107.94.51]) by imf10.hostedemail.com (Postfix) with ESMTP id 830C6C0009 for ; Thu, 2 Nov 2023 09:35:57 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=micron.com header.s=selector2 header.b=lEJQuqQb; dmarc=pass (policy=reject) header.from=micron.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf10.hostedemail.com: domain of Ravis.OpenSrc@micron.com designates 40.107.94.51 as permitted sender) smtp.mailfrom=Ravis.OpenSrc@micron.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698917757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pquRgceUHQEpluTkrmopiYyjq4xjUSGvH/6ZsJJQr4M=; b=KUQyeu3VU4bf+HuHOPxfL0Nz7XbeUFNwTTF57K0WEkzn/izl/EFaKv94q77dllEVzjsc6n r9WwO++hdWOVJUcvKP+Ag88cuwwTLAJmB2s3Cp4u7WaPuCV0pJ3U+Uf2eAD7kWJ5chwEzK YgLNt1ENA35YgJHvsHZsdHt/7jfpbcM= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=micron.com header.s=selector2 header.b=lEJQuqQb; dmarc=pass (policy=reject) header.from=micron.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf10.hostedemail.com: domain of Ravis.OpenSrc@micron.com designates 40.107.94.51 as permitted sender) smtp.mailfrom=Ravis.OpenSrc@micron.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1698917757; a=rsa-sha256; cv=pass; b=rQ7W+X97Cor6NT0VvJcsBsrh6T3r+C2SC7Gh7yonSKtlXHf6+za2EOjDZ2FEiwEnLTO0Ew 879qVsrSTp9zFDwWO2DfigPXy0puWMTyPHpR7pXqVuPwemKyujlUyz+gLi6fI38rYxYm0I yy1OIQqoq5GNCnpnILmGUtkc97uQ7tQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NlB8RKUm5QEc7UJdVvK/cG54kCK/m3yaGYxDq12RzyJL8LaGO2pKrSPl8JlWh/LoolEaPGRoZ14f7uhtGTYY3XbHP1umz/KEjxGEWO1kNeJ/QNi+gCGhd+OBtDOFDDjlN9aHEuQSAnmYdD+QRvRAt4wTeNjK9sivF/h9ToRCqOCBcyWJiwEcATBSNttbCjIpeFj2yxXhNydYp3+usB45tLF+E4nYQKuarTU+q+D2D20+H4n8qFfjOwOGOZ6Uc8mwkGbGSeTQ6NiKd0REGU4hUU7Wk7K+zAsB/+KDE1l4tfZnhlfqUzNPo092yMr3YKW7amoj954E+52ecjCDNq0C5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pquRgceUHQEpluTkrmopiYyjq4xjUSGvH/6ZsJJQr4M=; b=SeA56GwUWSelYDQWQIins/ac0U15zBwCVT8DLFE62Yr3hokqCmAL6uD3NzXdhQbsklW74Q/FX8W3UP0uZda78+fXFUIYcon1BUO05J1Ntw/K9YyfgRGiqngZaNSYw5PI8fXjH9PhXZxcioYeum1QV/Dh/udds4M7uCqPHrKkioliSv00135ZmeSI8jyrdc+8G6a0Lb7rg+jr1xgNal9z6LXS5GKZ6axqFJ5nQ0daKCmIrlWYarv3l69C4IfOtBichl5kldDcA1EahqLrBlfEwWOiSAJlY9T8sVGtSkohmrc9dWZdqOdTWfbKVoUEpdXviJUg7x//K/afa9lAljhZ0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 137.201.242.130) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=micron.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=micron.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=micron.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pquRgceUHQEpluTkrmopiYyjq4xjUSGvH/6ZsJJQr4M=; b=lEJQuqQbZDVyGDl4jTeQzRdsi3x+AxKAxcu6MJ6Rk3Got/wIMaiAaz9nlL3SlNIhpyPh8N8YY/tu7qW8ZNJgFAplOeojjc5U3OgTCEL7dqQajHhZf+/LGS9I9yoU+o2FtMHAKA5FUyjhdABrqdpMhp2tYW6WsMhJhgfs6Ls7G9qEMwQH+ploeRRbWvXNxsso+7DgBlYqtazBtWl7lwHY946KZn8u+jr4H43DvX2vE/6U30eMshSZXiasBFi4USDwN8dOhqbd/YIA6K8HMPOe+TUBZqR6EdzuhPtZMMZve5A2G7iOna/LiADC2H/GHmlG9vXYMM5ECgD0ziz6VMUSWQ== Received: from MW4PR03CA0001.namprd03.prod.outlook.com (2603:10b6:303:8f::6) by BLAPR08MB6884.namprd08.prod.outlook.com (2603:10b6:208:30d::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.20; Thu, 2 Nov 2023 09:35:51 +0000 Received: from CO1PEPF000044F3.namprd05.prod.outlook.com (2603:10b6:303:8f:cafe::bc) by MW4PR03CA0001.outlook.office365.com (2603:10b6:303:8f::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.19 via Frontend Transport; Thu, 2 Nov 2023 09:35:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 137.201.242.130) smtp.mailfrom=micron.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=micron.com; Received-SPF: Pass (protection.outlook.com: domain of micron.com designates 137.201.242.130 as permitted sender) receiver=protection.outlook.com; client-ip=137.201.242.130; helo=mail.micron.com; pr=C Received: from mail.micron.com (137.201.242.130) by CO1PEPF000044F3.mail.protection.outlook.com (10.167.241.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.19 via Frontend Transport; Thu, 2 Nov 2023 09:35:50 +0000 Received: from BOW36EX19B.micron.com (137.201.85.154) by BOW36EX19A.micron.com (137.201.85.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Thu, 2 Nov 2023 03:35:49 -0600 Received: from VENKATARAVI-LAP.micron.com (10.3.99.248) by RestrictedRelayBOW36EX19B.micron.com (137.201.85.154) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Thu, 2 Nov 2023 03:35:44 -0600 From: Ravi Jonnalagadda To: CC: , , , , , , , , , , , , , , , Subject: Re: [RFC PATCH v3 0/4] Node Weights and Weighted Interleave Date: Thu, 2 Nov 2023 15:05:42 +0530 Message-ID: <20231102093542.70-1-ravis.opensrc@micron.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <87a5rw1wu8.fsf@yhuang6-desk2.ccr.corp.intel.com> References: <87a5rw1wu8.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain X-MT-Whitelisted: matched X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F3:EE_|BLAPR08MB6884:EE_ X-MS-Office365-Filtering-Correlation-Id: 1ad04d71-4ffa-4079-5d79-08dbdb871a12 X-EXT-ByPass: 1 X-MT-RULE-Whitelisted: Triggered X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jLbthI9v9fXjYMs4boDz1xzYsb2R1RnNFYv9zPFAtIGhF16zi7u6oRulvuZV1RTEUukF731bjXhWOJcsglWF8isxQaVsni5JBcEQ2luJzFdfZO03bD/thUIGDfnJcZWWV5MOAFv9Q7pcbuikk8MLgmHOVQjEv9thmcES7UAB39JyBZBqn04Zn7LZiGabO6eF/dEFsaKyz6TlTwxC87hpXSaHeSLIRHtP5sUuTfbKygUBiiDjHlnrkJ+oOrNkDXPtrFkaBP/BCpDuh0p6rDCIr/Agg/4+UQlBhvKKEKJVGStAwXy5sFfGH/0XSsGPyYShSmGPrWYZg9hDwJHmpGOjKopm5p8SrLJoeqOkk72cPhjGkl6oev1yCNlcWB/VEUZj/sfZ+s56TD8ekwlzVSiG5LLHtA8JYkAEm8DiTBOa7R3kAKObmH5kS4B8r8KBYr8I3gq/QujmEEflbA2Zwg2F4mlgZetsxAg8H87LXTKLacVfRX/fMZ/Im7esuobJueNcoru9z0cNwqBebRLyyr3U5mNcI1dR62wlHUH1HjEqZ9eJ/WcMo7PHCal0A7Zam7q1pEdVFLDp2UOXlpOMEJuumuINVZbVHgcGOFDkBK38wBAJgAbDpRc0tbWWw1hO05d4l8CXjU78pv9pX8P7UN4xFrcxjVuKktj8dGoP5PoFKZFO/WjOluikJd8qQUrha0NZyD8Q/yBnH8ep5uYA6QdWeBSiQbxYLuMpSz1L27/3L9Kc5u7Pek6kFcoXbrhQkgaE4oVrgedp01Zfy+seC5YFFA== X-Forefront-Antispam-Report: CIP:137.201.242.130;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.micron.com;PTR:masquerade.micron.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(396003)(136003)(346002)(230922051799003)(82310400011)(451199024)(1800799009)(186009)(64100799003)(40470700004)(46966006)(36840700001)(26005)(36756003)(1076003)(54906003)(316002)(70206006)(70586007)(6862004)(8676002)(4326008)(336012)(8936002)(2616005)(478600001)(426003)(5660300002)(40480700001)(36860700001)(47076005)(7416002)(2906002)(7696005)(356005)(40460700003)(41300700001)(7636003)(82740400003)(66899024)(86362001)(16393002);DIR:OUT;SFP:1101; X-OriginatorOrg: micron.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 09:35:50.6442 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1ad04d71-4ffa-4079-5d79-08dbdb871a12 X-MS-Exchange-CrossTenant-Id: f38a5ecd-2813-4862-b11b-ac1d563c806f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f38a5ecd-2813-4862-b11b-ac1d563c806f;Ip=[137.201.242.130];Helo=[mail.micron.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F3.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR08MB6884 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 830C6C0009 X-Stat-Signature: 6puj8ptzaan5syz84m7xrp7x1gbexsfc X-Rspam-User: X-HE-Tag: 1698917757-678859 X-HE-Meta: U2FsdGVkX1+a51Wdj+WZHvhkH/Q48w/AtxOY5M0KUDXftiEnjmBOIOqQe2pehwPZFZM0vwEvzKOYUd4XRC//gxipPj1gjm9dG6jg6tpOT3+GG4wH5GOkXXx6kGBgq2EIgS3o+TafoVSMp6KD9CfRXjvUcgWVtlz/7k5pEmJ3Lb6VMP/qHa493tJpEIKQ/g35rL+gVrPk3n7DtsQdop9QRQ2lo/MwAUKZumhGbQIz9p4yuT1+y6ztZ706fBJuKIA5CBTXI0HO4MixNT5TB4eFm8Bmx8TWvw4UuqmzIAlf0wQ7jl/BS4a0v09lnCd1wo7JhOHOOf6dbLPsJUMxqZW57iJD7iV0ADt68rZ8eYpIqNXOQXAhpf62wU+/ilW7YtlVm88gpFreKeUn/BVwjQuNb3/NiknDaF/xanuXt2N02oTVAVQEpbBcuKU+e3fIhMJleRyQsZNFzak74/ztUVVWMQd+3Jo5owM8OmcLmhHPqgtVStCA/9eILGUqY54mV5XwQEtYAxtuWOs3/MrfQ6tl4OJz1V93okm26jJpNxbck9vtOa901JYJzmWGpNjPSFWG+mogMN3c9JTuay6evkNiGEjiXjXwUC9pE2utCGmybVFLXqlVTf0R7653bppBUcuTQDM3H57tu4sDJITxSp1+3da+ULb164FUT+qCV1LEVnFYCW0N69ZF3v9H0iT6HFdcisscu/k1bt/TICDnit6LhiYkfmUc2eYxex01hZlmG7tl8hYfVVQEWRhA8DoXaP7zkq7yKe72NeUWY23Gx9gGVkJ9MZ+zmn16L4UENUzLpsG6pDLfM++IClL7PmkzFeJhzUIiAjR//CoVUCAooeciLPfvab839D9qTACg2i7xAr8X0ZRUK1aI96uS++8LyZ+LZeCmStK7VHyeMuBUYFgIHjf3Ic4/430Vfjg2Uat6Sh/mPlkzjuiWZs8KH04ZfZq6kzXqq6NYrSVVIxX9tTC lQwaqDv4 0eNXtVHXROy1uzBYM+Mb/5PpzR8llV5yJZTQWqrxsGU1f0wd/wMH5WjnwVtY6vTJoDo8Ta+HojK8VcVYnYC7d/p3i+UGepi+kTfRo6s2ZtIHiKCmfo2SWfXcsn3Px8/BVox8cmCzUp2CNz7FMys3Zvz+1VdKD1Lo8PA4MUIRbdUbzmcrcq4qU2xT/tGlHBK0uyR6r1coD9bu8gy0dNABLl665fWQC0J3V2rRi1nYqpCV6FuC6JI9At9JEgWrryhgrqMP+ddmWbBeN+ycnWBCmks7H0+AVLN+iYOqh6NyGBBV3VaO7drZZxuKmOJYWsQgeiLMXp1rEepbl6ISQeAEtxa1cb6MyTMgWD5fB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Should Node based interleave solution be considered complex or not would pr= obably=0D depend on number of numa nodes that would be present in the system and whet= her=0D we are able to setup the default weights correctly to obtain optimum bandwi= dth=0D expansion.=0D =0D >=0D >> Pros and Cons of Memory Tier based interleave:=0D >> Pros:=0D >> 1. Programming weight per initiator would apply for all the nodes in the= tier.=0D >> 2. Weights can be calculated considering the cumulative bandwidth of all= =0D >> the nodes in the tier and need to be programmed once for all the nodes i= n a=0D >> given tier.=0D >> 3. It may be useful in cases where numa nodes with similar latency and b= andwidth=0D >> characteristics increase, possibly with pooling use cases.=0D >=0D >4. simpler.=0D >=0D >> Cons:=0D >> 1. If nodes with different bandwidth and latency characteristics are pla= ced=0D >> in same tier as seen in the current mainline kernel, it will be difficul= t to=0D >> apply a correct interleave weight policy.=0D >> 2. There will be a need for functionality to move nodes between differen= t tiers=0D >> or create new tiers to place such nodes for programming correct interlea= ve weights.=0D >> We are working on a patch to support it currently.=0D >=0D >Thanks! If we have such system, we will need this.=0D >=0D >> 3. For systems where each numa node is having different characteristics,= =0D >> a single node might end up existing in different memory tier, which woul= d be=0D >> equivalent to node based interleaving.=0D >=0D >No. A node can only exist in one memory tier.=0D =0D Sorry for the confusion what i meant was, if each node is having different = =0D characteristics, to program the memory tier weights correctly we need to pl= ace=0D each node in a separate tier of it's own. So each memory tier will contain= =0D only a single node and the solution would resemble node based interleaving.= =0D =0D >=0D >> On newer systems where all CXL memory from different devices under a=0D >> port are combined to form single numa node, this scenario might be=0D >> applicable.=0D >=0D >You mean the different memory ranges of a NUMA node may have different=0D >performance? I don't think that we can deal with this.=0D =0D Example Configuration: On a server that we are using now, four different=0D CXL cards are combined to form a single NUMA node and two other cards are=0D exposed as two individual numa nodes.=0D So if we have the ability to combine multiple CXL memory ranges to a=0D single NUMA node the number of NUMA nodes in the system would potentially=0D decrease even if we can't combine the entire range to form a single node.=0D =0D >=0D >> 4. Users may need to keep track of different memory tiers and what nodes= are present=0D >> in each tier for invoking interleave policy.=0D >=0D >I don't think this is a con. With node based solution, you need to know=0D >your system too.=0D >=0D >>>=0D >>>> Could you elaborate on the 'get what you pay for' usecase you=0D >>>> mentioned?=0D >>>=0D >=0D >--=0D >Best Regards,=0D >Huang, Ying=0D --=0D Best Regards,=0D Ravi Jonnalagadda=