From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6DBAC282EC for ; Thu, 13 Mar 2025 16:48:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF0AC280002; Thu, 13 Mar 2025 12:48:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA279280001; Thu, 13 Mar 2025 12:48:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91B01280002; Thu, 13 Mar 2025 12:48:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 705FF280001 for ; Thu, 13 Mar 2025 12:48:07 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 97190B1E5A for ; Thu, 13 Mar 2025 16:48:07 +0000 (UTC) X-FDA: 83217110214.19.4F84B52 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf11.hostedemail.com (Postfix) with ESMTP id A5CE940002 for ; Thu, 13 Mar 2025 16:48:05 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=qfPBdfHY; spf=pass (imf11.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.173 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741884485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6gWJHqAH2Ne74hFxjKFTiD545xNbJagX+7KeghzOK3g=; b=QwGnvBGJrKKVl7HiWdTnqI+BCk8gATZxtbByFFmAJDpxaUCpXxfkrgV9RfnRas9RcINOTJ DG/SoM/S8eqwu5ZUq6DMjDAR+e5w25EaUNUVddlo7Sj77afgXVuukMh2q5FfBhkcRCNglw ylUbA6witXiLt/f2jKbV4OWJa7jLwTs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=qfPBdfHY; spf=pass (imf11.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.173 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741884485; a=rsa-sha256; cv=none; b=VHQnkMrIvc4/RV8qyqjMjUv3ComRpnnoeJMKBQoQXtka4Ka2mfV1AspDRqLb8oTMneLaFQ 6QNAoTf0UbiuPNwwnNP6dWYl79nQ8gk5ktzrZEMbmOr180hjTs76he5IvY7uqh9xNaNXfg C0Yz73wodEGv5NzmVYLJL7PigVGe5Po= Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7be8f28172dso76858685a.3 for ; Thu, 13 Mar 2025 09:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1741884485; x=1742489285; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=6gWJHqAH2Ne74hFxjKFTiD545xNbJagX+7KeghzOK3g=; b=qfPBdfHY7RkWMODLhReTVLNaRmkN1gD742iV1IKNy4WWqfiXspd5qywvqPtAP80366 6wYBXUMZIeZ5wbp1+wVLXbH6GzywXLVXrkbk740bECIJt7zSLu6Mr3WUQORg/5IynM6/ hrl45MW2S9cNsolJpKOrO29qwOQJG5wr3s6OQxpW/EWVPYtuHph2cMNPi0spMNszV2gH 3riQfcFJBhrD5Ukxt028q6/s8J/rNPf2bHwxOa9dswQWt7PyUHoz38a+uw0NTzac609D FyzIDFxJesLJ3lcUNLysMS4+PqpAAA1D7rAZRZL9pR0v5IsMyXBhw1itgaEGIYu/hb5w RQkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741884485; x=1742489285; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=6gWJHqAH2Ne74hFxjKFTiD545xNbJagX+7KeghzOK3g=; b=iG/vKgmy4Z2yukf7PJIcr/X9PNafeejKqH2MfOaRtbLsi5x4tQSYYUaF3GlMmVQ9Mz Udr3T80lP6hISYsxE/tl6CyqV8AKY2Q9eSGad/Dsigx9tRVo/mAy+axeJD5reweUDD+B xdkrysF0UZuWcHZC3VDuVMpyIWphEwKbNx1CzyYaV/xrZzIdhVBhk/wnbF5gvF5KqS1B 4fNG+4oLvXDpaVRiVWlwFBE+RgsxEFlG3WR1+yzUwDilemhW3XzeeEYzFhf6BKot83hx jft6piy26jrD9cUOjsAjK4hwjXvQU+9LbA1sExdPsbjLlE+FAwFHrbtfE7uENyI/LUjw 2hnw== X-Forwarded-Encrypted: i=1; AJvYcCVVpdM2OPgttdYDTx26BvWMYzXBZB9CSkarDcPX5ucPkVLEA/wu0h9PHUdFXzUT7NpB+XQPPwoWHg==@kvack.org X-Gm-Message-State: AOJu0YyLbWdZlR/5EHbwm57x/KXLbmQO1IM0zMvPEthG8aNWSQmkEydV WwecFLPC70snrJEOn34l9lkCj2OfGVOtAg26m3YmwLZioHNYPBOawpDRNkUBFNc= X-Gm-Gg: ASbGncs33CaD4tWtLCWl19L6iGOPzucDkMSRUsyFT15x1xPPXlEoamOROtevCYA7GWx LIoLzGD1/IrwGlidOUEfhXAk9k5kZ2IuoRQdVYCxfEWpn1IvPwxdeGepzjVv2y83qMNA15N6Kvq B+AUbzJFiQcGle2Vs4ODMQc68aC7r2z/DFj24IOeJrdLEY/jW/2IDyGFznjpaoDq33Y9ZvU6U2T i+00ZRfXQ//0trTY2pl013ixIeNXYpTAoEcroXRauzesvR5NdPH9QgzeaEPLh5Yrxlq7Xprpwg/ /Cvrah5W1DYkFszNwR7Fs29kqM76y6/qPpKV53E0lQkfNsqQ57NywJNzqBqU4TENp5+p6sTyHbI KLJnFIThZvmSeRkl8Y7A/9d4kJQI= X-Google-Smtp-Source: AGHT+IHxF9ffTb10hBbkPjF4Gy9uvvCTgZFfxO5Mi/avamJ3LWPv9XNw4u9cLuY8UUZaXI1bajEgNQ== X-Received: by 2002:a05:620a:618c:b0:7c5:64a5:d17e with SMTP id af79cd13be357-7c579f1eebbmr83505385a.25.1741884484673; Thu, 13 Mar 2025 09:48:04 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c573c4f525sm118401585a.20.2025.03.13.09.48.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Mar 2025 09:48:04 -0700 (PDT) Date: Thu, 13 Mar 2025 12:48:02 -0400 From: Gregory Price To: Yuquan Wang Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [LSF/MM] CXL Boot to Bash - Section 4: Interleave Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: A5CE940002 X-Rspamd-Server: rspam03 X-Stat-Signature: qf8g59ij31pif6wzg7p1p4mnko83wnyt X-HE-Tag: 1741884485-664280 X-HE-Meta: U2FsdGVkX1+OmA4AQLiiwLmvCRjod5QPA3/Iu2fklJwd9RSXfhnpds51WVnheZ1M3W/pMAxaxGRDHWSxB01Ans2Mm0gcLRG+Zn9QrywEj1lKA6OqaK1c3pMGcY1eS451O6tWqaAa4K+fHevFbaejn4EQMcPwkCo9utUCJK4BpuyNxeIqrda7bW24T4rjiAEVcC1fIatI67d4N+VxRHx0MzJFsubKN0luu9cuSNUTUNH2a4sp4ZzNjWbmEg+dPSLqLpwNGk+uzn4IsWNj5ACEPXEDbxCIT0XAFQivW8b+bmF8J2R8CMApePz/3t8RT48L6CCqXAQu1CSpNk5a1S5zMhRkGX0D8ItaqfsPIfD+XPrNBHxxjvj+ZEn0hcA1bMuDJxIJzDiUC4G2ZoPfZyyoLCosRcAw0d6Z7rvTtq2OB1K2dQaVvQ+/1Ftja/dQBhVd0CMiimRz59gIcEjImjBiMAOCGquIWqwEABNdPt+ksgQDL0GXHXWWsaLpSp3ATKt5p7VtjHfMyufCIRC9KEpuzR6Z9yhol+onAzFkqVFY4WHLU098EVvwNfE1f/8tgmX/1us74EhjqESMjpttt8rzPjx04H8By+QL7cyyKm4WjEgHA6cal7NJ9eizVeJUiNMnTSCRT8K6GtD2RT/D3R3s6JUeoEp077+Jqy95r+WfkKksAgO/lsdIsazmmIJN7khzQ2ER4r0F0Srmpo7nTrTKDzZjnmwWCWZeGZFfLb7Wf47AZ7/HWXGJo9nJuzQglFa8tZs0vxtcNrhNaJDy3ZC1uhqtmPBvd9w6EO5j4dRePVwiwfI2/eKAIGJ2O3TjR4SiPTZMWVztYK4qtx/a4OgxeWTtAGDBQ01rJYqOh3hBKUnX2BdxEb5jjTBcuulvlCwnqZuRhyEj4DIga4m0icfzuSLFNibYV7wukky31wMINoLBlfnZL8zC+8POXYuYokol85HUkt9ny1jBPzn4wny ljkUdNkV qhRoP8V9j1GlfJLgPufPfxtFlUEaWz7Iyytwyt4ObuZHvKP9hAIWqBjsor4vlOcE/K1rKV1Vss8sZ+CijPOG5LcdryUDNc0vmFzfke3QmdabB+FgWq3AlSYC6uhhd5nighgmQKthEHlRZx9dc5b9mvasL4EDARSI2irDYbwbj15HvegcZAjDXpS+k0zJ2PUHMWSzbd1HYdhzOsbfSj+2T9OcbYA2j42TuUVb45/QVVC/PTYA+t3JgNNnELr/hnAGacAQ6aP2aPwFbCSVH0F7qdVWy1fmeIUY9UddeGftmfg8o9qeb39ntbNpLjsAc8W9VxGHuLTnjb6UOtsA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000104, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 13, 2025 at 04:31:31PM +0800, Yuquan Wang wrote: > On Tue, Mar 11, 2025 at 08:09:02PM -0400, Gregory Price wrote: > > Window size : 0000000080000000 <- 2GB > > I think is "Window size : 0000000100000000 <- 4GB" here. > Quite right. I am planning to migrate this all to a github somewhere after LSF for edits, so i'll take all the feedback and incorporate it then. > > There are two targets at each level. The host bridge has a granularity > > of 512 to capture its parent's ways and granularity (`2*256`). > > > > Each decoder is programmed with the total number of targets (4) and the > > overall granularity (256B). > > Is there any relationship between endpoints'decoder setup(IW&IG) and > others decoder? > I'm sure there's a mathematical relationship that dictates this up the heirarchy, but each endpoint decoder needs to be programmed with the same interleave weight and granularity of all other endpoints. Technically unbalanced configurations are possible, but Linux does not support them. > Hi, Gregory > > Seeing this, I have an assumption to discuss. > > If the same system uses tables like below: > > CFMWS: > Subtable Type : 01 [CXL Fixed Memory Window Structure] > Window base address : 0000000300000000 <- Memory Region > Window size : 0000000080000000 <- 2GB > First Target : 00000007 <- Host Bridge 7 > > Subtable Type : 01 [CXL Fixed Memory Window Structure] > Window base address : 0000000400000000 <- Memory Region > Window size : 0000000080000000 <- 2GB > First Target : 00000007 <- Host Bridge 7 > > SRAT: > Subtable Type : 01 [Memory Affinity] > Proximity Domain : 00000000 <- NUMA Node 0 > Base Address : 0000000300000000 <- Physical Memory Region > > Subtable Type : 01 [Memory Affinity] > Proximity Domain : 00000001 <- NUMA Node 1 > Base Address : 0000000400000000 <- Physical Memory Region > > > The first 2GB cxl memory region would locate at node0 with DRAM. > > NUMA Structure: > > --------- -------- | ---------- > | cpu0 |-----| DRAM |---|------------| Node 0 | > --------- -------- | / ---------- > | | /first 2GB > ------- | / ---------- > | HB0 |-----------------|------------| Node 1 | > ------- |second 2GB ---------- > / \ | > CXL Dev CXL Dev | > ``` > > Is above configuration and structure valid? > This is correct, the association between memory and numa node is pretty much purely logical. I'm not sure WHY you'd want to do this, but yeah you can do this (assuming you can get the BIOS to produce that SRAT). ~Gregory