From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB8E3C54E64 for ; Thu, 28 Mar 2024 07:54:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFC3A6B0083; Thu, 28 Mar 2024 03:54:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAD376B0085; Thu, 28 Mar 2024 03:54:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D75B26B0087; Thu, 28 Mar 2024 03:54:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B85E96B0083 for ; Thu, 28 Mar 2024 03:54:27 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2D9111C1292 for ; Thu, 28 Mar 2024 07:54:27 +0000 (UTC) X-FDA: 81945685374.29.E878A39 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by imf02.hostedemail.com (Postfix) with ESMTP id 95C2680007 for ; Thu, 28 Mar 2024 07:54:24 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="F0s/OWuS"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711612465; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QbC5cWd3WAP709PDHIqZn6WUv4KT91ju291e26yekHU=; b=8ADfIgs1OdHKh34MvisbuxuO/UiL4B5Z7H4q7anm7qFMKS1zzJve9XWxIG4c1ZZRbSKT3f TLcQMKaYKSq9D1t8DVyOIQiUqyWwUJYfoblOL2AnWgMC3+cqtMEJEtoWnWgvAiMGZ/KY26 cd8diuO6/0o+BjDo7/Tbz9k6EmIgIwk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="F0s/OWuS"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of horenchuang@bytedance.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=horenchuang@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711612465; a=rsa-sha256; cv=none; b=sO4v+Gl0TWLRUPFovwIumezcl28wkr6Svd09CoCH9wAb+HLQXoCfB8y+HnBBg/hgWceghi evVTDsDp63EGwWlkNaSAepJ2taK3Z3XUMxHDK5kg5BoXlChZvN4gXGRJe9GdXGqqe6R5us ZnysrT+91L6vwGCVEe++tucByICvnAw= Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-dcc71031680so604062276.2 for ; Thu, 28 Mar 2024 00:54:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1711612463; x=1712217263; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=QbC5cWd3WAP709PDHIqZn6WUv4KT91ju291e26yekHU=; b=F0s/OWuSveTjzrrtGUKVkw6rcQHYKgimhuys3ASbntVKaTqbwAFbT4EcbFbNYKb3MU q1hs7sPSw21GBf/esZRlSKx0l9KxfLphc8ydtkiFFCjaM3iLrcq4cmG/Knet+QsNa6ew lC2znXyiDgsXsK8h3tiSn6Hwq8lYxz9Us6VI4YoJn1e77DGrAn6ElFD7E1uloovbK0sr q52t7jdhCM+gCo8Hz+IfZFTjXEPxfGRxwMiPj52K/Fmre3zrg+UaJjAjxVHNUo9zw9Oe B3hYFT6XrOKn0no5ZYyGU1OuxHfhVC5xjIBtHGs4du7YgYlLWsSX9Y+PB49gDG3qDMpY J5Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711612463; x=1712217263; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QbC5cWd3WAP709PDHIqZn6WUv4KT91ju291e26yekHU=; b=AXu/8mWrxZ0Q+gveVc+71huan1yaEGCac+nbXMp+nHEiroVuDa1IbQz91kNT0JyesY x1L2KeTFKqh9UwgrD1ZGVW5eNnQLYOKOn5lhP4qIS4W9YeGQcg0yw2G06PSTg8EHujAF 9pTEXrdqpnqhVuLZqJIfSk2FsW2EDNnfkfCaWT9RTXSIR2zCes/BrlZG+qQvQ1LGx4UX YhyzqVx+OZCrAiY3E7Luf6tlVtoVrBDBg3mfY70AKEx7z35WZjhPlkyIFFdFvSGO0TCa sbcCM7qSxYaa1etgqiqqblexdEp2Y8wCCdRV0phUMr12U96eZyL98PER2MjvJGuvA1pt AIkQ== X-Forwarded-Encrypted: i=1; AJvYcCVk0JMFDW3+iy9Ic1El5/eUsANBRNmuuwgXBcDIQE6IQ3bjOPB3Yb5K0t8g5WOm8fAPtRdoFp3+x6OUeyVXWsAGERo= X-Gm-Message-State: AOJu0YygluPnzzzkzrz035WkoHPQGFWluglffuNBjDT6BOaVZU5yFHV1 inYyuNUFsQmITaSVzH3pZQPRSs+a3PkvXXCNQcjq7V2mIGKTWLRhSayC7LaLXStMXfiuit7I9YZ qQaqM2b5SGnkoZxl6/MAKr2a+kiesXuR8svs4Uw== X-Google-Smtp-Source: AGHT+IEdkLQhpRPn4k1IyjPDs5Vv2uuU8cYVmBxxxsVzJXO5QupQAGy5BtVXsboyybOzZeHVOtwZZhZHHoOjBTG7fIw= X-Received: by 2002:a25:ab26:0:b0:dc6:a223:bb3b with SMTP id u35-20020a25ab26000000b00dc6a223bb3bmr2099954ybi.46.1711612463662; Thu, 28 Mar 2024 00:54:23 -0700 (PDT) MIME-Version: 1.0 References: <20240327072729.3381685-1-horenchuang@bytedance.com> <20240327072729.3381685-3-horenchuang@bytedance.com> <87v857kujp.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87v857kujp.fsf@yhuang6-desk2.ccr.corp.intel.com> From: "Ho-Ren (Jack) Chuang" Date: Thu, 28 Mar 2024 00:54:13 -0700 Message-ID: Subject: Re: [External] Re: [PATCH v6 2/2] memory tier: create CPUless memory tiers after obtaining HMAT info To: "Huang, Ying" Cc: Gregory Price , aneesh.kumar@linux.ibm.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, Eishan Mirakhur , Vinicius Tavares Petrucci , Ravis OpenSrc , Alistair Popple , Srinivasulu Thanneeru , Dan Williams , Vishal Verma , Dave Jiang , Andrew Morton , nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , qemu-devel@nongnu.org, Hao Xiang Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: 6u4cti1dkezg8xs4dozuz3iihd58dkzf X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 95C2680007 X-HE-Tag: 1711612464-391842 X-HE-Meta: U2FsdGVkX1+Y2kfgYky8zBXcOUKFdl0shgGmLiImQqdHewEmFnEtowMHeg9//No/GXbdBwwEoXzByJuMpEx/HBb0TmRsQY3B1tMDITeMkJ7EVzJwXbMsO99UlDaa6WjnfNh0l1Uo+x4EDJHX5qA6Ds47gmsdxrRaGCPZggjNHq4I1Z7IZOA3GAFomLj4QowgJYyVlEUyvAnGERK6L9ptVLs0YsxMfLr3ye7mO+8CgIa856+bTRUG2oiGDEL9L/fkbx/wvV1M0d2KOHmXW1/kxoS4ZEVEUrkZXaD0ltY5eNU6nIyClS/LK7kgJNOKNScOL4Hn5bR0stJOGsyIS2Gs9pitwOLvCgTTnHX/KtJvvuamLKc7EYi6AKcgzdfuaZNWtt5qvr70DRH6lY03rhXkO25l8C1QQnlYCVQJ0TFbaZI0mjQophAerFLsB3GRHH1GcXxXRqFdFepYQbVxu1KBuBFOHRrqtZLsrEDQT2DomTWllwluwjsoKcsSs4a07Ee3a4+zFvL3U21ucnivJULfjH62cSkofVtCFYd9EAF69oAOflflWbQpqcVOYatWGzU5Fg+w3rn32SbhiD/OK919dj5JtZ9w+/HVffo7vOEeq4LA3DTjud+UAbg5xXvSvIG8yVDDtnmmX0SeqYKPgh1vQmiMf0ZpKy/P0tUc23VHBpnmtfOYaEzpC8qz4USz1JsAOzIsriKoAnJDFgidnkYPxhmtFkfZDHTJzI4/2naVVAo238tV5rjbOjxYblSf0+W9/dDB6G7D1mLKprDgEH/d4LpR1GThxKj0gL09qAH165UUdOS53z7CTXMo+VyARWRiQVfWAz/Zl02tULJJlw1Bf53Iwa00q0weAX8wWSuLmyyABjJ5irFo9/Rga6+/+XObGkTcy8d7N/0MzX8bdzQQlWsGQgYa/QWRNeFUrD8/rXz44J3oLh7iy8xWD4KvUK9VLnstNzVB0T0lYJmKPs3 M15ERJuC QrxEDkmCnA7+VaVuCZDL32p4UYCoc/Qjr0JOZ21Iz/w4AC0a9i3VKgoHY/aTog56yz+zLweFnCurmzH+M4mvvJWK2lwcxymgrh1Gc1RUl4Eok9M5JSv33AM6jCDELfds6QNHm8tmZIgTB+kN6lzoiQitYFj8wo3yvG8pWYxJXVV3goN2uUmXgrwfdzT1ImCVA+mGEdwkG3V/0mGwPRtOnqsWxfQ5AX1kiU4pXqUvud1IHCLfHRfKnlv6wPhw16mfjzBpWBOams4sznwbbcOL7nhhCm2xWjhSd8lyYkyW340q2bBdtDABKnuE00FdJmjmw0WMtRFFoID1gBBtXJfSETfcQ7/yktQTkSx6qAEt9Nmk0uYG/EIq9vs1PqcsdjnICzbQBEZw1QDY+R3Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 27, 2024 at 6:37=E2=80=AFPM Huang, Ying = wrote: > > "Ho-Ren (Jack) Chuang" writes: > > [snip] > > > @@ -655,6 +672,34 @@ void mt_put_memory_types(struct list_head *memory_= types) > > } > > EXPORT_SYMBOL_GPL(mt_put_memory_types); > > > > +/* > > + * This is invoked via `late_initcall()` to initialize memory tiers fo= r > > + * CPU-less memory nodes after driver initialization, which is > > + * expected to provide `adistance` algorithms. > > + */ > > +static int __init memory_tier_late_init(void) > > +{ > > + int nid; > > + > > + mutex_lock(&memory_tier_lock); > > + for_each_node_state(nid, N_MEMORY) > > + if (!node_state(nid, N_CPU) && > > + node_memory_types[nid].memtype =3D=3D NULL) > > Think about this again. It seems that it is better to check > "node_memory_types[nid].memtype =3D=3D NULL" only here. Because for all > node with N_CPU in memory_tier_init(), "node_memory_types[nid].memtype" > will be !NULL. And it's possible (in theory) that some nodes becomes > "node_state(nid, N_CPU) =3D=3D true" between memory_tier_init() and > memory_tier_late_init(). > > Otherwise, Looks good to me. Feel free to add > > Reviewed-by: "Huang, Ying" > > in the future version. > Thank you Huang, Ying for your endorsement and the feedback you've been giving! > > + /* > > + * Some device drivers may have initialized memor= y tiers > > + * between `memory_tier_init()` and `memory_tier_= late_init()`, > > + * potentially bringing online memory nodes and > > + * configuring memory tiers. Exclude them here. > > + */ > > + set_node_memory_tier(nid); > > + > > + establish_demotion_targets(); > > + mutex_unlock(&memory_tier_lock); > > + > > + return 0; > > +} > > +late_initcall(memory_tier_late_init); > > + > > [snip] > > -- > Best Regards, > Huang, Ying --=20 Best regards, Ho-Ren (Jack) Chuang =E8=8E=8A=E8=B3=80=E4=BB=BB