From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30FC3C5B549 for ; Wed, 4 Jun 2025 15:57:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C22018D0028; Wed, 4 Jun 2025 11:57:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD93C8D0007; Wed, 4 Jun 2025 11:57:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC1BE8D0028; Wed, 4 Jun 2025 11:57:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8DCC28D0007 for ; Wed, 4 Jun 2025 11:57:49 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3E04E5FEAA for ; Wed, 4 Jun 2025 15:57:49 +0000 (UTC) X-FDA: 83518173858.27.CB74C37 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf29.hostedemail.com (Postfix) with ESMTP id B539C12000F for ; Wed, 4 Jun 2025 15:57:46 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=dqNBvphO; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf29.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749052666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pz+EnW/D+yNH/lAm96nc0b+2F4MOWXDHQcHk92rc5NI=; b=RHpLMrVqcDI3m0Hj9gpUp8LiTI5AsyqYdlS+GLqBUpjLOGH2e4OykWGa4pajKDlN1BwqZP /f+TFKhaB1CYofL5giuLez/uUgzHIbO/6MDlowd2hLuGj8F12hxHvSZoufTtfl9eaZLbQ+ JrD7/9LPQR9+D3ddPKTznTJ+25TAYN0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=dqNBvphO; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf29.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749052666; a=rsa-sha256; cv=none; b=z6da2kV2nasyYHo6HQQ3hAD447EfOiHCGbh/GVYs2hNsIgksnJtMEpaSDxxaUeTyHrhFlw ty7blen2/u6Gc4NB4Nh/+FZrn03271U3TOcPzKrVUC5V+BQTXn3ZWv/MVluIyHX0Fqcuet VrCYTiLZhOT4OET4qwKu0nycHefhWh8= Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 554DGuZV024325; Wed, 4 Jun 2025 15:57:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=pz+EnW /D+yNH/lAm96nc0b+2F4MOWXDHQcHk92rc5NI=; b=dqNBvphOMDWXYrspYqvDIU wWMPFfxEUHSzA8DXllz4DhS/jH0jNd+ERl2YDnd6sUl7QMh1TZ4FN3A0MqQr8N4U rYMiArAf7OQtx2KDk+VWIHGlFwhMAeXxBKE4wsq/4vpoOglJTc1NFWrfcZPy1TfE jut00VjMu/VJigv0zu8TPTA2TYj3jcQo0EY2u9tHCMCFAXTsm80oWisj238DVUp8 rsnN1TJjguzmWEoPrlGgaKTEVXfOaj+CdSVp0VksGdrrB9t2171NDSG7GZN7NG09 RkOSTTo82mNlIw2BIkbO+9p5oLSOOez+rx3WV2OPf7CSMmpw/G6TgUjONvj+QWVw == Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 471geyurxh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 15:57:37 +0000 (GMT) Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 554FYtZY026153; Wed, 4 Jun 2025 15:57:36 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 471geyurxe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 15:57:36 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 554Ff1jE028437; Wed, 4 Jun 2025 15:57:35 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([172.16.1.74]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 470eakg9je-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 15:57:35 +0000 Received: from smtpav05.dal12v.mail.ibm.com (smtpav05.dal12v.mail.ibm.com [10.241.53.104]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 554FvXX623593548 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 4 Jun 2025 15:57:33 GMT Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6828D58067; Wed, 4 Jun 2025 15:57:33 +0000 (GMT) Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8FA2F58052; Wed, 4 Jun 2025 15:57:27 +0000 (GMT) Received: from [9.39.21.166] (unknown [9.39.21.166]) by smtpav05.dal12v.mail.ibm.com (Postfix) with ESMTP; Wed, 4 Jun 2025 15:57:27 +0000 (GMT) Message-ID: <3d28858f-4ec6-43ea-8a3b-b9ce9a27bac7@linux.ibm.com> Date: Wed, 4 Jun 2025 21:27:25 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7 1/5] drivers/base/node: Optimize memory block registration to reduce boot time To: David Hildenbrand , Andrew Morton Cc: Mike Rapoport , Oscar Salvador , Zi Yan , Greg Kroah-Hartman , Ritesh Harjani , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Cameron , Alison Schofield , Yury Norov , Dave Jiang , Madhavan Srinivasan , Nilay Shroff , linuxppc-dev@lists.ozlabs.org References: <2a0a05c2dffc62a742bf1dd030098be4ce99be28.1748452241.git.donettom@linux.ibm.com> <20250603200729.b7581e017e4ca63f502c795e@linux-foundation.org> <9f7ae0e6-4640-418d-a4db-dba594377ac2@linux.ibm.com> <8abecd5b-2768-49d0-afc3-561b95d77a24@redhat.com> Content-Language: en-US From: Donet Tom In-Reply-To: <8abecd5b-2768-49d0-afc3-561b95d77a24@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=Pq2TbxM3 c=1 sm=1 tr=0 ts=68406cf1 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=IkcTkHD0fZMA:10 a=6IFa9wvqVegA:10 a=VnNF1IyMAAAA:8 a=mxs0qDoQSYin1jJoqQcA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNjA0MDExNyBTYWx0ZWRfX8WJCyVYxYOx4 ZwXF2L0hT9TVa9DCSZd0HnDTGnBLFWEQfIxdH7zNVM7GUVTw9QNAEgXFvI/Xf2VFj702xdpkOwZ sfqoXfWav9Rh6Q7PyjfPbaOH7gSnB9W1ScAh5rm/TsIwa5ei31ohpf9YVdbG04aMpO/YATwLmKc Nu4l7OGkwJONLlda7cIGo5xN4tNoU5UWs7aWuldP0AxM+oI6kN/2TY/JBbQz7cKY93NcxIWW8tW WgHpy0zclU/Ytt607VyE1rKfvJD5sP9EHB1ztXnYinqH583LpBYXATXCdNBDhJPqPJ8R/vxLle1 avaH5sAL+9Pjua1nGH/CFx6EpV7DjmpkhInXsPuhnQOvipMOn0uGoH244A9l8nkKf4KiAHjDxo9 o+X4w74JpOyhMp51jOolPmtOZntcZfPPKt8xNLN8FXCuDhvXTNLasu1gWBqP7xtuOE954ekS X-Proofpoint-GUID: 9l28POHsiWfHdlg-x-L2inqY-xBTAJNz X-Proofpoint-ORIG-GUID: d6boryrX-Nv2vvbBUaTJBWLYcMY3GX1l X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.736,FMLib:17.12.80.40 definitions=2025-06-04_03,2025-06-03_02,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 clxscore=1015 malwarescore=0 mlxlogscore=999 phishscore=0 bulkscore=0 spamscore=0 suspectscore=0 priorityscore=1501 mlxscore=0 adultscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2505280000 definitions=main-2506040117 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B539C12000F X-Stat-Signature: u5g9wpd7y1zfm4zprqx6eg9yp15ga5nk X-Rspam-User: X-HE-Tag: 1749052666-983009 X-HE-Meta: U2FsdGVkX18JPFQiuVUi+pcKuwz15B0qSgX4dOZFrhhBxZLPr/gGmfNxktB4tiMc0KmjQsxmlIHHyP5tw/+KjAYeO10Kxrn+z3TyBDCy7oJR5Ij8KOngLE7Fa0ccnc1Q0MOvR6DVjBZE75zaQugKxv25mmq7vR/OVrRH/rh7zyNHvaxgYyaQoXMNKdm0gjuXhIdfVjXD9tgUKlcWEnKLb7TfwdY2SbW5w0OBCauxM/L0+XjebmNaAteZIr0g4NdYn0Zpopx4Hu0X+SOfwrSxKsDkw2ia27uv7VwTHeVnJVPn8tCfEZhHTj3hPqN3Y9FGwRh7GtyVehuzoRQgDX4LENy+cEZgCrjTI9hj6e6yxCvGDIzS75F5WfpxNGzDoRfWlMcxh9wtUBmCybYRQ0VoUP7NrRpnJzRb7a6K+U8JMSmbXz/YFFR8B+3uDqNITeyQlAvCxuk5RsnEOj+7S+mSY9VdrQe3T2uKw7fyW8gIMMdzFsEal+cK9tcvd8HkE4mnse4kMimiPTWc9Jmoi2miYdu2HbwzW3P4xjNiL2+n/PnIDEqxsh8u9TnY3i9h47sxiB1atlahNes3WjUs4bswDbix2Yt3GPs+5U78unxcovjewBUqvSzTABgwd5xOl77JoMn+EUcszIugiuNv979sUlD0zvWYJxMJcj7hiIcgkoWaustWHMSQQyiofpHgWOq36IG18ok/xvzWvoaqucBUVnU6InyojAmPPZs/lSXt9fhr4nKBVjW3A8Nn7tJKNVYGCWXmN3rWTVSw7nK/M2Eh3Ylq19KOwZ50DtaA0QYg2AMoLLgKOJW448bc9aLNiO268UjL9IIn3gSVmCsXmAOjhTI+BK9hEBI13huj9CwAW2MYjRyuyT76iuS6psSTwixVpUmFQVwX7jSQ1yOm2cVrf8onRwfIuSNjUvJHxOBb33MRIL0dT/1dzayWZ/bphS2grRKE9JlBQRHCEI7JNux CDSwL08E Wi/370zzjMXS1ZftLNBD1fJwq6aghkHALPlrgiCB/BccYLeuevaMu81+jERYgUgf3UCs6KKYXrDbwAkpBZbsTtQt6NQv7bVHETlY0f7ONHxlpS4y/+4qgSq+DxkXn7NhQrQXRVvTL3s6ptg4E4DE1UpRGwYAh3cj65B8F+4heHtZNHEHCg/t+HYCR6z5H+K+bI+yUyV1OAIGC3ul7NFiskaybhsqlgp8dHfyNWmMK3Sre1nP8lTaM15MmvbH1ByHvOKvp4aKwxHkF5etDg+ybvzOizlXLLx4SwNgA6aY9CntTgPw+nrwFqRcPp8Lswg+tpr2HKEuUN/qhyVLarkk/0W3BwTogu2KQb/9BmdNlMhMP7koAEIojoBHxfs2SqWHW+3/HoRKxtFJWITD+Me1PPXHFEzUmbrzVRcD5oPMskoldLU1sS8Ye8WBcpekTYP9Tb78H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/4/25 7:00 PM, David Hildenbrand wrote: > On 04.06.25 15:17, Donet Tom wrote: >> >> On 6/4/25 3:15 PM, David Hildenbrand wrote: >>> On 04.06.25 05:07, Andrew Morton wrote: >>>> On Wed, 28 May 2025 12:18:00 -0500 Donet Tom >>>> wrote: >>>> >>>>> During node device initialization, `memory blocks` are registered >>>>> under >>>>> each NUMA node. The `memory blocks` to be registered are identified >>>>> using >>>>> the node’s start and end PFNs, which are obtained from the node's >>>>> pg_data >>>> >>>> It's quite unconventional to omit the [0/N] changelog.  This omission >>>> somewhat messed up my processes so I added a one-liner to this. >>>> >>> >>> Yeah, I was assuming that I simply did not get cc'ed on the cover >>> letter, but there is actually none. >>> >>> Donet please add that in the future. git can do this using >>> --cover-letter. >> >> Sure, >> >> I will add cover letter in next revision. >> >> >>> >>>>> >>>>> ... >>>>> >>>>> Test Results on My system with 32TB RAM >>>>> ======================================= >>>>> 1. Boot time with CONFIG_DEFERRED_STRUCT_PAGE_INIT enabled. >>>>> >>>>> Without this patch >>>>> ------------------ >>>>> Startup finished in 1min 16.528s (kernel) >>>>> >>>>> With this patch >>>>> --------------- >>>>> Startup finished in 17.236s (kernel) - 78% Improvement >>>> >>>> Well someone is in for a nice surprise. >>>> >>>>> 2. Boot time with CONFIG_DEFERRED_STRUCT_PAGE_INIT disabled. >>>>> >>>>> Without this patch >>>>> ------------------ >>>>> Startup finished in 28.320s (kernel) >>>> >>>> what.  CONFIG_DEFERRED_STRUCT_PAGE_INIT is supposed to make bootup >>>> faster. >>> >>> Right, that's weird. Especially that it is still slower after these >>> changes. >>> >>> CONFIG_DEFERRED_STRUCT_PAGE_INIT should be initializing in parallel >>> which ... should be faster. >>> >>> @Donet, how many CPUs and nodes does your system have? Can you >>> identify what is taking longer than without >>> CONFIG_DEFERRED_STRUCT_PAGE_INIT? >> >> >> >> My system has, >> >> CPU      - 1528 > > Holy cow. > > Pure speculation: are we parallelizing *too much* ? :) > > That's ~95 CPUs per node on average. yes > > Staring at deferred_init_memmap(), we do have > >     max_threads = deferred_page_init_max_threads(cpumask); > > And that calls cpumask_weight(), essentially using all CPUs on the node. > > ... not sure what exactly happens if there are no CPUs for a node. Okay. I'm still debugging what's happening. I'll update you once I find something. > >> Node     - 16 > > Are any of these memory-less? No, there are no memory-less nodes. All nodes have around 2 TB of memory. > >> Memory - 31TB > > >