From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E94C41535 for ; Tue, 12 Dec 2023 22:14:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2C156B03C3; Tue, 12 Dec 2023 17:14:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CDC1E6B03C5; Tue, 12 Dec 2023 17:14:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B304E6B03C6; Tue, 12 Dec 2023 17:14:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9C8996B03C3 for ; Tue, 12 Dec 2023 17:14:55 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 625A61C0EFD for ; Tue, 12 Dec 2023 22:14:55 +0000 (UTC) X-FDA: 81559572150.03.6D51331 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf14.hostedemail.com (Postfix) with ESMTP id 8B518100009 for ; Tue, 12 Dec 2023 22:14:53 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zHS2w5GX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of rientjes@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702419293; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ai/6oa+3xTjAhJGHry+JbJcf+dfhAKDUCjsENuNGuQQ=; b=wihADT+QzOcxkYipz/QyrwxbRYnVoxJFxQgAQTk4WYNtPXk7mbOy9sAD2o0gwYjI/ON0N+ kws9CsYtOZ1qcDVG2COdqniLlwR3g5nKYUccuNJ2EVnQM9Q0m9oCDSsi7ghx65pc/0/dab BRY376fsZYl+GNsIKP3915g/8ii3bsg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zHS2w5GX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of rientjes@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702419293; a=rsa-sha256; cv=none; b=4QqeBzsIHXpgTAu4tX/GfFndC3WVK9BBYOXZl1BYBXUyctaWaEfAd8kqdkpBx/XEYpM8D3 edeBL9emmsYa3DogorKJiRn8YhHeHG9Y9y7N/yxaZyWgRu1h1cUuXrH0+iw8V9DpkBc4iO 1x72kT5g2RnYJ7el7XZYaYPK462xeN8= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d0b40bb704so57675ad.0 for ; Tue, 12 Dec 2023 14:14:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1702419292; x=1703024092; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ai/6oa+3xTjAhJGHry+JbJcf+dfhAKDUCjsENuNGuQQ=; b=zHS2w5GXLqLl5hKhv+oyzqfZzJBptOF1AdkOk229W51lpmAzAQX6UVw/DjAIDylM3R 0bOwl2WlTbrFkbRt4HdeV6WLmrs/L/Xo15YrRFT+02OvYF6f/tK/NKzgUXuzgbpiYg+V IJfUKu+uVRYurJmIeD3+lNL0nIiBXPioBX9GeENTi+Ud1xESuuu6laaXvKXQhSpB3qv7 LCMfPao+K723+6UhawMYcGXrCCX7/IxucLbsy9akCtNVbu0f0ceOMDKNqqK1727aAyW7 nkos1rSp1NWzPPmc8yhqQZ7iuItDzYfZ5TQImj4QU0G/MjgJjXknBb19tP4d0XBR/R4B Zh4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702419292; x=1703024092; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ai/6oa+3xTjAhJGHry+JbJcf+dfhAKDUCjsENuNGuQQ=; b=nsu2qsNOJdDZxYe1TFKh5bAics9v+joI3tOPKlCJ4emjS3akbFRfaxCxk0aH6I8F+A PUXUkiCgWIDbwtTM3S/Tp1LivzynZWza+sO/X246gLq3sMK7bukUhvObXeMMGtuoJAnQ gLO+DHxhfxNpNRp4ICV29xZNG7uNqfPFp+5LcZDLZeW3cpGC0U5ST990ZfR3Y1MlskD0 qlEcgtx1zuC+vJmuFjPEqA1L0luLEuNpU3uzKDd9i0LAGEOb/qSihf4RF0h69kk3kmdr 3eFFyNTDEgFfblzj72nWtlyw75PC79Twvc0t7OEVbgCidW8ehcBGyx9YUpAAK+y3khmS SF9A== X-Gm-Message-State: AOJu0Yymv6Jk95R3AUp8AAXsP5P/SuS4dZGVzXylJoyyugp4wcaKNjGp kWzabBfqrMclh4pmw/q9FYqskw== X-Google-Smtp-Source: AGHT+IGwh/A1nELtAxUKj+TAxd3RqTHBSSMVjxaO/S4KOsZf7yBKwvkT9TAlTopzChwQot1tEfqt1Q== X-Received: by 2002:a17:902:dac2:b0:1cc:6fa6:fb4a with SMTP id q2-20020a170902dac200b001cc6fa6fb4amr962337plx.22.1702419292041; Tue, 12 Dec 2023 14:14:52 -0800 (PST) Received: from [2620:0:1008:15:fc81:e0ce:797b:3b4b] ([2620:0:1008:15:fc81:e0ce:797b:3b4b]) by smtp.gmail.com with ESMTPSA id e10-20020aa7980a000000b006cef6293132sm6394390pfl.101.2023.12.12.14.14.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 14:14:51 -0800 (PST) Date: Tue, 12 Dec 2023 14:14:49 -0800 (PST) From: David Rientjes To: Gang Li cc: David Hildenbrand , Mike Kravetz , Muchun Song , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com Subject: Re: [RFC PATCH v2 0/5] hugetlb: parallelize hugetlb page init on boot In-Reply-To: <20231208025240.4744-1-gang.li@linux.dev> Message-ID: <996ba32c-78f0-1807-5e64-af5841a820e7@google.com> References: <20231208025240.4744-1-gang.li@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8B518100009 X-Stat-Signature: cn4rtjwfiqjddgug7ctnhztqhf6ejqmt X-Rspam-User: X-HE-Tag: 1702419293-504227 X-HE-Meta: U2FsdGVkX1+rqkpSZiq4jz5so7gXs2fX/N+EiqA3Bwsz6PVlsN+me4LHJ+ra/PNLOSGio+qcf3uvWsupziHNAb8XwrsJS4Gqis47rFrh0jYyR8aRNz1pVaRP28iOq+npSO8+TPqfQeHTlKlaPUULLZd5nqINtWyWfsMzNmYg1wuNPlcGRbw/NDOOCHapo8qxfftXLc4RwKB5m9fC33utEV1SC1th3nA3CvBtsRVvTZBpkihCdbWdPJVNtGvFlay/Pjk3tAtzyKsO0FUzj2+TxbMEXPMsL/mhKQGXw7WAHby+mG6UmX53I0YrT8yLsOB0VraCgdm5kvy2UhQ0uw9su9v8Z7O1+/ZRgc/xR+KQRNNNw9ug4T6ka52XoPAcs96O4LqaEvsOfHqc+YFn301XavPkUsA0eX790YoptaPs3351cyiWhRONqpw/ppOfXyIf72WdJPlPE0hahGJ+oCNAC9+TctMgy9WQm2PZ2Vtt1quTGq5ceH4GC+8i1cLaMnhebn97lP4lvrSrXHi2moc7A7xwmGzWHgH+BMmQYd3AZZfhxRXpF5nl76FRhfb9jN+fEl2lkxGeEVTJnJYlNFqnvVZFhg09YwJ0p0mse/w3bNXAKpicO9hAu23rn3ietELuWlJCRdqFwrtN/khYM/tQLkZMkKTTbBM+yIkpyiOzB98mhhpOnV8MIFI3xwCWkO6yvAlEPe8u0Sn8D3nGuqVt9iH23Ayq0Ci1PbMQjoWmQ01AlGGXbZfBJURJ9oFZ1Ky77xs5uDJncMlZ19U5xTbLKrauffJDvrBBuhyoUdlYrps4dXPCikaacKxsteFEjtcT16VeXIzANfowMB8UI4i6TVs/j5lrIhFxiHUW46c4gS+eGMfyP+mkbMHbxJejcq40gQr9jQnub2rCEnv4SkATSvAWVRPG7Tq5s4n+pB6cnmGLawjlcXf4W3dnR9FvBM+qAMZTIf1cnZGzOjU1KK4 xbhiywb7 5Z4XUlZOjtICEQAvlqv8E2gUbEaCOfT7rVWwritQKHETHSankFW5Dj3knCAdX/cpx5mVO2fV9NOerbcbVC1bkoCoWQISkMGSomKHwymPXMlhXEfwDedLX5VHEsH8Xn24jTfBBGblmZkK8znGm4tXgoLLRHaEPK2F2QNDpYcT5N9/rL0pU2Ku2n3RTAynLKpOEVpVyVKiRs+g1NVGF2u/kMXVNtKjjvCbAQq3258EkTcATjYrqYXLLyrF646rpWs4ykj0LXdF/fowh0OWgCSKszDNe2C88jL2VXNRJipBytGcP3CsWzj3CB7bB2g82AE/jzh3G5dS8svzkG+GyyWKkZY5tpE8SKFJg1bo702UAV7Dccpa+bRazP8+KGnpXZupQinxDKWdX+JxpiG6VvpafOt5LQHETZvzmUIDY4AbQt8AL9ivWrMkRnFEL88hDaz50nMif1KrbJ2avy/Gvmfwodug3eW6gf65YEcJ5pUNBe3P95AhlUTlaHz9sVdjYd++PGynQC/F57lKKsYMH7vZ/NwRH5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, 8 Dec 2023, Gang Li wrote: > Hi all, hugetlb init parallelization has now been updated to v2. > > To David Hildenbrand: padata multithread utilities has been used to reduce > code complexity. > > To David Rientjes: The patch for measuring time will be separately included > in the reply. Please test during your free time, thanks. > I'd love to, but what kernel is this based on? :) I can't get this to apply to any kernels that I have recently benchmarked with. > # Introduction > Hugetlb initialization during boot takes up a considerable amount of time. > For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2 > seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel > host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot > time. This is a noteworthy figure. > > Inspired by [2] and [3], hugetlb initialization can also be accelerated > through parallelization. Kernel already has infrastructure like > padata_do_multithreaded, this patch uses it to achieve effective results > by minimal modifications. > > [1] https://lore.kernel.org/all/783f8bac-55b8-5b95-eb6a-11a583675000@google.com/ > [2] https://lore.kernel.org/all/20200527173608.2885243-1-daniel.m.jordan@oracle.com/ > [3] https://lore.kernel.org/all/20230906112605.2286994-1-usama.arif@bytedance.com/ > > # Test result > test no patch(ms) patched(ms) saved > ------------------- -------------- ------------- -------- > 256c2t(4 node) 2M 2624 956 63.57% > 256c2t(4 node) 1G 2679 1582 40.95% > 128c1t(2 node) 2M 1788 684 61.74% > 128c1t(2 node) 1G 3160 1618 48.80% > > # Change log > Changes in v2: > - Reduce complexity with `padata_do_multithreaded` > - Support 1G hugetlb > > v1: > - https://lore.kernel.org/all/20231123133036.68540-1-gang.li@linux.dev/ > - parallelize 2M hugetlb initialization with workqueue > > Gang Li (5): > hugetlb: code clean for hugetlb_hstate_alloc_pages > hugetlb: split hugetlb_hstate_alloc_pages > padata: dispatch works on different nodes > hugetlb: parallelize 2M hugetlb allocation and initialization > hugetlb: parallelize 1G hugetlb initialization > > include/linux/hugetlb.h | 2 +- > include/linux/padata.h | 2 + > kernel/padata.c | 8 +- > mm/hugetlb.c | 201 +++++++++++++++++++++++++++------------- > mm/mm_init.c | 1 + > 5 files changed, 148 insertions(+), 66 deletions(-) > > -- > 2.30.2 > >