From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F0F3C64ED6 for ; Sun, 26 Feb 2023 05:33:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E935B6B0074; Sun, 26 Feb 2023 00:33:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E43CB6B0075; Sun, 26 Feb 2023 00:33:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D323D6B0078; Sun, 26 Feb 2023 00:33:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C50416B0074 for ; Sun, 26 Feb 2023 00:33:01 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A06F4120795 for ; Sun, 26 Feb 2023 05:33:01 +0000 (UTC) X-FDA: 80508324162.19.CC2993D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf10.hostedemail.com (Postfix) with ESMTP id F2939C0004 for ; Sun, 26 Feb 2023 05:32:59 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b="XZw/G//r"; spf=pass (imf10.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677389580; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PmuOF086m76vOIcwOcdUOik6UrEnMOJE0zBgEJ2YwbI=; b=5emN8CXzHZPKLh4xohMEfk+S9BKbqK6/N45wRm2WHC/QchMfmNtb7bR/qPvyPaDFa3MZRP +ywAuJObPWl06Get7Y+frZBKy+W6abnKyLASzfjBwzjdequ4nobrWPFR9t6B8GI8xC5uoP +zFuqiSw32iZKpVG+T95lorc1w+S53I= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b="XZw/G//r"; spf=pass (imf10.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677389580; a=rsa-sha256; cv=none; b=2EaaIyrphFQ+t6cBUNycmEYzbAe08+5Ft4JlSAxiARxs0jHeRiu+0r7RldKqAlOEIFX5jO 1G3OUQ2USXghqXVfen8jsy/AZYzz0dnkUXBQxZCrYkDFDHFZWP6ynch7I0Obl7ul0NUZa9 ZSFVYs0KCx/FzVwQUNIuagIU4i3KGc0= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2915E60BEB; Sun, 26 Feb 2023 05:32:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 673C2C433EF; Sun, 26 Feb 2023 05:32:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1677389578; bh=KKM1ZrLzii5xPBfX+EiAo3IMRvet3fzMbWHegv9LWiA=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=XZw/G//rOISP67m6YxwiKV43v0xiq6rvyjZA1H+4Rc2Vc3avd8OVJ6fJS3kKVJU28 5uk3D2OaYcMj3ZqMdNVAZyolCNy7JwndKijR0UdXn4Q22gnNC5KEigZoHy1XFW5idU DtVye/4yukFpxf6o1gAoVBxrbkTXEZIbsXcl49J0= Date: Sat, 25 Feb 2023 21:32:57 -0800 From: Andrew Morton To: y.liu@naruida.com Cc: bugzilla-daemon@kernel.org, linux-mm@kvack.org Subject: Re: [Bug 217022] New: Extremely Slow Hugepage Allocation Message-Id: <20230225213257.25f1f584bc73436dda42b62e@linux-foundation.org> In-Reply-To: References: X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F2939C0004 X-Stat-Signature: c8dfjhkjnp6r7dpfjm3z4pb3cnhiuubs X-HE-Tag: 1677389579-138332 X-HE-Meta: U2FsdGVkX19KiTo2F3iU8+KWt/tC7kRsXqM7X7H5knXi32/XasFlzvJ0nEVBGwCn27em6MQYvduTsyPAqj5ChPj1lj17djU+vV7Ue2n0DtkSRGgDHX/v+oI5Z54ZpALxIrQph1bjw2dTJRf10Ni8XqXyCvEG16jIr0X70C+tFW+MJu1V0GaNYSaf7e+UKW3dP3i0TNkpkeRXTU7xYQaIR3Dj8dvbXcXa8oAuQN24TLsbBUtA5RXCDQyRhr+HyCDQdRIhK14Q8EtgXL2X+tHeiVGFosQGWXICNHmxBk9FSag9aZjvMBzW7iebRKq9M/6JhRQXvMzgxG+BhOiKcQP8B+LFS+amTxcXON9S9QanFmFCpeIJBL8eEbB78isF1kFOLMZICy527zrZDz+jfD03ZPPvy/ivAPTyt5lrnvi7srcgO75vcj0F/kp0YD1sWD41eLAb1sxyKOktkVoo06L3MyV/Uo5qSWv3hDXyaubZqC7138yphHjS5Cx7qKHsuGK+goObHf/wgVrK9gxGv+jDse9ptwf8UrsXXFVQVCTpTT3oiG7k/dlYaN8pV6rUoH05jHWjUXtEds0oN5FSmChgqyzqKmKuJEqlndG2ePmcbokmpceoplVkXlq90owUUX5EfcK1x6ru7hZsA8RTWrhtAvcuUyMW1Ib2ZXeEyYLzhdU7SAhklVmAe4YPDs5QEUfdZtYvKlWJlbySOdzcymCpXInDL4dpO+QuZdYtAa84kLyGCmJvbFGOn73gyVJCa8nbyS/e1x6IlwTCOjtJIEHwD0XNz4av6lTFSshgAOfDKEZgZrSny15WBdMKD1IfS7QDjqX4n3jy4ekVbZ8kTkOKk4W/iUWDNFqGd/fafbSJjeID93H+vgrnr76TWy8CcGS49F+NEmJ7r+NPTPnPjCzfIOJ3xmoOlxCTv6ocO68HqSD2l+RgjwOKwHwwIy8H1hkWymVsnwZvXATBcAqVqIg lNjWYuQN t1eTY+vpuFFTJVFxuU+p8zndNDLizhQCVw9SjmWqobqImVaggsD9ZvcCWOo6FVTN2AC8HTbxT5GE2AwShaRkRuEVpnsl1fPd/sJcgT9P1KecoaSl5YdpkAax5IpUSF4EKysiD9rgRA9cS2hqLUrYJl4kb4rCYL3NkjjWCcx6Qk+SZzJgKePapJ7hQEg/HOA2p8OxB4OHWhT0H+q9wAKQFwu1nMpaHpTMbDlj2u79ujFsymgzirHp4PP1boCPtHYtJ0RRBaZdlQna0sz2hMInSva1bZjP6MG6WSL9XzJbWz/lMK/6CLyviGMbYDkK1SG+jN9fSDP9+Cn5oos7m/I+fyCQfw8dx02/syw/NAHxPrydCL8/RInjpRiMXd/KOG8p7PIzRjl+I1Kg6hwAkr1QL0y01soowEIZDJ0D/qHDm3rfxi+V7L2f9Eh6y9oG3/YiSzLhW64cvsdQXMo3lz6OuDnYK3BSpniQHRZn2Op0nO88RKaSMFRbBlbDH3QLnHmM7m2G+nNsLZQcyRfc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Sat, 11 Feb 2023 06:54:45 +0000 bugzilla-daemon@kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=217022 > > Bug ID: 217022 > Summary: Extremely Slow Hugepage Allocation > Product: Memory Management > Version: 2.5 > Kernel Version: 5.15 > Hardware: All > OS: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Page Allocator > Assignee: akpm@linux-foundation.org > Reporter: y.liu@naruida.com > Regression: No > > Created attachment 303713 > --> https://bugzilla.kernel.org/attachment.cgi?id=303713&action=edit > perf data and kernel config > > We have some ICE lake server with 1TB memory installed. They were all running > at 5.10.x branch LTS kernel and run fine. After we upgraded kernel to 5.15.x > LTS kernel, the booting process were extremely slow. After some analysis, we > realized that was caused by hugepage allocation. Our system used sysctl.conf to > allocate hugepage at the boot time. In fact, "echo 960 > nr_hugepages" had the > same effect. The only way to do a fast allocation is to use boot cmd option: > "default_hugepagesz=1G hugepages=960". > > Our System is Xeon W3375 with 1TB memory installed. But this bug also occured > with Xeon 8180 with 1.5TB memory too. Our OS is Gentoo Linux. With 5.10.x, the > allocation speed is around 300GB/s, and 5.15.x only had 30GB/s. We also tried > 6.1.1, it is the same as 5.15.x . > > We compiled 5.10.163 and 5.15.88 with debug option and used "perf -a -g sleep > 2" to catch kernel functions. Here are two perf outputs. I hope this can help. > > -- > You may reply to this email to add a comment. > > You are receiving this mail because: > You are the assignee for the bug.