From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B1FDC433E7 for ; Sat, 10 Oct 2020 06:11:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8B35B20732 for ; Sat, 10 Oct 2020 06:11:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aco4i24u" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B35B20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 955F46B005C; Sat, 10 Oct 2020 02:11:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DF088E0001; Sat, 10 Oct 2020 02:11:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A8226B0062; Sat, 10 Oct 2020 02:11:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id 4A3916B005C for ; Sat, 10 Oct 2020 02:11:35 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D35C8181AE867 for ; Sat, 10 Oct 2020 06:11:34 +0000 (UTC) X-FDA: 77354994108.26.skin97_420f739271e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 954E71804B66E for ; Sat, 10 Oct 2020 06:11:34 +0000 (UTC) X-HE-Tag: skin97_420f739271e7 X-Filterd-Recvd-Size: 5296 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Sat, 10 Oct 2020 06:11:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602310293; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IyGrTsuf+XbfRsOlZleQ8COsqo+RgEPBV6+boXBhvfA=; b=aco4i24upY52szACRORitb5ZYJflYxEqzQSQBmIpF3x+yzP/5+6wkxmXvS6HQybD/qO/yZ 7MrfZeQQhjkcpKjK2NcGBumMqWBhu5WBk+zGMc3Q7AAepFqSI+UdxU46KVDknR/S1nvUMb XjQ778oWLPpPfTdo76vaNg2kAWTNx8M= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-107-pc0cwYiJO-e5uiA-NDu72g-1; Sat, 10 Oct 2020 02:11:30 -0400 X-MC-Unique: pc0cwYiJO-e5uiA-NDu72g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3DEFF186DD23; Sat, 10 Oct 2020 06:11:28 +0000 (UTC) Received: from localhost (ovpn-12-89.pek2.redhat.com [10.72.12.89]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 19F4D10013C1; Sat, 10 Oct 2020 06:11:26 +0000 (UTC) Date: Sat, 10 Oct 2020 14:11:24 +0800 From: "bhe@redhat.com" To: Rahul Gopakumar Cc: "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" , "natechancellor@gmail.com" , "ndesaulniers@google.com" , "clang-built-linux@googlegroups.com" , "rostedt@goodmis.org" , Rajender M , Yiu Cho Lau , Peter Jonasson , Venkatesh Rajaram Subject: Re: Performance regressions in "boot_time" tests in Linux 5.8 Kernel Message-ID: <20201010061124.GE25604@MiWiFi-R3L-srv> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/09/20 at 01:15pm, Rahul Gopakumar wrote: > As part of VMware's performance regression testing for Linux Kernel > upstream releases,=C2=A0we identified boot time increase when comparing > Linux 5.8 kernel against Linux 5.7 kernel.=C2=A0Increase in boot time i= s > noticeable on VM with a **large amount of memory**. > =C2=A0 > In our test cases, it's noticeable with memory 1TB and more, whereas > there was no major=C2=A0difference noticed in testcases with <1TB. > =C2=A0 > On bisecting between 5.7 and 5.8, we found the following commit from=C2= =A0 > =E2=80=9CBaoquan He=E2=80=9D to be=C2=A0the cause of boot time increase= in big VM test cases. > =C2=A0 > ------------------------------------- > =C2=A0 > commit 73a6e474cb376921a311786652782155eac2fdf0 > Author: Baoquan He > Date: Wed Jun 3 15:57:55 2020 -0700 > =C2=A0 > mm: memmap_init: iterate over memblock regions rather that check each P= FN > =C2=A0 > When called during boot the memmap_init_zone() function checks if each = PFN > is valid and actually belongs to the node being initialized using > early_pfn_valid() and early_pfn_in_nid(). > =C2=A0 > Each such check may cost up to O(log(n)) where n is the number of memor= y > banks, so for large amount of memory overall time spent in early_pfn*() > becomes substantial. > =C2=A0 > ------------------------------------- > =C2=A0 > For boot time test, we used RHEL 8.1 as the guest OS. > VM config is 84 vcpu and 1TB vRAM. > =C2=A0 > Here are the actual performance numbers. > =C2=A0 > 5.7 GA - 18.17 secs > Baoquan's commit - 21.6 secs (-16% increase in time) > =C2=A0 > From dmesg logs, we can see significant time delay around memmap. > =C2=A0 > Refer below logs. > =C2=A0 > Good commit > =C2=A0 > [0.033176] Normal zone: 1445888 pages used for memmap > [0.033176] Normal zone: 89391104 pages, LIFO batch:63 > [0.035851] ACPI: PM-Timer IO Port: 0x448 > =C2=A0 > Problem commit > =C2=A0 > [0.026874] Normal zone: 1445888 pages used for memmap > [0.026875] Normal zone: 89391104 pages, LIFO batch:63 > [2.028450] ACPI: PM-Timer IO Port: 0x448 Could you add memblock=3Ddebug to kernel cmdline and paste the boot logs = of system w and w/o the commit? > =C2=A0 > We did some analysis, and it looks like with the problem commit it's > not deferring the memory initialization to a later stage and it's=20 > initializing the huge chunk of memory in serial - during the boot-up > time. =C2=A0Whereas with the good commit, it was able to defer the > initialization of the memory when it could be done in parallel. >=20 >=20 > Rahul Gopakumar > Performance=C2=A0Engineering > VMware, Inc. >=20