From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B7D5FA373E for ; Wed, 26 Oct 2022 14:48:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98CEC8E0002; Wed, 26 Oct 2022 10:48:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93DB98E0001; Wed, 26 Oct 2022 10:48:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 805748E0002; Wed, 26 Oct 2022 10:48:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 74BF08E0001 for ; Wed, 26 Oct 2022 10:48:11 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 234BD1A1052 for ; Wed, 26 Oct 2022 14:48:11 +0000 (UTC) X-FDA: 80063380782.23.75D23CF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id AA39010003D for ; Wed, 26 Oct 2022 14:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666795690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=waNXQeIHAKPovVRiEc+x+REE+g4+Bdakw7JoCJaSBPA=; b=a+cbGGQjF0aRPjdSshTKzf6F+UoPcRNlLqSUMzbyu/wZkf0P/CJT92s/Lczh3cno018EVM ONZlUIWlBKnljrVCilTSqXV1HzXcFxVm7ZFMooi08xpgeHTx7rq57FXWkEDX9S7Gnvyjao PQzQu3LajdQ5WTUr0gzBYveXpRWW6uY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-640-PXZwuAI1MSSxNwq7DZMmyQ-1; Wed, 26 Oct 2022 10:48:06 -0400 X-MC-Unique: PXZwuAI1MSSxNwq7DZMmyQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D7C86101A54E; Wed, 26 Oct 2022 14:48:05 +0000 (UTC) Received: from localhost (ovpn-12-35.pek2.redhat.com [10.72.12.35]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AE1B239DB3; Wed, 26 Oct 2022 14:48:04 +0000 (UTC) Date: Wed, 26 Oct 2022 22:48:01 +0800 From: Baoquan He To: Borislav Petkov , david@redhat.com Cc: Eric DeVolder , Oscar Salvador , Andrew Morton , linux-kernel@vger.kernel.org, x86@kernel.org, kexec@lists.infradead.org, ebiederm@xmission.com, dyoung@redhat.com, vgoyal@redhat.com, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, hpa@zytor.com, nramas@linux.microsoft.com, thomas.lendacky@amd.com, robh@kernel.org, efault@gmx.de, rppt@kernel.org, sourabhjain@linux.ibm.com, linux-mm@kvack.org Subject: Re: [PATCH v12 7/7] x86/crash: Add x86 crash hotplug support Message-ID: References: <53aed03e-2eed-09b1-9532-fe4e497ea47d@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666795690; a=rsa-sha256; cv=none; b=rz2gvIttMjU/WQInteHqMtzAVtVAwH5CfPRpsCe6BvhO0LUKRvORpqgqAiXihJ/JeOwi3T pTws6UwD/d31w5RRac+gPPXwTI5vvTt6HJsgp2NmTipmDwQACpFsGGe8C9NZttnp7QokWT LWJTvaVazkeVwCrOLbVy6L67WEUhTL8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=a+cbGGQj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of bhe@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=bhe@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666795690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=waNXQeIHAKPovVRiEc+x+REE+g4+Bdakw7JoCJaSBPA=; b=vNP+PCHoAEKCjtn1jbljFpwoWRhzpkRNFU1iWOHbC4hvyidKnTBTZCzWg1PR6v1qDqhcru 2SqRkISGReM9Ymabylx5uXIC+uifSkTuQln/l0UFmgBjfRUwatfqHHamsAdk4kJFJFBMhV durP3Uy2RBajBTnReWV8d6UeCzI09Y8= X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=a+cbGGQj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of bhe@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=bhe@redhat.com X-Rspamd-Server: rspam11 X-Stat-Signature: a1nythdzaju4pwfucznxdothqz4u8qot X-Rspamd-Queue-Id: AA39010003D X-HE-Tag: 1666795690-406638 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/25/22 at 12:31pm, Borislav Petkov wrote: > On Thu, Oct 13, 2022 at 10:57:28AM +0800, Baoquan He wrote: > > The concern to range number mainly is on Virt guest systems. > > And why would virt emulate 1K hotpluggable DIMM slots and not emulate a > real machine? Well, currently, mem hotpug is an important feature on virt system to dynamically increase/shrink memory on the system. If only emulating real machine, it won't be different than bare metal system. IIRC, the ballon driver or virtio-mem feature can add memory board, e.g 1G, block size is 128M, 8 blocks added. When shrinking this 1G memory later, it will take best effort way to hot remove memory. Means if any memory block is occupied, it will be kept there. Finally we could only remove every second blocks, 4 blocks altogether. Then the left un-removed blocks will produce 4 separate memory regions. Like this, a virt guest could have many memory regions in kernel after memory being added/removed. If I am wrong, Please correct me, David. > > > On baremetal system, basically only very high end server support > > memory hotplug. I ever visited customer's lab and saw one server, > > it owns 8 slots, on each slot a box containing about 20 cpus and 2T > > memory at most can be plugged in at one time. So people won't make too > > many slots for hotplugging since it's too expensive. > > There you have it - the persuading argument. > > > I checked user space kexec code, the maximum memory range number is > > honored to x86_64 because of a HPE SGI system. After that, nobody > > complains about it. Please see below user space kexec-tools commit in > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git > > > > The memory ranges may be not all made by different DIMM slots, could be > > firmware reservatoin, e.g efi/BIOS diggged out physical memory, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > I don't know what that means. > > If it is firmware crap, you want to exclude that from kdump anyway. Yes, now assume we have a HPE SGI system and it has memory hotplug capacity. The system itself has already got memory regions more than 1024. Then when we hot add extra memory board, we want to include the newly added memory regions into elfcorehdr so that it will be dumped out in kdump kernel. That's why I earlier suggested 2048 for number of memory regions. > > > Now CONFIG_NR_CPUS has the maximum number as 8192. And user space > > kexec-tools has maximum memory range number as 2048. We can take > > the current 8192 + 2048 = 10K as default value conservatively. Or > > take 8192 + 2048 * 2 = 12K which has two times of maximum memory range > > bumber in kexec-tools. What do you think? > > I still think that we should stick to reality and support what is > possible not what is potentially and theoretically there. Yes, agree. We should try to get a number which satisfies needs in reality. For Kconfig CRASH_MAX_MEMORY_RANGES in this patch, I have three items to suggest: 1) the name is not good, it doesn't reflect the fact that it's the number of program headers of elfcorehdr which includes the cpu note numbers and memory region numers. 2) default cpu number, I suggest 512 or 1024. The biggest number I ever saw in reality is 384. On virt system, it won't be too big. Below is abstracted from arch/x86/Kconfig. A smaller one is also OK, we can enlarge it when people really have a super machine and run into the problem. config NR_CPUS_DEFAULT int depends on X86_64 default 8192 if MAXSMP default 64 if SMP default 1 if !SMP 3) For memory regions, I would suggest 2048. Likewise, smaller value is also fine, we can enlarge it when a real system run into this. I made a draft here for reference, with my undertanding. Please feel free to change it. +config CRASH_ELF_CORE_PHDRS_NUM + depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG) + int + default 3072 + help + For the kexec_file_load path, specify the default number of + phdr for the vmcore. E.g the memory regions represented by the + 'System RAM' entries in /proc/iomem, the cpu notes of each + present cpu stored in /sys/devices/system/cpu/cpuX/crash_notes. Thanks