From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D20AACDB47E for ; Fri, 13 Oct 2023 12:34:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CB2180015; Fri, 13 Oct 2023 08:34:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27B518D0015; Fri, 13 Oct 2023 08:34:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1432580015; Fri, 13 Oct 2023 08:34:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 02E3E8D0015 for ; Fri, 13 Oct 2023 08:34:16 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B67D31A0258 for ; Fri, 13 Oct 2023 12:34:15 +0000 (UTC) X-FDA: 81340380870.28.76CCA97 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by imf29.hostedemail.com (Postfix) with ESMTP id 7E9F2120002 for ; Fri, 13 Oct 2023 12:34:12 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=mejeEkWn; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf29.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.136) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697200452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uYtmUhmwaaBTNa8kFVOzNzwocrXdB8aiXoGdyFWTayk=; b=4oBmRFgtgpJfDgLjUYIQWiLbHbutMTPNYAqc4lkx2Cffvv/lolUZ8p4PKYXqHuo36pIgyn 91pablkizkdLyh+k1t9f3H2dW3bBj/74U1LSv70RgADtX3yvgvM42crhey+utHQ9IFThIC ul3IrRLtJAEMAnBLipX/4RsBtQqoKvs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=mejeEkWn; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf29.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.136) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697200452; a=rsa-sha256; cv=none; b=kQ89U8ElyiEFVPihSOk8WIlHYmXMwDsAtIGHE5QqYluSCv55lNWvlqQ4FQHM+DH6Xx/RO3 i37VQVRp6HzyH+bVuQYyqLZ3ALCbUtt+cHoerkjd4Z3Lxg5r1m2B3tKZ8k3VhD17z3ioR/ hM39dUzHuby6JkUp8TMI6RhXXbk1wi0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697200452; x=1728736452; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=fGrVnxZmpyfEBqkF7piZXz/mFn6UmaNoaPJzfrOU+hs=; b=mejeEkWnlH5zKCRdFVYMYnwZyGKSWkqmMDx4fj27td+edW+jbh1ardj2 V1jZoE1y6hEbjmnCCqqFKpzUZuQWUlUGAJxG7K1mBvmKCsNKDIOGmRCbt 7rkzt7UMpvDEvGLBhcGtDWQNK7Z4aVNGmnNZEeo27/I5fvwUJBrrmZX8S AKx+mHGlYRVnJyms4nbKi4Gdw/9S1JA3OV12Xp3AqRnEshhbtj3CjWEfK fFP5CH90R4vYouli08yACYPzJ4M2bNFvK5VMwuwx3nUJuL9yPJele6DUJ Y3x3eS/ZvKybqnkI5UHq5C1Gj+vcSsjNb1RIoUnmhSXDYhs+N36uHEmc1 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10862"; a="364536433" X-IronPort-AV: E=Sophos;i="6.03,222,1694761200"; d="scan'208";a="364536433" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Oct 2023 05:34:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10862"; a="1086128325" X-IronPort-AV: E=Sophos;i="6.03,222,1694761200"; d="scan'208";a="1086128325" Received: from bgras-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.59.145]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Oct 2023 05:34:01 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 6E30C104A05; Fri, 13 Oct 2023 15:33:58 +0300 (+03) Date: Fri, 13 Oct 2023 15:33:58 +0300 From: "Kirill A. Shutemov" To: Michael Roth Cc: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel , Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCHv14 5/9] efi: Add unaccepted memory support Message-ID: <20231013123358.y4pcdp5fgtt4ax6g@box.shutemov.name> References: <20230606142637.5171-1-kirill.shutemov@linux.intel.com> <20230606142637.5171-6-kirill.shutemov@linux.intel.com> <20231010210518.jguawj7bscwgvszv@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231010210518.jguawj7bscwgvszv@amd.com> X-Rspamd-Queue-Id: 7E9F2120002 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 4k7h1yrz8z53sjny4r6q7utzezjdtdtw X-HE-Tag: 1697200452-941353 X-HE-Meta: U2FsdGVkX18xHNYV2cKgV9Ln1+NVR5V4lX4Aa7MvZdiaxm7T0cJ1zvESsUFmaOUFg+0+C5cQApFfZI2+DeDiuC5bX2xgpZMFoDH6yhTReCNzsEYuFhjkfwVy4jhUzVLT9cU080y7B4wC8BdH5EBwAoRAhaq+dI3982GvN/zhnHB3aDbSgDe8nMI0YzYs75d7+SqXvEwUVrfOIb6HavueMNXUNT/V4JMyorFY1a/ryvy5sWDCyZP1hOpKs2N33wMAE3Hk3jtMQ8acZBpQSx0Ga8Hl9C5q3OMmx1YDzQdApn0BRy1jFYKHubLL2iULqupfCdz4IKykQ7wSLA1k8KBMbTuk/XcPgVuJxdalDIUBrP+IepXRDMMtsWcw63Cez6B6h3ww1L6gvG9vC1krvSXoRWjtwWgncW0QIbyAWj7u7AuwfUHwaEavDXGd+J8jvR6utlLGnZe+q4BRhIOIj3Kov71RcZJIjQ9PghaS0h40HzM3fwze/Vx0ZdNkQp+ICEvVhGJtBWqejJt6l2q7LmdiY5WFzlYbBQROGDKaDR0zbD2TxwfuwD8qZJQuQINYnkYZjGq08XTAt4DHi1xEXtO/PKTZPkkVzGFfkPkY3+l3fHxRR3DCYcu+X4GTNMliCaHJHMEv44C0Ut60ycmbSwknnxrh5eyjN1/VpZZb5U1puQ/QMZ955V/Op7Fc074MXigK0HsmEB5k4aC0JAE8dCDwmAg3IKn7lC58/vKoDYne46G/4uD+IZum+X8rvM22cp8HJ5rTBmSKf+E55lZGjhfmVDI2ugLu5E3CKH4uYU63tEpYhkMYeh2QcaVQ8LUHie6y6PqZQsPowTT+JCIr2QoqMxADuj3VYx3hZ3G0dz8qWdSqBvp4d61/vUiLIH7VlBBApuCiB1gzjxuaeovQtmaPCMFbNdoeUpsx/gJsxy30LOReyJiOX41IVrP9rvbNgiQkmU1Jzf8O4wV/SDECrYy h/MEGK4f SAhQ4mQxU/ssDKenYwDWprjHSm8Meir8nTD5DOYjmpG2UVpgnR246JngmJyLn1oJWfA0XT043hfeWkVwpF69NFpYUXmZv6cEa/S9fDp+KMsz0Wpp+ROw+XuiCHJojsQggtppP9Tb/EU94P8ZvpnXHykN2Wb9YhOFtBzyjNcxsaBVu0jWZeqkxT1tkW0T+Wa3YPvG3ExfRMwzO/ZIyz3heMQGffIij34I/kSYa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Oct 10, 2023 at 04:05:18PM -0500, Michael Roth wrote: > On Tue, Jun 06, 2023 at 05:26:33PM +0300, Kirill A. Shutemov wrote: > > efi_config_parse_tables() reserves memory that holds unaccepted memory > > configuration table so it won't be reused by page allocator. > > > > Core-mm requires few helpers to support unaccepted memory: > > > > - accept_memory() checks the range of addresses against the bitmap and > > accept memory if needed. > > > > - range_contains_unaccepted_memory() checks if anything within the > > range requires acceptance. > > > > Architectural code has to provide efi_get_unaccepted_table() that > > returns pointer to the unaccepted memory configuration table. > > > > arch_accept_memory() handles arch-specific part of memory acceptance. > > > > Signed-off-by: Kirill A. Shutemov > > Reviewed-by: Ard Biesheuvel > > Reviewed-by: Tom Lendacky > > --- > > arch/x86/platform/efi/efi.c | 3 + > > drivers/firmware/efi/Makefile | 1 + > > drivers/firmware/efi/efi.c | 25 +++++ > > drivers/firmware/efi/unaccepted_memory.c | 112 +++++++++++++++++++++++ > > include/linux/efi.h | 1 + > > 5 files changed, 142 insertions(+) > > create mode 100644 drivers/firmware/efi/unaccepted_memory.c > > > > diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c > > new file mode 100644 > > index 000000000000..08a9a843550a > > --- /dev/null > > +++ b/drivers/firmware/efi/unaccepted_memory.c > > @@ -0,0 +1,112 @@ > > +// SPDX-License-Identifier: GPL-2.0-only > > + > > +#include > > +#include > > +#include > > +#include > > + > > +/* Protects unaccepted memory bitmap */ > > +static DEFINE_SPINLOCK(unaccepted_memory_lock); > > + > > +/* > > + * accept_memory() -- Consult bitmap and accept the memory if needed. > > + * > > + * Only memory that is explicitly marked as unaccepted in the bitmap requires > > + * an action. All the remaining memory is implicitly accepted and doesn't need > > + * acceptance. > > + * > > + * No need to accept: > > + * - anything if the system has no unaccepted table; > > + * - memory that is below phys_base; > > + * - memory that is above the memory that addressable by the bitmap; > > + */ > > +void accept_memory(phys_addr_t start, phys_addr_t end) > > +{ > > + struct efi_unaccepted_memory *unaccepted; > > + unsigned long range_start, range_end; > > + unsigned long flags; > > + u64 unit_size; > > + > > + unaccepted = efi_get_unaccepted_table(); > > + if (!unaccepted) > > + return; > > + > > + unit_size = unaccepted->unit_size; > > + > > + /* > > + * Only care for the part of the range that is represented > > + * in the bitmap. > > + */ > > + if (start < unaccepted->phys_base) > > + start = unaccepted->phys_base; > > + if (end < unaccepted->phys_base) > > + return; > > + > > + /* Translate to offsets from the beginning of the bitmap */ > > + start -= unaccepted->phys_base; > > + end -= unaccepted->phys_base; > > + > > + /* Make sure not to overrun the bitmap */ > > + if (end > unaccepted->size * unit_size * BITS_PER_BYTE) > > + end = unaccepted->size * unit_size * BITS_PER_BYTE; > > + > > + range_start = start / unit_size; > > + > > + spin_lock_irqsave(&unaccepted_memory_lock, flags); > > + for_each_set_bitrange_from(range_start, range_end, unaccepted->bitmap, > > + DIV_ROUND_UP(end, unit_size)) { > > + unsigned long phys_start, phys_end; > > + unsigned long len = range_end - range_start; > > + > > + phys_start = range_start * unit_size + unaccepted->phys_base; > > + phys_end = range_end * unit_size + unaccepted->phys_base; > > + > > + arch_accept_memory(phys_start, phys_end); > > + bitmap_clear(unaccepted->bitmap, range_start, len); > > + } > > + spin_unlock_irqrestore(&unaccepted_memory_lock, flags); > > +} > > While testing SNP guests running today's tip/master (ef19bc9dddc3) I ran > into what seems to be fairly significant lock contention due to the > unaccepted_memory_lock spinlock above, which results in a constant stream > of soft-lockups until the workload gets all its memory accepted/faulted > in if the guest has around 16+ vCPUs. > > I've included the guest dmesg traces I was seeing below. > > In this case I was running a 32 vCPU guest with 200GB of memory running on > a 256 thread EPYC (Milan) system, and can trigger the above situation fairly > reliably by running the following workload in a freshly-booted guests: > > stress --vm 32 --vm-bytes 5G --vm-keep > > Scaling up the number of stress threads and vCPUs should make it easier > to reproduce. > > Other than unresponsiveness/lockup messages until the memory is accepted, > the guest seems to continue running fine, but for large guests where > unaccepted memory is more likely to be useful, it seems like it could be > an issue, especially when consider 100+ vCPU guests. Okay, sorry for delay. It took time to reproduce it with TDX. I will look what can be done. -- Kiryl Shutsemau / Kirill A. Shutemov