From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A68CC433EF for ; Thu, 21 Jul 2022 15:49:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C8FE6B0072; Thu, 21 Jul 2022 11:49:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6797C6B0074; Thu, 21 Jul 2022 11:49:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51A2D6B0075; Thu, 21 Jul 2022 11:49:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 439A06B0072 for ; Thu, 21 Jul 2022 11:49:37 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 17A8940626 for ; Thu, 21 Jul 2022 15:49:37 +0000 (UTC) X-FDA: 79711541994.24.D8D41ED Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf26.hostedemail.com (Postfix) with ESMTP id EF1E914001B for ; Thu, 21 Jul 2022 15:49:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1658418576; x=1689954576; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=cxrvSMixUSva/vWiltebPsIj2OtNtkjOCGTWpSyZWFU=; b=RIOc0/JPv74kfcn7Bt4Ub238PFup6ThYirS82feMLGBK97OvtDmFYcyy OA/5n7noU8XWHl+cJlQi+vojfqT3aZu/WB9D11wEeqPoiVJDbHWKfdwH9 XNsMy+nemcs2oS7gAV1pjdPcyI/4as3zSO78lBLI2E7p6jGG8t3XIP96b egqpL/sD9GT8KJXVLbxs1UvwxXNQa1/gOlZdiEe3M7hn8wEwW1FbdH3HK dKHguHBJ4ZrAcNX2o2oZMFxL/+/NUrZEjJeWt2rpVlC0v/n/7q7k2yIW6 /65W+Cv3qSADHWx+tuxZKEPBXGS9UqoaquMn/Qnd6Q6xMSH8borLrGsZ0 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10415"; a="373385610" X-IronPort-AV: E=Sophos;i="5.93,183,1654585200"; d="scan'208";a="373385610" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2022 08:49:34 -0700 X-IronPort-AV: E=Sophos;i="5.93,183,1654585200"; d="scan'208";a="573792891" Received: from vasantgx-mobl.amr.corp.intel.com (HELO [10.212.244.191]) ([10.212.244.191]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2022 08:49:32 -0700 Message-ID: Date: Thu, 21 Jul 2022 08:49:31 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: [PATCHv7 02/14] mm: Add support for unaccepted memory Content-Language: en-US To: Borislav Petkov , "Kirill A. Shutemov" Cc: Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel , Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , Mike Rapoport , David Hildenbrand , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Mike Rapoport References: <20220614120231.48165-1-kirill.shutemov@linux.intel.com> <20220614120231.48165-3-kirill.shutemov@linux.intel.com> From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="RIOc0/JP"; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf26.hostedemail.com: domain of dave.hansen@intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=dave.hansen@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658418576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BHNqUikkfi6+yEZRvozlYvEZM0bWammPSemIVVZ9s9Y=; b=tpY3kYEV0jh6B5KBKch6jCzUZPa+20ZGHmArgP/oBqlOc8CR+uYIgzG0IzOVcPeqszFMV7 +dKxeYmq/KlniGSKgiIpDJo6q1hNVVl9lXz/KUPoBUJGI+6b6VITgLBkXSlK1zI50qpBAl 4qNJ5AlbPcCho1LHOn/3hQka2UB86N0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658418576; a=rsa-sha256; cv=none; b=bb8zpH7p11Y4nOjmplcauY6ueCeR3+5dyUPKDlQ+xjUUKOOlcP9tEBF+JapQ01fd/xmGt8 sYbQbsaCtIijvEMZvh2yb8MD+TUN61RKCV415kaOkQumHiLwNZm7w6CnbnTigq8DgLbawL Vc5n241zElq48YWdfe+hM7I7NU5OYdw= X-Rspam-User: X-Stat-Signature: dii9b3ihi6wjz433g3nkseaopo4it9wn X-Rspamd-Queue-Id: EF1E914001B Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="RIOc0/JP"; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf26.hostedemail.com: domain of dave.hansen@intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=dave.hansen@intel.com X-Rspamd-Server: rspam04 X-HE-Tag: 1658418575-766736 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 7/21/22 08:14, Borislav Petkov wrote: > On Tue, Jun 14, 2022 at 03:02:19PM +0300, Kirill A. Shutemov wrote: >> On-demand memory accept means latency spikes every time kernel steps >> onto a new memory block. The spikes will go away once workload data >> set size gets stabilized or all memory gets accepted. > What does that mean? > > If we're accepting 2M pages and considering referential locality, how > are those "spikes" even noticeable? Acceptance is slow and the heavy lifting is done inside the TDX module. It involves flushing old aliases out of the caches and initializing the memory integrity metadata for every cacheline. This implementation does acceptance in 2MB chunks while holding a global lock. So, those (effective) 2MB clflush+memset's (plus a few thousand cycles for the hypercall/tdcall transitions) can't happen in parallel. They are serialized and must wait on each other. If you have a few hundred CPUs all trying to allocate memory (say, doing the first kernel compile after a reboot), this is going to be very, very painful for a while. That said, I think this is the right place to _start_. There is going to need to be some kind of follow-on solution (likely background acceptance of some kind). But, even with that solution, *this* code is still needed to handle the degenerate case where the background accepter can't keep up with foreground memory needs.