From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 929A4C433F5 for ; Tue, 12 Apr 2022 08:15:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F9C06B0078; Tue, 12 Apr 2022 04:15:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AA276B007B; Tue, 12 Apr 2022 04:15:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14C2A6B007D; Tue, 12 Apr 2022 04:15:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 064A96B0078 for ; Tue, 12 Apr 2022 04:15:17 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id DB04D81A8D for ; Tue, 12 Apr 2022 08:15:16 +0000 (UTC) X-FDA: 79347517032.13.C665BA3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 58A1B80005 for ; Tue, 12 Apr 2022 08:15:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649751315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8HjmZzAIO50unb7pGZAQpRPqJO313RFErJC/+pOp/R0=; b=iKzXnDoLIcATeO1qukJjLuqfl4LeC0wGYWMGd7r97A4Pr6/bDqF91RyLBw9wC2eqK2K4pr Urc7gBRxbFp38F4UitrDv6yeUAIuwsfebezA0r/cdIWU9TwOVUTTwVtjQw2zAr9oF0C+3Z cEJlgy74N6/rtSlA9v0GlHLtqWcAUpc= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-2-ak50II0kNhabOgggRZ56og-1; Tue, 12 Apr 2022 04:15:11 -0400 X-MC-Unique: ak50II0kNhabOgggRZ56og-1 Received: by mail-wr1-f72.google.com with SMTP id i64-20020adf90c6000000b00203f2b5e090so3800845wri.9 for ; Tue, 12 Apr 2022 01:15:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent :content-language:to:cc:references:from:organization:subject :in-reply-to:content-transfer-encoding; bh=8HjmZzAIO50unb7pGZAQpRPqJO313RFErJC/+pOp/R0=; b=iH6CsGY/+G68AsMOXcZxzj2fVTS3stNVqefxcqroOBhLj/bWNA/SmKdJ9IaXMFZ7A7 CvY3O3dVycXM7voJqsIo66kfYmKZyayfLA2gzUd9lm4CqMagtD5ZvTy5ZmYPln8iDAA2 pH3o//s+vcH0OsZ5H2UN0cmo9R0FI7BidC5SGSw1JSiMt03c21Hp7g1nsh5VFOoIPx+p N3xgwkFuMpiVKSgTVQ05oT/66Q94vDemSiRohJKGgoz7OmJ6GjmHMMpgKdAM/wWXTWZp SYSQDIMhM2Mmvrhs6IPOoRimSUdCzjAhAlaewLMIlfcQHSR57HXM0X4vDF8y/EN3uaoq Ku1g== X-Gm-Message-State: AOAM53360jLS1S0fKFkK5s2X1s5/lKgH4eODq3w5ryUmaDQAID2HBtFx gmnItZcxiU77eN4QOeDSL/Ek0RQ/VmuQlIai6QPDLPLmXEbOuOkxZl5209VB61XFMcEjcyP072V oXp314si7Eg4= X-Received: by 2002:adf:ed8b:0:b0:206:1771:e373 with SMTP id c11-20020adfed8b000000b002061771e373mr28461577wro.84.1649751310601; Tue, 12 Apr 2022 01:15:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxZYmHGzijmWq4j1QgvJKMtTI4uehpfCeAgJZFkKQ0wTIJR1ucWkaH70j+SivEpEn8AXTnKWA== X-Received: by 2002:adf:ed8b:0:b0:206:1771:e373 with SMTP id c11-20020adfed8b000000b002061771e373mr28461563wro.84.1649751310358; Tue, 12 Apr 2022 01:15:10 -0700 (PDT) Received: from ?IPV6:2003:cb:c707:1800:7c14:16cc:5291:a9f3? (p200300cbc70718007c1416cc5291a9f3.dip0.t-ipconnect.de. [2003:cb:c707:1800:7c14:16cc:5291:a9f3]) by smtp.gmail.com with ESMTPSA id b14-20020a7bc24e000000b003899c8053e1sm1814456wmj.41.2022.04.12.01.15.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Apr 2022 01:15:09 -0700 (PDT) Message-ID: Date: Tue, 12 Apr 2022 10:15:07 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.6.2 To: Dave Hansen , "Kirill A. Shutemov" , Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , Brijesh Singh , Mike Rapoport , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Mike Rapoport References: <20220405234343.74045-1-kirill.shutemov@linux.intel.com> <20220405234343.74045-2-kirill.shutemov@linux.intel.com> <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCHv4 1/8] mm: Add support for unaccepted memory In-Reply-To: <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iKzXnDoL; spf=none (imf30.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 58A1B80005 X-Stat-Signature: 67bxi16zsdjg7gucyewosgomdzkcy76x X-HE-Tag: 1649751316-484245 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 08.04.22 21:11, Dave Hansen wrote: > On 4/5/22 16:43, Kirill A. Shutemov wrote: >> Kernel only needs to accept memory once after boot, so during the boot >> and warm up phase there will be a lot of memory acceptance. After things >> are settled down the only price of the feature if couple of checks for >> PageUnaccepted() in allocate and free paths. The check refers a hot >> variable (that also encodes PageBuddy()), so it is cheap and not visible >> on profiles. > > Let's also not sugar-coat this. Page acceptance is hideously slow. > It's agonizingly slow. To boot, it's done holding a global spinlock > with interrupts disabled (see patch 6/8). At the very, very least, each > acceptance operation involves a couple of what are effectively ring > transitions, a 2MB memset(), and a bunch of cache flushing. > > The system is going to be downright unusable during this time, right? > > Sure, it's *temporary* and only happens once at boot. But, it's going > to suck. > > Am I over-stating this in any way? > > The ACCEPT_MEMORY vmstat is good to have around. Thanks for adding it. > But, I think we should also write down some guidance like: > > If your TDX system seems as slow as snail after boot, look at > the "accept_memory" counter in /proc/vmstat. If it is > incrementing, then TDX memory acceptance is likely to blame. > > Do we need anything more discrete to tell users when acceptance is over? > For instance, maybe they run something and it goes really slow, they > watch "accept_memory" until it stops. They rejoice at their good > fortune! Then, memory allocation starts falling over to a new node and > the agony beings anew. > > I can think of dealing with this in two ways: > > cat /sys/.../unaccepted_pages_left > > which just walks the bitmap and counts the amount of pages remaining. or > something like: > > echo 1 > /sys/devices/system/node/node0/make_the_pain_stop > > Which will, well, make the pain stop on node0. > Either I'm missing something important or the random pain might just take a really long time to stop? I mean, we tend to reallocate the memory first that we freed last (putting it to the head of the freelist when freeing and picking from the head when allocating). So unless your kernel goes crazy and allocates each and every page right after boot, essentially accepting all memory, you might have random unaccepted pages lurking at the tail of the freelists. So if the VM is running for 355 days without significant memory pressure, you can still run into unaccepted pages at day 356 that results in a random delay due to acceptance of memory. I think we most certainly want some way to make the random pain stop, or to make the random pain go away after boot quickly. The "unaccepted_pages_left" indicator would just be a "hey, there might be random delays, but you cannot do anything about it". Magic toggles like "make_the_pain_stop" are not so nice. Can we simply automate this using a kthread or smth like that, which just traverses the free page lists and accepts pages (similar, but different to free page reporting)? -- Thanks, David / dhildenb