From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6209EB64DC for ; Tue, 11 Jul 2023 11:38:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47FE96B0075; Tue, 11 Jul 2023 07:38:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42FDD6B0078; Tue, 11 Jul 2023 07:38:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CFBF6B007B; Tue, 11 Jul 2023 07:38:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1A74D6B0075 for ; Tue, 11 Jul 2023 07:38:20 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E5A0D401E3 for ; Tue, 11 Jul 2023 11:38:19 +0000 (UTC) X-FDA: 80999132718.05.D35A966 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 8833A160003 for ; Tue, 11 Jul 2023 11:38:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FKBgOJz5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689075497; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l4lShu3YnxaIDGPIf6B9AQrZukxDqB/6TKJ5gYxYgPk=; b=oNJT5VoL694R30/b7IEqeAWCg7Vb1DQmRdPv4cepDnHMnJcol9NcF8m3KgpP+juD70dOVf Ubu0p9WOI7mCxtcuRe8s14eFgeMgZgV0QQ9jGJDlnbDU+TI4ExnEecAliYRp3xjTIwbWDh KHx/qqbWYApswCjpqYHfn5wZqOuI4fo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FKBgOJz5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689075497; a=rsa-sha256; cv=none; b=PCKNrvpknRtoUlqcImv+iGsvgywO5v3JHZmRQmcatvgfuwhGVmAglzLmn2RbZ7bz25YvHo nG+LccGs0IyFTz6Gvs3IZkHrntlubp8Bj7bxMrGffyNLnP1PF934AtTty/mrwOgxUMRDj8 nquZkUt5B5nB7Cpk1qA6Ef+RVTo+kA4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689075496; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l4lShu3YnxaIDGPIf6B9AQrZukxDqB/6TKJ5gYxYgPk=; b=FKBgOJz5G8rNKce1toTQ3KXvWn6bx3BjbDrmBUzPCYlE0WQ8EDnOjUd6KKGpUJBZ0pWGJc a8w9oNzuDCjWBFNA7BaY+u7mOfMKDX1tCe/8C+SqinIa6VayVGd0lXXCMOqZ7Pu++LX5ma VNOuHkEJQnDpeTAMABZFd7jasSS6vxM= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-275-LbaWrMToNqicDfk1rV9XRA-1; Tue, 11 Jul 2023 07:38:15 -0400 X-MC-Unique: LbaWrMToNqicDfk1rV9XRA-1 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-2b6fdb7eeafso50001121fa.2 for ; Tue, 11 Jul 2023 04:38:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689075494; x=1691667494; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=l4lShu3YnxaIDGPIf6B9AQrZukxDqB/6TKJ5gYxYgPk=; b=LoT4RwLHTcY+fHBU+ya8ggK0VHRS0qxLRYpPSzWZPbU6giu5wFvq4H2VUww4YY5zB2 ULphPM6+09HzmuCSNfNLPp10umVqiy95QlqHe+tad5fhN5K/8cxVAm4oXILGXXUyJqJo IJ+oGyA1YXq7GGyMjUUFZ7cbJUs0xJWCDKNKnnLs19wrqslO6g+RqXP8dgcxWZbJlB8e w+1LeY39cfyrb5Q5a/ukqlp/xIGKFxez9IbrCVla2oecHq7od+5u/U2A24PQoqKdY24W fsx/NXEtB5mW/PExM1ECRFRNVtSQqdQhJAR4+wMFyk0R+RERTKCkoSxrnVSBHStmdXpB u16g== X-Gm-Message-State: ABy/qLa5QPQuEAsFxNL1yTfeQtlBlLja8a+FMrdQmUgUDCl2OfY1CZf0 ZBVQQL1OJDNxLMMqZYkeNiBJjCVdw6H4vdgkvn6KPXt4i/gmYxmfQnS0r4gT/HOd8kwR++EXBtW 5Dx1yCPHrka4= X-Received: by 2002:a05:6512:1090:b0:4fb:74d6:6154 with SMTP id j16-20020a056512109000b004fb74d66154mr15031524lfg.37.1689075493770; Tue, 11 Jul 2023 04:38:13 -0700 (PDT) X-Google-Smtp-Source: APBJJlFvn23Gm82Z4IkNxxc3WPf8KDGAl3B01MTlzaouEW8vRTGxyN1gEu32SmXt4gyRICSlTY0HUQ== X-Received: by 2002:a05:6512:1090:b0:4fb:74d6:6154 with SMTP id j16-20020a056512109000b004fb74d66154mr15031482lfg.37.1689075493297; Tue, 11 Jul 2023 04:38:13 -0700 (PDT) Received: from ?IPV6:2003:cb:c745:4000:13ad:ed64:37e6:115d? (p200300cbc745400013aded6437e6115d.dip0.t-ipconnect.de. [2003:cb:c745:4000:13ad:ed64:37e6:115d]) by smtp.gmail.com with ESMTPSA id u18-20020a05600c211200b003fbbe41fd78sm2333643wml.10.2023.07.11.04.38.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Jul 2023 04:38:12 -0700 (PDT) Message-ID: <700c13ee-cf4c-69bb-7477-4f959d022b0d@redhat.com> Date: Tue, 11 Jul 2023 13:38:11 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, dave.hansen@intel.com, kirill.shutemov@linux.intel.com, tony.luck@intel.com, peterz@infradead.org, tglx@linutronix.de, bp@alien8.de, mingo@redhat.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, ashok.raj@intel.com, reinette.chatre@intel.com, len.brown@intel.com, ak@linux.intel.com, isaku.yamahata@intel.com, ying.huang@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, nik.borisov@suse.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: <999b47f30fbe2535c37a5e8d602c6c27ac6212dd.1687784645.git.kai.huang@intel.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v12 09/22] x86/virt/tdx: Use all system memory when initializing TDX module as TDX memory In-Reply-To: <999b47f30fbe2535c37a5e8d602c6c27ac6212dd.1687784645.git.kai.huang@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8833A160003 X-Stat-Signature: 558ngpjwakwmc4qrpihzw4h8z1jtmxmx X-HE-Tag: 1689075497-548207 X-HE-Meta: U2FsdGVkX19rb21KbpHcmS/WoLogFewHZSfv8suy3TAmk9F6yqQ8j8Qg8hfJaxofRq60Dp4cuxKe8MQTz/c4GZAQexlhD0eeXM8O221DNTBvpo9FhXSwg77oyM3pwugkVf9I6NijD3hXilikOyr8cM3SFEeF3RMibEJVuGADX7pwV2zOCGPSUr6auf3j37SGp3mnwbwjvUbs/YQD8RpURluq+ENUOIVslgaz3Pc5hgipmpVuixdNU/nIAk37Sd/4Sa7Z7xf9oQ09YEF8tGbXY/PI8B8UbAS8m6Ua/nk2RNmfiaQkBpe1F5erf/Sun8kAgi/u7ddiDdES4e/gDHrGZK5SjzDD0Pa/GoyUemrNm/SHuwaJqON9tR8+1iQKgPNG9DlTOt7ULF95hY2ag5NuOP9YcIhkpKrd42ZfNjxnTf505a6d3vz5WLRfSAwtT7dLnHfYLDKlokdsktwkasNk6IJJz72IDUuX3tSlQRhb3ftcGlNm0WcQuksBIPk+hF23heCvIj3IsFLHxXeZmw0w+09k5jP2ug+Amcbl4gpadbphsF7rOMZKKMtA2FnNIIWwaF13tteIqgtghD5qwLnMUG9zz/RbiIioAzjXANYAUfIpMF5xKh4NzQ8bqBVCoeQHrt1r7EOSrMeysAfwaN7cCN2DibDXseJtYs4GcEZjqLJ6C5yQNVuCmzekAhzx3wSoek0PjaAx9vpaJo5Pk8tbgAsEbEPFWIdhOwVbD/C0Lv2TLCTPICExLezxxn0Yv5isiukZwZdmZwu3uNLinVfky+zhj4X9juT+TfITRmjZ3chELMNiWY1Cpq2LYW1WdYFTojVTXLU8kYWysXa68pA5MBFeOGdLGCuNUR0urZFrbE7LHcuP7As+ZVnecHgqCn24ClTVwbE9EjOGx7K9r1/epLkkVBcMZTQQOTRtiySiui2FLPzZsLpTh+S4mTJmpyg7TBwWpGyE0XhfJcjnd8L vN+yVMY+ WGZ+teWiJ5nAMNdGy6ZYJqs9SS1E63LZ4vLs69C+IavhzTEGEiyk8vQ6PoLLUTT4Z1e4W8+1IAInRI7vLkvSCjIEG6KbV5wSgcFK4Mlm4yx++LKAOMh3IomAbwkPH6zcV0vcLioaMqix8KlDdrzasNUgCPHpSW4QBRhwVO2L4XX99mgBgLxqUoWkcc51z5iWZCWY8O5bOLx/ry2LfoFGWtLM9btDdOdpAq+k0Us9FOhKKh9Wt9SdRr7Zisma/9xiMrRZ18dYF2C6HFy9P5DS52+z2DYBcPfycxt+GV9xT7P2xtrMFX/OVNhzgZMe755G19TjDgEjZhx/bcvVWv3VNXAiKpIQYt9m6Glga/HsxoA3DcoA1HKF3r2yM4VTLWc/HTbA3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: [...] > +/* All TDX-usable memory regions. Protected by mem_hotplug_lock. */ > +static LIST_HEAD(tdx_memlist); > + > /* > * Wrapper of __seamcall() to convert SEAMCALL leaf function error code > * to kernel error code. @seamcall_ret and @out contain the SEAMCALL > @@ -204,6 +214,79 @@ static int tdx_get_sysinfo(struct tdsysinfo_struct *sysinfo, > return 0; > } > > +/* > + * Add a memory region as a TDX memory block. The caller must make sure > + * all memory regions are added in address ascending order and don't > + * overlap. > + */ > +static int add_tdx_memblock(struct list_head *tmb_list, unsigned long start_pfn, > + unsigned long end_pfn) > +{ > + struct tdx_memblock *tmb; > + > + tmb = kmalloc(sizeof(*tmb), GFP_KERNEL); > + if (!tmb) > + return -ENOMEM; > + > + INIT_LIST_HEAD(&tmb->list); > + tmb->start_pfn = start_pfn; > + tmb->end_pfn = end_pfn; > + > + /* @tmb_list is protected by mem_hotplug_lock */ If the list is static and independent of memory hotplug, why does it have to be protected? I assume because the memory notifier might currently trigger before building the list. Not sure if that is the right approach. See below. > + list_add_tail(&tmb->list, tmb_list); > + return 0; > +} > + > +static void free_tdx_memlist(struct list_head *tmb_list) > +{ > + /* @tmb_list is protected by mem_hotplug_lock */ > + while (!list_empty(tmb_list)) { > + struct tdx_memblock *tmb = list_first_entry(tmb_list, > + struct tdx_memblock, list); > + > + list_del(&tmb->list); > + kfree(tmb); > + } > +} > + > +/* > + * Ensure that all memblock memory regions are convertible to TDX > + * memory. Once this has been established, stash the memblock > + * ranges off in a secondary structure because memblock is modified > + * in memory hotplug while TDX memory regions are fixed. > + */ > +static int build_tdx_memlist(struct list_head *tmb_list) > +{ > + unsigned long start_pfn, end_pfn; > + int i, ret; > + > + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) { > + /* > + * The first 1MB is not reported as TDX convertible memory. > + * Although the first 1MB is always reserved and won't end up > + * to the page allocator, it is still in memblock's memory > + * regions. Skip them manually to exclude them as TDX memory. > + */ > + start_pfn = max(start_pfn, PHYS_PFN(SZ_1M)); > + if (start_pfn >= end_pfn) > + continue; > + > + /* > + * Add the memory regions as TDX memory. The regions in > + * memblock has already guaranteed they are in address > + * ascending order and don't overlap. > + */ > + ret = add_tdx_memblock(tmb_list, start_pfn, end_pfn); > + if (ret) > + goto err; > + } So at the time init_tdx_module() is called, you simply go over all memblocks. But how can you be sure that they are TDX-capable? While the memory notifier will deny onlining new memory blocks, add_memory() already happened and added a new memory block to the system (and to memblock). See add_memory_resource(). It might be cleaner to build the list once during module init (before any memory hotplug can happen and before we tear down memblock) and not require ARCH_KEEP_MEMBLOCK. Essentially, before registering the notifier. So the list is really static. But maybe I am missing something. > + > + return 0; > +err: > + free_tdx_memlist(tmb_list); > + return ret; > +} > + > static int init_tdx_module(void) > { > struct tdsysinfo_struct *sysinfo; > @@ -230,10 +313,25 @@ static int init_tdx_module(void) > if (ret) > goto out; [...] > > +struct tdx_memblock { > + struct list_head list; > + unsigned long start_pfn; > + unsigned long end_pfn; > +}; If it's never consumed by someone else, maybe keep it local to the c file? > + > struct tdx_module_output; > u64 __seamcall(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, > struct tdx_module_output *out); -- Cheers, David / dhildenb