From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37865C4167B for ; Fri, 8 Dec 2023 14:30:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4FDB6B008C; Fri, 8 Dec 2023 09:30:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB2046B009A; Fri, 8 Dec 2023 09:30:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A04FB6B009B; Fri, 8 Dec 2023 09:30:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8B7DD6B008C for ; Fri, 8 Dec 2023 09:30:30 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 502B340173 for ; Fri, 8 Dec 2023 14:30:30 +0000 (UTC) X-FDA: 81543886620.30.7901977 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf02.hostedemail.com (Postfix) with ESMTP id 6CE2A80011 for ; Fri, 8 Dec 2023 14:30:28 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=i68ETHyR; spf=pass (imf02.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702045828; a=rsa-sha256; cv=none; b=m8Od3L64tiSEUoPiKRHz6l2Y4IJkFIMY6nVAlBwjCpvkVK9INZFz7bfQpkya/yxBp/Vets tBoBs/H9VJ+yDMvl974ScL2QhTeJ0KGt98aeTtmwFAFoVZKUFVc0J7fvKCz1w0pT3/S3kV GPER64im2ZXYBRsOqbZg1VfhVg3pr2s= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=i68ETHyR; spf=pass (imf02.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702045828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9X8HQehrxqqdotEA60rYgF/5Z0BWv62JqBcRTG61sQ8=; b=eb5hfibz8FQ+ooOHKC+skDoRNEwwgzyqSaDNPpfhRsVKy13enMwypoVWfA1zPoIQUBq8yk XTVFH5g7G6YFomhxQNYURAch8ogVx3lByrAulnbX8yd+mnu8dCamJ/UtkhMYZTHpKQ8WPL 1nx7bYeMxu3SKcT3rQnYXmsPKaWTra0= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-40c256ffdbcso22329905e9.2 for ; Fri, 08 Dec 2023 06:30:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1702045827; x=1702650627; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=9X8HQehrxqqdotEA60rYgF/5Z0BWv62JqBcRTG61sQ8=; b=i68ETHyR5a/IHYcaH89Ohh7YSGkrHMKeT696BCtkbzZ9wvZ7i+MKdhu3DiTbsfDkSB RvV8CMeCYaowwyrhzVW6aOI02FpSRYzu3YRc4UwU+2YHBmOiz908GSCrIjxRifwr6cjb uwHdIpzBRGzdAICTZSJOKZwjvL1nefXPqZpp/xxEv/1FqTK80ioDW9qbEVt0RtANrZgb oQcDUphttyZXJtBPg/GvpfM/DZFuWDdkaCZTgYUpEgJSnU5BUmr2S3nGKFxBTqfMyJaF NJRlhzmJxhSyrIKVB2FFv/IbHE8AZm5WDnbqjyAJmQpZAr5OBeOuSgCVCOKgX2VQZzuG ircQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702045827; x=1702650627; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9X8HQehrxqqdotEA60rYgF/5Z0BWv62JqBcRTG61sQ8=; b=YrXR4NJz2m6aeklJH6t2WNa41pE06jfiTdh9j034kbFxwSL61aYnl1ha8rC+Z1G0FH jhh7yDyP7qq1xKhGIE4VcF5YiWW4lzFsdbfC/z8WjoUaAkmFu2Q3XBYgRlMgP8cLAC1v ibcOzPfIIXRab/MDKVxxb2ARmjI7wzd42a8/MRbQShI/VmHmkLBlJXohCr5LuAKBRlU6 B+ueGM2oXmxvnP+bvQ+gGqUDBf4jXwbk9RWm/tqV6RI5MqjNREi3n47sou20IjIf17pf Ln1/knPCVglbqqHcdUhJ5hP90E8BfMiSaAHLKr71DkxV4mEggaSDxA7gwopaYAL78Afg gf5A== X-Gm-Message-State: AOJu0Yx2mTLIiL6Oz6L3F99G3hHVu4haQs456E3Bo6NutwutpqcyBAjZ oM/1GBEz6tA73dXFgeC+96YPcQbFJRkmfipqzcUxRw== X-Google-Smtp-Source: AGHT+IF4tJfaa/fk4ZwgJmpqT0IJIacv8FURbcP3EIH//i3tM9hlmyqgmhJPetNv8trt2hsappah6dIbHJpD3YL5pXI= X-Received: by 2002:a05:600c:4d0e:b0:40b:5e1e:cf1 with SMTP id u14-20020a05600c4d0e00b0040b5e1e0cf1mr33402wmp.44.1702045826774; Fri, 08 Dec 2023 06:30:26 -0800 (PST) MIME-Version: 1.0 References: <20231207150348.82096-1-alexghiti@rivosinc.com> <20231207150348.82096-3-alexghiti@rivosinc.com> <6e3fb7b0-f47d-4d2f-b1b8-3ecc047b9ebf@csgroup.eu> In-Reply-To: <6e3fb7b0-f47d-4d2f-b1b8-3ecc047b9ebf@csgroup.eu> From: Alexandre Ghiti Date: Fri, 8 Dec 2023 15:30:16 +0100 Message-ID: Subject: Re: [PATCH RFC/RFT 2/4] riscv: Add a runtime detection of invalid TLB entries caching To: Christophe Leroy Cc: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-mips@vger.kernel.org" , "linuxppc-dev@lists.ozlabs.org" , "linux-riscv@lists.infradead.org" , "linux-mm@kvack.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6CE2A80011 X-Stat-Signature: sabbzj3pc16u5198f51j8pazp54cudxj X-Rspam-User: X-HE-Tag: 1702045828-481520 X-HE-Meta: U2FsdGVkX18Mp/siKot8NWWYOoMMZGOnTueH/KSvYp+gAQwyTgGllsNFNqpR/IAzzzQNOrLNNY2huaIvAuUTznseUc6mPGGNNPUqdYk/RwN0PbtIx7OTUZTjTTlb3A8Uh7gbA4G6yxyyNX4iZt0bnyqnX1ZUJwI1Gl++KGDXUFdBaqJSP0GFcxZAoieDmr9ZR7SoTDNKVjO20GMlqjOA4+gk8fOVBFOtrIHmnZfGio0QK68nyKahBhh6Q7fB9/sgl1MT7l5Y8FxVCphJJ1orKnqtgnVEufKMpBSMgmuiyV19POkGFgjUTtKnAOuQZFT97UACDcw4JqeZj9Zi+uzm/7aAoFZ5xerW9glTxRx0+uBakSQg0PEYuFDn8TgelcGvMBnQxQ/O8tSDEn+ayComcbok+jCQh6B/L45eg/QOMHmDrYk/kEWcwHn6sd1yBGBwvI7Z57LKbvcpXwM9GrCfmENDHMZztDU7bXp+5Z8wbvhxL4Q0jz2Mvgezl4aIRSacCxN3VB2JW1oGV6TnYiOYthzCjfNcEIw/ZbdMlFIb8gT0DG8eKWw2x9NXv5HiOQAtkicvTSL7UAs9jGLFtKve0uqSAfXADwrJKgwUcRnaCSM+fb89RTGSYzULeqgftVBDr3hih8L96lGMlFTREnZHziR011a+zMB13IZguF2GupL7IQPvunhypZNLsK70nSrYGhOijgSb4tPcydb80AK5x0XM1mybus64OQ4JDdA6GpWQPN0Y4yJ5TcfhtqVHKY/O211cQOI0GjI9s3m23dZsNpcYnHm0QnbdEzCS1czCdqmF7KXtH+JyuL6EED4yCuOWBNstzH2s45nCqlilUnk9pPleUv7AePvV5/5ONDzzmSx0X68GtY96Zh6XxE9Uomk0oPL1Ln/Zc4fcY78oKUc1QyzPcIcQE1R3i72eEt7Ge8b3Xteu7Jyjmm2wLvoqMV/Y5Coa0LvKwgZQwHp3kID 9zllk2+O bF30mOQqIs9ncVhjrvJo/wT2iA2px/diEVvqkKHpQojtNLTTcJRVGCEZtgXrjL8P4t/3ZhcK3W+1oSGCT5Lmw0BrcgwojiOghhenSzgpT4EtXZRWx2aGdZud91qoI7e5JiAGhriRFjTr7t2dggnjH/2peWWL87GIlKBpz0gJ2GhAAfKYsNbS/JRPvY7A5U7wXaDRREIqEykTyOYtoPQ1/O8hxJHgglFGhLLw+KRN5D4+Q7kX9IMLrw6ZU8L4CPOMFD31t9QM9b8Tn1Xo2Fl0m9embWf9M11bHe0Fn5auWSz/GCKV6SEk5S1WrJf+6BqJpZajYenz5/GXcu7LY74jL9zbwGUsvJFvXBW6I0QQe4ui0SHmY/wpk9DLq7kgnT9ezPev7CyBRSLwSNCvy5Ujo24E3WmIFSO5SrNAKrRpGp1qgtnDAj2SKEFRJqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Dec 7, 2023 at 4:55=E2=80=AFPM Christophe Leroy wrote: > > > > Le 07/12/2023 =C3=A0 16:03, Alexandre Ghiti a =C3=A9crit : > > This mechanism allows to completely bypass the sfence.vma introduced by > > the previous commit for uarchs that do not cache invalid TLB entries. > > > > Signed-off-by: Alexandre Ghiti > > --- > > arch/riscv/mm/init.c | 124 ++++++++++++++++++++++++++++++++++++++++++= + > > 1 file changed, 124 insertions(+) > > > > diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c > > index 379403de6c6f..2e854613740c 100644 > > --- a/arch/riscv/mm/init.c > > +++ b/arch/riscv/mm/init.c > > @@ -56,6 +56,8 @@ bool pgtable_l5_enabled =3D IS_ENABLED(CONFIG_64BIT) = && !IS_ENABLED(CONFIG_XIP_KER > > EXPORT_SYMBOL(pgtable_l4_enabled); > > EXPORT_SYMBOL(pgtable_l5_enabled); > > > > +bool tlb_caching_invalid_entries; > > + > > phys_addr_t phys_ram_base __ro_after_init; > > EXPORT_SYMBOL(phys_ram_base); > > > > @@ -750,6 +752,18 @@ static void __init disable_pgtable_l4(void) > > satp_mode =3D SATP_MODE_39; > > } > > > > +static void __init enable_pgtable_l5(void) > > +{ > > + pgtable_l5_enabled =3D true; > > + satp_mode =3D SATP_MODE_57; > > +} > > + > > +static void __init enable_pgtable_l4(void) > > +{ > > + pgtable_l4_enabled =3D true; > > + satp_mode =3D SATP_MODE_48; > > +} > > + > > static int __init print_no4lvl(char *p) > > { > > pr_info("Disabled 4-level and 5-level paging"); > > @@ -826,6 +840,112 @@ static __init void set_satp_mode(uintptr_t dtb_pa= ) > > memset(early_pud, 0, PAGE_SIZE); > > memset(early_pmd, 0, PAGE_SIZE); > > } > > + > > +/* Determine at runtime if the uarch caches invalid TLB entries */ > > +static __init void set_tlb_caching_invalid_entries(void) > > +{ > > +#define NR_RETRIES_CACHING_INVALID_ENTRIES 50 > > Looks odd to have macros nested in the middle of a function. > > > + uintptr_t set_tlb_caching_invalid_entries_pmd =3D ((unsigned long= )set_tlb_caching_invalid_entries) & PMD_MASK; > > + // TODO the test_addr as defined below could go into another pud.= .. > > + uintptr_t test_addr =3D set_tlb_caching_invalid_entries_pmd + 2 *= PMD_SIZE; > > + pmd_t valid_pmd; > > + u64 satp; > > + int i =3D 0; > > + > > + /* To ease the page table creation */ > > + disable_pgtable_l5(); > > + disable_pgtable_l4(); > > + > > + /* Establish a mapping for set_tlb_caching_invalid_entries() in s= v39 */ > > + create_pgd_mapping(early_pg_dir, > > + set_tlb_caching_invalid_entries_pmd, > > + (uintptr_t)early_pmd, > > + PGDIR_SIZE, PAGE_TABLE); > > + > > + /* Handle the case where set_tlb_caching_invalid_entries straddle= s 2 PMDs */ > > + create_pmd_mapping(early_pmd, > > + set_tlb_caching_invalid_entries_pmd, > > + set_tlb_caching_invalid_entries_pmd, > > + PMD_SIZE, PAGE_KERNEL_EXEC); > > + create_pmd_mapping(early_pmd, > > + set_tlb_caching_invalid_entries_pmd + PMD_SIZE= , > > + set_tlb_caching_invalid_entries_pmd + PMD_SIZE= , > > + PMD_SIZE, PAGE_KERNEL_EXEC); > > + > > + /* Establish an invalid mapping */ > > + create_pmd_mapping(early_pmd, test_addr, 0, PMD_SIZE, __pgprot(0)= ); > > + > > + /* Precompute the valid pmd here because the mapping for pfn_pmd(= ) won't exist */ > > + valid_pmd =3D pfn_pmd(PFN_DOWN(set_tlb_caching_invalid_entries_pm= d), PAGE_KERNEL); > > + > > + local_flush_tlb_all(); > > + satp =3D PFN_DOWN((uintptr_t)&early_pg_dir) | SATP_MODE_39; > > + csr_write(CSR_SATP, satp); > > + > > + /* > > + * Set stvec to after the trapping access, access this invalid ma= pping > > + * and legitimately trap > > + */ > > + // TODO: Should I save the previous stvec? > > +#define ASM_STR(x) __ASM_STR(x) > > Looks odd to have macros nested in the middle of a function. > > > > + asm volatile( > > + "la a0, 1f \n" > > + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" > > + "ld a0, 0(%0) \n" > > + ".align 2 \n" > > + "1: \n" > > + : > > + : "r" (test_addr) > > + : "a0" > > + ); > > + > > + /* Now establish a valid mapping to check if the invalid one is c= ached */ > > + early_pmd[pmd_index(test_addr)] =3D valid_pmd; > > + > > + /* > > + * Access the valid mapping multiple times: indeed, we can't use > > + * sfence.vma as a barrier to make sure the cpu did not reorder a= ccesses > > + * so we may trap even if the uarch does not cache invalid entrie= s. By > > + * trying a few times, we make sure that those uarchs will see th= e right > > + * mapping at some point. > > + */ > > + > > + i =3D NR_RETRIES_CACHING_INVALID_ENTRIES; > > + > > +#define ASM_STR(x) __ASM_STR(x) > > Deplicate define ? > > > + asm_volatile_goto( > > + "la a0, 1f \n" > > + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" > > + ".align 2 \n" > > + "1: \n" > > + "addi %0, %0, -1 \n" > > + "blt %0, zero, %l[caching_invalid_entries] \n" > > + "ld a0, 0(%1) \n" > > + : > > + : "r" (i), "r" (test_addr) > > + : "a0" > > + : caching_invalid_entries > > + ); > > + > > + csr_write(CSR_SATP, 0ULL); > > + local_flush_tlb_all(); > > + > > + /* If we don't trap, the uarch does not cache invalid entries! */ > > + tlb_caching_invalid_entries =3D false; > > + goto clean; > > + > > +caching_invalid_entries: > > + csr_write(CSR_SATP, 0ULL); > > + local_flush_tlb_all(); > > + > > + tlb_caching_invalid_entries =3D true; > > +clean: > > + memset(early_pg_dir, 0, PAGE_SIZE); > > + memset(early_pmd, 0, PAGE_SIZE); > > Use clear_page() instead ? > > > + > > + enable_pgtable_l4(); > > + enable_pgtable_l5(); > > +} > > #endif > > > > /* > > @@ -1072,6 +1192,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) > > #endif > > > > #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) > > + set_tlb_caching_invalid_entries(); > > set_satp_mode(dtb_pa); > > #endif > > > > @@ -1322,6 +1443,9 @@ static void __init setup_vm_final(void) > > local_flush_tlb_all(); > > > > pt_ops_set_late(); > > + > > + pr_info("uarch caches invalid entries: %s", > > + tlb_caching_invalid_entries ? "yes" : "no"); > > } > > #else > > asmlinkage void __init setup_vm(uintptr_t dtb_pa) I left this patch so that people can easily test this without knowing what their uarch is actually doing, but it will very likely be dropped as a new extension has just been proposed for that. Thanks anyway, I should have been more clear in the patch title, Alex