From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F2ECC4167B for ; Thu, 7 Dec 2023 15:06:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1C206B0092; Thu, 7 Dec 2023 10:06:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AA4BF6B0093; Thu, 7 Dec 2023 10:06:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7B36B0095; Thu, 7 Dec 2023 10:06:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76D406B0092 for ; Thu, 7 Dec 2023 10:06:17 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4C789A023C for ; Thu, 7 Dec 2023 15:06:17 +0000 (UTC) X-FDA: 81540347994.03.65D4DAC Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by imf27.hostedemail.com (Postfix) with ESMTP id 0CDBF4000B for ; Thu, 7 Dec 2023 15:05:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SSDtWNrD; dmarc=none; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961555; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=HcOU2U6azEG6440nkBUgKEkqovLWQ/kcJKYvvk5ZV0KZxst5BlJKgrVk3nikO+W+QRq7RJ Z0vThU1bDtcHKJCWK7d5DiCWqGMCOk/bF8mAOh/bR34dj3wZgndrpx5V7T/tna9fh9eVp+ dDsx/+1vOV36ywO9XfbR5p6jQOG5TKU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SSDtWNrD; dmarc=none; spf=pass (imf27.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961555; a=rsa-sha256; cv=none; b=w8PlaXyN6KBfvPQt9H3ez84Dr2fzK4nqnvfW0xVBBi4Rxk0ii2iWr7ZytosMZHSRAuraaf DAqCR1B0J00lDDfxzPB+ptlMJLJn55HtCRplDw3t9K5G9/NNeXMd7okmvK+OG8GZcguuG9 0tUUi6bhUkCavMxN9czEZflDEI1TJ80= Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-3332ad5b3e3so1059514f8f.2 for ; Thu, 07 Dec 2023 07:05:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961553; x=1702566353; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=SSDtWNrDem95axoCY9b+Xo5xwIg3OrfrzpExL7rPgm6Mvrqd58CgzeSA5BC9OSsOEZ saxXSGToJ/CailYtZZUUnU5gDtnMc58ccnpiNYLki8POOPitRVEyZpTcRVZ79Y7q+fva O65DzSk/7aC6lTtQMAzOtUQYEMLB4mMibBXWIRoYfdLTChkPBgM7sI2nwp5ngk2Jm8ob P7vlb6tNyrVPqWBqd516E3ubbBzcHSTA3kyXt2UWsh5NFoBDlGv86MuixuO4NC9fqd30 sJPvjsgKSYoSNZQa+Ex4dgXDjV6phgSNQB51BeY6nSTPgreKEQ4IhCU93zP0DMRDxcgm MPDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961553; x=1702566353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=BK+k5wBTIE268KBqMnb4pzLVVAsh7XVF5nBfQ+IBmk/EZJq8LikdzN9U9BQgo3BZ5N v0IscFDvqFI+cRGKc3w+Tf5eK+LCwKqpFPAUXjY+pHRH1UyE5Q1S2KBPmo4lLwdUqASK PEovqfy8HbbBw1aIYU3hAensAJq1vtLc36ITz2vdG8/+Vh9z7kxFYcJ8Aw6jacTpLgn0 h6GzVBdz9huvDTBiQ/nkc8UMdjKEuWlwYT4MX0S78Le4CC0FbxARnKVjI0o4kdsp7p9S VV61p10FBfXtB2Wp2p3VYRUb2GI51opCJp79WGRA69sflXy0mlFIZDkIqkRK03ugTI/3 CWng== X-Gm-Message-State: AOJu0Yx5jEvKwuuq7mOPfplYXRu+5nZsoPTFL7RKdpn1jDxv6iTYAN8Y xCVWwby9pyoa5DldlI12Wpcv5A== X-Google-Smtp-Source: AGHT+IEj2QTBGpOf3OHBs9Yni4aQareAokfWZfEZoiM8ehZh4/VLAukwcmzFcEIzO5CULnzn0YcNZw== X-Received: by 2002:adf:ea82:0:b0:333:1907:c2a3 with SMTP id s2-20020adfea82000000b003331907c2a3mr1503404wrm.21.1701961553264; Thu, 07 Dec 2023 07:05:53 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p13-20020a056000018d00b00333415503a7sm1644486wrx.22.2023.12.07.07.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:05:53 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 2/4] riscv: Add a runtime detection of invalid TLB entries caching Date: Thu, 7 Dec 2023 16:03:46 +0100 Message-Id: <20231207150348.82096-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0CDBF4000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nhpp3uoawhe3fdyj138qjcb1kpop5tg3 X-HE-Tag: 1701961554-922309 X-HE-Meta: U2FsdGVkX1+FlsYhol75AvnTVVrK9n6TX39qJI+pRhqpkwP7jClT5dNS3eAQhLQAxiRGVN1IVFh/Pu5T/dcpQxIR8Pjtx+5TGLOFPmlgT7EZBoIzZDT+cyYNLPElHxnI/fR3KRXI7ybHzdSbcFgMymgcv9ICgc8SxNck28YC81mIS9to4FWmusEV3Q3BiYVBHRS9H2vGum7wuMfrL9igAowquD1TrRhDaBLXDGRErqK8O/fJu4jDqSWaK8sIRVvzUTgM8de5SRWv5LRoY+PFFs4kaVxf+iOcDS223sgSm0phxlZU2zkCuRPWIvr8VWzmCmo7xpMGG+Eyf4wARxyiu9s5V75I1URauYmVWKYVh8KBlBv0jZVQNZjAmTaKpxvMMX4ILvCV7/T6+xalMKstDcp9mRB93X1+/fAbevYwmji9w8YZlIzWNtwWrDTsqweLbgekPEFqFRy7zGMv05zxydoZ39VfaNNWRaC6p6hBJRSpVUO71X45KqNriFZdOZ+cbl9gQ2tZjdaUA4w5ShC7RwXxJtshjkAqTVn550gjWOKYT0aIpCZ2K6yKxnGBnEYXjaU/EEQ1kQ6yfDZxMX5qlIXSPpRqHhUt4ppPvKu7qOtgKRTDTVek/Xs6SLbXud/5NikRoqolv0VppPzHbvVvzOdgNi4M0YeU003yvIKdtbZwNTE4fijkOLAVwy19BfWr9nosaEjXncvWnDQDyeddvbML+c9llJqJp63PMBmZDd2rhy9LdOQ/LXQdYtOnCnUJ1F/fshMMnzftLALp5mKTjPT6+baKOQOlyhuYeJGtnGC5YORLWg99OqJpGrksg4s3LYPfu4tc7NXaCOqU15wseo6Fqs/HkP+Rw2yF8VsSklj+5oq2hBaKPEhOqKGbdpM/2/qnivLjTT3GjjHo/X8/ajsyq682WoLdXamyfEr/ldI0J2YpFocyqeCM9n0OKEwFbD05AOjcQrwhbL7b0r+ 5I8law20 DWNipoqn2WKcovy+vuBHi1455iHtQKaL5RuKGkVnI/DOLwlMQ8tFDXqiGg88dG9kNyyWJebnmjFCM68E/Q5sMi0vwVd6cHpvDhCRZyf3z/N/STUKycJH0EAdDXVIkTsuEIG/8hg5EebVejjB/Rt6dZ7sVt22r3IB9PTKy4R8hpdl9spWco7vM4BTU71Ywfxa5pRiBfbyuZKrq5g3bzxVBQ1mE4rGuV3xcj3zvtQmMF/in0e1PN94vBXF466YYB7v5gzgYTWvCUOCnjq6WKuoJkWNwhHk9FKRWDNT/xWCtAXByHbluUq4/beQ+fsyxDN1amgAvMJGcVabQ6nVBoLhJ96p8hrabDrb9pavRvb/c5Nd8p1p13+LN4BFGwGLiK+X4evnRV5ynQDiiBtsMWIo9f+ve5kWhbFmKHoCD1TPpHd2buPMyzq28HSaP/v/OaGbvYQsBZuQJpmec4tc5xEovz79C90ouBvWIjQH7TEj/5N0pqmE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This mechanism allows to completely bypass the sfence.vma introduced by the previous commit for uarchs that do not cache invalid TLB entries. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/init.c | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 379403de6c6f..2e854613740c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -56,6 +56,8 @@ bool pgtable_l5_enabled = IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KER EXPORT_SYMBOL(pgtable_l4_enabled); EXPORT_SYMBOL(pgtable_l5_enabled); +bool tlb_caching_invalid_entries; + phys_addr_t phys_ram_base __ro_after_init; EXPORT_SYMBOL(phys_ram_base); @@ -750,6 +752,18 @@ static void __init disable_pgtable_l4(void) satp_mode = SATP_MODE_39; } +static void __init enable_pgtable_l5(void) +{ + pgtable_l5_enabled = true; + satp_mode = SATP_MODE_57; +} + +static void __init enable_pgtable_l4(void) +{ + pgtable_l4_enabled = true; + satp_mode = SATP_MODE_48; +} + static int __init print_no4lvl(char *p) { pr_info("Disabled 4-level and 5-level paging"); @@ -826,6 +840,112 @@ static __init void set_satp_mode(uintptr_t dtb_pa) memset(early_pud, 0, PAGE_SIZE); memset(early_pmd, 0, PAGE_SIZE); } + +/* Determine at runtime if the uarch caches invalid TLB entries */ +static __init void set_tlb_caching_invalid_entries(void) +{ +#define NR_RETRIES_CACHING_INVALID_ENTRIES 50 + uintptr_t set_tlb_caching_invalid_entries_pmd = ((unsigned long)set_tlb_caching_invalid_entries) & PMD_MASK; + // TODO the test_addr as defined below could go into another pud... + uintptr_t test_addr = set_tlb_caching_invalid_entries_pmd + 2 * PMD_SIZE; + pmd_t valid_pmd; + u64 satp; + int i = 0; + + /* To ease the page table creation */ + disable_pgtable_l5(); + disable_pgtable_l4(); + + /* Establish a mapping for set_tlb_caching_invalid_entries() in sv39 */ + create_pgd_mapping(early_pg_dir, + set_tlb_caching_invalid_entries_pmd, + (uintptr_t)early_pmd, + PGDIR_SIZE, PAGE_TABLE); + + /* Handle the case where set_tlb_caching_invalid_entries straddles 2 PMDs */ + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd, + set_tlb_caching_invalid_entries_pmd, + PMD_SIZE, PAGE_KERNEL_EXEC); + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + PMD_SIZE, PAGE_KERNEL_EXEC); + + /* Establish an invalid mapping */ + create_pmd_mapping(early_pmd, test_addr, 0, PMD_SIZE, __pgprot(0)); + + /* Precompute the valid pmd here because the mapping for pfn_pmd() won't exist */ + valid_pmd = pfn_pmd(PFN_DOWN(set_tlb_caching_invalid_entries_pmd), PAGE_KERNEL); + + local_flush_tlb_all(); + satp = PFN_DOWN((uintptr_t)&early_pg_dir) | SATP_MODE_39; + csr_write(CSR_SATP, satp); + + /* + * Set stvec to after the trapping access, access this invalid mapping + * and legitimately trap + */ + // TODO: Should I save the previous stvec? +#define ASM_STR(x) __ASM_STR(x) + asm volatile( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + "ld a0, 0(%0) \n" + ".align 2 \n" + "1: \n" + : + : "r" (test_addr) + : "a0" + ); + + /* Now establish a valid mapping to check if the invalid one is cached */ + early_pmd[pmd_index(test_addr)] = valid_pmd; + + /* + * Access the valid mapping multiple times: indeed, we can't use + * sfence.vma as a barrier to make sure the cpu did not reorder accesses + * so we may trap even if the uarch does not cache invalid entries. By + * trying a few times, we make sure that those uarchs will see the right + * mapping at some point. + */ + + i = NR_RETRIES_CACHING_INVALID_ENTRIES; + +#define ASM_STR(x) __ASM_STR(x) + asm_volatile_goto( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + ".align 2 \n" + "1: \n" + "addi %0, %0, -1 \n" + "blt %0, zero, %l[caching_invalid_entries] \n" + "ld a0, 0(%1) \n" + : + : "r" (i), "r" (test_addr) + : "a0" + : caching_invalid_entries + ); + + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + /* If we don't trap, the uarch does not cache invalid entries! */ + tlb_caching_invalid_entries = false; + goto clean; + +caching_invalid_entries: + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + tlb_caching_invalid_entries = true; +clean: + memset(early_pg_dir, 0, PAGE_SIZE); + memset(early_pmd, 0, PAGE_SIZE); + + enable_pgtable_l4(); + enable_pgtable_l5(); +} #endif /* @@ -1072,6 +1192,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) #endif #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) + set_tlb_caching_invalid_entries(); set_satp_mode(dtb_pa); #endif @@ -1322,6 +1443,9 @@ static void __init setup_vm_final(void) local_flush_tlb_all(); pt_ops_set_late(); + + pr_info("uarch caches invalid entries: %s", + tlb_caching_invalid_entries ? "yes" : "no"); } #else asmlinkage void __init setup_vm(uintptr_t dtb_pa) -- 2.39.2