From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F99C10DC3 for ; Thu, 7 Dec 2023 15:04:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 799DB6B0083; Thu, 7 Dec 2023 10:04:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74A246B0088; Thu, 7 Dec 2023 10:04:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EA9C6B0089; Thu, 7 Dec 2023 10:04:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4A4916B0083 for ; Thu, 7 Dec 2023 10:04:44 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E634E4027B for ; Thu, 7 Dec 2023 15:04:43 +0000 (UTC) X-FDA: 81540344046.12.BB1CC14 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf05.hostedemail.com (Postfix) with ESMTP id B6291100235 for ; Thu, 7 Dec 2023 15:03:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=PwmuMOo5; dmarc=none; spf=pass (imf05.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701961433; a=rsa-sha256; cv=none; b=wmDBUWnB26CtAKLTfF1ZSk9e23x0UL7xuEFlJnNpp3bE68apIn1369QFu584v/ezVdIp3X an36s4mcE9n8LypDjfE8iAn5cflXiUnqm45aaclwm47TzY4bI+pYC/Oony+56l0leEgjg4 5KMpuw6AmWrNiq9yRrUTP0vDLZaeuVo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=PwmuMOo5; dmarc=none; spf=pass (imf05.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701961433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; b=nGUCJalpwW5snUCMl7QyubOfuxzOBBry6xgIhKc9mUdPYBAPl1CXfb/3QleLU2TXfEGZLq gcmPggZ4NxsspJN4Db6/gHz4GPfoM5LBSijRnelhZUzfz2xFCh7auhMqVVaoxzXhu66jcC sULIn/HoE6/RA7Xn3clFZAYQ6L1z7hI= Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-40c09d0b045so12651445e9.0 for ; Thu, 07 Dec 2023 07:03:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961431; x=1702566231; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; b=PwmuMOo5QKf6yBzVgbvcI9xuiyh3VJe/fuIWLiReuS2sMiMpBWbKs7JzUthpmZgmxk 1TsU/rn3hHz74vi9g2X9tOIg1MYiHQ4nRNsBWgoCmJfJPca2MUN9zHFwHgLnDhLePjQ9 C1PEWAhRRatFzdn2paoEdLfwuzo+Hb28mJJFLsZz2jMhe3au+ie7MIOMepyoJhbi/0ev avE4+gKxNBxPB4iO/sWUq+Li67CTdPhJdgqkPzYn1qERBmuiyP+kCGTyFYNInklrn7sT zC8fmfb+E63IYzDXot7dTujIzFpToGUE8eJ7WS2KsOYgteBknZegZtrEuev6k/XufdF9 heLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961431; x=1702566231; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; b=G/uL+J1ODPo6C3Aqh669TEzcpglbMuoQf5A72BE7YOHc/TMniVRPHyqmD1vkO2Cfxe bmSpeaOGxOa29mhZwgZsSM6v1FfEqMiymDNnMTmuP7mzC5OoxwleF+TbKP2ebWCCIW3z BOpZKby3uioOtTKfHE9Kwi/tVnRnc/fXqp/KJQxDO6+G5W9o6m5ruzU25zWM+JR8WHHP UY2EAO0rgNksMnMN9mmRuQQeGe0rLp71NVZmS+1remQsTo/b8x9QjHSR7jip3UDaw3Yx 5h6lW/DckWmTVInlr3LnOufHhwZ7c1j6HMLOvjtkjORkm5ja0F8BgccuFT1MiwwrYH3R 9crQ== X-Gm-Message-State: AOJu0YwzulbhnH2zDNQRxBgJnXk2Fwm26ssbwdJpWDs723ec/yy5Dkt9 xmfm3ZuzbllKu6mQcTGHomYAFw== X-Google-Smtp-Source: AGHT+IEfU8bcWPltllq4UxddBOusN19bKPp7FZtm1tNb3JVSGuXfSF/1fbnNg3hbcLptFO8P4+JaSw== X-Received: by 2002:a05:600c:198b:b0:40c:5ee:2dda with SMTP id t11-20020a05600c198b00b0040c05ee2ddamr1118146wmq.177.1701961430948; Thu, 07 Dec 2023 07:03:50 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b0040b42df75fcsm2187533wmq.39.2023.12.07.07.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:03:50 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 0/4] Remove preventive sfence.vma Date: Thu, 7 Dec 2023 16:03:44 +0100 Message-Id: <20231207150348.82096-1-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B6291100235 X-Stat-Signature: ju4qdq3eaehbjo9pgr47bma5zwnwshd4 X-HE-Tag: 1701961432-312493 X-HE-Meta: U2FsdGVkX1/nIcMkSa1g8f9sdMtSjmYKSRNRhmizA2/WzxpvQdUXXCRF6FduPy3V+VAuDwoEBp2k0eIOgzZNsDO4GoKpVY8AnSRCIBpyXfQ58GfR0PpZV4IlHxg4MKq4lYjws210YXbngqvSJdYudTdPhRbOBhW3pxL6KII6fUWk3uIISdiM9InMVlLi6kemYthwxTOnhzTVshGHPDSUJoPvJi7xqiZa65W810iHJeHj77l3Nl8wMtrQeycQXCOlvA5itC3mhZhimIBrXt75R2HYyDCX749ZB6H+ZUAUGFyx5pQrt836O/AlBa7xbC4DaoLNnBM6GKr5z6XhGfqYBVkmQw8G5ahP5G231JqHj4quSooUHV05IbttXF1e2LHtybC0w3deLoZ7fThjrgCdumRGimvdj7uDmvf5IMrHrbsgHHIkNuwP7XLNHCXDSa2GlGlewfmQSlI/RwaoBrUESJSzGx7AQvKrDE26jmU2nA2JNFutR3LU+jAITQVJ9a5b89TuIVNxgGxwcK4mjpG3tDF16he4gzY3mqApx0QqTshtNOg69VF/qqpW+zddNA4VXkyj59kRlRkvEoLfwl2Q3QzqX09T4hIFwaErHWbzqyh09MogoTG65RFIi8kDF3k26sBzfMZm+jeojVYSvMpYD/zOgWg7MFsgmxlODugEbD6v6KNd+Ys3boV2miJglm9um4afliDI5JfJbghifG/4Bkg3/vptRGoKBTT3Oba/DoMHmw4o8/iTeZT9RoknkafiO0i0jZFPv03KKd7kFIztQ8ethJY2iUb4ZQG1sluk7xkfK78ZwIJ8QsNVNaUC3xywiOlSpHVGa/0bqKTT2w2yS/DHjCanVfu1HihuqrjI+/rV/Bh7gslJXv8YfSFlBFLVyVvF0TEQQpIbyQBtWON1A3Ipb58OVv2q9rgiv5Z7juoK1c4MejWkJl9zd4pmivMzzKI9Y3m0JMzpiuZsZee 10Mi2ldm JSaLZzJ0fmbaMQ83t1I4NNMKEyYkrIZNqK8DTruGkR4thQNyexxAl87US0n7O5n2by/fqL56BRG5AMNCT1XyHM4H36C/8m9Wg14XllNkSzbI/3F7G/O5Sa+6N//GBZMldJIllX/u0Gvf3jEsS2UBiIbmlurjNb6jELIGRdcxZhAtFBuCw3s4R3uLcyFe+bGt4QEYhcZWyDfv/hH+ERxYa1HdOdbDihc9GCBrJLHzCWv+SZhi7HM43qOxqU0MoNYwDu5RswImB/4DK4h2HM3o6hRNUqY6kfeQ4IWp1C2qRsLne+dboP1XhooIsS4gSXpOSHIqWLT6vH+YaZCWZtrwxXhv9cikFbJDUd2kVblV85NSUKxacdexcjnsyxftRXtxxWQt3Tpl/Em0w05eczjjSAqsInK3Ntvi7AG0Bmei0Zce78jQC+xZVKOF+Aw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In RISC-V, after a new mapping is established, a sfence.vma needs to be emitted for different reasons: - if the uarch caches invalid entries, we need to invalidate it otherwise we would trap on this invalid entry, - if the uarch does not cache invalid entries, a reordered access could fail to see the new mapping and then trap (sfence.vma acts as a fence). We can actually avoid emitting those (mostly) useless and costly sfence.vma by handling the traps instead: - for new kernel mappings: only vmalloc mappings need to be taken care of, other new mapping are rare and already emit the required sfence.vma if needed. That must be achieved very early in the exception path as explained in patch 1, and this also fixes our fragile way of dealing with vmalloc faults. - for new user mappings: that can be handled in the page fault path as done in patch 3. Patch 2 is certainly a TEMP patch which allows to detect at runtime if a uarch caches invalid TLB entries. Patch 4 is a TEMP patch which allows to expose through debugfs the different sfence.vma that are emitted, which can be used for benchmarking. On our uarch that does not cache invalid entries and a 6.5 kernel, the gains are measurable: * Kernel boot: 6% * ltp - mmapstress01: 8% * lmbench - lat_pagefault: 20% * lmbench - lat_mmap: 5% On uarchs that cache invalid entries, the results are more mitigated and need to be explored more thoroughly (if anyone is interested!): that can be explained by the extra page faults, which depending on "how much" the uarch caches invalid entries, could kill the benefits of removing the preventive sfence.vma. Ved Shanbhogue has prepared a new extension to be used by uarchs that do not cache invalid entries, which will certainly be used instead of patch 2. Thanks to Ved and Matt Evans for triggering the discussion that led to this patchset! That's an RFC, so please don't mind the checkpatch warnings and dirty comments. It applies on 6.6. Any feedback, test or relevant benchmark are welcome :) Alexandre Ghiti (4): riscv: Stop emitting preventive sfence.vma for new vmalloc mappings riscv: Add a runtime detection of invalid TLB entries caching riscv: Stop emitting preventive sfence.vma for new userspace mappings TEMP: riscv: Add debugfs interface to retrieve #sfence.vma arch/arm64/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +- arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 +- arch/riscv/include/asm/cacheflush.h | 19 ++- arch/riscv/include/asm/pgtable.h | 45 ++++--- arch/riscv/include/asm/thread_info.h | 5 + arch/riscv/include/asm/tlbflush.h | 4 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/entry.S | 94 +++++++++++++ arch/riscv/kernel/sbi.c | 12 ++ arch/riscv/mm/init.c | 126 ++++++++++++++++++ arch/riscv/mm/tlbflush.c | 17 +++ include/linux/pgtable.h | 8 +- mm/memory.c | 12 +- 14 files changed, 331 insertions(+), 32 deletions(-) -- 2.39.2