From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8515CC6FD1D for ; Tue, 21 Mar 2023 03:54:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9E006B0075; Mon, 20 Mar 2023 23:54:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C29486B0078; Mon, 20 Mar 2023 23:54:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC7C16B007B; Mon, 20 Mar 2023 23:54:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 99E266B0075 for ; Mon, 20 Mar 2023 23:54:49 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5E24040534 for ; Tue, 21 Mar 2023 03:54:49 +0000 (UTC) X-FDA: 80591539098.16.74EB02F Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf30.hostedemail.com (Postfix) with ESMTP id 8531680007 for ; Tue, 21 Mar 2023 03:54:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TmYhZNLl; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679370887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hoABFC0Xy1EF5COqSJmS8vasqT4dLgmRzbs++Rzv7g8=; b=aSQ1xXNHo4nsJMwcgcGKGz3ltk6GVX+CGD0bhNaYkTNxD52Q3Um3xagY3UV0TH+VaOB017 sqt4OcZI/hs/k1nA+COSWgQ0qIfLVzrd/CWVDkCqc+QbOa536yjel5LZ6QuhnxH9Rh9CWq x/P+OQW4fbUxdpclnCiwf1dBZVaI/m0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TmYhZNLl; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679370887; a=rsa-sha256; cv=none; b=2gKfG1r7d6B5keK4YcWZgotEhoyKvCg/PQLqSZEs8oopk9VKHzRNmY30NgGBVuOXD4EYRg d3eUmzv0gkgzMrT6CujD6Y+jK1oEyC+SH/cDG+O409Ya/TK9Cp6Nom91ISaz0gRI81EoMO bCvkwNnn3FZkFNM6wTdi957JS1Cmbbo= Received: by mail-pj1-f44.google.com with SMTP id lr16-20020a17090b4b9000b0023f187954acso14638530pjb.2 for ; Mon, 20 Mar 2023 20:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679370886; h=in-reply-to:references:subject:cc:to:from:message-id:date :content-transfer-encoding:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=hoABFC0Xy1EF5COqSJmS8vasqT4dLgmRzbs++Rzv7g8=; b=TmYhZNLluP9YJBu0GCH8ISUhXDlv7gr5BVpFGydQG0SbjYIa4Pe5xCrgs7NZ+dV/jx yw/i2n1cYrQWf6wKcdhfvFEcKP6OClbYe2TXvkVeGFYAuFPUhw/VHjYoqBoRHkku9Yyo YWWC3zkApLYJjIfkchNY0gSQ686GHeCh8LCT9SN4yJcp6kFJ7yk/++obcTGJ3T89oCYQ NUGzX2q87r3NVUfFZZVx/0cQMslHEuTRIDh3RqPyRRfEnZRpLDxVAQhSPAYgX7uX3RxX HmzZO0YicX5sEei74jmJ2OHFKaWouOtUX/Dp/VGusHFxDC3CxmnpbBoWnl6TCEib2N+C u+VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679370886; h=in-reply-to:references:subject:cc:to:from:message-id:date :content-transfer-encoding:mime-version:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=hoABFC0Xy1EF5COqSJmS8vasqT4dLgmRzbs++Rzv7g8=; b=39X4ZHuQlArRja6bbZvXRuqfjPpNWOt0Ljls2yqPsoswuZC1BXlYOagGmIJA0n9EoK vaSPPsk6PmEoohxfUqicHFHg/5maiOwVwISqeDlu7ZTeinT1Zf2KaUa+ar/PJYd+x2qF +22sLXDESO7u3O3PxU81MyNzidWXjC7XHxsevXvTN6m/CJ+nrx/bzGJtOuOkgLBZ61WO qdH3yJBiLXNOck0FLXR6YFLBHP5GMp6qXUeRUtIsdoUAFviksvMR1NviewgYL66FN6cv zJPobuG17aoqKoviLQqP7+Wd6YgczcCIDo+U9eUWv5+dYlIkDGbiMBaQ/cJwGwd9bfa4 L1Gw== X-Gm-Message-State: AO0yUKX38fLkVHyiriGcvc5f+yKBz7CywLSxAQ2uP17AzMhEbvbj4oOz zOrd/xe27ohci/0pzn54ju4= X-Google-Smtp-Source: AK7set9bW6MOw9JnTLFSfH/FO7ROJZl4Ze+GGQm42g/O8jmbrO7D12Z+lVRicSWvohwxG9WJlWXqGg== X-Received: by 2002:a17:90b:4f8a:b0:237:24eb:99d8 with SMTP id qe10-20020a17090b4f8a00b0023724eb99d8mr1184759pjb.19.1679370886224; Mon, 20 Mar 2023 20:54:46 -0700 (PDT) Received: from localhost (121-44-69-75.tpgi.com.au. [121.44.69.75]) by smtp.gmail.com with ESMTPSA id a3-20020a637f03000000b005034a46fbf7sm6872626pgd.28.2023.03.20.20.54.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Mar 2023 20:54:45 -0700 (PDT) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Tue, 21 Mar 2023 13:54:38 +1000 Message-Id: From: "Nicholas Piggin" To: "Peter Zijlstra" , "Andrew Morton" Cc: "Linus Torvalds" , "Nadav Amit" , "Dave Hansen" , "Andy Lutomirski" , "Catalin Marinas" , "Will Deacon" , "Rik van Riel" , , , , "Andy Lutomirski" Subject: Re: [PATCH v7 5/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN X-Mailer: aerc 0.13.0 References: <20230203071837.1136453-1-npiggin@gmail.com> <20230203071837.1136453-6-npiggin@gmail.com> <20230226141238.6ec5fdf7d75dcf2cd4c58ba0@linux-foundation.org> In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8531680007 X-Rspam-User: X-Stat-Signature: 5kzetndk438pk8nz31hspdr9e8m8ob7t X-HE-Tag: 1679370887-736662 X-HE-Meta: U2FsdGVkX1/8We6g9z99nRALQokCZsfAqXAOPuwSvKgkPwZa04WG3xobNQkBB0z0ql2gYF7HquarDEqqeApmr7bCcodrkDSQ3zvhNOTj/dU+691fyIPASxehzOOiCkYkUD8wmjOip8c0LkpFA+eKdQDnD8Wj34ZQlLpYUEzqfJo39zZ+v/yfLrqvT90I9dATBFCTYcdfjmITIx7kfcrCcWa9cOZUitpiMQE/QFH3sQnVofuJu27mUY7Kbz/kkw0rC1v527Yk1rm0qGicCBgGATESSoxblcPyfESIa2IKtg2XdlXZlD8oPIwoCrWcKBY40JtWMANdAtuRpAz6GoWWYx+KM2ozl1bubs0LT+cSUFjPb8gouY58nGnbRKXGTKjcnnpc5sw+zOqFsZkMJoFQaOp1h6aN2djAZMhiHiM8mRWB3uKkJ7wnmu91reBOlBm1SOfnyHWiz8u6ila3kXSkzmq377Jhws/8n4Tnqtfhc3rZ1Q8+nqLCYYV9FnirGYlbkx/inUkTO9r4PXBeTeMw7joMTy5JLRncbPjCPVkbvaR1V890sPdG8SGMntzkO1YfrFtw97tUgYnkuA6HCSq/GQJocUDWiGc9Mdul1nH3ni9oWAiNQCHeeNtuEdAzaPMMQqOWevGmOG+v9G1zC3IMhYrQPcOvz6B85HOj947SmNuwgoYqYXYuIHdH75X8KSx8jWbqffi8lPpqjTTFNngzzg2ueMz05os3KyR3sHFsRCXlynoOJO+slw6a2zOMPPFpPM9M/rYV3OFD8Y+9QgbISPA02UXb6OAXRxlwZdmqRi/cvl3uixhvtrpBZzrdTTDedy5hfQ4BFSOcQXhiiNVMh/qh4WjWRqCSykMfSEJsVPE4DhOnk2ImJNhbjO3xGqbFq8N6Tfnf+TXDaIMUD4sXiWlMujMno2QrBRXe/fCIgA8uxsNkKIidaVRGRRfmwyHNfBVfU8DItSYN4HhfYik A5xcnHSl BKFdByf/BPdP59r6RzZDyNQkwEPi5sAAOElH08vVFGawod2GBSfCODLl3IjDGgB7okotj0vvblQpIbQyKoh2qyshPfz5MBDe54DH8A3/ewGf7zryaF4k2rbBbCKv8cckGBNjOqTyh9LcS/KPqYPPVckiFr2QP2fP9Oi21fUxqsRjvR7JGzdeEdQcuYoCmI1T9wT12npxScCZ4Xkb7xpQjS1t6UThA4WzjqK7/mIiVtjx+gqM/vYz/B/0A7L0symENhqyrPX1LBQtLbiCKa7W4gbvq1S/8H94lGqS4fxnlq6nHag4uTykpjfQzFvt6mRgLYPN+A0YE99MV6Y/mP6XTbHjQFzh0iwTIGiNBfe5AOTFCPyn9HdscGj3N/pxf+IevV1JrGA7NSQ+pjA+ZH4Z90D8K6C+ETCw4ZYxWKonzNBDhTYxfLYHW3UG8s1MCRkwbJJsRac7K6s0Kg/+Us1WLUCuum+KTvzpGeBQiXBepFxYDz6pKXoJczad8OsGVqqihb5sSDNDsljP/F2bgDILiGkKENI2WfNkCTxcYU9/pJ8nghoPj6cPTrhH7thgo70dhOlgzTr0kztOIm+LLVIa47Trw/Mu2cRPm9cAZMbU2NwMOnZjBUawoJ+EtaHvnpX/ITO/oIBGixuYD2JKRm84D7o2080xbZ+3BfO6q3mdzr+1jkbU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon Feb 27, 2023 at 11:33 PM AEST, Peter Zijlstra wrote: > On Sun, Feb 26, 2023 at 02:12:38PM -0800, Andrew Morton wrote: > > On Fri, 3 Feb 2023 17:18:37 +1000 Nicholas Piggin = wrote: > >=20 > > > On a 16-socket 192-core POWER8 system, the context_switch1_threads > > > benchmark from will-it-scale (see earlier changelog), upstream can > > > achieve a rate of about 1 million context switches per second, due to > > > contention on the mm refcount. > > >=20 > > > 64s meets the prerequisites for CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so ena= ble > > > the option. This increases the above benchmark to 118 million context > > > switches per second. > >=20 > > Is that the best you can do ;) > >=20 > > > This generates 314 additional IPI interrupts on a 144 CPU system doin= g > > > a kernel compile, which is in the noise in terms of kernel cycles. > > >=20 > > > ... > > > > > > --- a/arch/powerpc/Kconfig > > > +++ b/arch/powerpc/Kconfig > > > @@ -265,6 +265,7 @@ config PPC > > > select MMU_GATHER_PAGE_SIZE > > > select MMU_GATHER_RCU_TABLE_FREE > > > select MMU_GATHER_MERGE_VMAS > > > + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 > > > select MODULES_USE_ELF_RELA > > > select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE > > > select NEED_PER_CPU_EMBED_FIRST_CHUNK if PPC64 > >=20 > > Can we please have a summary of which other architectures might benefit > > from this, and what must they do? Coming back to this... The recipes to enable are somewhat documented I Kconfig. If those weren't clear I can improve or.. not sure where else to add this stuff. It would be nice if all these options had more explanation and requirements, I'm just not sure what's going to work best (beyond what I did in Kconfig). Not much noise from other archs so far, so I'll take a guess and say archs that have large SMP systems might. x86 and s390 perhaps. Seems to be some work still ongoing in the x86 branch, I didn't hear if you found the docs inadequate or any suggestions to improve understanding? Some were very confused by it, but I was never able to help them grasp the concepts or get to the bottom of what the problem was, so that was a dead end unfortunately. > >=20 > > As this is powerpc-only, I expect it won't get a lot of testing in > > mm.git or in linux-next. The powerpc maintainers might choose to merge > > in the mm-stable branch at > > git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm if this is a > > concern. > > I haven't really had time to page all of this back in, but x86 is very > close to be able to use this, it mostly just needs cleaning up some > accidental active_mm usage. > > I've got a branch here: > > https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h= =3Dx86/lazy > > That's mostly Nick's patches with a bunch of Andy's old patches stuck on > top. I also have a pile of notes, but alas, not finished in any way. Great that a proof of concept shows it can work for x86, I guess that's an ack for this series from x86? :) x86 implementation presumably won't be merged until objectionable active_mm and other code in core code that makes things difficult for the arch is cleaned up so we don't get into the situation again where crap keeps getting built on crap and everybody else's nice clean patches gets nacked for years because one arch is festering. Will be great to see those cleanups. Thanks, Nick