From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2460DC64E7B for ; Tue, 1 Dec 2020 21:50:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 685CF208DB for ; Tue, 1 Dec 2020 21:50:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="cYaRYuHV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 685CF208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AC1596B0036; Tue, 1 Dec 2020 16:50:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A720C8D0002; Tue, 1 Dec 2020 16:50:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 960DB8D0001; Tue, 1 Dec 2020 16:50:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 7A1376B0036 for ; Tue, 1 Dec 2020 16:50:56 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 427732496 for ; Tue, 1 Dec 2020 21:50:56 +0000 (UTC) X-FDA: 77546058912.06.basin78_3417bca273ad Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 224C81003FB28 for ; Tue, 1 Dec 2020 21:50:56 +0000 (UTC) X-HE-Tag: basin78_3417bca273ad X-Filterd-Recvd-Size: 7242 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 1 Dec 2020 21:50:55 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1294422241 for ; Tue, 1 Dec 2020 21:50:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606859454; bh=R0u/t9dXTAtlGviFs2oYeCEEHjWlPsXgdRNYm6TSXfc=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=cYaRYuHVHFbMApZUMrH9vnKsdRglxiaUhXeHk2PFyc9icoIoHfWGY00FhNGuyu56E zh5mtAt+rmfJPCsCD3pvXYVLTLoYITBv6zabhI/ulLzoU09TWkFU9392r9O6XOFxPQ vmV3ti3MsO4EsPXsAzOp6NyNDn5IzSDbOYI4iEsc= Received: by mail-wr1-f53.google.com with SMTP id r3so5207265wrt.2 for ; Tue, 01 Dec 2020 13:50:53 -0800 (PST) X-Gm-Message-State: AOAM530/1JTCqjkWvZ9mn0rTGrROBExqWJ1J4I90OmYZCgYqzhds8r0N ZibrrtI3TZ9TXdQxpWKW07zcndsLg8vhEteSfPCPKg== X-Google-Smtp-Source: ABdhPJxU0mZkcyHFXZdJ4FhJRmKxyc24rWEoozINfeAF09HSf49gsQ5zWpDYiKpwtAGuhEHFY/rC7uAY3CBEYtIBXQg= X-Received: by 2002:adf:e449:: with SMTP id t9mr6472923wrm.257.1606859452360; Tue, 01 Dec 2020 13:50:52 -0800 (PST) MIME-Version: 1.0 References: <20201128160141.1003903-1-npiggin@gmail.com> <20201128160141.1003903-7-npiggin@gmail.com> <20201201212758.GA28300@willie-the-truck> In-Reply-To: <20201201212758.GA28300@willie-the-truck> From: Andy Lutomirski Date: Tue, 1 Dec 2020 13:50:38 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 6/8] lazy tlb: shoot lazies, a non-refcounting lazy tlb option To: Will Deacon Cc: Andy Lutomirski , Catalin Marinas , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Dave Hansen , Nicholas Piggin , LKML , X86 ML , Mathieu Desnoyers , Arnd Bergmann , Peter Zijlstra , linux-arch , linuxppc-dev , Linux-MM , Anton Blanchard Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 1, 2020 at 1:28 PM Will Deacon wrote: > > On Mon, Nov 30, 2020 at 10:31:51AM -0800, Andy Lutomirski wrote: > > other arch folk: there's some background here: > > > > https://lkml.kernel.org/r/CALCETrVXUbe8LfNn-Qs+DzrOQaiw+sFUg1J047yByV31SaTOZw@mail.gmail.com > > > > On Sun, Nov 29, 2020 at 12:16 PM Andy Lutomirski wrote: > > > > > > On Sat, Nov 28, 2020 at 7:54 PM Andy Lutomirski wrote: > > > > > > > > On Sat, Nov 28, 2020 at 8:02 AM Nicholas Piggin wrote: > > > > > > > > > > On big systems, the mm refcount can become highly contented when doing > > > > > a lot of context switching with threaded applications (particularly > > > > > switching between the idle thread and an application thread). > > > > > > > > > > Abandoning lazy tlb slows switching down quite a bit in the important > > > > > user->idle->user cases, so so instead implement a non-refcounted scheme > > > > > that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down > > > > > any remaining lazy ones. > > > > > > > > > > Shootdown IPIs are some concern, but they have not been observed to be > > > > > a big problem with this scheme (the powerpc implementation generated > > > > > 314 additional interrupts on a 144 CPU system during a kernel compile). > > > > > There are a number of strategies that could be employed to reduce IPIs > > > > > if they turn out to be a problem for some workload. > > > > > > > > I'm still wondering whether we can do even better. > > > > > > > > > > Hold on a sec.. __mmput() unmaps VMAs, frees pagetables, and flushes > > > the TLB. On x86, this will shoot down all lazies as long as even a > > > single pagetable was freed. (Or at least it will if we don't have a > > > serious bug, but the code seems okay. We'll hit pmd_free_tlb, which > > > sets tlb->freed_tables, which will trigger the IPI.) So, on > > > architectures like x86, the shootdown approach should be free. The > > > only way it ought to have any excess IPIs is if we have CPUs in > > > mm_cpumask() that don't need IPI to free pagetables, which could > > > happen on paravirt. > > > > Indeed, on x86, we do this: > > > > [ 11.558844] flush_tlb_mm_range.cold+0x18/0x1d > > [ 11.559905] tlb_finish_mmu+0x10e/0x1a0 > > [ 11.561068] exit_mmap+0xc8/0x1a0 > > [ 11.561932] mmput+0x29/0xd0 > > [ 11.562688] do_exit+0x316/0xa90 > > [ 11.563588] do_group_exit+0x34/0xb0 > > [ 11.564476] __x64_sys_exit_group+0xf/0x10 > > [ 11.565512] do_syscall_64+0x34/0x50 > > > > and we have info->freed_tables set. > > > > What are the architectures that have large systems like? > > > > x86: we already zap lazies, so it should cost basically nothing to do > > a little loop at the end of __mmput() to make sure that no lazies are > > left. If we care about paravirt performance, we could implement one > > of the optimizations I mentioned above to fix up the refcounts instead > > of sending an IPI to any remaining lazies. > > > > arm64: AFAICT arm64's flush uses magic arm64 hardware support for > > remote flushes, so any lazy mm references will still exist after > > exit_mmap(). (arm64 uses lazy TLB, right?) So this is kind of like > > the x86 paravirt case. Are there large enough arm64 systems that any > > of this matters? > > Yes, there are large arm64 systems where performance of TLB invalidation > matters, but they're either niche (supercomputers) or not readily available > (NUMA boxes). > > But anyway, we blow away the TLB for everybody in tlb_finish_mmu() after > freeing the page-tables. We have an optimisation to avoid flushing if > we're just unmapping leaf entries when the mm is going away, but we don't > have a choice once we get to actually reclaiming the page-tables. > > One thing I probably should mention, though, is that we don't maintain > mm_cpumask() because we're not able to benefit from it and the atomic > update is a waste of time. Do you do anything special for lazy TLB or do you just use the generic code? (i.e. where do your user pagetables point when you go from a user task to idle or to a kernel thread?) Do you end up with all cpus set in mm_cpumask or can you have the mm loaded on a CPU that isn't in mm_cpumask? --Andy > > Will