From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42BB6C64E8A for ; Wed, 2 Dec 2020 03:47:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9AD2A206A4 for ; Wed, 2 Dec 2020 03:47:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9AD2A206A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0935D6B005C; Tue, 1 Dec 2020 22:47:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 043CF8D0002; Tue, 1 Dec 2020 22:47:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E75C38D0001; Tue, 1 Dec 2020 22:47:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id D093B6B005C for ; Tue, 1 Dec 2020 22:47:49 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 95E1B180AD804 for ; Wed, 2 Dec 2020 03:47:49 +0000 (UTC) X-FDA: 77546958258.17.stove75_4801e04273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 7E74D180D0180 for ; Wed, 2 Dec 2020 03:47:49 +0000 (UTC) X-HE-Tag: stove75_4801e04273b0 X-Filterd-Recvd-Size: 9756 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 03:47:48 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id b23so351234pls.11 for ; Tue, 01 Dec 2020 19:47:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:subject:to:cc:references:in-reply-to:mime-version :message-id:content-transfer-encoding; bh=vso9fbKrp7rJuhxeZyuQMFTDzR5eCp91S2Q2UmwQNu8=; b=lNI3XBCFKt2LLxNR4G2aKs8QzeSwN8YoU49RmlOYwZwhWjw4zA9T4uk4bJt4038fio HF1AAiSIf+2LaxjPI/5ZoPjer3NeUrGzko76uTXIV1FVMhnXXBucKWcxcmpMUHkcI3eu C/bdlRXKe9gTSjQqv6gYD8Qv2XuBGcdRsZLea4+0Nexg8KNpYCVIplCa5rByseSCLDsg 8eAVsgDkJTG1nOljMO4ke/UQBBaBZT6ll4meohVY15rxBq4EOB/u5IwYekrurQNFMYKz FafYNj64nqYmZCsKlMjf2RnxlBW3z3m8fhVU1BPA39SZhrnG5PWaghQkMOMETRpBoz7j pqtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:subject:to:cc:references:in-reply-to :mime-version:message-id:content-transfer-encoding; bh=vso9fbKrp7rJuhxeZyuQMFTDzR5eCp91S2Q2UmwQNu8=; b=QZVwUp4ZCFJGSDmxTffNMQGjIL+F2WpG/42X/9d+vXFfak+ojUIe2yc+LiWvTFLcSL XuPyfeFFXRg9//pra2X0969gRxvME7tfPBXa57G+3HsTVAW5aqMwZNWKiR4qORdiQIpQ bWMAP211ELDpOypKOEU/LPT502ZH10mvSxwuxSXOkyZAIF+YwADBgJa2eVqroIYtsT3d jjVclshCl3QkH2AQBq2gESz3PMEM57pRF8nQMlTrxpN5btrjf19R1y5Eg6DEBYhr+Pxe yYQKkntxRNNQLwJR6uExiBj903RaorEJkVhRvU4hTMfv0jPhAph1hmuH8zs7s1dTT416 ZgXw== X-Gm-Message-State: AOAM532G0AJzeDmsOAGAsSdri6pE+PShnTaGwcQtnO0zfw5LCVguuSqq bscti/xl8dXiK6POIPlHUyo= X-Google-Smtp-Source: ABdhPJxXlpx2on+HCIB29S2yb5xH/4Wll0pcilrWx+Z8+P3dZi38+5/M+J7hVGfmNgjxUiHj+7OKCA== X-Received: by 2002:a17:90a:d249:: with SMTP id o9mr489390pjw.158.1606880867992; Tue, 01 Dec 2020 19:47:47 -0800 (PST) Received: from localhost ([1.132.176.4]) by smtp.gmail.com with ESMTPSA id q200sm384496pfq.95.2020.12.01.19.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 19:47:47 -0800 (PST) Date: Wed, 02 Dec 2020 13:47:40 +1000 From: Nicholas Piggin Subject: Re: [PATCH 6/8] lazy tlb: shoot lazies, a non-refcounting lazy tlb option To: Christian Borntraeger , Catalin Marinas , Dave Hansen , Vasily Gorbik , Heiko Carstens , Andy Lutomirski , Will Deacon Cc: Anton Blanchard , Arnd Bergmann , linux-arch , LKML , Linux-MM , linuxppc-dev , Mathieu Desnoyers , Peter Zijlstra , X86 ML References: <20201128160141.1003903-1-npiggin@gmail.com> <20201128160141.1003903-7-npiggin@gmail.com> In-Reply-To: MIME-Version: 1.0 Message-Id: <1606879302.tdngvs3yq4.astroid@bobo.none> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Excerpts from Andy Lutomirski's message of December 1, 2020 4:31 am: > other arch folk: there's some background here: >=20 > https://lkml.kernel.org/r/CALCETrVXUbe8LfNn-Qs+DzrOQaiw+sFUg1J047yByV31Sa= TOZw@mail.gmail.com >=20 > On Sun, Nov 29, 2020 at 12:16 PM Andy Lutomirski wrote: >> >> On Sat, Nov 28, 2020 at 7:54 PM Andy Lutomirski wrote: >> > >> > On Sat, Nov 28, 2020 at 8:02 AM Nicholas Piggin wr= ote: >> > > >> > > On big systems, the mm refcount can become highly contented when doi= ng >> > > a lot of context switching with threaded applications (particularly >> > > switching between the idle thread and an application thread). >> > > >> > > Abandoning lazy tlb slows switching down quite a bit in the importan= t >> > > user->idle->user cases, so so instead implement a non-refcounted sch= eme >> > > that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot d= own >> > > any remaining lazy ones. >> > > >> > > Shootdown IPIs are some concern, but they have not been observed to = be >> > > a big problem with this scheme (the powerpc implementation generated >> > > 314 additional interrupts on a 144 CPU system during a kernel compil= e). >> > > There are a number of strategies that could be employed to reduce IP= Is >> > > if they turn out to be a problem for some workload. >> > >> > I'm still wondering whether we can do even better. >> > >> >> Hold on a sec.. __mmput() unmaps VMAs, frees pagetables, and flushes >> the TLB. On x86, this will shoot down all lazies as long as even a >> single pagetable was freed. (Or at least it will if we don't have a >> serious bug, but the code seems okay. We'll hit pmd_free_tlb, which >> sets tlb->freed_tables, which will trigger the IPI.) So, on >> architectures like x86, the shootdown approach should be free. The >> only way it ought to have any excess IPIs is if we have CPUs in >> mm_cpumask() that don't need IPI to free pagetables, which could >> happen on paravirt. >=20 > Indeed, on x86, we do this: >=20 > [ 11.558844] flush_tlb_mm_range.cold+0x18/0x1d > [ 11.559905] tlb_finish_mmu+0x10e/0x1a0 > [ 11.561068] exit_mmap+0xc8/0x1a0 > [ 11.561932] mmput+0x29/0xd0 > [ 11.562688] do_exit+0x316/0xa90 > [ 11.563588] do_group_exit+0x34/0xb0 > [ 11.564476] __x64_sys_exit_group+0xf/0x10 > [ 11.565512] do_syscall_64+0x34/0x50 >=20 > and we have info->freed_tables set. >=20 > What are the architectures that have large systems like? >=20 > x86: we already zap lazies, so it should cost basically nothing to do This is not zapping lazies, this is freeing the user page tables. "lazy mm" is where a switch to a kernel thread takes on the previous mm for its kernel mapping rather than switch to init_mm. > a little loop at the end of __mmput() to make sure that no lazies are > left. If we care about paravirt performance, we could implement one > of the optimizations I mentioned above to fix up the refcounts instead > of sending an IPI to any remaining lazies. It might be possible x86's scheme you could scan mm_cpumask carefully synchronized or something when the last user reference gets dropped that frees the lazy at that point, but I don't know what that would buy you because you're still having to maintain the mm_cpumask on switches. powerpc's characteristics are just different here so it makes sense whereas I don't know if it would on x86. >=20 > arm64: AFAICT arm64's flush uses magic arm64 hardware support for > remote flushes, so any lazy mm references will still exist after > exit_mmap(). (arm64 uses lazy TLB, right?) So this is kind of like > the x86 paravirt case. Are there large enough arm64 systems that any > of this matters? >=20 > s390x: The code has too many acronyms for me to understand it fully, > but I think it's more or less the same situation as arm64. How big do > s390x systems come? >=20 > power: Ridiculously complicated, seems to vary by system and kernel confi= g. >=20 > So, Nick, your unconditional IPI scheme is apparently a big > improvement for power, and it should be an improvement and have low > cost for x86. As said, the tradeoffs are different, I'm not so sure. It was a big=20 improvement on a very big system with the powerpc mm_cpumask switching model on a microbenchmark designed to stress this, which is about all I can say for it. > On arm64 and s390x it will add more IPIs on process > exit but reduce contention on context switching depending on how lazy > TLB works. I suppose we could try it for all architectures without > any further optimizations. It will remain opt-in but certainly try it out and see. There are some requirements as documented in the config option text. > Or we could try one of the perhaps > excessively clever improvements I linked above. arm64, s390x people, > what do you think? >=20 I'm not against improvements to the scheme. e.g., from the patch + /* + * IPI overheads have not found to be expensive, but they c= ould + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and fr= eed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching awa= y + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence base= d on + * mm switching to avoid IPIs completely. + */ But would like to have numbers before being too clever. Thanks, Nick