From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DADCFC83014 for ; Wed, 2 Dec 2020 14:20:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1EF16221FC for ; Wed, 2 Dec 2020 14:20:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EF16221FC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2AA438D0002; Wed, 2 Dec 2020 09:20:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 25C068D0001; Wed, 2 Dec 2020 09:20:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14BD98D0002; Wed, 2 Dec 2020 09:20:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id EFCC48D0001 for ; Wed, 2 Dec 2020 09:20:06 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A7EC41F08 for ; Wed, 2 Dec 2020 14:20:06 +0000 (UTC) X-FDA: 77548551612.03.chair17_63120bb273b3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 5C5B528A4EA for ; Wed, 2 Dec 2020 14:20:06 +0000 (UTC) X-HE-Tag: chair17_63120bb273b3 X-Filterd-Recvd-Size: 6666 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 14:20:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=zrRkDip9syURLU9E0Xb2jeKaoBtCaWsSoFLaKvOtats=; b=Rsz1p5mkhJCX7CZVz3x+qAHtrt +IcT5x25jXpFmuWiw1PKnwQg2YBu/MuxQY8FpoPQ0RNIDZw1b2ewQQYueRC/wdqveh5rwAKK3WVJO s5vwq/RGYckBzfx7tiWTLX6l7qnh/l0w5qOlhW0WVz5q/EYUCIVDjzkE36UJ1A5wjT/PmIEvyitLR sP44sspX+zHD22KRxoHLpq4r8N74MZ8ZKqMVqmkSL4Fskoi2s3eyftVzrJjxmDUGAvKcCzJ1io6cG nlb/Mie3gyBSOCv7tR9HG/ubvCe5pQ7CAayYYCTZC8Kc5g8uMQqn3y0CoWn5u+Ash/btKS01xcNHz esFETNAg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkSz9-0003nY-UA; Wed, 02 Dec 2020 14:20:00 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CC1E7305C10; Wed, 2 Dec 2020 15:19:57 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id AFCB52143477A; Wed, 2 Dec 2020 15:19:57 +0100 (CET) Date: Wed, 2 Dec 2020 15:19:57 +0100 From: Peter Zijlstra To: Nicholas Piggin Cc: linux-kernel@vger.kernel.org, x86@kernel.org, Mathieu Desnoyers , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard Subject: Re: [PATCH 6/8] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Message-ID: <20201202141957.GJ3021@hirez.programming.kicks-ass.net> References: <20201128160141.1003903-1-npiggin@gmail.com> <20201128160141.1003903-7-npiggin@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201128160141.1003903-7-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Nov 29, 2020 at 02:01:39AM +1000, Nicholas Piggin wrote: > + * - A delayed freeing and RCU-like quiescing sequence based on > + * mm switching to avoid IPIs completely. That one's interesting too. so basically you want to count switch_mm() invocations on each CPU. Then, periodically snapshot the counter on each CPU, and when they've all changed, increment a global counter. Then, you snapshot the global counter and wait for it to increment (twice I think, the first increment might already be in progress). The only question here is what should drive this machinery.. the tick probably. This shouldn't be too hard to do I think. Something a little like so perhaps? diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 41404afb7f4c..27b64a60a468 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4525,6 +4525,7 @@ context_switch(struct rq *rq, struct task_struct *prev, * finish_task_switch()'s mmdrop(). */ switch_mm_irqs_off(prev->active_mm, next->mm, next); + rq->nr_mm_switches++; if (!prev->mm) { // from kernel /* will mmdrop() in finish_task_switch(). */ @@ -4739,6 +4740,80 @@ unsigned long long task_sched_runtime(struct task_struct *p) return ns; } +static DEFINE_PER_CPU(unsigned long[2], mm_switches); + +static struct { + unsigned long __percpu *switches[2]; + unsigned long generation; + atomic_t complete; + struct wait_queue_dead wait; +} mm_foo = { + .switches = &mm_switches, + .generation = 0, + .complete = -1, // XXX bootstrap, hotplug + .wait = __WAIT_QUEUE_HEAD_INITIALIZER(mm_foo.wait), +}; + +static void mm_gen_tick(int cpu, struct rq *rq) +{ + unsigned long prev, curr, switches = rq->nr_mm_switches; + int idx = READ_ONCE(mm_foo.generation) & 1; + + /* DATA-DEP on mm_foo.generation */ + + prev = __this_cpu_read(mm_foo.switches[idx^1]); + curr = __this_cpu_read(mm_foo.switches[idx]); + + /* we haven't switched since the last generation */ + if (prev == switches) + return false; + + __this_cpu_write(mm_foo.switches[idx], switches); + + /* + * If @curr is less than @prev, this is the first update of + * this generation, per the above, switches has also increased since, + * so mark out CPU complete. + */ + if ((long)(curr - prev) < 0 && atomic_dec_and_test(&mm_foo.complete)) { + /* + * All CPUs are complete, IOW they all switched at least once + * since the last generation. Reset the completion counter and + * increment the generation. + */ + atomic_set(&mm_foo.complete, nr_online_cpus()); + /* + * Matches the address dependency above: + * + * idx = gen & 1 complete = nr_cpus + * + * curr = sw[idx] generation++; + * prev = sw[idx^1] + * if (curr < prev) + * complete-- + * + * If we don't observe the new generation; we'll not decrement. If we + * do see the new generation, we must also see the new completion count. + */ + smp_wmb(); + mm_foo.generation++; + return true; + } + + return false; +} + +static void mm_gen_wake(void) +{ + wake_up_all(&mm_foo.wait); +} + +static void mm_gen_wait(void) +{ + unsigned int gen = READ_ONCE(mm_foo.generation); + wait_event(&mm_foo.wait, READ_ONCE(mm_foo.generation) - gen > 1); +} + /* * This function gets called by the timer code, with HZ frequency. * We call it with interrupts disabled. @@ -4750,6 +4825,7 @@ void scheduler_tick(void) struct task_struct *curr = rq->curr; struct rq_flags rf; unsigned long thermal_pressure; + bool wake_mm_gen; arch_scale_freq_tick(); sched_clock_tick(); @@ -4763,8 +4839,13 @@ void scheduler_tick(void) calc_global_load_tick(rq); psi_task_tick(rq); + wake_mm_gen = mm_gen_tick(cpu, rq); + rq_unlock(rq, &rf); + if (wake_mm_gen) + mm_gen_wake(); + perf_event_task_tick(); #ifdef CONFIG_SMP diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bf9d8da7d35e..62fb685db8d0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -927,6 +927,7 @@ struct rq { unsigned int ttwu_pending; #endif u64 nr_switches; + u64 nr_mm_switches; #ifdef CONFIG_UCLAMP_TASK /* Utilization clamp values based on CPU's RUNNABLE tasks */