From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E08F4C433F5 for ; Mon, 24 Jan 2022 09:49:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BD356B0081; Mon, 24 Jan 2022 04:49:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 56C936B0083; Mon, 24 Jan 2022 04:49:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45B7B6B0085; Mon, 24 Jan 2022 04:49:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 344C96B0081 for ; Mon, 24 Jan 2022 04:49:04 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E40238249980 for ; Mon, 24 Jan 2022 09:49:03 +0000 (UTC) X-FDA: 79064706966.24.5AD1CB4 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf21.hostedemail.com (Postfix) with ESMTP id 3B9E71C0010 for ; Mon, 24 Jan 2022 09:49:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=U9bFxdNKcDMMzkGN9dN7g8aA0mdb5+NJJNY3O8L2ma4=; b=B9qYNR+xIyROYlCnwaSlY6zegz bxZLShFS4UwlNBJCvIOLbwTcmBRhczE3DWhTiB32BwCxoyTMLpUeK1X47ePxCHyQsCOpF+ZsxMZRq 9GdaeQnzjLusTcfi9izTGzRsO/nV+XBefDo3PXM+TP2hTtBRAg2hT+xWBVjOJJ6tXqy8GkRsN1JMn N9XtS/8i/Jx228S76FhnXpOVRxU5zJR0HixKNTh6SjHf0WoKEb03UInibcW1YDf5nrY3ngUFPkz0C f7jDr6HiejuS6BCx0CGTeS86SRle8wjLCFBC9aj+BW2KHhFsi0eR4ca89Ck9pSLZ6UqFiGfp/s9mT G9VOp5QA==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=worktop.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nBvxv-00389R-Kb; Mon, 24 Jan 2022 09:48:47 +0000 Received: by worktop.programming.kicks-ass.net (Postfix, from userid 1000) id 3332A986245; Mon, 24 Jan 2022 10:48:46 +0100 (CET) Date: Mon, 24 Jan 2022 10:48:46 +0100 From: Peter Zijlstra To: Mark Rutland Cc: mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org, pjt@google.com, posk@google.com, avagin@google.com, jannh@google.com, tdelisle@uwaterloo.ca, posk@posk.io Subject: Re: [RFC][PATCH v2 5/5] sched: User Mode Concurency Groups Message-ID: <20220124094846.GN20638@worktop.programming.kicks-ass.net> References: <20220120155517.066795336@infradead.org> <20220120160822.914418096@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=B9qYNR+x; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3B9E71C0010 X-Stat-Signature: pmoj6i5aerzpb1x58he7yoixejustcgd X-HE-Tag: 1643017743-558015 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 21, 2022 at 04:57:29PM +0000, Mark Rutland wrote: > On Thu, Jan 20, 2022 at 04:55:22PM +0100, Peter Zijlstra wrote: > > User Managed Concurrency Groups is an M:N threading toolkit that allows > > constructing user space schedulers designed to efficiently manage > > heterogeneous in-process workloads while maintaining high CPU > > utilization (95%+). > > > > XXX moar changelog explaining how this is moar awesome than > > traditional user-space threading. > > Awaiting a commit message that I can parse, I'm just looking at the entry bits > for now. TBH I have no idea what this is actually trying to do... Ha! yes.. I knew I was going to have to do that eventually :-) It's basically a user-space scheduler that is subservient to the kernel scheduler (hierarchical scheduling were a user task is a server for other user tasks), where a server thread is in charge of selecting which of it's worker threads gets to run. The original idea was that each server only ever runs a single worker, but PeterO is currently trying to reconsider that. The *big* feature here, over traditional N:M scheduling, is that threads can block, while traditional userspace threading is limited to non-blocking system calls (and per later, page-faults). In order to make that happen we must ovbiously hook schedule() for these worker threads and inform userspace (the server thread) when this happens such that it can select another worker thread to go vroom. Meanwhile, a worker task getting woken from schedule() must not continue running; instead it must enter the server's ready-queue and await it's turn again. Instead of dealing with arbitrary delays deep inside the kernel block chain, we punt and let the task complete until return-to-user and block it there. The time between schedule() and return-to-user is unmanaged time. Now, since we can't readily poke at userspace memory from schedule(), we could be holding mmap_sem etc., we pin the worker and server page on sys-enter such that when we hit schedule() we can update state and then unpin the pages such that page pin time is from sys-enter to first schedule(), or sys-exit which ever comes first. This ensures the page-pin is *short* term. Additionally we must deal with signals :-(, the currnt approach is to let them bust boundaries and run them as unmanaged time. UMCG userspace can obviously control this by using pthread_sigmask() and friends. Now, the reason for irqentry_irq_enable() is mostly #PF. When a worker faults and blocks we want the same things to happen. Anyway, so workers have 3 layers of hooks: sys_enter schedule() sys_exit return-to-user There's a bunch of paths through this: - sys_enter -> sys_exit: no blocking; nothing changes: - sys_enter: * pin pages - sys_exit: * unpin pages - sys_enter -> schedule() -> sys_exit: we did block: - sys_enter: * pin pages - schedule(): * mark worker BLOCKED * wake server (it will observe it's current worker !RUNNING and select a new worker or idles) * unpin pages - sys_exit(): * mark worker RUNNABLE * enqueue worker on server's runnable_list * wake server (which will observe a new runnable task, add it to whatever and if it was idle goes run, otherwise goes back to sleep to let it's current worker finish) * block until RUNNING - sys_enter -> schedule() -> sys_exit -> return_to_user: As above; except now we got a signal while !RUNNING. sys_exit() terminates and return-to-user takes over running the signal and on return from the signal we'll again block until RUNNING, or do the whole signal dance again if so required. Does this clarify things a little?