From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D9A1C433EF for ; Mon, 17 Jan 2022 09:19:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B67CD6B0071; Mon, 17 Jan 2022 04:19:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B17426B0073; Mon, 17 Jan 2022 04:19:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A05C56B0074; Mon, 17 Jan 2022 04:19:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 906D86B0071 for ; Mon, 17 Jan 2022 04:19:52 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3870C181CB152 for ; Mon, 17 Jan 2022 09:19:52 +0000 (UTC) X-FDA: 79039231824.25.B2FDB04 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 4124A1C000A for ; Mon, 17 Jan 2022 09:19:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=FxsJjdfKuu5bMpXR0hew/7OYz/lzco51HZJYdo8kNlI=; b=UUjoeWidKnxmT0H8VFs6CIVpR2 1+Bu5gzqg+I9eL+G+KtR+IHQ2bIy+s8dn4Q2aV2B+H1iAfF5i1jlGPjhBX5Au0Hd0V/Uc/C1NZS3l ZSK2QHgJRH9R59nPcLm6P+Y5bHyuAtQ19JVYFq07+gzN4HrTPAtJ5PPhVxUZlZ1+jGl7aLFHQQZBr mGsSVGfbjjfgGxYyFGxtJfmciMtstDSqb9FYSEtOBrEZnmenlMFwzdYe8dtzPtB9CcdCC/bNjpS1y mA4autKfmCFWgs49z1PqRD2cd3avmn3DSfE8aPcSJvWKnVNzEJm6N2v+VTUi5WP+VOxZp1Sg2M1ur tRoUWEhw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1n9OAj-007zug-Bm; Mon, 17 Jan 2022 09:19:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0FE09300222; Mon, 17 Jan 2022 10:19:26 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id 8F4B82C707D18; Mon, 17 Jan 2022 10:19:26 +0100 (CET) Date: Mon, 17 Jan 2022 10:19:26 +0100 From: Peter Zijlstra To: Peter Oskolkov Cc: mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org, pjt@google.com, avagin@google.com, jannh@google.com, tdelisle@uwaterloo.ca, posk@posk.io Subject: Re: [RFC PATCH v2 4/5] sched: UMCG: add a blocked worker list Message-ID: References: <20220113233940.3608440-1-posk@google.com> <20220113233940.3608440-5-posk@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220113233940.3608440-5-posk@google.com> X-Rspamd-Queue-Id: 4124A1C000A X-Stat-Signature: 7yjkednpmihqbqcb1cke3ke1p7zhf8dr Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UUjoeWid; dmarc=none; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org X-Rspamd-Server: rspam03 X-HE-Tag: 1642411191-695584 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jan 13, 2022 at 03:39:39PM -0800, Peter Oskolkov wrote: > The original idea of a UMCG server was that it was used as a proxy > for a CPU, so if a worker associated with the server is RUNNING, > the server itself is never ever was allowed to be RUNNING as well; > when umcg_wait() returned for a server, it meant that its worker > became BLOCKED. > > In the new (old?) "per server runqueues" model implemented in > the previous patch in this patchset, servers are woken when > a previously blocked worker on their runqueue finishes its blocking > operation, even if the currently RUNNING worker continues running. > > As now a server may run while a worker assigned to it is running, > the original idea of having at most a single worker RUNNING per > server, as a means to control the number of running workers, is > not really enforced, and the server, woken by a worker > doing BLOCKED=>RUNNABLE transition, may then call sys_umcg_wait() > with a second/third/etc. worker to run. > > Support this scenario by adding a blocked worker list: > when a worker transitions RUNNING=>BLOCKED, not only its server > is woken, but the worker is also added to the blocked worker list > of its server. > > This change introduces the following benefits: > - block detection how behaves similarly to wake detection; > without this patch worker wakeups added wakees to the list > and woke the server, while worker blocks only woke the server > without adding blocked workers to a list, forcing servers > to explicitly check worker's state; > - if the blocked worker woke sufficiently quickly, the server > woken on the block event would observe its worker now as > RUNNABLE, so the block event had to be inferred rather than > explicitly signalled by the worker being added to the blocked > worker list; > - it is now possible for a single server to control several > RUNNING workers, which makes writing userspace schedulers > simpler for smaller processes that do not need to scale beyond > one "server"; > - if the userspace wants to keep at most a single RUNNING worker > per server, and have multiple servers with their own runqueues, > this model is also naturally supported here. > > So this change basically decouples block/wake detection from > M:N threading in the sense that the number of servers is now > does not have to be M or N, but is more driven by the scalability > needs of the userspace application. So I don't object to having this blocking list, we had that early on in the discussions. *However*, combined with WF_CURRENT_CPU this 1:N userspace model doesn't really make sense, also combined with Proxy-Exec (if we ever get that sorted) it will fundamentally not work. More consideration is needed I think...