From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78CE0C433F5 for ; Wed, 19 Jan 2022 17:33:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E46586B0072; Wed, 19 Jan 2022 12:33:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF55B6B0073; Wed, 19 Jan 2022 12:33:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE3AD6B0074; Wed, 19 Jan 2022 12:33:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id C08476B0072 for ; Wed, 19 Jan 2022 12:33:28 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 79822181CBC14 for ; Wed, 19 Jan 2022 17:33:28 +0000 (UTC) X-FDA: 79047733296.23.55BD546 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf16.hostedemail.com (Postfix) with ESMTP id 0D41B18000C for ; Wed, 19 Jan 2022 17:33:27 +0000 (UTC) Received: by mail-wm1-f45.google.com with SMTP id v123so6606536wme.2 for ; Wed, 19 Jan 2022 09:33:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=E5e/gbMD1CI1oNnHoXsVfQit0AqJzsZNFNoMFZIa3yo=; b=WmT8mo/Jdy5x0dloa8gkKdAaIf8hr7M/ebJkJWZ/RWfXSCx+/MARi8gUcqkp0XtFrz rM4cstaSkdu91SBqo5K1zXG+ti72XZ/IUGrNAP1vnrxuXWgBuJAYro2LtQFxLvNWTi2h 5ZLXWm28rZ68n3PX5lNuQSmhYOXMThUfK5Yym7fixUJZchEpMDZldGD9byToOQc830RZ ZsQUsTuwqNoAOm7g/BSTjFs/Igm3qC02l+4IiR33HANapb8mfs/42RNpC8utTrfLiEXt m6InW8giQaNDO6owF9gYElq7AsEsiL6GURWMQCvwfB9V+FDTY7KuNjzNK0sjcNIEQTL1 2fBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=E5e/gbMD1CI1oNnHoXsVfQit0AqJzsZNFNoMFZIa3yo=; b=h+MGbCy5NS0+cBOnomtDBBwUkt7JsPGQO+kjrirtQM3A0BBfG+2FZmpClaQXbNwooT zklA+ws9/vJ2BituynjRB7tWbaCK8TDBSuclUlgdsVvVZvcKCW6L2gS2fijEXsggYEgV I6C0osnNpgrAMvtM2qDZUIVxAs+degnl5rP+W5a3g1PaWZzrDb2Hk5WaZ8ilL8RLlMko bCfa39GzeIeLBE5ggkdH4oKG1Sp1ePcRnq2BZfURyp5FgUsxWKcMSb6rmKk/2Fu7eEex OAENyxkiGTwqqu92US0sgobzm3P5975kzDsxOLVa89Qz1p+LoLzcuUZitT3GjG0R7b8d Q8Vw== X-Gm-Message-State: AOAM531jJLXqE9CAMamr3aS5mOgAVf4d4euJH26s+/rFldb2xtmBqZBS 8syViYQkJRVmdmWiOng6xcPKh1vReAM5uLdC6uluuA== X-Google-Smtp-Source: ABdhPJwu7kUqdN9jzIk+hqALXPHegLrqP/weejs07XlgrmOWp0FWwVtOZH3CpffoM+v4k+wrQFDN0VBR42C/FK14AnQ= X-Received: by 2002:adf:9dc7:: with SMTP id q7mr6096353wre.148.1642613606589; Wed, 19 Jan 2022 09:33:26 -0800 (PST) MIME-Version: 1.0 References: <20211214204445.665580974@infradead.org> <20211214205358.701701555@infradead.org> <20211221171900.GA580323@dev-hv> In-Reply-To: From: Peter Oskolkov Date: Wed, 19 Jan 2022 09:33:15 -0800 Message-ID: Subject: Re: [RFC][PATCH 3/3] sched: User Mode Concurency Groups To: Peter Zijlstra Cc: Peter Oskolkov , mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org, pjt@google.com, avagin@google.com, jannh@google.com, tdelisle@uwaterloo.ca Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 0D41B18000C X-Stat-Signature: 3p7jgrnukui74wns7f1fhjjosefq4ffz Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="WmT8mo/J"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of posk@google.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=posk@google.com X-Rspamd-Server: rspam02 X-HE-Tag: 1642613607-565240 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 19, 2022 at 12:47 AM Peter Zijlstra wrote: > > On Tue, Jan 18, 2022 at 10:19:21AM -0800, Peter Oskolkov wrote: > > ============= worker-to-worker context switches > > > > One example: absl::Mutex (https://abseil.io/about/design/mutex) has > > google-internal extensions that are "fiber aware". More specifically, > > consider this situation: > > > > - worker W1 acqured the mutex and is doing its work > > - worker W2 calls mutex::lock() > > mutex::lock(), being aware of workers, understands that W2 is going to sleep; > > so instead of just doing so, waking the server, and letting > > the server figure out what to run in place of the sleeping worker, > > mutex::lock() > > calls into the userspace scheduler in the context of W2 running, and the > > userspace scheduler then picks W3 to run and does W2->W3 context switch. > > > > The optimization above replaces W2->Server and Server->W3 context switches > > with a single W2->W3 context switch, which is a material performance gain. > > Yes, I've also already reconsidered. Things like pipelines and other > fixed order scheduling policies will greatly benefit from > worker-to-worker switching. > > But I think all of them are explicit. That is, we can limit the > ::next_tid usage to sys_umcg_wait() and never look at it for implicit > blocks. Yes, of course - when a worker blocks, its server gets notified. > > > In addition, when W1 calls mutex::unlock(), the scheduling code determines > > that W2 is waiting on the mutex, and thus calls W2::wake() from the context of > > running W1 (you asked earlier why do we need "WAKE_ONLY"). > > This I'm not at all convinced on. That sounds like it will violate the > 1:1 thing. wake_only is a wakeup event, meaning the worker gets added to the wake queue, not scheduled on a CPU; we don't have to implement it in the kernel, though - the userspace may keep its own wake queue for workers like this. So feel free to ignore this operation.