From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1C54C636CC for ; Sun, 5 Feb 2023 19:49:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B0916B0072; Sun, 5 Feb 2023 14:49:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 260C36B0073; Sun, 5 Feb 2023 14:49:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14F286B0074; Sun, 5 Feb 2023 14:49:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 065D16B0072 for ; Sun, 5 Feb 2023 14:49:30 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C310B14042C for ; Sun, 5 Feb 2023 19:49:29 +0000 (UTC) X-FDA: 80434277658.16.B3FF9C0 Received: from out-131.mta0.migadu.com (out-131.mta0.migadu.com [91.218.175.131]) by imf02.hostedemail.com (Postfix) with ESMTP id C903C8000E for ; Sun, 5 Feb 2023 19:49:27 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=EiKIcs3K; spf=pass (imf02.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.131 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675626568; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zCXR2fdD5KUizgsD+r8xIXos+I6YOtrXfgJwLA6SDKs=; b=PRAimRLHVD/nNigsyl2bCCr9hJcx39koIEHk+oFGnJJKtrbmJPhQ3pTjCb4O+OnlxB1z8e 8oCp1YAi8WZ7PmQXE2uEcEE1+xPHMoHs00ncy5giotE5S38Y5Yix+JsWLrs2rxdxVKjUqt aBnbq9w4h2zP/TvrgqkIXmMkZsl8LjA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=EiKIcs3K; spf=pass (imf02.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.131 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675626568; a=rsa-sha256; cv=none; b=wk4D8Vz/KD95mwjuM+Og1J0pGzPXfx5tcHtzOpw/TBsLaAxVDEiikFB0eENLMIaF/XPBp2 qnOog5Z3rxEpWgjMmhOepm0YLKMtPMDISEh+ePFDZ8hB/Pv0TA+WJ+Av26ikno4dOZnAxg h7KVVEj1VkgAKieYc/YsCzmR+JlPUmw= Date: Sun, 5 Feb 2023 11:49:13 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1675626564; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zCXR2fdD5KUizgsD+r8xIXos+I6YOtrXfgJwLA6SDKs=; b=EiKIcs3Klx+YgeINRkbrjeT48IQMnlUunMLev5pz3djUhjV6YTD/1jYT489gdALY33FZCx dSbnndEJe4EaJCa9nw0etCNxIxWraIHACeTmUQS0DGR5VCNsZsBBcLx7vhAaIdbA07JJPu 9kXpNSVt5VA+kI0R8AAL2XXGsKB+Ues= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Leonardo =?iso-8859-1?Q?Br=E1s?= Cc: Michal Hocko , Marcelo Tosatti , Johannes Weiner , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 0/5] Introduce memcg_stock_pcp remote draining Message-ID: References: <9e61ab53e1419a144f774b95230b789244895424.camel@redhat.com> <0122005439ffb7895efda7a1a67992cbe41392fe.camel@redhat.com> <28e08669302ad1e7a41bdf8b9988de6a352b5fe1.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <28e08669302ad1e7a41bdf8b9988de6a352b5fe1.camel@redhat.com> X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C903C8000E X-Stat-Signature: tg7gowbiehmfeobxuwr6eesr9g13yj6y X-HE-Tag: 1675626567-589768 X-HE-Meta: U2FsdGVkX19N9YxBAmbBlkgJKvzaFdpYCnPzX+6Sxup/AogSO9M+11w/5yAgSumfL0xdYG74N7LvO9LtXuQ4ti9PuW73aoBaXr9YP2pJxyLfR1vwp5s5dJDAYL13q7EGJxPELszEKkUIZ9viRC9deeQRAtZL+vR2vNxvrpuidkSemTL6k7X4gQqvMIcpe17jizH4TV9PEn543KH1JfqfkCFR/v43qXZUKiMbNo/qRQli4fvmIk0VZINbcfIdIs4ViebgD6/eupeWNQWfvBcU2AsooRDz93WddVbvBCLu+5jUa9s5Eop9UUkqTOIXa0ziPMJ6nsMyyavLDQa5Qg5u4TPogDJlV+38g5SYtGAsZ174KBezmzMjSRnNlBMCEPiIL0gjzR1tzy01XXhW9DAqhBbeJXwnMpuSIPQuwUbY1n60FSKlpHBgCLeP91Tmyn8ucnB8co42IBLs4b2FHZrG8wVI5dQIl71tvK59G1F6LWv/ftJdj1vhJzlBF5mcD8CH3+1eEFlrAwPHdYmMIxdKVea5YtmICi/WtAePjdfd1rCs582B9xl+a1o/66CjIqdITzVEgslXC7etCS7lLBHuuGZlxInHl8mZYk+JlH9WcuxnB4hYWDefGCguYqdcG1XKuaBZpqALFOb9dNyNNBqCt7/HwO72Zof1aACCLXg9UZlONrnHaHSOAcmga6vAsxQrx47xX6binhJx4Yp8H2bGH+aTl1E2U7snYujQ94tv8hYhbjXLEOXssgjFoAMEh/v3qJqHzzezZOM5HXO4tostxpYlKryCvy+/xrusNLVY5Uk8aefwiSykf3/bJ8dX0S6MV3XSH3WkzSGsqbtHQ94bjoWxVPStrX4YiMbeK+qtDNCRdpajffB85Ajv+2fV46xpXBV6BEJ7cTjhO6HXO6mswT/cjle6nHSi7FZzSoasf6l7r1XQbdDYYd1l5/M4wZlHQTywH3f5b1a7gX21ZO4 YbA0sXBO G8YEpWj6SucvOR/UnafYqTpsHXgDNed9lpqNMJxRa5qGW2AQO68c5GkA5x2bgoat6FbEfBBSksKlX5iZNT0eV45Tqhk4MZDuH3AeVk7FWtlq/xyCkd3XlmCrlt1wn65S14I+S9XxPPPjsU2M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Leonardo! > Yes, but we are exchanging an "always schedule_work_on()", which is a kind of > contention, for a "sometimes we hit spinlock contention". > > For the spinlock proposal, on the local cpu side, the *worst case* contention > is: > 1 - wait the spin_unlock() for a complete , > 2 - wait a cache hit for local per-cpu cacheline  > > What is current implemented (schedule_work_on() approach), for the local > cpu side there is *always* this contention: > 1 - wait for a context switch, > 2 - wait a cache hit from it's local per-cpu cacheline, > 3 - wait a complete ,  > 4 - then for a new context switch to the current thread. I think both Michal and me are thinking of a more generic case in which the cpu is not exclusively consumed by 1 special process, so that the draining work can be executed during an idle time. In this case the work is basically free. And the introduction of a spin_lock() on the hot path is what we're are concerned about. I agree, that on some hardware platforms it won't be that expensive, but in general not having any spinlocks is so much better. > > So moving from schedule_work_on() to spinlocks will save 2 context switches per > cpu every time drain_all_stock() is called. > > On the remote cpu side, my tests point that doing the remote draining is faster > than scheduling a local draining, so it's also a gain. > > Also, IIUC the possible contention in the spinlock approach happens only on > page-faulting and syscalls, versus the schedule_work_on() approach that can > interrupt user workload at any time.  > > In fact, not interrupting the user workload in isolated cpus is just a bonus of > using spinlocks. I believe it significantly depends on the preemption model: you're right regarding fully preemptive kernels, but with voluntary/none preemption it's exactly opposite: the draining work will be executed at some point later (probably with 0 cost), while the remote access from another cpu will potentially cause delays on the spin lock as well as a need to refill the stock. Overall I'd expect a noticeable performance regression from an introduction of spin locks and remote draining. Maybe not on all platforms, but at least on some. That's my main concern. And I don't think the problem we're aiming to solve here justifies this potential regression. Thanks!