From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 307D4C77B7A for ; Thu, 25 May 2023 06:48:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6910D6B0074; Thu, 25 May 2023 02:48:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 640906B0075; Thu, 25 May 2023 02:48:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 508D7900002; Thu, 25 May 2023 02:48:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 427EF6B0074 for ; Thu, 25 May 2023 02:48:05 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EFFA41C72FD for ; Thu, 25 May 2023 06:48:04 +0000 (UTC) X-FDA: 80827847688.13.AF69D6E Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf29.hostedemail.com (Postfix) with ESMTP id 837D312000B for ; Thu, 25 May 2023 06:48:00 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b="ah0j6tO/"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf29.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684997280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H+juiSs8/r4vhTei6CJgEt//h9b/Y51gEXQ4E75Yfcc=; b=XWnbYBAmpvwnusEe6byAquwoVFLiLcBpuQ8NVc4ADg0esNBq1diyoi/gXmTG8/WQvjHXz0 MDnTZe8ubiJ+WJcmQfWIS2rFhBsm1NbdRGwYyaZcBXoSUlYC/fJV7WJjHfVWoWsRRdYFXX pCoXbH2YpaWStYVmDy/ZPOmEI/JnneA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b="ah0j6tO/"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf29.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684997280; a=rsa-sha256; cv=none; b=l4mgxUV+wPjybLbSHD9qMv+rlOeFXMFj9B07frVYCGlQAw8P2EttCAIESJLJIYd7W4xOzI CYjtJFLMM8PB2m52YIOE8zWdBZk0CUoGBYjsbcUMBv6RWfgzOsMki44t56Ti2V6p0PrLP+ CFu3BHMbDA4xK6znHP+aBKtA29nxkxU= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id CD37A21C45; Thu, 25 May 2023 06:47:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1684997278; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H+juiSs8/r4vhTei6CJgEt//h9b/Y51gEXQ4E75Yfcc=; b=ah0j6tO/OvF9OOdn3tDraQVrEAVJlxXWXvENO3yZhCIzUqKAk9B2HTVj0zFSNw5nPVO7Dy wN/lfth+QXJL783C3shhbBe+5DTkCQCL/MLbUZJWL5exmQdiHri7CsHLbohjvlWJsQmNzZ PeUG+4vyRbrcqKF/GpA9KBxpu7WIZZ8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 9EDC313356; Thu, 25 May 2023 06:47:58 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id A3xzJJ4Eb2SKBQAAMHmgww (envelope-from ); Thu, 25 May 2023 06:47:58 +0000 Date: Thu, 25 May 2023 08:47:57 +0200 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v8 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 837D312000B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: d4fm9bosdfyw5j5dyaozab3znqohjejo X-HE-Tag: 1684997280-357430 X-HE-Meta: U2FsdGVkX1+SHIZcwcprzW0FtGxNlb1KQQKLjTVLRuQ6YQVjjE3b/C+h0948TtXuqadu6vZQ0x/TwqMDADvucCfzo5/4g2Ua4K/VWBHbPh9+UKCp1XLjmyemqxDh5HeKDpMtsQhttttI4+2p8KDfjExEoHSCMlNaZil7m48JrPgw1CrlmhXzMhMWdUBDK7E8+Hmycm5cBapYcHi5GWO7M9PSvUa8kVQJkwGVdL6iitit2VlXJ9U/C7/55+F68h+s5r0DZa68njkIQlxrxAzTBfXWxP3nIPP5pqTW5bwjWPhR6uAZPahsdVHx2vArrrl7jZbTdDRq9Vw0+SqoYlFQCgAMBselx33LINTbWv6JIiLHxc1dVTpuqeiK67QR4tZNwsJJfIW9TDLfaGgxS0UMtU0I1cnIqNj4P2AZ7JZLhMYVFJgUHGGDAVkaBe4Jc0PCgUFZsm9JYnp+6CTcFOZYV+0CqEMCYCIfis4wDSaGF9c6RjCrnb67mmgUgC+UNTy6caeVfBl+K2Qg3E4RoYFF+PHwBVeKC28wdlP8/LNTUWdnnpgPB0Bn/mIzwKnjd7UOKMeaHlBNVNHc9b2lqQAChz0L6Oz2PBfIi94y1p7oggPV9+AdNJVw5MFVYNzmoWD7G+msV5Ztql4rNhlmwaF4uroQayTwERhPYfXfyPVemviHVEYUVWs1cOQHkczB9cwrZWxlXJBcyF1g4FGRvYG8Ev2FAjbY7W7ZhbjqcgPOkhCERdm0nL01j3t9QQKZsOhhkcxvTQOc4b+ybHb3VZ8drWGHW8fYBW+pgo3Z/V3yVHj0onqmhvBCYZRDvwZ2ySRW1iiwWueRaKhjZxxdJvKmSmvQ5+hYVs1vQXTcvccApvZUuE5Jtab0MmhFSBfBJBRIadOESoyc2NVK8E1Y49Cs6w9Di0etu9mvckmbyZd9vl+FFyiqKxir/x2qJA8hEwCGyMW07MbzuSFo0oyZiAl q8dOGLGD d1MNxVcGiQnwU1Acq5zp04y8cw8QkmmJcmoOSUk7ApALImXa6AkIVQHERkShttoWU6/DDGI2WUO3i9WU2kdXpixpCl0TMNSNRoK76cht9ukGweO/Bn2iJBxkAYyqrAygQg3RkDaskjdf5NJ0CmSk6sGD5Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 24-05-23 10:53:23, Marcelo Tosatti wrote: > On Wed, May 24, 2023 at 02:51:55PM +0200, Michal Hocko wrote: > > [Sorry for a late response but I was conferencing last two weeks and now > > catching up] > > > > On Mon 15-05-23 15:00:15, Marcelo Tosatti wrote: > > [...] > > > v8 > > > - Add summary of discussion on -v7 to cover letter > > > > Thanks this is very useful! This helps to frame the further discussion. > > > > I believe the most important question to answer is this in fact > > > I think what needs to be done is to avoid new queue_work_on() > > > users from being introduced in the tree (the number of > > > existing ones is finite and can therefore be fixed). > > > > > > Agree with the criticism here, however, i can't see other > > > options than the following: > > > > > > 1) Given an activity, which contains a sequence of instructions > > > to execute on a CPU, to change the algorithm > > > to execute that code remotely (therefore avoid interrupting a CPU), > > > or to avoid the interruption somehow (which must be dealt with > > > on a case-by-case basis). > > > > > > 2) To block that activity from happening in the first place, > > > for the sites where it can be blocked (that return errors to > > > userspace, for example). > > > > > > 3) Completly isolate the CPU from the kernel (off-line it). > > > > I agree that a reliable cpu isolation implementation needs to address > > queue_work_on problem. And it has to do that _realiably_. This cannot by > > achieved by an endless whack-a-mole and chasing each new instance. There > > must be a more systematic approach. One way would be to change the > > semantic of schedule_work_on and fail call for an isolated CPU. The > > caller would have a way to fallback and handle the operation by other > > means. E.g. vmstat could simply ignore folding pcp data because an > > imprecision shouldn't really matter. Other callers might chose to do the > > operation remotely. This is a lot of work, no doubt about that, but it > > is a long term maintainable solution that doesn't give you new surprises > > with any new released kernel. There are likely other remote interfaces > > that would need to follow that scheme. > > > > If the cpu isolation is not planned to be worth that time investment > > then I do not think it is also worth reducing a highly optimized vmstat > > code. These stats are invoked from many hot paths and per-cpu > > implementation has been optimized for that case. > > It is exactly the same code, but now with a "LOCK" prefix for CMPXCHG > instruction. Which should not cost much due to cache locking (these are > per-CPU variables anyway). Sorry but just a LOCK prefix for a hot path is not a serious argument. > > If your workload would > > like to avoid that as disturbing then you already have a quiet_vmstat > > precedence so find a way how to use it for your workload instead. > > > > -- > > Michal Hocko > > SUSE Labs > > OK so an alternative solution is to completly disable vmstat updates > for isolated CPUs. Are you OK with that ? Yes, the number of events should be reasonably small and those places in the kernel which really need a precise value need to do a per-cpu walk anyway. IIRC /proc/vmstat et al also do accumulate pcp state. But let me reiterate. Even with vmstat updates out of the game, you have so many other sources of disruption that your isolated workload will be fragile until you actually try to deal with the problem on a more fundamental level. -- Michal Hocko SUSE Labs