From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 381DAC6FD1F for ; Wed, 22 Mar 2023 10:13:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE8D56B0075; Wed, 22 Mar 2023 06:13:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A98CF6B0078; Wed, 22 Mar 2023 06:13:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 962576B007B; Wed, 22 Mar 2023 06:13:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 84F896B0075 for ; Wed, 22 Mar 2023 06:13:06 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 53F471C4E91 for ; Wed, 22 Mar 2023 10:13:06 +0000 (UTC) X-FDA: 80596121172.14.A5981A5 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf22.hostedemail.com (Postfix) with ESMTP id 7115CC001E for ; Wed, 22 Mar 2023 10:13:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=nTml06Ve; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf22.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679479984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aqEzHI7CsoF2HtxRsuwP9+5W2QuYHXY8VrSFADFSMLM=; b=LMI4D5mcIjLE5j25/RkIRkBZsLP7BV4stkuXyxxju6vF67NBR4TgWdmDa/yx9Jq5+c6nRC TZ38MdzXys3uFvrf6OvzXuM2lxz07J2HyXi+Cl5hYkUomEkyIs1Yc6eaz/6cOqpQsD9E23 f6BhXVYUm2zx5l5mqJsHLNMPWA6R5Vc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=nTml06Ve; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf22.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679479984; a=rsa-sha256; cv=none; b=khkZ1IQEedBPC1Ns7IT3Uy/S7cgKN0f5hQRsWNhCfdMl8tUhSGdJ1GLiqUBgkxICwgrCaV mjWoh1QJ5QrelqkivZwAPv6yBNdmA4TB2JHoY7pDgJvgO/H1TVrvJVU9RdO5rnlGWWmcF+ XAYehV4H2CygmohRbBQBBQXLp68Hw90= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E1CE121B45; Wed, 22 Mar 2023 10:13:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1679479982; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=aqEzHI7CsoF2HtxRsuwP9+5W2QuYHXY8VrSFADFSMLM=; b=nTml06VeJDqK9PEr/pRjJsAJ13bkHpV7QyzOi3MENEr+VHVAZfGLpu7kGmaBJDt56cpijR XFCxNyk/Y+5V0PanR6I9OwVkyv1QVrxosPmUVQVW9B8jXleMxpxNNu3STvpHnpzJoXVLIA +QRg5R4/1vUhgJI0ZuXXbspr/NhFqOE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BC5C7138E9; Wed, 22 Mar 2023 10:13:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id yuIuKq7UGmT5CQAAMHmgww (envelope-from ); Wed, 22 Mar 2023 10:13:02 +0000 Date: Wed, 22 Mar 2023 11:13:02 +0100 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7115CC001E X-Stat-Signature: q7mghb8p8rynpx7bej9zyfk8kob3ocg3 X-HE-Tag: 1679479984-528548 X-HE-Meta: U2FsdGVkX1+qffryMIOBQg03A2jcANoaPMUsTKb/HRK5sXJB4XRmdlEAyIj0zXSFyHnOV1N1cIDz5WlOsKG8h08m6xcUISodeDP9rYs/Nci2xqZ2T4ynHv/eGAqmfV7FhYZGSReNTIXIPiCm37pIqmOWYsVpr16dsRTC91rLTUJw6S7cwPoxH8AkjbKw8bZpPT4EgTuGRLigPWyGZpz0166FGS9n1/T+dxNqAmAwbEpwFtGIjVoixCNyjlJ4qK7eaA/P/HfyPTfVWZV2BB09rYd8S85tlN39DQMbfmPyuJdXNXkH9xnZlFc+H2khBEzMtPokLo6iG4Cd5sAJn5BJfL5zXPHwULecta+JDMmpURsDy+s+aodRkW9e2z5QRSTPAddD2QIY+bh0lJx5psZ6HUKJLTMOcZrv5s0EQsbGn+xnfnPqNJKAttKCf7+RL9wWj+quohu0wOR5uDZDZZvwLup3bd70ZGKHoRyzbiky4SXB4BrEUFcDilEycmzEvZI2elGhXtnLynmbZZVo/octpIJZOsQ3CMFXh72vmd+QCua4VoSTd+50MBaDOVjT8gRjfy+oFwZMoClAHPve2ujB8PLLxqdEc8bYebw6K4v70OxH8T8jx1ESDxYEhvz44UM9PZecsdjzwd5g5eaEmzz1gukmkw7AoIMPTl1PpDGCeZ2zmdhruItPi+ciCsKayB3UNes5nqkVeps6PJhs8/k1fT8gsFfvxnOV4NkLMN3NiXESZN2fIx3QhXoaaegf7wPZtRLmr+yNRxXXpzl0345VWbg5J0Pc+iVBP+bwP2zQ8QemfIq5b8u0uPaaYwRYPqOKuGP7UNP2nCAplxXWA3xa4KNYNLJqyiVdyNAyEGCLhVvX9CY9S4wC7ajohG9aAlVar7rwvpHHvy/CNhG5K+N6BGNUN8eIfjT9NfYswctiG4fVOw7Hi+b+kveEbDPVvSryGHxc7jdNRDuiEJpPR/S iBO7xk3L mYqtOIDPLhETEa4Uf0MZbsPCQXMzYOOO34u6+Xg/0HgdljR3bvNU13TWr5j/Z+IzXzpjVbMUfU/ehAQM5TU/fNRz1HJTByOw+i3V+INjcy9nULTBWYozjzvLbht+Ov/xV6oPGZGNP9+BKbyYHtbOe1afZ/WcH3i4Zylq7E8y3YexbaSlxnE9ylEdeqfznIGRBWR/a8JlbB+UraF+UtO2z8UnQJwWtU0oxk9ZQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 20-03-23 16:07:29, Marcelo Tosatti wrote: > On Mon, Mar 20, 2023 at 07:25:55PM +0100, Michal Hocko wrote: > > On Mon 20-03-23 15:03:32, Marcelo Tosatti wrote: > > > This patch series addresses the following two problems: > > > > > > 1. A customer provided evidence indicating that a process > > > was stalled in direct reclaim: > > > > > This is addressed by the trivial patch 1. > > > > [...] > > > 2. With a task that busy loops on a given CPU, > > > the kworker interruption to execute vmstat_update > > > is undesired and may exceed latency thresholds > > > for certain applications. > > > > Yes it can but why does that matter? > > It matters for the application that is executing and expects > not to be interrupted. Those workloads shouldn't enter the kernel in the first place, no? Otherwise the in kernel execution with all the direct or indirect dependencies (e.g. via locks) can throw any latency expectations off the window. > > > By having vmstat_shepherd flush the per-CPU counters to the > > > global counters from remote CPUs. > > > > > > This is done using cmpxchg to manipulate the counters, > > > both CPU locally (via the account functions), > > > and remotely (via cpu_vm_stats_fold). > > > > > > Thanks to Aaron Tomlin for diagnosing issue 1 and writing > > > the initial patch series. > > > > > > > > > Performance details for the kworker interruption: > > > > > > oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) > > > oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... > > > oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... > > > kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... > > > > > > The example above shows an additional 7us for the > > > > > > oslat -> kworker -> oslat > > > > > > switches. In the case of a virtualized CPU, and the vmstat_update > > > interruption in the host (of a qemu-kvm vcpu), the latency penalty > > > observed in the guest is higher than 50us, violating the acceptable > > > latency threshold for certain applications. > > > > I do not think we have ever promissed any specific latency guarantees > > for vmstat. These are statistics have been mostly used for debugging > > purposes AFAIK. I am not aware of any specific user space use case that > > would be latency sensitive. Your changelog doesn't go into details there > > either. > > There is a class of workloads for which response time can be > of interest. MAC scheduler is an example: > > https://par.nsf.gov/servlets/purl/10090368 Yes, I am not disputing low latency workloads in general. I am just saying that you haven't really established a very sound justification here. Of course there are workloads which do not want to conflict with any in kernel house keeping. Those have to be configured and implemented very carefully though. Vmstat as such should not collide with those workloads as long as they do not interact with the kernel in a way counters are updated. Is this hard or impossible to avoid? I can imagine that those workloads have an start up sequence where the kernel is involved and counters updated so that deferred flushing could interfere with the later and latency sensitive phase. Is that a real problem in practice? Please tell us much more why we need to make the vmstat code more complex. Thanks! -- Michal Hocko SUSE Labs