From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADCB6C7618A for ; Mon, 20 Mar 2023 18:26:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36CBA6B0078; Mon, 20 Mar 2023 14:26:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 31C4A6B007D; Mon, 20 Mar 2023 14:26:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 231BC6B007E; Mon, 20 Mar 2023 14:26:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 17AED6B0078 for ; Mon, 20 Mar 2023 14:26:00 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CF008160F1D for ; Mon, 20 Mar 2023 18:25:59 +0000 (UTC) X-FDA: 80590105638.13.14661CF Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf16.hostedemail.com (Postfix) with ESMTP id E93C918001E for ; Mon, 20 Mar 2023 18:25:57 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=KryhIWdP; spf=pass (imf16.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679336758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6NyzBHrEFFt3x3wldfOH2t3owxOVtgcPmDv2HFH2MaU=; b=6LvPqihqsSiqp6fK58/XGn33e6K5rQfZI87zE3dvg7YFK7lBhtz5qP+Cj0EE58WEHIGUKV nNb5m6fcsgZ16+L1wfX5A46mOhPfQtW41vu6M7SqZmsEgiFF1imJuik6FbBqfJJHZg3U2r 3rpc6ILu7/RWboecA37rHijWw+qTfSs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=KryhIWdP; spf=pass (imf16.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679336758; a=rsa-sha256; cv=none; b=jmLiaQCu5O76IfuZUGPnAwE1mEPGmP8U3WfnVwVlvxMSb1hCeFy2DxBIrJ1iPp5d1JeUAl pEoEiNrD+fBRVbSbaD+gDnCbcXYETTU+JaqQ8v+8Wbrteva7gn0izZoaYw/Sw79KGc5pHW wnr+tGLJy5TqDK3TNxVkWqI/rZdMoDw= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 97BB121A97; Mon, 20 Mar 2023 18:25:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1679336756; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6NyzBHrEFFt3x3wldfOH2t3owxOVtgcPmDv2HFH2MaU=; b=KryhIWdPnkDXrKEVLwDhapgy4b89AdPaHUXhLEd8LGuRasPsuJuEhANGgml6FxbEFO8qVs DJmxoWjBSyQVvkrrXOEsiRWW5dSJwWtlEnI384hbyv8/2UnEWtLJnJ/ifZ4eVrbjcv80+d ZLDbxx8GuJ2iHrjnPpJup6CgbZuNaZY= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7760313416; Mon, 20 Mar 2023 18:25:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 7406GjSlGGRsKAAAMHmgww (envelope-from ); Mon, 20 Mar 2023 18:25:56 +0000 Date: Mon, 20 Mar 2023 19:25:55 +0100 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230320180332.102837832@redhat.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E93C918001E X-Stat-Signature: ka7poayu1ii6iae14ejpt7od5ji7wby8 X-Rspam-User: X-HE-Tag: 1679336757-276761 X-HE-Meta: U2FsdGVkX19pYWG7bNajbzzEfebYO7hRdQPhMpKHoig32dSYrorzK7XhFkbOSZ/+5GsDkSI4O51p7o89+c+reZyUXfY5tnIbDzyAgPLjIRHwy4RQMZng638y0CmKKkM7qcBvPETKE9FpvT2bAxp+NjcDk0y42pOVOA0b+jt1CRnCAIQVEopVraMlw9Qa7lpZ1z0yR0vJD7hS8+C+arGK2lsFn6HwNW36Yid+HNVQl6NuJ8plekeUb+vDt4MDxqMrf12yiYvJtMw0t0TPFfUo16niA0lsDdr7XlRNmE8GW86DYBG63T5icDdPQvYdjufCEVevRB9+5GjLIS7yroMjyzipAq1PUpydbLHTyT4IwrwyZhGw0otTvYQK5Mzet+VSy+7qytzB8ryZ0pW7uaIUBfth/L49JCPoL/g9X0FSN7eTVw/fCx+k1pTR97hX51X9GawE4tFTtSgJQPe7vMPB61hHVcdLpqr1t+wpYKCNWiGH3qE+WTAKruJmnzTpT/iQ2fM85vuFUsA5JWOJqMsRT/gWkAOYtrhaXk9mn5S1MQTuoZSGK3vDBk18fOTAfxwtw36WCSXBNC/xobuyYnzwsHw7+IbXQPYWmzej4L/Na3zphiszUvZlhJe3zKPCu9Wfpa4j9v9DzD2mH7cGLG3jNVsU1ZkDiHV3zaMBCL3LtEdfptZ7l7EglfylD9hyWjsjgoOOXLeeHonj4usKKfyXCLsJPrm7DAsYGdutGuTm8s+GbyrK0mIV2h3QU1M+TH2T53lH3BPYtkzU7ZdtfSKzxBLG9pZhO0xcdbHDGJm7HVAe752FEp4boY7vJYm+rvd4yYOGyD2rEzJiCr6NkbkfEmgUKvEwPT0tcZllkPH2c9pOGRiQQzb337Quu62wQAdublugHe1eqeTgde+yOSqoWLtKpSO+AERllcr+xWXAX7ieBkR/fguS/eZ4VLZuKXsn1LSMOuax0w/dDpZCWgC 3iWwMIAB u5EzynqsbK3cjAM2pvOX18hmvC0cirlsN9Dv139Ast17fy6iKYP27BkVSHx27bqFkMLXxaVPSRVRQGCMs7msXEK1zY4FhWvovp8UHCGDMDDr5CorUagGaSSaCjHfh2OFZaO5pQvOcKWExBPBonG3RTQKHYqlcC/HvsDDHN1N3PH2iOYM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 20-03-23 15:03:32, Marcelo Tosatti wrote: > This patch series addresses the following two problems: > > 1. A customer provided evidence indicating that a process > was stalled in direct reclaim: > This is addressed by the trivial patch 1. [...] > 2. With a task that busy loops on a given CPU, > the kworker interruption to execute vmstat_update > is undesired and may exceed latency thresholds > for certain applications. Yes it can but why does that matter? > By having vmstat_shepherd flush the per-CPU counters to the > global counters from remote CPUs. > > This is done using cmpxchg to manipulate the counters, > both CPU locally (via the account functions), > and remotely (via cpu_vm_stats_fold). > > Thanks to Aaron Tomlin for diagnosing issue 1 and writing > the initial patch series. > > > Performance details for the kworker interruption: > > oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) > oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... > oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... > kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... > > The example above shows an additional 7us for the > > oslat -> kworker -> oslat > > switches. In the case of a virtualized CPU, and the vmstat_update > interruption in the host (of a qemu-kvm vcpu), the latency penalty > observed in the guest is higher than 50us, violating the acceptable > latency threshold for certain applications. I do not think we have ever promissed any specific latency guarantees for vmstat. These are statistics have been mostly used for debugging purposes AFAIK. I am not aware of any specific user space use case that would be latency sensitive. Your changelog doesn't go into details there either. [...] > mm/vmstat.c | 440 +++++++++++++++++++++++++++++++++++++++++++++++------------------------------ This requires much more detailed story why we really need that. -- Michal Hocko SUSE Labs