From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2CA7C77B7D for ; Mon, 15 May 2023 18:03:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65E9590000B; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60F0F90000A; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B29590000B; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 360E6900006 for ; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0722A1613EE for ; Mon, 15 May 2023 18:03:52 +0000 (UTC) X-FDA: 80793262704.22.581468C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id CB2E11C003C for ; Mon, 15 May 2023 18:03:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V0z1gkZ3; spf=pass (imf20.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=jkb4dlPC4zyep6CJ8cMLWwNNaJdusa90HtfOUuDaHEw=; b=ZxHovGdNGoBf3eq6Ed+mhbcrNu+jgNDBS909wGJekkekHfQLNDsOmotb+PJGsT+sEmAg3a 2vH7pHhahRVWtq0z2YDvIVnEEUO+YrFX6leiXxlB+pv6SFnFJVwy3zLKi4+Cbrv4yVo7Lp AqhjMHs+FpkgHOMPvRDs4QfEJ1DHi2U= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V0z1gkZ3; spf=pass (imf20.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173828; a=rsa-sha256; cv=none; b=AagfsiybRqupYc7kPFC/NK9Z7vW3Fc8E2S0VG8TYF4s1Bq9idVbq3IUXogDtt6x4bWvtVP R3nJC9FCaAJthGRquJ4CR+bM3T5/8TYrS1Ud4MVyt/HifzHZwFqMe+2ITbME3AjL0KucTd Y8fucudx9JwYLqWddLTZRxz+6UpboCE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=jkb4dlPC4zyep6CJ8cMLWwNNaJdusa90HtfOUuDaHEw=; b=V0z1gkZ3uh3k9q0poPJcxm5RP+UILH9flOUGPR6wa1ksv7L8pLflJm5jcOC97mlruRUUDd QXF5E8wH/qHxa7DJs3dwqHsbcq2EjGBip8+J+Rb3C0bVu00hmr5BJYbP/3lFRatGPltKNP XgmO1GcFa2pM6dcOShPZzhXpJAFu1GQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-gp1bspFaPzeFbxO02PyHjg-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: gp1bspFaPzeFbxO02PyHjg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 27D73185A79C; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F1C7340C2063; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 81D6B4161E52E; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.692835888@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:26 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 11/13] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Stat-Signature: uo4ps5kq5689oeno63aespod5onbhkiz X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CB2E11C003C X-Rspam-User: X-HE-Tag: 1684173828-829019 X-HE-Meta: U2FsdGVkX1/ZH7+HkeRfM2+cvbHIdUUxKkLA0E++cx6+CCBXMTSxNyybPZGgURusMdxRTrmk57GF8ibfc74vbrFCS+y0cHTMV8+4aGi8wVpZKoiWsC0K2eHFAizqzdN5w9ic+/dfmG+WCDbPGb5u51rK3jCOIObfQTMWxExtnTeOYzVjpknBT6+t748bR48JgwqVMixT4jiWKeJ6d0VZt6vvBIRC5tAouqiQvI+8nZ5NrY2dRee2Dno0jw302HkDP8e3ssD+6H13RpkWh7HM1x8hQrpRc6nDde3M47ZWAABpWjZcdG+mTl4pwBlGn9+7/IZ/A1xVoQ89wg1UtPccVaflMbKUrHGFmqo8SURtkP95QwYFqj4JGb3zzuA0jDKbbZ3cR2I9ONgS7pUn9q4O0mgiwyE3BoyVy45AwOTkh82DPxrTuaIvlfpYp+z2+YGlHDRNu5SBSzZfXOFrPtRNuCaPxl8HTh/knt97NJCUzyBanPvncWXSdV7xueAKBJRcKKufK3Hfb0H8N59YfOoVlsRg45bX1PasXVRXKMvTbhH05IaHsOi72s2NiconFXgYa2u3hz7PXuFYrV8dpXmYojoQdNzZRtkp+lzXdbAGNowMeXmNByC+htYaBUsF6ChkFlmY9UphPcaJOguPqE00zG7WE8aWLJqS+tzrNLf0HCW3mlzcGfNTaSxemF/GgFVuOs4A6k8DLlNOiceBFMl237MKvxSYcvBSw3qEfWsLslHzS/7jmCESFcp4PUVZ3fjHXwi1DbIWSPnuKfle6fNkG1vetMuAhqM5/h2KbWfbLiAutvPjiHsfQdHZVv7oHIfJ8Vm7wSCBGI/cJ204nqATrrU7ggvUbdY0qbsfUQPnhiTXHtb+fQUGVQD2ib2SfBf92dsgH4cMg+/p4Q6yCYNJv4OCNicEMp16PTO8fYbo1EUv09Z0zd8IxpzjiDvhjY75ce3lLr68y0c7gmSuyM2 1PlzdwnW 14O3zuAor2SJgfYcpq1Xs4rWxdaVh5iyFfILGZt8PfNN+B2SRhZEg6jIzqJQV2vwf58LbBrSRBvEQ+bKnQEA6YO2abWy73Kz3ssElcm70cVyJ9x00Roe7g7kI44lUxfPW1EQQFJgKRjh2zZiqSHYNV51ooNKBiUevWSaoXkXxpbPxFguDJIcK0BE6+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications. Performance details for the kworker interruption: oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... The example above shows an additional 7us for the oslat -> kworker -> oslat switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold for certain applications. To fix this, now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. Signed-off-by: Marcelo Tosatti --- Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2049,6 +2049,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2068,6 +2085,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) {