From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE0EEC77B75 for ; Wed, 19 Apr 2023 13:44:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C3F6900003; Wed, 19 Apr 2023 09:44:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 37304900002; Wed, 19 Apr 2023 09:44:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21356900003; Wed, 19 Apr 2023 09:44:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0E434900002 for ; Wed, 19 Apr 2023 09:44:42 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A84151A026D for ; Wed, 19 Apr 2023 13:44:41 +0000 (UTC) X-FDA: 80698260762.16.DE82035 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 28B47140013 for ; Wed, 19 Apr 2023 13:44:38 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Sr+nxd7b; spf=pass (imf09.hostedemail.com: domain of atheurer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atheurer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681911879; a=rsa-sha256; cv=none; b=TciZWeYsYKgXQy6PGLNiqbZtZ00aBsVQxS3tA+HxI+iALeWZijyzCqkcNFoJ7OygmFLx5E 7mRTPk8Y/xNJJLwsrF+5XcIp6DSCvhl3kJZqya4xSXN9GJfjDPX/Q7byAWR4H9+j+tb842 SQhiVcWb2+X73YZi4zkYXCz752Rhm8w= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Sr+nxd7b; spf=pass (imf09.hostedemail.com: domain of atheurer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=atheurer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681911879; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nqxlKnTYtu/1HhRoT6vkF1bV6JuDwglY0N8GO6DZu7w=; b=49mEIhvKGqidPIoiLyqG7yrnonbwwtEUKCmvjBahyrRt7EpJKXYAI6CIwi/qeL8KUysBBE olmuWr+z/Aii76TSZuuHabaFVGDBIZcov8NE2nIfPYEHfHjmFu6sc9HmoHW0C/SJ7xsEKR tYehH5QnyfTrdVfe6Asd08j22y7jvXU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681911878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nqxlKnTYtu/1HhRoT6vkF1bV6JuDwglY0N8GO6DZu7w=; b=Sr+nxd7bjUgF6CeT3aNny00y34jz09vyPLGPYp/cK7Thm8GA2/b0XTC7PckTArtAwdXRg3 Br1d+WW+J6KwDQf/RGOWDDM5WJs2LGoBR3wocQfTDd9Qh/GZNzU4KDg+bDNbw1DvnmjVDT s171+25Dre9icbjcNcirtufUTp0gq6U= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-1DSPOqSXMVGbu5swfaC-vg-1; Wed, 19 Apr 2023 09:44:37 -0400 X-MC-Unique: 1DSPOqSXMVGbu5swfaC-vg-1 Received: by mail-ot1-f69.google.com with SMTP id 46e09a7af769-6a604fbda57so1076783a34.2 for ; Wed, 19 Apr 2023 06:44:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681911876; x=1684503876; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v8JKAFfWzA+HrvLNdRkbuEhcM/1yqc32dGHnjCBEcPk=; b=eUSFV9boPalwFpNh3jpgP0r3dscmaCg/paN5S5JvvvF4jDJB5ceA3jIC7J3oOT3Ayk AgSiuktPGqXMRRScoN0jBtNtV+DL8Rmm+/IBNYFVaJsOYSsGyHVwaKaLFxkCXnijfbSl Bp4lQaSgif6fDnMWdryAxG8X0N6+4sk/asqizTIqcBJ/Vsqofo2FJ68skFYUC5xLpEy8 go0YxddP/WYf+SgR4Tg7lhkfsGlHeb+VAx6EZ3jJg0DdRSMmJ1P7UEnT0tCaqtmXjQbK ATSuw7mcGWeGSQUfxetZ9H2scPI4vz5v0rVX7u9vYvueRvMrIJa4wo4HTGkT/ifXrVpD QH2w== X-Gm-Message-State: AAQBX9dJA+pNu9jemIvXUNN/I7JJmIOBc2B1KesWQWzLdYzZ8a0WPXM4 mAYLM5gU1E1MjfTjScu183ViQ7MUTlHmjjcfZIiMzHeeFpmTsJUm07exs+6E07lUzD33cGbSxGS dgX/0Opv0Guo= X-Received: by 2002:a05:6830:129a:b0:6a5:f503:aeab with SMTP id z26-20020a056830129a00b006a5f503aeabmr2947252otp.6.1681911876245; Wed, 19 Apr 2023 06:44:36 -0700 (PDT) X-Google-Smtp-Source: AKy350ZM1GVqQYv0BdCOoAH4vTMkZ0Fnnoaokpull2fJw9MugBZhRm7c2vOI1arU/DUOzItjTKyEQQ== X-Received: by 2002:a05:6830:129a:b0:6a5:f503:aeab with SMTP id z26-20020a056830129a00b006a5f503aeabmr2947231otp.6.1681911875902; Wed, 19 Apr 2023 06:44:35 -0700 (PDT) Received: from smtpclient.apple (cpe-24-28-76-31.austin.res.rr.com. [24.28.76.31]) by smtp.gmail.com with ESMTPSA id z10-20020a9d7a4a000000b006a4244d2a7asm6611275otm.9.2023.04.19.06.44.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Apr 2023 06:44:35 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.4\)) Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely From: Andrew Theurer In-Reply-To: Date: Wed, 19 Apr 2023 08:44:23 -0500 Cc: Andrew Morton , Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko Message-Id: <98E08059-6607-43FF-84E3-BFF3FF193D09@redhat.com> References: <20230320180332.102837832@redhat.com> <20230418150200.027528c155853fea8e4f58b2@linux-foundation.org> To: Marcelo Tosatti X-Mailer: Apple Mail (2.3731.600.4) X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 28B47140013 X-Rspamd-Server: rspam01 X-Stat-Signature: ci38jsdjg8fa33jdyr5aubnr84i9ru1q X-HE-Tag: 1681911878-438345 X-HE-Meta: U2FsdGVkX19XgcieUaWTfrgTDAHBgguFwDODRL8uyYeULFkzpd/h0HUFZXDdluXn96OcrMGrbfdM7o0bYT5joup/F8gcjac2lcCcRKU/a1kYfq62WALIDmkyh2GtpcXJiSvqr8O+hTQ+7cwbB0Pd1awE6Hy+23pqKZcE7TsZVF+PreMoaq+xl+qnwJwMjhMfNo+LqOBOHTKPx8jBgXAdkDbWy2CnatgHpzB4Oz+Gb7mvnJdC4/PlGgUeLPxWgPR+gkvYoziBw6cdK9DMHJRxYD2sI5ISOfatsFBNNAYp/1CUQHyJ2gObPs20Ux5Ysx4rAuyS3zlk9lwJImwCdbGUfn/yKJAaEWUEhLMFk9wkmGyUlWuHSDpwK36iclWpl0fEeavNTfgpJ6YgoVCXWags07f46ggJw/vPL/zqtIA771TrUUAVZhMifCmkQ3HkBZwLg2pYOgf7jAWvI4uocFQMkoblold3EQb49+mpP1dYqywpdp5MiTTRkhDVz0yQ6PTdbRLgKtyiTF4w3YNOeY70h+wZXNkpP+SgzGllbVR88LSWHKeSjAfZFPr2w+BbpO5+gWPTsrgK4ads50yO6mWVT9z2kMxNy60ReqK++zk2mBh0KlUcVI/1+NOtLuvCJwTV+Iksx+DIcubsebQEyX1q6t9+akwTRkQX2LMf1bx37AxW04Vri29I/dQ0SP5QmlMK5O283zCu4sQI30xjV98BdfhHOuuJcPxoai6MW876MFdH45w/mc+X1KwR4UxBQjSynD1ly//6Uv3TU+Lq6wjmctgm7vVFgHoTMjKcN7FGtCdD3hKEyuL4hrf8R/2fEuw8HVz9jtlpPgfbLZwiAfTPr6thbHZfdztWNDWdXHEt52cb1LFweCesxMk/o2a4x5djTGpTl8btV6WXQo26WGL7X7IlWDcOUztYoX8IPIe7JfV22SbgFtlFNu+sH1keSA6BpNqSXdVGhKgHczEPGEf EgtuUwxw 9v5xhSPeNEgA3kg0d8Ypatcb2PIqAl5zj1+2pUCZansy4PU5bk01Ikb9upxbf00sNWIgnwoG7v15ISbyt+PkgKHlFFnjyxXccSlkWqTU63hL8CGM+ILnprhjAUiInTc7O42YsCf5AlxF8EEM5nzLnNZrNIilQstOFkQOvO3BuoX0kb3liSfG0cfmAUBxGZn+ltobBRST+v6IgVo01NTficL9ohakg/OfJcuJL5GJ5oJ75SHqY4AIcyGmOYXkYkQDE2qkXO2d4Z6h/kNnaElrWwQHzJZ30OlFLiQlwf4o0ugQrWoHz16HDxvTPrB03rP+VuAsVnwGZhgVgLWxy5ST7DCQ5RajH911pS0wEvILbC5qBgIUzlAC7vFkz4hdecuRPT/Ch1C818Mevv6lyn4EfrC7jXbINT8/1MpxmP9sDliLPrUn98ifosauCea2mTvgON68oErCtzsY5Sm+dhe6RegRMiG9eqClIAW0Ua6FbvZfT5uoLHniALmC1Hhxs5R+eiGMpsRq8v2V1VlRK05tFnVcyd01aNwPaOrjxG5TEHyqOORI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Apr 19, 2023, at 6:15 AM, Marcelo Tosatti wrote: >=20 > On Wed, Apr 19, 2023 at 08:14:09AM -0300, Marcelo Tosatti wrote: >> On Tue, Apr 18, 2023 at 03:02:00PM -0700, Andrew Morton wrote: >>> On Mon, 20 Mar 2023 15:03:32 -0300 Marcelo Tosatti wrote: >>>=20 >>>> This patch series addresses the following two problems: >>>>=20 >>>> 1. A customer provided evidence indicating that a process >>>> was stalled in direct reclaim: >>>>=20 >>>> ... >>>>=20 >>>> 2. With a task that busy loops on a given CPU, >>>> the kworker interruption to execute vmstat_update >>>> is undesired and may exceed latency thresholds >>>> for certain applications. >>>>=20 >>>=20 >>> I don't think I'll be sending this upstream in the next merge window.= =20 >>> Because it isn't clear that the added complexity in vmstat handling is >>> justified. >>=20 >> From my POV this is an incorrect statement (that the complexity in >> vmstat handling is not justified). >>=20 >> Andrew, this is the 3rd attempt to fix this problem: >>=20 >> First try: https://lore.kernel.org/lkml/20220127173037.318440631@fedora= .localdomain/ >>=20 >> Second try: https://patchew.org/linux/20230105125218.031928326@redhat.co= m/ >>=20 >> Third try: syncing vmstats remotely from vmstat_shepherd (this >> patchset). >>=20 >> And also, can you please explain: what is so complicated about the >> vmstat handling? cmpxchg has been around and is used all over the >> kernel, and nobody considers "excessively complicated". >>=20 >>> - Michal's request for more clarity on the end-user requirements >>> seems reasonable. >>=20 >> And i explained to Michal in great detail where the end-user=20 >> requirements come from. For virtualized workloads, there are two >> types of use-cases: >>=20 >> 1) For example, for the MAC scheduler processing must occur every 1ms, >> and a certain amount of computation takes place (and must finish before >> the next 1ms timeframe). A > 50us latency spike as observed by cyclictes= t >> is considered a "failure". >>=20 >> I showed him a 7us trace caused by, and explained that will extend to > >> 50us in the case of virtualized vCPU. >>=20 >> 2) PLCs. These workloads will also suffer > 50us latency spikes >> which is undesirable. >>=20 >> Can you please explain what additional clarity is required? >>=20 >> RH's performance team, for example, has been performing packet >> latency tests and waiting for this issue to be fixed for about 2 >> years now. >>=20 >> Andrew Theurer, can you please explain what problem is the vmstat_work >> interruption causing in your testing? >=20 > +CC Andrew. Nearly every telco we work with for 5G RAN is demanding <20 usec CPU latenc= y as measured by cyclictest & oslat. We cannot achieve under 20 usec with = the vmstats interruption. -Andrew