From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E044C77B75 for ; Wed, 19 Apr 2023 11:18:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D32C8E0001; Wed, 19 Apr 2023 07:18:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53232900002; Wed, 19 Apr 2023 07:18:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 337CA8E0005; Wed, 19 Apr 2023 07:18:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1C2B58E0001 for ; Wed, 19 Apr 2023 07:18:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A9DE91601FB for ; Wed, 19 Apr 2023 11:18:56 +0000 (UTC) X-FDA: 80697893472.10.153D005 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id D88AC80015 for ; Wed, 19 Apr 2023 11:18:54 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XEq9UGQl; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681903134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A5NZuRAf3bA6RMcpxMcuMmjJScXbqJXX42QQW6vSBm8=; b=ydqhYVkmkHnQNsaep2eFYgvvPs5eLvJQFYETB+CkAbLbKEBHAR0vSN8ZaUfMB+K6e4A4g4 W4dTBOV1eTDrNgtM9McLdTdJZAqYClPVLHylj/jtYxIorCHR5QSdFg15xCAQFJr1Fh4Ua9 OeD+t4EjI0ujohVOm9obKLXTxih7QE8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XEq9UGQl; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681903134; a=rsa-sha256; cv=none; b=GhrFmwyQsajrIO/2o/FDalIthYJWIQA+6IuhAsUjzYYBAVCEDiSBdlVCW1IeB2YdLMsxIC GUDn5PHzVSF1Hdc56LQczr5618TqUVPz0QN1qGKyaiRwHglV9SSIeRhev4nUGtET3Yu7Sp dGWnxlPT/hlimRfiw3IkRjh69XhoOwU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681903134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=A5NZuRAf3bA6RMcpxMcuMmjJScXbqJXX42QQW6vSBm8=; b=XEq9UGQlh9j8hsK3EYBNG7LxKa2PAr5JOdeXxkAZgMIYwP9KvWmrTJwOZTWDqSzivwlDVN vBmH7CyBSztHGC4l1fs4ZqDmgz1LZn+mIDotdSKj9we5xRO+L32s3/Yx21oitipL2LykhC jkjQSN1M0Jwgb+th++WJfg48hQ1pc4g= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-378-dUtHiiTHO_6yhoxJauysHQ-1; Wed, 19 Apr 2023 07:18:50 -0400 X-MC-Unique: dUtHiiTHO_6yhoxJauysHQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3FAA73C10C64; Wed, 19 Apr 2023 11:18:50 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 065462958; Wed, 19 Apr 2023 11:18:50 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 9744A400E056C; Wed, 19 Apr 2023 08:14:09 -0300 (-03) Date: Wed, 19 Apr 2023 08:14:09 -0300 From: Marcelo Tosatti To: Andrew Morton Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> <20230418150200.027528c155853fea8e4f58b2@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230418150200.027528c155853fea8e4f58b2@linux-foundation.org> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D88AC80015 X-Rspam-User: X-Stat-Signature: ky8dpufxwskcf5ise1mutn8s5xouwpzq X-HE-Tag: 1681903134-339307 X-HE-Meta: U2FsdGVkX1/6r7fznovdlQpOulD23ZaJXu4ZJh0cCgAra8q1RAxy558ZKP9OrWupjSfOlK4JSROW2BAhpsr2IGLvhBe1F/IUC8jSOB22sd+iaKA74lwrw2qFa3s6xrLRN+hwzb7PF15RzeqiMAklULqeIEUeXRbI9MtPV1dHfFGb+uLla6t4iTl1PG7MV0LJvZ+F8yIkDSarrUw0LGDcn8dCQ7AntijU3lMK1pJTrakigJ65MT67MfM2nM2asGkhGBij7IRprkpKDDVrwvCBSigMWhGUASZmYjhHKg1h/VKnUmiU668jRbMPFVezwY7q0/woPPh06FFogZivwpveIrZPDncI5/6Ru5hPIZ8pQ3MXPh85b95IG7m3VMul9n8mihCg0AjNxOnNd1kqffuBaSrb+QpcA/cTOZcuDDGNNTW7WK+Ndd4sEWXMQa+lXsPi6zpSYHx0trQguCOXVSMHWFvd6XD8BDUTLw0zUf3Lq1Zb6ZRAzaGP99apF+8j0pHy5+AC/Dvj1ZwR6cXXKUWqg4bR5VTYJLomc9i8P26Fm6P2ogO/VT4sq8vaupslUxvtBi8d3ZWfrE05OBAW57SQkrP+TKPNzpCIzPZ/wcL0/sgj4sBV+rmqYQpmTZbSdpbG6nNHVDewu1hbG2W9dp1FTd8NPSUi3r+r2rBtiqFQnHpz3h5pb4gOEklNMi//8Tn/VTo3NiAPp4CQ4o4VXvW2iNoCH5X0qi0nXLYbxXE4UWzIAQKCYEcvkXLxIhtuehUOOT+cRyFYkdcrjjs9Q6+LxGUdfbdwmXXM21vJU3ypUhNygjD0DHkjQBt8a9uE5heRU5MAz6lq3xE8PH4Qz4Gx9Jr1xY2/u8Et/Bcnrx7rzFyVh/OGErxgWECK+hMCosQqaoPYlgRYAH+W9SdqYeyDsEFB+HUyLiU4t2+AnPzOduAnj/8xZqUgs8JmtNywRW2KDcvqXcWb5305xJ4EnOk ahyM1X85 08C7JOUlhe985bOzbrKMkZJxuVdJulrRDK5ezuUMoVLBOtisNV9N+Uv+u56uJwq9aubO0MAdxWexRkyb/4emBhSZR5k0YadGze7Hn2czOpc9EZq/BAEo9HvW/bRbMHETTaGqjY3NsyYs8/qCuye3sk9fEjMOgu14xvn29RCNTYLB4pFu/XiYphg+Kc0R5aPOUn8Qa1bEkYYR6334188V0RnR9QgjBKf8IspCz5eSS4KlBCjcBqx9UWbA2mZeMxc0qgxkbwSj9ws2nW9HPbqX2NPUs1AqNd441GiCs7uMszyy+uYFdiW8JMuTYounc/xqTUAYZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Apr 18, 2023 at 03:02:00PM -0700, Andrew Morton wrote: > On Mon, 20 Mar 2023 15:03:32 -0300 Marcelo Tosatti wrote: > > > This patch series addresses the following two problems: > > > > 1. A customer provided evidence indicating that a process > > was stalled in direct reclaim: > > > > ... > > > > 2. With a task that busy loops on a given CPU, > > the kworker interruption to execute vmstat_update > > is undesired and may exceed latency thresholds > > for certain applications. > > > > I don't think I'll be sending this upstream in the next merge window. > Because it isn't clear that the added complexity in vmstat handling is > justified. >From my POV this is an incorrect statement (that the complexity in vmstat handling is not justified). Andrew, this is the 3rd attempt to fix this problem: First try: https://lore.kernel.org/lkml/20220127173037.318440631@fedora.localdomain/ Second try: https://patchew.org/linux/20230105125218.031928326@redhat.com/ Third try: syncing vmstats remotely from vmstat_shepherd (this patchset). And also, can you please explain: what is so complicated about the vmstat handling? cmpxchg has been around and is used all over the kernel, and nobody considers "excessively complicated". > - Michal's request for more clarity on the end-user requirements > seems reasonable. And i explained to Michal in great detail where the end-user requirements come from. For virtualized workloads, there are two types of use-cases: 1) For example, for the MAC scheduler processing must occur every 1ms, and a certain amount of computation takes place (and must finish before the next 1ms timeframe). A > 50us latency spike as observed by cyclictest is considered a "failure". I showed him a 7us trace caused by, and explained that will extend to > 50us in the case of virtualized vCPU. 2) PLCs. These workloads will also suffer > 50us latency spikes which is undesirable. Can you please explain what additional clarity is required? RH's performance team, for example, has been performing packet latency tests and waiting for this issue to be fixed for about 2 years now. Andrew Theurer, can you please explain what problem is the vmstat_work interruption causing in your testing? > - You have indicated that additional changelog material is forthcoming. Not really. Do you think additional information on the changelog is necessary? > - The alternative idea of adding a syscall which tells the kernel > "I'm about to go realtime, so please clear away all the pending crap > which might later interrupt me" sounds pretty good. > > Partly because there are surely other places where we can use this. > > Partly because it moves all the crap-clearing into special > crap-clearing code paths while adding less burden to the > commonly-executed code. > > And I don't think this alternative has been fully investigated and > discussed. This was tried before: https://lore.kernel.org/lkml/20220127173037.318440631@fedora.localdomain/ My conclusion from that discussion (and work) is that a special system call: 1) Does not allow the benefits to be widely applied (only modified applications will benefit). Is not portable across different operating systems. Removing the vmstat_work interruption is a benefit for HPC workloads, for example (in fact, it is a benefit for any kind of application, since the interruption causes cache misses). 2) Increases the system call cost for applications which would use the interface. So avoiding the vmstat_update update interruption, without userspace knowledge and modifications, is a better than solution than a modified userspace.