From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18E7EC7619A for ; Thu, 23 Mar 2023 12:17:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67C906B0075; Thu, 23 Mar 2023 08:17:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62CC76B0078; Thu, 23 Mar 2023 08:17:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F5006B007B; Thu, 23 Mar 2023 08:17:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 416A96B0075 for ; Thu, 23 Mar 2023 08:17:38 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 02AA4160678 for ; Thu, 23 Mar 2023 12:17:37 +0000 (UTC) X-FDA: 80600063796.08.C56A683 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf27.hostedemail.com (Postfix) with ESMTP id 014C940011 for ; Thu, 23 Mar 2023 12:17:34 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=gR445mg0; spf=pass (imf27.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679573855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Uu+N5jOhken1tfZs5kVQQIsrfUWmzwdgGp/YYNK2ddI=; b=59zxIzL5o2IobhujNJTfvQDKGcPiT9PMRp6AL01WmHB3ygSSdc4XxGxYEHPNN6v1x7TJY+ fCkFp6MtcxjwHzZHcVPynlnYfqbUFPkGWaNijtreEzjn8oQKVbYtvAjro0HBf8H+x0hgW2 FfsgPBba/vnGTzBxPkF91W3uXK9ke0g= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=gR445mg0; spf=pass (imf27.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679573855; a=rsa-sha256; cv=none; b=O28dICjXa1TBYa78l5VFgSHFck6yPP7tQh64Su0hQPG3O8Wjh5ICACOTEslNgsQ/7/XiMO 82ednc+KSKgr40h/cwXPc/le68SJMYH8iG2jRGDxWgpHRhhb9o7/KClClp54QiWEzxdkAR ye12/JmHA6eZsCrIZRZVP9Um1kOzfvU= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 92BE3339C9; Thu, 23 Mar 2023 12:17:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1679573853; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Uu+N5jOhken1tfZs5kVQQIsrfUWmzwdgGp/YYNK2ddI=; b=gR445mg0TOBYGbWjZ1RG3k1s2MieH6HxpCRvqcr4BUjL+mUSWQ1cqVgaxefHLMtEZKy6lB BJlTrqxjFaJUJWNygldd5h4KoE6jq54y+25iWhMOLvPZw22iGS4/9UmdpzdI8+IR6bI8CV HvsNaeYcN1mwyIzCWKUG6wR1OLlci3g= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7368D132C2; Thu, 23 Mar 2023 12:17:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id skRCGV1DHGQfLwAAMHmgww (envelope-from ); Thu, 23 Mar 2023 12:17:33 +0000 Date: Thu, 23 Mar 2023 13:17:32 +0100 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v7 00/13] fold per-CPU vmstats remotely Message-ID: References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: etxzj5jdtftarmieg4dwd113cdp8ogk9 X-Rspamd-Queue-Id: 014C940011 X-HE-Tag: 1679573854-554698 X-HE-Meta: U2FsdGVkX1/EhWtWpDaaTEYd2zfYCac4n3scuncO+QUf+KS27ousJg+d05GmI726CTdO1xKbeWPus7HDjjyvqJGZ4FWdnH42tJQZoBvIYgtfbo0QG32VkcOC3DaHfBoDuYoEnj7zbLwPDIekmLnllUiPgytiEfVhivjBBPTf1AuM/J9XqSPQkN8xBpqNtw4IO+ACKg10V/JomSeb8k4S8uczyD3qBybNTJyroixCt3ZdI5xIf/MSZgvMnw2835uPcyaOwyGBtyBIeLP16UcXHxbtIg8vMStk9R3XOdp/aFiGZFstBDiePZHhTDA84Q4D8CC8sNTKe95IuQgqWJgW5eAC3D9XNvxfYAlQczkFyHJfs6tuAyFb2l0pbM9fqOss1gl3NMJ2m+mx3rYvjjMqa7K8kvBRXSrE4p+CCcpkZedsbEyidsaTn3poQTfiFDtlYrflz71u4L5ujT+i3rjLEWnNd2jlS+QrEwVc1OvlFfLiYhBM1WTR4VxoevcSWGUda5Uai/ZeetrWWWnMCxA7SPh5DvPvnOf5kSpaw6nVrr5DSkBmJjm497yUrtnjAVThgp3FLbKZUp7+ZmUWjEig5EkL8AJZu/ubXy1PO+P2wY4lH4mjRYnKNoEq3dvqZYbGw+IRZognp0L/GoI/fP3e/gWfv1FqHpLxo828YvikEcHgKkBOomXBoclO5j6zAqmEsDlEC/xQoo/DQy4lHw6OUGh4P4DsYhQDv+WuyVEPrWBMe1b5sbIw/FFylS23NyysZP71YdDekYUUm4ndvRx9+4SCAWuQNwlISG5UKQolwJh0z9WKuWhIsVZ+8xbOBSHe0Rrjq95a3fV2mIkBibwXJP8nSBgp5Uij4HkkW2JWlt35AZ/BLDChgMu1lir7W9r8qYNyb+c1XUUBUQuX5KEnIAdd40wxyBgXtSh88lqvbpI1I1wrLjPJBeHF2lXxoxPrmvHgme5hIkgipIKrZNp TGVswOPT IZH4Hzxb1t4cnVlydj4ye5eC6Ll1J8oFADhp1znjCMHPTqbE2vU76ULRkc0T+IILan2OHK/idrhAesKmI9CUSCHAoMyD3JOTzAGxXlj5XdeGYoM7gQmXN2SMC951OD1PHiMfyqs3iksk3BQOwtee3ymx+FPM3It6NkzHZy1usWfjYH/sRy8dQwE1ZZScjrFqnpSSfk2LV1T2AKHkp2QRw/08M3Q+cnX6cPTCY1cLVOCTAn3wyS659iIcFFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000830, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 23-03-23 07:52:22, Marcelo Tosatti wrote: > On Thu, Mar 23, 2023 at 08:51:14AM +0100, Michal Hocko wrote: > > On Wed 22-03-23 11:20:55, Marcelo Tosatti wrote: > > > On Wed, Mar 22, 2023 at 02:35:20PM +0100, Michal Hocko wrote: > > [...] > > > > > "Performance details for the kworker interruption: > > > > > > > > > > oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) > > > > > oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... > > > > > oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... > > > > > kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... > > > > > > > > > > The example above shows an additional 7us for the > > > > > > > > > > oslat -> kworker -> oslat > > > > > > > > > > switches. In the case of a virtualized CPU, and the vmstat_update > > > > > interruption in the host (of a qemu-kvm vcpu), the latency penalty > > > > > observed in the guest is higher than 50us, violating the acceptable > > > > > latency threshold for certain applications." > > > > > > > > Yes, I have seen that but it doesn't really give a wider context to > > > > understand why those numbers matter. > > > > > > OK. > > > > > > "In the case of RAN, a MAC scheduler with TTI=1ms, this causes >100us > > > interruption observed in a guest (which is above the safety > > > threshold for this application)." > > > > > > Is that OK? > > > > This might be a sufficient information for somebody familiar with the > > matter (not me). So no, not enough. We need to hear a more complete > > story. > > Michal, > > Please refer to > https://www.diva-portal.org/smash/get/diva2:541460/FULLTEXT01.pdf > > 2.3 Channel Dependent Scheduling > The purpose of scheduling is to decide which terminal will transmit data on which set > of resource blocks with what transport format to use. The objective is to assign > resources to the terminal such that the quality of service (QoS) requirement is fulfilled. > Scheduling decision is taken every 1 ms by base station (termed as eNodeB) as the > same length of Transmission Time Interval (TTI) in LTE system. > > In general: > > https://en.wikipedia.org/wiki/Real-time_computing Thank you, but not something I was really asking for (repeatedly). I am pretty aware of what RT computing is about. I am not really interested in a generic fluff. I am asking about specific usecases you have in mind when pushing these changes. > For example, for the MAC scheduler processing must occur every 1ms, > and a certain amount of computation takes place (and must finish before > the next 1ms timeframe). A > 50us latency spike as observed by cyclictest > is considered a "failure". OK, you are claiming that much but you are not really filling up other holes in your story. Let me just outline few questions I have. Your measurements talk about 7us overhead the vmstat processing might add. This is really far from > 50us above. You suggest that this is an effect of the workload running in a guest without more details. I am quite surprised to hear about RT expectations inside a guest system TBH. All that being said, it would be really helpful if you were more specific about the workload and why there is no other way but making vmstat infrastructure more complex (it is quite complex on its own). Thanks! -- Michal Hocko SUSE Labs