From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF558C4345F for ; Fri, 3 May 2024 18:11:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A6086B00A9; Fri, 3 May 2024 14:11:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 455886B00AA; Fri, 3 May 2024 14:11:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 344166B00AC; Fri, 3 May 2024 14:11:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 135D26B00A9 for ; Fri, 3 May 2024 14:11:30 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AFAB51C0393 for ; Fri, 3 May 2024 18:11:29 +0000 (UTC) X-FDA: 82077877098.27.5138A76 Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf05.hostedemail.com (Postfix) with ESMTP id B82D6100011 for ; Fri, 3 May 2024 18:11:27 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=oiMT1nYh; spf=pass (imf05.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714759888; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v43bZUtmgn/Fa1lECh+eb1qKS1YLdkiy2ArNZeurJLk=; b=ksfg3eBO6I0/+sApX2koT4A/iR8WwPFBkVZ+fnY5pgJiClxp0Bsrn1FBxg/jTuEcR8KvLo MQnxvQvsso1MVXLNEuD+WQ146Jf3zjZm5Dlh0eHJ9WS9nLkEqa6B1EKIhs+VVb7YSAltYd 0RyxKiPkTZYmiUYsm8aeeB8Xeaq0tj8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=oiMT1nYh; spf=pass (imf05.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714759888; a=rsa-sha256; cv=none; b=SlMceFUE+PAqe5XoLFqRvdiEcBNY3hAKXUMmVcY4b3azls2Bzd/qcbJoDhiA/ttDEj4w85 5PFKpaSIP6PEFip49CKOHWnMKBoFFad/sJyu3puaCBXloCGnngTguE483/n36Pf0vhgW8m EX3bWslfdhKXVGEZRpPHiiB81RiG2l8= Date: Fri, 3 May 2024 11:11:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1714759885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=v43bZUtmgn/Fa1lECh+eb1qKS1YLdkiy2ArNZeurJLk=; b=oiMT1nYh62rxDYQnwrF57dx5GghIQOtDjIqKfWRkmQH3x2e7AosUA3Y7zwhMyvYdveFFQ/ IzfGaEUD0xtJ3vsnWclQvbASZgdji3FWoQxrHEqqbYyDftzz5yNC6qwMtIxhkTSlDWBfHQ K2/tiY0enRpvq2UUgKQIf6jJuFDRqSA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Jesper Dangaard Brouer Cc: Waiman Long , tj@kernel.org, hannes@cmpxchg.org, lizefan.x@bytedance.com, cgroups@vger.kernel.org, yosryahmed@google.com, netdev@vger.kernel.org, linux-mm@kvack.org, kernel-team@cloudflare.com, Arnaldo Carvalho de Melo , Sebastian Andrzej Siewior Subject: Re: [PATCH v1] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints Message-ID: References: <171457225108.4159924.12821205549807669839.stgit@firesoul> <30d64e25-561a-41c6-ab95-f0820248e9b6@redhat.com> <4a680b80-b296-4466-895a-13239b982c85@kernel.org> <4m3x4rtztwxctwlq2pdorgbv2hblylnuc2haz7ni4ti52n57xi@utxkr5ripqp2> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: B82D6100011 X-Rspamd-Server: rspam06 X-Stat-Signature: 9te7q455m51b4ujhhej85sse9ikz5hjz X-HE-Tag: 1714759887-675299 X-HE-Meta: U2FsdGVkX1+yPuKcbp1A1lt35+G8JfYbbnk1mnZm0Uplh+VrgVRYxUVogKzbU2wW6Cj5X9QcjV+OP5o6dRPFG+YjVx1gfNpZioIhWOG9zNY5vrzaxkhJutOjLGh9ZlmkpjuiDJGa6SzHMjZ+qGXIx8Ok7dpArPnsxRaG0xGPt5bAKnQYW2hBrMUEtwwbJ4tzJP0f7MpBllrSdjekuEKw7KUGvte7o8Lb54lygkraDPknP1Vp4oJ8WtMawTP22WXYxDJ8RFOwFTAEM+lCm8qIjd+R5ntYFzJ2zcvH9iB75eu/Tz0uO1zedquUF14VYM601hRwVbS25U2BDqw3PSKc4oHfwEuyNgbF+4H5LQk1D8quwxl4rhC4bBnlLs+UCx77JdvmDhCyvz2SPW02tQxaNMKY5xqZ/TDmlAAVeXRAP4toz3vt6qmnf189FdJKyOsiiI0th5hzq4dij5tIb/KxA5B6PbZ21vBSwD6k6XlH+gEuEKs6VbCe/8k1niS+ubXVA6lIi4xgwXUPtHhIwRLw/fTAYa/k6k8yGR8Q54nNoeuwkoMqRI0H7u7qcxMIiQxQ8BhPvHwVQwmCVDzxC97zwIHUbEJxThQd7ylBQY0iNL01gFcDlNXGyuWm7xZ93E3f+7iIef/aC1872Xyqajk6cEZ+wGWmljqogBdAVkCp0A2r2wZu6L2UjLpjoZ18/0MCEcQq/kNhh1JBLc5UZ6tITZeLAoxNr/3m+i5sxhWApXNqlyaYbJurkFyiRzNTFdjnPUvO16ZivMD176TtMt8k8WiwhlkkNowIOOyncJ+WGnXDfL7uEtjpsd0feDWlbT/mZEhqS+ggO/RJk3ke2cfZ3TYIM70ap8EAZ5qGXkSNxSgLQ2lapHsjR7krbkQFFwYWEcVW9uJdVsUUOlvuQW8zA21596uf5sahAIH8xfltnRX4NTxnngs9GCIf0VFkc6k0WSZ3jhZLQY0bOsxhAWc jEBoLGpL QA9ei+aF57njlPGUhTO/VlwKbi3/fGGIPU3kwam5hL2+56IzK8ZONl6ZSsrMeyXQIFbl+hbX4knfti2mYnHH5Wdk/daLTUTs/N0Iq1G841zjX1CvMl4pzRrDt5yIpleuKAzoLzgUxExxwusc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.010071, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, May 03, 2024 at 02:58:56PM +0200, Jesper Dangaard Brouer wrote: > > > On 02/05/2024 21.44, Shakeel Butt wrote: > > On Wed, May 01, 2024 at 07:22:26PM +0200, Jesper Dangaard Brouer wrote: > > > > > [...] > > > > > > More data, the histogram of time spend under the lock have some strange > > > variation issues with a group in 4ms to 65ms area. Investigating what > > > can be causeing this... which next step depend in these tracepoints. > > > > > > @lock_cnt: 759146 > > > > > > @locked_ns: > > > [1K, 2K) 499 | | > > > [2K, 4K) 206928 > > > |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| > > > [4K, 8K) 147904 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > > > [8K, 16K) 64453 |@@@@@@@@@@@@@@@@ | > > > [16K, 32K) 135467 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > > > [32K, 64K) 75943 |@@@@@@@@@@@@@@@@@@@ | > > > [64K, 128K) 38359 |@@@@@@@@@ | > > > [128K, 256K) 46597 |@@@@@@@@@@@ | > > > [256K, 512K) 32466 |@@@@@@@@ | > > > [512K, 1M) 3945 | | > > > [1M, 2M) 642 | | > > > [2M, 4M) 750 | | > > > [4M, 8M) 1932 | | > > > [8M, 16M) 2114 | | > > > [16M, 32M) 1039 | | > > > [32M, 64M) 108 | | > > > > > > > Am I understanding correctly that 1K is 1 microsecond and 1M is 1 > > millisecond? > > Correct. > > > Is it possible to further divide this table into update > > side and flush side? > > > > This is *only* flush side. > > You question indicate, that we are talking past each-other ;-) > > Measurements above is with (recently) accepted tracepoints (e.g. not the > proposed tracepoints in this patch). I'm arguing with existing > tracepoint that I'm seeing this data, and arguing I need per-CPU > tracepoints to dig deeper into this (as proposed in this patch). Ah my mistake, I just assumed that the data shown is with the given patchset. > > The "update side" can only be measured once we apply this patch. > > This morning I got 6 prod machines booted with new kernels, that contain > this proposed per-CPU lock tracepoint patch. And 3 of these machines have > the Mutex lock change also. No data to share yet... > Eagerly waiting for the results. Also I don't have any concerns with these new traces. > --Jesper