From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69191C4345F for ; Fri, 3 May 2024 12:59:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A9356B0083; Fri, 3 May 2024 08:59:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 758866B0085; Fri, 3 May 2024 08:59:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6479F6B0088; Fri, 3 May 2024 08:59:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 46C496B0083 for ; Fri, 3 May 2024 08:59:11 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ED9921C17CF for ; Fri, 3 May 2024 12:59:10 +0000 (UTC) X-FDA: 82077090060.10.7C7280A Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf07.hostedemail.com (Postfix) with ESMTP id B760340004 for ; Fri, 3 May 2024 12:59:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Hd4GZDPD; spf=pass (imf07.hostedemail.com: domain of hawk@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=hawk@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714741149; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qO28MLZn7mcydNfP0G5bQLlFbD6V6eeA7yYSzlusOjc=; b=HR18V4w4Fytg+QkbwgURx4OFohvk0YSC5GqpkxaKovmedpwe57y/8/3HRO1ZtDkACsQawT /gMPaI/thuqvVHFO03WGQsDxmNZA/6iw0cfYGS46Zo9HZ0mIWZMGGN3ZrJZoneQ1AqZnAr LnsJcb1/CLRvNd3OrgrdCLNmzLuyiv4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Hd4GZDPD; spf=pass (imf07.hostedemail.com: domain of hawk@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=hawk@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714741149; a=rsa-sha256; cv=none; b=wHbQUimtLbjn0uV85w9X8T+g6/hUEFdmMTBXorsg/zTcagyZv3+MgGYGkole6Yscrslpob wGF2jOwg38OcleSHiMniJ3HvqwD0gzzUawnRYdQ7Ma5qZ84PmLwq5T/Rcq4TMLjnaD6PT2 Slxpj10kB5NCYPbmqPN0/d0uqVWSgqE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 8B111CE181F; Fri, 3 May 2024 12:59:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF816C4AF14; Fri, 3 May 2024 12:58:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714741141; bh=Uj1cwNv+UCHnN7nqmvGy0hMOZ7WAjQ3MkfqneQLAISk=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=Hd4GZDPDvredSG5fm6vJ52DUbvyC4aJvst2KyRAfrpZuByknfcR9mjqmxATyx7fsg c2FLGnPpsCc1VT3sD3B4wMOB4NKbSiJBGAooFYWR1wgHYwEcXbaTNaM6AquolWuYDY jfSZeahdvQECMks1g2M4YvD7POxUpzDWyoiL6yDzQXLdBOIOzJ8j/Dw2yXss0z5PaC XlTOsuUHWca/Jp3SR7v1vbu/mtjtI4iTLziqoOpZ+mJI2SBH+opZ3hjMpqOBWgZ4O4 4zifmO9dDEBFkMxv471+w0AVTp33PFpH55YfgiGwE6q8/kbWdFSoNd3tUyfjWXToKO uL1EqL/JYv3aA== Message-ID: Date: Fri, 3 May 2024 14:58:56 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints To: Shakeel Butt Cc: Waiman Long , tj@kernel.org, hannes@cmpxchg.org, lizefan.x@bytedance.com, cgroups@vger.kernel.org, yosryahmed@google.com, netdev@vger.kernel.org, linux-mm@kvack.org, kernel-team@cloudflare.com, Arnaldo Carvalho de Melo , Sebastian Andrzej Siewior References: <171457225108.4159924.12821205549807669839.stgit@firesoul> <30d64e25-561a-41c6-ab95-f0820248e9b6@redhat.com> <4a680b80-b296-4466-895a-13239b982c85@kernel.org> <4m3x4rtztwxctwlq2pdorgbv2hblylnuc2haz7ni4ti52n57xi@utxkr5ripqp2> Content-Language: en-US From: Jesper Dangaard Brouer In-Reply-To: <4m3x4rtztwxctwlq2pdorgbv2hblylnuc2haz7ni4ti52n57xi@utxkr5ripqp2> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: B760340004 X-Rspamd-Server: rspam06 X-Stat-Signature: h79if4m5gjd8q15kmobdhxgf4f7mwo46 X-HE-Tag: 1714741148-436348 X-HE-Meta: U2FsdGVkX18OEryUgH3NSRZC8xBjqQ/RWDtC+lwfYuzwnGve1d/Gjwa937w2KVwCIi1kFVoBVRJaIphW/+NNuBihUUPKWNXGXGwYwk6gpZaBCJ8X2pHvw4Au/f9wJzHEnIzTgt9OJrC1GQ1mYJG5ubbg7PkSGkCDkiFjymS5UWUoZd4qnILKFVdLH7jLw/lOu6rstzI9J2sQt+cs9C+ZtHHyrbkZoZgEPznHNjkvJ48/Eozn5S3WZHbJd6DkUBuO8J0pfgX0yA5aUCkFrHd0JkyK83gFsG0qA+KLE66hosBn+dx5HU9Z6nIKEgK56yBNy4dZjWhOdLR/qDTz+sduJa4W/Zi9fJEaQmtHRrC5jn+pSg9JJgHUOmqbX2KLcf1s3lblbqG9LRw5m5xdHZarqs+XkMqugH7l5O29RPO8nwgYGd4mrhpDNXkfV7QuOQSMCEm03qEyiXg7QNnEl28YVeigy4D2ln0plEkdARvfLgyRWIr9znBIrac2O46PJGuG/24msbXWXK9NYja/GgQ/smJq357n5cKD0zK1l20ltQ8cKDdUGFfgbzEveK8T8u+EpAThA1VLxf2g7cUrwSiuoKG5XB5QEXOrmaTYmy0CjLiCxkQak/MlhpgStXtD0vxJSv5uhXZBAdce0QIzpoRtLWQJJwIxcuCda8DyUtBOV6l+VM8ODwC2kvUskfPzNibqsvXVLDqdC9kgDOWl2qwlms3u62IF/XsjK/9jMD4ggwShD7Kiq0l9dERAbaRtRV0eyQh8EO9RYrYHg6r2ywDx34VVeNeaIOMjVpcmb9Jj9dL4hYViacph2BpxU+zNhVkT0j5vpnoiV7muXek/abgnjiC6YC5/7tr+oM97sVrXGVBfqGR/y+mytqstM4O0Qm+LttYfTsIMI1NLj/73tI1pEYkHNweSMu+BphvP0HRa2t2Nxi/sBDC/mfD6VaTORQAUG9IfZDYosxZ9o850BOh x9fHbLvJ DBIj+/toRiHwdBKhhR+F1HYMBmU1oM2ubCOe0nu0LxrFrUz5cGiMDXf/Lqk81sF3ouNCrDZo3h+C4j9+xj2dKhIfqm3TdJW5ilnmGOHxl7akCEnttZukIU+IrrmKC2324JiCwteiOVDZ37Y0xBcoBrJWA8q5557NIBfkxJ8O16Rn3BMeo+nq+S3uDxigX9/9tgAkiBsj9Z+wgfvq7u0BSqQRpxXpiTAgJQ43H+1PKZjBpLwimKkbTV9Qr8o1sgZYunL8SkInflMwxTMm8ZF9BJ3W1wA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002059, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 02/05/2024 21.44, Shakeel Butt wrote: > On Wed, May 01, 2024 at 07:22:26PM +0200, Jesper Dangaard Brouer wrote: >> > [...] >> >> More data, the histogram of time spend under the lock have some strange >> variation issues with a group in 4ms to 65ms area. Investigating what >> can be causeing this... which next step depend in these tracepoints. >> >> @lock_cnt: 759146 >> >> @locked_ns: >> [1K, 2K) 499 | | >> [2K, 4K) 206928 >> |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| >> [4K, 8K) 147904 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | >> [8K, 16K) 64453 |@@@@@@@@@@@@@@@@ | >> [16K, 32K) 135467 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | >> [32K, 64K) 75943 |@@@@@@@@@@@@@@@@@@@ | >> [64K, 128K) 38359 |@@@@@@@@@ | >> [128K, 256K) 46597 |@@@@@@@@@@@ | >> [256K, 512K) 32466 |@@@@@@@@ | >> [512K, 1M) 3945 | | >> [1M, 2M) 642 | | >> [2M, 4M) 750 | | >> [4M, 8M) 1932 | | >> [8M, 16M) 2114 | | >> [16M, 32M) 1039 | | >> [32M, 64M) 108 | | >> > > Am I understanding correctly that 1K is 1 microsecond and 1M is 1 > millisecond? Correct. > Is it possible to further divide this table into update > side and flush side? > This is *only* flush side. You question indicate, that we are talking past each-other ;-) Measurements above is with (recently) accepted tracepoints (e.g. not the proposed tracepoints in this patch). I'm arguing with existing tracepoint that I'm seeing this data, and arguing I need per-CPU tracepoints to dig deeper into this (as proposed in this patch). The "update side" can only be measured once we apply this patch. This morning I got 6 prod machines booted with new kernels, that contain this proposed per-CPU lock tracepoint patch. And 3 of these machines have the Mutex lock change also. No data to share yet... --Jesper