From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B6B5C54E41 for ; Sat, 24 Feb 2024 17:55:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0B126B00A0; Sat, 24 Feb 2024 12:55:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CBB206B00A1; Sat, 24 Feb 2024 12:55:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B83836B00A2; Sat, 24 Feb 2024 12:55:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A62576B00A0 for ; Sat, 24 Feb 2024 12:55:55 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2CC79C036C for ; Sat, 24 Feb 2024 17:55:55 +0000 (UTC) X-FDA: 81827450670.11.3ECDAFA Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf03.hostedemail.com (Postfix) with ESMTP id 691CC20016 for ; Sat, 24 Feb 2024 17:55:51 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=L1KE+2OK; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=none (imf03.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708797352; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XR8nVZl4LpgltHQGkRhk6jUPN/LLTQOfmkb1QXGjz5k=; b=Ezw8UhVdCih0ddXAok1noV1TROR0EVaaLESvM9cnNg+gslzwLmqqRAmg0UMGZ+JfdwYdZ3 3SeSi2p7Q93t1OtcathtvUhkiP3f6GeI5WlxI6Kfm+ArWLfjLHiLhD0QYUm7bTRDftK84y 0YSkuUwv740WqLWBWYt/CZPpTH2HtVE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=L1KE+2OK; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=none (imf03.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708797352; a=rsa-sha256; cv=none; b=KEbZzvLuFZHLMIgtIKTJXx++n392V7iA8wnuTqMcqc2bvQPEE7F83Y2m+Yad3omf1Nm+PQ AuSZ4/aLx5hA1L0pSn86yLyM5eULl2/8WUWbSRorSwVvzIBiEG5Q12iUw0PFxTP+ja4UPK 3TaVJihPzbYOiOKAn0Qz3NPaX88Agkg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=XR8nVZl4LpgltHQGkRhk6jUPN/LLTQOfmkb1QXGjz5k=; b=L1KE+2OKmBtW7kaH2huYiLBXD6 2NWnvoTDFS+WzF8QJqhCRKS1zAlocYZQlT14QgM6ztqse4ZlyoeO5z8xwoikQDkMaAcT39dloBQRU HE/aO/oolctuUL/MO/3kojJtzeC413g3uUXdl/7z1f+Wonbqo/zRX8W5SMzQjcthkG2M1Y3NjdNg4 6/NthdY31g4BwQeBgxA80JayH4ACwziEA5pfL1+2K2EuBJobhswwM9Fx45zt1n298AQMvtdKuMNMy 3+6wL7x3mWVkB4PSmllZPg0XICpmCKDY55/5liUk/nHhjOjKjgxTzwRMGBfjWcgesISgxeTIhnCUy vDhFYumw==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rdwFX-0000000DNTS-4Adp; Sat, 24 Feb 2024 17:55:47 +0000 Date: Sat, 24 Feb 2024 09:55:47 -0800 From: Luis Chamberlain To: Matthew Wilcox Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-mm , Daniel Gomez , Pankaj Raghav , Jens Axboe , Dave Chinner , Christoph Hellwig , Chris Mason , Johannes Weiner , Linus Torvalds Subject: Re: [LSF/MM/BPF TOPIC] Measuring limits and enhancing buffered IO Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 691CC20016 X-Stat-Signature: 466ey3oijfdtk49tggaxjfmq4yjn6tot X-Rspam-User: X-HE-Tag: 1708797351-605487 X-HE-Meta: U2FsdGVkX19t27UcyuRpTSyUI6k4iMgoUzgIpdnmOSGvPF4faxPyj2MyG5SRL9U9sMhj5YKwLETOseRq8Zd2HSCtZG79XiZuq0bgLkdu7tL8bH3HbJK2yeqfDNcQK81lKqOdX2X2rtB4K/AY4lOXzOTzLvND/buW3Cy8HaDIlaAHIB43E/NpoJIlZRAjt7mSkx4AtSRgiGh3WAGw6fQWr6AMFKe0A/Z5brNNzXaDYkM5eliiyA2CfRhrxcbmRWosmKkVV5Pl3+OybG03QEzMXLEv6IPWxMiYCb+A4vqYkteQosoBhKc8YGQE53Q8YigPMpemeMevuhKZ6sN+vtAde5CO671+wagrjIdP0FWuKMAE7DKrU/R2DGxFBJXbrfHYmQjMFhCwpflWWBU2ekxPwa9Qdl4r89A5PD9+58U+dJHj8ojam5d9scajFnrVm47ASuT/Ll0cj+Hd+RXNy79xRG4RSQ/wzqWFbWLL6cOpj9wdj9I7YjEqAdjsRznaSaHze05QqlGKpbxRdQKAj1fgOPk8d/aav05SPa3I8PlvIemAvTaswBnCfjLqg7AXNUUGF4CqKbgR7I5zKpHOddNsBbfEHDYZQ5K+ymCkdXoQ4JI7egZ9KDavsxEL09pORQCXPYFJDLt5BouhhOWvoHX2U31gki/lsNMaCSX337yhrhF1O9QJUBDqBGMlRzGEuiJfN+quPXxylyqSHDJui+sWwFqZ586yTLxbZu0t5gBTI/DMcQFvslJmOsyH1PBWTLa+5KEcfT82gmJ1x6eyPEJdE3zJZSlffUf/fxZ5gCqKDzIQ2zEphXUymnXQcHfrOHgs/vC5KB4+3F4oqtP7z5mjeBypOOoszd0Z5B0oUVlqcdnEmFN6Q6J+OxwBRKKmMUDw/QJpHy0BpE/2zBRo6KzvTaz8J1Kho1AUVwNGAVDp91T2+BzQWlgWyz+PECrU4BfQsTDMRR+fyIDz+IkXHRm NyW+qrBu rWYXjg6JiMubxt1HRMdpunz1f0+Vjc1TtMyvz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Feb 24, 2024 at 04:12:31AM +0000, Matthew Wilcox wrote: > On Fri, Feb 23, 2024 at 03:59:58PM -0800, Luis Chamberlain wrote: > > ~86 GiB/s on pmem DIO on xfs with 64k block size, 1024 XFS agcount on x86_64 > > Vs > > ~ 7,000 MiB/s with buffered IO > > Profile? My guess is that you're bottlenecked on the xa_lock between > memory reclaim removing folios from the page cache and the various > threads adding folios to the page cache. If it was lock contention I was hoping to use perf lock record on fio, then perf lock report -F acquired,contended,avg_wait,wait_total If the contention was on locking xa_lock, it would creep up here, no? Name acquired contended avg wait total wait cgroup_rstat_lock 90132 90132 26.41 us 2.38 s event_mutex 32538 32538 1.40 ms 45.61 s 23476 23476 123.48 us 2.90 s 20803 20803 47.58 us 989.73 ms 11921 11921 31.19 us 371.82 ms 9389 9389 102.65 us 963.80 ms 7763 7763 21.86 us 169.69 ms 1736 1736 15.49 us 26.89 ms 743 743 308.30 us 229.07 ms 667 667 269.69 us 179.88 ms 522 522 36.64 us 19.13 ms 335 335 19.38 us 6.49 ms 328 328 157.10 us 51.53 ms 296 296 278.22 us 82.35 ms 288 288 214.82 us 61.87 ms 282 282 314.38 us 88.65 ms 275 275 128.98 us 35.47 ms 269 269 141.99 us 38.19 ms 264 264 277.73 us 73.32 ms 260 260 160.02 us 41.61 ms event_mutex 251 251 242.03 us 60.75 ms 248 248 12.47 us 3.09 ms 246 246 328.33 us 80.77 ms 245 245 189.83 us 46.51 ms 245 245 275.17 us 67.42 ms 235 235 152.49 us 35.84 ms 235 235 38.55 us 9.06 ms 228 228 137.27 us 31.30 ms 224 224 94.65 us 21.20 ms 221 221 198.13 us 43.79 ms 220 220 411.64 us 90.56 ms 214 214 291.08 us 62.29 ms 209 209 132.94 us 27.79 ms 207 207 364.20 us 75.39 ms 204 204 346.68 us 70.72 ms 194 194 169.77 us 32.94 ms 181 181 137.87 us 24.95 ms 181 181 154.78 us 28.01 ms 172 172 145.11 us 24.96 ms 169 169 124.30 us 21.01 ms 168 168 378.92 us 63.66 ms 161 161 91.64 us 14.75 ms 161 161 264.51 us 42.59 ms 153 153 85.53 us 13.09 ms 150 150 383.28 us 57.49 ms 148 148 91.24 us 13.50 ms I'll have to nose dive some more.. but for the life of me I can't see the expected xa_lock contention. Luis