From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46AFCC7EE2A for ; Tue, 24 Jun 2025 05:59:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D39326B007B; Tue, 24 Jun 2025 01:59:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE8936B009A; Tue, 24 Jun 2025 01:59:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD78E6B00CB; Tue, 24 Jun 2025 01:59:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A896A6B007B for ; Tue, 24 Jun 2025 01:59:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7A0A11D9AEC for ; Tue, 24 Jun 2025 05:59:43 +0000 (UTC) X-FDA: 83589242646.17.B184B3A Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf21.hostedemail.com (Postfix) with ESMTP id 99FB31C0004 for ; Tue, 24 Jun 2025 05:59:41 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jtsReQnJ; spf=pass (imf21.hostedemail.com: domain of kundanthebest@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=kundanthebest@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750744781; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m6010/ZxGOxc/GtHfiUtuCWiZZWt6dGaK+o3fK6MDmU=; b=VTlLdHPGKh/tewfWlFyDtP4OtWO+1fD9nGjeBD29b5FV/c1fScj0/DsImFSi8e0SHECji6 9E3JI3l+ozOBNtf+QwzSlbDLYCAIxfAX8vuSFdAAlc9eQBKMhlZxuAy7XbyMynm/ncQ+Lg 1fE4zNjm5+MBCVFDUN0hKBQh4iG8OPk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jtsReQnJ; spf=pass (imf21.hostedemail.com: domain of kundanthebest@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=kundanthebest@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750744781; a=rsa-sha256; cv=none; b=siljFs052N2nUvsBzEZcqY5TYUsBzuk3zmL5FGMNWyKJhdzcFG34ME//ALbdZFiHi1jtVv u0/1DJmAaEFLNXKy4NoEvxK54nTmJqJqhpVzHJbnNkd7UezX03mlDrwi0Wn8I0U1nZU3ux qSH7ZI7Hxdp2dowXt0QNp0lzMQZE738= Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-ade30256175so877728166b.1 for ; Mon, 23 Jun 2025 22:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750744780; x=1751349580; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=m6010/ZxGOxc/GtHfiUtuCWiZZWt6dGaK+o3fK6MDmU=; b=jtsReQnJn0T09lu1Z1i5KTNYuA5MA/0Je60s42kU1IVUnvw2/FR//DARR54QiixqWc skxSILsTRM5HSkTq0nGlKipGkVrT7gR4ZABKLCh6U3oD9DK/Tm+KF8p1l+ImwhPOp+gC oKgDyprIHJ51uCFKWokKnYjVH//D+raLf4vmXklY99wK67DjcU2e3gGxZfY1DwOrq2Cj zfyaIQPULJe4WHLLAx+ucC/Lv6zxvyNjNU767rLrulqPJUgjNjcn3iECH/CPF68kcUV3 Qk9a/5Dw+of4EnvDr9PuevsgfvAC2mp9dmwJzFYPRLUDi27IUlF468dwBbCeJaL6LYe/ VrdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750744780; x=1751349580; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m6010/ZxGOxc/GtHfiUtuCWiZZWt6dGaK+o3fK6MDmU=; b=YXGDDVgovV6z+HcepB0FxyCqtbwDYMv4z88H9RrTeDJTh9VGUhkbVi9X4bNCO4j/+N MKrOsIwqb7bdy+bwlwvqfdLJnIHCoGqFNTVZ8yDwaxOxVtog0fOopAfKajqFYzwQhaDF WLeYumsSVQM9dnHUqvcE/MON3WHyClU/syogFHDUQ4aSVk3TpUgh9kVJgaYZIuRMppYP Uq2F9VpFgkq6+Kn2IjTLvHt8kxY3fxrwSJbRH6Qc/VojgXmo6Y7oI6u2pPGHDi7h5gpv 4o7iprl+nW9/ijww4D+n5+z2DwQZD1r3d6LT52X/dfnW5C+t+6rdaM4V4nYZpw21cbSc 6T5A== X-Forwarded-Encrypted: i=1; AJvYcCW4mR/waKkguCp93S54muU5qcFdv8TjccCEDzaLGI6OX1WSnuyQcVJmxzpmuy+xo9e+8O2H+UhV2g==@kvack.org X-Gm-Message-State: AOJu0YyVJDNbcOrq8BHPAC21a8Pp0UmxBZye/IdkO0L8rrg9AgC5yxYG lpzL1nzSIVkqKLBVSF07sV8jgkcMQXs0XXjjRd1Lb41S0MT1udnoG/AOeQAizGyT3o/Bkw+wP4x 7ju8hfwRchRZ2Nv+HlCWpuDsuHqPH4zM= X-Gm-Gg: ASbGncuUqzkjDNj7HbUSbSKaNrPTVPTZU/CmB4OvZr2PiQM9EwVEodnLVye3baU7VvT QC1UhkRZHsJP8krq/ZCTfpuSM3/eNtLvPHanPlDui+9KpSP1CBBtTi+aK1XsYf6pyeFXZFpWVdC BQziOIDuLI989I0qISFPpwAJetYEcPXL/otw9KCwQ6OuCtf0uVQsUgXt0= X-Google-Smtp-Source: AGHT+IE+xtk8mCeLX4OzNGlO9SwlPPmDyNCsobTZwfKxxEZdu+lYg/k7JddDFzyJ8a/vNVxMgcZ5qdSBpvnFKoOwczc= X-Received: by 2002:a17:907:7e92:b0:adb:229a:f8bd with SMTP id a640c23a62f3a-ae057b6cc1dmr1547486566b.29.1750744779560; Mon, 23 Jun 2025 22:59:39 -0700 (PDT) MIME-Version: 1.0 References: <20250529111504.89912-1-kundan.kumar@samsung.com> <20250602141904.GA21996@lst.de> <20250603132434.GA10865@lst.de> <20250611155144.GD6138@frogsfrogsfrogs> In-Reply-To: <20250611155144.GD6138@frogsfrogsfrogs> From: Kundan Kumar Date: Tue, 24 Jun 2025 11:29:28 +0530 X-Gm-Features: Ac12FXzjJtXhWP5qaSTxOaM8kfMlrnXKIbZrGlp-NuqDUifHrwujHPqe26XNyh8 Message-ID: Subject: Re: [PATCH 00/13] Parallelizing filesystem writeback To: "Darrick J. Wong" Cc: Anuj gupta , Christoph Hellwig , "Anuj Gupta/Anuj Gupta" , Kundan Kumar , jaegeuk@kernel.org, chao@kernel.org, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, miklos@szeredi.hu, agruenba@redhat.com, trondmy@kernel.org, anna@kernel.org, akpm@linux-foundation.org, willy@infradead.org, mcgrof@kernel.org, clm@meta.com, david@fromorbit.com, amir73il@gmail.com, axboe@kernel.dk, ritesh.list@gmail.com, dave@stgolabs.net, p.raghav@samsung.com, da.gomez@samsung.com, linux-f2fs-devel@lists.sourceforge.net, linux-fsdevel@vger.kernel.org, gfs2@lists.linux.dev, linux-nfs@vger.kernel.org, linux-mm@kvack.org, gost.dev@samsung.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 99FB31C0004 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: r3jkyi5wfn33fpx3zbmz7985bf3f3hk3 X-HE-Tag: 1750744781-533616 X-HE-Meta: U2FsdGVkX1+L3oyhdXFbpf9DpyP9dM+xN6FSqRC3ZRIje31o64Rv+UYR7iAZkgKpZYYr+b+qSxJGLd9ZVqrjhOlp8ceoxzqyw/NVTJatktnFuanhUSGR5Aenu0NaaPeHg1brfv0HapIny5ORIdjQp94+Lxq6ZB71uRiqGO9SRmK7AUhKET4cxgn2qaa4KhKaEuJDm+uwPvzoslvS4RnB3I5g27Q3cCnuyDSfgVKApjIHPNK4Hx/YviXbYcg/Q5qwH1R52PzQKhzA9EkOfcXYosPF0z1Xn74kOhE4Z8HT2mmGTyYpGFMqQp7i59Xz6uCSJ2rcuEwrnsxt78e/rCfmRH4GvBG7hroXDjS5e2vUy97e6UUu1umlswUqSb9bKpNm/9824YcmMg916i1VuDe+N3Cl4hihn5insWcTXXi5d/nP+mtfX2wFolcJki0r8VyrGsVkGlpgrWGjVLfu2tub1GtULnRJsC3eBWndI6zkUQF9PyL4P+r9cbMhsOxSGh4SomW7UchYJjTUTgp+xrQzz4AIbhKunpO55fNqw/Lx7y059a86BpODxgkJDCJV0L5TrOZzlTpJpIlRbQzGhaMfucZE7aN5NRWZe8uaBNz1s8rgdpowySl3P+5RiLvzHlDIolA0aIieOfDqAhqffJRarP7zBnHeqozUrwgjIxLRfTQkxNJrCII+s4V0sADTzia6RMp65R3GCFJAgWAnZy1ftt5FwZG9wMTlxpKogqf4t9xyyxFKxP83Zz0KbuGA1op0mbGFd2hSbGeBEr1rvr0XLgxGvWK3c7oGtCcPauMJ/DSZ+edgaX4zqbQjcUeFJeb9qUzc9YKVZiBtqD+k8ofUf2HVGJqsKVgQm5aRESYkxO866D9FVfnKBH+feEKovTd0j8nWbAnTSdO9sYiQpYCSJpZXSv0MPI780h+ZvcxdP66NgS1d+EK2NfVX6u1p0Q9bV3nxm7+ef69cTJj3SLs mzmQo80R DSl2u9/UjJLZGyvOCn9YvqBhrMEvLQvG3mHshq49TW96ejDvh+Xf3nK9wwxZtRWCgAztUtMgxXEcPYacz8qJWzbHT3lmlALizeYBGbJqEVtj+LUKBVB3RrmV74KYPQa+A4GHTkEwXDo/XGmZq+V8HvMryb2KCXVeZzuCOhqLmcUPoaJkPCcVxOfD3QEn5LZdhRYz9xZT/CH9vfDaSAyh1cIXPqhWiimUzNLjR+oVg6sLifNXjEXRVoioY+LIMy7jMCulXY1z59jJohzp5QaDbsIcDPx4XaPHNSzC6TY3l+L98bGqjUZJ5N7kiA38/UChf9okSsOmcpw+Q+xn8qrwSt493xl/3z5NdbZtgowlE2jxL+lKY28xZTq4qnbg1JTWEeetc4bwkRdEc22IOnHKwCaNIFu/IfFqHOikZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jun 11, 2025 at 9:21=E2=80=AFPM Darrick J. Wong = wrote: > > On Wed, Jun 04, 2025 at 02:52:34PM +0530, Kundan Kumar wrote: > > > > > For xfs used this command: > > > > > xfs_io -c "stat" /mnt/testfile > > > > > And for ext4 used this: > > > > > filefrag /mnt/testfile > > > > > > > > filefrag merges contiguous extents, and only counts up for disconti= guous > > > > mappings, while fsxattr.nextents counts all extent even if they are > > > > contiguous. So you probably want to use filefrag for both cases. > > > > > > Got it =E2=80=94 thanks for the clarification. We'll switch to using = filefrag > > > and will share updated extent count numbers accordingly. > > > > Using filefrag, we recorded extent counts on xfs and ext4 at three > > stages: > > a. Just after a 1G random write, > > b. After a 30-second wait, > > c. After unmounting and remounting the filesystem, > > > > xfs > > Base > > a. 6251 b. 2526 c. 2526 > > Parallel writeback > > a. 6183 b. 2326 c. 2326 > > Interesting that the mapping record count goes down... > > I wonder, you said the xfs filesystem has 4 AGs and 12 cores, so I guess > wb_ctx_arr[] is 12? I wonder, do you see a knee point in writeback > throughput when the # of wb contexts exceeds the AG count? > > Though I guess for the (hopefully common) case of pure overwrites, we > don't have to do any metadata updates so we wouldn't really hit a > scaling limit due to ag count or log contention or whatever. Does that > square with what you see? > Hi Darrick, We analyzed AG count vs. number of writeback contexts to identify any knee point. Earlier, wb_ctx_arr[] was fixed at 12; now we varied nr_wb_ctx and measured the impact. We implemented a configurable number of writeback contexts to measure throughput more easily. This feature will be exposed in the next series. To configure, used: echo > /sys/class/bdi/259:2/nwritebacks. In our test, writing 1G across 12 directories showed improved bandwidth up to the number of allocation groups (AGs), mostly a knee point, but gains tapered off beyond that. Also, we see a good increase in bandwidth of about 16 times from base to nr_wb_ctx =3D 6. Base (single threaded) : 9799KiB/s Parallel Writeback (nr_wb_ctx =3D 1) : 9727KiB/s Parallel Writeback (nr_wb_ctx =3D 2) : 18.1MiB/s Parallel Writeback (nr_wb_ctx =3D 3) : 46.4MiB/s Parallel Writeback (nr_wb_ctx =3D 4) : 135MiB/s Parallel Writeback (nr_wb_ctx =3D 5) : 160MiB/s Parallel Writeback (nr_wb_ctx =3D 6) : 163MiB/s Parallel Writeback (nr_wb_ctx =3D 7) : 162MiB/s Parallel Writeback (nr_wb_ctx =3D 8) : 154MiB/s Parallel Writeback (nr_wb_ctx =3D 9) : 152MiB/s Parallel Writeback (nr_wb_ctx =3D 10) : 145MiB/s Parallel Writeback (nr_wb_ctx =3D 11) : 145MiB/s Parallel Writeback (nr_wb_ctx =3D 12) : 138MiB/s System config =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Number of CPUs =3D 12 System RAM =3D 9G For XFS number of AGs =3D 4 Used NVMe SSD of 3.84 TB (Enterprise SSD PM1733a) Script =3D=3D=3D=3D=3D mkfs.xfs -f /dev/nvme0n1 mount /dev/nvme0n1 /mnt echo > /sys/class/bdi/259:2/nwritebacks sync echo 3 > /proc/sys/vm/drop_caches for i in {1..12}; do mkdir -p /mnt/dir$i done fio job_nvme.fio umount /mnt echo 3 > /proc/sys/vm/drop_caches sync fio job =3D=3D=3D=3D=3D [global] bs=3D4k iodepth=3D1 rw=3Drandwrite ioengine=3Dio_uring nrfiles=3D12 numjobs=3D1 # Each job writes to a different file size=3D1g direct=3D0 # Buffered I/O to trigger writeback group_reporting=3D1 create_on_open=3D1 name=3Dtest [job1] directory=3D/mnt/dir1 [job2] directory=3D/mnt/dir2 ... ... [job12] directory=3D/mnt/dir1 > > ext4 > > Base > > a. 7080 b. 7080 c. 11 > > Parallel writeback > > a. 5961 b. 5961 c. 11 > > Hum, that's particularly ... interesting. I wonder what the mapping > count behaviors are when you turn off delayed allocation? > > --D > I attempted to disable delayed allocation by setting allocsize=3D4096 during mount (mount -o allocsize=3D4096 /dev/pmem0 /mnt), but still observed a reduction in file fragments after a delay. Is there something I'm overlooking? -Kundan