From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C262C52D7C for ; Fri, 23 Aug 2024 12:20:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBE416B01A9; Fri, 23 Aug 2024 08:19:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D47366B01F3; Fri, 23 Aug 2024 08:19:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC0AF6B01FB; Fri, 23 Aug 2024 08:19:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 99FD66B01A9 for ; Fri, 23 Aug 2024 08:19:59 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3EE9FC18BA for ; Fri, 23 Aug 2024 12:19:59 +0000 (UTC) X-FDA: 82483416918.23.F4B54A6 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf07.hostedemail.com (Postfix) with ESMTP id 6033F40002 for ; Fri, 23 Aug 2024 12:19:57 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="TUawN1J/"; spf=pass (imf07.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724415580; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UNUCMHj/hyaQMFc8KHdZF7HyFyUqAA9VUiUHkYXLnpM=; b=rGGzPA2m/sAv8z+doUb1fdAwbeFpVuJrcXNtM5EGgB8CYCaLAk4/mN/4oCJNy3EeUBKdNA 14CcL3VZtIVkV9dRaymsK4K8iSMTy/rcEt3ia7oDyixi0UoQeqVMuVdsO7vyZNvBL76aSO JRCUqOSNVKs9AY/qd9orHMu1BFiMtqM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="TUawN1J/"; spf=pass (imf07.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724415580; a=rsa-sha256; cv=none; b=TyuiDagu1XGq5ldIOJokmiZDiOVt9TWo76IHk+H5/E2XFFRE/2sV0LDNBqvcjK6D4e8OBB G8lCzmvXb2lOyKlLW9gs40Uh1x1lnJCslzsfaEfzd4VjaNrJpW8NxfIeTmXR3psO8I1vI5 KfbFETU5B8Bb1+PRY7fUUcl6WYXEjm0= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-713edc53429so1541494b3a.2 for ; Fri, 23 Aug 2024 05:19:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724415596; x=1725020396; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UNUCMHj/hyaQMFc8KHdZF7HyFyUqAA9VUiUHkYXLnpM=; b=TUawN1J/NcIYAIeV7vD+Mr5At7njrAlJ5jO48bwxgei7thVCubZnqodvbhPWGxtxpJ uChwL6ImRCpXhPJ22ac2h/NdLRvImyip758Z1w2p70qs5urHaJXZelTejBrv+OoTPjFS pksZXL4L2konjQKJ59kpXpgnEXRu+8w2YC+ScM7ePcDuefGD4EvlTVp02TCPWRdgyyy9 wuX7muJ4foHNGYYcwuY9lJJAQEmhed7A/IYMdMEj2YunCaw8Bn53FBUDjqfbH77QVvQO DB1Vv8EdBnSurAYgdA7bJGZwEzN3nPDDB2Sn5wUec4VQsTOnJ7i1yXazutBFstW2MnkY 3crQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724415596; x=1725020396; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UNUCMHj/hyaQMFc8KHdZF7HyFyUqAA9VUiUHkYXLnpM=; b=RfZizDiZVPXIbIqgqf71MAcDzgZB7hMv8EZtps81uUu1zE6oofZifwmPG195i6ZRaD jFD0lF94BdOAZAt55/mCsZsxcBIbr7rihpPoCE0Aj71CChi5BVislefUCHPMp8Gp7c1S LEXu2nbZUXh5DlQ2BJOfcOmqjjRsxOjcCbBYh7Vu+LjvEfDguL6pLgkg7l1Ns6CycesO 9oGn1i5aboy8y2V8hAiCCDIC+hJ8UpmW1u4nH23g6jsG2PnlxXHPEHqMmWfmy9ZXRFFw CoRf1euyTZmAynUGGo1rvUKQEeLyFMEk2dt8PRAYNqfcXaNUdPZsOLYUySLz+wjozPnj THeg== X-Forwarded-Encrypted: i=1; AJvYcCWjls+AvEMsNI5lF0EU/uDtKJAiVbIp4o1eXT24IKX50qhoFHSQk4yLauZCi4zEQRVJVhYT4ZfefA==@kvack.org X-Gm-Message-State: AOJu0YzL/BLekLUxk+9CKrtPKAVRtHdKFt2ICQdbcJ+h32TfKQTfoYfj Co21Nej7j1VM5a//s06av6GjxnncVEUg3qOph+W5K3UDqZpV/w+N X-Google-Smtp-Source: AGHT+IGzc7ioPCbQCUlxcip72zBvSMtYRgGsnomQD6AMKfga1Ybtd8vRCH2kemIvGrXuqleOIpl3tQ== X-Received: by 2002:a05:6a20:e18a:b0:1c3:ff33:246b with SMTP id adf61e73a8af0-1cc89ed999cmr2643659637.40.1724415595687; Fri, 23 Aug 2024 05:19:55 -0700 (PDT) Received: from EBJ9932692.tcent.cn ([124.156.216.125]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ad5529csm2955543a12.68.2024.08.23.05.19.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 23 Aug 2024 05:19:55 -0700 (PDT) From: Lance Yang To: yukuai1@huaweicloud.com Cc: 21cnbao@gmail.com, a.hindborg@samsung.com, axboe@kernel.dk, baolin.wang@linux.alibaba.com, boqun.feng@gmail.com, cgroups@vger.kernel.org, david@redhat.com, fujita.tomonori@lab.ntt.co.jp, ioworker0@gmail.com, josef@toxicpanda.com, libang.li@antgroup.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mkoutny@suse.com, paolo.valente@unimore.it, tj@kernel.org, vbabka@kernel.org, yukuai3@huawei.com Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops Date: Fri, 23 Aug 2024 20:19:39 +0800 Message-ID: <20240823121939.65934-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <7c3499ac-faa7-cc0c-2d90-b8291fce5492@huaweicloud.com> References: <7c3499ac-faa7-cc0c-2d90-b8291fce5492@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: qaouo1k673fspjkptq96ttcj3au51j5o X-Rspamd-Queue-Id: 6033F40002 X-Rspamd-Server: rspam11 X-HE-Tag: 1724415597-141713 X-HE-Meta: U2FsdGVkX1/FtrtgE8WfOOw2GEJBXLyo8bPkiWIYjTK3cuUbuzMpnU54Q3Z/JKq93tXUZnCCOyTk+8bH9aj8qEfPPKc6LSEL+gW6BwyG0lm2jaO2lLvak8cPkcDvd5FazcXFDkM43gv/AU+dH/C5m0YS7YU50UM5TJEyKj1Iw26H90jpNKueaIL1CMa2ytWCgLlnPKHIUUL6MDZrxttZJKXBUIdroiREhk565ijzwk2XqfNaGdRFtHQJ2ECElmrxtmbWWBlLUZ38cY1lj3stKFQyXl6KUhdptNhgp9WZ0oMuOgDiG7r4cr4Ce5PbyF9YVYFR3U6jhuDmsC4EiJibSp80SsOhPpj7DLOCo1g7uCJEi5e4ayVQuQG3d04wK9Td6DaxDQ2LCaydNARVlEVmxvgjnUl8Y5LjwyIF43GjnUKHNWvE/Z+7vQeE1k9igk5mlKB7FSR/Yz/2c4gVRO7B8cqBsDX/x+dnE0GRdqpf3V/bjYOz/bAcd9HJjJ6ChBDFX47bV7nYoP+KAesO3Wfv1P7bKBwSLTWuhUWIoXW/1XGevi7ZvkwG4tBtJHSDqQceoy+XLQvOoBlGefiggXW89QXnSkLLq+Z8RkyNWxzAyk1dxhIyi6IdpYVZnswOsWvk44JUVOz0ZqAQWWwizGQUcw/iV4G7An2e2xbGpzg5FtOmyAXoKNj4XeruX8W6G8BAdIF0hue8VH1aF2sqJxqB5uIVGOZe8tQyyobZWKDLm0ppV+fzsIIq+pi61JvCg7Rk3LWCU1ML6f0xccVcykriel/IbMAUIDpdz1GCwOxXZJFUfj6m1L/r6GKG6iAFvYhJsYrPQrfqa4kPRWHEBnL2Jgu7B7xc4a3IPocQG02Joq3UjgXt4QcFeW5xPgn6jL2BSRPBAAhQaie5TIfdJWGHlmbBKtHkz01jRlpkk7V3jYnBnJEC8SRIKCHkkCYth3QoAqi+31r/jOSH+wBmbrj kW8bgKcZ iT0tq57FS7g+C3D9wbXQ5Tb53LEGjVJu5c6yLNDoN1usnrBC8Ytjx1hI54OOEqRWgsL3rOtEmvvOt1lquhYGqd2b3U27i05zLAW+tff++Fx/Ijoc7iJTdxX65stFzEVJ3DIsW38W7zu3eS7drDckhXMEaXJgsBwe+bWvw2nRMLDI8oZO9YxEaYjzcp3EHBP+np5vP69L2uXHpqcBIlCYzBktY/9EZXhxwnbbtWBCExBwQE1OD2giEPRXrjixi4PmUivyZlo/QqzuqJ+Vt0L1AFJ5ZSx5kRQOHf6KMK1Hr73VOFQMK+K/jC4tTPpbwqSj8cIIFA11LDRlvLa6NtqFJZd4VTM2GCqoPVwdPEcIBrP5/GrfHynEYuN1mkOagQ9KG6GP4CyMoLxqgfvqRra7QJsFhAQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.007651, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Forget to add the test result of buffered IO: With wiops, the result is as follows: ``` $ echo "8:0 wbps=10485760 wiops=100000" > io.max $ rm -rf /data/file1 && dd if=/dev/zero of=/data/file1 bs=50M count=1 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.062217 s, 843 MB/s $ dmesg -T [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5ac500 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5adb80 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5ac140 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5acdc0 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5ac280 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 16384 ffff0000ce5ada40 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 4096 ffff0000ce5adcc0 [Fri Aug 23 12:09:10 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5ac500 [Fri Aug 23 12:09:10 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5ac500 [Fri Aug 23 12:09:11 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5adb80 [Fri Aug 23 12:09:11 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5adb80 [Fri Aug 23 12:09:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5ac140 [Fri Aug 23 12:09:12 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5ac140 [Fri Aug 23 12:09:13 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5acdc0 [Fri Aug 23 12:09:13 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5acdc0 [Fri Aug 23 12:09:14 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5ac280 [Fri Aug 23 12:09:14 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5ac280 [Fri Aug 23 12:09:14 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000ce5ada40 [Fri Aug 23 12:09:14 2024] __blk_throtl_bio: bio start 13824 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 4096 ffff0000ce5adcc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1536 ffff0000ce5adcc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1536 ffff0000ce5adcc0 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5ac500 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5adb80 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5ac140 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5acdc0 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5ac280 [Fri Aug 23 12:09:15 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 11264 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 11264 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 8704 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 8704 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 6144 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 6144 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 3584 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 3584 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio start 1024 ffff0000ce5ada40 [Fri Aug 23 12:09:15 2024] __blk_throtl_bio: bio done 1024 ffff0000ce5ada40 ``` And without wiops, the result is quite different as well: ``` $ echo "8:0 wbps=10485760 wiops=max" > io.max $ rm -rf /data/file1 && dd if=/dev/zero of=/data/file1 bs=50M count=1 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.0791369 s, 663 MB/s $ dmesg -T [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87ca3c0 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87ca000 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87cb2c0 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87cb040 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87cac80 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 16384 ffff0000f87cb400 [Fri Aug 23 12:16:50 2024] __blk_throtl_bio: bio start 4096 ffff0000f87ca640 [Fri Aug 23 12:16:51 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87ca3c0 [Fri Aug 23 12:16:52 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87ca000 [Fri Aug 23 12:16:53 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87cb2c0 [Fri Aug 23 12:16:54 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87cb040 [Fri Aug 23 12:16:54 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87cac80 [Fri Aug 23 12:16:55 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000f87cb400 [Fri Aug 23 12:16:55 2024] blk_throtl_dispatch_work_fn: bio done 4096 ffff0000f87ca640 ``` Thanks, Lance