From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62CE4C52D7C for ; Fri, 23 Aug 2024 12:05:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B40FD80092; Fri, 23 Aug 2024 08:05:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF21980084; Fri, 23 Aug 2024 08:05:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91B5880092; Fri, 23 Aug 2024 08:05:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6B47280084 for ; Fri, 23 Aug 2024 08:05:33 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1208FA9BF7 for ; Fri, 23 Aug 2024 12:05:33 +0000 (UTC) X-FDA: 82483380546.29.5F1E89E Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 18C1AC002E for ; Fri, 23 Aug 2024 12:05:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dG6D9d1A; spf=pass (imf28.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724414650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D0VE6+ZYrN9BjyYYtqs8go1pj9o6RckfxkFxS0nV9kY=; b=ZVubNAgRZ1cjWXhafVKzYAp2Q8MfdIMJ0GsECnsJtomDZWZEeLKdze5H6HOAqR8PkSeULA MAUsWL6V5ci2ejwjvR8P4eFEQf9y3hgTJ06diSMri43laKgyswHSpECS/3QIwhmOiT9oU9 XNzEWLMK7/H6L6pTM1LRY3SCv+1P1rE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724414650; a=rsa-sha256; cv=none; b=SHr8tbDuzRwq8iESjlBtkmEKbeJV1f0v958Zh96MdsW18MyErDgQo3JuLG+dFoCvONY/4v 3tQe2T+IaNYkLh/oze6hA2GLuMp3xy54ikM62EMf/wuYD9hHHVzFc85ZPSRVFw2aKpeFKr bG3k56x1TavpcD4zWcsrXtBvsX1tNGI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dG6D9d1A; spf=pass (imf28.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-201e64607a5so13668405ad.2 for ; Fri, 23 Aug 2024 05:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724414730; x=1725019530; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D0VE6+ZYrN9BjyYYtqs8go1pj9o6RckfxkFxS0nV9kY=; b=dG6D9d1AGJfE5rcdQkW/jD3XQ2ru8g8quZgF5ReOvXIkRVeMqsP8HnvQmPuydS3Spp eD3bqc6CXFjSM7buAJoxh/g2lt6MUm+WrcVzeDcdg9A+OW6JZ4FQ/ThBOjEZjPGw+NgM hxCVxICMXdfn93ZdYXG8KZ90d3l2V1dqqiPUISqq3Z3C6RXcAbCUo3RSM+EmH/5M+ygK UKFbZS3CDD5UU8EOa4B8OFM5ocrDrAI/3P1sjQX3B2gIXPKBsbDonh2beW19n+p+AW8s gsGnuHOCG+lPM8OJhjGK1wU8CGHa1ySLueDns9w12JZF9rPfJQkChD+KMm1eeAKEoR5b +qwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724414730; x=1725019530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D0VE6+ZYrN9BjyYYtqs8go1pj9o6RckfxkFxS0nV9kY=; b=Si3/DF/ThQ5lBuYH83gQwRdNm3zP9jeMAEubOipT82TCpZejpbqfqskEQtjf4bDhhh UlGvBHNvMj5aBaHO2W91OmNCLF2+joOgG27D2zSAHgehSGuqH3E+G/R5MojE7bmyey0w Vxogm4LGtxiJJmrFzzsFYfVePrDAzQEDfbUbwrNMOAYvnhWo/59oVDRRVau/N8POOJQF 8Hn8AuW4sEVO8hTmszxtBbTOS9ZJEzTFOyOOeDFPlIA0oe1H7QCkTwFuiFOXDJET4evs W0JHMO20+t3QTUVHG+ZchivcFhjQnwn40c2N48x44VtTTsll/we4l5zw+PxXOGnUkW5o x4KQ== X-Forwarded-Encrypted: i=1; AJvYcCXaBDQAZ04iDmZoFDT1gwynkosdq3CfCnsZH1pUNaX6nitHskVqNqAaahdUomEfJk6cGFrdma4aBw==@kvack.org X-Gm-Message-State: AOJu0YwVGjrGY/YB7BtormkCO/JUuhzS2t0fp5AtqLvAKqiOVRBhqm+h bowSeaoz2UGGoSX6CLnhHNG4T/1YjrHDX3SF3/JxnR1Mx5/xW/Eh X-Google-Smtp-Source: AGHT+IEO6vi7aDxUStR4vKn6XDyg1NSSxU2zhFU6iMHn0Wj3RSHkrpPNZRQJloReLTJaYDsR8rgAIA== X-Received: by 2002:a17:902:e88f:b0:1fd:96e1:801e with SMTP id d9443c01a7336-2039e4f6ae6mr17650795ad.51.1724414729187; Fri, 23 Aug 2024 05:05:29 -0700 (PDT) Received: from EBJ9932692.tcent.cn ([124.156.216.125]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20385609dddsm27009595ad.196.2024.08.23.05.05.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 23 Aug 2024 05:05:28 -0700 (PDT) From: Lance Yang To: yukuai1@huaweicloud.com Cc: 21cnbao@gmail.com, a.hindborg@samsung.com, axboe@kernel.dk, baolin.wang@linux.alibaba.com, boqun.feng@gmail.com, cgroups@vger.kernel.org, david@redhat.com, fujita.tomonori@lab.ntt.co.jp, ioworker0@gmail.com, josef@toxicpanda.com, libang.li@antgroup.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mkoutny@suse.com, paolo.valente@unimore.it, tj@kernel.org, vbabka@kernel.org, yukuai3@huawei.com Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops Date: Fri, 23 Aug 2024 20:05:02 +0800 Message-ID: <20240823120510.61853-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <7c3499ac-faa7-cc0c-2d90-b8291fce5492@huaweicloud.com> References: <7c3499ac-faa7-cc0c-2d90-b8291fce5492@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 18C1AC002E X-Stat-Signature: op6fwfjp9rg8c6btni11914skc4rbtsa X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1724414730-451959 X-HE-Meta: U2FsdGVkX1+wsdNMIjLFbBJRFR/iFdz5DfV6YPgzuhcpBKG8VJPK0lx4zP0Eu1F1+i7IOafIgbPcgW2nCoDvX9Fb8NntQCGF/5ng8l1Dscti2aLxfgAP/NC5KKdObNuUc6noOM4u080mGF0jB9/AxibyNGfhC/XS0coFuCVgd8HulrOnrCqcmlihvrYxaK9MvtKEpSPh6yUSGZ1s3u2uCG41GU07Y9FxnvOQdFaF+/0FdAMf/hg6Ma1Bj8ZqlKMFTA+KqDD201v6wcvnRb4DBM6M5P36tjmcbX6o71azawflVzh2Z+6ALeYvqTwQ4z1tZ6bQooEwqCbKKvx9rzYPe7rY0QHnpTkVq6DaUDle9mTusOLuRPxV6R1xvCLYNWCYZQTZRtIIOVCwdgX1KVvMezWXoOY5dxEvL42LJXTk5DPyrGar4c2a1jtyckI+0rGRITVV/4tdNbW/l/8dGY+ciZUqlBpq/QZOZMtYWTko3SiBEfp6RRuvyltx6pTOeEPJXF7Ww/D3qTZZjLhFindt9F38Fiq15ispg8aeem2Jjjn1HtmLUXc8mWu+T45NNyI3hqyA1D65wJKN6gsaGPEOoRnhki01GMDj/gxNb8jEVpnUjDv8c7jbB8mf1qMnvT4LYqb9+eOVOesFCp8zH+P8OvSntfl8t9PKl1XjY3eT28Has9KF9YW/uxb9wc7mLc7zsb5GmuzIqBYGdIuMXuYFYYHaQHz4QUh98lK8C+dZlS1qoMmb8RdE9da9aePiN90rZmA29UNrr7zYwyN8eFHrWl0bFaH9UOKuNb3Hk5mXFqc/1WmmFl/YKAXETPO4q0d1KSPO11k4hiMIaR/QNXn/AsefLMNudZBcjEyJpSUJoa28C0ZjMzp6lvvXE1QMY3RgAEvEA5U0jEIzY+kuMkzLagsc2dc1jWrg9NW2K8GtlcwsuX8Jg+bRFROnmgWBzWHtaqUmsMzCIz8ZbfMtOWn yVnfU65s isV9GkZEj16GdFLFzom4k8J2xmWNt6abGeB5k1l02a7OYYQo6GBybPh9tL+X4lqJZh7KJ5KE7PDQt5mEVv3G88xkVPayiGDoiDZ9eXA+tOCzdUqMhvwnfap+1fWQ0UMREAMf419Q0/3vFSVy2zxvRGgPrARo9zKNkP3TE35ftdXLIOJRTyAHxf2JFd4qt1kN1NEkt5z876mI91inN2P62RczXGr0W8SuQJyyW97WFbxfjgu7qRuqOqegHnKFXaohxkr8sx2c9b4vjpUDe5nRaWuvgrgx6a5tdqzfRV8Nj6fsVa0Sx0TV1rFdm+Ep1Xbdud9tP81PbGUEObqEPPPQ7/jzvCEWSfhPpAYG5b+UwQpOVG95QleCJDg2OP1O8hSI83T+vcQFp4AfK+sCyYkyic/iRBpp992HvHRvCTTpeuqbqDYU8SC9oMhQamg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.003105, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: My bad, I got tied up with some stuff :( Hmm... tried your debug patch today, but my test results are different from yours. So let's take a look at direct IO with raw disk first. ``` $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 90G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi └─sda2 8:2 0 88.9G 0 part / sdb 8:16 0 10G 0 disk $ cat /sys/block/sda/queue/scheduler none [mq-deadline] $ cat /sys/block/sda/queue/rotational 0 $ cat /sys/block/sdb/queue/rotational 0 $ cat /sys/block/sdb/queue/scheduler none [mq-deadline] $ cat /boot/config-6.11.0-rc3+ |grep CONFIG_CGROUP_ # CONFIG_CGROUP_FAVOR_DYNMODS is not set CONFIG_CGROUP_WRITEBACK=y CONFIG_CGROUP_SCHED=y CONFIG_CGROUP_PIDS=y CONFIG_CGROUP_RDMA=y CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_HUGETLB=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_BPF=y CONFIG_CGROUP_MISC=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NET_PRIO=y CONFIG_CGROUP_NET_CLASSID=y $ cd /sys/fs/cgroup/test/ && cat cgroup.controllers cpu io memory pids $ cat io.weight default 100 $ cat io.prio.class no-change ``` With wiops, the result is as follows: ``` $ echo "8:16 wbps=10485760 wiops=100000" > io.max $ dd if=/dev/zero of=/dev/sdb bs=50M count=1 oflag=direct 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.05893 s, 10.4 MB/s $ dmesg -T [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 2984 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 6176 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 7224 ffff0000fb3a9180 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a8640 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a9400 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a8c80 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a9040 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a92c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 4096 ffff0000fb3a8000 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 2984 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 1400 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 6176 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 3616 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 7224 ffff0000fb3a9180 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 4664 ffff0000fb3a9180 [Fri Aug 23 11:04:09 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a8640 [Fri Aug 23 11:04:09 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a8640 [Fri Aug 23 11:04:10 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a9400 [Fri Aug 23 11:04:10 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a9400 [Fri Aug 23 11:04:11 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a8c80 [Fri Aug 23 11:04:11 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a8c80 [Fri Aug 23 11:04:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a9040 [Fri Aug 23 11:04:12 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a9040 [Fri Aug 23 11:04:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a92c0 [Fri Aug 23 11:04:12 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 4096 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1536 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1536 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 1400 ffff0000fb3a8f00 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 3616 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1056 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1056 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 4664 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 2104 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 2104 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a92c0 ``` And without wiops, the result is quite different: ``` $ echo "8:16 wbps=10485760 wiops=max" > io.max $ dd if=/dev/zero of=/dev/sdb bs=50M count=1 oflag=direct 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.08187 s, 10.3 MB/s $ dmesg -T [Fri Aug 23 10:59:10 2024] __blk_throtl_bio: bio start 2880 ffff0000c74659c0 [Fri Aug 23 10:59:10 2024] __blk_throtl_bio: bio start 6992 ffff00014f621b80 [Fri Aug 23 10:59:10 2024] __blk_throtl_bio: bio start 92528 ffff00014f620dc0 [Fri Aug 23 10:59:10 2024] blk_throtl_dispatch_work_fn: bio done 2880 ffff0000c74659c0 [Fri Aug 23 10:59:11 2024] blk_throtl_dispatch_work_fn: bio done 6992 ffff00014f621b80 [Fri Aug 23 10:59:15 2024] blk_throtl_dispatch_work_fn: bio done 92528 ffff00014f620dc0 ``` Then, I retested for ext4 as you did. ``` $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 90G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi └─sda2 8:2 0 88.9G 0 part / sdb 8:16 0 10G 0 disk $ df -T /data Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sda2 ext4 91222760 54648704 31894224 64% / ``` With wiops, the result is as follows: ``` $ echo "8:0 wbps=10485760 wiops=100000" > io.max $ rm -rf /data/file1 && dd if=/dev/zero of=/data/file1 bs=50M count=1 oflag=direct 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.06227 s, 10.4 MB/s $ dmesg -T [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 2984 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 6176 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 7224 ffff0000fb3a9180 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a8640 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a9400 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a8c80 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a9040 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 16384 ffff0000fb3a92c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 4096 ffff0000fb3a8000 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 2984 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 1400 ffff0000fb3a8f00 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 6176 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 3616 ffff0000fb3a97c0 [Fri Aug 23 11:04:08 2024] blk_throtl_dispatch_work_fn: bio done 7224 ffff0000fb3a9180 [Fri Aug 23 11:04:08 2024] __blk_throtl_bio: bio start 4664 ffff0000fb3a9180 [Fri Aug 23 11:04:09 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a8640 [Fri Aug 23 11:04:09 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a8640 [Fri Aug 23 11:04:10 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a9400 [Fri Aug 23 11:04:10 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a9400 [Fri Aug 23 11:04:11 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a8c80 [Fri Aug 23 11:04:11 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a8c80 [Fri Aug 23 11:04:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a9040 [Fri Aug 23 11:04:12 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a9040 [Fri Aug 23 11:04:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000fb3a92c0 [Fri Aug 23 11:04:12 2024] __blk_throtl_bio: bio start 13824 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 4096 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1536 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1536 ffff0000fb3a8000 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 1400 ffff0000fb3a8f00 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 3616 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1056 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1056 ffff0000fb3a97c0 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 4664 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 2104 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 2104 ffff0000fb3a9180 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a8640 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a9400 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a8c80 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a9040 [Fri Aug 23 11:04:13 2024] blk_throtl_dispatch_work_fn: bio done 13824 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 11264 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 11264 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 8704 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 8704 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 6144 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 6144 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 3584 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 3584 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio start 1024 ffff0000fb3a92c0 [Fri Aug 23 11:04:13 2024] __blk_throtl_bio: bio done 1024 ffff0000fb3a92c0 ``` And without wiops, the result is also quite different: ``` $ echo "8:0 wbps=10485760 wiops=max" > io.max $ rm -rf /data/file1 && dd if=/dev/zero of=/data/file1 bs=50M count=1 oflag=direct 1+0 records in 1+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.03759 s, 10.4 MB/s $ dmesg -T [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 2904 ffff0000c4e9f2c0 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 5984 ffff0000c4e9e000 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 7496 ffff0000c4e9e3c0 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 16384 ffff0000c4e9eb40 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 16384 ffff0000c4e9f540 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 16384 ffff0000c4e9e780 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 16384 ffff0000c4e9ea00 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 16384 ffff0000c4e9f900 [Fri Aug 23 11:05:07 2024] __blk_throtl_bio: bio start 4096 ffff0000c4e9e8c0 [Fri Aug 23 11:05:07 2024] blk_throtl_dispatch_work_fn: bio done 2904 ffff0000c4e9f2c0 [Fri Aug 23 11:05:07 2024] blk_throtl_dispatch_work_fn: bio done 5984 ffff0000c4e9e000 [Fri Aug 23 11:05:08 2024] blk_throtl_dispatch_work_fn: bio done 7496 ffff0000c4e9e3c0 [Fri Aug 23 11:05:09 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000c4e9eb40 [Fri Aug 23 11:05:09 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000c4e9f540 [Fri Aug 23 11:05:10 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000c4e9e780 [Fri Aug 23 11:05:11 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000c4e9ea00 [Fri Aug 23 11:05:12 2024] blk_throtl_dispatch_work_fn: bio done 16384 ffff0000c4e9f900 [Fri Aug 23 11:05:12 2024] blk_throtl_dispatch_work_fn: bio done 4096 ffff0000c4e9e8c0 ``` Hmm... I still hava two questions here: 1. Is wbps an average value? 2. What's the difference between setting 'max' and setting a very high value for 'wiops'? Thanks a lot again for your time! Lance