From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E7A9C52D7B for ; Tue, 13 Aug 2024 05:01:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02A636B00A2; Tue, 13 Aug 2024 01:01:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F1BCB6B00A3; Tue, 13 Aug 2024 01:01:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D95AF6B00A4; Tue, 13 Aug 2024 01:01:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B6B426B00A2 for ; Tue, 13 Aug 2024 01:01:35 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 404F28058A for ; Tue, 13 Aug 2024 05:01:35 +0000 (UTC) X-FDA: 82446024150.06.32A185E Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) by imf03.hostedemail.com (Postfix) with ESMTP id 27AEE20004 for ; Tue, 13 Aug 2024 05:01:31 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="IIz/0jjA"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.167.45 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723525235; a=rsa-sha256; cv=none; b=0rIYCw42ILxyomc8uZieuOLf4hp+Kgz2TT6s7/V9O6REj5Em5pFts3WRAmBRKhJ+YiQpfq KwM/ZlMtk3QPha4aWIV7CcB/jb9d3bDkgaaQKtrX5M6sZss+9pcZEKo8fE+qFjzQpckB1f 7c1J8a+lf1qyKhKpujF76Qz3DzF+pQE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="IIz/0jjA"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.167.45 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723525235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=faS0v3HI+ptxTJXy/8GMg5Iu1t2r0HjJ/yVEjDqFytQ=; b=Z116kAx8LSHeF6GsRsXiDn5ktZqWhllbtY/SZQYouzpD4uchujKkjFPSM9aC244jovrkAD MUJ2IF6YTdL8uYFq7wLTHGOV53BtcgbFjMQ8HLzi6k1M4NJrPkoZEKayLYl0OwdnRximCc PCSsYvY4e9jJosn85h372BwxKpjIzug= Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-52f024f468bso5933927e87.1 for ; Mon, 12 Aug 2024 22:01:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723525290; x=1724130090; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=faS0v3HI+ptxTJXy/8GMg5Iu1t2r0HjJ/yVEjDqFytQ=; b=IIz/0jjAVJHXs9VYHtVKmlhTDFhRhN2VrpTvd8SGx99Cawc3tJsVbwWCFPV4cfQDPZ BHZ5vvtU9EgE+rRr5nTRRDcGMExvfZuPCadCaQLD6qTRaJkO0qduhS0GGwKpK1ycb+64 TEf+7TdwZ0Kq7FCMOT6XqxC3sSxN06gN6NoWmsbwHtJ8XtbgcD++WRfwm09E0908XyCX RZYoj175vDMg6CysKm3c9s2ejlzGZ6q+UKPqnj/fyAQfca+Kkc+mtsrqwoWJzpxmD9KP T4i0Kh+bO7ixf4HynxHa5FjPAqqZ+OhqRYT98Kg3xpa8lJ21UoAfy7dMN13WTdA9OA5Y cwCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723525290; x=1724130090; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=faS0v3HI+ptxTJXy/8GMg5Iu1t2r0HjJ/yVEjDqFytQ=; b=hP+EXR7qrtwCA41Lnn1l6bpgkD1p/mO8CnytDcs2XDzHYYvgwSa7Hml1YSxqAPJpoA IfEklEfhiH8VFpyFD9EzP4SEfvrwKa8hYkpqx3Zi3BfF5ptr+9DHzgHtB4ILbyFK04/N Kuyj6USWR1tRTjj2zK6zyK9xi2/zqYG8IC7g19TA/qNx2E3w3tjaaMswTBAYnbLhOoQt FgujbEre0TRNFQZwX17JW9L3iSX5jvmKMgwo+/ArvAG7XA8KDaUSpf0qBo937eJ+vy35 LcSDOdMY7K57qd+L4GW1f4D6IqUnhOqYEiR8qEyU7GV5EF/kg0Ui5obni5ItcBTP8Dgi yeIw== X-Forwarded-Encrypted: i=1; AJvYcCWLZ1ivw2QREyEe+EKM8OnSR2kRqCbvL88ISXlfIxRXne/tphqeUtbpVZk+wj2WBwjSijg1skatHSeQuurQRH6zegA= X-Gm-Message-State: AOJu0YxhlKpsVcba6eQgYtakwWw5+fvPLe7z6w5RLtdgNmUsflUMP3Yn kZZw0vXGDGqjxrUfobtTKvEE/oYcSedkHtVXFJUQl4zoBjnxqu9GsRTYflfzZyWQIMLuxrjLSDO g90NSRnRK+gYYJh5w46gG8q/xvCE= X-Google-Smtp-Source: AGHT+IHg4HejanXu81YcwxgqcvuepXRrvMPaHjRu9QiPBBuRooRXcC8Yxq/ke7UZdeY3R4JwxgIS7Gqm2nE6xQI+vJE= X-Received: by 2002:a05:6512:224e:b0:52e:bf53:1c13 with SMTP id 2adb3069b0e04-5321364a52fmr1386641e87.7.1723525289864; Mon, 12 Aug 2024 22:01:29 -0700 (PDT) MIME-Version: 1.0 References: <20240812150049.8252-1-ioworker0@gmail.com> <9ede36af-fca4-ed41-6b7e-cef157c640bb@huaweicloud.com> In-Reply-To: <9ede36af-fca4-ed41-6b7e-cef157c640bb@huaweicloud.com> From: Lance Yang Date: Tue, 13 Aug 2024 13:00:53 +0800 Message-ID: Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops To: Yu Kuai Cc: =?UTF-8?Q?Michal_Koutn=C3=BD?= , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, cgroups@vger.kernel.org, josef@toxicpanda.com, tj@kernel.org, fujita.tomonori@lab.ntt.co.jp, boqun.feng@gmail.com, a.hindborg@samsung.com, paolo.valente@unimore.it, axboe@kernel.dk, vbabka@kernel.org, david@redhat.com, 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, libang.li@antgroup.com, "yukuai (C)" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 27AEE20004 X-Stat-Signature: if1kzzk9zg8j83c3m66r5i5aoz95poqo X-Rspam-User: X-HE-Tag: 1723525291-894448 X-HE-Meta: U2FsdGVkX18mDENVd4SuY7kXxK4FGK+X2mFqWeacxe/mkhyAMrg4VGUaOCa+hvSRN+zHJxs0kWx3Gr5UJMZg1LCx/wmsKNaa/AWS/nKbqa0b/MbQtdqCujk/sV/TYqdNv5yp1syHGDa+Bf0OvBh/AjmYaEC7PmIRtRBZhhyv3956BS6X6mIZX7tefKIXT9L2zmCrDR8nUAJWBOUEcETciE4eycxwUC0KI9YnN1rCB275n2Rnr7H6qxgTSNGjYB+2dMMC25c12fnKyXUdLaTtcPPYmXFPsp7qTOpXvfctpr5IDBZy31Zmc/GbBYYvTOWGRplCPnjcou62kHUuyozHWgt775QllCdH27gSuBfC2JDbxBoIZxcXyQzRxShlwHID6LrhiivdHc9XnrhPiJ2u3Yi37t97crru9mpJNIHKYh3wcDX414PmRHeoRmdGk8RiFhlWANzEVpfF4WLNwpWTmzKW9N0KhvnMVYoU0lwvRudMhx/FMQeMW3q4+Xw6R3JsduM5y3X22XUHGxzk7oly3uB3HNwSUq2ySnT7szHJAngkIkLeMFDFe+llLG/6cmHPIoLdleydqtUCc5Pv6v66JCU4cwKrGK7lmL//n0/Z6mRZnX5LU+GsELrlkyJPv60bFA1bjMfYxaFhmaTQq9QofnP52OW9jZEAzxf89fNhiSbmshA9Bx73ZWQQZMANhjpnNF9lhatN4L8Rimu6vxHXSaNkChajiDK6LuoqO3lT5SI8rx/FqApMw8hmYyB8Q/4PSjOfsmWw/3/pWz/HdNIprO0CAQtHWIxdOyZwaL8Vq7xK6nbSnO5ygIuJcoGmyepf2fSAOTeGpMZcBIzE+R0DoVIkKXFPPgYwwO+FKIXztEeZ8IgR0TU8JxsnCba+E0/QX60sa0bMwzYfLtuyiutT8v+u7Ur6zeqa5fb90BzQeuDtRml+MUOsZm+bilXR/A8t2HnjeX/IqHxSPODc4r8 ZioCl27i iqeIeYDfr/gBtIvvDNb6tt72PdaZNB1ZzIQB58GCRWgaC/qbQ0RYk6HTh4NP5Jqff+/TYm0olLX1E1mqF57EY0AuwUVXjCZjRA9iuQjUpMmbs8k6MQvYCodQZN1uIh7WLbl3VfODCAM0vna5MVmxh4MVjG78d5u4B8xTtQ52hURj1FzpuV1t4DkieEdfzmBBr17cemgo2CuTtj40J+Ki3SBLjcg5qtT05UI1avcRXuIjuo7zohoig6t1DLwZHPj8qQVxBnDZ3NFFPh9EaAbvAUOm1Hsr8ryk0tPwg4OVIKHHN2frfSPA8m0u5MUF5l8kBFmdguDbkl9cY4sx0ovT6J+NoJ5GKrELrytG+uYLrAGH31o+GJH0qhwHQeZ6j+l9pxuDdPL7sllMsiiuCfF06nDsMBj7lPdE/xB2QJHm7tMX8LOWCYtJjXPLoacvpyzgA9k4El30FPY3lmpbLaKuhIX9ZrIm5zGp6znxOZJwDoxferoHiEvYj2/aPDCSOe+Al8G4HSCV3wA3opGS/NorTXiPKWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002914, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Kuai, Thanks a lot for jumping in! On Tue, Aug 13, 2024 at 9:37=E2=80=AFAM Yu Kuai w= rote: > > Hi, > > =E5=9C=A8 2024/08/12 23:43, Michal Koutn=C3=BD =E5=86=99=E9=81=93: > > +Cc Kuai > > > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang wrote: > >> Hi all, > >> > >> I've run into a problem with Cgroup v2 where it doesn't seem to correc= tly limit > >> I/O operations when I set both wbps and wiops for a device. However, i= f I only > >> set wbps, then everything works as expected. > >> > >> To reproduce the problem, we can follow these command-based steps: > >> > >> 1. **System Information:** > >> - Kernel Version and OS Release: > >> ``` > >> $ uname -r > >> 6.10.0-rc5+ > >> > >> $ cat /etc/os-release > >> PRETTY_NAME=3D"Ubuntu 24.04 LTS" > >> NAME=3D"Ubuntu" > >> VERSION_ID=3D"24.04" > >> VERSION=3D"24.04 LTS (Noble Numbat)" > >> VERSION_CODENAME=3Dnoble > >> ID=3Dubuntu > >> ID_LIKE=3Ddebian > >> HOME_URL=3D"https://www.ubuntu.com/" > >> SUPPORT_URL=3D"https://help.ubuntu.com/" > >> BUG_REPORT_URL=3D"https://bugs.launchpad.net/ubuntu/" > >> PRIVACY_POLICY_URL=3D"https://www.ubuntu.com/legal/terms-and-pol= icies/privacy-policy" > >> UBUNTU_CODENAME=3Dnoble > >> LOGO=3Dubuntu-logo > >> ``` > >> > >> 2. **Device Information and Settings:** > >> - List Block Devices and Scheduler: > >> ``` > >> $ lsblk > >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > >> sda 8:0 0 4.4T 0 disk > >> =E2=94=94=E2=94=80sda1 8:1 0 4.4T 0 part /data > >> ... > >> > >> $ cat /sys/block/sda/queue/scheduler > >> none [mq-deadline] kyber bfq > >> > >> $ cat /sys/block/sda/queue/rotational > >> 1 > >> ``` > >> > >> 3. **Reproducing the problem:** > >> - Navigate to the cgroup v2 filesystem and configure I/O settings: > >> ``` > >> $ cd /sys/fs/cgroup/ > >> $ stat -fc %T /sys/fs/cgroup > >> cgroup2fs > >> $ mkdir test > >> $ echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max > >> ``` > >> In this setup: > >> wbps=3D10485760 sets the write bytes per second limit to 10 MB/s= . > >> wiops=3D100000 sets the write I/O operations per second limit to= 100,000. > >> > >> - Add process to the cgroup and verify: > >> ``` > >> $ echo $$ > cgroup.procs > >> $ cat cgroup.procs > >> 3826771 > >> 3828513 > >> $ ps -ef|grep 3826771 > >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color= =3Dauto 3826771 > >> ``` > >> > >> - Observe I/O performance using `dd` commands and `iostat`: > >> ``` > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > You're testing buffer IO here, and I don't see that write back cgroup is > enabled. Is this test intentional? Why not test direct IO? Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEBAC= K was enabled. $ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK CONFIG_CGROUP_WRITEBACK=3Dy We intend to configure both wbps (write bytes per second) and wiops (write I/O operations per second) for the containers. IIUC, this setup will effectively restrict both their block device I/Os and buffered I/Os. > Why not test direct IO? I was testing direct IO as well. However it did not work as expected with `echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max`. $ time dd if=3D/dev/zero of=3D/data/file7 bs=3D512M count=3D1 oflag=3Ddirec= t 1+0 records in 1+0 records out 536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s real 0m51.637s user 0m0.000s sys 0m0.313s $ iostat -d 1 -h -y -p sda tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 11.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 55.00 0.0k 1.8M 0.0k 0.0k 1.8M 0.0k sda 55.00 0.0k 1.8M 0.0k 0.0k 1.8M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 14.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 14.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 14.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 14.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 13.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 13.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 13.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 13.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 18.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 18.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 12.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M 0.0k sda 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 4.00 0.0k 24.0k 0.0k 0.0k 24.0k 0.0k sda 4.00 0.0k 24.0k 0.0k 0.0k 24.0k 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda1 There are two things that confuse me. First, initially, neither the wbps nor the wiops reached their limits. Second, in the last second, the wbps far exceeded the limit. But if I only set wbps, then everything works as expected with `echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max` > >> ``` > >> ``` > >> $ iostat -d 1 -h -y -p sda > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wr= tn kB_dscd Device > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3= M 0.0k sda > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4= M 0.0k sda > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1= M 0.0k sda > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1= M 0.0k sda1 > > Looks like all dirty buffer got flushed to disk at the last second while > the file is closed, this is expected. The dd command completed in less than a second, but flushing all the dirty buffers to disk took a much longer time. By the time the flushing was completed, the file had already been closed, IIUC. $ time dd if=3D/dev/zero of=3D/data/file5 bs=3D512M count=3D1 1+0 records in 1+0 records out 536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s real 0m0.578s user 0m0.000s sys 0m0.576s $ iostat -d 1 -h -y -p sda tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 74.00 0.0k 664.0k 0.0k 0.0k 664.0k 0.0k sda 74.00 0.0k 664.0k 0.0k 0.0k 664.0k 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 15.00 0.0k 1.1M 0.0k 0.0k 1.1M 0.0k sda 15.00 0.0k 1.1M 0.0k 0.0k 1.1M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 13.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 13.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 12.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 9.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 46.00 0.0k 1.7M 0.0k 0.0k 1.7M 0.0k sda 46.00 0.0k 1.7M 0.0k 0.0k 1.7M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 11.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 6.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 11.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda 11.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 15.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 15.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 8.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 15.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 15.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda 10.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 49.00 0.0k 1.6M 0.0k 0.0k 1.6M 0.0k sda 49.00 0.0k 1.6M 0.0k 0.0k 1.6M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 53.00 0.0k 1.6M 0.0k 0.0k 1.6M 0.0k sda 53.00 0.0k 1.6M 0.0k 0.0k 1.6M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M 0.0k sda 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M 0.0k sda1 tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k sda1 > >> ``` > >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to se= veral > >> hundreds of MB/s. > > > > What it would be on average? > > IOW how long would the whole operation in throttled cgroup take? > > > >> > >> - Testing with wiops set to max: > >> ``` > >> echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > >> ``` > >> ``` > >> $ iostat -d 1 -h -y -p sda > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0= M 0.0k sda1 > >> > >> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrt= n kB_dscd Device > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2= M 0.0k sda > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2= M 0.0k sda1 > > And I don't this wiops=3Dmax is the reason, what need to explain is that > why dirty buffer got flushed to disk synchronously before the dd finish > and close the file? The dd command operates in the background, and it seems that the dirty buffers begin to flush after the command has completed. > > >> ``` > >> The iostat output shows the write operations as stabilizing at around = 10 MB/s, > >> which aligns with the defined limit of 10 MB/s. After setting wiops to= max, the > >> I/O limits appear to work as expected. > > Can you give the direct IO a test? And also enable write back cgroup for > buffer IO. > > Thanks, > Kuai Thanks a lot again for your time! Lance > > >> > >> > >> Thanks, > >> Lance > > > > Thanks for the report Lance. Is this something you started seeing after > > a kernel update or switch to cgroup v2? (Or you simply noticed with thi= s > > setup only?) > > > > > > Michal > > >