From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C424C52D7B for ; Tue, 13 Aug 2024 06:18:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D15A86B009A; Tue, 13 Aug 2024 02:18:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C78F36B009E; Tue, 13 Aug 2024 02:18:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2E1F6B009F; Tue, 13 Aug 2024 02:18:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7A9EC6B009A for ; Tue, 13 Aug 2024 02:18:12 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C00A1A05F5 for ; Tue, 13 Aug 2024 06:18:11 +0000 (UTC) X-FDA: 82446217182.02.BC5F431 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf06.hostedemail.com (Postfix) with ESMTP id BB908180017 for ; Tue, 13 Aug 2024 06:18:09 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FxObiexN; spf=pass (imf06.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723529796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F+g+z1RGD7NU2gbGbBKpzfSa2hMVvKkEjAdSNcJPLjs=; b=GFjckAZVrBB7Hw6yej98ZCzGiY3CKC2XtH38z3wSvlGcodPOHQidGyXI6D0nk5s8fo8hrn Cvg+qDHLxJKjo02cEzLlSv0nfjR4fB+9XeC5vMzKrDwqR7zASlLiZwBZeBj/eTFCvMMRkp soS+Z8ZNF363idbbmMQtAuWgs9qWjVU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FxObiexN; spf=pass (imf06.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723529796; a=rsa-sha256; cv=none; b=zzMocQo0KystRdWA8vQWzSwTL/eCV/W8pAj+36XoCHcU7bej2ac+pw301vZGFMsO63YPfO cYbAne1t6QRuOWcT/2FEsjC6yaWrtLHmSj5SempJU49gTotsTkmpxBgIbYsM7yZGQ6k4zk dTm0lh1hHAeoktY7F0MowM6E8TO55wA= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5b5b67d0024so5812360a12.0 for ; Mon, 12 Aug 2024 23:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723529888; x=1724134688; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=F+g+z1RGD7NU2gbGbBKpzfSa2hMVvKkEjAdSNcJPLjs=; b=FxObiexN4COEvjoSjHO03dNPtBXPSIK9bVM7rce/UMv2LW2tmq7i7ttblfDH25Kctw 3aLaoU6q7jxBkR2uZZLZav/u3VWMiyTPliF6meO1kv3ua/vEYNhit5dn9P/zfAoyp8RH SlE4hTtspNcqwvga6k6NsAQVHlkg24xWUgwwXUM3lfDBRqwV5T8GYiMKckiXCW41XOUp 7yxYVM9n5Uis9sr3YVF64jjUD8DofTkrCe9nmUXMSHyl9GyeRnQUe8WDYIqwAAcRK8Sz rW575wStdMtd6H56Vc0Pk0SoSVZhgOHW1xSFjTTGgNNMQCUZOscfOX9BTdIlLbPuxM/4 lNhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723529888; x=1724134688; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F+g+z1RGD7NU2gbGbBKpzfSa2hMVvKkEjAdSNcJPLjs=; b=mGh0FzEHJJnWX1YpJtKlFdO4Zuar+gKqt6m5BR5sYsiq4SmjNn9tXXC2wbKVqTRUzR t62l/4kEmU733EdUsGHzhzsmtvkyZoRVtpFiBlsyLycm2HTWoxuD8mjMvRIdpsRTflu7 koimPZ1nFCEimUsBjNlMb7ZOOAXixv5qqomprOEmtNyOuRzKSGjk3sGniKHS6ft0nGnZ 6HyaAs8mFWPiuWcJbmL0wFc3XoWKy5ocQtGwufMg1leftdPS72TJOWVq5a1HW+Ltoem5 6axGAHys1b2P8DYCdVuq+l2ALekZH5BO3R1RJ5QEyLkThgCqiGNDzdH5hVLSu4QzEFLz KMCQ== X-Forwarded-Encrypted: i=1; AJvYcCVJ9GtYlbtP7QP0J6066wPA+rqOBkmDFQ/MwPlgAhR1Ppurwh3/+2CDa3XCdvcy9A+LEyETDS1gWBMSniZVQiDrHCA= X-Gm-Message-State: AOJu0YxNfJ85Yuk018w0EZqjN9EEZtDxlRj2BFj/ngvT2haVkFG3qYo5 0pqFWBoCXUWptSg5TA2RG7/PM5SwQMDQ8uAvGH6tdaaHVmraWpJG3/05GTi0ak0Z5a1O3c0+Pmf gppgtsPdcNYGFKhtqETohGjD1ccs= X-Google-Smtp-Source: AGHT+IGrfFiMMvx5Z+gBLeKE4Pf/N/gMzc/KSiWz2TjQS59VsD0MNowfq5IKWZeZ3owo5g4ohGYRpmnU6PfQIGhe4UI= X-Received: by 2002:a05:6402:3489:b0:5a7:464a:ac0 with SMTP id 4fb4d7f45d1cf-5bd44c30b9dmr1589244a12.11.1723529887659; Mon, 12 Aug 2024 23:18:07 -0700 (PDT) MIME-Version: 1.0 References: <20240812150049.8252-1-ioworker0@gmail.com> <9ede36af-fca4-ed41-6b7e-cef157c640bb@huaweicloud.com> In-Reply-To: From: Lance Yang Date: Tue, 13 Aug 2024 14:17:30 +0800 Message-ID: Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops To: Yu Kuai Cc: =?UTF-8?Q?Michal_Koutn=C3=BD?= , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, cgroups@vger.kernel.org, josef@toxicpanda.com, tj@kernel.org, fujita.tomonori@lab.ntt.co.jp, boqun.feng@gmail.com, a.hindborg@samsung.com, paolo.valente@unimore.it, axboe@kernel.dk, vbabka@kernel.org, david@redhat.com, 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, libang.li@antgroup.com, "yukuai (C)" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BB908180017 X-Stat-Signature: ugm7zoe7sc3w6nhgbzora5qsoo57gk9b X-Rspam-User: X-HE-Tag: 1723529889-243791 X-HE-Meta: U2FsdGVkX1/lieJ99SGzVEZAhFDjyPTnq4Av8crSvb/7JxniNAI/Rj0JK1IePpnfK27d2uZ4IN0Yuj4TkjBZX8rAtTbzLKNlFz4LYZ1tShkHPIFQAqMhfQ+GACfpbnPnA6jMCypu/5N6NHI0QjlOPhSu4OVjuRKZKEjwV0cp1eBSN6HC3i/TVpkIxR6ecM06+UI6SHRJYFhnAPQKYvUO32A5ObiUev+QDNh9F2hrtPyZUJZdFhWL/wMjBfgHdtI4baokQHxqUERazLDY55s6TKNUdl6/4KyXBqRND+Tnvy1B3EUTk8k9DfTzdyHN/x13cgBlA60CVDk/EACeK+hJxtCWFpBDl+Oj1E3oANzyoU0ftcoWDThnyq5iQwTjrYCqulEzNRw7AkXyn0vY8UUqKEFApMwKEMOmepAu/PORi25UkwbiC4XI2cTrkajegRjwN0JltNFgqhqVwePeJoBHjQMDDd0MifQMyEwDpG1IqmJ7FMuta9XJ2hogiBh4oX6CluxlbAX+OD6PnmcNbzUElW2FhX2vHramIV1X5raDtQVQZinbCD3AC+q1sGOiR9anNvqGEHTCPMDu6wY0cyH1VHdvl0tT+fKgvLKRiDV4JnjTCCWjuHPjIdY/M3KTO11NWR80VAZI6F3DJSV2hIqZ7VOpzuqGRhor07Rti4qt2CySE1RFlJGFrd9DxaUyoK/l7mzWBOC0cjNE7WdUqU5prRHHoqEMfosOTa+hLDtfyVeI8hpK0FOZwcI/Pm0z/2nyOzJieNL3UdbtQRrvQNW2iWvqWLR3iJstzaLVNjwJfbrvTLNQHxQOQ5b6X2wuf4ZuVIJP9IUDk4Qu+3k4rJlhNwM5/x7tawSkZlzx8/6jUMyvKgX27epiSsm/gFZb2fA8Ckg/wpAgcV4o9xD6UaoBt5tpbo5o9BLXP7UabEtUWMiJVjdW5JUWYQZ9Qq85ZYsyU6QvjM/jGKmtzhb0oQw a98aLmDh kFL8UCY4zs5v1wfGyTshGz3aAG7uSaJYNUMNCHra32YW1q343CesfVV37RkEYhAzQ58zMVsReMCpL04aALTUA4T625t+YCaueqpLdFZDK2k2iYc/kjPPHh7k90pS8l9Q4ydaC2OmbqleN+QdXKRW4RJfw1hhHLO8xBS6saJUwuZRZvqNC5a+AodpMbPXPUEQy4Z54xUDG5w57Qk/30Xp+FIiBpj45uM0swfFkdpMBWNS62qo+6+875Zd86UOeH6Ba3FJ5ocHbYPF7dYy+dnF+JzSMYFZ8cZpur9gCtlU3o3FGr4ATAC1U1yF5RTsSzJ6KoproPSn2hWR7Ahqx3/Mw1Ii098YgJpKAqNb/xp5tx1Gff6VNuyk6izv4AYLekrrUXzAIF0WtbPs1fUekuf0gFioOBoopbvWW8PftSdFYVSh9on5OSwSRDmPN2lErD7Y5U3V3V8qjxYELXiXb3m36ERFv7sZxFYeBqlvx/Ww6y1/NpCA9A5UXGThm2BfzzAKUl81qVLMQM4V02b7Y4bu7/OxY0Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000368, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: I just realized that bursts appear to depend on the maximum values of wbps = and wiops, when wiops is not `max`? Thanks, Lance On Tue, Aug 13, 2024 at 1:00=E2=80=AFPM Lance Yang wr= ote: > > Hi Kuai, > > Thanks a lot for jumping in! > > On Tue, Aug 13, 2024 at 9:37=E2=80=AFAM Yu Kuai = wrote: > > > > Hi, > > > > =E5=9C=A8 2024/08/12 23:43, Michal Koutn=C3=BD =E5=86=99=E9=81=93: > > > +Cc Kuai > > > > > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang wrote: > > >> Hi all, > > >> > > >> I've run into a problem with Cgroup v2 where it doesn't seem to corr= ectly limit > > >> I/O operations when I set both wbps and wiops for a device. However,= if I only > > >> set wbps, then everything works as expected. > > >> > > >> To reproduce the problem, we can follow these command-based steps: > > >> > > >> 1. **System Information:** > > >> - Kernel Version and OS Release: > > >> ``` > > >> $ uname -r > > >> 6.10.0-rc5+ > > >> > > >> $ cat /etc/os-release > > >> PRETTY_NAME=3D"Ubuntu 24.04 LTS" > > >> NAME=3D"Ubuntu" > > >> VERSION_ID=3D"24.04" > > >> VERSION=3D"24.04 LTS (Noble Numbat)" > > >> VERSION_CODENAME=3Dnoble > > >> ID=3Dubuntu > > >> ID_LIKE=3Ddebian > > >> HOME_URL=3D"https://www.ubuntu.com/" > > >> SUPPORT_URL=3D"https://help.ubuntu.com/" > > >> BUG_REPORT_URL=3D"https://bugs.launchpad.net/ubuntu/" > > >> PRIVACY_POLICY_URL=3D"https://www.ubuntu.com/legal/terms-and-p= olicies/privacy-policy" > > >> UBUNTU_CODENAME=3Dnoble > > >> LOGO=3Dubuntu-logo > > >> ``` > > >> > > >> 2. **Device Information and Settings:** > > >> - List Block Devices and Scheduler: > > >> ``` > > >> $ lsblk > > >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > > >> sda 8:0 0 4.4T 0 disk > > >> =E2=94=94=E2=94=80sda1 8:1 0 4.4T 0 part /data > > >> ... > > >> > > >> $ cat /sys/block/sda/queue/scheduler > > >> none [mq-deadline] kyber bfq > > >> > > >> $ cat /sys/block/sda/queue/rotational > > >> 1 > > >> ``` > > >> > > >> 3. **Reproducing the problem:** > > >> - Navigate to the cgroup v2 filesystem and configure I/O setting= s: > > >> ``` > > >> $ cd /sys/fs/cgroup/ > > >> $ stat -fc %T /sys/fs/cgroup > > >> cgroup2fs > > >> $ mkdir test > > >> $ echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max > > >> ``` > > >> In this setup: > > >> wbps=3D10485760 sets the write bytes per second limit to 10 MB= /s. > > >> wiops=3D100000 sets the write I/O operations per second limit = to 100,000. > > >> > > >> - Add process to the cgroup and verify: > > >> ``` > > >> $ echo $$ > cgroup.procs > > >> $ cat cgroup.procs > > >> 3826771 > > >> 3828513 > > >> $ ps -ef|grep 3826771 > > >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > > >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > > >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --col= or=3Dauto 3826771 > > >> ``` > > >> > > >> - Observe I/O performance using `dd` commands and `iostat`: > > >> ``` > > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > > > You're testing buffer IO here, and I don't see that write back cgroup i= s > > enabled. Is this test intentional? Why not test direct IO? > > Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEB= ACK > was enabled. > > $ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK > CONFIG_CGROUP_WRITEBACK=3Dy > > We intend to configure both wbps (write bytes per second) and wiops > (write I/O operations > per second) for the containers. IIUC, this setup will effectively > restrict both their block device > I/Os and buffered I/Os. > > > Why not test direct IO? > > I was testing direct IO as well. However it did not work as expected with > `echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max`. > > $ time dd if=3D/dev/zero of=3D/data/file7 bs=3D512M count=3D1 oflag=3Ddir= ect > 1+0 records in > 1+0 records out > 536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s > > real 0m51.637s > user 0m0.000s > sys 0m0.313s > > $ iostat -d 1 -h -y -p sda > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 55.00 0.0k 1.8M 0.0k 0.0k 1.8M > 0.0k sda > 55.00 0.0k 1.8M 0.0k 0.0k 1.8M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 18.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 18.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 12.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M > 0.0k sda > 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 4.00 0.0k 24.0k 0.0k 0.0k 24.0k > 0.0k sda > 4.00 0.0k 24.0k 0.0k 0.0k 24.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > There are two things that confuse me. First, initially, neither the > wbps nor the wiops > reached their limits. Second, in the last second, the wbps far > exceeded the limit. > > But if I only set wbps, then everything works as expected with > `echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max` > > > >> ``` > > >> ``` > > >> $ iostat -d 1 -h -y -p sda > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_= wrtn kB_dscd Device > > >> 7.00 0.0k 1.3M 0.0k 0.0k 1= .3M 0.0k sda > > >> 7.00 0.0k 1.3M 0.0k 0.0k 1= .3M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 21.00 0.0k 1.4M 0.0k 0.0k 1= .4M 0.0k sda > > >> 21.00 0.0k 1.4M 0.0k 0.0k 1= .4M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1= .2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448= .1M 0.0k sda > > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448= .1M 0.0k sda1 > > > > Looks like all dirty buffer got flushed to disk at the last second whil= e > > the file is closed, this is expected. > > The dd command completed in less than a second, but flushing all the > dirty buffers to > disk took a much longer time. By the time the flushing was completed, > the file had > already been closed, IIUC. > > $ time dd if=3D/dev/zero of=3D/data/file5 bs=3D512M count=3D1 > 1+0 records in > 1+0 records out > 536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s > > real 0m0.578s > user 0m0.000s > sys 0m0.576s > > $ iostat -d 1 -h -y -p sda > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 74.00 0.0k 664.0k 0.0k 0.0k 664.0k > 0.0k sda > 74.00 0.0k 664.0k 0.0k 0.0k 664.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.1M 0.0k 0.0k 1.1M > 0.0k sda > 15.00 0.0k 1.1M 0.0k 0.0k 1.1M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 13.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 46.00 0.0k 1.7M 0.0k 0.0k 1.7M > 0.0k sda > 46.00 0.0k 1.7M 0.0k 0.0k 1.7M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 49.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda > 49.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 53.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda > 53.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M > 0.0k sda > 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > > >> ``` > > >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to = several > > >> hundreds of MB/s. > > > > > > What it would be on average? > > > IOW how long would the whole operation in throttled cgroup take? > > > > > >> > > >> - Testing with wiops set to max: > > >> ``` > > >> echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max > > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > >> $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > >> ``` > > >> ``` > > >> $ iostat -d 1 -h -y -p sda > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 48.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda > > >> 48.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 40.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda > > >> 40.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 41.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda > > >> 41.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 46.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda > > >> 46.00 0.0k 10.0M 0.0k 0.0k 10= .0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_w= rtn kB_dscd Device > > >> 55.00 0.0k 10.2M 0.0k 0.0k 10= .2M 0.0k sda > > >> 55.00 0.0k 10.2M 0.0k 0.0k 10= .2M 0.0k sda1 > > > > And I don't this wiops=3Dmax is the reason, what need to explain is tha= t > > why dirty buffer got flushed to disk synchronously before the dd finish > > and close the file? > > The dd command operates in the background, and it seems that the dirty > buffers begin > to flush after the command has completed. > > > > > >> ``` > > >> The iostat output shows the write operations as stabilizing at aroun= d 10 MB/s, > > >> which aligns with the defined limit of 10 MB/s. After setting wiops = to max, the > > >> I/O limits appear to work as expected. > > > > Can you give the direct IO a test? And also enable write back cgroup fo= r > > buffer IO. > > > > Thanks, > > Kuai > > > Thanks a lot again for your time! > Lance > > > > > >> > > >> > > >> Thanks, > > >> Lance > > > > > > Thanks for the report Lance. Is this something you started seeing aft= er > > > a kernel update or switch to cgroup v2? (Or you simply noticed with t= his > > > setup only?) > > > > > > > > > Michal > > > > >