From: Lance Yang <ioworker0@gmail.com>
To: Yu Kuai <yukuai1@huaweicloud.com>
Cc: "Michal Koutný" <mkoutny@suse.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
linux-block@vger.kernel.org, cgroups@vger.kernel.org,
josef@toxicpanda.com, tj@kernel.org,
fujita.tomonori@lab.ntt.co.jp, boqun.feng@gmail.com,
a.hindborg@samsung.com, paolo.valente@unimore.it,
axboe@kernel.dk, vbabka@kernel.org, david@redhat.com,
21cnbao@gmail.com, baolin.wang@linux.alibaba.com,
libang.li@antgroup.com, "yukuai (C)" <yukuai3@huawei.com>
Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops
Date: Tue, 13 Aug 2024 14:17:30 +0800 [thread overview]
Message-ID: <CAK1f24=R9tgxGB3T3JPsHAhg+hAgvRg+-K-gWT_+WYVOTRtSpA@mail.gmail.com> (raw)
In-Reply-To: <CAK1f24mwzXa8Az5WFYu+1UopTCStDWx3yDr1RugLwphS-hWizw@mail.gmail.com>
I just realized that bursts appear to depend on the maximum values of wbps and
wiops, when wiops is not `max`?
Thanks,
Lance
On Tue, Aug 13, 2024 at 1:00 PM Lance Yang <ioworker0@gmail.com> wrote:
>
> Hi Kuai,
>
> Thanks a lot for jumping in!
>
> On Tue, Aug 13, 2024 at 9:37 AM Yu Kuai <yukuai1@huaweicloud.com> wrote:
> >
> > Hi,
> >
> > 在 2024/08/12 23:43, Michal Koutný 写道:
> > > +Cc Kuai
> > >
> > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@gmail.com> wrote:
> > >> Hi all,
> > >>
> > >> I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit
> > >> I/O operations when I set both wbps and wiops for a device. However, if I only
> > >> set wbps, then everything works as expected.
> > >>
> > >> To reproduce the problem, we can follow these command-based steps:
> > >>
> > >> 1. **System Information:**
> > >> - Kernel Version and OS Release:
> > >> ```
> > >> $ uname -r
> > >> 6.10.0-rc5+
> > >>
> > >> $ cat /etc/os-release
> > >> PRETTY_NAME="Ubuntu 24.04 LTS"
> > >> NAME="Ubuntu"
> > >> VERSION_ID="24.04"
> > >> VERSION="24.04 LTS (Noble Numbat)"
> > >> VERSION_CODENAME=noble
> > >> ID=ubuntu
> > >> ID_LIKE=debian
> > >> HOME_URL="https://www.ubuntu.com/"
> > >> SUPPORT_URL="https://help.ubuntu.com/"
> > >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> > >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> > >> UBUNTU_CODENAME=noble
> > >> LOGO=ubuntu-logo
> > >> ```
> > >>
> > >> 2. **Device Information and Settings:**
> > >> - List Block Devices and Scheduler:
> > >> ```
> > >> $ lsblk
> > >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> > >> sda 8:0 0 4.4T 0 disk
> > >> └─sda1 8:1 0 4.4T 0 part /data
> > >> ...
> > >>
> > >> $ cat /sys/block/sda/queue/scheduler
> > >> none [mq-deadline] kyber bfq
> > >>
> > >> $ cat /sys/block/sda/queue/rotational
> > >> 1
> > >> ```
> > >>
> > >> 3. **Reproducing the problem:**
> > >> - Navigate to the cgroup v2 filesystem and configure I/O settings:
> > >> ```
> > >> $ cd /sys/fs/cgroup/
> > >> $ stat -fc %T /sys/fs/cgroup
> > >> cgroup2fs
> > >> $ mkdir test
> > >> $ echo "8:0 wbps=10485760 wiops=100000" > io.max
> > >> ```
> > >> In this setup:
> > >> wbps=10485760 sets the write bytes per second limit to 10 MB/s.
> > >> wiops=100000 sets the write I/O operations per second limit to 100,000.
> > >>
> > >> - Add process to the cgroup and verify:
> > >> ```
> > >> $ echo $$ > cgroup.procs
> > >> $ cat cgroup.procs
> > >> 3826771
> > >> 3828513
> > >> $ ps -ef|grep 3826771
> > >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash
> > >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef
> > >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771
> > >> ```
> > >>
> > >> - Observe I/O performance using `dd` commands and `iostat`:
> > >> ```
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> >
> > You're testing buffer IO here, and I don't see that write back cgroup is
> > enabled. Is this test intentional? Why not test direct IO?
>
> Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEBACK
> was enabled.
>
> $ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK
> CONFIG_CGROUP_WRITEBACK=y
>
> We intend to configure both wbps (write bytes per second) and wiops
> (write I/O operations
> per second) for the containers. IIUC, this setup will effectively
> restrict both their block device
> I/Os and buffered I/Os.
>
> > Why not test direct IO?
>
> I was testing direct IO as well. However it did not work as expected with
> `echo "8:0 wbps=10485760 wiops=100000" > io.max`.
>
> $ time dd if=/dev/zero of=/data/file7 bs=512M count=1 oflag=direct
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s
>
> real 0m51.637s
> user 0m0.000s
> sys 0m0.313s
>
> $ iostat -d 1 -h -y -p sda
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 55.00 0.0k 1.8M 0.0k 0.0k 1.8M
> 0.0k sda
> 55.00 0.0k 1.8M 0.0k 0.0k 1.8M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 18.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 18.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 12.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
> 0.0k sda
> 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 4.00 0.0k 24.0k 0.0k 0.0k 24.0k
> 0.0k sda
> 4.00 0.0k 24.0k 0.0k 0.0k 24.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
> There are two things that confuse me. First, initially, neither the
> wbps nor the wiops
> reached their limits. Second, in the last second, the wbps far
> exceeded the limit.
>
> But if I only set wbps, then everything works as expected with
> `echo "8:0 wbps=10485760 wiops=max" > io.max`
>
> > >> ```
> > >> ```
> > >> $ iostat -d 1 -h -y -p sda
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda
> > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda
> > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda
> > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1
> >
> > Looks like all dirty buffer got flushed to disk at the last second while
> > the file is closed, this is expected.
>
> The dd command completed in less than a second, but flushing all the
> dirty buffers to
> disk took a much longer time. By the time the flushing was completed,
> the file had
> already been closed, IIUC.
>
> $ time dd if=/dev/zero of=/data/file5 bs=512M count=1
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s
>
> real 0m0.578s
> user 0m0.000s
> sys 0m0.576s
>
> $ iostat -d 1 -h -y -p sda
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 74.00 0.0k 664.0k 0.0k 0.0k 664.0k
> 0.0k sda
> 74.00 0.0k 664.0k 0.0k 0.0k 664.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.1M 0.0k 0.0k 1.1M
> 0.0k sda
> 15.00 0.0k 1.1M 0.0k 0.0k 1.1M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 13.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 46.00 0.0k 1.7M 0.0k 0.0k 1.7M
> 0.0k sda
> 46.00 0.0k 1.7M 0.0k 0.0k 1.7M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 49.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda
> 49.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 53.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda
> 53.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
> 0.0k sda
> 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
> > >> ```
> > >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to several
> > >> hundreds of MB/s.
> > >
> > > What it would be on average?
> > > IOW how long would the whole operation in throttled cgroup take?
> > >
> > >>
> > >> - Testing with wiops set to max:
> > >> ```
> > >> echo "8:0 wbps=10485760 wiops=max" > io.max
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> ```
> > >> ```
> > >> $ iostat -d 1 -h -y -p sda
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda
> > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1
> >
> > And I don't this wiops=max is the reason, what need to explain is that
> > why dirty buffer got flushed to disk synchronously before the dd finish
> > and close the file?
>
> The dd command operates in the background, and it seems that the dirty
> buffers begin
> to flush after the command has completed.
>
> >
> > >> ```
> > >> The iostat output shows the write operations as stabilizing at around 10 MB/s,
> > >> which aligns with the defined limit of 10 MB/s. After setting wiops to max, the
> > >> I/O limits appear to work as expected.
> >
> > Can you give the direct IO a test? And also enable write back cgroup for
> > buffer IO.
> >
> > Thanks,
> > Kuai
>
>
> Thanks a lot again for your time!
> Lance
>
> >
> > >>
> > >>
> > >> Thanks,
> > >> Lance
> > >
> > > Thanks for the report Lance. Is this something you started seeing after
> > > a kernel update or switch to cgroup v2? (Or you simply noticed with this
> > > setup only?)
> > >
> > >
> > > Michal
> > >
> >
next prev parent reply other threads:[~2024-08-13 6:18 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-12 15:00 Lance Yang
2024-08-12 15:43 ` Michal Koutný
2024-08-13 1:37 ` Yu Kuai
2024-08-13 5:00 ` Lance Yang
2024-08-13 6:17 ` Lance Yang [this message]
2024-08-13 6:39 ` Yu Kuai
2024-08-13 7:19 ` Yu Kuai
2024-08-15 1:59 ` Lance Yang
2024-08-23 12:05 ` Lance Yang
2024-08-26 1:31 ` Yu Kuai
2024-08-26 2:15 ` Lance Yang
2024-08-23 12:19 ` Lance Yang
2024-08-13 5:11 ` Lance Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAK1f24=R9tgxGB3T3JPsHAhg+hAgvRg+-K-gWT_+WYVOTRtSpA@mail.gmail.com' \
--to=ioworker0@gmail.com \
--cc=21cnbao@gmail.com \
--cc=a.hindborg@samsung.com \
--cc=axboe@kernel.dk \
--cc=baolin.wang@linux.alibaba.com \
--cc=boqun.feng@gmail.com \
--cc=cgroups@vger.kernel.org \
--cc=david@redhat.com \
--cc=fujita.tomonori@lab.ntt.co.jp \
--cc=josef@toxicpanda.com \
--cc=libang.li@antgroup.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mkoutny@suse.com \
--cc=paolo.valente@unimore.it \
--cc=tj@kernel.org \
--cc=vbabka@kernel.org \
--cc=yukuai1@huaweicloud.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox