From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B617AC52D7C for ; Tue, 13 Aug 2024 05:12:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A3046B00A0; Tue, 13 Aug 2024 01:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2532E6B00A1; Tue, 13 Aug 2024 01:12:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CD476B00A2; Tue, 13 Aug 2024 01:12:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DBB1C6B00A0 for ; Tue, 13 Aug 2024 01:12:04 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 815B0A6FE5 for ; Tue, 13 Aug 2024 05:12:04 +0000 (UTC) X-FDA: 82446050568.04.B6253D7 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by imf15.hostedemail.com (Postfix) with ESMTP id B0051A0004 for ; Tue, 13 Aug 2024 05:12:02 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xr4cQRS7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723525911; a=rsa-sha256; cv=none; b=AHGLDE0XmCHQzdSysjULICdwiXQyqsaRjyQaAVguE2ntUIQHtguJd4eaGTT32Oek95sXrv WK2v0m1HtDoQJbyMHkj+hkGemCI3yR3sLDcwDiDrI6qoMbRSRclqlYt32eOKRthk6ZRH31 e0Hi9gQKyHGN8G3PpnUQucIJVIopb/c= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xr4cQRS7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723525911; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xhT2Y2oF/uJ6fCpK+SpF85ANnuI7c6Ysyk5Jyojf0og=; b=U1AtFyrwpSWGUCw2UST9Ya4KNY76bWXZ6p7SEnAYPrsiYfrG8S2VP15n3+TLM3g4u3a1dl xyzftWlko23KvM7IrHcnZwcfve8Wf77jBp3Dr6kpN0ehkT8idP6kfSke0zDrLnqy5YPPte lJBvoMNpTKLu3Yt7Z+w3/56AL1IlOgM= Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-5bb8e62570fso6071402a12.1 for ; Mon, 12 Aug 2024 22:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723525921; x=1724130721; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=xhT2Y2oF/uJ6fCpK+SpF85ANnuI7c6Ysyk5Jyojf0og=; b=Xr4cQRS7e1zJhw4Q5im6rsbSvogHfqnCn3Ov3s50v0Q0S9CZoEy8gLX5xxCtVl0tNy tI1zvQaGYNLNg/70H6uUT5UL43x1J3lnabyMRxmdK4uXaxiy+9FHQtLDw11XmosiD23j TLJI9cnWF6AYMxCd5JKu0LiqvyKl16eqcKRjOawVvYQCOnQH34ddYCxRJ+NSx455y5sq dyRLlKDY5YZXOxL0Ya/jtcDCu9wOhEz55th9NRfeJFrHIKAi+Bn2dh5EeJp5IE0jo+Ia UFgM1GqGC8dOTT9/+XBELYjd4Xcr9Fu9Kkmn1Wu8VzrAavFuM3fH+aCSHYYmtAIZVvpe hClw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723525921; x=1724130721; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xhT2Y2oF/uJ6fCpK+SpF85ANnuI7c6Ysyk5Jyojf0og=; b=m0RLshaEZmMai4E7AOseVADjrgbGexgR4M8t9gu+C2rTZE4bynAJtuBzj+g5ld+rL3 yckEK5UdNdkqn9D0iP0G4qPy8jZM7H9yW+z8j6jVHCZfTBA0dCqOwJ4kI7k/56dSzqr1 /bYBRwN3QGPB/bYx5sEAIcjal6VHGrsu0pjryRNnZcSwxggB6KR8rFqz92RY2E1B1PZ/ WpOwUHcOgURL/tW0TOEGqAb9YyHj/xyE541/UdnEy0IirxmGSrD4e3JHhTBea7vMCROE bm3EWYDwjO+/V0dxyZStqwSsEdUraQ7K71i6ETriaii+i2BugMR2iM+XsCfMrHM2/8UG 5Ozw== X-Gm-Message-State: AOJu0YwNvYV7fXDKilNkXjAFIZ1a9tExZMisijbBloGd91IuACRsnRW+ rz8IEjS2PXE62ZBR1YMGDjIGqHWrlCX1z7DQZWHg4RDyF5SkUm3/EHoQ3YzUURYi6eAWQQ54NLI Ee6ackDNziwBnIvNTinYZu0Befq8= X-Google-Smtp-Source: AGHT+IHY9qm5hOo0rihOZU0LIYFyLAAviDBaIbzofcC8Ywjoc5SlO7vwoHvwiGbR5dMn7tBFWDnGueFQeDifIUQjjIY= X-Received: by 2002:a05:6402:3511:b0:5a0:e62c:61bd with SMTP id 4fb4d7f45d1cf-5bd44c698b1mr1424586a12.29.1723525920619; Mon, 12 Aug 2024 22:12:00 -0700 (PDT) MIME-Version: 1.0 References: <20240812150049.8252-1-ioworker0@gmail.com> In-Reply-To: From: Lance Yang Date: Tue, 13 Aug 2024 13:11:24 +0800 Message-ID: Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops To: =?UTF-8?Q?Michal_Koutn=C3=BD?= Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, cgroups@vger.kernel.org, josef@toxicpanda.com, tj@kernel.org, fujita.tomonori@lab.ntt.co.jp, boqun.feng@gmail.com, a.hindborg@samsung.com, paolo.valente@unimore.it, axboe@kernel.dk, vbabka@kernel.org, david@redhat.com, 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, libang.li@antgroup.com, Yu Kuai Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: B0051A0004 X-Rspamd-Server: rspam01 X-Stat-Signature: yxaa8jyuf94y96xdsg136zdiiwkce64s X-HE-Tag: 1723525922-351761 X-HE-Meta: U2FsdGVkX1/Q9Glx1c92NOdevK6YcOVbFfWYd7gTvhTGnNM7ECxHxPvdRm4rsyp3hhmq5oUZ4gZD1FeQJ4I1AIiBrrkr3y01cPmQrYRW32tkItFCZy+KlndP+vgCLEnL9Ek0a0sONSAkDmC/ePa/5uOOwoi0fV0EOP8ZXYKKl1d008HTTQd4Zhpn652HfeL1aYn/xwPulBo2WdgaHekX1CYtedORoO1bYGRUINxD+gP8ihDC3x2WrUhGJZmqWd8M8aIOTtPmPSEHtKtymkwgI0lC5fsSgiM6IVUZ3wAxnZmogQfxImnM+pwPgZTj8vbt39KMtgStgd7sXSTq3orJ3vkJjS7w+7LSKEpSGf06hmlZVKdYglojxGZiyNHXWSuEc9eSWRqwOKBckaqbwq5J1h65TQrrOcTFGJQEOvLsvf1i5FJnzyK2FUC7RtVbU52vipJxSB21hnVTR3Y0bZ3IVN6CNXu8xvz8WuS03OuHpB3XZ94MGbgS48gJE7P9LvNq5YuOnTbI8gPSMZHCpKSSFGiqkKbH0b3ElL77/sw0+eSl3ojUf3UqR+TjNneYvgNDnKOgQWBDU9NMhOToRuB1F3k0xox7EH+tKu1HEWPa0Mh1e/txRgDm+LrKTCZLm4tSlWmfBxoLLKhyWTO63P5aPGPep30McF2rwWSGsy7gvmrsQsPVOug0OIsCIq+yP8VVNuEtROlXqLtHmhdJSL8ExfFhY9eSTG9rk+fFBn2wvGNYkf2xeB1FV4Qzc+jJbbJ7dqi5vGxp2IhrDh8VcCv3JRntG+/Ccv4SzUe1xkPA/6chX67kxzrhvv30AnPJbJu6+8gDWE22/U/L54v75Q1C4+aj6niu2CESDPymVvfKkESkGTQNsm+fz4Q47wbLe72vhYNDZyHkUSvrHA1HHusXg7u/LvOngjt9nKJy/GJcmgyxwDkmybfCZ7ed1mPP4V6NQCpsy8LQ0nyhFMxPvQe HtBL4Mdj yehQyOeGkyt8QdtaoDiZcYca4Hd2PAwZFxB/ARdNxUe75hIh9Ff6nsKEeFF8e90C2VbFaRg8ZqFKPEhWAXx370/n/Ds8Fs48f5yB0tI0SlkzIXpY4xlHYH1egrwtHU6Xeb3xSTKU5BVLPIEDXES8/RclT4qToKuC6+vollKcQtPCMErKxP3czd8tqj+IkRl0O2SZc0tcya+RSYz7fMbKivq+Kt6OtAUu8xVg1w9XdYfzCpc8QtlcSUEs9nVPi+bfKgCndn4VtZmjWlhEyRXr0ZJ6clnl82pPjL5CCth7pUR7/4Yzw/ejZLpKsIZ2mOyN5dzztT/Nh9h8w9cdkKntieDb1XTBhk0CgIjhuY2rrC+bY/4vPEhPwwYPihpARjQ+esu/+02XuprkHDc26Xv7jv9V7g+tqd/XVKvFPFzr37Ojm6fQUalxBb+6Yo1vtrADIQNydSDF0Lho5VBKLAUfcixZDrVMjL0NgzdgcnkXybGjFLwdY8M/AElIM5l7SLH3dv815sW+tNDzvrrnlZlk68LsWGw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.017018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Michal, Thanks a lot for jumping in! On Mon, Aug 12, 2024 at 11:43=E2=80=AFPM Michal Koutn=C3=BD wrote: > > +Cc Kuai > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang = wrote: > > Hi all, > > > > I've run into a problem with Cgroup v2 where it doesn't seem to correct= ly limit > > I/O operations when I set both wbps and wiops for a device. However, if= I only > > set wbps, then everything works as expected. > > > > To reproduce the problem, we can follow these command-based steps: > > > > 1. **System Information:** > > - Kernel Version and OS Release: > > ``` > > $ uname -r > > 6.10.0-rc5+ > > > > $ cat /etc/os-release > > PRETTY_NAME=3D"Ubuntu 24.04 LTS" > > NAME=3D"Ubuntu" > > VERSION_ID=3D"24.04" > > VERSION=3D"24.04 LTS (Noble Numbat)" > > VERSION_CODENAME=3Dnoble > > ID=3Dubuntu > > ID_LIKE=3Ddebian > > HOME_URL=3D"https://www.ubuntu.com/" > > SUPPORT_URL=3D"https://help.ubuntu.com/" > > BUG_REPORT_URL=3D"https://bugs.launchpad.net/ubuntu/" > > PRIVACY_POLICY_URL=3D"https://www.ubuntu.com/legal/terms-and-polic= ies/privacy-policy" > > UBUNTU_CODENAME=3Dnoble > > LOGO=3Dubuntu-logo > > ``` > > > > 2. **Device Information and Settings:** > > - List Block Devices and Scheduler: > > ``` > > $ lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > > sda 8:0 0 4.4T 0 disk > > =E2=94=94=E2=94=80sda1 8:1 0 4.4T 0 part /data > > ... > > > > $ cat /sys/block/sda/queue/scheduler > > none [mq-deadline] kyber bfq > > > > $ cat /sys/block/sda/queue/rotational > > 1 > > ``` > > > > 3. **Reproducing the problem:** > > - Navigate to the cgroup v2 filesystem and configure I/O settings: > > ``` > > $ cd /sys/fs/cgroup/ > > $ stat -fc %T /sys/fs/cgroup > > cgroup2fs > > $ mkdir test > > $ echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max > > ``` > > In this setup: > > wbps=3D10485760 sets the write bytes per second limit to 10 MB/s. > > wiops=3D100000 sets the write I/O operations per second limit to 1= 00,000. > > > > - Add process to the cgroup and verify: > > ``` > > $ echo $$ > cgroup.procs > > $ cat cgroup.procs > > 3826771 > > 3828513 > > $ ps -ef|grep 3826771 > > root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > > root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > > root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color= =3Dauto 3826771 > > ``` > > > > - Observe I/O performance using `dd` commands and `iostat`: > > ``` > > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > ``` > > ``` > > $ iostat -d 1 -h -y -p sda > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wr= tn kB_dscd Device > > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M = 0.0k sda > > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M = 0.0k sda > > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M = 0.0k sda > > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M = 0.0k sda1 > > ``` > > Initially, the write speed is slow (<2MB/s) then suddenly bursts to sev= eral > > hundreds of MB/s. > > What it would be on average? > IOW how long would the whole operation in throttled cgroup take? > > > > > - Testing with wiops set to max: > > ``` > > echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max > > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > > ``` > > ``` > > $ iostat -d 1 -h -y -p sda > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 > > > > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M = 0.0k sda > > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M = 0.0k sda1 > > ``` > > The iostat output shows the write operations as stabilizing at around 1= 0 MB/s, > > which aligns with the defined limit of 10 MB/s. After setting wiops to = max, the > > I/O limits appear to work as expected. > > > > > > Thanks, > > Lance > > Thanks for the report Lance. Is this something you started seeing after > a kernel update or switch to cgroup v2? (Or you simply noticed with this > setup only?) I just switched to cgroup v2 to begin testing, as we intend to have containers run in cgroup v2. Testing on both the 5.14.0 and mainline versions ;) Thanks again for your time! Lance > > > Michal