From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0512DC3DA7F for ; Mon, 12 Aug 2024 15:43:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B3C06B008A; Mon, 12 Aug 2024 11:43:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 164746B008C; Mon, 12 Aug 2024 11:43:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 004D56B0095; Mon, 12 Aug 2024 11:43:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D5A396B008A for ; Mon, 12 Aug 2024 11:43:10 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 541DDA4D25 for ; Mon, 12 Aug 2024 15:43:10 +0000 (UTC) X-FDA: 82444012140.13.E730312 Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by imf28.hostedemail.com (Postfix) with ESMTP id 54CF3C000A for ; Mon, 12 Aug 2024 15:43:08 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=fP8fZ9dU; spf=pass (imf28.hostedemail.com: domain of mkoutny@suse.com designates 209.85.208.43 as permitted sender) smtp.mailfrom=mkoutny@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723477320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Kd2QxG6rPBQQCGR+f4zRNeOOb8mEAgkCnPSorAh8lE=; b=qXSKKbOfhtFNWrCT7fbW8ZYI+xz3eUb2r8HviNVmW4LttlYCtTJOahzVi+63B8GACBc2Qy mmSdpOHOynKVK32ReIKgkKMjBhbVUH38aHMJ/J81d5iPQ+pbcCRQpCwHfqvqQ6mAUvqXO/ T8tSW0uHtr9iK1AoqivQD7m+KIu84nk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723477320; a=rsa-sha256; cv=none; b=Udr1lDSi6RHDWoC6h18ZlWv/MOERdR74PxvI6Rnr0i+hCA6ZhuVuWsn4PR6nvNm/02PURJ EmF3DowRjlQQqhharIzqRUSYVrjp0kcm34Cx/q0W/CztyskITXMsToG5ilfVvFcouLvrqL N8UM8pQx0HqOdNE8q8I4YCnScFmuYDM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=fP8fZ9dU; spf=pass (imf28.hostedemail.com: domain of mkoutny@suse.com designates 209.85.208.43 as permitted sender) smtp.mailfrom=mkoutny@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-5ba43b433beso4702335a12.1 for ; Mon, 12 Aug 2024 08:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1723477386; x=1724082186; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=+Kd2QxG6rPBQQCGR+f4zRNeOOb8mEAgkCnPSorAh8lE=; b=fP8fZ9dUONVbfu6Ioc92eOfjxzmS77jHX1MHioLeZ+YBtViolQRkx44kiwEe6rZxsA 7FBQLGP7D1hlQLolDAyUA55jtqObAWWhw3tWAmaRXK/97YJyyS5hTtGuTaLTX/gNeVwu p5C+DgrNylj0sZIYR6dJLktLJzHU+YKkbuVgGQbIokau278OLo69/0iBIBHw3TuZDVRr 1VpmnIFCWwrA1K2FqMxKOG4RRqLLKgBUyN3CSqtrZ6vy0af+s+nM2MwhW2b7S9OC58cg WErkSVEYXkFoP22cXLznDgq+wCS9IpXFCqfQb+A0furnc5wSo8CWJakTQpWIPgpL9doW WKKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723477386; x=1724082186; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+Kd2QxG6rPBQQCGR+f4zRNeOOb8mEAgkCnPSorAh8lE=; b=RcoadbZgZL7GLoLz2tN08Sps2H4sJkxwR/SHOnEfPn4RNz23OHXblcQk5peuND5aSu 4xSibEh/atVANN5FlLfa4kK87K6n+Iazi7uf1hMD6ErQDqWpjO+Ukc96vjKHpy4PcDvX axmZcolrZVhzjvahkN/ymq0+VnczVuH0d74ehddq/7OkJhq6ulFOBWE61GWP+VCc2TYC JKjr+6dDSZZWIn402bGAYJ+6ioxB9ssSS/XShxc1FIzuWyajAlcdqkhYrx5opyzWVoR/ bc0RHe9UTdNYqumjAtPkMJ20IcwgIltCahNf6Y/odbRycHPtRLNLy0lNZKHMJZsVv2D0 HD8A== X-Gm-Message-State: AOJu0YxixGYQiY7fnULkFz95p+GrBOOiu+sVhDnh2vIaBpeIdg/l0PaT PbilxSUs3MfxWAIcq98dDkPd1mFFp1Ba8GFkjrsES41UsJ2pSoNOr9TJgqHuwYo= X-Google-Smtp-Source: AGHT+IHZmIqej7ltDGnyTqGaSC79yZq9Lw/8TPyw7oq1fSnTNGiUK0CDNwWrb22vPVedQEiictjEFg== X-Received: by 2002:a50:8dc7:0:b0:595:71c7:39dd with SMTP id 4fb4d7f45d1cf-5bd44c79eafmr415317a12.34.1723477386380; Mon, 12 Aug 2024 08:43:06 -0700 (PDT) Received: from blackdock.suse.cz ([193.86.92.181]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5bd196a756esm2190058a12.52.2024.08.12.08.43.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 08:43:05 -0700 (PDT) Date: Mon, 12 Aug 2024 17:43:04 +0200 From: Michal =?utf-8?Q?Koutn=C3=BD?= To: Lance Yang Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, cgroups@vger.kernel.org, josef@toxicpanda.com, tj@kernel.org, fujita.tomonori@lab.ntt.co.jp, boqun.feng@gmail.com, a.hindborg@samsung.com, paolo.valente@unimore.it, axboe@kernel.dk, vbabka@kernel.org, david@redhat.com, 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, libang.li@antgroup.com, Yu Kuai Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops Message-ID: References: <20240812150049.8252-1-ioworker0@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="mkaklzxm72eg2bej" Content-Disposition: inline In-Reply-To: <20240812150049.8252-1-ioworker0@gmail.com> X-Rspamd-Queue-Id: 54CF3C000A X-Stat-Signature: 5g7yj5jooccdzog3udoguh9erscqw5qd X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723477388-269865 X-HE-Meta: U2FsdGVkX1+j3gpRIr/fYmKGswoWTxHJNXOAYWScuCD0hYY0uHZxMdFkU3c1LZHx8w7MS3mV1Fgu93c3p/BRO9BgLNCx3DLgjQEJ6e4ObMSZw7h5V8COpuJzKY3sQekQLC4m7K+gbdmc3wBjFAQm8a4Gcq/RDMAXAwv8XpJOAmdPciRf6hhi8IPxrwnWOgCb+iIyXejPUY0h37DL5buUxpegMoUSHU3tPuzVpMxK5/uNAILunYcsvTcBp9c9Hxc+CiAqkeSSKJTZsiP4TqP+MgArSq1YPykZ6ISFq2gjr68ngImH3aNuX7j0493eyBKfOsNGjYTFT1Vsl0eqlmi/9uvW9BXyRvhP8GKpko5OWILy3zqhQpQ/FuS4AaFwVLwsVxUUoT1tnl8pRXOhJUY4QdKNUakzq/5jWWhZbDWU9VI6Zuee6hoWfowixZEVMWjMWJK8uxRkoGGU35TuZq5hQhXd8foR5sPeo5LEKwa5vR6vUHuhFZ+F3olxJrqiVaNq0GLlVITgPSc4CO32E8PFj1cwtF2witNZIs2fp7S0qtPNIz815q9cTVMM2D2cIX9ZQaRC99FzNWvlrwb9ujtbF3Lqd1tsp0vVd00g3aVmdJk4hfNO4sThGxZLk9HbUJLI3XZSZhhLIuCwQh4mBi3GDsQhGjw/FvlO6ghecHJZp2n8LiHZ3Vy+mlc/fq4ami1PaFGT2TxeOAjBttLfZ0Mh1GquMVqclGHD4ERust3YU1F+d0t0eJIxiWA4DiYPDFP4cCvqFpj7rCLSzhBpL2CD7jNjTru8iCCGJcx+ml+9Ejq0k6uJoJJugLUmE2zZpFiBf2e1VN8UBy+RMcLoGjcA0V+kHoq8UG8yHge2HJon45KyRlq/khgFqJE5e+BYZC3ta+CYL+TuIvMI5H9juntrlH++doFP17IZhP8VC61rr+UDdxr77Gg99qqch2i6BtK+vzoWjETSd4PnvD49RvZ dpnkRsHK /WygD9vcbKdpt7S3QM3xcmgqpK2XEtSmRG1Uz7kmxkmpep1y33TDQYISfp1QKWFT/kMSVh1etXP03MPcNvkMFXpBCj4ODi4+CaHJgkC1FmHDXTr98b08KV25XNJfsipJIlhemOJZFZHt7qSx+ntzFXsqx7CSsjZVZF4iB6UnV4mp2DFt88ejchsqREWK/GRkI3Awr7p9z4406HVSfyDNZZKbhb9yTN902AGsxvPBc9Fp8kb1slSvG9vytFRpfF2B+b/EtsP3enL523Pv2ydiSS1KwHAEEdgqPCcS1Kq9rDTTpWLN59I/4V3PUZz2ODZ74t4Pt5goBE+ft5Rfpg83XkTh/+Qx0TX0PnNUG/vlEwnARVX4KWBz3h+EFPi8W9eXmW6AAWH+VZydqMmgS6WuxTmS2IL55lfy0pAaDWJZJz3TIBdwWs3ofHEIwp6LXv26BF+cspqE8iWurSeNXDXrJrdqkYcj076nn+9Ypd3RWkAMPQKnppNVK9+GnW0p3gslbXqFj50lR+R3ZFuDLMAc7IK7Wsl+0iFf3SNZW2MW9HKApdvyB6X4yDHMskJsmFj+CrB79zPNrqUoaQlw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001186, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --mkaklzxm72eg2bej Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable +Cc Kuai On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang wr= ote: > Hi all, >=20 > I've run into a problem with Cgroup v2 where it doesn't seem to correctly= limit > I/O operations when I set both wbps and wiops for a device. However, if I= only > set wbps, then everything works as expected. >=20 > To reproduce the problem, we can follow these command-based steps: >=20 > 1. **System Information:** > - Kernel Version and OS Release: > ``` > $ uname -r > 6.10.0-rc5+ >=20 > $ cat /etc/os-release > PRETTY_NAME=3D"Ubuntu 24.04 LTS" > NAME=3D"Ubuntu" > VERSION_ID=3D"24.04" > VERSION=3D"24.04 LTS (Noble Numbat)" > VERSION_CODENAME=3Dnoble > ID=3Dubuntu > ID_LIKE=3Ddebian > HOME_URL=3D"https://www.ubuntu.com/" > SUPPORT_URL=3D"https://help.ubuntu.com/" > BUG_REPORT_URL=3D"https://bugs.launchpad.net/ubuntu/" > PRIVACY_POLICY_URL=3D"https://www.ubuntu.com/legal/terms-and-policie= s/privacy-policy" > UBUNTU_CODENAME=3Dnoble > LOGO=3Dubuntu-logo > ``` >=20 > 2. **Device Information and Settings:** > - List Block Devices and Scheduler: > ``` > $ lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > sda 8:0 0 4.4T 0 disk > =E2=94=94=E2=94=80sda1 8:1 0 4.4T 0 part /data > ... >=20 > $ cat /sys/block/sda/queue/scheduler > none [mq-deadline] kyber bfq >=20 > $ cat /sys/block/sda/queue/rotational > 1 > ``` >=20 > 3. **Reproducing the problem:** > - Navigate to the cgroup v2 filesystem and configure I/O settings: > ``` > $ cd /sys/fs/cgroup/ > $ stat -fc %T /sys/fs/cgroup > cgroup2fs > $ mkdir test > $ echo "8:0 wbps=3D10485760 wiops=3D100000" > io.max > ``` > In this setup: > wbps=3D10485760 sets the write bytes per second limit to 10 MB/s. > wiops=3D100000 sets the write I/O operations per second limit to 100= ,000. >=20 > - Add process to the cgroup and verify: > ``` > $ echo $$ > cgroup.procs > $ cat cgroup.procs > 3826771 > 3828513 > $ ps -ef|grep 3826771 > root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=3Da= uto 3826771 > ``` >=20 > - Observe I/O performance using `dd` commands and `iostat`: > ``` > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > ``` > ``` > $ iostat -d 1 -h -y -p sda > =20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn k= B_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M = 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M = 0.0k sda > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M = 0.0k sda > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M = 0.0k sda1 > ``` > Initially, the write speed is slow (<2MB/s) then suddenly bursts to sever= al > hundreds of MB/s. What it would be on average? IOW how long would the whole operation in throttled cgroup take? >=20 > - Testing with wiops set to max: > ``` > echo "8:0 wbps=3D10485760 wiops=3Dmax" > io.max > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > $ dd if=3D/dev/zero of=3D/data/file1 bs=3D512M count=3D1 & > ``` > ``` > $ iostat -d 1 -h -y -p sda >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M = 0.0k sda1 >=20 >=20 > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn = kB_dscd Device > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M = 0.0k sda > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M = 0.0k sda1 > ``` > The iostat output shows the write operations as stabilizing at around 10 = MB/s, > which aligns with the defined limit of 10 MB/s. After setting wiops to ma= x, the > I/O limits appear to work as expected.=20 >=20 >=20 > Thanks, > Lance Thanks for the report Lance. Is this something you started seeing after a kernel update or switch to cgroup v2? (Or you simply noticed with this setup only?) Michal --mkaklzxm72eg2bej Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQTd6mfF2PbEZnpdoAkt3Wney77BSQUCZrothQAKCRAt3Wney77B SSR7AQDYbqxZijycGMxpcelGXyctNdIBM47M8Why3R2HzYQBSQEA47s0N6yTrlnt VHfknkNUobpvCMF9U3kDAj3zaVwfMw0= =+ZWt -----END PGP SIGNATURE----- --mkaklzxm72eg2bej--