From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18F02C4345F for ; Thu, 18 Apr 2024 02:14:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86E606B0093; Wed, 17 Apr 2024 22:14:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 81EB16B0096; Wed, 17 Apr 2024 22:14:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E6056B0098; Wed, 17 Apr 2024 22:14:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4D16F6B0093 for ; Wed, 17 Apr 2024 22:14:20 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 08B03A0435 for ; Thu, 18 Apr 2024 02:14:20 +0000 (UTC) X-FDA: 82021033080.30.0503A19 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf17.hostedemail.com (Postfix) with ESMTP id 221C040005 for ; Thu, 18 Apr 2024 02:14:17 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UedQJEF0; spf=pass (imf17.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713406458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7EWy3InHHcdr6Et3oKPqHiA/efKMXLdl9NWj5a3LIKA=; b=rohAWQqzrPDSRgRcxayLvOZM6d3IuMKdJ06rUR4ADfurU0w0YjLH5lNHZN6DCaCb/WPfGz kI4S45oPRUsBgbTH0Cugb3jWSlOUpplVElToZwOmEq7OU04zDsjjVmuz4Db3CRTCwaqQPy tCYznwBjBIwIa4aReUX11lLXz80Rk6w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713406458; a=rsa-sha256; cv=none; b=DpttobtpQoB8ypIS0cDEvRA8uKoPW3errRhXhqoYHLyFWvemkn4S0UXeiAIdtmKzSURZwk vBqCLloEZgACinFxYMPWInU3hZxPF5fduAt7C27ngOTkiLuvQlIBXGvvs9H8z9pQcjfXMl zlWqm9AIX2j1RPVK1ckUcxS30fQXC/E= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UedQJEF0; spf=pass (imf17.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-a526d0b2349so27459066b.3 for ; Wed, 17 Apr 2024 19:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713406456; x=1714011256; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=7EWy3InHHcdr6Et3oKPqHiA/efKMXLdl9NWj5a3LIKA=; b=UedQJEF0MHALuAg34Ah2ZQzq9gVxdP1kxjb7XPIcv9Ku7yadX5w1SDj9MaM4s29UCf oAPYIAY7Ifb7J4kVuAFKDOxnGoKz0+ZCPF24mMsjoYvf1SYzRf/vxUk4G8MfkIuf7BLJ jdFS6o63tWy+k++lj4fhm2O7p4DIqhbsYn06YcX1SYYohUETJU/MUxojnItf4xNE/kJu jQmqd9ohAFNXMjiWOIRYcla3qdHsHH+JwDd+buCf9Ns7tWk2xCDzTBCQY1F4hC3W4Asd lEDsg/emXQS/fXq/fOd3qrJJpALTpSiS4ZkkAXUcvzLdVeOdM6dk1fNDqy+gix8NPmyn WMAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713406456; x=1714011256; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7EWy3InHHcdr6Et3oKPqHiA/efKMXLdl9NWj5a3LIKA=; b=lOQ4iKxeAJjtIiKvJqBKwZCmH4Bx5c+Pu2nD0KJo8Gd9N7tsRH920hPt3mJbtr84iM XCCPSzY5vvZU7ecy9fn5P7BQWBoP2kMcJbZQYzOHoVxZRoEt8VXj3+OTXhiVqawLM8ec 8EdiNLfgt8gMc3FRTaTThHM3ufp3bCiNJFintdTxHIFRgytYUAgo9lW0H6kDezpTIsQl fqfw8ap0LXPBymKjl1ZKtuKkaz7jBmJ7+Kdoeos8LtZXL7+VApgOCnn77SkbtUPjaOOi DYo+ENauUBlMTiEGFU3OeYng7KAxXb6KFVlrXKQy3kQZD6UiN2KPnPqIkTdInJExB0yS 1U4A== X-Forwarded-Encrypted: i=1; AJvYcCWxyagiZSIzl++YRuciuHzVAZJi75LaOstI2XZ+SSv70JbvARUJH5nC2z9LecUOQLywpPoX9bhIs5FICdJXAiMrKEE= X-Gm-Message-State: AOJu0YzmHckN25vl1ETVklnIXYdworp8jNzYuvzGORB695hQ+rWnOn5j WCScpVgEatA06ELAirxi0iGfoCJJGu1llNKwK4K4ibQhce4iPtHNyga9Z7X+zaDCQlHXnv7I/RJ BsdYd4znlaUAIyDWx7/nQ+6/3GBvvAfHj1rhY X-Google-Smtp-Source: AGHT+IE/Nf4C5eaH5s4T2RRQ0jDt7/fjhylkOElaSpZipRuSnyI2rIKGrPh8cigEE9JO5l6XGaOIHOMiew3CNftZbYw= X-Received: by 2002:a17:907:6d07:b0:a55:6f39:3364 with SMTP id sa7-20020a1709076d0700b00a556f393364mr296261ejc.15.1713406456189; Wed, 17 Apr 2024 19:14:16 -0700 (PDT) MIME-Version: 1.0 References: <171328983017.3930751.9484082608778623495.stgit@firesoul> In-Reply-To: From: Yosry Ahmed Date: Wed, 17 Apr 2024 19:13:40 -0700 Message-ID: Subject: Re: [PATCH v1 0/3] cgroup/rstat: global cgroup_rstat_lock changes To: Tejun Heo Cc: Jesper Dangaard Brouer , hannes@cmpxchg.org, lizefan.x@bytedance.com, cgroups@vger.kernel.org, longman@redhat.com, netdev@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, shakeel.butt@linux.dev, kernel-team@cloudflare.com, Arnaldo Carvalho de Melo , Sebastian Andrzej Siewior , mhocko@kernel.org, Wei Xu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: phemumdmnwanyqtnzbt7azsikft8fomh X-Rspamd-Queue-Id: 221C040005 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1713406457-994646 X-HE-Meta: U2FsdGVkX186iIgLcMjDkadpK+H/48pDwVT6RhkcQbmUkU+Wc5oRytWacZahjsiizfpxuQxZ9Kkfdg7DRSrMlB2IBzfCMfgWyoJVdz5zff+6Qkg2gJDVHptK2YknWbozOKXvRfECK8MPohSC+5UcPcz/IlbOWC5rtPdTiT40Y1YlCyJt+xdsluv8Ecn79zQTShfrm4KAm+IwsXly8YyaT1IXHZvPBRQ6BsLQxLTLX46olVpRWGMbv21xCnbGPDsu5nFxhaTlGCv55BQkfoGnziafiRjP6x49zCGNpr2RcNkjRf+eZJ22d4cV3KAqp+YDzJbI6juHbJfdyqMUAEDhr8ITpQSaQgyder3B8A0EgGP6JngnSaQMgtPaosVx1YZh12oIHetpJDy6Uj/JEzdYCDTRN/CYoc9R5kSalbCfKVgmeBwzLzju32XUNqvAzVXjtjrrVbXkyE19CMcn2nNhck8VxbajxILQguwFPYevv+WsqSNj++aEVXPwFpFsuqR/xKaLsfnjHnpFiIZZrky6r0R7miwKbrLGBF126wX08JxmZd3uVgyUKHbGdvwwiI2JPeuuUPbDXm6N++rhH1VHbxEx8nQJtmQERRFLN6jFvQW/H0OOxui85eD1LvaaOqaOjgnMqHS+dglS05r99JBZZ1LdxZ6EENasXH53FOycPD6XbvHN3Fbxgnuaw7rBsTuDEdO25ZwQTMkgkoehlrtmt6HO7HgHZNG1WfhN+PMOQB5Pc4DBQ2B6oYsjQJBFSQDqqqFaxXJ+pKZ2LKYKGLekhZRxAh80m+xaTWG6i4tagkEHCet1gHO3lpk7tbIAHEpzwZKwQyunmdhJ1pn4C+ZSZktASK7hijT7xnF+YSWAz7pDsFUlS1F434mhqrlw/gRatbiMz8pvLlaOiCPyeFhJpli9j0sEKgzfdbFQCzlgrpA+tJwYyX6SlC+bgWQprG7fa1Or+WlugCHKbueaedV fefmyrmp NRrBuHjWjEacAj6TxlJOHmbVyukVPq6/5SIyiAD70JqAkt0ShFH/Ov7UsUBpkKQbsnt+SirHGFXq4XQ6IFeFGAPxcpBnnNGOMkYVzjIvqmtZpP2iSYte5DVpc865WsQGwTibFpuwuVGo7lmJauUjWQJxgMsrylX6LaapFYXIOkult65hilQc6+98rzdNQN20HRFoZUUNWuY8CV3CtXwshOsBQ3mzkDkkMQbckniPYWpPz0oFZRmrhx4Yu4YLMlzIyxd8pvSAE7bq3mZmINC3mv134pPdR4LchBfVtmOAxwdvxFDKn2XntsAFFRa4AUhFR1+Z4VqaL1tAb5Lw2Lu/WyvN1YAnUPRdbIiF4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.002147, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 16, 2024 at 2:38=E2=80=AFPM Tejun Heo wrote: > > On Tue, Apr 16, 2024 at 07:51:19PM +0200, Jesper Dangaard Brouer wrote: > > This patchset is focused on the global cgroup_rstat_lock. > > > > Patch-1: Adds tracepoints to improve measuring lock behavior. > > Patch-2: Converts the global lock into a mutex. > > Patch-3: Limits userspace triggered pressure on the lock. > > Imma wait for people's inputs on patch 2 and 3. ISTR switching the lock t= o > mutex made some tail latencies really bad for some workloads at google? > Yosry, was that you? I spent some time going through the history of my previous patchsets to find context. There were two separate instances where concerns were raised about using a mutex. (a) Converting the global rstat spinlock to a mutex: Shakeel had concerns about priority inversion with a global sleepable lock. So I never actually tested replacing the spinlock with a mutex based on Shakeel's concerns as priority inversions would be difficult to reproduce with synthetic tests. Generally speaking, other than priority inversions, I was depending on Wei's synthetic test to measure performance for userspace reads, and a script I wrote with parallel reclaimers to measure performance for in-kernel flushers. (b) Adding a mutex on top of the global rstat spinlock for userspace reads (to limit contention from userspace on the in-kernel lock): Wei reported that this significantly affects userspace read latency [2]. I then proceeded to add per-memcg thresholds for flushing, which resulted in the regressions from that mutex going away. However, at that point the mutex didn't really provide much value, so I removed it [3]. [1]https://lore.kernel.org/lkml/CALvZod441xBoXzhqLWTZ+xnqDOFkHmvrzspr9NAr+n= ybqXgS-A@mail.gmail.com/ [2]https://lore.kernel.org/lkml/CAAPL-u9D2b=3DiF5Lf_cRnKxUfkiEe0AMDTu6yhrUA= zX0b6a6rDg@mail.gmail.com/ [3]https://lore.kernel.org/lkml/CAJD7tkZgP3m-VVPn+fF_YuvXeQYK=3DtZZjJHj=3Dd= zD=3DCcSSpp2qg@mail.gmail.com/