From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C73ECC77B7F for ; Fri, 27 Jun 2025 19:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 532DB6B00B2; Fri, 27 Jun 2025 15:14:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50A816B00B6; Fri, 27 Jun 2025 15:14:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4474E6B00B7; Fri, 27 Jun 2025 15:14:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 358396B00B2 for ; Fri, 27 Jun 2025 15:14:23 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A3671C031D for ; Fri, 27 Jun 2025 19:14:22 +0000 (UTC) X-FDA: 83602131564.11.A24F6BB Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173]) by imf13.hostedemail.com (Postfix) with ESMTP id 393A420016 for ; Fri, 27 Jun 2025 19:14:18 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=TVlierNB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751051661; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zZ4E07hhGNy4W9XkbKFt8clyjUYqy4CKRl6nAq2dZWw=; b=KGjg9K+VtGAy8O2vsFJQnZvdkKappiwfzHCYODZGKTdAYn7JPIeebtl+l7+3VqD+I5BS9L 2i+bX8cqF6RKrP6vaC/iCRUQOPueSlreOE4D8wIJBiy2z4vxyv6p70/2lAFLKbaqu9qepD nwYYHqk6NJVnR6RrKHX4bq9m00jsmgc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751051661; a=rsa-sha256; cv=none; b=e84RRd6GFVCRvEgz7J+WUXTKeP5XKGvFW9kj7Mp1uyusnFFSqEZ5vPGGxgjp92GNkKv+eL 2TXgqyXNXbtNYskAvw7pBl1TmDIVsD5P2+2y55RTxqXOmF/GiALGFQCPSjFAutoDzROddc Q8x/lx1rIHWNxCrXuv+IDkk7Ga8ACYg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=TVlierNB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev Date: Fri, 27 Jun 2025 12:14:07 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1751051653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zZ4E07hhGNy4W9XkbKFt8clyjUYqy4CKRl6nAq2dZWw=; b=TVlierNBVW0hk2uG4e4Hfd2RJEOyJ1xHNvKYwESP/Z3WLhH4ziqomfaXtukklglcCLtBVG mGOAL9R2nU0qDRaH71npFYP36DufqumXe/5KuAZcDz3cszljWBSxVl0RHAW2mOtn2K5HlR 85eYSCG6B2JhnQ8CCFnxdz+XRscwgqA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Kairui Song Cc: Chen Ridong , Roman Gushchin , Muchun Song , Muchun Song , hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, yuzhao@google.com Subject: Re: [PATCH RFC 00/28] Eliminate Dying Memory Cgroup Message-ID: References: <20250415024532.26632-1-songmuchun@bytedance.com> <5f622eec-a039-4e82-9f37-3cad1692f268@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Stat-Signature: b71jxk8s5b47ttrg13af7i6f55d868qq X-Rspamd-Queue-Id: 393A420016 X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1751051658-756803 X-HE-Meta: U2FsdGVkX18H5Pjwntt7FQzkPVOwVX4G7y9KHfMqRc7lR7IHIATvSvd78QuLmludKmYMNnQ+qlFBZUOpNDmHGmOiCyTW5OpPnmy32jTqR9YY28itlHKuluBKrhzGU5ji/F8QG3mgZsfTGDWDZh2Dbd7xnVS1t+GjW9AZt8uS9BFGtGoaksm1VqzHNy8UaxF7AAlttNV4VTwzNVWTtqc4dpTJBIzDUH05FvSr/Hg/CAh9xZVhZ6TgXAwp17UIxWRqLE3vuO3CZl35Y2dd3B4a9+3kixkxqtIiXzBy1N55ZWrCfXxxhsHOyoijucNvnVSQA4LftI1y+D1Lm9BNHdg/vQEn+yTAWMvS4MxWqFYLqF25fyUhm4nKFcw951I1p2dsYa1aML+4pZ5kzVp2MdngBrq08s8EB3brjp6Br89Dy9zTEjF9NW3jxU5agMS7qxmHp1jl3oDHKKk8gkwlxb53MAgvM8uE8Ge5ixY79eKO0LrUwQUNFLFpqtL+WsT690DL/ou0TI0D+1DmdEWF8sbIGU7W5BzhNk/k5YN+89WKjVTCgq1ji/q+E3OCQvYKx7O8uzIA02xY+xMPkZYO+zJ/PPutrT+bDejW9IkwTr0cTKcS5ze8ElKHpYVnH5BOGXQKu166Yp/dlzVV4SPRgvFI1xGw9jDMq+EiZHIrSgKVD8K3i+kyourDNlPkNtVnZ4MaqrdUEalshD9Wv/cjYJp5yZqZWQXkSBKHn16fBbNzlQH11ZwcY9P6UEhIVXqWNfw3mva4BeBlO9yeR3gJA97/yWvDqxALF+UZES4P4+MdQR5jdJbevQ2UdIZmeS0pZDGI52Zqc6YIkDAic602RYdXbQXwRnIOt8cayPEzWUB/PEAlEoUln38lnW50OvKTZNZGkANxqUd0H4HRg9NcbBTNDXmUBZBlEi1I8vfC6YankVHNlzAVTSLiehH9RJrFTFtMkJcgcBg1+4lwYqfonTl EHwR3nbn mxKB5JwnRrXR0kzih4oDmPK0sGnTS8FzXHkBgMvNWzICvBVTVtPjQiWdGzNuaij33HYvc8LuDltL6UR2JMk1XmwT/PoOMxMSnKgWuz7LRLwdrUEnZYeKmi2PCH8YAwRi0BmCJRByDbnteIeP48KgCsILMha1irNXQ+kF68gno18pxPIkzZRE7Fhs5XTlt4BUWrLWVjZo8JUCUFMDp1Qaar63cDuseUOZj+HYRC+loi8+8m+BMcGgqnDSf2mnCYQVpEdatPPG6Ra8OV8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Jun 28, 2025 at 02:54:10AM +0800, Kairui Song wrote: > On Fri, Jun 27, 2025 at 5:02 PM Chen Ridong wrote: > > On 2025/4/28 11:43, Kairui Song wrote: > > > On Fri, Apr 18, 2025 at 5:45 AM Roman Gushchin wrote: > > >> > > >> On Fri, Apr 18, 2025 at 02:22:12AM +0800, Kairui Song wrote: > > >>> > > >>> We currently have some workloads running with `nokmem` due to objcg > > >>> performance issues. I know there are efforts to improve them, but so > > >>> far it's still not painless to have. So I'm a bit worried about > > >>> this... > > >> > > >> Do you mind sharing more details here? > > >> > > >> Thanks! > > > > > > Hi, > > > > > > Sorry for the late response, I was busy with another series and other works. > > > > > > It's not hard to observe such slow down, for example a simple redis > > > test can expose it: > > > > > > Without nokmem: > > > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > > > SET: 16393.44 requests per second, p50=0.055 msec > > > GET: 16956.34 requests per second, p50=0.055 msec > > > > > > With nokmem: > > > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > > > SET: 17263.70 requests per second, p50=0.055 msec > > > GET: 17410.23 requests per second, p50=0.055 msec > > > > > > And I'm testing with latest kernel: > > > uname -a > > > Linux localhost 6.15.0-rc2+ #1594 SMP PREEMPT_DYNAMIC Sun Apr 27 > > > 15:13:27 CST 2025 x86_64 GNU/Linux > > > > > > This is just an example. For redis, it can be a workaround by using > > > things like redis pipeline, but not all workloads can be adjusted > > > that flexibly. > > > > > > And the slowdown could be amplified in some cases. > > > > Hi Kairui, > > > > We've also encountered this issue in our Redis scenario. May I confirm > > whether your testing is based on cgroup v1 or v2? > > > > In our environment using cgroup v1, we've identified memcg_account_kmem > > as the critical performance bottleneck function - which, as you know, is > > specific to the v1 implementation. > > > > Best regards, > > Ridong > > Hi Ridong > > I can confirm I was testing using Cgroup V2, and I can still reproduce > it, it seems the performance gap is smaller with the latest upstream > though, but still easily observable. > > My previous observation is that the performance drain behaves > differently with different CPUs, my current test machine is an Intel > 8255C. I'll do a more detailed performance analysis of this when I > have time to work on this. Thanks for the tips! Please try with the latest upstream kernel i.e. 6.16 as the charging code has changed a lot.