From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAC2BC7EE2A for ; Fri, 27 Jun 2025 18:54:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 869D56B00BA; Fri, 27 Jun 2025 14:54:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 81A6B6B00BB; Fri, 27 Jun 2025 14:54:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 708F86B00BC; Fri, 27 Jun 2025 14:54:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5D9896B00BA for ; Fri, 27 Jun 2025 14:54:52 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ED93D584C8 for ; Fri, 27 Jun 2025 18:54:51 +0000 (UTC) X-FDA: 83602082382.18.E0A6C6E Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by imf01.hostedemail.com (Postfix) with ESMTP id EEBB740004 for ; Fri, 27 Jun 2025 18:54:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B0fPswS0; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751050490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MkZf/rGrOyK/HyrljMZNxwNo/mJruqVicCPTs40b2Ts=; b=CHwNTaUPmEc87WfoaCsNRVFQt/+X7GNwt7HEORLGmeVZWwGc9vA+zxo4cQbaMbOYpRDbGZ UFaYOISvmmLH0OF/Qs6Yp3gm7VWx9UG0kxZrM6XdBS1lRGYhICLLfctoDr0qnf/CcG06ce OH3jI5YmTPEzuXbs5jhJb3AKKMKSUKY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B0fPswS0; spf=pass (imf01.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751050490; a=rsa-sha256; cv=none; b=kAFH2JYHT60Jf4N/ojSh2Kp6Io6nhScK0OyMGwz1yDA8wnO9INLDIu4FVkednP/7Uc+xAb FzkRGhJCSzwy7kRwy2j+PJPAmd+i18A+dh0GJA1bdY+ID9bigF+Z/bPpdvtBOfjAR4Gl6T loWemwoA5gUo6nk3FFBuj9w/RUjZi84= Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-32b7cf56cacso23798311fa.1 for ; Fri, 27 Jun 2025 11:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1751050488; x=1751655288; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=MkZf/rGrOyK/HyrljMZNxwNo/mJruqVicCPTs40b2Ts=; b=B0fPswS01WoSIhsvDo/elvOicaZuUcKVose2Ow+EZUevgN8I6arQ7+nDgEct5VqbOc VaAsngzoLCOfXqzJitc/YsKASqqa6RkdGM1RsuxNDqLr34E8Npvru4adVmUiLBlfOt/S I3qSRqzpRBLiWPckNH5nWnrJs0KdyUWYTCnXwU6zsg7+sZKKWjgjU99MRU+ffLIAyT2/ 7Vx8lqW5nEJPIcM1CXg/ZkEz4jytwPlHVvR+4wedfihTFdVCOHruiHoYdoyiAjCn/n5Q pK9zDQZslBjy233bV122MVZtqy9mCxqVOeg4oAlIaHBhs/rUr8TRSJABeqS+CjTt+boz vdSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751050488; x=1751655288; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MkZf/rGrOyK/HyrljMZNxwNo/mJruqVicCPTs40b2Ts=; b=gSRYKocvY1Mi213DK/rQ4qEXjqOhBfHlilir+PGtZ1HVqjW8Wzh1xL+hadpXidlz6G wivc+7q9/kqh1ah241i9j0zxpSJD55RCpCJBUWZOKARmp8e35dbRh9KDWev0Dv+skdPj r+HujWPzhxWVGTEllZaCmnNvTp5a7sl1plfmjpagXyukcZp6AmHYQnAuMRSAKLx74bmb VM0Iu9E3d82g2YXIuLC0oDdmTOhurSKg8o9Mdp5ewZKVDBdxjQW4pssxeL9xxbtos6Ej rNTaF83fE45qdgq2CNkcgJiV7gEIiBJY6vImn1Kgr8zvhTRG13DuKOcAYlZtgmxX8tk8 XiAQ== X-Forwarded-Encrypted: i=1; AJvYcCXE7vYBu9Za+805Ge45J7M43pBJyltK3Oj+jKtbc33y0GuucOgRnm3VC97RWFO9c73R2J0Hmdrv9Q==@kvack.org X-Gm-Message-State: AOJu0YxtMCLIzsbqFSe/FDfcoIoUo7Rb8En7PpT2Dh3ejOPZ+P4yJh2b Y9WYEbYBj5qs2QJZHhLo8QsPHtknLvSJ9Z+b2DQNbKGGARYNgX+RpWcSJ2zIlmhHYyhOQl8/WPI QsBie3Nve51Cxtwv+KGOZkl5nHIqmAuo= X-Gm-Gg: ASbGnctI++CNxfyWYarqlmfAULK/Gc3kPNRMONhnx/d9wVPBqzIOHXRzIJOuQNA2ZB7 NDfUgMSQHE+o9Dyzvv8sMPmO6PLImTZsWoddALlKjT7SsDQocg+WnYtNnJAbXadF5KEPocfrSa/ HiSJX0paoC1vRIyG4dkJunOuVP/6NX/LASzL9YKObo7GHWjcgdg1Df6w== X-Google-Smtp-Source: AGHT+IEEWJL2fzTT6sp/iFnXIOYPN7CNjGrDkAoMvUyzyseYP2O8kfk5drk0YGN9ll3GTdOreVTaWBEIEhSZBaQSC1w= X-Received: by 2002:a05:651c:32a:b0:32b:5eb3:280 with SMTP id 38308e7fff4ca-32cdc50e821mr13050961fa.29.1751050487729; Fri, 27 Jun 2025 11:54:47 -0700 (PDT) MIME-Version: 1.0 References: <20250415024532.26632-1-songmuchun@bytedance.com> <5f622eec-a039-4e82-9f37-3cad1692f268@huaweicloud.com> In-Reply-To: <5f622eec-a039-4e82-9f37-3cad1692f268@huaweicloud.com> From: Kairui Song Date: Sat, 28 Jun 2025 02:54:10 +0800 X-Gm-Features: Ac12FXwDAS0QUytrD_AyHHgfvBJCFweeNF9Ytnf0eONhRRe-M_Ek30p9WmZQNCE Message-ID: Subject: Re: [PATCH RFC 00/28] Eliminate Dying Memory Cgroup To: Chen Ridong Cc: Roman Gushchin , Muchun Song , Muchun Song , hannes@cmpxchg.org, mhocko@kernel.org, shakeel.butt@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, yuzhao@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EEBB740004 X-Stat-Signature: aqjt9gi56p1cp1cohsrpr8hz8h61gmrm X-HE-Tag: 1751050489-961427 X-HE-Meta: U2FsdGVkX1/Dy4a+y+SkLdZJZW/itWOaPUOYZPE2aa5ZKoIvXRKsNlAr6RFUbkILOYo3OYOSF5y9mvCjwZf5gx9JomjQJxkcr77xcb1JF4f3cUFvvM6Wfu8IcfevFSs7A8NOjfAp7UjsHIr6h4vItSdcT93fW20wkRPH1l6AXbxiNDVYa4xQvc+iNgxqq4LxIHtXP3bXqbYa6sCqgtg4xQHPbvSGHenAkRKbm7hmgjRrMzGIx4o4btBYV0KZJqp0SNEnyISHLkDX2hXtUAil6cL95PEL3riTCeQgaMwzKQDjS7qAiHhrzdOYYLwxTagma4tFnoRGLMUDtghnOWv2SXsdQH1JwyUYZHZ0vcltkN5AyAzxz6P5zXKr278CoPgF0CY9X529Zb23z+16wdxZ+WX3SHfZXoz4lO4FwP+Xjs8mJyX0lPs40xdUnNl/encfGWLo6r3XBPBEb6aDRkxXGH0LCimjTc3mR0q5SNRsA+nn+x8MGcG5y2yuMls06X1mi12qquC3+79CKrFu7G+NWFRSHOjZsR+/kDi4Pv0vwb6sLBPYPGlgfjeGl5HeqN2oBlASxdZcqB26+jWvC9lAPW3NXVDlj/YI7AmwIL9j2ovVS/CcNpCHLJ73aXy8cLHsjnhJhZS7lMEfb0NGLlGa9q7XfN8fV2YuysukyXQxdR3f+OJbdN30HWjjFKqhnVrvMeduUpPvZQ9W+0jgS1KNAYZDlt0eIJ0fmjccGzL4MwNVJGl8Xqv/hvaML8J/H067ZNlwkqZPg7qOhPu89ubB2urd3+iOT9w88zCPNNF7WnVvT8dRuFe8/AbsOLjby5bxNsr/z8Pr0/lXx3STkuOUS1YoFzF1NCKv+OJmHCou6ohEGBJZj9ydrccHQPhwA4vm8zR/4t9QBCSpuxSHZY+m7We2w/2xHXken2uFTArIMwwy3DLJ6dW/R3zJ7uUyhRIPmXa+8sKjVElRuMPrVEb g3GK9nwB 8i/UdNkJSgunNz7O3+2A4HCSB0ZqpETddv/yZPp36cdIH4d6mBdDiIgy3vY1sgVVSaaI8LW79/iHe53OiLZzdXUvR72XAJD1pO03OdtKiV8e9rx2NUuF9YY4FwL4WpEYzB6ENlt483J12ZN5wY7gWcoV44AtHDN3GBiF7dSlgVuKVshOfj3v/2OAup3/ax3eYhsSKJCnQCAInXKfMpWwxvah0af+F0UzebGKnm3TcF29kr48sdmwbSRwSb88CQT0RdfW3vWyOhEcv6nQLQVo520FWvi6MAn8aA85LWbq8BynkyyJZYcP0XsHtKFSzwWH1uLI6hjqPFxteRkSeE435kOS1Q8/RuFVu5rv/e1x0DHbhk288U92Q8wTe0EiUSfm9aGTP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jun 27, 2025 at 5:02=E2=80=AFPM Chen Ridong wrote: > On 2025/4/28 11:43, Kairui Song wrote: > > On Fri, Apr 18, 2025 at 5:45=E2=80=AFAM Roman Gushchin wrote: > >> > >> On Fri, Apr 18, 2025 at 02:22:12AM +0800, Kairui Song wrote: > >>> > >>> We currently have some workloads running with `nokmem` due to objcg > >>> performance issues. I know there are efforts to improve them, but so > >>> far it's still not painless to have. So I'm a bit worried about > >>> this... > >> > >> Do you mind sharing more details here? > >> > >> Thanks! > > > > Hi, > > > > Sorry for the late response, I was busy with another series and other w= orks. > > > > It's not hard to observe such slow down, for example a simple redis > > test can expose it: > > > > Without nokmem: > > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > > SET: 16393.44 requests per second, p50=3D0.055 msec > > GET: 16956.34 requests per second, p50=3D0.055 msec > > > > With nokmem: > > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > > SET: 17263.70 requests per second, p50=3D0.055 msec > > GET: 17410.23 requests per second, p50=3D0.055 msec > > > > And I'm testing with latest kernel: > > uname -a > > Linux localhost 6.15.0-rc2+ #1594 SMP PREEMPT_DYNAMIC Sun Apr 27 > > 15:13:27 CST 2025 x86_64 GNU/Linux > > > > This is just an example. For redis, it can be a workaround by using > > things like redis pipeline, but not all workloads can be adjusted > > that flexibly. > > > > And the slowdown could be amplified in some cases. > > Hi Kairui, > > We've also encountered this issue in our Redis scenario. May I confirm > whether your testing is based on cgroup v1 or v2? > > In our environment using cgroup v1, we've identified memcg_account_kmem > as the critical performance bottleneck function - which, as you know, is > specific to the v1 implementation. > > Best regards, > Ridong Hi Ridong I can confirm I was testing using Cgroup V2, and I can still reproduce it, it seems the performance gap is smaller with the latest upstream though, but still easily observable. My previous observation is that the performance drain behaves differently with different CPUs, my current test machine is an Intel 8255C. I'll do a more detailed performance analysis of this when I have time to work on this. Thanks for the tips!