From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C649FC4332F for ; Mon, 5 Dec 2022 17:25:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4864D8E0002; Mon, 5 Dec 2022 12:25:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 436138E0001; Mon, 5 Dec 2022 12:25:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 325878E0002; Mon, 5 Dec 2022 12:25:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 244088E0001 for ; Mon, 5 Dec 2022 12:25:57 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ED82312030B for ; Mon, 5 Dec 2022 17:25:56 +0000 (UTC) X-FDA: 80208930312.23.8C1C53B Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 97BFDA000A for ; Mon, 5 Dec 2022 17:25:55 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FpTRKX8U; spf=pass (imf25.hostedemail.com: domain of shakeelb@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670261155; a=rsa-sha256; cv=none; b=wB2cRWtzSRn9RR15UayAGUq0TPefI98UwpEdvuo+JKKM4OQhtLxWUJZYe+nX6mTrrC85EE l8YqxeRbY+gLfToHtk4z04p+eoRibOprhkW6HvrE2tQSW36LALlY1WCkTlhF7ADDzL/i5M w1zjPWKhByDRoDMfKQZBO7ptKg1AsGQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FpTRKX8U; spf=pass (imf25.hostedemail.com: domain of shakeelb@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670261155; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Edkl3+qJAeg+/U50a0cxOdfQg98c2hJcXAmKpe7H710=; b=zD+gkvEBokD/UuNJA0oAOcbJT5JYdmBGZcS0jl2Ec68gLy3OhxTtvAbaoP1JRK4voPF13c B1E5zngfcOjciXSRoDUhrj+VnL7qVYlu4/g3GO3NLUKdiShg4zl0d903m2NRCB6wZIEGcC 9p4uWIQHKBRH5jO5bx7nQq5+kRms19w= Received: by mail-yb1-f173.google.com with SMTP id d131so11963768ybh.4 for ; Mon, 05 Dec 2022 09:25:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Edkl3+qJAeg+/U50a0cxOdfQg98c2hJcXAmKpe7H710=; b=FpTRKX8UJUk9Op/KK/uXUUhZ6S5JK8MyesWCc4GCpXfqngc25OS1MFr3Ri2m4P+3KF JEkGVmny3vAWqXRowqwU8mqo9EFHYv332iGeFtVv8YMTZF1fbfHQZbo2lLf1eKmbVFuP aMECVitbiW1LmseQvTaaPEKfuJpCG84qxwsarkmyocsZwr1HWjdyjf48+GRUZfsfS/+z gT69a/gMeHf30XPTkUrvXl/1xM+AqdpU61+dFzWOilydAtPICWAtWE1zLWiAnNAO+SMp tAjbNPWXWQtOD4eQCbnZM1ik3f+9RIx79QYBUorEQIjttnmjPsE9BsUbEqslchMa0utO pWoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Edkl3+qJAeg+/U50a0cxOdfQg98c2hJcXAmKpe7H710=; b=fcHnpI518sozlDwevYGrxWwDgpGr7IMLPEAyzt8y/b0N1Genp8UbEhKDFcuWExEDrI S1H5fmtvqhVvZloNmD2ZagGRSp64QVNY2PJlIuLzcTc8kKQ/AKR8ETSiZByENsmk/3N+ EVPPBJsklG8/gtPe0iNl3UX0hIIIyPTUL1ZN/F/IKWiJzYSH/jEmY3PMfkRWzE2+IvTG 6qM7a9M/xWh5bi+yrqvYi1slqFfkVbS56lnRXm5DRMK9I8SbBe7OCCrJ1MgPxKmHsBM7 BS963yCdmKDVWeiVhMMv2Nu8T4oaGAVgKuJpGphoyJyX/vIbbmZUfCBWXGFepkmVMh4o KgBg== X-Gm-Message-State: ANoB5pkXFguCVcuXVtJEma5Q9lfBpUs8MIw1FOXHJFGQiF9HgKojDD58 WRkac4szq52bjMs0Zu1XI+qhzxVbqVsq9XiGCz86sA== X-Google-Smtp-Source: AA0mqf6DwTZNLDhsdfcEkb2wWE5R6b32ryRzEyZrdKlofhuY67qZdSlztP8r0l3+UZteuF8ZRujA9JwtimR9iLWGiIo= X-Received: by 2002:a25:ae92:0:b0:6f9:d605:9f6a with SMTP id b18-20020a25ae92000000b006f9d6059f6amr27618176ybj.294.1670261154521; Mon, 05 Dec 2022 09:25:54 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Shakeel Butt Date: Mon, 5 Dec 2022 09:25:43 -0800 Message-ID: Subject: Re: [Regression] mqueue performance degradation after "The new cgroup slab memory controller" patchset. To: Waiman Long Cc: "Luther, Sven" , "linux-kernel@vger.kernel.org" , "regressions@lists.linux.dev" , Roman Gushchin , Andrew Morton , Christoph Lameter , Johannes Weiner , Michal Hocko , "linux-mm@kvack.org" , Vlastimil Babka , "kernel-team@fb.com" , "Eric W. Biederman" , Muchun Song , Alexey Gladkov , "Bonn, Jonas" Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Spamd-Result: default: False [3.10 / 9.00]; SORBS_IRL_BL(3.00)[209.85.219.173:from]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; BAYES_HAM(-0.00)[35.21%]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[16]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; DKIM_TRACE(0.00)[google.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; FROM_HAS_DN(0.00)[]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; TO_DN_EQ_ADDR_SOME(0.00)[] X-Rspamd-Queue-Id: 97BFDA000A X-Rspamd-Server: rspam01 X-Stat-Signature: 3xyby9i8p6nuq9e1og4bo6tx3ydw9wqk X-HE-Tag: 1670261155-304645 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 5, 2022 at 8:31 AM Waiman Long wrote: > > On 12/5/22 11:06, Shakeel Butt wrote: > > Hi Sven, > > > > On Mon, Dec 5, 2022 at 6:56 AM Luther, Sven wrote: > >> #regzbot ^introduced 10befea91b61c4e2c2d1df06a2e978d182fcf792 > >> > >> We are making heavy use of mqueues, and noticed a degradation of performance between 4.18 & 5.10 linux kernels. > >> > >> After a gross per-version tracing, we did kernel bisection between 5.8 and 5.9 > >> and traced the issue to a 10 patches (of which 9 where skipped as they didn't boot) between: > >> > >> > >> commit 10befea91b61c4e2c2d1df06a2e978d182fcf792 (HEAD, refs/bisect/bad) > >> Author: Roman Gushchin > >> Date: Thu Aug 6 23:21:27 2020 -0700 > >> > >> mm: memcg/slab: use a single set of kmem_caches for all allocations > >> > >> and: > >> > >> commit 286e04b8ed7a04279ae277f0f024430246ea5eec (refs/bisect/good-286e04b8ed7a04279ae277f0f024430246ea5eec) > >> Author: Roman Gushchin > >> Date: Thu Aug 6 23:20:52 2020 -0700 > >> > >> mm: memcg/slab: allocate obj_cgroups for non-root slab pages > >> > >> All of them are part of the "The new cgroup slab memory controller" patchset: > >> > >> https://lore.kernel.org/all/20200623174037.3951353-18-guro@fb.com/T/ > >> > >> from Roman Gushchin, which moves the accounting for page level to the object level. > >> > >> Measurements where done using the a test programmtest, which measures mix/average/max time mqueue_send/mqueue_rcv, > >> and average for getppid, both measured over 100 000 runs. Results are shown in the following table > >> > >> +----------+--------------------------+-------------------------+----------------+ > >> | kernel | mqueue_rcv (ns) | mqueue_send (ns) | getppid | > >> | version | min avg max variation | min avg max variation | (ns) variation | > >> +----------+--------------------------+-------------------------+----------------+ > >> | 4.18.45 | 351 382 17533 base | 383 410 13178 base | 149 base | > >> | 5.8-good | 380 392 7156 -2,55% | 376 384 6225 6,77% | 169 -11,83% | > >> | 5.8-bad | 524 530 5310 -27,92% | 512 519 8775 -21,00% | 169 -11,83% | > >> | 5.10 | 520 533 4078 -28,33% | 518 534 8108 -23,22% | 167 -10,78% | > >> | 5.15 | 431 444 8440 -13,96% | 425 437 6170 -6,18% | 171 -12,87% | > >> | 6.03 | 474 614 3881 -37,79% | 482 693 931 -40,84% | 171 -12,87% | > >> +----------+--------------------------+-------------------------+----------------- > >> > > Is the last kernel 6.0.3? Also we know there is performance impact of > > per-object kmem accounting. Can you try the latest i.e. 6.1-rc8? There > > are a couple of memcg charging optimization patches merged in this > > window. > > It is known that per-object kmem accounting regresses performance. I had > submitted a number of optimization patches that got merged into v5.14. > So the regression is reduced in the 5.15 line above. It looks like there > are some additional regressions in the latest kernel. We will need to > figure out what causes it. > In 5.18 PREEMPT_RT patches went in and 559271146efc ("mm/memcg: optimize user context object stock access") got reverted.