From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06FABC4332F for ; Mon, 5 Dec 2022 16:31:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 524CE8E0002; Mon, 5 Dec 2022 11:31:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D5558E0001; Mon, 5 Dec 2022 11:31:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39C758E0002; Mon, 5 Dec 2022 11:31:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2B1058E0001 for ; Mon, 5 Dec 2022 11:31:02 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E768D1A070E for ; Mon, 5 Dec 2022 16:31:01 +0000 (UTC) X-FDA: 80208791922.21.9ECB1F9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id AFFFD140015 for ; Mon, 5 Dec 2022 16:31:00 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EdZOwcBD; spf=pass (imf26.hostedemail.com: domain of longman@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670257861; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OHhzQw/CjxDjwlNJmwDQzqyO8fri5N61JkSmsbVVt1o=; b=HDlKPm8Atx9tr7MnrL6vDa0ApKCAf8EByAWzrGS6wwACLm5vNJTVrMy4bVJuKDNLlDBNjM 46fmehI3/4b3MxGlgQJgzR8nrXRjfF4vJCQaQgyqmLnM+QKFe6n5VV+RIhD9QIS7AdqLrT VfIwbbbY6Nxet3oaj2vU2GCgAEW8u3M= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EdZOwcBD; spf=pass (imf26.hostedemail.com: domain of longman@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670257861; a=rsa-sha256; cv=none; b=4gPLZfQe1PnQ2apkrMOj1j7dmWUdgBZoX3gT32XVLo4vyLs/KY+Dzk4G068vj6Hz/FNZEh KmtqA9ZWxs1EiwqF2Cgb8FNOKL47pTdx2xmTihdQsdHZwR+h0fKOXxVNBqNYLJI71VMvsb oS1ycZQL2ELEn7q8pwAuS+YDKTY68ok= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670257860; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OHhzQw/CjxDjwlNJmwDQzqyO8fri5N61JkSmsbVVt1o=; b=EdZOwcBD25p55ELNPgf9LhpdhTGme8gNCtXLmx2mu2OoEP2QtVscYPXlbefeKUuiyAwgvo chJ9IBiLWCAvFhtKsGqiyVAZF3s9Mep6JMlbx4nyWNCpkbSS0O6OQtKbC0ExBlG6ZV2orY m9hyFQrbw2KHMHbd41posw5CqdsxoZ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-73--vGpsqClNt6fYfui6UQkLA-1; Mon, 05 Dec 2022 11:30:56 -0500 X-MC-Unique: -vGpsqClNt6fYfui6UQkLA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BA47D800B23; Mon, 5 Dec 2022 16:30:55 +0000 (UTC) Received: from [10.22.9.203] (unknown [10.22.9.203]) by smtp.corp.redhat.com (Postfix) with ESMTP id AACF52028E96; Mon, 5 Dec 2022 16:30:54 +0000 (UTC) Message-ID: Date: Mon, 5 Dec 2022 11:30:52 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: [Regression] mqueue performance degradation after "The new cgroup slab memory controller" patchset. Content-Language: en-US To: Shakeel Butt , "Luther, Sven" Cc: "linux-kernel@vger.kernel.org" , "regressions@lists.linux.dev" , Roman Gushchin , Andrew Morton , Christoph Lameter , Johannes Weiner , Michal Hocko , "linux-mm@kvack.org" , Vlastimil Babka , "kernel-team@fb.com" , "Eric W. Biederman" , Muchun Song , Alexey Gladkov , "Bonn, Jonas" References: From: Waiman Long In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: 9u7ymap3yu3anp1tfkepc7u3yjqcxwkq X-Rspam-User: X-Spamd-Result: default: False [-0.90 / 9.00]; DMARC_POLICY_ALLOW(-0.50)[redhat.com,none]; R_SPF_ALLOW(-0.20)[+ip4:170.10.129.0/24]; R_DKIM_ALLOW(-0.20)[redhat.com:s=mimecast20190719]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; BAYES_HAM(-0.00)[40.81%]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWELVE(0.00)[16]; DKIM_TRACE(0.00)[redhat.com:+]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[] X-Rspamd-Queue-Id: AFFFD140015 X-Rspamd-Server: rspam06 X-HE-Tag: 1670257860-14250 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 12/5/22 11:06, Shakeel Butt wrote: > Hi Sven, > > On Mon, Dec 5, 2022 at 6:56 AM Luther, Sven wrote: >> #regzbot ^introduced 10befea91b61c4e2c2d1df06a2e978d182fcf792 >> >> We are making heavy use of mqueues, and noticed a degradation of performance between 4.18 & 5.10 linux kernels. >> >> After a gross per-version tracing, we did kernel bisection between 5.8 and 5.9 >> and traced the issue to a 10 patches (of which 9 where skipped as they didn't boot) between: >> >> >> commit 10befea91b61c4e2c2d1df06a2e978d182fcf792 (HEAD, refs/bisect/bad) >> Author: Roman Gushchin >> Date: Thu Aug 6 23:21:27 2020 -0700 >> >> mm: memcg/slab: use a single set of kmem_caches for all allocations >> >> and: >> >> commit 286e04b8ed7a04279ae277f0f024430246ea5eec (refs/bisect/good-286e04b8ed7a04279ae277f0f024430246ea5eec) >> Author: Roman Gushchin >> Date: Thu Aug 6 23:20:52 2020 -0700 >> >> mm: memcg/slab: allocate obj_cgroups for non-root slab pages >> >> All of them are part of the "The new cgroup slab memory controller" patchset: >> >> https://lore.kernel.org/all/20200623174037.3951353-18-guro@fb.com/T/ >> >> from Roman Gushchin, which moves the accounting for page level to the object level. >> >> Measurements where done using the a test programmtest, which measures mix/average/max time mqueue_send/mqueue_rcv, >> and average for getppid, both measured over 100 000 runs. Results are shown in the following table >> >> +----------+--------------------------+-------------------------+----------------+ >> | kernel | mqueue_rcv (ns) | mqueue_send (ns) | getppid | >> | version | min avg max variation | min avg max variation | (ns) variation | >> +----------+--------------------------+-------------------------+----------------+ >> | 4.18.45 | 351 382 17533 base | 383 410 13178 base | 149 base | >> | 5.8-good | 380 392 7156 -2,55% | 376 384 6225 6,77% | 169 -11,83% | >> | 5.8-bad | 524 530 5310 -27,92% | 512 519 8775 -21,00% | 169 -11,83% | >> | 5.10 | 520 533 4078 -28,33% | 518 534 8108 -23,22% | 167 -10,78% | >> | 5.15 | 431 444 8440 -13,96% | 425 437 6170 -6,18% | 171 -12,87% | >> | 6.03 | 474 614 3881 -37,79% | 482 693 931 -40,84% | 171 -12,87% | >> +----------+--------------------------+-------------------------+----------------- >> > Is the last kernel 6.0.3? Also we know there is performance impact of > per-object kmem accounting. Can you try the latest i.e. 6.1-rc8? There > are a couple of memcg charging optimization patches merged in this > window. It is known that per-object kmem accounting regresses performance. I had submitted a number of optimization patches that got merged into v5.14. So the regression is reduced in the 5.15 line above. It looks like there are some additional regressions in the latest kernel. We will need to figure out what causes it. Cheers, Longman