From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 238ECC4332F for ; Mon, 5 Dec 2022 20:08:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C0E8E0002; Mon, 5 Dec 2022 15:08:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CB308E0001; Mon, 5 Dec 2022 15:08:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56ED98E0002; Mon, 5 Dec 2022 15:08:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4821A8E0001 for ; Mon, 5 Dec 2022 15:08:35 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 16B4812027F for ; Mon, 5 Dec 2022 20:08:35 +0000 (UTC) X-FDA: 80209340190.07.5BB5AC1 Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) by imf03.hostedemail.com (Postfix) with ESMTP id 1BB7A20017 for ; Mon, 5 Dec 2022 20:08:32 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GiQWgTJ6; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670270913; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RM7CfezPPGUav4Lu0t/qwJdsYnLC3YJVlzAQP3YTuPI=; b=wA6OoA30xYrcB1rrv8SAuVngiDTY5Yg+A3gPiOnHtEWga4QfNjxjIyTE2mo6+DWMY/R0pb 90+W142r0heRaMR+WY/x2FDYdYDkrbg9ZpqDmaRjP9QZ4Ir6XPwsehhXDfQNkhx1crTpM8 4TIpp1N7xILebAW5gc/DUKsQ2GqrOH8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GiQWgTJ6; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670270913; a=rsa-sha256; cv=none; b=u2pdQjFh874Dz1tpbHunf90vhXjxf2uIs7a9+VVEF4m+QndPxj7Y3qFDROqS4fyZgZmGxs biMGoOi3le0/VG7Yw4+N75Bgg+WLcW+/7Pw8LCe1cnbNtTKtdu3awEvvmdixMJogoGr4Vv +ZqrKypYaWjgGq4wI6yEEWgIaZDgHGk= Date: Mon, 5 Dec 2022 12:08:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1670270910; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RM7CfezPPGUav4Lu0t/qwJdsYnLC3YJVlzAQP3YTuPI=; b=GiQWgTJ6UTtJOmb5w3cPc9P2E7cWItMRPqs8Se1Z6wFfmT+yiKILuinJ1nWYlLc9UCuonp 1rquDwpn88+qs6w3Qt79G2pnr9tNFJm/uC31fX+nnJMTXpA5RkxtLTg3iISSilDgk9Sxo+ 8hR4g/SU4Ba738nIjpy4oa7S/9CpuS8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: "Luther, Sven" Cc: "linux-kernel@vger.kernel.org" , "regressions@lists.linux.dev" , Roman Gushchin , Andrew Morton , Christoph Lameter , Johannes Weiner , Michal Hocko , Shakeel Butt , "linux-mm@kvack.org" , Vlastimil Babka , "kernel-team@fb.com" , "Eric W. Biederman" , Muchun Song , Waiman Long , Alexey Gladkov , "Bonn, Jonas" Subject: Re: [Regression] mqueue performance degradation after "The new cgroup slab memory controller" patchset. Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1BB7A20017 X-Stat-Signature: jh781mja76zn7u86tm7wob3c3xrb9wdr X-Spamd-Result: default: False [-0.50 / 9.00]; DMARC_POLICY_ALLOW(-0.50)[linux.dev,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:91.218.175.0/24]; R_DKIM_ALLOW(-0.20)[linux.dev:s=key1]; MIME_GOOD(-0.10)[text/plain]; BAYES_HAM(-0.00)[39.30%]; RCPT_COUNT_TWELVE(0.00)[17]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[linux.dev:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[] X-HE-Tag: 1670270912-224078 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 05, 2022 at 02:55:48PM +0000, Luther, Sven wrote: > #regzbot ^introduced 10befea91b61c4e2c2d1df06a2e978d182fcf792 > > We are making heavy use of mqueues, and noticed a degradation of performance between 4.18 & 5.10 linux kernels. > > After a gross per-version tracing, we did kernel bisection between 5.8 and 5.9 > and traced the issue to a 10 patches (of which 9 where skipped as they didn't boot) between: > > > commit 10befea91b61c4e2c2d1df06a2e978d182fcf792 (HEAD, refs/bisect/bad) > Author: Roman Gushchin > Date: Thu Aug 6 23:21:27 2020 -0700 > > mm: memcg/slab: use a single set of kmem_caches for all allocations > > and: > > commit 286e04b8ed7a04279ae277f0f024430246ea5eec (refs/bisect/good-286e04b8ed7a04279ae277f0f024430246ea5eec) > Author: Roman Gushchin > Date: Thu Aug 6 23:20:52 2020 -0700 > > mm: memcg/slab: allocate obj_cgroups for non-root slab pages > > All of them are part of the "The new cgroup slab memory controller" patchset: > > https://lore.kernel.org/all/20200623174037.3951353-18-guro@fb.com/T/ > > from Roman Gushchin, which moves the accounting for page level to the object level. > > Measurements where done using the a test programmtest, which measures mix/average/max time mqueue_send/mqueue_rcv, > and average for getppid, both measured over 100 000 runs. Results are shown in the following table > > +----------+--------------------------+-------------------------+----------------+ > | kernel | mqueue_rcv (ns) | mqueue_send (ns) | getppid | > | version | min avg max variation | min avg max variation | (ns) variation | > +----------+--------------------------+-------------------------+----------------+ > | 4.18.45 | 351 382 17533 base | 383 410 13178 base | 149 base | > | 5.8-good | 380 392 7156 -2,55% | 376 384 6225 6,77% | 169 -11,83% | > | 5.8-bad | 524 530 5310 -27,92% | 512 519 8775 -21,00% | 169 -11,83% | > | 5.10 | 520 533 4078 -28,33% | 518 534 8108 -23,22% | 167 -10,78% | > | 5.15 | 431 444 8440 -13,96% | 425 437 6170 -6,18% | 171 -12,87% | > | 6.03 | 474 614 3881 -37,79% | 482 693 931 -40,84% | 171 -12,87% | > +----------+--------------------------+-------------------------+----------------- Hi Sven! Thank you for the report! As Waiman said, it's not a secret that per-object tracking makes individual allocations slower, but for the majority of workloads it's well compensated by significant memory savings and a lower fragmentation. It seems there is another regression between 5.15 and 6.03, which is a separate topic, but how big is the real regression between 4.18 and 5.15? The benchmark shows about 14%, but is you real workload suffering at the same level? If the answer is yes, the right thing to do is to introduce some sort of mqueue-specific caching for allocated objects. Thanks! Roman