From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C061C4321E for ; Mon, 5 Dec 2022 14:56:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFF408E0002; Mon, 5 Dec 2022 09:56:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD5DD8E0001; Mon, 5 Dec 2022 09:56:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A01418E0002; Mon, 5 Dec 2022 09:56:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8E1FD8E0001 for ; Mon, 5 Dec 2022 09:56:17 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 426A2AAD2D for ; Mon, 5 Dec 2022 14:56:17 +0000 (UTC) X-FDA: 80208553194.12.0A5160F Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by imf05.hostedemail.com (Postfix) with ESMTP id 285F0100017 for ; Mon, 5 Dec 2022 14:56:13 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=windriver.com header.s=PPS06212021 header.b=XJcXyjNf; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of "prvs=93383f03cf=sven.luther@windriver.com" designates 205.220.166.238 as permitted sender) smtp.mailfrom="prvs=93383f03cf=sven.luther@windriver.com"; dmarc=pass (policy=none) header.from=windriver.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1670252176; a=rsa-sha256; cv=pass; b=bvszSz/skcIm8inlNF3VXxD8Ki32iNbfQB3E13ijxuf/TlqJtecu6hk4Ba7pr4ZKvbcevw 8o5nfxdHPjsjP9SSJQVCHCl4oUCUBwWp3jO3AGXzhl00fLNqynWfNRza8FjnOJQl3XZH3J H8gzgUdgPt0uxYu46dlUPOIWb+7Ej6Q= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=windriver.com header.s=PPS06212021 header.b=XJcXyjNf; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of "prvs=93383f03cf=sven.luther@windriver.com" designates 205.220.166.238 as permitted sender) smtp.mailfrom="prvs=93383f03cf=sven.luther@windriver.com"; dmarc=pass (policy=none) header.from=windriver.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670252176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=hx6lmKf5fsRfVUH/ZnR3vRj5X/W/P1VIKBUM5zK77lc=; b=F1YYDDFOrUxxbp8nb1VZyQVDuHy2R5CnWH10C4A1tXWbBvFTPlzJ+QjnwnpxL1aW55H9Ep SvzA+k7Vmz+HPiwBp8yhHmNQKgJ3VMSibqeaqDGIzkVCiBaMoe93f/t9AH9QLiV5e8akpz KUYNP2/Kju1ggYXSuPTRPztH6q2ZkMo= Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5Dkg00013811; Mon, 5 Dec 2022 06:55:56 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : cc : subject : date : message-id : content-type : mime-version; s=PPS06212021; bh=hx6lmKf5fsRfVUH/ZnR3vRj5X/W/P1VIKBUM5zK77lc=; b=XJcXyjNfqrXAiXiFORmieliM2AqYPFiwGleD1KynN/LHpmFsJ1mQeGwnoTTJplTq8Rg0 +1HVrDs+pxbIVxNmxKvt0jLkFycrh+WsVH/HrfxoNtq2p0WO82vEJnoghEluc03hkEOa wnLW37I1CKofm7GrnUbinN9NW8WvSwoNUmcU++rO2r9c4xlTv9wbd15qHNf3HLpt/Cyg tjO/0iO36UYwoePMMyxkhMsFGBFEqVFHRITIO8/5onao7XChUYSmWHqV3uC5XfQ43Xz/ 2Xwa85ARRXssc/6UG6VTIlAnOZ/Kf6sTCtyk3NSmqaSqZNzYxf+fZP9tMslskLswXnsy Vw== Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3m82m6se5j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 05 Dec 2022 06:55:56 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VN5/WQx9maDMdzrEKRuEePNl2RRLdJiL9K+5jB9qHXsskM/rWkWp+Z8bNiMEd5drAoNLSIcwhGWrB99K/QBjWbsujT9ITEJdQWKVtFbrjv6gSm7m4oxpgWECBPw8OhrDzMzdoXR25wSrkAFPDF2fsbnQ11Prpz4hIKNBLBE6cxK4UOFxQlX2XAeY8GfKsTOLI1spZeRdtEScFaqb4WxAFnTa34a+8FGE1+kNKvBtgqAlCT4u7Bg7nnrK47JeFW9rhJy2SHq1XUQaAqXQK6VrpxdOplC/Eqgk5O7Aegf430lpVbk8ySPM1unCU7K+oHRL5tPIYPquj8fNNy7kO2K6Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hx6lmKf5fsRfVUH/ZnR3vRj5X/W/P1VIKBUM5zK77lc=; b=H/BPjKOFXiwFU/GfgY8A65zDUPNNK4Rre3wN/tuecdMfexfoYRWiJwOBG4kAv9uLEm0dratHzrKWpXmBlPfKc+lgyiBPKz0wFizISUkr9IoNBD8ZElsRm4v1T38qXumtgdGv+RWjnQ21DQhGfMf43YwecYJWxVInJVUfil19rmnyQc3sEfDfwU6ofyNhPEFE+1pJVR6HuKmxTzFrIEeZTH21/XviJp42syiPuBdiH3hTuSLEEt/aIoStW6f4UoIsFBO3cPTWfev7cb8qEUJBRXWjx0Jh0bDvJVVyGamuSmi/samUDUHKP8w/eKQK4DYT5mRU/sO/vt/Ox4JiOWzC9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from PH0PR11MB5626.namprd11.prod.outlook.com (2603:10b6:510:ee::15) by CY8PR11MB6914.namprd11.prod.outlook.com (2603:10b6:930:5a::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.14; Mon, 5 Dec 2022 14:55:48 +0000 Received: from PH0PR11MB5626.namprd11.prod.outlook.com ([fe80::f05a:5750:9d7b:7f92]) by PH0PR11MB5626.namprd11.prod.outlook.com ([fe80::f05a:5750:9d7b:7f92%3]) with mapi id 15.20.5880.014; Mon, 5 Dec 2022 14:55:48 +0000 From: "Luther, Sven" To: "linux-kernel@vger.kernel.org" , "regressions@lists.linux.dev" , Roman Gushchin CC: Andrew Morton , Christoph Lameter , Johannes Weiner , Michal Hocko , Shakeel Butt , "linux-mm@kvack.org" , Vlastimil Babka , "kernel-team@fb.com" , "Eric W. Biederman" , Muchun Song , Waiman Long , Alexey Gladkov , "Bonn, Jonas" Subject: [Regression] mqueue performance degradation after "The new cgroup slab memory controller" patchset. Thread-Topic: [Regression] mqueue performance degradation after "The new cgroup slab memory controller" patchset. Thread-Index: AQHZCLlPJRgR3N+NBE2RcqkRsyxPQQ== Date: Mon, 5 Dec 2022 14:55:48 +0000 Message-ID: Accept-Language: fr-FR, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR11MB5626:EE_|CY8PR11MB6914:EE_ x-ms-office365-filtering-correlation-id: 73f5addc-1527-47b8-ad3c-08dad6d0cb6e x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9JqWqzV8OamKo+AYknCuPD8v6Z2zoXfuz8hJ7SaZGkQmVJkmMUWQ0IDhYkBSVP8GKTK3T+G2dKjAf39O9uCBrOqjuzYD1zWaThKr4Ju81X0EzaxTI7yzCTV5x+yGJaOz/XVAa++mkO1MFBLvRetJKEnw2BzMuK8RgkDMI8sulLqRE94dPb15n7NjDhk9qPdwEnJjf1WpN4BUcEGnEvg6eG9iabVoAdpsuDH+IUN5yfmfQN20n1mM9QnY5tvqIMf1MU0MrsPZ9YmN/m1jpFqL3UNQ/QpiwfvaDxqejxVB77WMlaWCACMJTZI73le3QLk4O0TYN7mG2v/WkJWuzlY0ALKh0EIA6LVno+zYLIaGPDqTBLCw/ePu3IgHoYrlm4bPgnI67Swm7EUgcB/sZeqXxmEeZc1m8+SrhNcJ1sgiS7k6kgjLsyM5cVDyzCQopoMcEd6d94AhkkaqdIenrpwyn9A0gPnPNciySA6keLysjs4Xe4fXNrGsANCa2D7zh3SAld417Zs1uIR4DQE6Y5+Mi1ht6AW5bNlynX3s5rs+L5MvfN8gNnrOTWzrSZnvm7bKuz39aH6atVMKHnh9RV0vlmLRPycJSIscH3abo9pn/4kup03gMtTYMlJhu9Ep6VcJnWRupKKO9p+vavKAWxoE6w/tjz/H4sp/ekxVBkH7bB8KUix0uenyWg27oTWlsYeIT3xbD8wiAxcTLOkjerKeKFhxPe1rNmpRcXlCBijl9gQ= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR11MB5626.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(136003)(376002)(366004)(39850400004)(346002)(451199015)(19627405001)(33656002)(86362001)(6506007)(7696005)(478600001)(71200400001)(966005)(38070700005)(55016003)(9686003)(83380400001)(186003)(122000001)(38100700002)(40140700001)(8676002)(4326008)(5660300002)(41300700001)(7416002)(107886003)(52536014)(66946007)(91956017)(66476007)(66556008)(64756008)(76116006)(316002)(8936002)(54906003)(66446008)(2906002)(110136005);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?p/Gg6ihXO2fdKO9/3lMQZlI2A7kvigIsACD5zATguhJP5UZRbDdvSnYw2M?= =?iso-8859-1?Q?peOuGiToz8MT0gXNKmSuHPck6lSMuULt2VKFQcPFzViVoQL32RLbgni4Bx?= =?iso-8859-1?Q?+/mgd/VLtdkIngq9WhkIX+uMxTJ5tLt0cTdVH1XRXSy7Bf+jlmaLtW33uz?= =?iso-8859-1?Q?HLVSQkVg7BFv1YXQdQ24jRtZVN36VFir8gItK0sWBRm4/zF9kPfGM5MwxG?= =?iso-8859-1?Q?7xP6KuAsiMNp0lN+Ftu/ierq+m2ujCnm0GTowdw4AlVSUt9/xXqYkTZq0u?= =?iso-8859-1?Q?GV2+OfrRHQkexZGvXEIeRlllW6nBKAYjFhDFK8rLkQgwd6nVGII/KTk+Bo?= =?iso-8859-1?Q?qJmTKyUQSHk/I7yeT/qEuW4SudfMPmlGfrujEOALVpzOg7XnwaoTQ0PPZ/?= =?iso-8859-1?Q?iYqnbpdRD+l1+EZfX3us2Bht70GBrJny2azrrZxWGPRkgudQLBMFqqZJ+E?= =?iso-8859-1?Q?e0+Etj1+UYNygn88XNds1CAC0tdt2dgqWNK6wNKocpIOFyKXWJA7kjNOd6?= =?iso-8859-1?Q?6TI/lGookysgNqwDQUgo2PYkfVTU9yw/W602X7D34vL7O5Uc21uv/CNAfR?= =?iso-8859-1?Q?9Q5ywDUFwTzpRrtETIiHc9m3Gtq2iBAL/nVhx+Ty2CjH/b2M6F9+MOvLb6?= =?iso-8859-1?Q?JE0lGaHyoBRARUxPzGyBR2EoQ5JZUk2/mShJ+zdDMfrnOU8t5KihYYKrUz?= =?iso-8859-1?Q?Otk/GTl/KUxOjhB91zNqvTTpbYPbOLSEsJvA3l955sPGKW3n+K8uooLwaj?= =?iso-8859-1?Q?u2LGxPeeIezAUbHQNtTChhJqr9AfngzrRAcnGoikYZWa4hl3iihoJk7WF8?= =?iso-8859-1?Q?bFiISIie/SDCAqy4VJOIGAiuSORZGBvpUmr2Tv1EmPQGea8268aFywxecA?= =?iso-8859-1?Q?ZOHMyTQlfJ3DSlMWk1/xAaxouc/e4SHAy9KujOVnoiJH5qkbskrBtP9bq2?= =?iso-8859-1?Q?pkllM9nHqf4pbr3drep+rvTVISQCRoZAv9Hhk8dU7ogVy6JoqrbkpV+w8l?= =?iso-8859-1?Q?Lmc/jQxylEPtKRdOeyprE2c3B+tVqAUqogrwBj0/ci2LAgoGnJnlsBZyy+?= =?iso-8859-1?Q?4iirQWKM28KJyHhvBzPBSr1bRgv3ONtgvWenuAIdnq7gAnBW/H6KuAeri9?= =?iso-8859-1?Q?jCc3htgSOGYZ9sxLGwCX9+TqA4gXSdEA1O1TuDwRuJag1lvlS1cLqUqEbd?= =?iso-8859-1?Q?L6Nv4sdAo9jQiP0q4Bgd0sy2NhX0LxPBCAwMxC4QzmSr4i9ohy8VtsSvS7?= =?iso-8859-1?Q?KznXev2jYHCZkoeyzmr20EE8Cz5PElajxYOK3KLZuzycmdku4mKdojnmdX?= =?iso-8859-1?Q?GDCeoAyf3/XD+YAiXLhCLDGkm8NOqhZNNvvhLnMPZlopkodcgcK8Ds9zCs?= =?iso-8859-1?Q?+fehuKbUM7fTT4TJBIh201GTudTv9DrLx0PB+OHtqRvhf4YoU8yOdIypYc?= =?iso-8859-1?Q?UHs2Tbll47Pu5+pfkRPG8yzHVx+o1qBJxg2LeVkthM3+JBhIA/cWQ+GayA?= =?iso-8859-1?Q?u81nFhoywt4I4uk6MYtDYAoC2j6AJhAAk5uJNrx2jRAnviVL0OLLOxCyeg?= =?iso-8859-1?Q?tjuZ+KRIs0lmfEI66BjQcKT8rj7MBT4XSRQmLVZBMmF9jk9F43V+EWxJ48?= =?iso-8859-1?Q?7qLHpGkWJUD80dgH+4cF/4w/1Kivz/xcUWi+is69x6z2FS/QlK3zJErDlS?= =?iso-8859-1?Q?y99GhXbxaCfiehixt/241fJL2roBLoCcqCeglPv8?= Content-Type: multipart/alternative; boundary="_000_PH0PR11MB562641BC03630B4B7A227FD7E9189PH0PR11MB5626namp_" MIME-Version: 1.0 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5626.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 73f5addc-1527-47b8-ad3c-08dad6d0cb6e X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Dec 2022 14:55:48.0603 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: i6JEPoJgqYUb01n48GhLWEnaNN2LQrW2vVa4WAdIMb3uqZgaSTUAMxY2XO430O3dVl5QldMmTqM6Tn1/xXAHG2OPgL56ads29zAK8alCh5s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB6914 X-Proofpoint-ORIG-GUID: ZbZCOfUKP3lCeFZFg8b9pz47hH1QK1X- X-Proofpoint-GUID: ZbZCOfUKP3lCeFZFg8b9pz47hH1QK1X- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 impostorscore=0 suspectscore=0 adultscore=0 phishscore=0 clxscore=1011 malwarescore=0 priorityscore=1501 mlxscore=0 bulkscore=0 mlxlogscore=993 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2212050123 X-Spamd-Result: default: False [-7.90 / 9.00]; BAYES_HAM(-6.00)[100.00%]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; DMARC_POLICY_ALLOW(-0.50)[windriver.com,none]; R_SPF_ALLOW(-0.20)[+ip4:205.220.166.238]; R_DKIM_ALLOW(-0.20)[windriver.com:s=PPS06212021]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_NO_TLS_LAST(0.10)[]; RCPT_COUNT_TWELVE(0.00)[16]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TC_DOMAIN_MIX_CASE_VALID(0.00)[]; DKIM_TRACE(0.00)[windriver.com:+]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[Sven.Luther@windriver.com,prvs=93383f03cf=sven.luther@windriver.com]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FORGED_SENDER_VERP_SRS(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=2]; TC_DOMAIN_MIX_CASE(0.00)[] X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 285F0100017 X-Stat-Signature: x7gswo8iq11u9txmrp6neokf6nenb8ye X-HE-Tag: 1670252173-147897 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --_000_PH0PR11MB562641BC03630B4B7A227FD7E9189PH0PR11MB5626namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable #regzbot ^introduced 10befea91b61c4e2c2d1df06a2e978d182fcf792 We are making heavy use of mqueues, and noticed a degradation of performanc= e between 4.18 & 5.10 linux kernels. After a gross per-version tracing, we did kernel bisection between 5.8 and = 5.9 and traced the issue to a 10 patches (of which 9 where skipped as they didn= 't boot) between: commit 10befea91b61c4e2c2d1df06a2e978d182fcf792 (HEAD, refs/bisect/bad) Author: Roman Gushchin Date: Thu Aug 6 23:21:27 2020 -0700 mm: memcg/slab: use a single set of kmem_caches for all allocations and: commit 286e04b8ed7a04279ae277f0f024430246ea5eec (refs/bisect/good-286e04b8e= d7a04279ae277f0f024430246ea5eec) Author: Roman Gushchin Date: Thu Aug 6 23:20:52 2020 -0700 mm: memcg/slab: allocate obj_cgroups for non-root slab pages All of them are part of the "The new cgroup slab memory controller" patchse= t: https://lore.kernel.org/all/20200623174037.3951353-18-guro@fb.com/T/ from Roman Gushchin, which moves the accounting for page level to the objec= t level. Measurements where done using the a test programmtest, which measures mix/a= verage/max time mqueue_send/mqueue_rcv, and average for getppid, both measured over 100 000 runs. Results are shown= in the following table +----------+--------------------------+-------------------------+----------= ------+ | kernel | mqueue_rcv (ns) | mqueue_send (ns) | getppi= d | | version | min avg max variation | min avg max variation | (ns) vari= ation | +----------+--------------------------+-------------------------+----------= ------+ | 4.18.45 | 351 382 17533 base | 383 410 13178 base | 149 = base | | 5.8-good | 380 392 7156 -2,55% | 376 384 6225 6,77% | 169 -11= ,83% | | 5.8-bad | 524 530 5310 -27,92% | 512 519 8775 -21,00% | 169 -11= ,83% | | 5.10 | 520 533 4078 -28,33% | 518 534 8108 -23,22% | 167 -10= ,78% | | 5.15 | 431 444 8440 -13,96% | 425 437 6170 -6,18% | 171 -12= ,87% | | 6.03 | 474 614 3881 -37,79% | 482 693 931 -40,84% | 171 -12= ,87% | +----------+--------------------------+-------------------------+----------= ------- Below you will find the following appendices: - raw GOOD bisect results - raw BAD bisect results - test code source code - bisect results Friendly, Sven Luther GOOD bisect results: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # uname -a Linux axxiax86-64 5.8.0-5-286e04b8ed7a04279ae277f0f024430246ea5eec-good #1 = SMP PREEMPT Fri Jul 29 11:01:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # ./mqtest RX: 380/392/7156 TX: 376/384/6225 Accumulated time for 100000 getppid, 16924653ns Average time per getppid: 169ns # ./mqtest RX: 383/397/6300 TX: 377/385/8258 Accumulated time for 100000 getppid, 16915580ns Average time per getppid: 169ns # ./mqtest RX: 385/397/7690 TX: 377/385/4244 Accumulated time for 100000 getppid, 16922910ns Average time per getppid: 169ns BAD bisect results: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # uname -a Linux axxiax86-64 5.8.0-5-10befea91b61c4e2c2d1df06a2e978d182fcf792-bad #1 S= MP PREEMPT Fri Jul 29 11:01:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # ./mqtest RX: 524/530/5310 TX: 512/519/8775 Accumulated time for 100000 getppid, 16921865ns Average time per getppid: 169ns # ./mqtest RX: 520/525/8204 TX: 514/522/7420 Accumulated time for 100000 getppid, 16931493ns Average time per getppid: 169ns # ./mqtest RX: 520/525/8128 TX: 512/519/5340 Accumulated time for 100000 getppid, 16921535ns Average time per getppid: 169ns Test programm: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #include #include #include #include #include #include #include #include #include struct sysc_meas { int64_t min; int64_t max; int64_t sum; int64_t count; }; static inline uint64_t __cputime_get(void) { unsigned long l, h; asm volatile("rdtsc" : "=3Da" (l), "=3Dd" (h)); return ((uint64_t) h << 32) | l; } static inline int64_t __cputime_ns(uint64_t cputime) { /* Note, it is assumed thatTSC clock is 2.2G */ return (cputime * 5) / 11; } static inline uint64_t __cputime_get_ns() { return __cputime_ns(__cputime_get()); } static void sysc_meas_init(struct sysc_meas *sm) { sm->min =3D INT64_MAX; sm->max =3D INT64_MIN; sm->sum =3D sm->count =3D 0; } static void sysc_meas_add(struct sysc_meas *sm, int64_t value) { if (value > sm->max) sm->max =3D value; if (value < sm->min) sm->min =3D value; sm->sum +=3D value; sm->count++; } static void sysc_mq(uint32_t count) { int fd; struct mq_attr mq_attr; uint64_t ts; struct sysc_meas sm_tx, sm_rx; char dummy[256]; if (!count) return; mq_unlink("/sysc_mq"); memset(&mq_attr, 0, sizeof(mq_attr)); mq_attr.mq_maxmsg =3D 10; mq_attr.mq_msgsize =3D sizeof(dummy); fd =3D mq_open("/sysc_mq", O_RDWR | O_NONBLOCK | O_CREAT, 0666, &mq= _attr); if (fd < 0) { printf("ERROR: unable to create MQ, %d\n", -errno); exit(-1); } sysc_meas_init(&sm_rx); sysc_meas_init(&sm_tx); dummy[0] =3D 0; dummy[sizeof(dummy) - 1] =3D 0; while (count--) { ts =3D __cputime_get(); mq_send(fd, dummy, sizeof(dummy), 0); sysc_meas_add(&sm_tx, __cputime_get() - ts); ts =3D __cputime_get(); mq_receive(fd, dummy, sizeof(dummy), NULL); sysc_meas_add(&sm_rx, __cputime_get() - ts); } mq_close(fd); mq_unlink("/sysc_mq"); printf("RX: %ld/%ld/%ld\n", __cputime_ns(sm_rx.min), __cputime_ns(sm_rx.sum / sm_rx.count), __cputime_ns(sm_rx.max)); printf("TX: %ld/%ld/%ld\n", __cputime_ns(sm_tx.min), __cputime_ns(sm_tx.sum / sm_tx.count), __cputime_ns(sm_tx.max)); } int main(int argc, char *argv[]) { uint32_t loops =3D 100000, cnt; uint64_t ts_a, ts_b, acc =3D 0; if (argc > 1) loops =3D strtoul(argv[1], (char **)NULL, 0); cnt =3D loops; sysc_mq(loops); while (loops--) { ts_a =3D __cputime_get(); (void)getppid(); ts_b =3D __cputime_get(); acc +=3D (ts_b - ts_a); } printf("Accumulated time for %d getppid, %luns\n", cnt, __cputime_ns(acc)); printf("Average time per getppid: %luns\n", (__cputime_ns(acc)/cnt)= ); exit (0); } Complete bisect result: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D commit 10befea91b61c4e2c2d1df06a2e978d182fcf792 (HEAD, refs/bisect/bad) Author: Roman Gushchin Date: Thu Aug 6 23:21:27 2020 -0700 mm: memcg/slab: use a single set of kmem_caches for all allocations commit 15999eef7f25e2ea6a1c33f026166f472c5714e9 (refs/bisect/skip-15999eef7= f25e2ea6a1c33f026166f472c5714e9) Author: Roman Gushchin Date: Thu Aug 6 23:21:24 2020 -0700 mm: memcg/slab: remove redundant check in memcg_accumulate_slabinfo() commit c7094406fcb7cdf4fe1de8893f0613b75349773d (refs/bisect/skip-c7094406f= cb7cdf4fe1de8893f0613b75349773d) Author: Roman Gushchin Date: Thu Aug 6 23:21:20 2020 -0700 mm: memcg/slab: deprecate slab_root_caches commit 272911a4ad18c48f8bc449a5db945a54987dd687 (refs/bisect/skip-272911a4a= d18c48f8bc449a5db945a54987dd687) Author: Roman Gushchin Date: Thu Aug 6 23:21:17 2020 -0700 mm: memcg/slab: remove memcg_kmem_get_cache() commit d797b7d05405c519f7b62ea69a75cea1883863b2 (refs/bisect/skip-d797b7d05= 405c519f7b62ea69a75cea1883863b2) Author: Roman Gushchin Date: Thu Aug 6 23:21:14 2020 -0700 mm: memcg/slab: simplify memcg cache creation commit 9855609bde03e2472b99a95e869d29ee1e78a751 (refs/bisect/skip-9855609bd= e03e2472b99a95e869d29ee1e78a751) Author: Roman Gushchin Date: Thu Aug 6 23:21:10 2020 -0700 mm: memcg/slab: use a single set of kmem_caches for all accounted alloc= ations commit 0f876e4dc55db5fafef774917fd66e1373c0f390 (refs/bisect/skip-0f876e4dc= 55db5fafef774917fd66e1373c0f390) Author: Roman Gushchin Date: Thu Aug 6 23:21:06 2020 -0700 mm: memcg/slab: move memcg_kmem_bypass() to memcontrol.h commit 4330a26bc4527f1d8918c398ebc983574f761cca (refs/bisect/skip-4330a26bc= 4527f1d8918c398ebc983574f761cca) Author: Roman Gushchin Date: Thu Aug 6 23:21:03 2020 -0700 mm: memcg/slab: deprecate memory.kmem.slabinfo commit f2fe7b09a52bc898ec030d4fa6f78725439c7c2c (refs/bisect/skip-f2fe7b09a= 52bc898ec030d4fa6f78725439c7c2c) Author: Roman Gushchin Date: Thu Aug 6 23:20:59 2020 -0700 mm: memcg/slab: charge individual slab objects instead of pages commit 964d4bd370d559d9bd8e4abc139e85d2753956fb (refs/bisect/skip-964d4bd3= 70d559d9bd8e4abc139e85d2753956fb) Author: Roman Gushchin Date: Thu Aug 6 23:20:56 2020 -0700 mm: memcg/slab: save obj_cgroup for non-root slab objects commit 286e04b8ed7a04279ae277f0f024430246ea5eec (refs/bisect/good-286e04b8e= d7a04279ae277f0f024430246ea5eec) Author: Roman Gushchin Date: Thu Aug 6 23:20:52 2020 -0700 mm: memcg/slab: allocate obj_cgroups for non-root slab pages --_000_PH0PR11MB562641BC03630B4B7A227FD7E9189PH0PR11MB5626namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
#regzbot ^introduced 10befea91b61c4e2c2d1df06a2e978d182fcf792

We are making heavy use of mqueues, and notic= ed a degradation of performance between 4.18 & 5.10 linux kernels.

After a gross per-version tracing, we did ker= nel bisection between 5.8 and 5.9
and traced the issue to a 10 patches (of whic= h 9 where skipped as they didn't boot) between:


commit 10befea91b61c4e2c2d1df06a2e978d182fcf7= 92 (HEAD, refs/bisect/bad)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:27 2020 -0700

    mm: memcg/slab: use a single se= t of kmem_caches for all allocations

and:

commit 286e04b8ed7a04279ae277f0f024430246ea5e= ec (refs/bisect/good-286e04b8ed7a04279ae277f0f024430246ea5eec)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:20:52 2020 -0700

    mm: memcg/slab: allocate obj_cg= roups for non-root slab pages

All of them are part of the "The new cgr= oup slab memory controller" patchset:

  https://lore.kernel.org/all/2020062317= 4037.3951353-18-guro@fb.com/T/

from Roman Gushchin, which moves the accounti= ng for page level to the object level.

Measurements where done using the a test prog= rammtest, which measures mix/average/max time mqueue_send/mqueue_rcv,
and average for getppid, both measured over 1= 00 000 runs. Results are shown in the following table

+----------+--------------------------+----------------= ---------+----------------+
| kernel   |    mqueue_rcv (ns)   &= nbsp;   | mqueue_send (ns)        |    g= etppid     |
| version  | min avg  max   variation | = min avg max   variation | (ns) variation |
+----------+--------------------------+----------------= ---------+----------------+
| 4.18.45  | 351 382 17533     base &nbs= p; | 383 410 13178     base  | 149      base =  |
| 5.8-good | 380 392  7156   -2,55%   | = 376 384  6225    6,77%  | 169   -11,83%  |
| 5.8-bad  | 524 530  5310  -27,92% &nbs= p; | 512 519  8775  -21,00%  | 169   -11,83%  |
| 5.10     | 520 533  4078  -28,33%=   | 518 534  8108  -23,22%  | 167   -10,78%  = ;|
| 5.15     | 431 444  8440  -13,96%=   | 425 437  6170   -6,18%  | 171   -12,87%  = ;|
| 6.03     | 474 614  3881  -37,79%=   | 482 693   931  -40,84%  | 171   -12,87%  = ;|
+----------+--------------------------+----------------= ---------+-----------------

Below you will find the following appendices:=

  - raw GOOD bisect results
  - raw BAD bisect results
  - test code source code
  - bisect results
 
Friendly,

Sven Luther
 
GOOD bisect results:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D
# uname -a
Linux axxiax86-64 5.8.0-5-286e04b8ed7a04279ae= 277f0f024430246ea5eec-good #1 SMP PREEMPT Fri Jul 29 11:01:19 UTC 2022 x86_= 64 x86_64 x86_64 GNU/Linux
# ./mqtest
RX: 380/392/7156
TX: 376/384/6225
Accumulated time for 100000 getppid, 16924653= ns
Average time per getppid: 169ns
# ./mqtest
RX: 383/397/6300
TX: 377/385/8258
Accumulated time for 100000 getppid, 16915580= ns
Average time per getppid: 169ns
# ./mqtest
RX: 385/397/7690
TX: 377/385/4244
Accumulated time for 100000 getppid, 16922910= ns
Average time per getppid: 169ns

BAD bisect results:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
# uname -a
Linux axxiax86-64 5.8.0-5-10befea91b61c4e2c2d= 1df06a2e978d182fcf792-bad #1 SMP PREEMPT Fri Jul 29 11:01:19 UTC 2022 x86_6= 4 x86_64 x86_64 GNU/Linux
# ./mqtest
RX: 524/530/5310
TX: 512/519/8775
Accumulated time for 100000 getppid, 16921865= ns
Average time per getppid: 169ns
# ./mqtest
RX: 520/525/8204
TX: 514/522/7420
Accumulated time for 100000 getppid, 16931493= ns
Average time per getppid: 169ns
# ./mqtest
RX: 520/525/8128
TX: 512/519/5340
Accumulated time for 100000 getppid, 16921535= ns
Average time per getppid: 169ns


Test programm:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <stdint.h>
#include <sys/types.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <mqueue.h>

struct sysc_meas {
        int64_t min;
        int64_t max;
        int64_t sum;
        int64_t count;
};

static inline uint64_t __cputime_get(void)
{
        unsigned long l, = h;

        asm volatile(&quo= t;rdtsc" : "=3Da" (l), "=3Dd" (h));

        return ((uint64_t= ) h << 32) | l;
}

static inline int64_t __cputime_ns(uint64_t c= putime)
{
        /* Note, it is as= sumed thatTSC clock is 2.2G */
        return (cputime *= 5) / 11;
}

static inline uint64_t __cputime_get_ns()
{
        return __cputime_= ns(__cputime_get());
}

static void sysc_meas_init(struct sysc_meas *= sm)
{
        sm->min =3D IN= T64_MAX;
        sm->max =3D IN= T64_MIN;
        sm->sum =3D sm= ->count =3D 0;
}
static void sysc_meas_add(struct sysc_meas *s= m, int64_t value)
{
        if (value > sm= ->max)
            &nb= sp;   sm->max =3D value;
        if (value < sm= ->min)
            &nb= sp;   sm->min =3D value;
        sm->sum +=3D v= alue;
        sm->count++;
}
static void sysc_mq(uint32_t count)
{
        int fd;
        struct mq_attr mq= _attr;
        uint64_t ts;
        struct sysc_meas = sm_tx, sm_rx;
        char dummy[256];<= /div>

        if (!count)
            &nb= sp;   return;

        mq_unlink("/= sysc_mq");

        memset(&mq_at= tr, 0, sizeof(mq_attr));
        mq_attr.mq_maxmsg= =3D 10;
        mq_attr.mq_msgsiz= e =3D
            &nb= sp;   sizeof(dummy);

        fd =3D mq_open(&q= uot;/sysc_mq", O_RDWR | O_NONBLOCK | O_CREAT, 0666, &mq_attr);
        if (fd < 0) {<= /div>
            &nb= sp;   printf("ERROR: unable to create MQ, %d\n", -errno);
            &nb= sp;   exit(-1);
        }

        sysc_meas_init(&a= mp;sm_rx);
        sysc_meas_init(&a= mp;sm_tx);

        dummy[0] =3D 0;
        dummy[sizeof(dumm= y) - 1] =3D 0;

        while (count--) {=
            &nb= sp;   ts =3D __cputime_get();
            &nb= sp;   mq_send(fd, dummy, sizeof(dummy), 0);
            &nb= sp;   sysc_meas_add(&sm_tx, __cputime_get() - ts);
            &nb= sp;   ts =3D __cputime_get();
            &nb= sp;   mq_receive(fd, dummy, sizeof(dummy), NULL);
            &nb= sp;   sysc_meas_add(&sm_rx, __cputime_get() - ts);
        }

        mq_close(fd);
        mq_unlink("/= sysc_mq");

        printf("RX: = %ld/%ld/%ld\n", __cputime_ns(sm_rx.min),
            &nb= sp;  __cputime_ns(sm_rx.sum / sm_rx.count),
            &nb= sp;  __cputime_ns(sm_rx.max));

        printf("TX: = %ld/%ld/%ld\n", __cputime_ns(sm_tx.min),
            &nb= sp;  __cputime_ns(sm_tx.sum / sm_tx.count),
            &nb= sp;  __cputime_ns(sm_tx.max));
}
int main(int argc, char *argv[])
{
        uint32_t loops = =3D 100000, cnt;
        uint64_t ts_a, ts= _b, acc =3D 0;

        if (argc > 1)<= /div>
            &nb= sp;   loops =3D strtoul(argv[1], (char **)NULL, 0);
        cnt =3D loops;
        sysc_mq(loops);
        while (loops--) {=
            &nb= sp;   ts_a =3D __cputime_get();
            &nb= sp;   (void)getppid();
            &nb= sp;   ts_b =3D __cputime_get();
            &nb= sp;   acc +=3D (ts_b - ts_a);
        }
        printf("Accu= mulated time for %d getppid, %luns\n",
            &nb= sp;  cnt, __cputime_ns(acc));
        printf("Aver= age time per getppid: %luns\n", (__cputime_ns(acc)/cnt));

        exit (0);
}

Complete bisect result:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D
commit 10befea91b61c4e2c2d1df06a2e978d182fcf7= 92 (HEAD, refs/bisect/bad)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:27 2020 -0700

    mm: memcg/slab: use a single se= t of kmem_caches for all allocations

commit 15999eef7f25e2ea6a1c33f026166f472c5714= e9 (refs/bisect/skip-15999eef7f25e2ea6a1c33f026166f472c5714e9)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:24 2020 -0700

    mm: memcg/slab: remove redundan= t check in memcg_accumulate_slabinfo()

commit c7094406fcb7cdf4fe1de8893f0613b7534977= 3d (refs/bisect/skip-c7094406fcb7cdf4fe1de8893f0613b75349773d)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:20 2020 -0700

    mm: memcg/slab: deprecate slab_= root_caches

commit 272911a4ad18c48f8bc449a5db945a54987dd6= 87 (refs/bisect/skip-272911a4ad18c48f8bc449a5db945a54987dd687)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:17 2020 -0700

    mm: memcg/slab: remove memcg_km= em_get_cache()

commit d797b7d05405c519f7b62ea69a75cea1883863= b2 (refs/bisect/skip-d797b7d05405c519f7b62ea69a75cea1883863b2)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:14 2020 -0700

    mm: memcg/slab: simplify memcg = cache creation

commit 9855609bde03e2472b99a95e869d29ee1e78a7= 51 (refs/bisect/skip-9855609bde03e2472b99a95e869d29ee1e78a751)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:10 2020 -0700

    mm: memcg/slab: use a single se= t of kmem_caches for all accounted allocations

commit 0f876e4dc55db5fafef774917fd66e1373c0f3= 90 (refs/bisect/skip-0f876e4dc55db5fafef774917fd66e1373c0f390)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:06 2020 -0700

    mm: memcg/slab: move memcg_kmem= _bypass() to memcontrol.h

commit 4330a26bc4527f1d8918c398ebc983574f761c= ca (refs/bisect/skip-4330a26bc4527f1d8918c398ebc983574f761cca)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:21:03 2020 -0700

    mm: memcg/slab: deprecate memor= y.kmem.slabinfo

commit f2fe7b09a52bc898ec030d4fa6f78725439c7c= 2c (refs/bisect/skip-f2fe7b09a52bc898ec030d4fa6f78725439c7c2c)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:20:59 2020 -0700

    mm: memcg/slab: charge individu= al slab objects instead of pages
 
 commit 964d4bd370d559d9bd8e4abc139e85d2= 753956fb (refs/bisect/skip-964d4bd370d559d9bd8e4abc139e85d2753956fb)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:20:56 2020 -0700

    mm: memcg/slab: save obj_cgroup= for non-root slab objects

commit 286e04b8ed7a04279ae277f0f024430246ea5e= ec (refs/bisect/good-286e04b8ed7a04279ae277f0f024430246ea5eec)
Author: Roman Gushchin <guro@fb.com>
Date:   Thu Aug 6 23:20:52 2020 -0700

    mm: memcg/slab: allocate obj_cg= roups for non-root slab pages

--_000_PH0PR11MB562641BC03630B4B7A227FD7E9189PH0PR11MB5626namp_--