From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E45C7EE23 for ; Wed, 7 Jun 2023 14:24:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C7BF8E0003; Wed, 7 Jun 2023 10:24:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 578D78E0001; Wed, 7 Jun 2023 10:24:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43FFB8E0003; Wed, 7 Jun 2023 10:24:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 338A58E0001 for ; Wed, 7 Jun 2023 10:24:49 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DB6EF140251 for ; Wed, 7 Jun 2023 14:24:48 +0000 (UTC) X-FDA: 80876173056.07.F051D88 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf18.hostedemail.com (Postfix) with ESMTP id 7D79A1C0022 for ; Wed, 7 Jun 2023 14:24:45 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="HcU/AaEE"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf18.hostedemail.com: domain of yu.ma@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=yu.ma@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686147886; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E/9ARJok8U8ujnPVe3o5BwOiMElSPTaQAQBe0ZpmtLo=; b=fvmM6LAgBvaFPOVDv56HeHvmV1P2sukSmlrW6Ta3G3TbL+ftXxxv7xXDLNhc3ql7AJQEp+ 2O1ntqQNKvhblvyzpPx6h6hNBoQFwTdZT/cqfTwYowX0yALcf2UysEVVqvPratKxX3iDGU q/Pz25Jpm+XCsyu9QIJRXcMs8lh5qMY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="HcU/AaEE"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf18.hostedemail.com: domain of yu.ma@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=yu.ma@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686147886; a=rsa-sha256; cv=none; b=gqBlsV/doP4GVIg4Fa2UzkZIJb7TKgyMa7FjLOXjwVIjF/WOfPFmuOYWi8xOThUSORhZ3J 9/y+Z6dnLpSQi54rBpatiNCGJob2mY7AGysrvd9885QUmwDyy5DWZz9QQMoFikDJ1tGaQH U1FQhuOr5zLjJVxlLchkIl3JzxxRPfU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686147886; x=1717683886; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qOS33Z45j4TI2OKl5FBe0hF56pv3kMKz0o+IBc1D+4Q=; b=HcU/AaEEpaRrVtTniI8JisPPtruT3xKC7WTt2y1nOXB2OLuAaIe/5lfc sRsnlU/N4+fMrTZ80ZzxBNFCVG0Z2gIUsS+BUMxgPEwBWfyRgc0v3rvLE uMIhogvj60vg8mz27n9cL2CQ2A52bFoFNdzbhN78U7xZlbO06zoE7rfY7 B7B2aIxF8rYlu8jXwr+NXYV6edZ1gvVPU7wVlf8hTpIv7GU9ypC3mNpEt h2AgwPL4LCh1Y/cGM0ogAsFkL7ruLKO9YiDwWJo2FBWyYXAUxH6sP0TKg fdLBuOef4Pl/O3/sLegE+GOc+cfKehBDrC2WJQHC+3YGC/fLhp2ExJoWM w==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337361333" X-IronPort-AV: E=Sophos;i="6.00,224,1681196400"; d="scan'208";a="337361333" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jun 2023 07:23:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="659973610" X-IronPort-AV: E=Sophos;i="6.00,224,1681196400"; d="scan'208";a="659973610" Received: from linux-pnp-server-20.sh.intel.com ([10.239.146.185]) by orsmga003.jf.intel.com with ESMTP; 07 Jun 2023 07:23:23 -0700 From: Yu Ma To: yu.ma@intel.com Cc: Liam.Howlett@Oracle.com, akpm@linux-foundation.org, dan.j.williams@intel.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lipeng.zhu@intel.com, pan.deng@intel.com, shakeelb@google.com, tianyou.li@intel.com, tim.c.chen@intel.com, tim.c.chen@linux.intel.com Subject: [PATCH v2] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Date: Wed, 7 Jun 2023 10:50:09 -0400 Message-Id: <20230607145009.58899-1-yu.ma@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230606125404.95256-1-yu.ma@intel.com> References: <20230606125404.95256-1-yu.ma@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 7D79A1C0022 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: w6ogeu1i9wycy3f539ebrxfhkmxx8nwb X-HE-Tag: 1686147885-882443 X-HE-Meta: U2FsdGVkX1+tWYW9UemOeqifOFkt9wUt+NSS6Qz+YoM/6mQgyfZCFUcVS4gzLTkeJQWNdbuUKhh1viT5ysIbCgQP+OJmO8RdmWWdLqbG5IaJm4iBBlYEjhf+tEdIouuNpca/33tV8Ao8krq9vZfqJ+JppNepW/EN6W+o1Gb586JVgPw6vYM/46oaN1XSuJZsGnAWWERIhWk0UlZCg5hG1BMuOG2kWnuynRO+lLZT/rCgJ6OPZbSIxqRMf2rIE9O7894hEhuBBDey3+ZR2Ral7Qm4LypfxecLy/UpfhRr6ZbLWqJxvxJURSR0a51rp/CkhWe9/nFt9fcoSIkSQ+3UI8mssTrcEtinR9xtTUNYbbYp/czX8WeHD7+U4TrGIPCdklBtbis2HXopGJIyiSK57rwpCJEyfi6TsHNEw0tigEiJ3wtGupktUm4xFJkMgspTD7JGijs3K6GQ0th8O8fXeCe1U4bEagg2nIPqi/0HT730LoQIeZG82eO9OTNzwV2F9jt4Vmsb+VK8/nZKq+uRqS5rkq/5vysDjieG8P+6bR2RmiknBYlJ0OEAB8aQs0ryrVVN2E6Ez9Y9mMYjZY8atVxF7fAsilI7xnilM8Lr4PuPNFWVPzFy2JeSKP96U96tGMqIwWqlikCkcdpZ/mzAWX0IGMQKf3PpI1z42skLwEDb9wvZHULBMAeBotdFRt/tyd1xVxVIR357k6v5Wp9Sizr859qdwqdmRSyHtOc/TrDioVlutNV17WzExPevwqLlXtU7LLjHaukcWnByXuLr9mJQU2HtmczJOCVp3kz7Gv/aFVMnSy5ZRdqiRnoilygHXzmV1JFSPkO8zbrJ7/6tNag39+UmBR7tdSeSYwLIwU+Qn3o9KF2INhgRqbPVdMgeCO46R3voDUQO2sc6mF+EYgKfPM4kS23GOS+zOtd+YGDIBxSbIf4FjqqZRzht7Gj/8pIeNhsXW21J2ydPcn4 WNjnTjoK hqyx/TKUsLChcdst1to+vmFOZvy/ihGo94GnKYVOs4cDavhxCZ00qZMsu6F5WQ4IAXjc8X3DpyjxHj/tG1z5HO6NEE+XHEYwL565JY1JiOoYPGf0m8ZpnfWO+TbSfgqcVv17IfcvwC23qWfXFKupO4FWHar+5T92krHw6pKt4OfbjOjbatlDywEwSe6HzOBQT6EJ4SHkZd95sNR0763nSHZ1kiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When running UnixBench/Execl throughput case, false sharing is observed due to frequent read on base_addr and write on free_bytes, chunk_md. UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. It will do system call on execl frequently, and execl will call mm_init to initialize mm_struct of the process. mm_init will call __percpu_counter_init for percpu_counters initialization. Then pcpu_alloc is called to read the base_addr of pcpu_chunk for memory allocation. Inside pcpu_alloc, it will call pcpu_alloc_area to allocate memory from a specified chunk. This function will update "free_bytes" and "chunk_md" to record the rest free bytes and other meta data for this chunk. Correspondingly, pcpu_free_area will also update these 2 members when free memory. Call trace from perf is as below: + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock - 53.54% 0x654278696e552f34 main __execve entry_SYSCALL_64_after_hwframe do_syscall_64 __x64_sys_execve do_execveat_common.isra.47 alloc_bprm mm_init __percpu_counter_init pcpu_alloc - __mutex_lock.isra.17 In current pcpu_chunk layout, ‘base_addr’ is in the same cache line with ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the last 8 bytes. This patch moves ‘bound_map’ up to ‘base_addr’, to let ‘base_addr’ locate in a new cacheline. With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.4-rc4, the 160 parallel score improves by 24%. Reviewed-by: Tim Chen Signed-off-by: Yu Ma --- mm/percpu-internal.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index f9847c131998..ecc7be1ec876 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -41,10 +41,16 @@ struct pcpu_chunk { struct list_head list; /* linked to pcpu_slot lists */ int free_bytes; /* free bytes in the chunk */ struct pcpu_block_md chunk_md; + unsigned long *bound_map; /* boundary map */ + + /* + * To reduce false sharing, current layout is optimized to make sure + * base_addr locate in the different cacheline with free_bytes and + * chunk_md. + */ void *base_addr; /* base address of this chunk */ unsigned long *alloc_map; /* allocation map */ - unsigned long *bound_map; /* boundary map */ struct pcpu_block_md *md_blocks; /* metadata blocks */ void *data; /* chunk data */ -- 2.39.3