From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5D0EC7EE25 for ; Sat, 10 Jun 2023 02:40:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78888E0003; Fri, 9 Jun 2023 22:40:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B28B38E0002; Fri, 9 Jun 2023 22:40:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3EB28E0003; Fri, 9 Jun 2023 22:40:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 958488E0002 for ; Fri, 9 Jun 2023 22:40:55 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4BDA91203F7 for ; Sat, 10 Jun 2023 02:40:55 +0000 (UTC) X-FDA: 80885285670.22.BB07F7D Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf03.hostedemail.com (Postfix) with ESMTP id 7555A2000C for ; Sat, 10 Jun 2023 02:40:52 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UVlZ9JL8; spf=pass (imf03.hostedemail.com: domain of yu.ma@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu.ma@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686364853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PszLhsnto48yTB8KgmrixkKhTCYUSXUUbkyj+qEXEjA=; b=imZPrniz051iGG071IkfKOxNOAmPj5K60KtxppGWQ0QBY/1Jp1DExrLjd2ue8ItLUi7wJG 2v5WfZm4qLLxETdzZfG/Hves8MYhwy0r3FJWHr9QWPICr7jW2F9GJLVm2GH2fuDII171hI faaCFGFT9RUg0V7kxn9lDFUtn7QxWyc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686364853; a=rsa-sha256; cv=none; b=UUUOxYkyxiybkZKBS4K4dhw1n9aUp6ixTQejTS1eyIDtUDuU4j96shycy8dE6v0u1Byel4 ch1ss0sMbAoIh6O6BSEe4nP9yxhfXYlYnBZQYhzVBl/XNW+CbrIIoGsuhEv5yEnZiMenR9 RdHkyYo/mA6vZG8XCIyl2MkLElfuGp8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UVlZ9JL8; spf=pass (imf03.hostedemail.com: domain of yu.ma@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu.ma@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686364852; x=1717900852; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8jqWFsMfbwzSoZ5zFqaX3i8XF+2E5kDtejebv/Ye4BE=; b=UVlZ9JL8h0l2NP1D4OGhxxPBpiN4GtfJLNhoGsBog1rz5NbPF/KWF8IB GWC88fu6nXZU7TBwm8vR8DF9NbOH7B2MM+gXkV0Uj4ruy8fre3b2wnKGL gVeg9oWn8PC5uGzE9Rb/3Pmz6InRdiO2t5ooRIMgg1Q93D1M+Hhsc+F8H DxQNPtdM22ao6DJ8HdLPPRFL7/XMUIikHV0zwTpYwIwS15NY2aWiVTcTK uCjB32uoS8mz2YtLs2W7PJ7hN5UHmWYTmfDhDZWWMKPlfmGk5kxbmbnOP T3bY4BM4XmYHFYA23MZTXLhLvi9hyfcihfxtQlYKD4KRmgpW8+3rThFGt A==; X-IronPort-AV: E=McAfee;i="6600,9927,10736"; a="337361526" X-IronPort-AV: E=Sophos;i="6.00,231,1681196400"; d="scan'208";a="337361526" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2023 19:40:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10736"; a="854935287" X-IronPort-AV: E=Sophos;i="6.00,231,1681196400"; d="scan'208";a="854935287" Received: from linux-pnp-server-20.sh.intel.com ([10.239.146.185]) by fmsmga001.fm.intel.com with ESMTP; 09 Jun 2023 19:40:44 -0700 From: Yu Ma To: dennis@kernel.org, Liam.Howlett@oracle.com Cc: akpm@linux-foundation.org, dan.j.williams@intel.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lipeng.zhu@intel.com, pan.deng@intel.com, shakeelb@google.com, tianyou.li@intel.com, tim.c.chen@intel.com, tim.c.chen@linux.intel.com, yu.ma@intel.com Subject: [PATCH v3] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Date: Fri, 9 Jun 2023 23:07:30 -0400 Message-Id: <20230610030730.110074-1-yu.ma@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Stat-Signature: qfpthafxcgs491d6nyk7ghuggqf7t7fw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7555A2000C X-Rspam-User: X-HE-Tag: 1686364852-410343 X-HE-Meta: U2FsdGVkX1/k6bRLwFNrX9SUP+L/WpDdJFHFxr82SYZQiDlyfcfSI9bGS13eHn5mBaqr/Ij1qrCQJ9dO1BMgc/h6ky3OeCJj7rBbychQudpsbThTsOTbhUfqhHLMf42EtFBAu5kMslGBj6qMXBZMIH9PdHl0c024+6hiMnTqIy9cgGPM/oNOMr7rEV7OHdPZQOC59DtJowmpmT5GzMwKR+DIVPj51y63Am/PS8Hc/tQpQVncEzFXq/yqt22NjKDXTUIkkwkEsjlYLn9fhOKr1vCx8LmNmBrh5P0JmYrT4C9fsaq0JGUirMRGSsFwiICFfxUmlV6cBK+a/b+VumuqDOQsCNCcMnaFppWCZz34UYgRGBa0l1esh0Zj71eRG3FcTbmB48JuWdEjMaFrVqdj5LDTt55zfYY7Xo0i/x0yBNTEgHyRjJPIgo9Xq8NSJ7z3PwmUBVGVj8zyag78ykEIJ38xqTIT5Q1puBeMJ4wcofZ4npQdDd31KNqtNbR95FZVZBCcKiAInMtn435PVkc58XoCUnrUgTuW96iwwuONK1X3WLGqt2sfg4z6gA/aJYrwQPvD87JVRmqnkg+7rPTgvdatIQB4xvdfSEv7xaCHXoM5JlgEpCvjyWm6UbIKdPUMEkErOB5GET/WlQbqzW+YwWsHKLKF/whJ8fm04KPVDnd8RDadDXIlvwaJLySMdxuYllihbY7LdlQ0jBjeJJZe8Nm+qbGyJajQ3IzN2BKIt7uYuVVyFE4rcZBpab6TPvNuGj2/jASYCmdAztfciknIYw+gQ4wQN7TfNktybUm4Cl2UGUPuuOCr3C5SJsnaCvIKhv08wO/nkNOddm8AvpDhF3mVVepnF6It/Fkdk3xy566PgUYQJcgQ2geoj4SlP6G4GTfImsrE7u1c7j+vFj/8KdHrw8rPuq+Ku0Czs/p6wVRIVDyQWNBGbkcqCavZsda2wGw8w5tk/8eagmRoCM3 WZBqREeY WGMKGN5dLaYHJMPjoEiGKWXkF3/YN9rKqpEYheLRxwZXTTkumUlC5MtkkYvNHPGV6LRP8/E5n1LqHw9dQrbzL7jVqyIZtjXH++pIbdm6fxDNWleFcOi5WMlw51UC/Yh3Zo0bAKjY1uLcIVSDJB2nj6RmBUieRH3nsEJq3SUvv0ydjZpbGYfpE4yahwuiJf4szEGTSZ/hXMetj55Wmv24fI1QwXA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When running UnixBench/Execl throughput case, false sharing is observed due to frequent read on base_addr and write on free_bytes, chunk_md. UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. It will do system call on execl frequently, and execl will call mm_init to initialize mm_struct of the process. mm_init will call __percpu_counter_init for percpu_counters initialization. Then pcpu_alloc is called to read the base_addr of pcpu_chunk for memory allocation. Inside pcpu_alloc, it will call pcpu_alloc_area to allocate memory from a specified chunk. This function will update "free_bytes" and "chunk_md" to record the rest free bytes and other meta data for this chunk. Correspondingly, pcpu_free_area will also update these 2 members when free memory. Call trace from perf is as below: + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock - 53.54% 0x654278696e552f34 main __execve entry_SYSCALL_64_after_hwframe do_syscall_64 __x64_sys_execve do_execveat_common.isra.47 alloc_bprm mm_init __percpu_counter_init pcpu_alloc - __mutex_lock.isra.17 In current pcpu_chunk layout, ‘base_addr’ is in the same cache line with ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the last 8 bytes. This patch moves ‘bound_map’ up to ‘base_addr’, to let ‘base_addr’ locate in a new cacheline. With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.4-rc4, the 160 parallel score improves by 24%. Reviewed-by: Tim Chen Signed-off-by: Yu Ma --- mm/percpu-internal.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index f9847c131998..7f108b25bb93 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -41,10 +41,17 @@ struct pcpu_chunk { struct list_head list; /* linked to pcpu_slot lists */ int free_bytes; /* free bytes in the chunk */ struct pcpu_block_md chunk_md; - void *base_addr; /* base address of this chunk */ + unsigned long *bound_map; /* boundary map */ + + /* + * base_addr is the base address of this chunk. + * To reduce false sharing, current layout is optimized to make sure + * base_addr locate in the different cacheline with free_bytes and + * chunk_md. + */ + void *base_addr ____cacheline_aligned_in_smp; unsigned long *alloc_map; /* allocation map */ - unsigned long *bound_map; /* boundary map */ struct pcpu_block_md *md_blocks; /* metadata blocks */ void *data; /* chunk data */ -- 2.39.3