From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C34A6C7EE43 for ; Mon, 12 Jun 2023 21:43:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 038C16B0074; Mon, 12 Jun 2023 17:43:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2B326B0075; Mon, 12 Jun 2023 17:43:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF2C08E0002; Mon, 12 Jun 2023 17:43:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CC32D6B0074 for ; Mon, 12 Jun 2023 17:43:37 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8F4A4120359 for ; Mon, 12 Jun 2023 21:43:37 +0000 (UTC) X-FDA: 80895422874.09.FA0976D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf16.hostedemail.com (Postfix) with ESMTP id C60AF180015 for ; Mon, 12 Jun 2023 21:43:34 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=ZJPFlxxS; dmarc=none; spf=pass (imf16.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686606214; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8LGtYNPhYMeH+LnTP4Y5y6WgJiTZazxaHtcerKwbpvk=; b=xwt5pIFckHWqxl+nC2tcdpyHPHJmapJqdN7QKanYK+dfvWkm3DnIa5myyXeqpiPFeShv2L NpM9pgcNOH0OJZZw5IP8enU7r1c0srK2sn87QhYF5XywgrbnA+OVpGWFqpSfqmw4WNvVBO fUssV6S5cCHq7jqck0YoZkERi1VPyp8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=ZJPFlxxS; dmarc=none; spf=pass (imf16.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686606214; a=rsa-sha256; cv=none; b=MAi0TeqVeaJMhIJ4HiANojfPl13Dwt+OG2J4rVnOaEm236U//mqFH1I85p/7XTmlL3qXA8 d8Me48kaeB64ZYCQ6XLGVzE1PYqsfMZWW5zcxrgJYvbdou8tZ4LozqoJBzKhuaMzN/fC5y HDHkJwWyULpy0XS/lv15me6bAcw5I7E= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CDA8462055; Mon, 12 Jun 2023 21:43:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCC68C433EF; Mon, 12 Jun 2023 21:43:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1686606213; bh=pJu03XMdmPE7SbWuyufLP/c1f6vs0Y+aafTE9PXHw1Q=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=ZJPFlxxSuK1z4No0psN3Y/lGE4KZCsRmSZq3Cu3v28aUB/M8v5Y4HtJkXMIB2UQIg NoVN3BIsmeDWXYSplidLRSNzvE4vq2BgLKN6XVQB6ENTsiKGwseQLqCb+qzAb5MytN C0YpByv+OpVN4WEG4V0nJa/craec2ZW7r/bp92Bg= Date: Mon, 12 Jun 2023 14:43:31 -0700 From: Andrew Morton To: Yu Ma Cc: dennis@kernel.org, Liam.Howlett@oracle.com, dan.j.williams@intel.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lipeng.zhu@intel.com, pan.deng@intel.com, shakeelb@google.com, tianyou.li@intel.com, tim.c.chen@intel.com, tim.c.chen@linux.intel.com Subject: Re: [PATCH v3] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Message-Id: <20230612144331.b1d069bce4ba3800fdd62738@linux-foundation.org> In-Reply-To: <20230610030730.110074-1-yu.ma@intel.com> References: <20230610030730.110074-1-yu.ma@intel.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C60AF180015 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: j81k8eob5wygsa9arh3zoeirnd5gxz1a X-HE-Tag: 1686606214-651447 X-HE-Meta: U2FsdGVkX18Q6gdYySyDu08y0A0W3Wi+MJU1Msjx4Ax4UX32l1J4ORJVxWZZqtdQ/5qShocFUTcuDwS6wh3n96wB4GRQyHROP+qGsB8SKeyfLytomcKi7zz4CNh/PIezq4DwrjkrhO9/F5ezFeMBlaTDLWWi9c3BenBvYwSqJ5YS2UqhS89/qp0REL496RBO/bcVU3dNcwKXPTQUl3gKhf5akPFLaCYGxKa1csW3qdc2z7tiwQzF0Wc8/2o40RmVXZ04BjM6QHu3+8RR4HHuVedmcDaYg0I6NCYQcxp+tRBd+bl20wRqYqU28l9bM+zOPG3NVz72l3rK6fd/V5gi6KdumjxeHHMB6WbB09UgIGv7o0+EGHx4BWZvwlsGcCp3l33TkkwlrFFDAlvwCzqa7Ys9dJg/Az8bYxgTrSgsMjrM8nI7yfnxWLYPWNjdhhwbj6bW/y2ak5FcJULT5g2xo7vGsr6c0XMtiMY4dcYb0ePhul2NAu6JhoT8sU4P+XD18bGjqQ42EGHUjyHioEqQEMxMdGcrzZMC/CCQayK1Ncu/wxjVQeVdlet7ogGVPa92pT2JPR9jCWsslloi6JfIirAyBc5eF3QWlyxhthukAvIhlUxslZ1kqgY9niL6cpIAPIqR21g6dwdCkZ+DFqyIQTu0gJ26FJd49CVjiQyMazCLsBdpGZfYabsX/581xSsM+BEmy6v1KmUjQX9C3JQITOiGVdnpuDwpFG2KbXaDksJxUIIZgAFCH4YY5Tkx+XchFt1sk6GMUdLYg0VS7dpIfvFvp2ng51D8/uDzhJGcNPmBJnJ4HzqT6rHNnX3+sJfzF1q6lXOiz+WrEWRIAU417XeNTxaAY45SI+iB0UTFvCOXdBtzDcU7zzp3mu9nPgLwQIGPfMltg2eZ00BMstdmY1np7ijr8v9GqCXT6vkGxwVCgvhISWmOY3Ku/s3WnozhiTo73lUxKIYqntJdH6W WeqSDKPU d2/xAZ9D8GSOPdNWIrNAN9B8rLXCHdxwWjrcZQNTFZjn93QjwuYpb6uxAswoIQ5irfevYUW9KSONzVkGWYn+WPmENnfHIjusWlQkujiVejPlEdoxkC5fVB/N/FlwxgcmqH6KBOVjgIpzrlFmKaSQLUohxRJeF7CdZkYb/3IbuRpMhPNuR/ci5hObBujCWEuRToyFI2rex4PZuidj1M1OgW5nnAMEb2QIPePnnbs+i+24u7u1GXTAxmW8vpHP8HgFjtaHLmEBIxJgU9/qKdymUfHKVUtPs85/kQdi15wiUikmRDY+Arkw5CzVIp/9ahB1WO0c8tetTqwqubIPTEJ6OVpVSPJg6X3Ol4NhJT/6JW8MPnEDlyL3DA81YA+XkTzxerpa7zRiMgZlec/1h06022E42jg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 9 Jun 2023 23:07:30 -0400 Yu Ma wrote: > When running UnixBench/Execl throughput case, false sharing is observed > due to frequent read on base_addr and write on free_bytes, chunk_md. > > UnixBench/Execl represents a class of workload where bash scripts > are spawned frequently to do some short jobs. It will do system call on > execl frequently, and execl will call mm_init to initialize mm_struct > of the process. mm_init will call __percpu_counter_init for > percpu_counters initialization. Then pcpu_alloc is called to read > the base_addr of pcpu_chunk for memory allocation. Inside pcpu_alloc, > it will call pcpu_alloc_area to allocate memory from a specified chunk. > This function will update "free_bytes" and "chunk_md" to record the > rest free bytes and other meta data for this chunk. Correspondingly, > pcpu_free_area will also update these 2 members when free memory. > Call trace from perf is as below: > + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init > + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc > - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock > - 53.54% 0x654278696e552f34 > main > __execve > entry_SYSCALL_64_after_hwframe > do_syscall_64 > __x64_sys_execve > do_execveat_common.isra.47 > alloc_bprm > mm_init > __percpu_counter_init > pcpu_alloc > - __mutex_lock.isra.17 > > In current pcpu_chunk layout, ‘base_addr’ is in the same cache line > with ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the > last 8 bytes. This patch moves ‘bound_map’ up to ‘base_addr’, > to let ‘base_addr’ locate in a new cacheline. > > With this change, on Intel Sapphire Rapids 112C/224T platform, > based on v6.4-rc4, the 160 parallel score improves by 24%. Well that's nice. > > ... > > --- a/mm/percpu-internal.h > +++ b/mm/percpu-internal.h > @@ -41,10 +41,17 @@ struct pcpu_chunk { > struct list_head list; /* linked to pcpu_slot lists */ > int free_bytes; /* free bytes in the chunk */ > struct pcpu_block_md chunk_md; > - void *base_addr; /* base address of this chunk */ > + unsigned long *bound_map; /* boundary map */ > + > + /* > + * base_addr is the base address of this chunk. > + * To reduce false sharing, current layout is optimized to make sure > + * base_addr locate in the different cacheline with free_bytes and > + * chunk_md. > + */ > + void *base_addr ____cacheline_aligned_in_smp; > > unsigned long *alloc_map; /* allocation map */ > - unsigned long *bound_map; /* boundary map */ > struct pcpu_block_md *md_blocks; /* metadata blocks */ > > void *data; /* chunk data */ This will of course consume more memory. Do we have a feel for the worst-case impact of this?