From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5626DC7EE2E for ; Mon, 12 Jun 2023 21:55:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB8D58E0002; Mon, 12 Jun 2023 17:55:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C67B66B0075; Mon, 12 Jun 2023 17:55:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B310D8E0002; Mon, 12 Jun 2023 17:55:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A3F5F6B0074 for ; Mon, 12 Jun 2023 17:55:50 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6C7A914034D for ; Mon, 12 Jun 2023 21:55:50 +0000 (UTC) X-FDA: 80895453660.10.BF01838 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf29.hostedemail.com (Postfix) with ESMTP id 88B56120012 for ; Mon, 12 Jun 2023 21:55:48 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf29.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.219.48 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686606948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NdjVUJ6/scsSqpoSTprToMI3M6bYhDxCkzPi9uXpyWQ=; b=PQHPYprjZP9y9rj2fqizT6/7MGji/Vd3zzQ0SxVnfEjZi8/E3W/q/ajXFvwwKH0tLk99R6 E8qzMXOA0DTyKMRdhOqWzZNrKllG3kLrlmz+7TD2Ufvgwue20zycosYOo5ImHWPb4/hqlG RIohery5q2uSjv6OKHV3HsMtjTxx09o= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf29.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.219.48 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686606948; a=rsa-sha256; cv=none; b=7qPPV0MYQn0vhbiV1bhHsKlo7qSW1SrVPr8SaV6VJr05XKjTu9rateoLofQJajwo6w9rcR 7CIoiFoBUSNm5Hx+P29gkGEm5F0hYP/B1MJNPO4duz3i4OgbCvoq6XzHkns55iZ6vifaeQ fFkn75YtEqzZc9ky07ZQkjUUamcg71M= Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-62de508705dso2817196d6.1 for ; Mon, 12 Jun 2023 14:55:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686606947; x=1689198947; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NdjVUJ6/scsSqpoSTprToMI3M6bYhDxCkzPi9uXpyWQ=; b=SSaHqtb5bvjtf9FPbC5mZQZeuxFWY8cQSwbIMW57IO/ShoviYfrHKZ0lHHRev3/Uoj w9a9zvgVQR38wE2Bcubnq3owvL0QmUn24tRx5MalISz6JfWCboHvNiZP12GkT7bKCqi0 1lHONXoLIGTR7Og9SP/veal6wbi9YjFVU6zFxsKLcsFJF/ypFLP3xKAdcSe7iBt+DL3d esJMiutwEMN2FHcmiwQev1x+4fZ3HHYSOUpogSsh0PzCIclZ+J2lxcJitQ/8Cczzjjuf 0in75TVWDYYWnI/UE98RTzwmcjcz+SOXwqlAiGREnZq237nlfAJrSv80NnftYQw9iIlp YSFg== X-Gm-Message-State: AC+VfDyZkSwh0/ZMynqKaENx70wviDgnqkxzdS1fsB2itljJFkI/wt0o IuxrNCsNf4akoK+giAdXat8= X-Google-Smtp-Source: ACHHUZ4m5vStrs0sioVKbOYDwj/2oyC5bo7y8pq6ntqGS4HJej45p5Csu8B3wr92BynaH+AKYVHjSw== X-Received: by 2002:ad4:5ca8:0:b0:626:1984:2461 with SMTP id q8-20020ad45ca8000000b0062619842461mr11872904qvh.28.1686606947610; Mon, 12 Jun 2023 14:55:47 -0700 (PDT) Received: from V92F7Y9K0C.lan (c-73-228-235-230.hsd1.mn.comcast.net. [73.228.235.230]) by smtp.gmail.com with ESMTPSA id l5-20020a0ce085000000b0062439f05b87sm3539394qvk.45.2023.06.12.14.55.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 14:55:47 -0700 (PDT) Date: Mon, 12 Jun 2023 14:55:44 -0700 From: Dennis Zhou To: Andrew Morton Cc: Yu Ma , dennis@kernel.org, Liam.Howlett@oracle.com, dan.j.williams@intel.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lipeng.zhu@intel.com, pan.deng@intel.com, shakeelb@google.com, tianyou.li@intel.com, tim.c.chen@intel.com, tim.c.chen@linux.intel.com Subject: Re: [PATCH v3] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Message-ID: References: <20230610030730.110074-1-yu.ma@intel.com> <20230612144331.b1d069bce4ba3800fdd62738@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20230612144331.b1d069bce4ba3800fdd62738@linux-foundation.org> X-Rspam-User: X-Stat-Signature: 5ux6qcjooj6ci577q86nswjdkdmbj1o8 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 88B56120012 X-HE-Tag: 1686606948-767412 X-HE-Meta: U2FsdGVkX1/1Cvgprc92Ry+Tc7iBkQb/a8WWzUO/11Ot8Ytjjava2lfCP0nPCZfflpBKqD0TyamTGi+eUA3wAGkz/PwHxkMv9fHh5dX7jJ70wfPQYaKjoN89Tnx6aQW+uWwgCHLyMVlcawVtmaod/IKLGvfOYnfHweZmSCZaPq4ddU/z5naBOqywhsOGZeDuqZdmR7vj3WvQXeg2ym9/JuULayAHEHr/Bg9Yw09l3ptRCxkOAie3J28MyfUDGWe0L1tZwv8qhMRNNYkux9963XgqzLBlm0ZAPVO99SXWRSnYg72xhBOrVH2yqqAtD496hqNHM0msLCnVCvHqNCdhfkAqlpcvRMqlOGrfW/Wu9LMetQbhvh0yeObICrWHCU/AciacxEWjPBHE4eM7fWfPms7mtPOzP9fqLWaWz/MhZOa5WWx7zGK3Lr900t6OplANvsOB1HadIhC1QPRVKpKdWaW3mS6QXJq3xJn9+0MlxnIXJiM+Y+S+6OdNmbYVuWMBaUnpNbNmO71Ax6jWRM0hVzoLyfkyPOg/DheW6AvPpWq/kuorl7G27rN/mL75w3efSpDtgtmC9x69IZZFh1tx1gZfENLMWBf5vNfJjHKLSIczxAgzAmjtYz2M5PP/jLNHY6xjXPUCJZxxOzcEKKl9n8B8OwabxcN2kzVzFVPab8FKH5hCPsQbMx7YYmcP0gQK/LOEt5We2mU/fSN5G0PnfsXvaC1nwWcB4Gnm/UTR/JyQabZdAPuOSa2NBZy3NRdev3vfR+Stre+jYJnR0i8On5LP+3/9dbAifXjveoH8CcBT0jSE4BtmPTyQmwBtoRt2rUCVjI1l5oS+/KT1hrulo/1K9hfFV59Q3rvP7MycCr1j0xr1ZX+0sz5zudNJVw0qaSPr6pVwrGkt02gluWloBMql9vefJLLoiV5Ufx2uDbTMsPm1h9xM4yu2OEyjy37IQRPwgppTG1u2TAAEKa3 X6nDnrnt HRiMlay2QipVFOGKkAc8G9IBrwXASzRxOOLNvXeZbVT394dDme7cTtyZ7XuSPtvt9vL5fnbbWFksPACG7SJdjip+Pr/27GGoFmCe2c90yBKPv9ipQcDAyD2kprisxhWbvUkCbX2xuq25X+66/Rnh+1SlEZnxEt91MSwMz1xqVDmLoSlm7WYVCCIGK7HOd7t8qw8vi/s15e7KGoNOwE65TjVpv8a3ppPB57lnPND607bSSm1CgO0/hwWgOq8UmC91nQqdXQJdLkuqwDMA77t0FN2wU+mvcQRFo2+2k8HFJ4wy8z5+wtfrpUGz0HtMWj5gFRtIdcw1sBxekV3X3mev5coiNYjpG3xGzoXa7g5SEyM3UkAXkKZLbxsegGlLbsobTVt6tU64joe1VBmQ9pxRd0OWs3lX9YuhyrSS0AsXCMbp4Q2KBG+VfdtHNLmOUW9I/ULr/ENxKYc5hOUiP1sdcgp/AGA3nTGfbZ/q9MXyTPuYhaEN1oqb2/Gser/UexiWcTfu38EezazKVQqSg4sETg6QtfpoAJ0QjWPXMVbcXyGxRuKRSQ0h8YvvACgnw8Kb6KASFaWFYb8yXO1bS0U8LyBSS5EFor5bqEBw/iksqiXyaeTmWcZLhAa7IdxYHLC6Th1TKuzDm8X2sLV98hmTXpVENjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Andrew, On Mon, Jun 12, 2023 at 02:43:31PM -0700, Andrew Morton wrote: > On Fri, 9 Jun 2023 23:07:30 -0400 Yu Ma wrote: > > > When running UnixBench/Execl throughput case, false sharing is observed > > due to frequent read on base_addr and write on free_bytes, chunk_md. > > > > UnixBench/Execl represents a class of workload where bash scripts > > are spawned frequently to do some short jobs. It will do system call on > > execl frequently, and execl will call mm_init to initialize mm_struct > > of the process. mm_init will call __percpu_counter_init for > > percpu_counters initialization. Then pcpu_alloc is called to read > > the base_addr of pcpu_chunk for memory allocation. Inside pcpu_alloc, > > it will call pcpu_alloc_area to allocate memory from a specified chunk. > > This function will update "free_bytes" and "chunk_md" to record the > > rest free bytes and other meta data for this chunk. Correspondingly, > > pcpu_free_area will also update these 2 members when free memory. > > Call trace from perf is as below: > > + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init > > + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc > > - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock > > - 53.54% 0x654278696e552f34 > > main > > __execve > > entry_SYSCALL_64_after_hwframe > > do_syscall_64 > > __x64_sys_execve > > do_execveat_common.isra.47 > > alloc_bprm > > mm_init > > __percpu_counter_init > > pcpu_alloc > > - __mutex_lock.isra.17 > > > > In current pcpu_chunk layout, ‘base_addr’ is in the same cache line > > with ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the > > last 8 bytes. This patch moves ‘bound_map’ up to ‘base_addr’, > > to let ‘base_addr’ locate in a new cacheline. > > > > With this change, on Intel Sapphire Rapids 112C/224T platform, > > based on v6.4-rc4, the 160 parallel score improves by 24%. > > Well that's nice. > > > > > ... > > > > --- a/mm/percpu-internal.h > > +++ b/mm/percpu-internal.h > > @@ -41,10 +41,17 @@ struct pcpu_chunk { > > struct list_head list; /* linked to pcpu_slot lists */ > > int free_bytes; /* free bytes in the chunk */ > > struct pcpu_block_md chunk_md; > > - void *base_addr; /* base address of this chunk */ > > + unsigned long *bound_map; /* boundary map */ > > + > > + /* > > + * base_addr is the base address of this chunk. > > + * To reduce false sharing, current layout is optimized to make sure > > + * base_addr locate in the different cacheline with free_bytes and > > + * chunk_md. > > + */ > > + void *base_addr ____cacheline_aligned_in_smp; > > > > unsigned long *alloc_map; /* allocation map */ > > - unsigned long *bound_map; /* boundary map */ > > struct pcpu_block_md *md_blocks; /* metadata blocks */ > > > > void *data; /* chunk data */ > > This will of course consume more memory. Do we have a feel for the > worst-case impact of this? > The pcpu_chunk struct is a backing data structure per chunk, so the additional memory should not be dramatic. A chunk covers ballpark between 64kb and 512kb memory depending on some config and boot time stuff, so I believe the additional memory used here is nominal at best. Working the #s on my desktop: Percpu: 58624 kB 28 cores -> ~2.1MB of percpu memory. At say ~128KB per chunk -> 33 chunks, generously 40 chunks. Adding alignment might bump the chunk size ~64 bytes, so in total ~2KB of overhead? I believe we can do a little better to avoid eating that full padding, so likely less than that. Acked-by: Dennis Zhou Thanks, Dennis