From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBE3FC7EE37 for ; Fri, 9 Jun 2023 18:21:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 271C28E0005; Fri, 9 Jun 2023 14:21:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2212A8E0002; Fri, 9 Jun 2023 14:21:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EA4C8E0005; Fri, 9 Jun 2023 14:21:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F299C8E0002 for ; Fri, 9 Jun 2023 14:21:05 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C0177A02CF for ; Fri, 9 Jun 2023 18:21:05 +0000 (UTC) X-FDA: 80884026090.20.D2C8A3B Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf22.hostedemail.com (Postfix) with ESMTP id E0A15C0020 for ; Fri, 9 Jun 2023 18:21:03 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf22.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686334863; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IaWjrGNhqSByWN0/XqnLQhJI8MLJxrR5fieFPN9nYXE=; b=zdfaPM+B0og07dzYxNdRweRaG6LRGs/CDMg+yE5mN28Ac5hcJyjFu+ElSEPydTtlCUNZEZ K8GnxZx6HUF/ME7nU03JI3K1xO3uz1qzwtsXPYSVk8P+ii8aoxmBzi1MJTxdi0qg2xNocf fzoJ+8LBde9TcKngAA0+vaqpo2IRomo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf22.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686334863; a=rsa-sha256; cv=none; b=dpAMRv7la3OO/0/l9kPlwZ4TmnCwnm8uLzMP26T5VnFUMGViSbvlKOJ33UyCLaHiAhocQe tXDI1r6kpGfeLPxHHCOxYvCRoYl9dwgeklM5mPJtdQoV3FfrspV0Exi04qLzqb1oAfIHCD bjnEwgiUrhM/ScGAF7Z8r+mL8w0ZnRI= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1b02497f4cfso8199465ad.3 for ; Fri, 09 Jun 2023 11:21:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686334862; x=1688926862; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IaWjrGNhqSByWN0/XqnLQhJI8MLJxrR5fieFPN9nYXE=; b=aHW631yYsW04Yq9o/ITzHP9Hv9WPZS1WKqt255K72ryuClh8Ianmp7KDtPF28xRD75 sCtJti84D9wokhXI362v3QkaSSufrXr9mwMaHp/k8sPcOrBccUWNqKbqvOBt+6YZfGyZ WZlo9de1O4hN7eNllo41lfRnHoZsa7Xz2ltILrw0nSah0BYLKeg2dt475qcrqneNow3a 6XL4Tpi1qbJfceiGQxIZrLnqw1dTF46Oj2ceQypTB/xgqeuPH+1iUu0Z/ZOZWraJX4b/ HF7KK1CO7q0MM5jTiwfw6oWClJFLZKMmap3NRutjcH3lbhMIBrOIXaKtLwYe1ozIOBJL aK5w== X-Gm-Message-State: AC+VfDze5ktQKqwNo5+Sa70QSfvBDbbSHKp56QcMAmstcMlqiPAM3skJ Qws+lV0wfYrp952EZXExk1k= X-Google-Smtp-Source: ACHHUZ4BhBmkbsnJosS7fjxzReTQSXqEYezEONtWNid7NI7UpCLzdo4A4kG2zBTNRVrdS4HK4HdY6w== X-Received: by 2002:a17:902:ecc9:b0:1b0:fe9:e57e with SMTP id a9-20020a170902ecc900b001b00fe9e57emr2117151plh.0.1686334862459; Fri, 09 Jun 2023 11:21:02 -0700 (PDT) Received: from V92F7Y9K0C.lan ([136.25.84.107]) by smtp.gmail.com with ESMTPSA id b4-20020a170902d50400b001ae0a4b1d3fsm3589234plg.153.2023.06.09.11.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Jun 2023 11:21:01 -0700 (PDT) Date: Fri, 9 Jun 2023 11:20:58 -0700 From: Dennis Zhou To: "Ma, Yu" Cc: "Liam.Howlett@Oracle.com" , Dennis Zhou , "akpm@linux-foundation.org" , "Williams, Dan J" , "Hansen, Dave" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "Zhu, Lipeng" , "Deng, Pan" , "shakeelb@google.com" , "Li, Tianyou" , "Chen, Tim C" , "tim.c.chen@linux.intel.com" Subject: Re: [PATCH v2] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Message-ID: References: <20230606125404.95256-1-yu.ma@intel.com> <20230607145009.58899-1-yu.ma@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: E0A15C0020 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: bq6rjcsnoizyb7qsiefhd679mrahxhfz X-HE-Tag: 1686334863-321501 X-HE-Meta: U2FsdGVkX19TKdARLFtZ/vovnQBeq9JI6RiG+hJDmknBGVYn1Grd67y27BBXPduGb7u7MAt/23s8MXxNdcHqe7AQHY9kSn5N6SIksPfwpBOClNTOAxf8B5886N1gH86ax+g5aWo78b35oJ14uIzn8YFj8BhylGvWUvrlFhaidziIUzn/O4wye+ycsGUE8QcXMeR81hR24C7aPODDt6JZ5jugjokc5g2XLkqdZNVuh717mDx97SB5ALrfG0XAje3DXDikKloAVbfm07oiNRL0KQ3igFD3JZhDRfTTfxZc5vJK8eOKNr2PM6RePeDHwPHWT/muUP3G9XMun/o91Y5uU4AxITRNNd9qs7MP+/cqTBARsFlfOh1ZRx+HUBguTIY3mE4Gy5iDDYupJB42xb30cow5R7tZF/bNnj+EpU9PqucdX67EjA4nGtoVzIAN+nxNEjahYPqAF8GNpzop+sEuzL+iBwxih9Op/TDfBoRTlk/9G+OtNwHyx9ZG+Nx6yHygHscL4w3lXgo60PdCFKUHy1bu/J20f3At7T2SCftf27m/dWHVESHaEv9ugPLMZqsozOWIQR7orAcqMc2xEW+GjE4T7Tqf/LCThUCo/8kL95N5rD20x4b3ua/9QHJP3SsZFUjKyCybNxXzMZMV5/FpNy7N+owdOmSSxHC6mVUwhq2XvV3KIakjyX49e9oJbmg8XlpDSKhP86RaNtB4kE37AsPH+bBz16kcbSqXyDg2o2faOqmSuY7ouMeEMYNFStSisz3JN8DSuuBHSRJqvEpIDmHQGBWSznJLlnpjgNi3YH5Vjh28FWydfnqveVkCyakdJlqsnwGPyE/JWW+mJE9uzzqF01O98HTPlZMA1mbQikrGjDnXcg6C5yhhAmQOCAYyGP1bf5UC5d2Q0mB2fYSz5TsFQoWJ3LJ/5f+JsSkBMTbNXHAzDnGNFUXdKROuG0cvW2m6fQZCJdZqD/DKCS9 uhIw1tX2 wtU0D6CysJ7SdzFQsM5VFmxK72yv5EFd1LVBnVwXpNGgHASM1fEwAK2SVF9Avl3IZPl0yAE6Rus1Zalv5mk5c+WPYbef34LyAYq8zCY/NNijh9rNZChlwZfm7d5X0Oyb4GoqP4OgtscuK+6Jp/yvoJMtrtoy5DDW8ybC27J1PjyGlO9bhg9WPYeRYoUCWtc4Gg7elMo5Fpn4cXS3FU6NB7ZXSQRoxt84GLofQKt/YbARgavCkv3sqpeMxnAyx5/XLgGCf/dRjREYo01R+nQcbhOfewGW2NLdyURfz7fyplwIKAkdpgVhTcOUsjIQHqAvRt5t4PjNY4YwyJB2g2O5YnmASJuhvQCla9GDfD8Ydc7KPwXqO8z12cIAh9pcasnDprUqtswjsqPiNeDoXf7xebwvOanmaVy3mJ0bYcD52iCUxjiNNmMiaW71hFio+n5s6/mzlFvBIroCgIJE5k++MiBQRApP/viGQQAaBaO206YTDRFPe0WcJoQJ3+hjQwVYX9B8OzyQZbi2fVJzlLMOYuDPX7bl4kyylB3iO9of3vYIdC19Wnom051ti2Qv2XQ3KQsRURDPwU7MjSW2ThFP52QqalaAoM6KSTE+TVsi1td1YBGaqTaIhjrqcrt+xj4pDCaoN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Yu, On Wed, Jun 07, 2023 at 03:02:32PM +0000, Ma, Yu wrote: > Thanks Liam and Dennis for review, this is updated patch with comment around: > > > When running UnixBench/Execl throughput case, false sharing is observed > > due to frequent read on base_addr and write on free_bytes, chunk_md. > > > > UnixBench/Execl represents a class of workload where bash scripts are > > spawned frequently to do some short jobs. It will do system call on execl > > frequently, and execl will call mm_init to initialize mm_struct of the process. > > mm_init will call __percpu_counter_init for percpu_counters initialization. > > Then pcpu_alloc is called to read the base_addr of pcpu_chunk for memory > > allocation. Inside pcpu_alloc, it will call pcpu_alloc_area to allocate memory > > from a specified chunk. > > This function will update "free_bytes" and "chunk_md" to record the rest > > free bytes and other meta data for this chunk. Correspondingly, > > pcpu_free_area will also update these 2 members when free memory. > > Call trace from perf is as below: > > + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init > > + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc > > - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock > > - 53.54% 0x654278696e552f34 > > main > > __execve > > entry_SYSCALL_64_after_hwframe > > do_syscall_64 > > __x64_sys_execve > > do_execveat_common.isra.47 > > alloc_bprm > > mm_init > > __percpu_counter_init > > pcpu_alloc > > - __mutex_lock.isra.17 > > > > In current pcpu_chunk layout, ‘base_addr’ is in the same cache line with > > ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the last 8 bytes. This > > patch moves ‘bound_map’ up to ‘base_addr’, to let ‘base_addr’ locate in a > > new cacheline. > > > > With this change, on Intel Sapphire Rapids 112C/224T platform, based on > > v6.4-rc4, the 160 parallel score improves by 24%. > > > > Reviewed-by: Tim Chen > > Signed-off-by: Yu Ma > > --- > > mm/percpu-internal.h | 8 +++++++- > > 1 file changed, 7 insertions(+), 1 deletion(-) > > > > diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index > > f9847c131998..ecc7be1ec876 100644 > > --- a/mm/percpu-internal.h > > +++ b/mm/percpu-internal.h > > @@ -41,10 +41,16 @@ struct pcpu_chunk { > > struct list_head list; /* linked to pcpu_slot lists */ > > int free_bytes; /* free bytes in the chunk */ > > struct pcpu_block_md chunk_md; > > + unsigned long *bound_map; /* boundary map */ > > + > > + /* > > + * To reduce false sharing, current layout is optimized to make sure > > + * base_addr locate in the different cacheline with free_bytes and > > + * chunk_md. > > + */ > > void *base_addr; /* base address of this chunk > > */ > > > > unsigned long *alloc_map; /* allocation map */ > > - unsigned long *bound_map; /* boundary map */ > > struct pcpu_block_md *md_blocks; /* metadata blocks */ > > > > void *data; /* chunk data */ > > -- > > 2.39.3 > Thanks for adding the comment, but would you mind adding ____cacheline_aligned_in_smp? Unless that's something we're trying to avoid, I think this is a good use case for it both on the pcpu_chunk and specifically on base_addr as that's what we're accessing without a lock. Thanks, Dennis