From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA7F5C7EE37 for ; Tue, 6 Jun 2023 21:26:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 376CC8E0001; Tue, 6 Jun 2023 17:26:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 326376B0072; Tue, 6 Jun 2023 17:26:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C7768E0001; Tue, 6 Jun 2023 17:26:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0E9986B0071 for ; Tue, 6 Jun 2023 17:26:06 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C21831203B0 for ; Tue, 6 Jun 2023 21:26:05 +0000 (UTC) X-FDA: 80873605890.05.205D4B8 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf10.hostedemail.com (Postfix) with ESMTP id C5FCDC0002 for ; Tue, 6 Jun 2023 21:26:03 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf10.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686086763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zC4QoBDS/PVcd+xuGELUqjLK2nruU3LnP3t1Jluqdlw=; b=sUZdUP7H32I93QaOyHuCWThBUV8zu4annDDaoLYvqDwe+KxQlZHoo6PSqppidJaS7GK3Mm v3M4cotJ0DL7jATOSNufBcTglrKydR2aa/mB1k0tVPlBnLPGEPy5bxNK1fNuFryuXWwhFZ W2dOvpzJoHz3WrHDRGNr5LtSBAAGVZE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none); spf=pass (imf10.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686086763; a=rsa-sha256; cv=none; b=sGmYfsP/d8dSfhFG2oKwP1VNTbWBi/J965+Dthb9DGRy+p1a6Bh088VCnkYcFn2JMkeAgL OIATIqcW/mMCP7aw+LnKwTe03WnWQ61lzq37nDdj6AkOJQT0dekQWJTukAB0kqokaPawz3 yP0myZoakTKkW6nDMJcblT/K7zBqLq8= Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-53482b44007so3459650a12.2 for ; Tue, 06 Jun 2023 14:26:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686086762; x=1688678762; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zC4QoBDS/PVcd+xuGELUqjLK2nruU3LnP3t1Jluqdlw=; b=E6tIhkA0f8TsDrRPrFNS7sJiBPrKu0qiIJdmwsxfWFCAdGK72QFtchSz5gGhx3nYme 7zVTTFcFn202HvS47eFNYeHjGlkqQ7O6Z4DIXe99VxVB3pojQtbzm8h2z87F1eyWlIzp PO8G1fm63LcVpxETHZlcu6ARNAr5NepKYzaeX1HshNdROWtGfJYKB6TUPBU94mM415XA YS/kFiUHDplBI95eCsL678IV913ymKQoFKvf0djdd3r43aBvt2r/vv0wK7shzzjQw5fY OYc8MvuYvyh8OO92H0sDukkBCg5xNI7aZVsqNiWUNuVWYVPcHJNhuzBoExSdY0zTiCvD xjHQ== X-Gm-Message-State: AC+VfDxc8WpXKNxN2Ntw4A1YXnjYZLMJGhtOh64m7KEgI11pqdgHFGSv u/lGZIAYIVOeB8qF5sWEacE= X-Google-Smtp-Source: ACHHUZ5KL2dkTrDBHB5xFhsizQ21blEeXqp67Pe2LQdYDiY/iIvys/JkvRO5W6BkaVZJEHCTVOZdKQ== X-Received: by 2002:a05:6a20:9191:b0:113:fe95:da8c with SMTP id v17-20020a056a20919100b00113fe95da8cmr712667pzd.14.1686086762296; Tue, 06 Jun 2023 14:26:02 -0700 (PDT) Received: from V92F7Y9K0C.corp.robot.car ([199.73.127.2]) by smtp.gmail.com with ESMTPSA id 12-20020a170902c24c00b001afebec96basm8992514plg.148.2023.06.06.14.26.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 14:26:01 -0700 (PDT) Date: Tue, 6 Jun 2023 14:25:56 -0700 From: Dennis Zhou To: "Liam R. Howlett" Cc: Yu Ma , akpm@linux-foundation.org, tim.c.chen@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dave.hansen@intel.com, dan.j.williams@intel.com, shakeelb@google.com, pan.deng@intel.com, tianyou.li@intel.com, lipeng.zhu@intel.com, tim.c.chen@linux.intel.com Subject: Re: [PATCH] percpu-internal/pcpu_chunk: Re-layout pcpu_chunk structure to reduce false sharing Message-ID: References: <20230606125404.95256-1-yu.ma@intel.com> <20230606192127.w66indpq74ec6eyt@revolver> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20230606192127.w66indpq74ec6eyt@revolver> X-Rspam-User: X-Stat-Signature: t1yxyau8nr9zqc79acmeji1zibqecabm X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C5FCDC0002 X-HE-Tag: 1686086763-141473 X-HE-Meta: U2FsdGVkX19Jayk2HqEkAC++Pokdxo3B1rvnKXztSEj2LKzmrIefHnGXkz7uB2BT7MsZzKu1I6GYwKum7zVcFViVNIfjpEyFVftymOeIrP3qkcHlaCKT0zrQnuFnrMyrMtRxgSMh4eqR90jx2LEMRgWU2RwMas18bECbVO43OB1wKKwhEfnDuw0DnWjN+rhambPxmXHEtssyQIkPerlfLYhhlN3phyMFd3uZRRkNA/sXnm12Q3y81IfgrSqawICoIF9MoUrin25O3L8ieOKuRx31kOzLsBxjj8dusy2Wg8Do/bqXnUS/u3+x61HpBF/iRzgDJpTe59D0LSHxO3tp6EcQTliZ/rZ4dmx10MtkX1GKyTvCLM9gCjsYlIYkzbpfIfdtHlCooLxkgVAJxjRQVhIZW8OJXcZ71ocQGOrdwvwJJOtMR0Q5sbsOMlEXSrXf14KE9tiDPpi5ASxtyfNswAFQiR7ApUk4EtmwmyTWOoZHGG2rSaQ2SeNun7QArboOCwXrPILma0N0fzOjWx4cEPTp5/4dxLdIRQFVK2pW6u1m09i8bCss1RX/Bc5zNvKc0O22uZIZrVaxWVLmTy31Ym60Bbz03Ph7SJ+aEVSjprO1IqxzRbeVdYbgqTrEtVcEUsGxblwt088dEvD3jB2xReUmiF1RV2oAEbkoO4UnmiMCgmQPdJAk+YgG5mEk7LJP2G7Brm9PcLy726Y6iyBec7qtijE1ZzBwO0G9dI3FwL98WO+CPTR1G58MymL3AZkoJT31iyRzjK6xYkMAyxHoJIAdMMqLCaBYFdotu1Kx/VxUaYvm3VHq8gyJu4BjfZXg14DYGe3ChnGQ4EZmQKzBiHBAyW7al+B1ehcET7cgqa/2OoUXx8PYm1XFsFTX5Fc7N9AXeWzHXKxG3vsS2ZuGZsantUkJ9ZIhO/hH93kuc03kkrl0Obi3lvveTdQpYh8GbROkHD1PysoemRUyyb/ Lv6qPvqM aBmu538n1ua6vrQnfRwg96wh7ZHF32DbqJuck/7C2X2RXeaEqYxmqrHKiXdIy0gCF0muEE62qh5p1Gy6TDag8nhx+CNpF90iucNcMh1SCEoKCzedbwIWC1H9hFgxxocn94xHziJXQRqny7BIfjY6q09KPxs2PXKiGvaYLiV+7vvkIXHCEqfaE+8hbVlijPDf0FpVAJmZq52Qr7cEDypFvu4JW0SLmvGreNtZfcKIW1WV5HFtzfy8d+SA/6eqOM8YI5OPM31IUKRY5AG98p5IYtVOCOZA9KR4apn/6c0O2NTmQD75qfOmxt9UQhfz+dortn9FSsGjbaKQd6uEyRtRFv10rozSUqLZlNhiTvZd92JDA7bFn5jDLFgoN0ctu4q0sbiMyGPb4RSus1aPpq/ZUnKrmgxvA2gnmKOrebi6Ij40Rse9WQBlUq2ZeB3o3NFAk8kjNNNmqxEyJ89JcztcwJf+ykNUOfVc9PyydpiTp5a7xl04rudTJK+3cCZhg1bgQZaCaqadr8e0IbLnMWcHDLbNHj10PodAL1NLPINoltVe6qTw4tXEJMMA4wN1Kj5xWDvXaIMDUo0ibIEECOQlnPa0bFQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, On Tue, Jun 06, 2023 at 03:21:27PM -0400, Liam R. Howlett wrote: > * Yu Ma [230606 08:27]: > > When running UnixBench/Execl throughput case, false sharing is observed > > due to frequent read on base_addr and write on free_bytes, chunk_md. > > > > UnixBench/Execl represents a class of workload where bash scripts > > are spawned frequently to do some short jobs. It will do system call on > > execl frequently, and execl will call mm_init to initialize mm_struct > > of the process. mm_init will call __percpu_counter_init for > > percpu_counters initialization. Then pcpu_alloc is called to read > > the base_addr of pcpu_chunk for memory allocation. Inside pcpu_alloc, > > it will call pcpu_alloc_area to allocate memory from a specified chunk. > > This function will update "free_bytes" and "chunk_md" to record the > > rest free bytes and other meta data for this chunk. Correspondingly, > > pcpu_free_area will also update these 2 members when free memory. > > Call trace from perf is as below: > > + 57.15% 0.01% execl [kernel.kallsyms] [k] __percpu_counter_init > > + 57.13% 0.91% execl [kernel.kallsyms] [k] pcpu_alloc > > - 55.27% 54.51% execl [kernel.kallsyms] [k] osq_lock > > - 53.54% 0x654278696e552f34 > > main > > __execve > > entry_SYSCALL_64_after_hwframe > > do_syscall_64 > > __x64_sys_execve > > do_execveat_common.isra.47 > > alloc_bprm > > mm_init > > __percpu_counter_init > > pcpu_alloc > > - __mutex_lock.isra.17 > > > > In current pcpu_chunk layout, ‘base_addr’ is in the same cache line > > with ‘free_bytes’ and ‘chunk_md’, and ‘base_addr’ is at the > > last 8 bytes. This patch moves ‘bound_map’ up to ‘base_addr’, > > to let ‘base_addr’ locate in a new cacheline. > > > > With this change, on Intel Sapphire Rapids 112C/224T platform, > > based on v6.4-rc4, the 160 parallel score improves by 24%. > > Can we have a comment somewhere around this structure to avoid someone > reverting this change by accident? > I agree with Liam. It was only recently percpu was added to the mm_struct so this wasn't originally on the hot path. It's probably worth reshuffling around pcpu_chunk because as you point out base_addr is read_only after init. There in general aren't that many of these structs on any particular host, so its probably good to just annotate with ____cacheline_aligned_in_smp and potentially reshuffle around a few other variables. Another optimization here is a batch allocation which hasn't been done yet (allocate essentially an array of percpu variables all at once, but allow for their lifetimes to be independent). PS - I know I'm not super active, but please cc me on percpu changes. Thanks, Dennis > > > > Reviewed-by: Tim Chen > > Signed-off-by: Yu Ma > > --- > > mm/percpu-internal.h | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h > > index f9847c131998..981eeb2ad0a9 100644 > > --- a/mm/percpu-internal.h > > +++ b/mm/percpu-internal.h > > @@ -41,10 +41,10 @@ struct pcpu_chunk { > > struct list_head list; /* linked to pcpu_slot lists */ > > int free_bytes; /* free bytes in the chunk */ > > struct pcpu_block_md chunk_md; > > + unsigned long *bound_map; /* boundary map */ > > void *base_addr; /* base address of this chunk */ > > > > unsigned long *alloc_map; /* allocation map */ > > - unsigned long *bound_map; /* boundary map */ > > struct pcpu_block_md *md_blocks; /* metadata blocks */ > > > > void *data; /* chunk data */ > > -- > > 2.39.3 > > >