From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CF18EB64D8 for ; Fri, 16 Jun 2023 22:39:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3BCC8E0001; Fri, 16 Jun 2023 18:39:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC5296B0075; Fri, 16 Jun 2023 18:39:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B16C78E0001; Fri, 16 Jun 2023 18:39:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9C1606B0074 for ; Fri, 16 Jun 2023 18:39:26 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6170BC0D09 for ; Fri, 16 Jun 2023 22:39:26 +0000 (UTC) X-FDA: 80910078732.01.D35A7CA Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2055.outbound.protection.outlook.com [40.107.95.55]) by imf29.hostedemail.com (Postfix) with ESMTP id 4F436120013 for ; Fri, 16 Jun 2023 22:39:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=dXB7bKie; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.95.55 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686955163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vvOsdAY6gGm4Rtv+3E3NpKHGFvxylQMnWXp0OvXi+i8=; b=fxw+ENbq+zdZNFlACi5IiskgUhMQzMV0fPMvcnm+o4VCWwPJftuJoc7Ko9O8CyQcan2glK Zn1IAoPmKrNp+ONVApyGK42eKgUbZDJXJlvns0d5N6nkF26zYxDRUkqzkq/D4OvNBdaSCh sEiSKWXNNv8qj5OwzoB0oh2hj3QJ08E= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=dXB7bKie; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.95.55 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1686955163; a=rsa-sha256; cv=pass; b=KoCijHjAodIuImJNqRYhEZZzzq7SyURk3dIOJKuYMoynlwKpV7OtcHQgNnbmaY3QjRxifA IxzNpq9K6tG98wyg32r2rBStIimd4w+iCidDJhfpJ3Dt5+zPFZU7uRyKRx2XGM4JqA0zba jcg9rhxyAtyuMg6OKHBN1iCawNPk7gc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nlf77ZPkJT0LXe+H+0ftqjawdPfu/t9MiPTeB0QTDVn0RLe/vl9Nq/XigWJTE7vMe1/4jNkmwkOuVNLV3EPtaG7ubX4uT47lBUlmXCM3AMocIhck7L2k+qWDSGbv+fTNhheCFdzx1/J/+5eY3kwJtKvF6K4IKYHuluTkFg9ZJN4/VLLR43ArqaS4GfPVuPUVABz3Ohp77CruHcZoustkQm2CDPffc3lnac176ZWK8u/QiPn/cYRTA1qKUjSzUPxzlYxIugXaZeD5+3ZhdQjHxZ4LMdRIX4xMWGCeMuRIuDCsYHXO2L0U6M7odImlMGtsALA+unX4Mho2LImVLJbVCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vvOsdAY6gGm4Rtv+3E3NpKHGFvxylQMnWXp0OvXi+i8=; b=iF3Tbm8qiYlNSZeZpmCGAUmvX4qKEsmdlx01dB9+KAwHdQtpLZJ06j54OG3QtvDrubidhj3swla3sTuvoGzl8QIXu6rogiLaIFGEsdb26p+x4yKhRer2ArflWSKKgFtS4GBBMYhEVIRRzqyTMljDEjcsWQ2NNdm92vbrpYYNurl53EE+1+XdLIKxy4Oqd6IRb3ofjkb/E7jttscNktWbFLgX4gewYFzP8Ei96YsXNxN/hYipIlWCQ8KUeCDCcGJMdrFbf2RAseJcVGcBF1TkJRB+lDATdt9f443wtlR7vGKoV78nmpL/+FQTXv4tbOwXhAxbC/6a6JA7wXg208mJDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=efficios.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvOsdAY6gGm4Rtv+3E3NpKHGFvxylQMnWXp0OvXi+i8=; b=dXB7bKieoVGRw6U+MGM6Ut9YF0mIHGR/WJrpEzE+jxJMVns/Iu95XCtA20dFJHUdSMWwd//mNnxwjCyAsBTsxLEBreA/r7EBD3peQqF6iCs9pyPXhr5HUZ1dmGQUZ0y8a+ZROSmwzyg+770HgQc5QbxQ8VmtsVVsRFVzTsDTl/vbXmDU7VCk7EuCl1zZk/mDlIxc+VzRYxbArjgDeA9mSxctR0wub7zejBua5INR+Rm28/oEx7yH6jbcBBPtmtSP4SEnz7Yi0CAluB3enejmYYivKaDNvYvrXsqzmjkGgAQQ5/6xif39jpstY/CuEfV/FPIbUjPViDzZNccM22yj5g== Received: from BY5PR04CA0020.namprd04.prod.outlook.com (2603:10b6:a03:1d0::30) by BY5PR12MB4035.namprd12.prod.outlook.com (2603:10b6:a03:206::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.27; Fri, 16 Jun 2023 22:39:18 +0000 Received: from MWH0EPF000971E6.namprd02.prod.outlook.com (2603:10b6:a03:1d0:cafe::8e) by BY5PR04CA0020.outlook.office365.com (2603:10b6:a03:1d0::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.27 via Frontend Transport; Fri, 16 Jun 2023 22:39:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by MWH0EPF000971E6.mail.protection.outlook.com (10.167.243.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.27 via Frontend Transport; Fri, 16 Jun 2023 22:39:15 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Fri, 16 Jun 2023 15:39:03 -0700 Received: from [10.110.48.28] (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Fri, 16 Jun 2023 15:39:03 -0700 Message-ID: Date: Fri, 16 Jun 2023 15:39:02 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH] mm: Move mm_count into its own cache line Content-Language: en-US To: Mathieu Desnoyers , Peter Zijlstra CC: , kernel test robot , Aaron Lu , Olivier Dion , , Andrew Morton , Feng Tang , Jason Gunthorpe , Peter Xu , References: <20230515143536.114960-1-mathieu.desnoyers@efficios.com> From: John Hubbard In-Reply-To: <20230515143536.114960-1-mathieu.desnoyers@efficios.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000971E6:EE_|BY5PR12MB4035:EE_ X-MS-Office365-Filtering-Correlation-Id: 46a123ac-82c7-45c3-a5d0-08db6eba83fd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ogy0J6GQ9d+p8Wlzwb/FBZxFY3AFywpUKP5CbuvCHPafFLFfS192Bp92Tk9hVtCinArA2u2VW/H22f72beHZv+Wpnn29XXvJkqL2mYbWFglNmcdZ6+uNzBo0eTKweW+QHbQ1hbQz+QJr5r5TQuSkjm+9E4hCywL/h1+ANNPapkW9O35WD/9OZKgLDW4bA8x2QbY5deearl9NOstIkfABBb8lfJ0KG9mZboy08uEim2/9gqwlDhI/WG5KstSFdGQVi0N5NAQFPEjbX98ztoQTS+xWQ/Mi/f2gFR+GxEJ4BRvJl6G8oZM85Pwz/y5tkzDQ94G/lrG6vgbqfLD7XjwR7JqUrNyX92L2N1Z6TdzlND9IKL36XLnrb1VpGnpJM53BhSM5dNwK/w4OdfKCtbKOPtPYQkB7HDexJz0lAt/t4ORPueIeUqzovEm+6Uxw2roUl0avhCr7pXDfvsCMoACABTDNjGHK1Bex+gt4MDqL3qqlOI+D1RtALb8wBS3mkqETssYwSvxmmcm3uLwyUSgIThSYcMmDMqN+u7F2nHRaUkbDh5MILZlxjpW8gRJ4g4f42QdRRIsVJnP98y6UessYGWsOZY5IBh6uokNsdZaCQrdrpAeEcALX15dgs8oIT4axbBrKaELKgnSQyOuV/zc+GvpPviCSsMhxAqXWpCloxyb0TJM0ebGgjcRvOOSRMevjP/ASlBRVUpKPDgp5LY1++qxxdzEji79ddNyPmSOX4WGBU4OPhRO0aNaeCjfUZa1pso2FioErxxca8wB6FBlVNW9YVYkBZ6idlp0ZY3+r56y+9D1CjamrAOL2LNobJfFHXlt15zzQySGH8K852ogsi7pbVVi3WH3DV0sVW+X7V3U= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230028)(4636009)(376002)(346002)(136003)(39860400002)(396003)(451199021)(46966006)(40470700004)(36840700001)(36860700001)(26005)(2906002)(186003)(16526019)(40460700003)(336012)(83380400001)(426003)(47076005)(40480700001)(2616005)(70206006)(70586007)(356005)(7636003)(31686004)(4326008)(82740400003)(8936002)(41300700001)(8676002)(316002)(16576012)(54906003)(110136005)(31696002)(36756003)(478600001)(86362001)(82310400005)(7416002)(53546011)(966005)(5660300002)(43740500002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2023 22:39:15.9530 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 46a123ac-82c7-45c3-a5d0-08db6eba83fd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000971E6.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4035 X-Rspam-User: X-Stat-Signature: 3dsi8cwo1uyfs98tm3xxzc9bbiy1kyfy X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4F436120013 X-HE-Tag: 1686955163-488836 X-HE-Meta: U2FsdGVkX1+eC6j4PSo54zJeXiqMgB1RbbfvDxOaAWWKHYjw144Foj4F6mjgQq0AH1tGLEiXFexSUfu4QAMFH5KGTv1trUTjzwf4cMPy8A4PtHAqQVtA5lsXxvaJMLCwo25WLnNV8q5LXR0Gmj/d5F/qz5v+dFvq5JkgH49W1iWw+W1CxezRw5ECKXK3+ucUhrGxfS1HnRIMBVlP3ypmG/eteDsr28bJhA3hnWfvEqyJeXie8BSqXA1HSVZ5CNV8FiQbqu10U6YtIWHdXZDeoZYbhPpxGaGl0t8ntIGyurNWYG+0Bh7klXJw1n3NBC7hKLc/DOk3CTf58GeZwA+OPHZdZdqh0mQeLGxf50l1zNNA7IijcqpfU/iMMA9pIRjrowECnnwan63ypQSdy/wSmStO8UXMwYDVjZc+z2o3y3n/VVrsNBOgPNSYFyNjNFAdUFfbYk7Ur0ND2ZDckoCfe+9Bt5mzhyPLdOydHNphGP1KzYLrfQ0AFG0WX1gVGuU03aU2IpVkrKWW1DqsnV8dmEWwgMQD1oH6Gh0/reUz2ll5UXPV3nxcr6q0+7h7SBLkmEy2WdHPyY+CwyWuBsOu6uWtv5RK+3yIafRVFVBzrXjvx/vPaBjM/6q/eUcrFahHVXk4GVTniaf8pu98sppj72V21tb8+B6MfGW6hNrujNJmGRco5l5/NLzzaJFjANUmU4wQ4hhiF6mbsq2DSWVp5d4EpZ8w+MectxdTwlbHPKsv9sItlpPBHa/hN3l0dYGTmrGwHgiH4UVj2N6Ou0jVrMMQIWdmSs4mQai7F0ccRLLDNkGQ9Tr/oWZOljmJhV+g/NDqS20iqDa26X3mkiomXBT95sa5BJHDeNK8g5JOvePxfVMteEIYTa7KN/2pGA9kRlto0rqN9cevFqzoKYaFg5SERAIobxCtbjPEvvvnClyWq48U2phCiQV1cgHTKgCQUzM9IKXJuincBObWcxX Dqe+4sWy rm+07oMJKFMOZoHsM00f/Uvq8a2Kmi/bbmuANYMZpSImo7PwSjmzFU5u15hXSfPC2PBpYPv687uBnoBo0HZJhHPtnTP9YxS422eOtPQQ6UHurJvf01NOLeTQu0HiEBGOLlN/mxO1dbXc7A6ARUJDRrSjr8pVkvlrOpA7vyhSzzjgiEfomlsWJlVbv6iDXDhCPV3eF7m1XtpEz3s/123+yqGGJOhiEGOdIgou+3Udj4p7NOKxqI6ENinuDOzz0pSWYKA/V7alRQ/ElP5dI06G5kFX6o6qV1wrdtyCDGmlqDxpN+xaglWstjYmCOXn2L2trwsBucX3/7HfPpMk3hDJA548CYp2Pk9pchId4674xfxITGdyePgBWuT0qq+QQ5Xp8k1Taye1scM1K8urrUve9tNTwW3Xl2pheBDcnJoy008nViIqhr/RwPtSjL0/ZXpq+P4NXTmSp+K/xMZFFJjXbeGKn+vbSc+VBAc/vAb+qtycVOpqcpqHFardhp/j6mRu0Ih8r7fb2G9JQ+I6I6QzocB6/k3PBnPC0w6Tc4obZ3zzLgUaarJ0te6aCu+maEFUE72bE2UrAzwRPxShFWvtrMgt39auFtZqxiO2a9wCsSzmQnNzsxbHcdH6Qdxyg/p57ThlqQdIekH+NvKKaTeLHO/SKyjT5F04iO7wLVeA/PX3C77lOe5ww++FO/xrpW54vASTy9Ru7swPFQB4jW+LUbIpdoA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/15/23 07:35, Mathieu Desnoyers wrote: > The mm_struct mm_count field is frequently updated by mmgrab/mmdrop > performed by context switch. This causes false-sharing for surrounding > mm_struct fields which are read-mostly. > > This has been observed on a 2sockets/112core/224cpu Intel Sapphire > Rapids server running hackbench, and by the kernel test robot > will-it-scale testcase. > > Move the mm_count field into its own cache line to prevent false-sharing > with other mm_struct fields. > > Move mm_count to the first field of mm_struct to minimize the amount of > padding required: rather than adding padding before and after the > mm_count field, padding is only added after mm_count. Oh, and I almost forgot to add: Reviewed-by: John Hubbard thanks, -- John Hubbard NVIDIA > > Note that I noticed this odd comment in mm_struct: > > commit 2e3025434a6b ("mm: relocate 'write_protect_seq' in struct mm_struct") > > /* > * With some kernel config, the current mmap_lock's offset > * inside 'mm_struct' is at 0x120, which is very optimal, as > * its two hot fields 'count' and 'owner' sit in 2 different > * cachelines, and when mmap_lock is highly contended, both > * of the 2 fields will be accessed frequently, current layout > * will help to reduce cache bouncing. > * > * So please be careful with adding new fields before > * mmap_lock, which can easily push the 2 fields into one > * cacheline. > */ > struct rw_semaphore mmap_lock; > > This comment is rather odd for a few reasons: > > - It requires addition/removal of mm_struct fields to carefully consider > field alignment of _other_ fields, > - It expresses the wish to keep an "optimal" alignment for a specific > kernel config. > > I suspect that the author of this comment may want to revisit this topic > and perhaps introduce a split-struct approach for struct rw_semaphore, > if the need is to place various fields of this structure in different > cache lines. > > Fixes: 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid") > Fixes: af7f588d8f73 ("sched: Introduce per-memory-map concurrency ID") > Link: https://lore.kernel.org/lkml/7a0c1db1-103d-d518-ed96-1584a28fbf32@efficios.com > Reported-by: kernel test robot > Link: https://lore.kernel.org/oe-lkp/202305151017.27581d75-yujie.liu@intel.com > Signed-off-by: Mathieu Desnoyers > Cc: Peter Zijlstra > Cc: Aaron Lu > Cc: Olivier Dion > Cc: michael.christie@oracle.com > Cc: Andrew Morton > Cc: Feng Tang > Cc: John Hubbard > Cc: Jason Gunthorpe > Cc: Peter Xu > Cc: linux-mm@kvack.org > --- > include/linux/mm_types.h | 23 +++++++++++++++-------- > 1 file changed, 15 insertions(+), 8 deletions(-) > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index 306a3d1a0fa6..de10fc797c8e 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -583,6 +583,21 @@ struct mm_cid { > struct kioctx_table; > struct mm_struct { > struct { > + /* > + * Fields which are often written to are placed in a separate > + * cache line. > + */ > + struct { > + /** > + * @mm_count: The number of references to &struct > + * mm_struct (@mm_users count as 1). > + * > + * Use mmgrab()/mmdrop() to modify. When this drops to > + * 0, the &struct mm_struct is freed. > + */ > + atomic_t mm_count; > + } ____cacheline_aligned_in_smp; > + > struct maple_tree mm_mt; > #ifdef CONFIG_MMU > unsigned long (*get_unmapped_area) (struct file *filp, > @@ -620,14 +635,6 @@ struct mm_struct { > */ > atomic_t mm_users; > > - /** > - * @mm_count: The number of references to &struct mm_struct > - * (@mm_users count as 1). > - * > - * Use mmgrab()/mmdrop() to modify. When this drops to 0, the > - * &struct mm_struct is freed. > - */ > - atomic_t mm_count; > #ifdef CONFIG_SCHED_MM_CID > /** > * @pcpu_cid: Per-cpu current cid.