From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68D0CC4332F for ; Mon, 13 Nov 2023 14:52:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD0306B0185; Mon, 13 Nov 2023 09:52:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B7FA96B0187; Mon, 13 Nov 2023 09:52:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FA026B0188; Mon, 13 Nov 2023 09:52:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8CF366B0185 for ; Mon, 13 Nov 2023 09:52:39 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5B8F3120896 for ; Mon, 13 Nov 2023 14:52:39 +0000 (UTC) X-FDA: 81453222438.25.7FD14BE Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by imf28.hostedemail.com (Postfix) with ESMTP id 559E5C0021 for ; Mon, 13 Nov 2023 14:52:36 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=gJ5++TSY; spf=pass (imf28.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699887156; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6HnzFUsKGlLs8f61Ya/dUj1IWTJeXcHbwSJLwNqDuQ=; b=S8MhZrwMRGFRQ/OYZsZ6Tl8gRxsYElVEDGdXQ2bD0cfvTTU/KHFwjxQg0YOdExrXHSkTUu c/fbsRs+KyA5zitt1PB2itHItluU/A+U1VB3baqoO9UUXZHfm6PBcAQMkeL6iQO7ykCA1v RpzEuEjKZdgFJyip2uLZOqGe6L1Apvw= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1699887156; a=rsa-sha256; cv=pass; b=sSHy6I3KMvZDYsn2Lu3TMqwg9SRqkj9UcgGAPEojIsfQz5oJGqnzFXiJlZmtNQDgwxWLC7 poRV54RhLoIn1Wc83ETY4ajWUE5RNsfdJlU9t7RVcn6e5UCoG1eefhEPI3NQwz37ya430N 2E34dCPYHuZBVuyCZn54hDccBymcs7E= ARC-Authentication-Results: i=2; imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=gJ5++TSY; spf=pass (imf28.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LwgEFA7gLJmt5MzoONDAYL4rVrd2JDetn8Er+Puq2Wt7+tnLKl4BvqqojDzdvfyn+mZjb10aFTOCP5GicCe3UFKMt+zuehlpCQ3rgTJiRXFbuXicqSk7/Tj+YZRE9KMZxcNHoPehKJo8LiFNlaxpbJHEt51DZhdwji79dbHwiCYA+WKo0JYrzC0M3sX830NuXyLY6cf1KqXHgVGWJLYOt7CEQs/hmrgkxQVaiDUN7DkKKtE28Kj9aRjHwUn8mksjzdgMlX+SsfyKimnqmtoSUq8QtaFQWzN61TFJoZ9Wxg+9Q9jW1pI+wJArUUxNqdOGM4e036upaIsNoMZ2LFZuag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=z6HnzFUsKGlLs8f61Ya/dUj1IWTJeXcHbwSJLwNqDuQ=; b=N2xMZ8W0W30nAqLI1qS/pz8HzSKvOlhwtbfKvTt+dSHudodVX9rkEs956/iQINdIyhAGX+7Lj1d6t9krXqbwB9pEix3AAbiG3NtrniE0IiW/y3bWQQziOctmrbo75EraiaQY8ke8yHG/KYBvyyecc5j3BkY4kRSnnzaTwhK2D3ZYYxYN8OaoNKd2d/cKcVc0pvhqKArgi0hNvSZzafv2ioh95yJ/zNRei7GibY9lm6jOIOcIjlMvq9eLIf/6Y4iDp+u5Si7DHZSYcbqW3myFOYjHULzptiyPoD1aYCHa33NoT+HDyrZyPoUY/zW1aqvAgM3DyzdVk6fOITydQuERGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z6HnzFUsKGlLs8f61Ya/dUj1IWTJeXcHbwSJLwNqDuQ=; b=gJ5++TSYy7VoZGreGLHWHtxk8PkrQgOA8qdca2xkzTeZ9lpJtymI3GnIG+HngpuTg9kS0I4svc6QEs5w+OXTyEEGviPYZwKQaRMlQg+zDYMFq8PV/xdvxqim4iey7eepnEWYook2l/jrjgAJXqIX1qs+vs2o4RvIzRQk28ISaOOHrdsDHN303cxd+hUf40YpJTCG4mtXByIo0zRZ1dVb8gxGsKzJUXR4BQ80rDTzhMxmQUKXgE7J3uKVEpY8RsgloCGs3rIXaJ9XVJ0+xxioDae0V6Jlnw7CxMk8COXk2wXtj33Y4K7ER6hj6agVm1tlIjt1EOMBMUuvS0J7enrcuQ== Received: from BY5PR12MB4130.namprd12.prod.outlook.com (2603:10b6:a03:20b::16) by PH7PR12MB7018.namprd12.prod.outlook.com (2603:10b6:510:1b8::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.31; Mon, 13 Nov 2023 14:52:33 +0000 Received: from BY5PR12MB4130.namprd12.prod.outlook.com ([fe80::6b9f:df87:1ee2:88ca]) by BY5PR12MB4130.namprd12.prod.outlook.com ([fe80::6b9f:df87:1ee2:88ca%6]) with mapi id 15.20.6977.019; Mon, 13 Nov 2023 14:52:33 +0000 Message-ID: Date: Mon, 13 Nov 2023 09:52:29 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: [PATCH v6 0/9] variable-order, large folios for anonymous memory Content-Language: en-US To: Ryan Roberts , Matthew Wilcox Cc: Andrew Morton , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , David Rientjes , Vlastimil Babka , Hugh Dickins , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230929114421.3761121-1-ryan.roberts@arm.com> From: John Hubbard In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: MN2PR05CA0028.namprd05.prod.outlook.com (2603:10b6:208:c0::41) To BY5PR12MB4130.namprd12.prod.outlook.com (2603:10b6:a03:20b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR12MB4130:EE_|PH7PR12MB7018:EE_ X-MS-Office365-Filtering-Correlation-Id: 78cf25e9-a98d-4fb8-d6bc-08dbe4582ad1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m3Wc3Q8W0YyKQ+F1/oTf+FBopA6HTtnAI6qHK+Lq+IRfMx8bBZg+EtbyD3cTASRVXm+/+qH/v6DDy03FyZIibCQTHNAb+U4z1euJ4u6TuGt68vOlXrhs0FLNpNCx5+Qo1+465ltsamabQb+LjSAzYLEb9/bJATH1AJXk/Nwwy3WNfr02es7xVG+A+Ew6Ds0xp46vBCQUqVTjh+vNgUsd8NylOMZOhvY6DOEgcYN0QgOII1hrLiJpr3zsHqkfWFpP68yJ4gwoPP7i4j8PI/GXAxtTmp7zNNJE6/rTG8T3n13nXf972gANFWKq/dbRHmPxjX337e9inKzxZtfr00iX3kdmmDepqLVy6ECRpACa6fbbZiLD0maBtqcOCxRoSDCfRqAfhPxD6owTdZSUwc4rkDc0E3W75KbT3y0NQmA+Kh+tg9wHeQsf4res100YhUilahtwW8j/DYCq4/8JzGyKUtqZzsNlc5Zzy3EK0J5iadVk55BrFVivF5Tj2PtzDkEXN34fkAGx7Jrm5eE0PjsSGbyOdzUu/BG4K3q/RpWxkg4PJffbiTQRhCgwn+im7DvOEa06kek0ArrmWS4FpUlPIo1zcQkApKYyQvdbXzaBD1fwSui/rvUTqGn2iN8bJ22rraeOkbnWKx5Sulxki3dybg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB4130.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(136003)(396003)(366004)(376002)(230922051799003)(186009)(1800799009)(64100799003)(451199024)(110136005)(66899024)(316002)(54906003)(66946007)(66556008)(66476007)(478600001)(6486002)(6666004)(7416002)(86362001)(5660300002)(31696002)(41300700001)(36756003)(2906002)(4326008)(8676002)(8936002)(2616005)(38100700002)(83380400001)(26005)(31686004)(53546011)(6506007)(6512007)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TnBwU1hlM1dvcCtuNzJvT20rM0ZhdStpUnJORVlJNTh6VXdNZkhvRnEyazg1?= =?utf-8?B?dnJsMm5LQXBzUDZEcFRrd3dLMmhFQkRFTFdlTHFzL1FaM3JWWk9URDYyRWxO?= =?utf-8?B?ajhkQWtZTEdHUUNXaGhrTTJlNUltcHFjQUdXV2x1SFFJMVZqLzBtRjJCY28r?= =?utf-8?B?RUlKYjI3b2IraVBrdlhhWWZVaEJlb08zQW5LN3czeXJBSWtzUGxjeElmOGgv?= =?utf-8?B?bUR3ekppelhINGswaDlyWUp5eHRSRkVQUEhEVTJ5aVFyUE5qem1CUktnK1B5?= =?utf-8?B?NGNGeldUYVp5MWh5YTE1eTAxUm51dE1QQitBQlMrWld0VWtBZDU5bzRJcE14?= =?utf-8?B?ZCt6QnBOdlVmeC9mYnltTU1IcmJ4Nlp4TUhvNDVSeFpLaytNdGxrbHVjaGpG?= =?utf-8?B?Uks5SjRJZ1pEY1Erdk9KV1d6SmZsUmsxMXNhc1VnaDhkZHJvcW9MUUVUNEtp?= =?utf-8?B?cmp4TlRyZkhITkpDQlhEbXNPLzV4WXdheFpuZ1dkOUIzRVR1bzRLNGZoMXhI?= =?utf-8?B?U1p4K1IrN1FxN0Jua0c2MGo2TllVMnIvaktTOE5sNGxwcy9CaVZCbFM5bytU?= =?utf-8?B?clpqSkFxOTU5YnZsWjV0SWkvM2piSnF0Z00wSng4UUxIRHJxd3dMOUFRSFBF?= =?utf-8?B?aWJVRk9FbGdoNkhtUWJwd2I5VWhxWjJKcndXQUU5NHNWUjJ5UGxmNmo5Q0Fi?= =?utf-8?B?bEs0Zkg4UEs1MTJ3cmgxNStKV3p4V0dKTWZMUDhubElhRmw4ak13K004c1N3?= =?utf-8?B?L2FmOTQwUzBzRFQ5a1NMR0o2TTNGQmpjeWhoeko1RUJLSGFGNS9ORDFYODRx?= =?utf-8?B?ZUZmYTlxU2hTTkt0d2RsbVlzU2Fhc0dpR0RLN0JIV3N0SWpvKzhSQ2ZNK0U4?= =?utf-8?B?L2k4N0k3MkVRcnFhdm1rdjZtQVE1cWxLU2xQNG1HRXE2bUltZ0hPejN4MjRK?= =?utf-8?B?V0orMUduTzc3YTRvN2E0OUVSOUtXdm9WRTFrWDZ2cExhV0hHUVRDRGFkL0ZC?= =?utf-8?B?UkRRSDJvY0tlNTBNWHpLY09BNXpwLy9ia2lpUGhPZ1kvdFluTXhEcTV0WXAr?= =?utf-8?B?YWxia3hBbWxNMWhPWUs5RUl4Wk1tVWZ4enY0QklSWmVyQWRrdHlseWdRUmZY?= =?utf-8?B?ZnI3QlFnZlhobWJCU2thcytsNm9RbW9WNFErTzBRNzFJZGU0WXg5UndlTVdl?= =?utf-8?B?bVQrdHBUVUlRKzRiTFJSckdyb3lOczdhdnFraUozQ21LUjlBTTdkNFBMdDlO?= =?utf-8?B?Vmsvdno5YmpNUWZyR25YR1A1R3dJQ1JYZ1JPSzRVZWdXTEdNbFF4WUE4cmdE?= =?utf-8?B?c2MvV2tjMDJ3bDBOSkFMSGdVY3VMOFpES2UzNE5Lb0prTHEyekR2MWx5YTd6?= =?utf-8?B?ZHZSUytsV3R4WGo2K0lmUFFIUkpERUh0Zkl3OUZuZGEzUVJTa3Fsb1JPdmxX?= =?utf-8?B?ZkVOa1k1QmFXOFVtenEyb0lZaTVSSU1EdWt0aFZQRWpQbjRzS1hQVGRrRTd3?= =?utf-8?B?cnEzaTduU3o4S0d6UkFmdjNZYjNqZGdrTW9yVFFITFhRZUw1a3hFVWViY1lH?= =?utf-8?B?TFJxK09nbUNCcE56U1IwdjdGNTNZaXM4S1J0SFB5SzREc0t6SGxTRGhpMzFw?= =?utf-8?B?bzc2NjdTazVJcXFPVnBVMzhEWFV4YjEzaGhKakZET25rR1hlVVQ1U2pkdjYy?= =?utf-8?B?MGdMVWNGeUxLY3N0dmhBRkhzOUY3RXZFMGdvUG5jRnIxYmUxOWRHQnN0aWps?= =?utf-8?B?UHgrOUZVRUxrMG5GV2FYa3JONksyMVRPSUdpTUtQNklsbDBXSllKbFZDeWRx?= =?utf-8?B?VElJMVNHRG1nUzB0ZnZNVEZ6c3F5TTZoelo5VG1UVUZSRVBhejkrcm43TERz?= =?utf-8?B?TzZqOEYyWkxjUDJEQWVudzRCU2hGbHRBWFhXVkNKcWFZS0RxamwwaVVuTWgv?= =?utf-8?B?djFqc1VnT1J6TENIMGF0ZjR1OW9QWjVOTUhOVm1yMFd3Q0ltMDVWeGk2TDFz?= =?utf-8?B?SmdDMnNhMis4NzNzNTUwb3pFZkg2OVJJc0ZYWnN3ZWNuWXQ0b0sxUFlabEVw?= =?utf-8?B?OXNteGJOQjlOc3c3aHlWa2hwTHlleXg1RCtKTTNsd3Q3aEFCNW13bEJ3QW1P?= =?utf-8?Q?kgppOUyFF7MxX0IEhLHpMk+9i?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 78cf25e9-a98d-4fb8-d6bc-08dbe4582ad1 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Nov 2023 14:52:33.1706 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7wsGP/ws27wkJWrp5LTIJdDwos4oAivzuT3LNK7HCyf31EsXsYJAf7NOQXM95xP9WmxQzTExUmEk97cB4xiCoA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7018 X-Rspamd-Queue-Id: 559E5C0021 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: hgnnf6ezhej4izumjjojyok85hocyboi X-HE-Tag: 1699887156-685859 X-HE-Meta: U2FsdGVkX1/nspAkzD/tBWXDnqsunPr2V2dKEbo+klKsS6NEV6bv+WqHkAr5A1wznDGuN3ktv4FRimXzVqMHGQUnDEhlpit8GcOjsziwkTuQHqpsHWjfZ9l2ZjjSTkFy61mj4P47ya6kDTLahXaGtwK2NbnXXT/sgRhlkzHdhE2uHvokC1lqahwuaJQaLP5JJKW4IEudlVIVSKK+YV6wtEtB4Sue3hM3i6ZeTJmolGaBtsLbUNDebmLK8jCQXDW+FopTIThuKyqcqP5u3ptBTojsHo00hYZYxv2RLN03gL71nShzVb7S+gNqUB+wunXaxS+bqvr+sZ3NCDpahHFRyu2TYH6mW180pdjX2z0pGFO8F6O2+ktuANMu+8uK0cVyI2652SjYnHCubuDhDovxs0S9l6ObJRISWdIMH2ZsuZeNYedf7ImtjouYwBXJ4I8MQxuceoyQjJGTH4CjxxLyx0wHJ4mrp+hGkM1DjP0+zJgxhahMD8YdbM9VG3FzyAY2h3UJbrEiKIyYYNNs+6gqrQR1/MIArgIHxbdVfqmNrO3L+jItInJ4fPUAhDQ5/sf50w0YI+Dobldaxjrt3+bEIOotcSQkTUdOFL6XXXAJzjIaltVEoPxF3UU1yAOOL1LToAHHgHaskuyv5vScDxbBKYugUam/zfQhQefx1VsUhUxg4IIHuBMOHGO+PwrHzYtKSie/gGWuMEGXeTMKoDvbBSsKCycEvGXA4Tq5b3sQtXO7Y8aHHmlTMKEObTIcJELM4JC5Hhvdmp7fYgRRQOIuyv/N8HbzLpiyNRl2qZROS38vMP7wWi/mr+EsGdElJUHGGU493AhsPA9BM8yiVJ360fgNvebhdBUVRHrO2u0wamZsp22CNHK+lTiE9C4eiSyQhhVtpi7cfORmavtEjtT9jdlVTm9vAt1MqhYeqyq/rOhMka+GVMLegnvI23AbKBrcL4Roz82aGdFt3Lu0RE0 XiYrlyZ8 W+8Qqg5rvN5Gj2phW9vrR3vmw3zxYaZbL/SHYHufJ11W0V4gX/wxM+1e6DjS0tIe0T3n6zVeoEwdGzyoipa6XtqmoQYgb9YV9Wrs8+YazLsyYUnPIA/QbsKzuOBFRlm7rDh9N2qyulK+iEe/5moSI8dzCLH+JQa9I0zRgHJvByvX+JR9KcMxE5fqOepX4paRSS0Xc4eHCmCV311FQz2JfUWaLAS7PSojkVP8V4hBMeYkbeewuMyRH5ZwTfrYFbJ0DwUu63zRRfuTqkWs447b2od8IFVflXp/P3eXn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 11/13/23 2:19 AM, Ryan Roberts wrote: > On 13/11/2023 05:18, Matthew Wilcox wrote: >> On Sun, Nov 12, 2023 at 10:57:47PM -0500, John Hubbard wrote: >>> I've done some initial performance testing of this patchset on an arm64 >>> SBSA server. When these patches are combined with the arm64 arch contpte >>> patches in Ryan's git tree (he has conveniently combined everything >>> here: [1]), we are seeing a remarkable, consistent speedup of 10.5x on >>> some memory-intensive workloads. Many test runs, conducted independently >>> by different engineers and on different machines, have convinced me and >>> my colleagues that this is an accurate result. >>> >>> In order to achieve that result, we used the git tree in [1] with >>> following settings: >>> >>> echo always >/sys/kernel/mm/transparent_hugepage/enabled >>> echo recommend >/sys/kernel/mm/transparent_hugepage/anon_orders >>> >>> This was on a aarch64 machine configure to use a 64KB base page size. >>> That configuration means that the PMD size is 512MB, which is of course >>> too large for practical use as a pure PMD-THP. However, with with these >>> small-size (less than PMD-sized) THPs, we get the improvements in TLB >>> coverage, while still getting pages that are small enough to be >>> effectively usable. >> >> That is quite remarkable! > > Yes, agreed - thanks for sharing these results! A very nice Monday morning boost! > >> >> My hope is to abolish the 64kB page size configuration. ie instead of We've found that a 64KB base page size provides better performance for HPC and AI workloads, than a 4KB base size, at least for these kinds of servers. In fact, the 4KB config is considered odd and I'd have to look around to get one. It's mostly a TLB coverage issue because, again, the problem typically has a very large memory footprint. So even though it would be nice from a software point of view, there's a real need for this. >> using the mixture of page sizes that you currently are -- 64k and >> 1M (right? Order-0, and order-4) > > Not quite; the contpte-size for a 64K page size is 2M/order-5. (and yes, it is > 64K/order-4 for a 4K page size, and 2M/order-7 for a 16K page size. I agree that > intuitively you would expect the order to remain constant, but it doesn't). > > The "recommend" setting above will actually enable order-3 as well even though > there is no HW benefit to this. So the full set of available memory sizes here is: > > 64K/order-0, 512K/order-3, 2M/order-5, 512M/order-13 Yes, and to provide some further details about the test runs, I went so far as to test individual anon_orders (for example, anon_orders=0x20), in order to isolate behavior and see what's really going on. On this hardware, anything with 2MB page sizes which corresponds to anon_orders=0x20, as I recall) or larger, gets the 10x boost. It's an interesting on/off behavior. This particular server design and workload combination really prefers 2MB pages, even if they are held together with contpte instead of a real PMD entry. > >> , that 4k, 64k and 2MB (order-0, >> order-4 and order-9) will provide better performance. >> >> Have you run any experiements with a 4kB page size? > > Agree that would be interesting with 64K small-sized THP enabled. And I'd love > to get to a world were we universally deal in variable sized chunks of memory, > aligned on 4K boundaries. > > In my experience though, there are still some performance benefits to 64K base > page vs 4K+contpte; the page tables are more cache efficient for the former case > - 64K of memory is described by 8 bytes in the former vs 8x16=128 bytes in the > latter. In practice the HW will still only read 8 bytes in the latter but that's > taking up a full cache line vs the former where a single cache line stores 8x > 64K entries. > > Thanks, > Ryan > thanks, -- John Hubbard NVIDIA