From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1091AC369CB for ; Thu, 24 Apr 2025 01:49:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 223F26B0007; Wed, 23 Apr 2025 21:49:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AF1D6B000A; Wed, 23 Apr 2025 21:49:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF2026B000C; Wed, 23 Apr 2025 21:49:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C8F086B0007 for ; Wed, 23 Apr 2025 21:49:47 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 07BAEB862F for ; Thu, 24 Apr 2025 01:49:47 +0000 (UTC) X-FDA: 83367256014.13.53C38D5 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf17.hostedemail.com (Postfix) with ESMTP id 9B3544000A for ; Thu, 24 Apr 2025 01:49:43 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=MYYkVp18; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=uKCnBtmq; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf17.hostedemail.com: domain of jane.chu@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=jane.chu@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1745459383; a=rsa-sha256; cv=pass; b=RuXJLn9sp0HaA+8h0VIa1OQZ7uYTqUYLUka2rgH78vS6Ro4BOHvpOOaXl89kh4kVPo7H6X 6aRUtSuFD9L+S8mC1VP/bviHBdqm4YpyrUYXEVAcHU4pJ46PlJAbn8DhPCE5HHBw0BsRjq cpSrz+mRCx3QUzKzdZcZHgHJooGNzFY= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=MYYkVp18; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=uKCnBtmq; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf17.hostedemail.com: domain of jane.chu@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=jane.chu@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745459383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nYu3qUJbhusnloRkvjRMq9LbrDuGr9+VaJu/yAE1kuI=; b=w3BjXXFaKMqE5MSjqyzhO5aoX7TCzjJypdXQZ+XRRlZsWG3RK+b8Y97A+/D43Chy8iDOKx sx0487AOIT7PEQHERHfv4UsCYS97tDmgaFiO5yLi+0n3OMOhIaPK5ewTjPaRGpCym9KVTd 90L0bqWszZorggBuwOxEF006SU7oDL8= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53NLMuog013977; Thu, 24 Apr 2025 01:49:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=nYu3qUJbhusnloRkvjRMq9LbrDuGr9+VaJu/yAE1kuI=; b= MYYkVp18WUF8MAUncs3G3iOwpH+rQqMJ2NsBeMhciViiCiKhhNPYO6w6GI90MzPW y6HvIyY35pyX8bbj7kFXEmJQZc/eNUW+yOorc2soEwgPyORmyoxEWXdxmRAsu+j9 nEydXEU/DX7mcfYz4FN1E+HVwRXeosemWzp25ujzthcBM2SZITCOkLJnrafbpOlc GftZ6s9OEyLSirP/XlT2WTe4RV0F6Wx5az/o8P8MK2MKnUCkCX2tcxTEFFseA3C1 FKWDD4k3hk/8pneuh0BM4rTzl7a5c6xUL/z3Q8ezgOLW6ZVr5K+Jw8KkMnq7w7C3 fVf0Fr5F0CM6iELs2CgwUA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 466jhdjtqd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 01:49:36 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O0MnaG017418; Thu, 24 Apr 2025 01:49:35 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 466jvfwr1r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 01:49:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GvqIvExkleUkEodH1lffNcON6GgmLeu4cHTnUZvS0sOVrThNdwzU41nBFQLEV+33aw/6ZFlYfjUpKubzZWgA571MChxQfhl6Mi4PKyr6hni0BdwcwZcbF6qwKh0OBOlxU83vQptokuXDF8TQpgYKC7u0KqHEkOOlN4oR5dyVjsiCkdmPMXVVnhZ9jnGOkPYSuUR5l23xGB6D8HEE2xKjDbWLDP4FZl/SEQusp/uJSFoLMX5S2Gbij1d6GpPtpWb2MDb6COPFmke5f6KqRa1J3o2ObCPwOn1U1kaqvuCyWzehTsA9l65EGORBe9zB9gOtdR2MBpMtTmyKlmsoZG8byA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nYu3qUJbhusnloRkvjRMq9LbrDuGr9+VaJu/yAE1kuI=; b=QBSGxHtJh1Ygc3peBp3+ogSgWzo5bsS1moHYjWX0Apvzn1Pg+xHy4xqN2/TdX7xWVUTFVH6dlC5Rs3LS1tJPeJ+yhDQfkjoYmbzdIdChvYgiShazECpkxboQcn1lFlOc2DwZhIhSjb85UH8SswcqXvTFLACnFbDfikasUJWp3DuRDWOP1LHpCgl0K95hZAUvzONGS591WT1FJoXpYtEHW4YJ/MMBsSGiAFsnLWuRTuB8X1eNJxdqeOOrpgMm23Tgz0G0GRZVvw4krAtAHPbctNVZMM9xdGPdXywqED6gMSRvU4ONGwhsuC1aOdw9gfi8C711v2jG+rTgmpAWTkQYhQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nYu3qUJbhusnloRkvjRMq9LbrDuGr9+VaJu/yAE1kuI=; b=uKCnBtmqoaCOKotj2J0dZ37cyB5hIRxISQRubJ5R3A1eh0X5Zoke2K6RxmU1wqk9QXY5j6R93Hj12iOwUj4N75BmcBp3a2yTq3vFK9DUJylnlW36Chu0rcmeVIsWLWAIvJthvPXWVqNff+Ub6o8ZxmTSTlbggHyxm/1N6PdwxH0= Received: from SA2PR10MB4780.namprd10.prod.outlook.com (2603:10b6:806:118::5) by CY5PR10MB6024.namprd10.prod.outlook.com (2603:10b6:930:3c::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8655.33; Thu, 24 Apr 2025 01:49:33 +0000 Received: from SA2PR10MB4780.namprd10.prod.outlook.com ([fe80::b66:5132:4bd6:3acb]) by SA2PR10MB4780.namprd10.prod.outlook.com ([fe80::b66:5132:4bd6:3acb%3]) with mapi id 15.20.8655.030; Thu, 24 Apr 2025 01:49:33 +0000 Message-ID: <914dfa88-d36c-44c2-a7d6-22f6fbd2b86f@oracle.com> Date: Wed, 23 Apr 2025 18:49:30 -0700 User-Agent: Mozilla Thunderbird Subject: Re: Report: Performance regression from ib_umem_get on zone device pages To: Jason Gunthorpe Cc: logane@deltatee.com, hch@lst.de, gregkh@linuxfoundation.org, willy@infradead.org, kch@nvidia.com, axboe@kernel.dk, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org References: <20250423232828.GV1213339@ziepe.ca> Content-Language: en-US From: jane.chu@oracle.com In-Reply-To: <20250423232828.GV1213339@ziepe.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: SJ0PR03CA0223.namprd03.prod.outlook.com (2603:10b6:a03:39f::18) To SA2PR10MB4780.namprd10.prod.outlook.com (2603:10b6:806:118::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PR10MB4780:EE_|CY5PR10MB6024:EE_ X-MS-Office365-Filtering-Correlation-Id: 0f49b051-4f14-4917-390b-08dd82d242b2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?b0ZiSWl2d05ITnBBdzR4WTFUM3FXRmIrOHIyTHFOcnhhc1E2YkdUUW5IRmNr?= =?utf-8?B?dEowSFhHbFFnMXZQdkNleVU4YlZLRktQdy9vUUtVYWFOTzZNNllWdkFLNXpv?= =?utf-8?B?WE4wRlVpdGxBVnF4T2doVmdSdURJalFsV1VQQ09hNjVIc3FQSVdSY1pCYlNF?= =?utf-8?B?My9wbDN1NXp2eFlpaHRMaWxQV1VBQlMvenJxTHZ4SkZYcUI2Yk5mT2k0RGsr?= =?utf-8?B?ZWY0SW5HSGYrZzZORDU1TFdpY2hBZFllMzUwck5rSFVpNjJsQ2NwR0t6OE9W?= =?utf-8?B?S1dMWm9EUzZrYXJ2ekFHVlk1cDB3ZFU2R1JsVk5JcXJ6RjRDWElKRUNsWHpP?= =?utf-8?B?VWE3Y3NjZ0Q5R2p0bmZ6RDcyQ2hkdWYvUmJIK3dCSEFIb2tQdTV0bUF3SC9N?= =?utf-8?B?aFNwZkQzRUpsajVsU3NpTXQ3ZGZJWWE2ZVJNNTdwYmZZUnRvK1daTXVBRGpo?= =?utf-8?B?R3pjeC9KLzdEODltTy8wb2ladHJQZHpJanJXbUxiSldGUnVmU0l2bysxbUl5?= =?utf-8?B?b2VNeFdHc2tMd0VITnV2d3dZL3h2Tjl4UnpyS24yRmpoNkIxN2Njekk3WFZM?= =?utf-8?B?c3g1M2NGRk15RjQ4bnZxR0lPd3ZNc1VUTjVJNWRZWWlBSlZwSjFUK0REMHVV?= =?utf-8?B?Ny9xbnc2RVVYZHVQb3hhc1BKUXFtT1ZDYWtacFl6MEFvWnNTWmtqR1p6WDBq?= =?utf-8?B?MEhmY3BVS2hGN3BaYkFwbm1MYkRRVVNzSHMwc2N2ano3SWdiakJ6RitqdCta?= =?utf-8?B?ZEhkd1JQa1hXSGluL0EvamZ2WVdqU1gwZXBNclhwNHFwa3ovaEFyQnRjT2h1?= =?utf-8?B?a0VlWCtPenhlcDQ2VWx3enUyM3VaV0MvaXlmaHB5TjJGMTcyMzlORXhOY3F5?= =?utf-8?B?Q1N0ZHBrOG1mYVVSMVo4OW13ZHlQWDFKQnU1Y2hNN3hySDdUdnZoMDYzVklT?= =?utf-8?B?MTNybk5rL3hZU0RER3hxWmJLWDlZTEp0UDlZUFBIRzJtUUthU3laUlllanpr?= =?utf-8?B?UDFuVDNWUE1OWndGZ1BIRWIyRGQ4MGpTcXFLTVZrYTRPRGVBUVhQQ04wakN4?= =?utf-8?B?ZDBjdlJwM1lJd1ZvK1lKbGJCUmFhZllzZWl4aE9Oc0dzM284aVV5UDhJT3lD?= =?utf-8?B?OEN1enhmTExZaE9mcEdFdWtEZFg5WGVMMUVtK1FXVStZZUZmcCtrZ1E0UGtu?= =?utf-8?B?RFZjQmZOdEp2blBxZ2RINjNVdW5aWkV1REp4OHp1NHNrL1BBVGJHZjhZUXdS?= =?utf-8?B?ZUsrUWZJSXBGT0Y2TURNTlEwVjFDT3BuVHBpKzlzOXdYUXNocy8zSTBhazdW?= =?utf-8?B?dWQzZUtHS3B5eGZxY3Urb2F5ODRseXVxdFhIWjF1WGJvQklIQzIrT1BwM05m?= =?utf-8?B?Z0V2QXpXRVhiTCtRRTdnd0ZpeVNSeVl3SklMY3hqQTNZaFMvOWVVOHZJcDFy?= =?utf-8?B?clBQZFhoQlZLZlRCNXduNnhGcXQ4cHVwY21XKzdSbHd1eU50U1pRV1k1REgx?= =?utf-8?B?cVl5NEZaM3N3ZlNnY0t1VVhFa2hFRGZGSWVialJ0L0VjV2IvNTNyb2pOTTNw?= =?utf-8?B?VXJrMTc3eDFWbmdKZ1BjSm9FRFpKNE5hSU1weDJKZmhPZnV2UWNobkh0VTZL?= =?utf-8?B?M3N5OTkxZlFoV1UxanJRU1NZR2x4ckhMa3hSNy9KSStmSkhiNU9wcVIxSHNS?= =?utf-8?B?WmVxVHZWQWxDOFUvSTlUbFpDWitsV3JJRVhsV2ZEeGxVUDZ2dlFvSkNDTm5C?= =?utf-8?B?bk5tekI2dUFaMFBOdUR6a2p4aU0zUm5qK3NUY2hsVTl4aTg5NWpqNHJFeGts?= =?utf-8?B?cVNuSzk2QkQxWEROR3FSaHBhM0s5d0grQlJvWFQ0U3FqcjBXUTFEam5uQ1Rl?= =?utf-8?B?RWtJb3NVT3VyelVGNGNOYnp6ekpheGVxQ1p1VEJjVmR5ZWxDZ0Z5ai8vMGIz?= =?utf-8?Q?Nii/LzCfim4=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA2PR10MB4780.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Yjc3UERPVFNWNXhIUHA5NGREbk1RZUEwQVgvQURKUWN1ZldFd29GazI4aUJs?= =?utf-8?B?cFI0T1NNYllsb2lxZWNTeUZ4cHRqUmJiZjdwbEdaUUcxa2h4WnFPcmhQZ2FT?= =?utf-8?B?OUdBV3RsSUNMQ2RaUm1DSnNXUitaSk9GejQ4ZS9LVmllQWxqVWZuTjNMMTRS?= =?utf-8?B?QXBPNzcxS0ZxOXVqRkxhZVpTRklsTHJKaGlMajVxeFNGdEFNL3o4U2ljd0kx?= =?utf-8?B?L0ZpSXY0Wm1idjlJbm42aEFZY24vS0JocXJydTVTK29UR1Eza21BODEzWjFT?= =?utf-8?B?Ym5GOTVPdm9OSXUrc3pybHpuMVZlQjVna2FHRndLVy80a0NFVUJIUlF2Wnl5?= =?utf-8?B?Vm9naTdTRjFnZWtKU0FwR043S2liV0VtdlludDhTNzlyaVhHQStmcXJqRkNp?= =?utf-8?B?ZDBBVlFtbUllbDdTT3BQVXlxMUR4cVV0VFd1TmQrcnlmbmh2ZW1oVGwwMGcw?= =?utf-8?B?NU9vcUNBaGgxaDFwczBVdzNERUtDSTJwclRyczBKdk5taHVZbGxaUXpWS0ZS?= =?utf-8?B?YkxyREEzU2VNRjVxSkZ0M2hlUUFHQjlhUnRRM3ZqeVNLWklCZVNvY3RoWStp?= =?utf-8?B?cFVrRCtpWVRNSjN4Z1RQYWlxRnlGaHRhZDdtb1RLU2h5TXAvSmxYaVZPQUNW?= =?utf-8?B?UnlacitDdGVUdVlRN01yazNRKzJIaGFHeWxqMFNiUGUyRDJNcjhMamlOMDZy?= =?utf-8?B?WXFCK1IrU2pYUm4yTlQ4U3p4bDZGTmlPU2VUUTBwZFRGZEtGYzJQcE5LS05t?= =?utf-8?B?OVJYSGVYUHR5SXVJaWxhL3pTM1craUtiNnhwZ1RVMHhCL0puVWE0dU1RVVFu?= =?utf-8?B?Qk9OdVA5dUM2bms4MHBRUVlGT2Qxd3lZTmFzM09ZbGtqYThoSGJ4bzlrb3VE?= =?utf-8?B?Q3g2THVHRW81dC9yQjhhZXEwY3BabWJFZGhpL2YvTUk2SExGS09FRjBDWFVL?= =?utf-8?B?TDQ1U0F0QS91VmkrUHcvU0pMTjRQbjNSVWZXcXkxSFMwR0hlc3E5S3VIUG51?= =?utf-8?B?WWg2bGl4ZXA3UjV0anFlYUZrci9BL3hOMXpybjZ1L2haeHVmUW45eVA1T3hM?= =?utf-8?B?ekF1bVJTYUZ1SzZVRG1JcjdKVm81Y0hoY2NDMjRnc1NRZHFXdmExVzNsbjBk?= =?utf-8?B?Rk9KdVpoVmhUK2pCcWxkc29BRFdRNE5hRk9kUVZ0c2VPOWErU0JpMFFZV2c3?= =?utf-8?B?eEc3VHAvV1QvZTNLc2x6N0dNKytlMlY5OGVJUk9PYjRCeFk2bjhoZEs5SGJz?= =?utf-8?B?V0dWa0hLeHlxZURHcnd3TE5oRjNJVmhIVXVBTmY4d1g2SUlPdTcveXFkSUpG?= =?utf-8?B?V3NmU0ZhQ01JRkNua1E1UTg1WGlBV1ducm5xYW83b3FGdmJnVkxjeVAydTN6?= =?utf-8?B?VUt3SUY5SzExZWZUcTc3ZnczeVFWZGUxQzJBaTBVbmVXZ3dzVVYwOWc2MkRR?= =?utf-8?B?VEpDamE0SGN1aXZ1elNXYjdYV05jbDdST21vc1NyaXRKNkd0NXQ1UFhTZEVt?= =?utf-8?B?NG9tcU04MTVvZTV5dURlUFVxNkdmUDNIOWJob0pYdHJOaGluYTRIY1JyRXFW?= =?utf-8?B?Uk15bWtoV0pIYWJlVlJ3aUl2bm9ucmVJdndDbnA2UE1rckYrLzE4RkU2L0dT?= =?utf-8?B?VUxXZXV6ditmenM2Y2tzTWM3VGl3QnM1SWsyV0V6VlN6cGsxRmtzcnhUd2h1?= =?utf-8?B?N21SVmRlcFYzRnpybzdGWG9pMDlvM1RXQmpIcnJSRGZBZEVVSVdseHJGRjRy?= =?utf-8?B?TTdGNzZmazVSb0RaczRad2xXTUt3Q3RMMTBUeEtjbWR3OWIvOG9lZVZiRFJX?= =?utf-8?B?U2E5QVNDS1J6RkZkMk41OGt3d3QzM25lZUtzOEd4c2h4ekpxczhjNExPT2Zu?= =?utf-8?B?aHpYMmVZZmt2SlJURXlKTC8vWDdGRzZTb1RlT3g0Nk43cjhsZDUrMkFPaUlt?= =?utf-8?B?YlZWT28xcTYzcitkd2xleWZUQVJnYkFVT0xXem4wa3RHOVYxbHNzMm9iK2Rm?= =?utf-8?B?TW0xUy90VmYxckcwNHdacUs0NUE0NzNjN0lzcG1QTnpCVzBhTWpoQW5CcUhW?= =?utf-8?B?MXAyTXhWWnM5YlN2dTRoUWdCVVlpWHBSVEFab0NjSlVqSzZ4LzdIcExPOU45?= =?utf-8?Q?cReQWMzByYhOJ0YcU7aADS7HR?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: sTCsRy+iAzGM0cB2lNEk7FoMf+4y1Nw9EGwHrFl6Aiv6Syx8Ti5AKKN3gqXvpEzlBssPrh19wX84fKpzwJ0QvvA9nFUyDp0XR5ZLYc/0PfOr1ZM2CdmAT5J0EgDIMmLukcFH6JQbkQytMzRejvwb4BK2uUNHIJdo/nilG2bkuYL/MwJ63VnXvnuIjQe3Q1cdsyVJ9pdWWuJVK+KS0hVGlCYau1m75xReaE3+xQG/apz283BZPOuPp4g8mg8rOgvna1APN4fHE4a64/vFNA8hETfo/zPPpelj6ehYIJNsFyebL65v78wpK/ErRRLbb3cGLonSffbK5u58PuhC7NAztuZDcwfdyWT/BjNBAbdihgREI4LcqWr5mbUVk1YbhPGuHhPGFKQ+df+U1sNQY2FHIL2qY2bI1rGWIfBxpnbFfU9+K2mnk8EzIUrNPVRylKxf3IZHHBr0giU6iOrFa4ZprcrHaNQSE5GFYFQ+WS8BrpP08rjBYXq2kA/Zqn/csuFaE78mDWOKJp+j0o9HuTsNGgocW3wQAnBmMohsof5oQSve95hqD9vzzMSNoqAuxBSXmqr9xZd+d8y7tWrZuY02pacEN4zsmSDD2vQp5V7Y6LA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0f49b051-4f14-4917-390b-08dd82d242b2 X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4780.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 01:49:33.2234 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YyWvy4GeajadcdfWEW/BBiuqhru4cJ1yrd7KzAXWGGCh+nQ+8pT7WtvAjFrMppx4ONHmx6qPOjA/1QfRt1Gx9w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR10MB6024 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_01,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 spamscore=0 suspectscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240009 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDAwOSBTYWx0ZWRfX5PlOThrX7yeY Zs1Aj3xeIRLC8hORwKGYJ+m1r+s1V6SxNR4smeU5Ps1dzfDeiNZNu9XxXIgC4SQtov2EfODspRu h5u4+9hUUJvMaafyvZnlbcvQtW8A2fYNWXmCfVTzCB7btsVAPahZmHZVEEw+Hpk2m4StoEalFSp 19cZyt7xH1Y6xTGCqIPGsrmROHyCBN24xRbLAh42X9qAAWi4kRk5bzCRewvBuLComWLcljroGm7 l0M4iK2qTRpnjL4r+xnQAUDhu+lSBQ8bK8dEgA3lIFQglC5RDOqbKTQH5nGqPP2tqNeKtpOv4um 9XzYCrctio7xuMgao7tHweV+XtmFDTfXxjHzxpBwrx2e61xjomUKS+gCeLU7BH319sTB1KnfTP9 6IWiNTkb X-Proofpoint-ORIG-GUID: rxo0B5AohoNfJ67Rhegdu582KtUyhw5m X-Proofpoint-GUID: rxo0B5AohoNfJ67Rhegdu582KtUyhw5m X-Rspamd-Queue-Id: 9B3544000A X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: r5dokko4xq7ys7g1op5u1ths5g164djx X-HE-Tag: 1745459383-850088 X-HE-Meta: U2FsdGVkX19oQ8XnM1XPF/sViNO7IavA9KPzulsB51+y2wxLc03IJc1Gs2IOHuqpC2oSvlEi/Wj/wIn+JTO21Q6c0qOu71nO+nx0vQeza8d3QKdMCFtgOv8VAxwK/jrziXXcvmFRectC59BnJ8t+Rn/XyGlEYvi/RmUsyDJluk0sj0RoVKuE9L8YaTDN0WyaIbQUhx8NUCY6DS6PEzgTFph1yv2P9VoIkUn7Vbin15MzIVZsNhSxoGEkcOdrcIj2wyDLqDIVBimoYlrGymQdQAK4pGh9aEPzJc8QhNwcK67RlOsf7AhHLvoJ+OcQrB8SKHh6AuB5M3AC00sggO/3Oh6YKHBhll5MgYeDw1bxqClVbb4rKkGAQNp+SVcxgPszgJRErdsn1UBCR25XQm1xotmj728fl0rcTY4vLQRkGPKP90asiXeqIwrHzGgpq8ds3szytabwrW1LTRgCsP/Giozmr1Bbu9BPCdj/PrRQ5Im08JnjTDaCTvqrzdQ5GLa0gWPqp7/nvLkaTSakYv8aYCO729e80gl8wx6X/uKXix++fduxgV0emSQleO9WL8EoE5ELHFLPQBGsqXPIed8FcZANk6u5xDHyR8N8ke09Pqf8xxCbUgpoWcij0v+hFBgr0N3ZsUJltBzMFHhys91blOTFQiDPzBN2jkBR/sEE12HEwGKcR1KTkh5zEc6G2Vj+xnOHZaQDwN6r9h9IDx/KyPA8y84NYO74JwPdXibCfc49+KDyXtBKX9b4tDYvYogNZ55GPJ8E9rKiVh4/lZIfk+56sEk/YYGTnvwNqmNvYCTM9wIZfYbgsOGnq590veTCcdaJ6E+bkY6TWQAeB25aTe51XVg1PaVWuDXAJmQOixSTVferG8/zskTrpzp6uZ36HDecniQNomsHWxdRaeIcskPw+t3NjcjuqHtjjhXJ2dqhXVKRdGV7Naz2pUMBZuPDhBuDe+rG4rkq4z6F9M8 NW/FJDq1 wxkGs0XXU9bwwi+WdnFMi6ltVwVzC2Ul7XUGDlpwVER6dk8h15KDpWpnxl9QQJ5s78Rx84pKQLxFpMJ/FyxA4AQZcRgZ4df1jHVJJXqrMT5gqlbsPMM5VMgY0/atl7+ltQFNrX8TtYOEM82Vx7Y8Zj2GNblgMApEUlWbTPKCLa/ahRWR4p2DMHX9BtP39+XU0dqvoZqmV4z1RkcryQKiscsTXIMrH4zqowrkZhCHqozXvwur711vLBcejyu50J5Eokjvd4wqXg6W0/8KFN8IIIbS+tib54qNZ9hzR2FrkPj/zdhBuE0mc4S1a+R0qBY0pcA0eCUnkJbbqZOhexLJW+PoXSHj7Qk6bGj7oGdga666fNvfpUdpIS6WnzxenKtiwnJ9vGNqOBLdrDOL+4X07mCeS99FudsB5NMqcCNevjQ9ZYQX6ah2Q3MqINKyCRR0oWOU9RHfr1sp1aRePHQ82TpF5gtdLiCmP5p5lDX4A4k/YMjH1vRjP5vrFz6T5AtlP2xdlbeKm01wUIQGTShlRxFignriYvbEPnpfKWzavrh+IL7Zq1bM3c8vTHsqCCdpuKyUH41yNYutTArWoPt9zudFoMnC5qAOh6SMZjxv2/jfWd8WzWRwGl2AJVRGQLBS/dvXDf9V3TGjgryje88sqEJmlFDxFVDTGwHtWRJbMvy4FgDSoIA/kC23sPRSUN0ideLdYU3OJ13Wl9W95NXY9A3dl3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/23/2025 4:28 PM, Jason Gunthorpe wrote: > On Wed, Apr 23, 2025 at 12:21:15PM -0700, jane.chu@oracle.com wrote: > >> So this looks like a case of CPU cache thrashing, but I don't know to fix >> it. Could someone help address the issue? I'd be happy to help verifying. > > I don't know that we can even really fix it if that is the cause.. But > it seems suspect, if you are only doing 2M at a time per CPU core then > that is only 512 struct pages or 32k of data. The GUP process will > have touched all of that if device-dax is not creating folios. So why > did it fall out of the cache? > > If it is creating folios then maybe we can improve things by > recovering the folios before adding the pages. > > Or is something weird going on like the device-dax is using 1G folios > and all of these pins and checks are sharing and bouncing the same > struct page cache lines? I used ndctl to create 12 device-dax instances in 2M alignment by default, and mmap the device-dax memory in 2M alignment and 2M-multiple size, that should lead to the default 2MB hugepage mapping. > > Can the device-dax implement memfd_pin_folios()? Could you elaborate? or perhaps Dan Williams could comment? > >> The flow of a single test run: >> 1. reserve virtual address space for (61440 * 2MB) via mmap with PROT_NONE >> and MAP_ANONYMOUS | MAP_NORESERVE| MAP_PRIVATE >> 2. mmap ((61440 * 2MB) / 12) from each of the 12 device-dax to the >> reserved virtual address space sequentially to form a continual VA >> space > > Like is there any chance that each of these 61440 VMA's is a single > 2MB folio from device-dax, or could it be? That's 61440 mrs of 2MB each, they came from 12 device-dax. The test process mmap them into its pre-reserved VMA, so the entire VMA range is 61440 * 2M = 122880MB, or about 31million 4K-pages. When it comes to mr registration via ibv_reg_mr(), there'll be about 31million of ->pgmap dereferences from "a->pgmap == b->pgmap", give the small L1 Dcache, that is how I see the cache thrashing happening. > > IIRC device-dax does could not use folios until 6.15 so I'm assuming > it is not folios even if it is a pmd mapping? Probably not, there are very little change to device-dax, but Dan can correct me. In theory, the problem could be observed by using any kind of zone device pages for the mrs, have you seen anything like this? thanks, -jane > > Jason >