From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 418D2C4345F for ; Thu, 2 May 2024 01:16:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6768C6B0083; Wed, 1 May 2024 21:15:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 624A76B0085; Wed, 1 May 2024 21:15:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49E376B0087; Wed, 1 May 2024 21:15:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 27CA86B0083 for ; Wed, 1 May 2024 21:15:59 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C7E2E40C33 for ; Thu, 2 May 2024 01:15:58 +0000 (UTC) X-FDA: 82071689196.15.87590B3 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2055.outbound.protection.outlook.com [40.107.93.55]) by imf11.hostedemail.com (Postfix) with ESMTP id 27C0D40003 for ; Thu, 2 May 2024 01:15:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=I4xzD8Fz; spf=pass (imf11.hostedemail.com: domain of apopple@nvidia.com designates 40.107.93.55 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1714612556; a=rsa-sha256; cv=pass; b=x6dYh5OYKv2Y4n6Y2QAYETiwcs5UmxsjxGRYIVZuSK4+WsxQFqJy6xxXIvh7m/MZ6uPwo8 DaynA6SepJrfZJNiXMpCTCC0IbowPwHgVKs/VC/BN8JiVc0FKvofRs0R6T4mFs3dkxHGmx oJc5PU2mX3u0NI6ZSnUQ9a0FLFWHHPQ= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=I4xzD8Fz; spf=pass (imf11.hostedemail.com: domain of apopple@nvidia.com designates 40.107.93.55 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714612556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6Db46LSTcxXVrW0eWH2GLVHgDS6Tt7EHsI/0VtQ2UX0=; b=SdBAKgyb8a7Zgle+mFHDRI/aeL2Dhs+DtQt4zhEHR16jyDZ1nRsp0fSPYLuAercO6GQvKN XrULXxweYgRwaiAtH8gYLJtyWI1+ERgAFpTU69HIwlfhvLw98OeUKltZq3JDYVBFyCnB5V iZNnTv7wBGpZ1elqZUE8xMqqtGQltRU= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oQPzOyU7EPoPadctxxFHP+Q8FgiVLMpMtZZ5Q78J70VbuX8OZP39cxPgi512Z+BfX2IBe9exRUhE0mRHCT11ItpG+gULLKirO/kGSZgxs/ULUAzI8EttIuzI8S1kj7kL6V83UyotsQM2mGkiNAcPVn84NwdzbtvSvy/qBfpNaB+bUipfaIfyj0eNgQj3yNgsYnR5Tf5ftXM9kfeWF2W25XdvqQ7WOQaXqUwI6t7EwWfgVzTG/088mXQ+Av+pffjrryS7LxDW1zCg5oXhgQHwEyCs9JcjEx333he7GkfjqSa7SIj5elbRk81m+e3Z7EGpaeL7xbxnnMAbargAvzrhQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6Db46LSTcxXVrW0eWH2GLVHgDS6Tt7EHsI/0VtQ2UX0=; b=aklRFsqtg4Mvc2t8ictaN6P6hbW//fnyWFLOV/C1HIvhqEb+6nc/SzyCZTGkYGNPR7svbk+xMUFhbtzZDStCFhJ1q0oAiWVHg//7lkMlZjAh6RP7EfU/MdFFlvT6VRg2pxJVSTTj1VfjUH1y1JzMMS1FbfRdnUtECgMM4aWaWGuzSzxRnN3dRllB4LwnkQ3pEoJI1+popZ+oGmkd5D1cDlWywgfmZb+Eb2Eugg+osxLt5YzppNvB0qfgrHBM4wGRw+0WFngHUw3rItwetVRQKBiy6ZIqxdKpnmLkpjPMpOLPurJD6Q2Zpv+BAwM+pYoKDAf9jg/oF7/9ZRnrr+4lQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6Db46LSTcxXVrW0eWH2GLVHgDS6Tt7EHsI/0VtQ2UX0=; b=I4xzD8Fzhsg9wdwqqZCFZWD4LP9ivDqg0t1QHAr05fAtt21edwoOsr3pwJl1E0KkNOIHtSOzC7AFx0+egPVPhWxtE9/AomlXokBmVWRf3cRRGhT9hei/obkzbkK1MABmRmMQiCzylfiUDFCTdElp2pTIQnLoYYX9pbttMZLzJExwn3bLSLKDtkAs9vqLsQ0VSOKZh1Gg2QT9skQGenznir3LqQY6Opeh+q+GJurkd5QkeGvvVinnbQC6IS13Uu83GpDid6FmqDY/OW0N6vQrubdIleb1HSX0416CQ2Pvx+mwdXdbCPZTwu4dRnwQeXcQQ41I+SzFf4DluaTH0qgqEA== Received: from DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) by DM6PR12MB4436.namprd12.prod.outlook.com (2603:10b6:5:2a3::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Thu, 2 May 2024 01:15:53 +0000 Received: from DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::80b6:af9b:3a9a:9309]) by DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::80b6:af9b:3a9a:9309%7]) with mapi id 15.20.7519.035; Thu, 2 May 2024 01:15:51 +0000 References: <20240501003117.257735-1-jhubbard@nvidia.com> <20240501121032.GA941030@nvidia.com> User-agent: mu4e 1.10.8; emacs 29.1 From: Alistair Popple To: Jason Gunthorpe Cc: Christoph Hellwig , John Hubbard , Andrew Morton , LKML , linux-rdma@vger.kernel.org, linux-mm@kvack.org, Mike Marciniszyn , Leon Romanovsky , Artemy Kovalyov , Michael Guralnik , Pak Markthub Subject: Re: [RFC] RDMA/umem: pin_user_pages*() can temporarily fail due to migration glitches Date: Thu, 02 May 2024 11:05:46 +1000 In-reply-to: <20240501121032.GA941030@nvidia.com> Message-ID: <87r0el3tfi.fsf@nvdebian.thelocal> Content-Type: text/plain X-ClientProxiedBy: SYBPR01CA0064.ausprd01.prod.outlook.com (2603:10c6:10:2::28) To DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB7726:EE_|DM6PR12MB4436:EE_ X-MS-Office365-Filtering-Correlation-Id: 6df38e2a-4635-4470-580d-08dc6a45682e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|366007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?1ZS+AJT6Q7Fdqt424Rd3dEJkMdQx+BqeGiawWU9FGozZnV2k1otCGOSHsrvb?= =?us-ascii?Q?2pRFndGCF8kygFD3KJ8YYuRCQ70c9h/n83SZWd7i64Z1k5v9HFkRbmiAh+PA?= =?us-ascii?Q?xzXbCes7fW9v47kfAKp34vEO+G63BKB2qCmLjWEeZUJhsu3RCYEA1yYBA4JD?= =?us-ascii?Q?7k2ckV/caZnTV3+F65srkiO5TL9SaJt/aYm7RReoDpPrg8xY8AOi6Hx4mrKV?= =?us-ascii?Q?pvGt1wZeQ/j/XAf3+Tcx/7BvbsFWhdq2BpSWJOTf/2FvAeCLAxtOePb/lNPn?= =?us-ascii?Q?04dY8Kr3/6A3HaUruIk0ayEwGjydvpjLcjSbkelS+QA0BD8O7+xpgFuBMjPh?= =?us-ascii?Q?kZUKY98H2m5yjzv2Q7zpbH+ZD3Xhu+w+ukBpKljF/Ym1uxwk39WaakrytBxi?= =?us-ascii?Q?nWP7pTuU27hD3RQQyrA/Ax9X2emjQlDYhPCtkC0pWR2LHp/PpFpSgcSThZcQ?= =?us-ascii?Q?pq5lrGjMcZ6q3XUNKN/K4PLfnZeuoUugZ37rb5eRnQhp305o3Fiw/rOTY9hh?= =?us-ascii?Q?G9YpUTHJPB2XCVoIfxV8YwRDeuSe39bKDzQfkWXXN95LM0Ntoo1LV/yJ8xMw?= =?us-ascii?Q?P6fhfDBJihMuCyZjvrcpX5/CxRBZ7yiVzL4D81q7pcjLeLOvAYOASL7hpAK6?= =?us-ascii?Q?jykw8n4fDDX1ZcvvIiTHVIk0KGswhoBdzZdYu7RL6xW5G0mvzHydI+H2Jo6F?= =?us-ascii?Q?fkoK2QzS9XlvACjBO0jEcLSb9YLDj4O7+Ch4/LnxRGuNp5NPzHGM7KABP+yn?= =?us-ascii?Q?KgeMTmwpicrYfA029T3cZ/eaHQPJWCuW6xuOMLG4U0L/at9c7mY8Zr+A8zdS?= =?us-ascii?Q?5jyY43jRB/CEmnyRsjrOJuq8kaLE35Q+dL/oUELIZAfFQ9qfeyQlIR4vkJyE?= =?us-ascii?Q?zEbzeDMjlxxS6z8rvKEOvHzZFOE+xYPvuzqPPYMGhvKKDn/KgcWE2mMbCUvV?= =?us-ascii?Q?LJcI+1LQvRbNvSaUU44ZptUEly3hJoQuXFreDqo2UxRvRRJnPqemmNA4YXUA?= =?us-ascii?Q?Xr6DGRYZcHRSrFKKaSFIfUiAn2VWtkPObOuNvorW7pX4A2rNuGLmFl0BCKo0?= =?us-ascii?Q?ZrRXF/OwhLT/Lrli6ngVcFZg1xtlQOBPP6U6V4J2nmvOOf9PRH6dKq+2jW8l?= =?us-ascii?Q?TW8LuBeGAiSUwt0NFGBeLAy4C6h5V//6gLXsopaTZhY00Oo8wE6Z2XPyW4g7?= =?us-ascii?Q?BEVXG3gaGzN/gUUhJeA1QSDusR6bTHdsvsOdS9QX20bkv30WCeIyqDmlFmAg?= =?us-ascii?Q?tymTsOtR8IqH1leWQK+XNRVQ921tlvYp8oYysBWMkA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB7726.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tL5SS4eKB9Vlduc7yw0nsBG4eLx5U6e/mdh+8Ut+9qR69uvG05K2hRweXfGF?= =?us-ascii?Q?7sK3GDzJHeBaQM8PU7F/gdcMShKvIgDCX1ZGHldaOeHLvUI4sTNoovYiAIcL?= =?us-ascii?Q?a0jvjObcfpYTuPnefupUaTlEMZwQ1voMl3RUg/mvhetzYKM7qs+pUy9S7lEE?= =?us-ascii?Q?tn+y9wMqSzBNTwA+lcVabn6EQRUn1sl0MRx+czreBxoVhVOIU66ZItKsR0lC?= =?us-ascii?Q?GhVjvRm7WH/YnNbzLK/vfcV2vxdBasBNI6aGYvIz0hMzWSJ+AvmbhjnWF8zg?= =?us-ascii?Q?uTziuMfVxPHRwWLbSuR3wAX/nfONLWjss6z+t+dvgm552w2MlYekKD7KW2pc?= =?us-ascii?Q?lEGd4SgUec8+qe79R+OTxxPGq163x9e2NKjqpN5DAT6v7R9WVjqyKvdJinWt?= =?us-ascii?Q?ImNUl8tBsDoB11qAjWLhYLlD8I186nUnzcuC8MQ3YXzUU+Zl1fjlOla0O+40?= =?us-ascii?Q?rcfdHKJhsFtV6v+iBjSRl+de8Hq/jHRK2TAsga0NGzpnuwhRqkBav39GdUMY?= =?us-ascii?Q?2aOxphAF8bxmZOGuRE0tpI26ilJPn8tqDOiVe4/TZARZHYF8uN8tFvNrW+Um?= =?us-ascii?Q?RBn/LT4G3KKgHUm/1Eh1zoScLlHSHHxZbKnMuE5x1qAx/wNyJYIFEH5RkbUn?= =?us-ascii?Q?P4yRNNHJRy/8Ht72Buy0tdeFdaJDetdhj/bqFCBrYRFRw5K2XJ/DxeLbJRvE?= =?us-ascii?Q?hegsAouhHZftMeQAaJVkCrgqmq7iaW6lyl4MnYzGMFdvGVfavZUV0pdqe7A7?= =?us-ascii?Q?VEkNCfFP0ZU91SrRJ2TOVRSEgCOMLJpkEzicFruzB4DoFuQQSJnFXwtmfSWI?= =?us-ascii?Q?0+vZD8v0iFXU5CAc+uTLRJKnNqL7CmLnFbEIl26cEV6nD9aq/ic0RgQaJoaV?= =?us-ascii?Q?ZLvjYtO0aBKu10Rbz9p1n+6FTsvKlApeRVnJhIgqEU7MD/JBpuNTx8L+fXrY?= =?us-ascii?Q?qvZK1LWGl1+22apJGhJM36GzWP9xVdxO2YHrE97iYM3SKjh2iDywaSSm3UZo?= =?us-ascii?Q?mIwnPIaRCyxjnRnf5qnI/v0HnqKfq8MPnP/y6mdEJIdHpzO4YBS4LtJ4w+q6?= =?us-ascii?Q?aQwy0uL43FKXp7uDbbfybERCEmMr+FfCPnOk9WFGC0B21Mo4ciAFFS9MSeV6?= =?us-ascii?Q?68LMV/WMZ9wpZ645gli63+IOBPr9sLznehK2TWv6WPKILeftImLDqNk26vaD?= =?us-ascii?Q?iMLpZLbvTSb1rsPa7fr2UxVHKGSBlRX6A/SmCJzIJIUTXY6uj7W9aGo6uCjQ?= =?us-ascii?Q?LiwjMZa28W06C49lOQXA0wWAXnoyI3vTo6S44E0KK5svYMOYqvzcaNEBWS8x?= =?us-ascii?Q?2zOUcUA/6975qsQZ6Xus0c6QBJzhOn9kPg9x2mPizBn6PlVoOAYeQVbnKd76?= =?us-ascii?Q?w3ZWSOC+ImAMD3kyN80QJ2slpD1s6xzfbU1d5BkgGmPWioUYQl+hMDBLY30O?= =?us-ascii?Q?M37JPIIEovs/bP4SImu/8wGWH4t5cZA041IXJ58rz+oVL22eLB/1Gkw+adgp?= =?us-ascii?Q?oxx/kf+FlMlyoshSn3X7MDpFMS76yda8PYlRWI5SU0bW3rAtO1FGlAEdi0nV?= =?us-ascii?Q?cw5UlS91AjwXwR/nLn0Qr9tzPDgC2LntV7X9lHo3?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6df38e2a-4635-4470-580d-08dc6a45682e X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 01:15:51.4285 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Jr/A8ATBP+mu5jt56wJOpPsR3ABGeLXJTshxeabHMVDoyxGo93KqRhwU5Rec19m6Wtu5yTqrMVj8MKQWfynO5w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4436 X-Stat-Signature: agjqkb7mfxs7d37wpjpm3wmuy1gxcd9p X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 27C0D40003 X-HE-Tag: 1714612556-336323 X-HE-Meta: U2FsdGVkX18OMKfBei2goPGi1hFO9uDIVdLPPps5wEPp+yoy/2cjOn973dYfdfudT9nAHUByt5Jo1KSFJGFFht2+y8HkmnH0jZKnb9Xaglrn9hhA4UKmqi+gL3N4npObEYRFQIB7f/WlNZS6MAA/b66QEgeDCx4AbgrsJfKPnHIecVtB/Z/II1uOVh5oHEW46k9MoM1h9lEyILWdHxf4Pu6Ytas23vqKucI0YGgKxj3/98+M7/fefH45pvhT/IbpPIjSjYnqd5cJ03nT5WCIyFN5xvm+c33CfnRKgffN8gOsIekTRxI3X/yrdrbLwYiK5uGm7Iq1xMtNCe+Psd1guSRUrNK583FytinT3VZHfXoOg7TPH/QFrC0a97Z636lt0yPspff7Cu67EZnW3B1UVs9vK6K5WHDoINxfA0fX7F51m8umkt72GCcDYtPPjMlkl8oBFuxARFH7ngLih8FRNo0eoPqtL9D42zj7dJ8MYDQv66pyfV7UiGcPlMSqbCwyHpY21H5ynm5D5qc9JWpv7m7g8EgUCxicRqnZ4hBlmVQaR/35Fwx+6O6vmg+PZWKIckv11OYk19KBBcguvM2P51bPNJIjA/+iD3ciAzJqNpjNMXxN/jKB4MMKLp5dxCUmOqGNOVQx1ivd1cFztFmJEX7tDWdrT68mK4gwWNhPEt9jMFjdi4i9kGw5WCiI7CFKPHhMt6icTvxwHqnUGNtSxuWYwLyIiXpg82ME656aErrXN0cplZKUW15L0pI1w4V6iGE8W3PPwOgElZnaflFlrgAcTfSqoxquiQFh7QE+N9YQnLpXt38KFFV4jcX7zRuJ1hrMeelnFZtEWfCPqtQxOTJYL2oHGAjfxsqn/3FNqvX6hyzcUU1ok2UxjEUVWmghMzbArdanXkUjzhCjdGq+mrgSU9ybukm9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Jason Gunthorpe writes: > On Tue, Apr 30, 2024 at 10:10:43PM -0700, Christoph Hellwig wrote: >> > + pinned = -ENOMEM; >> > + int attempts = 0; >> > + /* >> > + * pin_user_pages_fast() can return -EAGAIN, due to falling back >> > + * to gup-slow and then failing to migrate pages out of >> > + * ZONE_MOVABLE due to a transient elevated page refcount. >> > + * >> > + * One retry is enough to avoid this problem, so far, but let's >> > + * use a slightly higher retry count just in case even larger >> > + * systems have a longer-lasting transient refcount problem. >> > + * >> > + */ >> > + static const int MAX_ATTEMPTS = 3; >> > + >> > + while (pinned == -EAGAIN && attempts < MAX_ATTEMPTS) { >> > + pinned = pin_user_pages_fast(cur_base, >> > + min_t(unsigned long, >> > + npages, PAGE_SIZE / >> > + sizeof(struct page *)), >> > + gup_flags, page_list); >> > ret = pinned; >> > - goto umem_release; >> > + attempts++; >> > + >> > + if (pinned == -EAGAIN) >> > + continue; >> > } >> > + if (pinned < 0) >> > + goto umem_release; >> >> This doesn't make sense. IFF a blind retry is all that is needed it >> should be done in the core functionality. I fear it's not that easy, >> though. > > +1 > > This migration retry weirdness is a GUP issue, it needs to be solved > in the mm not exposed to every pin_user_pages caller. > > If it turns out ZONE_MOVEABLE pages can't actually be reliably moved > then it is pretty broken.. I wonder if we should remove the arbitrary retry limit in migrate_pages() entirely for ZONE_MOVEABLE pages and just loop until they migrate? By definition there should only be transient references on these pages so why do we need to limit the number of retries in the first place? - Alistair > Jason