From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DFD3C77B7C for ; Mon, 1 May 2023 19:21:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19C3A900003; Mon, 1 May 2023 15:21:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12563900002; Mon, 1 May 2023 15:21:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB9BE900003; Mon, 1 May 2023 15:21:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D5EED900002 for ; Mon, 1 May 2023 15:21:09 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8F3D6AD809 for ; Mon, 1 May 2023 19:21:09 +0000 (UTC) X-FDA: 80742654258.06.3615064 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2081.outbound.protection.outlook.com [40.107.102.81]) by imf13.hostedemail.com (Postfix) with ESMTP id A25042001D for ; Mon, 1 May 2023 19:21:06 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=hYRcvhoF; spf=pass (imf13.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.102.81 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1682968866; a=rsa-sha256; cv=pass; b=bxKu3IdHVBcJm1TT7LfD/oMOp55B/c4Pvk5YnibFrWEHkZblz4hwKWpiAy5Ch2dlr27xom uaTaLHAs3Gmb5GSlgRahQEhRqu4EpNhrhw+eYepoBiEj9ZUD6idxD1TQy4eDWvzuTJ+/9d 6EYLasI1ya9eDQAkCTgZPx969YCELtc= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=hYRcvhoF; spf=pass (imf13.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.102.81 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682968866; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PagNhKcdVCDaKirdBeWKkiuQhExtcfc/pZOVOChF4ug=; b=QepWT26f5GBW+BWLtg58XdYqsfou5d5nGMss0BlFrMPdEnCgWKXDPLhSwQxCTIg2HOoisZ b/iWhbEc6nlNM9/2m4BC5m6BbZEw+s2jyRz4ravEagyUUtCWAOwxZ7VGNxvr4dnEcdcRc8 YnuGQnuXXStCYRPgR9Rx4OhLSEcD8VE= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kWi9uxn6IQR2n0EfMDHbGr662L/3Jymc7odxOCMt4g/EJLuZB40FHlVquz38cQmtY581yLzAnJnGXhUOurFSRwg0gCKDBWw2+KyrfFM2ZtfI7mY4bpfUyjHaM/G/tiRVjtCTDvGkwJ0XBKRNOZbKfzO15CJnXyyxyLROwO3QEjrxUV4K8yo2bia67Ley69h7VLXDVRyiyt5/SWZLjHBFZgecxRUxj7MR1rfFJjn0mj0SZlBQoW6iVc8W/CLjwJ+58ncsETDtwF8CzEICi1cG6JP2U00M9Grq4ohvFvSxH2NfGEaFvmQboaC9YhXUDQNKWMiTSgC7XhCbxEx/i3loPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PagNhKcdVCDaKirdBeWKkiuQhExtcfc/pZOVOChF4ug=; b=IL0YqJoOJL2M0F4ko62LQpFMFMutPoAj6J+8Ttg0j/TrkS2FPNfO8GhJveYOLqECg+/FJ1W5vBOQa3v7Jsx/rUYxf9wMSYHKoKGudghEMYq4aJyBsYtD0qwerL9a/JqxTeSeqjKhkK+0fkwOR1temdVhxF7RsGMLaghcVYlHFafkfMieyPPlhzYwSavlu8iMiDOn+LLJDC0stEDqizdbrDuAgL4XDCY3CtrjYY6CIChXxbF7G5agdUungLwYHBec7Ev3g/jmwaIx9hCKzAswN+pZWxjNZ8LMkqPnhiOwxeitmQRW0vZCjadQWBeafDLTMraXT8UaCSxfQSqgQ5OTYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PagNhKcdVCDaKirdBeWKkiuQhExtcfc/pZOVOChF4ug=; b=hYRcvhoF50f8x5W2fyEHUazxjvCuD5krHK76hdWC/Vg308IfuwVP/3I3728cT78ELaEjqxfeaVXb/zNynPHen11Fb9IjUdNgNgiYw5K/lRycGLUUVT03KWDAy4KF0W3VQSpLvc34SBYQKPVsaweICgA2Qow7gApEwh9MIfR8UOWCuDmqhtTSngnPQ6XiZWkXCVnQ3xqL/BRF1P0LJNcV9zAHHaOpRaJQe5m1M6+vYpQKtJSssKA3Vj/iIYTtf4IkNVrFtBfclaRHsS5VJMG9uZuCqNSeEKo3N4vgdZxNtMSHto7ZRxxD7m8EXEtVZcF6pIhNpKJ0Xq7FvuqVBKnrbQ== Received: from BY5PR12MB4130.namprd12.prod.outlook.com (2603:10b6:a03:20b::16) by MW3PR12MB4476.namprd12.prod.outlook.com (2603:10b6:303:2d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 19:21:04 +0000 Received: from BY5PR12MB4130.namprd12.prod.outlook.com ([fe80::4514:cdc2:537b:889f]) by BY5PR12MB4130.namprd12.prod.outlook.com ([fe80::4514:cdc2:537b:889f%3]) with mapi id 15.20.6340.030; Mon, 1 May 2023 19:21:04 +0000 Message-ID: <227f2880-7f6d-115f-6958-0d195057d567@nvidia.com> Date: Mon, 1 May 2023 12:21:01 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: [LSF/MM/BPF TOPIC] The future of memory tiering Content-Language: en-US To: David Rientjes , Michal Hocko , Dan Williams Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Wei Xu , Frank van der Linden , Johannes Weiner , Dave Hansen , Huang Ying , "Aneesh Kumar K.V" , Yang Shi , Davidlohr Bueso , Jon Grimm References: <7443f0e6-6be2-3320-60d9-03da0cca2987@google.com> From: John Hubbard In-Reply-To: <7443f0e6-6be2-3320-60d9-03da0cca2987@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: BYAPR02CA0049.namprd02.prod.outlook.com (2603:10b6:a03:54::26) To BY5PR12MB4130.namprd12.prod.outlook.com (2603:10b6:a03:20b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR12MB4130:EE_|MW3PR12MB4476:EE_ X-MS-Office365-Filtering-Correlation-Id: 69ef3035-075b-455b-ab7e-08db4a7934eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pr5XyVc3D2N8Ij8tXwxW1kZ49Jj/je81JIpAlr/TLUf+ZWGtJpEBduaA1JxqmnNL7tFpGge83vsYBWzCrXXuxPeR8yspWV0gZF5w8dWD82wCmSgFemBdaioBxhlx+VXQ8yaWHJTOxYXXnVF7TnBMvOtdlzuvbGVZBW/1FAUWhLlpQI+3yaXTuJP52h6YifrO9M1HCBTHvowaU2a+zZSFwwG9TBcZHK7Gga4R9td2PF59y5EXycNmxhqXsZXRLWttFkuybKwURgNlW3NauZVVRi3RiY4TeWEuKsQaMRcfvRtqFC5c8D+7o5/iXPB/Hte1nrOYTT50FSsX8P6IonxDhjYe1qwxFyDSO79fNPoe+CF2qhUmB3jPIzyW0R6cE/+x3vA53xY1jKSEgiWd85tWqTAo9Jxu+93Ue/lJ3rJ4zP3AMv/MpZQjqnoZPh9SLSvnpeuOTVRoRf0of9lHwJ1GM77eJBWiWkLYMxjMEJDUJ5XUaDG7BVW5jdwfx5Njt27WNJ1x7hHd2kYUNJRDmmN41t8uOeTGqd2JrjuNccKB6A7WPFEx2o4K1WP59CbSeiEumqkB929hnD9JnhstGXtSgrNuSBQ6RmTsA3PjvjNVTRsmgsQ8XPifeK1uJLnLI3OwDgh0AvDj88LqdphQwZQFtA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB4130.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(346002)(396003)(366004)(136003)(376002)(39860400002)(451199021)(478600001)(31686004)(36756003)(7416002)(5660300002)(38100700002)(2906002)(8936002)(8676002)(86362001)(316002)(31696002)(66556008)(4326008)(66946007)(66476007)(41300700001)(83380400001)(186003)(6506007)(6512007)(26005)(53546011)(6486002)(2616005)(54906003)(110136005)(6666004)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YjU4S2hKZzk1SDhpYllIWDFIbGpyelNVOEliRTBVRXJJTnJXdW04ZHFvbXow?= =?utf-8?B?OU9BR0FSZFQ1YWp4dHlUSjJaQjAxSzhEOHZETDhJMXZ1V2VBSjBGV0NhNDVa?= =?utf-8?B?ZFVQbTFsVDdubitoMUlmY2o5bXFlSkFhTEVkeU43M2tYR2gzNE55MGdSN3Zu?= =?utf-8?B?V3lORThsSTFGK1YwdVdGSjlWRUdQWkptekw3YUNrdGk0ODRmaEwzMkNnUDZ5?= =?utf-8?B?YkN2aDhWaXhiUk04QWQ1L2o4UC9FLzltSHpRWktsZ2Q0cWVkU1ZYQmdKc1Rs?= =?utf-8?B?WTVVczVMbzgxQ3FocXVFVDg3dGtBMjd6elk5Z0NnTDVXcEUvYXVWSFhhTkEw?= =?utf-8?B?bXc5akxpQ3VIZks4a1ZOODVFR0tXdlhnT3NCbjk4VTRBUEZkSnJ3ZkpJRzZ5?= =?utf-8?B?cnhYd1hRdTIyYVlQTUxkTTRxY0JtS3R2MEViMUpZc3liSC9mbmlFL0NsN2Za?= =?utf-8?B?bGJ2cTdqam5kTUR3ZWkvWTBEeVFlTVdVTHg4d3gwSGhLSHFUemZVMmRzUmp1?= =?utf-8?B?d3I4cklWVFJwdGNqeFhTcnQrdVlRUWNubzRPRWZiYVArUDFBUjN1eldYQ2F1?= =?utf-8?B?TGdZbmw1Z2s1QmlCemxqTFdpeGRhUVUweWNmWnlJMFI3MzlJQTZERHNEdlNk?= =?utf-8?B?eERrR2lFU1BTTWdHN0gzTFVlY0wvMjBmbnlOYkdCQ0Y0RTQycnJFc0VFNVBQ?= =?utf-8?B?TmpXSTdzUGtXNW5weFdjaTJXN1hVM1VoSEtKMkRydTU5VHpTalZpbHRvS3hX?= =?utf-8?B?Y1ZKcnE0Mk0xYjNrRldoRUlmR3B4ekZzZDV5TkwyRUM5NGQzQWpSVjdRdGo2?= =?utf-8?B?ZFJNazltL2kzQmtCZWNJMUdJd0hkVkh4V09vb2xzc2FadVhXeUdCMjlmZTJ6?= =?utf-8?B?L3FJTml0Q0lFeUVrUTlDSVd2ZStwaXJQbWdENG5Zd2tTd1RucG0xSEV5eFFH?= =?utf-8?B?RmZQVUsxbU1JRVhGNzIzbWpiSFZ3L1JFVUVUbnZUWVBxY2g3azNHcVpGM3Zm?= =?utf-8?B?QTBZV0dHVmxWY2h3NXJLcSt2UlB1cGgxNVNUakFQZWV3VFRhVEVUeHFkN0tv?= =?utf-8?B?YW16ZEZEclVTUGwyQUpsZ0VWTkFnZ3dTOVVaN3NkbjRJTzh6Q2Vlb01KNkxz?= =?utf-8?B?RmRMN3VZRGRUb0t4V1JsWnZhMDVzTkxKd0N0dDAvMXRBM1JIbzdlYXFZUWZF?= =?utf-8?B?U0V1TXhUYzVPTmI1N3lBNzlRdkdLZDFEUmFuR2oxZmtLeDNDVCtURVhiVjFo?= =?utf-8?B?Z2JBRzd2RHhRQnRqcFFLYjVuN3A3V1JXNWRYcU5pVllSVlBSTEcvNE1xc0Nz?= =?utf-8?B?Zi90TTEzdStEQWJXSjJyczBWRzdiL01BZzI3OE9DakVJaXJSY29BZEtiSm5q?= =?utf-8?B?N09VWHB4YnFkQ2pTcXZKcC90RDdZVVNUZVRNV1BJWWdPa1VPcE5kNnVVbUMz?= =?utf-8?B?YzlQYVZoUHdXRHc4Z3ZuTmw5dHk1THRUc1NwS21FOVRrcy9pWmVwaTVnNGFy?= =?utf-8?B?R3NPUUxJQmZQd0Ezai9YbHRPYkhMWGdEZ2J6amI2REkxVU5idngydlBZL1cr?= =?utf-8?B?akZNZVJqcWJjZ1N2Q0tSbmJpMS9KTTdaOHIrVmw4dXJtQSt2MHJRV0RLcUs5?= =?utf-8?B?SlVlSG5XMHg4c0VCRUhzekZhQ1lhZWVSUG41dEVEUGloeUkvWU4xOStwMStL?= =?utf-8?B?dnZ3UDhpMisweUpJcFJaUW8vU2pzNlhxdnRsdE1mSlM1TEkrTG1PcmxUQzYv?= =?utf-8?B?eGorMVVwV1hITlpVbkgwV2s3MUFDOUgyMXlOVlJGVEZPSzRkMysxRXVOOGQ2?= =?utf-8?B?TzJzUml1eVJhL2NMRXA0dXRNaUZqaEdpTTNLNW5JN1V0L0o5djZtUHpKMFox?= =?utf-8?B?eWRrQVM0d1lld21iSitLZnlNcFcrT2pNWm5PRlJjTTBqV0JjNTJpYjlSKzZy?= =?utf-8?B?NHErejdYUUx1YXcxcnlrTzdBSERwWkNSNWkrRVhBY1dnWThvMStpYUZsRnVR?= =?utf-8?B?M1pZRTk0TWNPTWlLQ3NacGVKZ3ZFZHpYQmhOWTJheUlVck1ZSUlZVmJPdUZO?= =?utf-8?B?bVA2aGhHTUs1d2FyT24rZHhhcHZuQ3NWOU9OS0NaaU9weEZkMHJWRDE5MnVz?= =?utf-8?Q?/Mo72ee1r+9mkmKlQ0nzqA0pT?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 69ef3035-075b-455b-ab7e-08db4a7934eb X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 19:21:04.3547 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jNodjOTg/s0LOhX3wcK/Ca7ycvJfMUBKNcZ1jc+xbtjBv3FzAvj+pztbN3Z0XJiivYxB+ko1Eq80ih+ZmqaSfA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4476 X-Rspam-User: X-Rspamd-Queue-Id: A25042001D X-Rspamd-Server: rspam01 X-Stat-Signature: 78yuczjm64p5nyxjp9dh9kfdiurh67n6 X-HE-Tag: 1682968866-172795 X-HE-Meta: U2FsdGVkX1/Wz7sTGbjbYDW5wdsznDR/w9EsTBINgFJnDthp/xaHr5QyAkL3dfDoolJLivyS7Fj44o9Hv8IPVzHoZLUr30gYsAKs+zALGyibTLCB0y1wBk9TwNNE5QY4WKR8ZphRR3kVLy6slwhodKdQT+1S1STHePR55HyyU1IGOCW2JrXq7O/n3/FiX2AYXnrGl0BqF+Tm2vEz7vy5XeJSAIKmpzOvfIsmynTQ3OoPzBnjAtZbBmqzrcSdB9aILnlzxsyvDyOoRzc5CpC/MTRCIKSWt8Aj4pVA6bEHlrlaLkFz+wKh5Iw9x7TOT53ogp01Fjtg2aA6orYfQ7bFYJfQjMv080Dir8UZmOdMaAYtlw8T9e56pUis7xvemNPFbxEOtdZjiqssSpnJXATI5rxuW1++L7ItfXg/3HTPy/3ZYiQdzETYkgkOTbfjSfCK+TyLjnuizPzf7RqRNtGijgsBReJL1dpoIDHqUcqbY+DRfCB6s4Kt5oAc4TzdAzYn42XzIoDticels/BoaZBaL5Modod/VPm0bmxPDDwuz+2eXSx5df8SWqPdlNvEtwUTv85bYUbyTC+cdepdFxoRvTGg9n/i/4dRvYwYfyZPAxoaX+1GV1Dofw54GYKmbxFkOm98rmm1dxIbn52Wxri8/ZMS7ZoZXsT0A+g3qw1OwPu8FWD7pEfex5sWtfpcWtEccENuCI5coU2XjWZ6bKz7Unv17vhAWEM/6GPCwc+CymQHv94q+/mtOdPSWWV1oX44pSUlnR1tkM216NDGZGXE0omUc/hVvSALvEG5SgFmpjGL+vttTBF4LMtuXQ7xT2qHy8PhTHzWsRB5yvL4Fqlw4LOMgkcPYeWktcZdiZ76v6cFe9J2uaqHTVR3NgamRjg1zt+WwDZkFNH4ejYn6VHcGmK7eLKYmY7gF5Sfa1NCYLs3lyBeZzYJ/nI15G7QlP8mBq7doiuBu0jMn5vm5ox tmHg3+XH VtSGzt8jxNIqKf/huGo5oo9WT9fP1LnvQNhm9zZAe/55ZFeA3oSCCMBKEi0MW1hDSuYtrW2gObvvVw816w3mVN0CEIXpm4hs4afVK05XG4n9h7Nsqkg3odKL6qCaRB7Fs6Rk7f/MnqM6rw2lKIkK6FPu0+Pm+RktBc/E+g65GjfeIXq2TSCYctKyAgVMFmTOg1znxA6oTUhnJjJeRuL+wTInFDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/26/23 21:30, David Rientjes wrote: > Hi everybody, > > As requested, sending along a last minute topic suggestion for > consideration for LSF/MM/BPF 2023 :) > > For a sizable set of emerging technologies, memory tiering presents one of > the most formidable challenges and exicting opportunities for the MM > subsystem today. > > "Memory tiering" can mean many different things based on the user: from > traditional every day NUMA, to swap (to zswap), to NVDIMMs, to HBM, to > locally attached CXL memory, to memory borrowing over PCIe, to memory > pooling with disaggregation, and beyond. > > Just as NUMA started out only being useful for the supercomputers, memory > tiering will likely evolve over the next five years to take on an > expanding set of use cases, and likely with rapidly increasing adoption > even beyond hyperscalers. > > I think a discussion about memory tiering would be highly valuable. A few > key questions that I think can drive this discussion: > > - What are the various form factors that must be supported as short-term > goals as well as need to be supported 5+ years into the future? > > - What incremental changes need to be made on top of NUMA support to > fully support the wide range of use cases that will be coming? (Is > memory tiering support built entirely upon NUMA?) > > - What is the minimum viable *default* support that the MM subsystem > should provide for tiered configs? What are the set of optimizations > that should be left to userspace or BPF to control? > > - What are the various page promotion technqiues that we must plan for > beyond traditional NUMA balancing that will allow us to exploit > hardware innovation? > > (And I'm sure there are more topics of discussion that others would > readily add. It would be great to have additional ideas in replies.) > > A key challenge in all of this is to make memory tiering support in the > upstream kernel compatible with the roadmaps of various CPU vendors. A > key goal is to ensure the end user benefits from all of this rapid > innovation with generalized support that is well abstracted and allows for > extensibility. > Yes, this is an extremely relevant topic from our point of view, as Jason already mentioned. I'm very interested in a system that works well in the presence of highly capable devices that can handle replayable page faults and can co-process along with the CPU. Eventually, the kernel should, arguably, be more aware of what a GPU or smart NIC is doing with both memory and (device) processor time. I have lots of examples of that, and one of my favorites is the current autonuma behavior: unmapping a lot of pages, and waiting for *CPU* page faults, in order to decide which NUMA node those pages are best placed on. Of course, if a GPU or other page-fault-capable device had those pages mapped, then the MMU notifier callbacks will force the device to unmap those pages, and then fault them in again, at a truly huge, and unnecessary, performance cost. Also, when thinking about designs, sometimes it helps to think about memory from the perspective of these devices, just to kind of shake up the mental model. thanks, -- John Hubbard NVIDIA