From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F4175C43334 for ; Thu, 21 Jul 2022 23:00:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 040E28E0001; Thu, 21 Jul 2022 19:00:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F31D16B0073; Thu, 21 Jul 2022 19:00:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAC588E0001; Thu, 21 Jul 2022 19:00:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CDF326B0072 for ; Thu, 21 Jul 2022 19:00:29 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4E6B41C655A for ; Thu, 21 Jul 2022 23:00:29 +0000 (UTC) X-FDA: 79712627778.30.D4EA3B9 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2075.outbound.protection.outlook.com [40.107.223.75]) by imf08.hostedemail.com (Postfix) with ESMTP id 871F616007C for ; Thu, 21 Jul 2022 23:00:28 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hhwYyPyffkWGHB1GuCjwXLK2ioNVr3BrDwvjudlqjKecdlx0OCBPutf4RHYbogE+AMkAK/2BUWGfn+w8OfNIxn/zw3LroYY8Ab4t4doUuyPQ6m3M1CW1ykeTwvWnIBU9VCvtvN7kNM4vFOOU/ELTdUOgVjGwX8531PByVtda6fGDi2QuOZ/jn6Bg4tCRAwXj+oG5cH63G7RGh9gntSfSSmCfbVPHMtI6jn5rSKErc+55WrRse+ygkr5Ar127ph87CutfiqktGe/3dtMdanh/uA8VgiRsSy4mRFeo3aZDnCvgw3V7LD3HJTZLWRbSXc+3Pmp7joPl+jUTGXpKh5fIPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PEqVtFXBmLsIBkHrerqbCLvmeJuTONa/12j+qhkHvEw=; b=e8uZUhr02h2QpSaz55342l+7aXchZgoQXiDKvQPvMW7AUyJrJGpRkSruCUMaOKc2YCkE86DKBmoWhPGTbmhqbLg0RpT85OonaQwV2x51wmDZ9fC5sauqR6EZyfBuPW6HOskAdVFWusvlWjEuGdbLSQgLMwsSZbXuIRiBvdRaE44RimVJJFABjQcgcl2vz1D6J2W61e3Gd6K7DUN8jvYCdyCH7lpko/FR5CN9S/QNQC5RrgMmcXwVOB+R7nEQ+JNMX3oXYBlpuVwU+TG738/TWJYVuXvrLmXJbAYVA7jSLU0lFKwUHn6ZS3ZUzywpuQvf6I7bdnW/AczGEOmF38a9jA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PEqVtFXBmLsIBkHrerqbCLvmeJuTONa/12j+qhkHvEw=; b=Tlw5xA2PAVoAvK0+IgxMfd/JPEQTHkWCya1fCdDLsWk4PgpV2FS3BWhpWOG2Osonp2nfYqtm9xqW0IOMnzL6m4ZKdC20klaqRd7w/WrTadEmi6sq/6uxlVLKhm7ZrMVezUTdgPPQbXwxJFo9W1+NmWtciHzJu1T32mtOFctudbo= Received: from BL1PR12MB5112.namprd12.prod.outlook.com (2603:10b6:208:316::16) by MN2PR12MB3198.namprd12.prod.outlook.com (2603:10b6:208:101::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.21; Thu, 21 Jul 2022 23:00:25 +0000 Received: from BL1PR12MB5112.namprd12.prod.outlook.com ([fe80::2d9d:f26a:77a4:c19]) by BL1PR12MB5112.namprd12.prod.outlook.com ([fe80::2d9d:f26a:77a4:c19%5]) with mapi id 15.20.5438.024; Thu, 21 Jul 2022 23:00:25 +0000 Message-ID: <0d9a7320-e639-ad7b-c45a-644914d73c2e@amd.com> Date: Thu, 21 Jul 2022 19:00:23 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 From: Felix Kuehling Subject: hmm_range_fault interaction between different drivers To: "linux-mm@kvack.org" , Jason Gunthorpe , Alistair Popple , "Yang, Philip" Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: YQBPR0101CA0179.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:f::22) To BL1PR12MB5112.namprd12.prod.outlook.com (2603:10b6:208:316::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 146ba86e-6704-4336-6c6d-08da6b6ccc33 X-MS-TrafficTypeDiagnostic: MN2PR12MB3198:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5J+O3DnUhdZnzndiQiczUqajjtHIckETlB/dHS/GCsZOY6ouiDHy+LH78DC7LUai3U8yq/W6GkvoWRr1/cT+1kVcnfllawlUkoUNh7NlSWMEugccNB04vpJrkM5jLXWrdZyLKMY29E6FViaPPj2Z+MIlgfq9T78vaIFEcIb4KgLIHCrPuvsUmsCL8gBDMKm9QxVY6xcYxC6HuUoAwxFj90uzi0Zlt57e2sH/LH8rQDgttLrj6BOi62NRboRHdnVeW0QcANnTggA6y+7/EYye+ppntYmbvhwLMtSm4dD8W4q+siSs+DiJ1iSqlQpgw5CK8gbLFkLD4fUtMNnfffqCwpXQZeODIJo9MpE4eKvgAEdBIW278vL6djfEICAdYuH1tNaNSk4R2IyYw3QyIukGIHbBLpvwhWMM9U+Xn+/hypyMdUzfhAA+9LNS5rFuju7V8QgXdAfj2xVN/2BQ9K2TYMs8LdSl9w48yE+YDlj/kTix0Z+mlEDyb7TQSIArfz8fDKjQaGuv7k+lk3rzI5Hp/WeXjf9QtI8Unjy+hYyQaZJr6/AiggZ3Vxkq/XiHsdut7jPfC28PFpKJstuEWnBwH7ZZhIZOZu/kO0ZpLBXXmrSF+lvUD5CMMdsUScY4ehJdAX7kiHuaqI4X9N2UQwoJsf2M3vY3xyitQPkzBNcBB4scV6nfcYGkVzxjofHI4zKdAGy2qWxrLXl1A2p/2gi8UFqjtWKrltf1LxOfXSGfne+Z8/jP702cBIn2fOcdA0+qCD4rtRqeTC0HCSPE1JYCTzLB79OtaNU72vb9lFWr6x+9B7S6ZuPdRXNRJK/hVSAxtAAeOjWQj0CUoj/tWPq8Vw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL1PR12MB5112.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(39860400002)(346002)(366004)(136003)(396003)(83380400001)(2906002)(5660300002)(6506007)(44832011)(41300700001)(6636002)(478600001)(4744005)(8936002)(2616005)(6486002)(6512007)(38100700002)(86362001)(31686004)(186003)(36756003)(66946007)(66476007)(66556008)(8676002)(31696002)(110136005)(316002)(26005)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bFl1eEJKNmI5RThZK3B1eFhLejVERkRtSEwxVEVlMlBXK1BDa1RmZmtBN0Z6?= =?utf-8?B?RDhkMXlhbkxCN21TYUpQNkVNemV3R1NzMy9xRS8wMklTR0dVUndhbDBoSVpO?= =?utf-8?B?bCsweXpUNWNiV1I0WlQ3KzRkR3lqNWFQOFhHS3NzZk4rVWVlTC9JWjkvK0R2?= =?utf-8?B?L1N3OXordGF5Rjl0bjJidHBYOG9ieForbCtuUzVxUWwzMnV4QWt2d0l6TDkw?= =?utf-8?B?NXB4YmlvZ0twbEhKNVBWQ0VmUzZLTVhYRFhWbENybEFpc1p1cU0rRDRLVXpK?= =?utf-8?B?dlRRZ0RFQjRwZlNhaE0wbDBIdnY2N0lFQjYvS2pObmdNMEpPWHdLc01EK0Y3?= =?utf-8?B?L0JxekNRbGdtQWhDOEJHQXVsalRSTEZHQUVTSVlZOGVSVEs0VHorQytRaHJY?= =?utf-8?B?VmNHTW85TWRGdENLWnUzNEFVcjEyUEl6UDIzQjkzamxyMzJHKzlmQjQwUzFI?= =?utf-8?B?eElyNEZlejAyMklEaEFLTG5Yakg1NUQ3REFkRGFZMEMwRDFyZXVpQUZPQnFS?= =?utf-8?B?VjgvODJ5WmtyTGdJanFVT0VJdEIvYi9CaVRRVVlJNVlQZUxIazVBVTIzR3hE?= =?utf-8?B?MUk2a3lHbG9LVmUyelNhQUJZYXFsbEJ4YTlkdTh6VmJkcXI2Q2JQb3Y3Tno2?= =?utf-8?B?RDhzREo2TDNNaU5tMTJ1RENVUzFvMjczRTNFOExqUEhRMkZaT0tYMTk0Znd0?= =?utf-8?B?ZWZJOWpmS2Z4RlB2UGVIN1laMEIrNkRXVGxhVVlqTEMwamwzK2hTUVlXVE82?= =?utf-8?B?bU05MmhPM1VCWW1pTmJSSXZNaGFwM2w2eEpGZTVEVWZKQVFibzdDbk5UOFdl?= =?utf-8?B?WmhSVlFRZjVlUVBZU1VLaFR3eFROZUJDR282UzRIeTFITlYrSTY2Um96SDJy?= =?utf-8?B?NC9sNFc0UnNVTUhsQ0J6eStLYlhTQWhnSHVWVXYzRi8xcC82emYySjdCWjV5?= =?utf-8?B?elRjZ1RNcFExNWU4dG5HYzJrNEtieTl3OGR5YjlwQ20zUmJDMElZcUd5MUpU?= =?utf-8?B?bEFnc21UaU9FV2wxMTRhd0d0VmdSSk8zSHBMckYwdEIrU2Rhc2RXaTB3bFdy?= =?utf-8?B?YUJmOVAvWFVqUUdCdWY5bE5NNDFmR1FnRjVleUNYckRsZVRHSTJvejV2RWNp?= =?utf-8?B?bGNnSGUwZGRHOTJRNXVJMk5yUVNDY0VWTHgwRmx0YzBhVDRlSDNkdUFrbW96?= =?utf-8?B?QlJOTFpLS1dSYmYyUXFKM1pmREhYcktwVkZBWGp0WWZnQnIrMjUyeExKcmcx?= =?utf-8?B?V3pDZWJOWFNWbmZocWkvcEs2ZVFKdVVnL0kxM3FHUVZraC9YMEJrdVdxYWZa?= =?utf-8?B?MkVXekY1aHNKdjRwZHZSYXVyZjZUWDNHYSsxMmJHT1IzSC9OMWhGVFRQbExz?= =?utf-8?B?K0FhWWVBd1ZzdnpqMmliNHdpSFJLMFZkb1ErVmxKdjF4STZmR3gzd0Jkd1o4?= =?utf-8?B?d0ZIczFONTJJQ2pWVG9IQzNNRlBqekNJdTVTaHppTCttRlByWC9qd3J1TnNR?= =?utf-8?B?a1ZXWnlRMGU4T1cyRFl0ZmNiVkVpVG4yVCtmVVA4b3d2WjV5dGNxOGVSUXhz?= =?utf-8?B?MENrK2JpSkhLenNBUklaU2U1Mzd2SnlmekZ6Y3VYL2QvSUFvN3Zxb0RPcWh4?= =?utf-8?B?VHFuYlFCVTkzU2daZE1CeStwWU1RN3JtWFR2NXcxRmpCWm9RWmVsblNHQnZ0?= =?utf-8?B?cDdNaW5qY2F5SUl0NHM4aUxQTTBub29IdldRTkdId3pUVVA5aW5hU3JCVUxt?= =?utf-8?B?VCtSZVdlaVVIK29LbmNWc2NZQmNGakp4QlVZZWkzcHplTTRYNHZLY1Z3Tm5v?= =?utf-8?B?bllIc00rREhoT0lMYS85VWZ2MWV2TTk5UVlwMVdkamxiUkF6OVAzRGFsN3F2?= =?utf-8?B?Vi95ZGdDOTlqQUpJclFCbGFpMGczY0pldkkwT2d6UFpXVU5vYTFOWG53Y081?= =?utf-8?B?Z1ZhSmNPNzh2M0JRR1gxb0hZREpKTkZlSCtuRC81NWl3NmwwMzBZczY1SmNB?= =?utf-8?B?UXpkTGZFZHl1MERmMHM3QURVZkh3dDZKdFFmdlN0N3ZYaG1Sc3UzaWNFRE1n?= =?utf-8?B?SDRFU3VmYkZHazN2RWo1aDkvQkI2M2RIVGc0T1VYM3V3VTlpZFM4Z2t0d3I3?= =?utf-8?Q?l17eQzjKQK2gpWbEKYkobUpn0?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 146ba86e-6704-4336-6c6d-08da6b6ccc33 X-MS-Exchange-CrossTenant-AuthSource: BL1PR12MB5112.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2022 23:00:25.4002 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /1GHGM/7lCssTIj1qVe41JVn8GuwVmK1iY37B9dGrp4uOgLeWDkvggQiENO9FfYdXIhLfet3h/23/i3cNKDyug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3198 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1658444428; a=rsa-sha256; cv=pass; b=Ftvw2bkkp7v2IzOB3odvJlivbP34F6TEelFbMGUM+YT2oOhPPtaWYMtUPgJXujG8hdnUWi Ur/YNadVxj9Bj5UDJ8A5JZVorT9AJxrD+keauyKxVH6Ozv9e/FkdUa0mX/KWvf0lsPE/8H ZU1hVK5af307iXPHNSISr8Pl6HZaREY= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=Tlw5xA2P; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf08.hostedemail.com: domain of Felix.Kuehling@amd.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=Felix.Kuehling@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658444428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=PEqVtFXBmLsIBkHrerqbCLvmeJuTONa/12j+qhkHvEw=; b=8S7PiuCWgoybHUCX5peKEjq19jfoLQg3SHcXJXs5/ZcOEzJmyF+xbNtINrhaj4b6vbMlXK ofSxL48C9URRlMGZOLPPIFhjhDbEMYwkxZqAwJC/8E30H3/cvPk3MyieAtwgO7d2uaD4VQ pNKI6uGZI+AFsyySMTRvKErCHdhi0Fc= Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=Tlw5xA2P; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf08.hostedemail.com: domain of Felix.Kuehling@amd.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=Felix.Kuehling@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 871F616007C X-Stat-Signature: rhu4k6jgarazh9975jete9hokr4dykas X-Rspam-User: X-HE-Tag: 1658444428-946413 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi all, We're noticing some unexpected behaviour when the amdgpu and Mellanox drivers are interacting on shared memory with hmm_range_fault. If the amdgpu driver migrated pages to DEVICE_PRIVATE memory, we would expect hmm_range_fault called by the Mellanox driver to fault them back to system memory. But that's not happening. Instead hmm_range_fault fails. For an experiment, Philip hacked hmm_vma_handle_pte to treat DEVICE_PRIVATE pages like device_exclusive pages, which gave us the expected behaviour. It would result in a dev_pagemap_ops.migrate_to_ram callback in our driver, and hmm_range_fault would return system memory pages to the Mellanox driver. So something is clearly wrong. It could be: * our expectations are wrong, * the implementation of hmm_range_fault is wrong, or * our driver is missing something when migrating to DEVICE_PRIVATE memory. Do you have any insights? Thank you,   Felix