From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C12CC43334 for ; Fri, 22 Jul 2022 15:34:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A2896B0072; Fri, 22 Jul 2022 11:34:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92A7B6B0073; Fri, 22 Jul 2022 11:34:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 756006B0074; Fri, 22 Jul 2022 11:34:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5DF7B6B0072 for ; Fri, 22 Jul 2022 11:34:17 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 37CF9AB6C6 for ; Fri, 22 Jul 2022 15:34:17 +0000 (UTC) X-FDA: 79715132154.20.20A7CD3 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by imf09.hostedemail.com (Postfix) with ESMTP id 98DC6140005 for ; Fri, 22 Jul 2022 15:34:14 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cauj7cU/yEPD3To30879o2d0vBABKPL0EMA4iHKyyuQ3+LMTA13c/NmBJM+RqN77pLUCJGzPtgwsPlkcypa+ycf5E9lfe4LoiU9/XoeMYJG6DNC1TWJvNTZd8w+KR1tTUnzmAAZuUt207Quy4pv2bMpWO2oCkgbIkwhz4GWfcgEoO5tESSjjVuImztA9X3Zgi8REF4Mgf8ZmFY/EyGq6EaMhT9qwCzd+vlsWeQwNqr/GMEUkBvhU8m/MI0ablTg2EU0HSoeO+MRY88pHbFY2Xd5B6s/p30Cx/N2Zu11lbqwpckPu+XDqXXohhHJA0JOrGfZzFjeHVwBvRj9K6rmrLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/e4CEkVNicDqCkDlTkQ4YQWhDG69eJEfKkvUrLJuESg=; b=Rif5ZrQOkdDVM+wS1On6TEyrfJAsmRNAdQLNIJw9AzHtwjt0F1abRVdmX+dQ7yE0WHgrp8k7EZVbgGgVghW3VR9N6hOUVDAiZmwILcao7TY5hP15TXCHBPejIB+xniTWEf0wT0sR5EX9FlGr3ABGbjKvWJ5c3TOqpip/ggJfwtAY01pDyHsLYvQl6/3OcvxPgPsY1P7u5YkounSKN1Q64WTasxpCV3nqRCHYSF0jepp2/wPRa3Sn5NgY5EKDHBLbspX3Dv/QQ2dEnld9JQFc6wK4kDAfMw/sK9yYX4xOZSL8G/RmdhJpx1izCLYrSkXccLYIdywI4XHNJXKpamk6Sw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/e4CEkVNicDqCkDlTkQ4YQWhDG69eJEfKkvUrLJuESg=; b=f4NyUy9Vj2y4uGmAdEIE+9wfgJ1wYjB38IyD24t12Zzaj7/mVqYNFQxIIBuVuWS+SOf6gP0ENe9IoN7nm74x/J+fFNq5daz1j4ZkQ7FzKM1dgMD6jD+IcgghdrB/DHIZhqnlkDQBp+zmB25EczjpMvMJEIZ9oxRcYSNM0ZcGpBJkNFEpRV6aG/w4iL1rxg8I6b2vzFLgix+7YHnCSqV40ne1Z+rYJ9JP7fyA2w6wEPtg9TPwu0LCos78Suvx/gN4fmbmtCOZOeYYNZd5vuAVEiKQeahhGQAUOsN4WMHW3lUiL+tgrIhmME6BPT0Zwb8gEYS2zy22kjQ5Nl54DfLqKA== Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by SJ1PR12MB6123.namprd12.prod.outlook.com (2603:10b6:a03:45a::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5458.19; Fri, 22 Jul 2022 15:34:12 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::ac35:7c4b:3282:abfb]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::ac35:7c4b:3282:abfb%3]) with mapi id 15.20.5458.018; Fri, 22 Jul 2022 15:34:12 +0000 Date: Fri, 22 Jul 2022 12:34:09 -0300 From: Jason Gunthorpe To: Felix Kuehling Cc: "linux-mm@kvack.org" , Alistair Popple , "Yang, Philip" Subject: Re: hmm_range_fault interaction between different drivers Message-ID: <20220722153409.GB58735@nvidia.com> References: <0d9a7320-e639-ad7b-c45a-644914d73c2e@amd.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0d9a7320-e639-ad7b-c45a-644914d73c2e@amd.com> X-ClientProxiedBy: MW4P222CA0025.NAMP222.PROD.OUTLOOK.COM (2603:10b6:303:114::30) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c4f0dc62-b130-4c4b-01d5-08da6bf7a0be X-MS-TrafficTypeDiagnostic: SJ1PR12MB6123:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cDlX/1rF+bv1ZwRM7rxrfJOZ8dqaHYy51P3m/e9wBUEOHoGEEXkaa7B4ZLMCMlmDFq2Lfuhy+82LXcvPJPasRUoporyZbBGfTny6V4rTV/lkB1eNUSNLyfrYPQiSQSRghNklS97UOaVK9CNqBcbT9tO2eooedx/rLnOcECbm7aw8u1ugb9/sN72VseceNLG1pHkbatAyK15KC4Uy8f8p/DtdCReG/hCYyk/SiMw6RoqrNL5Vj9Vu0m9/k3mtRyVQQrbZEMSPdUw/oq1RjNvtySOSdUW81oYyL17GO1B4fZnsokqy0xQQvRHOJGMwFVkuK6FlUx/qQ4eECQDGyMPstVqGxY7ekwugDF8TvtgG7IVnopGj/W96vIGkn0qRIPco7klhbwXqH+U6IsuHE3fE9WazM3VtIRFE36uuczaXMhhO9ugycyYkw1pzwxXOeMcl29WeTiKtJGNeIOTHuZ0ZRNbgr8Iyqg93XhQsdGKVlgRYj6Q5B948MwZI0IaWue7NDMC30kDmdWDlkfVdoAgfhRKdAfJqqYBLJ0o9+VugRW7ean1jaLe9bRlKjLib9YkdD0ZpSe+TqaNvYKqtp8Y2cA/kCreQCvrDew7TTqf8U7wKJmc/4XALCDt2tEqCGVL+KHC0mzWc1BlePGqRyj6B+RS6QYWVIeTTeDTl8IWbvvr3iIoP+Oq6+BI9x+AViezPJWrnvAUwvuBDRGEn8ft1bBLBX9ywwpTgaiQIP8/FH7P/fQDEp+QuYohWJV0a56H3 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB4192.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(376002)(39860400002)(396003)(346002)(136003)(5660300002)(8936002)(86362001)(2906002)(6666004)(6506007)(38100700002)(41300700001)(26005)(2616005)(6512007)(478600001)(6486002)(6916009)(1076003)(4326008)(8676002)(66946007)(66476007)(66556008)(186003)(316002)(83380400001)(33656002)(36756003)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tUg+z855SeuMToa3aZl4Tj/ZVqene9NbvV5zf/QxaMM6SgNN9F30wULK6EZS?= =?us-ascii?Q?wJ+4wT8DF6oy46aFQWRZJqTKHTnpqOQ9rg6tByNN75shy5n+5zchLgvgzwXs?= =?us-ascii?Q?u40blqyLidaMdR6XQ78osTcam24/v2Ofa7mhSpFzBdcMZU3nleTk656Gn3rm?= =?us-ascii?Q?RFIe4lcAE796HCtt4zUnx8Hfu40Jeys8ghjQJ5dtaDn5WZ6/wcCL9gtjX40p?= =?us-ascii?Q?5sNZO0p9bAtX56UcDxsiQ55RCIV4kPss59hCqw2qAxZEhHbX4ienqsyOGSCi?= =?us-ascii?Q?EBoyMiUzIdyKYjaNKO2ahp+r76cLvUMiEjVCUf/hw1bHIfs/BqlDxR9nANrI?= =?us-ascii?Q?Au6SRUA8+D0vdqfZcNR5Ed5jz3oTlMQBDYsRhEujHzECr8EOyqgBytEQazaA?= =?us-ascii?Q?1bjBCZyZTFqYQEUzotgJovbPKActYF2NiO/DHwD3VySjzq1ae5jsmqQh7CAq?= =?us-ascii?Q?HQI5LBgq3A7TSom00khNPHxwLnaFKtEAAyu3p50NhMkjK1XmyssecCdcQ46m?= =?us-ascii?Q?shBq6FgFOAIVwE+J/uzLuwgEntxVnRaI6k59gwOcfH0ckALIHpdlgdwRRKag?= =?us-ascii?Q?dCb0PiFqz0+9XLJdGasyyfqoyjuSSHnilbfJDvT0Gdfrh+fwmIF9rH0HdujD?= =?us-ascii?Q?xQlhjdKDMKYe8hCWPmFZnIzj902Hx0Swt9tXATiIthzg/gwQ+8okAExHTcKx?= =?us-ascii?Q?izX61y8KtODBn/kK8rt/765sBp5SwSw2Ou+4mfbI1C0GGeH4To+MeAENyWKq?= =?us-ascii?Q?Bzy3iPOhFrKAEnwrLPu8G7scMVHKMpv9B2mkBsJZkIC68kAE6s8eiFfZgtV0?= =?us-ascii?Q?IFq4lQ+3ybRfUhLVAsM5JX3nz9VFK6wkhs/0UA54On/h2V7D0NtI3Svakf6W?= =?us-ascii?Q?EZPtaGZiV9+FfUwZaDkGj68uLryW8s3TbHBkem8X85piVKcbwe+zO0uFmRa0?= =?us-ascii?Q?gOpRJkfXcLKVqGC2qBsuY7x43If/7LUCApebSEkrO4ZPHix1l2ART9V6FmDO?= =?us-ascii?Q?QCF6OesU1IAxyK4hWZG0Zk9Idi4NbDWvFhjJMpER6BRtiXf6Nb9thO9aA9WP?= =?us-ascii?Q?zoxDxfv+VUpxbujcgFi2ldpd53E9KFjBFhOCQqLY0/Sok3urwuzH5GYlIhkr?= =?us-ascii?Q?X/EazonzaZDpWvhcsnFwup4wNjfU74O+CXdVgtXSXaHoWpp7Dz0KtIus6f/x?= =?us-ascii?Q?bhAp57Wnaga2LPj48i8br9Fi6IuZkauatdr4FCUKK+WhjSXkatlpz58DAhr6?= =?us-ascii?Q?bgJ+nZ+ujWJBvttH2esDDVQjcFyaMcUjSzEaPhoOmkPunjTtgvwd4c32zcq1?= =?us-ascii?Q?R8mS4mX6wb/bzsQSNFbLRZuPDdvSxihfhQ75WNBqlDZOD6ssvsLkQ/T4y+I3?= =?us-ascii?Q?N6Mu9kMAA2BfZGWeSacw/QPeepl/7+Oz95XI2OyQ1Y6bPN2iGnL9uxGQKVqS?= =?us-ascii?Q?hq3qDJ9IAhIrrPlkAUwHMiJ+94Wf3H5h8ehyFIcVrf9k3uI9uHsqlNZj8GgH?= =?us-ascii?Q?aVlmgsYW4Cc+9KzPUaoAvu2n95+hguPAO/db3s4x4Javo4oZbSATxAEXjxGK?= =?us-ascii?Q?EphHBTnZ4MrTODgow6Qle0kGgrSDZvCgS6c37a7b?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c4f0dc62-b130-4c4b-01d5-08da6bf7a0be X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2022 15:34:12.5810 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /sGegr+TDg88/dSEjnYKHh7RTnnzuwhBNruFKd2+tT4RheoLHoqaj+/LPGUKeGRq X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6123 ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658504056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/e4CEkVNicDqCkDlTkQ4YQWhDG69eJEfKkvUrLJuESg=; b=qHLV7G7llkINdpQPHJ0dwrRbDEt0dfnwUojDC9QE6UfFPLNkYn84xrwsScSaldqLIrBQRt 6drPIgGmTN15dskp9POYKWn9g5nZLySSNExigeZp4FRIrlfJs/q/A7mp2PKsjHIpyFdkg3 VHy6e/aqlVyfanD/kiyOLMssxsWz+qM= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1658504056; a=rsa-sha256; cv=pass; b=ChP7d6Yxf1CYDAWwI+NwcRIm9qRM7xnlf4D/RWT0+eHLa8aXVeg5Rxne/l29Wo1/npIRFQ p2SfNFs6fMOB2CgzV3HnAYB4gtPuPu3wQbHEr0Byr7y2aB0/B2UFw8rn5nD1rKls8HICXk Y7kA4Sd46KaqX/SXbhizd/Zq4N97IJg= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=f4NyUy9V; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=none (imf09.hostedemail.com: domain of jgg@nvidia.com has no SPF policy when checking 40.107.236.41) smtp.mailfrom=jgg@nvidia.com X-Stat-Signature: t1p8qt3gatnm8s7p9d4fr5uqomc81wes X-Rspamd-Queue-Id: 98DC6140005 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=f4NyUy9V; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=none (imf09.hostedemail.com: domain of jgg@nvidia.com has no SPF policy when checking 40.107.236.41) smtp.mailfrom=jgg@nvidia.com X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1658504054-315781 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jul 21, 2022 at 07:00:23PM -0400, Felix Kuehling wrote: > Hi all, > > We're noticing some unexpected behaviour when the amdgpu and Mellanox > drivers are interacting on shared memory with hmm_range_fault. If the amdgpu > driver migrated pages to DEVICE_PRIVATE memory, we would expect > hmm_range_fault called by the Mellanox driver to fault them back to system > memory. But that's not happening. Instead hmm_range_fault fails. > > For an experiment, Philip hacked hmm_vma_handle_pte to treat DEVICE_PRIVATE > pages like device_exclusive pages, which gave us the expected behaviour. It > would result in a dev_pagemap_ops.migrate_to_ram callback in our driver, and > hmm_range_fault would return system memory pages to the Mellanox driver. > > So something is clearly wrong. It could be: > > * our expectations are wrong, > * the implementation of hmm_range_fault is wrong, or > * our driver is missing something when migrating to DEVICE_PRIVATE memory. > > Do you have any insights? I think it is a bug Jason