From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27E82C433C1 for ; Thu, 25 Mar 2021 18:24:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA5D6619F7 for ; Thu, 25 Mar 2021 18:24:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA5D6619F7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4D94F6B006C; Thu, 25 Mar 2021 14:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B07D6B006E; Thu, 25 Mar 2021 14:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 304416B0070; Thu, 25 Mar 2021 14:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0045.hostedemail.com [216.40.44.45]) by kanga.kvack.org (Postfix) with ESMTP id 163906B006C for ; Thu, 25 Mar 2021 14:24:47 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C2437181AC9C6 for ; Thu, 25 Mar 2021 18:24:46 +0000 (UTC) X-FDA: 77959222572.28.F7F2745 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2056.outbound.protection.outlook.com [40.107.243.56]) by imf10.hostedemail.com (Postfix) with ESMTP id 8FC8A40002CC for ; Thu, 25 Mar 2021 18:24:44 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ScNHyoBsZoTNWfhjtb0YmD8QeeTtJZcnmtUuzyj1agyp/Jboonu62fE9UobVKdpRPQppmdoSnz4yB/xdJFR9Ak5mXqSzKLyrJxE2N42egGJD9L4kSAEOalWHTGIRzIF5AwdOLQwImllyhChAaJaaDuqPPw17KrVZj4FhB/3HOLNo6oyRZO9LegRU5aipMHh3ee5aAQ35op9R67eqUOmw0uFpSKL5aBaY2kc8SBbjkckDYrEOAT/+E177AOWqq6tHXTqXk3R44gq3kRZRRQpgo4gIdRW0X/7yk1ClOT9Eksjf3c0WgMM+vWUHDj05wqTNFCwvVBtav98jD2ngd2SR0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3SqROZLZCOOjBoKMVH+WjvAQ7Q0h2QgpzUijtUHTYMM=; b=S2g9JQnQgNbTIsEzJvNp/QPlDuKZcJhENHrGys/mM4F50v2Hp++XRdgiAjcm8qNKAxN5lpu4zA3tb31FxpslObFRnjXUDAqXnqOfvBARwkqJL08Dazvo/K48p1ytEN2tHy7KqL7CfmhSmTWgvOiLuMQpwt2jHy6V2MFeah6onuQBhRTE92fdR1Qgdx7TrPPUBvPeQdseyh0fd9jD73Xh0TrmyS9bta0uvX049ctjjUy01kBJMrJC2bkjjhvGAgs55htKF/9kdS51jXDxjVrArvdik0sV9ZZqXvbzEsbYbKm6tw85CGikPHs9bKVH2yet/KWyDCUwjSpoUkwCS93z2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3SqROZLZCOOjBoKMVH+WjvAQ7Q0h2QgpzUijtUHTYMM=; b=F0l8XvZAJJO8/y5M1bbNLbZUZEl1mas9T/K689DSfyxzW1Gsu7ugw7BrgaEDKqVUtc+JENbf/MiRhxpuiqdEGfIJ8zhhPIWnluElM1FILtqNFX7ZWhbHlnwQdD8l0X696Cf1DDh6ARgPx+LAh3MI1aCwDo1s4FLMbZKOxYC9/34KmYsW4fyB9GQN5V2ITEYpCi6Er9wjxwSDBh6FXECYxzhbcZ8F9cD7mbpIKLMTXyr4mu6lkevGtiTkt48qpPYXnLWfezZaHWZ0kGUS6/ntnrMVJAkAQyscwNOkj7fzEu1hoh9+KesN/1kmsCZZ9XkBMlVpRAmM2e3sYhwdN4upVQ== Authentication-Results: shipmail.org; dkim=none (message not signed) header.d=none;shipmail.org; dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) by DM6PR12MB3514.namprd12.prod.outlook.com (2603:10b6:5:183::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Thu, 25 Mar 2021 18:24:44 +0000 Received: from DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::1c62:7fa3:617b:ab87]) by DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::1c62:7fa3:617b:ab87%6]) with mapi id 15.20.3977.029; Thu, 25 Mar 2021 18:24:44 +0000 Date: Thu, 25 Mar 2021 15:24:42 -0300 From: Jason Gunthorpe To: Thomas =?utf-8?B?SGVsbHN0csO2bSAoSW50ZWwp?= Cc: Dave Hansen , "Williams, Dan J" , "dri-devel@lists.freedesktop.org" , "christian.koenig@amd.com" , "airlied@linux.ie" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" Subject: Re: [RFC PATCH 1/2] mm,drm/ttm: Block fast GUP to TTM huge pages Message-ID: <20210325182442.GI2356281@nvidia.com> References: <75423f64-adef-a2c4-8e7d-2cb814127b18@intel.com> <6b0de827-738d-b3c5-fc79-8ca9047bad35@intel.com> <9f789d64-940f-c728-8d5e-aab74d562fb6@shipmail.org> <20210325175504.GH2356281@nvidia.com> <1ed48d99-1cd9-d87b-41dd-4169afc77f70@shipmail.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <1ed48d99-1cd9-d87b-41dd-4169afc77f70@shipmail.org> X-Originating-IP: [206.223.160.26] X-ClientProxiedBy: CH2PR04CA0003.namprd04.prod.outlook.com (2603:10b6:610:52::13) To DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (206.223.160.26) by CH2PR04CA0003.namprd04.prod.outlook.com (2603:10b6:610:52::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 18:24:44 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lPUew-002xTZ-P1; Thu, 25 Mar 2021 15:24:42 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 87b372a8-7844-4409-9ba6-08d8efbb4359 X-MS-TrafficTypeDiagnostic: DM6PR12MB3514: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GYdPxnX5s3a9G9N7ZgYZQH+Ez32r3RXm8u9k1vyQe1rmCkkZL2F0RKub8tApqXFuHWkw6h+Pgdx0Mz7139Qz45LiHlpEwbd3Xp8KHT3CQUyl5JXwRs7pnVRm9mxkdc2X2/fZcLYcvxGdvXvsZ85SGrM5MHp310rXxtzO0w02HNbY99Cg56Tfe++CIwZZCZZLSbraHwidtF+RaKRrJ5C5O8Jy4VfC6w401dqtFnl8tW5bYrv/bhFZkmde1z9vgzYQqyRwNImE/NmdJ1ulg+cAlo9wKqm/mWFgUe6szIBoDqUG5EQvHe6yXP6G65zN25KBKShK3ww5yKUtVsT3GqkF2XGnxrrdcVKUxZCQkGo/AtoAYAUA61ePwmC0/G/VRpJ+DqBULf+6MycJMyjOLvVoa0VCMno8yjSj94zkx6Vze5zh/up+LWF1LwgN+ilyRVVHyxP5FcbRkrgVyv7vRjNE49n5qwZdVzPgKr6HallaJondo/qkAA5gk8HT1zUgxVxdB1Iat7Kle4blCr1mBrqiHnFtLqVqbm4GJHDVhnh43U5vQdMKtTa8w7XQP+57rHho6KVv1U8F53ZO/nbvjp3xAwFmGzTsFQqvbmL3qdwQn/iq0y4Fh4A/KLzMR078Ud74fzchiopCa/1XgB5+ib0kqteSWMiNNsYE1xjWk2d9plmCi4HVNB9Ok58mIumKNOzm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(376002)(366004)(39860400002)(346002)(396003)(136003)(66946007)(8676002)(86362001)(36756003)(66556008)(66476007)(2906002)(33656002)(4326008)(54906003)(478600001)(316002)(8936002)(426003)(2616005)(9786002)(9746002)(83380400001)(186003)(1076003)(53546011)(5660300002)(38100700001)(6916009)(26005)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?b0c0djdQSS9QZXRuVnM2ODlySEExYlU0L2VpQ3VGbi91RjZOZWVHYm4xR1hS?= =?utf-8?B?SHVCc29hN0ZBMnIrUXI4YlM3REJXT0JHSmdsTlhLNnNmUlYrQWdGVlRpbHg1?= =?utf-8?B?WE9zUkt6ZmkwWGcxWjZ0VUZoekhick94RHd6WWwyOFVOc0pnd2UrNm1TM3pi?= =?utf-8?B?RVpvQXVNVmJ2bGZVdi9NdFc2NzJGWFV4RCtpYmF1NUoyNk9hRS9abFVjd0J1?= =?utf-8?B?bG5IOW91ZllMdDZBTUFjRkhhcG9ycmludDJodkJ6eEx0NTV3TUtFM2JVMFpn?= =?utf-8?B?dXZEMytRYUUxdDM0S3YwSFpOZ1hCQU4vY3R3b2d2Zy9kWURtN2VIMThnNmNv?= =?utf-8?B?SkE3OTNyeW11VWpIbFJKZkdVNExrYlRhWEpuT0dpTXFMSlZiVHh5UnF6aGc1?= =?utf-8?B?OEZNU3hqVENwekdFSmZKUDNOMk01eTAxS2daejl1UDZMTVlseGd0YVNMamhU?= =?utf-8?B?NW9nbk10Qng4ZlFpQklSSTNjOWhwL1U5ZFlpSVFHZDF0S3c0NjRjOVRqSVo3?= =?utf-8?B?N3gyanlRUTEzVXR1L1FpbjRpUGhkUkl3b0tOSG4reU9yS1R6blVXcnpRTnZu?= =?utf-8?B?bjJueWhqMFVaeGFHTXR6WWFMbXlUOWVHT1JPRUJUcC9GcDR2WlMzL3ZOS3Q1?= =?utf-8?B?cDFVM2dtdXVJZ1VCRmQ0aTRuWU5uZzkraUp3TE5wQS9DdTRKcHRkd3IrRk9j?= =?utf-8?B?ODgxUHo5Nm9KSHdFRGpFbEFDRFJXcDVSWmg4cGdmaXg0eDRxckNjY25waHNo?= =?utf-8?B?dXF4MTJBTmtxbnFJa1hCMHhVRzFkL1diYmoweVhObkxqNVFXNnNkS3V2N1F6?= =?utf-8?B?YzFWT2ZER2h5L1lXUENHVkxnTWFrRGtscnY0Y05iNEdZTU9MNDdYQUNYb2NS?= =?utf-8?B?aU5RNXVjN3cvb3pSY2UzRWdVVGxFTk13b1BFY0ZqcEZBbWRwOXYxMTRLQUlH?= =?utf-8?B?bVBvQVdWOENDeTBXeEc2cU4rRHRSeGhabGdwTThyQXRaNkxrOWdtdEFJOHB6?= =?utf-8?B?OEtCa0hiUVNLZWFTQnMzY1M0eE1JcW5QWnFMN3A2eE0zS0xiU0ZpS3FsY1ds?= =?utf-8?B?aU95ZEZGd1dwdEliL0NJQ091UTY3bTNUY3FadXBVUE5vSmZ1SEloMDY5b3Uw?= =?utf-8?B?cExCSlFBcGxoU3docVNndk5BSUlqM0xYdFY2WFkzNEJxN1dzTlNRWk9tTm5K?= =?utf-8?B?elpkbk12Z0dJbGFsdEFXRnZZd2xka3d5WEpIZ0NaVDN5NHhzdEdkeGJ0NG9h?= =?utf-8?B?UHNXeFUxRWhKWVIwREV1RDhQM2d4NU8rMnJXUkVsVk5NblZDQmd1bDF0TW1a?= =?utf-8?B?bmRHaWw4VnhZZEk1Y3BMUzZXZ3IxNVp6cUtJMXEvNEg5VWNRQWJnemxMSldG?= =?utf-8?B?dUlGQlJPRGJNck9qenh5QUFTV290bk9hVFBQY1ptODB2eE5aVENUZHViUy8y?= =?utf-8?B?VGgyamlzNnpzZnBsUHRqZmduNWJUQUVSTlBGeVQxbi9LOVU5UHk5Y1RrMlFp?= =?utf-8?B?VEtCTjlEQURaRTZOb1lsYjRHaS9UdC9FQ25OY2RVL21ZOGh6WTR1bjE3c1d2?= =?utf-8?B?RG1zcmpEcC9taEhTTXV4RlVvbUI5bGhqNUpPRG1qcmthQXdZWXdCcDZodDYy?= =?utf-8?B?Wmt1ZUg0OXRkdEsyM3NVbVhhM3UrYkVxYVV5cFNZRlZXV3dYclFEWHZPd1dS?= =?utf-8?B?eE1GMVRQNnJYU3hvNFhtUXluQ0VFcHJ2SUtXMFpZZnJQV0Q4QVBLcHJUTVRk?= =?utf-8?Q?yEr38YIpfxn4fmwtZNdR2GXWg4j+cE1NgRUbxM/?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 87b372a8-7844-4409-9ba6-08d8efbb4359 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 18:24:44.4232 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oJ0SUKQ0zxs5emj9lVs3SfnFZaxGIeYa+ZNOPWJi015r2d9/D5XOgAlS6HGw9L5l X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3514 X-Stat-Signature: n7xw4t4eaissshpiwrkd935khnzbcgn5 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8FC8A40002CC Received-SPF: none (nvidia.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=NAM12-DM6-obe.outbound.protection.outlook.com; client-ip=40.107.243.56 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616696684-695209 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 25, 2021 at 07:13:33PM +0100, Thomas Hellstr=C3=B6m (Intel) w= rote: >=20 > On 3/25/21 6:55 PM, Jason Gunthorpe wrote: > > On Thu, Mar 25, 2021 at 06:51:26PM +0100, Thomas Hellstr=C3=B6m (Inte= l) wrote: > > > On 3/24/21 9:25 PM, Dave Hansen wrote: > > > > On 3/24/21 1:22 PM, Thomas Hellstr=C3=B6m (Intel) wrote: > > > > > > We also have not been careful at *all* about how _PAGE_BIT_SO= FTW* are > > > > > > used.=C2=A0 It's quite possible we can encode another use eve= n in the > > > > > > existing bits. > > > > > >=20 > > > > > > Personally, I'd just try: > > > > > >=20 > > > > > > #define _PAGE_BIT_SOFTW5=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 57=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* available for programmer */ > > > > > >=20 > > > > > OK, I'll follow your advise here. FWIW I grepped for SW1 and it= seems > > > > > used in a selftest, but only for PTEs AFAICT. > > > > >=20 > > > > > Oh, and we don't care about 32-bit much anymore? > > > > On x86, we have 64-bit PTEs when running 32-bit kernels if PAE is > > > > enabled. IOW, we can handle the majority of 32-bit CPUs out ther= e. > > > >=20 > > > > But, yeah, we don't care about 32-bit. :) > > > Hmm, > > >=20 > > > Actually it makes some sense to use SW1, to make it end up in the s= ame dword > > > as the PSE bit, as from what I can tell, reading of a 64-bit pmd_t = on 32-bit > > > PAE is not atomic, so in theory a huge pmd could be modified while = reading > > > the pmd_t making the dwords inconsistent.... How does that work wit= h fast > > > gup anyway? > > It loops to get an atomic 64 bit value if the arch can't provide an > > atomic 64 bit load >=20 > Hmm, ok, I see a READ_ONCE() in gup_pmd_range(), and then the resulting= pmd > is dereferenced either in try_grab_compound_head() or __gup_device_huge= (), > before the pmd is compared to the value the pointer is currently pointi= ng > to. Couldn't those dereferences be on invalid pointers? Uhhhhh.. That does look questionable, yes. Unless there is some tricky reason why a 64 bit pmd entry on a 32 bit arch either can't exist or has a stable upper 32 bits.. The pte does it with ptep_get_lockless(), we probably need the same for the other levels too instead of open coding a READ_ONCE? Jason