From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CCF1C432BE for ; Tue, 17 Aug 2021 00:42:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B35C160187 for ; Tue, 17 Aug 2021 00:42:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B35C160187 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4DEA56B006C; Mon, 16 Aug 2021 20:42:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 466FB6B0072; Mon, 16 Aug 2021 20:42:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E0866B0073; Mon, 16 Aug 2021 20:42:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id 0D4FD6B006C for ; Mon, 16 Aug 2021 20:42:47 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 98A21182AE898 for ; Tue, 17 Aug 2021 00:42:46 +0000 (UTC) X-FDA: 78482722332.21.8F4589B Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2068.outbound.protection.outlook.com [40.107.223.68]) by imf17.hostedemail.com (Postfix) with ESMTP id 0A75CF00CC8F for ; Tue, 17 Aug 2021 00:42:45 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CzAXRwwYxWVd2C7Njd9z3/Qi9/GkNgUUXAwmwjWxU+ubIHrDYwZEizhbaGr6XQSsv2z+vMqAu95th2GYp1aiU5W3Gx7XunSMUrqfXaUU/7XSG+QuK719i0dA4/aaTl9CyaYst18wDpPaBi7IC2Gxyvj8pctvF9v07K9CNM3C3PaHE6IGm42QSj7fDNKWLIWWiiWRuz2MMHK8S5a4HwSeFNsfgFRMD7xKY93CtiDRnuhe73U7PiLyozuIcFOZu0oOFR0uc1Au1N/U+ZSc8RAAuxvkDqFUiX39CoNWM12ngaIxxRzUKp1SMnemXbKepaADIOmmlgJAg67M/+QOfm6PBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AWujEKxUIya+IDDKvwsoxAXIzzfs0U77W0RgfmSaw9c=; b=jmfzbYWeD+C2KxCs2Egx44XEnXnBr3pN7nrLizWgMIyusjxvahIIIJsxZDoDQEpMdfbgFtX59SWGFZn7oawljb5P4xzBl1VXPuefyq7L8tOd7arqsQvarC1UvvCPSYt6fdMEWJSX8+xLunFe93PEFo3HltFrtWY7sD6hdwHOxF5kbzK5u94fhg4U9thTz6CBJSXbA04/WTtzEr3M0JvU7QRedJhbAb5L++jkC0VU7M064s95R1+k2zN91/HEhEnNWWAFhBiEt5SSTyixBeWkD9hoDpA0t7o80LDZnHuNIKwFw9t47m88UMW2SjZqwUvtbssLM2DJs362Lz9lcjvt1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AWujEKxUIya+IDDKvwsoxAXIzzfs0U77W0RgfmSaw9c=; b=hHYalWn1qq+qUs5q498zH1l9ponXCQy2FoHOsnDaO5ycWxvfdc0GOukFBc69qfHvQDjevz3ZDM38DBstkMEBA8L/U/FiILLW+YSu8g3hlcUqdTQSVYuYXVmDL4F/tV1cSSFpvQhCjouhV+3yMqnWm+cFyrh5HLxgbzEjt6yi+Ms= Received: from BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) by BN9PR12MB5100.namprd12.prod.outlook.com (2603:10b6:408:119::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.17; Tue, 17 Aug 2021 00:42:43 +0000 Received: from BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::b891:a906:28f0:fdb]) by BN9PR12MB5129.namprd12.prod.outlook.com ([fe80::b891:a906:28f0:fdb%4]) with mapi id 15.20.4415.023; Tue, 17 Aug 2021 00:42:43 +0000 Subject: Re: [PATCH v6 05/13] drm/amdkfd: generic type as sys mem on migration to ram To: "Zeng, Oak" , "Sierra Guiza, Alejandro (Alex)" , Christoph Hellwig Cc: "akpm@linux-foundation.org" , "linux-mm@kvack.org" , "rcampbell@nvidia.com" , "linux-ext4@vger.kernel.org" , "linux-xfs@vger.kernel.org" , "amd-gfx@lists.freedesktop.org" , "dri-devel@lists.freedesktop.org" , "jgg@nvidia.com" , "jglisse@redhat.com" References: <20210813063150.2938-1-alex.sierra@amd.com> <20210813063150.2938-6-alex.sierra@amd.com> <20210815153825.GB32384@lst.de> <694ea624-9dc7-7a25-78a6-308ee0debaea@amd.com> <771FC2D4-15CF-464B-96FF-837533A85886@amd.com> From: Felix Kuehling Message-ID: Date: Mon, 16 Aug 2021 20:42:40 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 In-Reply-To: <771FC2D4-15CF-464B-96FF-837533A85886@amd.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US X-ClientProxiedBy: YT3PR01CA0067.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:84::26) To BN9PR12MB5129.namprd12.prod.outlook.com (2603:10b6:408:136::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.100] (142.186.47.3) by YT3PR01CA0067.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:84::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Tue, 17 Aug 2021 00:42:42 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9eb97e19-ebe0-4271-9851-08d96117ec6e X-MS-TrafficTypeDiagnostic: BN9PR12MB5100: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z1QR+TsAa+3dxtWXAvxYqhG5MtlvWBv/ue8ICEE/dicPKtoSwRKB04IYVtjY+nRUEdqxtQZV54n30nZ4i3lBkQrcLZo6NeMpIJrZwgl8qZXkbumIqdNYDO3t2VN+WFDICsF14AP/fLj62nCkjVAMSovG/enJeTrWnJW7N4qa8q8B8F9V6lDd+IBJJOWw6uRsQMW8pqe2Hkkn4KPAvzQNBdSph9WKaO2bbO38YNXAH9aLXdwGKNmt6zDsutA1dODUfKhfG/qakf6izXvZElS0UsXxr2Dhndpnn/ivwHs5EtOG4RcnHk6DdUrRxKMKc51oRllyzUw1Lpbl0hyx2b7xblZDRSxsBVqTNGCSyzhHupP8NTzgkwVYxXe4jdarRhJaisziV1sTlDuu3Tg5kvwHDhIV4KTGhdeXz7h1bpk3n54pA/ZrB1NRL6wBniXLBCCxOeZkQK1cYb/x/77NLABtarfd6NtTmczpfVT1AYVq1rtqqLjzhKeq0d65u/+GkMG5d2fkrKAm13gu9KTTvI1m9JpWoxKyobWDwLM7GBI4gJybKWrHARyHHSjysVAGookxM0YxRXhOodfeTcC93x4RqHt9hPUDzi9GiaXLpdfYnRWC3WAwZ5zJ4sOB324lDD5Ij+s9kZsW0xcAd+8hc+LvzLBrjP40pt8k77iqVepQRw4pGwJD0e7cMaGJllCv85ZiDFqSuAetg6/jraC+YLYg/hRFmK2OuCDxmwzD4p65Myl2zuVXmiutIkHi8Ju4WPkRdKTiN1L1k5eARfzwyqlBWA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN9PR12MB5129.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(366004)(136003)(346002)(376002)(39860400002)(31686004)(66476007)(66946007)(54906003)(110136005)(86362001)(316002)(53546011)(66556008)(31696002)(16576012)(44832011)(4326008)(83380400001)(956004)(186003)(2616005)(478600001)(36756003)(8936002)(7416002)(26005)(38100700002)(6486002)(5660300002)(8676002)(2906002)(41533002)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NjdOK3BFakNQWVk2QmlSUmZ6UnJjL245Z2hoajdIOWxkMzh6UFVTUGxPbUhM?= =?utf-8?B?a2VKN2NMbVdkUXFsQTRKV1lURFNrSmlMOEVScFROWHhTN0o2YWlaWURBV2tx?= =?utf-8?B?bnZrOWI1ZzFidGREaEJGeHAxNDNmSzZzV2M3RmdIYVNNZ3ZmY252OXV3VTlw?= =?utf-8?B?V2o3ZmxEcHEyRitGc0t4UExqbUhaU2ZaaVdtMFNtZGE1QURFRmR1MnIvNUov?= =?utf-8?B?MVN2VG5Ia2dXKytLNFdML0w5SFRENW4ycG45L3Qyb1RGWGRwcG9JUkdFZ1Ay?= =?utf-8?B?QU1DdjRjNktEN3p5cTlreFVKQURtVmxibUEzc3M1YmJFdVFDcXpjYnNKR2xW?= =?utf-8?B?MmM3TklESW9MRXhydldxOGlQdGo2a0J3bWlrYkFNM00rYWswdWpyYUFxU2FW?= =?utf-8?B?TmRxczNFRjhJbmo0QnFCVHo5Q2szT0t2OVIrTjlLcUc1ckNQNUdydzJVVWpj?= =?utf-8?B?MWlDemJlZ2E5SVJCNHhkMTMyOFVuVFh0WElNb0w0N3FETmNpNVNjK0VUSTBo?= =?utf-8?B?YnY4WW1SazhoR2R0VGR0K0hXRkhlVkVYOVRQTmdKbjBaTGF5OEpwc1VjMTFj?= =?utf-8?B?OEhlSEZaalIyV25aYUdEbGpzMUwvMFhOeTRWbEllV1o5UWVIWGp1U3U2RWd1?= =?utf-8?B?bGlhV0Y4WE9RMGg0VGYrL1FYbzgvRGo0Vi9saWFyUkxKUUlrWjF6SFZ2N2Iw?= =?utf-8?B?cTlVaFFtTUhBMUNJQjVkcHFpMGJvdkhWYVRBbnZ5QWptQ0pveW5nMkFVbVVj?= =?utf-8?B?QmNycHltL1FlM1VzeXRtdTZrTm8zWjJGbThkRlpaL0g5NmlUQnQ2M2p0ci9O?= =?utf-8?B?WHM1cG9abjdTTERFWUs2YllRMkNEZ0ZNMGNLRWx1VHZNeVJ1K0hXMDBJMG5a?= =?utf-8?B?cktydFUxQ3ZMNzhvVUh5VmplcSs2YkFMZVo0ckpSKzQ0UzJaUzRlUWRsSkhJ?= =?utf-8?B?c1ZnOFJHYnNxaHlWS2hyZGdQU3V5amtUL0xQZld3OGN1QWxYVWhtUU1aL1dz?= =?utf-8?B?Z3ZBdUxTbGY5a3FuSG0vUklMNHNlUmUxbFI2MzFPVTMvbG9RZkJOVDMrNENw?= =?utf-8?B?ZjJTVDR1d3VRUDN5U1NIajlVdk93LzNRSlRSeDhMYi95OUZyQ2ZIV2orNzRv?= =?utf-8?B?cWpkeDdVWXpVaGlNYjFEVFZKTTQzZ1BucjRJUm9TaVZIZVBiRXl5OTdERm5I?= =?utf-8?B?Z0d3dGoxV3p2NTZoMCtkeFdDVS9uMk0vSWpRQmtLcDhXOStEL0FNR1lBeFAx?= =?utf-8?B?Qk5iZEIrQ2crYmlscy9TQnI1NzI0L3o0VWM5dGhjVVdSUUZWUTRoMmtsSUg5?= =?utf-8?B?RTBoTWU3ZHI5NzFGQVYxeVRxdHF1N2w0eFRLcWpoaGczSVpHMmFnbG5MYzFC?= =?utf-8?B?ODlCSFU1d1JhZjhCTng0aTFIY3J2M0hSUCtOcjVjRUxMcE9YY1p1aHBITmVn?= =?utf-8?B?bHZkdnNwN3p2TXc2MDNCTUxqb0grVGlpbE8wbEJGbzNXRFJVK0N2a05FMVN6?= =?utf-8?B?aVZld3FXVUpiWG9pcDBYNWlsUVkrdkhvM09GSm5VTGREbDdYdHgyTy96anNP?= =?utf-8?B?QnNOUjdPbWpjSi9KSjhuNWVCWFhmQlR4TjU0NXM1eHA0OFo0YmFjNzNiQzdN?= =?utf-8?B?dXkxMTR4VFJGNDFMdkxseGcrN0lOcStxcndYdVJFcEtsVFk0UWdtOEFXUG9k?= =?utf-8?B?M0dxa1h4MEtuQ203aFZqMTVTbisxQ2hsU1B1MWIvUnlkeStRcjNEb0V4S1Iz?= =?utf-8?Q?Om8fpo/gHkhPqE+CXOAovPngn9AL87qsiRmY0pw?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9eb97e19-ebe0-4271-9851-08d96117ec6e X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5129.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2021 00:42:43.0161 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 33SussHXnbbHfQMgqzrMWgAqMh521pIUOmQEZsuFY+VHi0C565Nf4zVTsO9uG14PfZHDw6Jy+WByP8bo5DVsbg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5100 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=hHYalWn1; spf=pass (imf17.hostedemail.com: domain of Felix.Kuehling@amd.com designates 40.107.223.68 as permitted sender) smtp.mailfrom=Felix.Kuehling@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 0A75CF00CC8F X-Stat-Signature: 3ubyk5eno96p47ajrdsjgntujzwtb68g X-HE-Tag: 1629160965-916907 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Am 2021-08-16 um 6:06 p.m. schrieb Zeng, Oak: > Regards, > Oak=20 > > =20 > > =EF=BB=BFOn 2021-08-16, 3:53 PM, "amd-gfx on behalf of Sierra Guiza, Al= ejandro (Alex)" wrote: > > > On 8/15/2021 10:38 AM, Christoph Hellwig wrote: > > On Fri, Aug 13, 2021 at 01:31:42AM -0500, Alex Sierra wrote: > >> migrate.vma =3D vma; > >> migrate.start =3D start; > >> migrate.end =3D end; > >> - migrate.flags =3D MIGRATE_VMA_SELECT_DEVICE_PRIVATE; > >> migrate.pgmap_owner =3D SVM_ADEV_PGMAP_OWNER(adev); > >> =20 > >> + if (adev->gmc.xgmi.connected_to_cpu) > >> + migrate.flags =3D MIGRATE_VMA_SELECT_SYSTEM; > >> + else > >> + migrate.flags =3D MIGRATE_VMA_SELECT_DEVICE_PRIVATE; > > It's been a while since I touched this migrate code, but doesn't = this > > mean that if the range already contains system memory the migrati= on > > now won't do anything? for the connected_to_cpu case? > > For above=E2=80=99s condition equal to connected_to_cpu , we=E2=80=99= re explicitly=20 > migrating from > device memory to system memory with device generic type.=20 > > For MEMORY_DEVICE_GENERIC memory type, why do we need to explicitly mig= rate it from device memory to normal system memory? I thought the design = was, for this type of memory, CPU can access it in place without migratio= n(just like CPU access normal system memory), so there is no need to migr= ate such type of memory to normal system memory... > > With this patch, the migration behavior will be: when memory is accesse= d by CPU, it will be migrated to normal system memory; when memory is acc= essed by GPU, it will be migrated to device vram. This is basically the s= ame behavior as when vram is treated as DEVICE_PRIVATE.=20 > > I thought the whole goal of introducing DEVICE_GENERIC is to avoid such= back and forth migration b/t device memory and normal system memory. But= maybe I am missing something here.... Hi Oak, By using MEMORY_DEVICE_GENERIC we can avoid CPU page faults triggering migration back to system memory on every CPU access on the Frontier system architecture, because such pages can be mapped in the CPU page table. You're right that this is the reason for the whole patch series. But we still need the ability to migrate from MEMORY_DEVICE_GENERIC to system memory for reasons other than CPU page faults. Applications can request migrations explicitly (hipMemPrefetchAsync). Or we can be forced to migrate data due to memory pressure from other allocations (evictions in the TTM memory allocator). Regards, =C2=A0 Felix > > Regards, > Oak > > In this type,=20 > device PTEs are > present in CPU page table. > > During migrate_vma_collect_pmd walk op at migrate_vma_setup call,=20 > there=E2=80=99s a condition > for present pte that require migrate->flags be set for=20 > MIGRATE_VMA_SELECT_SYSTEM. > Otherwise, the migration for this entry will be ignored. > > Regards, > Alex S. > >