From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC198C636D6 for ; Wed, 22 Feb 2023 16:40:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 513586B0074; Wed, 22 Feb 2023 11:40:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C4F76B0075; Wed, 22 Feb 2023 11:40:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 363916B0078; Wed, 22 Feb 2023 11:40:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 234146B0074 for ; Wed, 22 Feb 2023 11:40:24 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E4968140592 for ; Wed, 22 Feb 2023 16:40:23 +0000 (UTC) X-FDA: 80495490726.28.85F03C8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id B4DEA1C0015 for ; Wed, 22 Feb 2023 16:40:21 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dNQo9NV6; spf=pass (imf20.hostedemail.com: domain of dakr@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dakr@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677084021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sKvLH39Md6VFASwNimh3Ijve7a+l/n+mf16qXiFcEQs=; b=N5MlPr844YfLaF2KZH7KXM1b3v3uCtvqjoXHFaGfxZh2ntHQ+fkQ2moZuPBEGZcepwcvvz KCkMdRCGLNGiL0Br2ACbxn+bTaVTyslfzaVzDK1x/UfpAG9fM94SnirGySxA2dW/V7aCDR PLaC3N7NIJ/EZghYKKooJP+hHDLVSos= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dNQo9NV6; spf=pass (imf20.hostedemail.com: domain of dakr@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dakr@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677084021; a=rsa-sha256; cv=none; b=IYt0JMzBQJda3KrmnjkwtioiJPDUY+zVxL9sGn78y7jQXDms0vtUeP00ltY6EXjGxhJcmk inYxZ5MR5VVLAeSCEracNNHs7XDIHYMQluXS9DGMLjcO4E0qwptkbjRaYF1D71Si7sJOIg 4mQ4emlVgqtPz4hx8y5sUd9M0edDBBQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677084021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sKvLH39Md6VFASwNimh3Ijve7a+l/n+mf16qXiFcEQs=; b=dNQo9NV6a26VtC4fu6Vx5sSoqATrNkwQZSxZR+HK9uHKkH6dWZ2u2qNUj1e6Q6CBxK/DeZ Bw+BhemKhJcfA2bsxcrddgQZP3TV+c/sRgUhqxhwTGNINnhc6UftsoVTZXr+9SGDCgbuqx VcMwWQPt5Bu2PM5cv6q/NmYzC6MsKcI= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-589-sLIj9U4bNuqXk18P6f1T8Q-1; Wed, 22 Feb 2023 11:40:17 -0500 X-MC-Unique: sLIj9U4bNuqXk18P6f1T8Q-1 Received: by mail-ed1-f69.google.com with SMTP id dm14-20020a05640222ce00b0046790cd9082so12000956edb.21 for ; Wed, 22 Feb 2023 08:40:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sKvLH39Md6VFASwNimh3Ijve7a+l/n+mf16qXiFcEQs=; b=42YMQx/Wj+gqySY9jVQLW65XgZAN27T6Kj2eO5CnfknfV+X1Ey0ynUY0Is6yaVDnOo mqXXfpoCpdvRD4xHy7gHXgbOhpS4+GqO6TcTmeiiIyktsIadpn13MI29ZjkU6of7gNFW xf2j2jTBiNWl6IBl+MbAlQR2lYe9rc9NURvWAEu9HSWAc47p7W4dY9kt8hxnVIkqvB5k OD6VdscmgBaO5T/cDhQWZgQdv0SxycprVpKk9iqQ0QRsWfGAPiPsqu4fZ0HYLX/6m8SE +bUofRFh6V8FdUCjXnK9TCpSgw2l7UbNUzRRc35EFb+v8HqwC/J4MAfdGxTXH6qOCs3w Slvg== X-Gm-Message-State: AO0yUKVUUUYNUTxoS3jvA3e7IyjV5WIKhZNH61XKnD5Tieps1XSU+e5D s9Yhn6jHWqIwyjMJq0ggaSnqOlSRWkYaX8wBrk0iBjalE4lx2ROwtRfRZ/Uk4dLIxrtr/XmvJKU SOS/Mz1/KOn8= X-Received: by 2002:a17:906:58c6:b0:8ea:825:a5db with SMTP id e6-20020a17090658c600b008ea0825a5dbmr1975993ejs.76.1677084016602; Wed, 22 Feb 2023 08:40:16 -0800 (PST) X-Google-Smtp-Source: AK7set8y1aIvy6/M9izm4ZRhyULPWeIfqPOxVNzTRzQDPPo3HY43nOvrd3fmMTJNtsFtgPLgH4e/yw== X-Received: by 2002:a17:906:58c6:b0:8ea:825:a5db with SMTP id e6-20020a17090658c600b008ea0825a5dbmr1975964ejs.76.1677084016324; Wed, 22 Feb 2023 08:40:16 -0800 (PST) Received: from ?IPV6:2a02:810d:4b3f:de78:642:1aff:fe31:a15c? ([2a02:810d:4b3f:de78:642:1aff:fe31:a15c]) by smtp.gmail.com with ESMTPSA id v18-20020a509552000000b004a23558f01fsm3756214eda.43.2023.02.22.08.40.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 22 Feb 2023 08:40:15 -0800 (PST) Message-ID: <83755119-083d-7d66-fca0-ca306c841d9c@redhat.com> Date: Wed, 22 Feb 2023 17:40:14 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [PATCH drm-next v2 05/16] drm: manager to keep track of GPUs VA mappings To: =?UTF-8?Q?Christian_K=c3=b6nig?= Cc: airlied@gmail.com, daniel@ffwll.ch, tzimmermann@suse.de, mripard@kernel.org, corbet@lwn.net, bskeggs@redhat.com, Liam.Howlett@oracle.com, matthew.brost@intel.com, boris.brezillon@collabora.com, alexdeucher@gmail.com, ogabbay@kernel.org, bagasdotme@gmail.com, willy@infradead.org, jason@jlekstrand.net, linux-doc@vger.kernel.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, Dave Airlie References: <20230217134422.14116-1-dakr@redhat.com> <20230217134422.14116-6-dakr@redhat.com> <70ba382f-1559-289a-4922-ca9c371aaf59@amd.com> <29ea3705-5634-c204-c1da-d356b6dfbafc@amd.com> From: Danilo Krummrich Organization: RedHat In-Reply-To: <29ea3705-5634-c204-c1da-d356b6dfbafc@amd.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: B4DEA1C0015 X-Stat-Signature: 1y5aihne1hjy8gucuuzqhw7jxqu8qg4a X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677084021-434821 X-HE-Meta: U2FsdGVkX19H7ox+7htHtqdcT6LQn8Z+UL2y5bnyqYrjeHKithag4aTZjtDvKgrdPlvGVRLIYbfDubd8R0FzMQnPd+2Mgf5I+PTLETftXH1m/IHsWkamqgQ9ntpdd7udRUcRg2wfMlvLPsiO2gbc8I5/mLQqic21lpU5xpse8Le+u5ytdn2LJ2NCErlTA+XdzwfTMGFglOPWqLFewfLp0FBrXNOoDsa6xJOG6s7C+UXa40m9sRtnyD8CZKFM15K7hZsLOB0uerkiID/ycUNMK9NyfiOfdEFXjHlSAvAHNABNOgDOtM9uOmU9qZLCiweXGhI+p8s4hv6RUlLMcCOZDe8igpH/Nuy34DFv0nfZxfgCSOcCwUmAGUIbPfZHQ1tEsdd1OySPzkHhis1cBN9DCOrTmWfw7zq2AQek/aTASbNwFd7oINkQM2ZnRLzVxO+Gt5r/R2e06dNjZZvtsKPXpC0tT1RwXzccZ3oSw+rCel9I5yk+jP5TEhXZ3yc/Jr3EMqSzYwbhUBirqb1Vt1jVek5i5pBa7NurLaIZh+ljRH9aYAwW+ns3v6X2p8RgrG0KXFYAATj0tbl4tBX6VTzo/BhbU9T5zRQo1mtS3PNTCDtPABRomZtqZSYWPeOGun7tJZNeZJOY9NfYrmu10cdiwygojq++6ZKuSARH0sR77EGqpK7hb8yg9Cuw3xjndaMqry3ho/hwJbUX5ENK+0j9snUHyHKkNypUeMo3InShgzI9IE0XDLSICRwO+IE3aAcKniU82N1/74fJjfdIEITV4TEAjLx8wooqCxFlSrEzD8t24oNIbhBtTjrsjRL135dAd7W4Xi0YenGRmlitDwmVqiv3SB93yIBvHZj0sTPE46lHqDCr+QdnuEtbFO9M3H/LYNCDNBqFd78tBGjwGdx0t3OHPFKVHv+yMOO9OJA8I4Xja1ODJ2gB+axiK9Fet2Zo1kAdIH6kFG5p+4ydzEr d9WkTs29 42qKx3Tyfps3aidFtns8e+BKBI0GxqjWWX+j3lDzYUDzV/qklBUwEolki/BKryePzDLT6+hV/TZLPKXbpZcAQ2lIXfGb/ZgZcgiRDKR5GOh6aDoip7aBV67jOAdku1h/q7bjRtYKux9iicpgPHX89QNaOxNyUpwGboH2mFu2J9SfdL7i+YArI9OfQBqTjBxm0t9/awPQ+8D9Vof0SYwA7YHgzmWoZez9PRAK/DsW3Ku516jYMeFzC4brGP6Yg0tS5aTMEV4wlfKDQmfKIdxfNtN6JReyGooDtqT1hgGacq1bNMTKlmp1F1+aiKzs5pN3SqTCdbB6ymdGwk18SUukUTQCkWTBrt4JHZZEZPoO2FtqwLH9YREtKKpGNUmre1J1JE3WTtjacL9Wmpjw1zVP14jJ3LQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/22/23 16:14, Christian König wrote: > Am 22.02.23 um 16:07 schrieb Danilo Krummrich: >> On 2/22/23 11:25, Christian König wrote: >>> Am 17.02.23 um 14:44 schrieb Danilo Krummrich: >> >> >> >>>> +/** >>>> + * DOC: Overview >>>> + * >>>> + * The DRM GPU VA Manager, represented by struct drm_gpuva_manager >>>> keeps track >>>> + * of a GPU's virtual address (VA) space and manages the >>>> corresponding virtual >>>> + * mappings represented by &drm_gpuva objects. It also keeps track >>>> of the >>>> + * mapping's backing &drm_gem_object buffers. >>>> + * >>>> + * &drm_gem_object buffers maintain a list (and a corresponding >>>> list lock) of >>>> + * &drm_gpuva objects representing all existent GPU VA mappings >>>> using this >>>> + * &drm_gem_object as backing buffer. >>>> + * >>>> + * If the &DRM_GPUVA_MANAGER_REGIONS feature is enabled, a GPU VA >>>> mapping can >>>> + * only be created within a previously allocated &drm_gpuva_region, >>>> which >>>> + * represents a reserved portion of the GPU VA space. GPU VA >>>> mappings are not >>>> + * allowed to span over a &drm_gpuva_region's boundary. >>>> + * >>>> + * GPU VA regions can also be flagged as sparse, which allows >>>> drivers to create >>>> + * sparse mappings for a whole GPU VA region in order to support >>>> Vulkan >>>> + * 'Sparse Resources'. >>> >>> Well since we have now found that there is absolutely no technical >>> reason for having those regions could we please drop them? >> >> I disagree this was the outcome of our previous discussion. >> >> In nouveau I still need them to track the separate sparse page tables >> and, as you confirmed previously, Nvidia cards are not the only cards >> supporting this feature. >> >> The second reason is that with regions we can avoid merging between >> buffers, which saves some effort. However, I agree that this argument >> by itself probably doesn't hold too much, since you've pointed out in >> a previous mail that: >> >> >> 1) If we merge and decide to only do that inside certain boundaries >> then those boundaries needs to be provided and checked against. This >> burns quite some CPU cycles >> >> 2) If we just merge what we can we might have extra page table updates >> which cost time and could result in undesired side effects. >> >> 3) If we don't merge at all we have additional housekeeping for the >> mappings and maybe hw restrictions. >> >> >> However, if a driver uses regions to track its separate sparse page >> tables anyway it gets 1) for free, which is a nice synergy. >> >> I totally agree that regions aren't for everyone though. Hence, I made >> them an optional feature and by default regions are disabled. In order >> to use them drm_gpuva_manager_init() must be called with the >> DRM_GPUVA_MANAGER_REGIONS feature flag. >> >> I really would not want to open code regions or have two GPUVA manager >> instances in nouveau to track sparse page tables. That would be really >> messy, hence I hope we can agree on this to be an optional feature. > > I absolutely don't think that this is a good idea then. This separate > handling of sparse page tables is completely Nouveau specific. Actually, I rely on what you said in a previous mail when I say it's, potentially, not specific to nouveau. This sounds similar to what AMD hw used to have up until gfx8 (I think), basically sparse resources where defined through a separate mechanism to the address resolution of the page tables. I won't rule out that other hardware has similar approaches. > > Even when it's optional feature mixing this into the common handling is > exactly what I pointed out as not properly separating between hardware > specific and hardware agnostic functionality. Optionally having regions is *not* a hardware specific concept, drivers might use it for a hardware specific purpose though. Which potentially is is the case for almost every DRM helper. Drivers can use regions only for the sake of not merging between buffer boundaries as well. Some drivers might prefer this over "never merge" or "always merge", depending on the cost of re-organizing page tables for unnecessary splits/merges, without having the need of tracking separate sparse page tables. Its just that I think *if* a driver needs to track separate sparse page tables anyways its a nice synergy since then there is no extra cost for getting this optimization. > > This is exactly the problem we ran into with TTM as well and I've spend > a massive amount of time to clean that up again. > Admittedly, I don't know what problems you are referring to. However, I don't see which kind of trouble it could cause by allowing drivers to track regions optionally. > Regards, > Christian. > >> >>> >>> I don't really see a need for them any more. >>> >>> Regards, >>> Christian. >>> >> >