From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96221C47083 for ; Wed, 2 Jun 2021 14:37:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40ACF610E5 for ; Wed, 2 Jun 2021 14:37:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40ACF610E5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C0F296B0092; Wed, 2 Jun 2021 10:37:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE60B6B0093; Wed, 2 Jun 2021 10:37:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A86E96B0095; Wed, 2 Jun 2021 10:37:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id 76E816B0092 for ; Wed, 2 Jun 2021 10:37:39 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DEA1F824999B for ; Wed, 2 Jun 2021 14:37:38 +0000 (UTC) X-FDA: 78209037396.22.70BC873 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id A83D7A000269 for ; Wed, 2 Jun 2021 14:37:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622644657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=m/QwAefnsRXdsP3/z6cvtajoG5iQ+RdDqTRbmixBgeQ=; b=PZxBvMZW1FC2WzalZzxVNddLZQ7A8lL3M9+gn5JxFs8gcOsXl4jp+uwZDmEFGcycCV1aA1 NoZlD+OkmC94VOE5IPfx/QXX28A0TiS8Mw3SEqtfGaRF686w4LNvdDX6MiWP+qgmDDEMZx mVG1cOH2v/Ksqd7KRdLI0kLj1dN0lZ8= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-388-x7vV4ZTAN42CqFcZl__83A-1; Wed, 02 Jun 2021 10:37:33 -0400 X-MC-Unique: x7vV4ZTAN42CqFcZl__83A-1 Received: by mail-qv1-f70.google.com with SMTP id b24-20020a0cb3d80000b02901e78b82d74aso1804656qvf.20 for ; Wed, 02 Jun 2021 07:37:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=m/QwAefnsRXdsP3/z6cvtajoG5iQ+RdDqTRbmixBgeQ=; b=q/lWetGbzH9IFP2rph9nFl4khxDlncoL3B2a3/4zN+89O2R+ITpF6jE0RtRHAcX3LW b74Bbktw0EF17kLqbKGIyBNgiBJZWCbycfCHCRTBs+E4WK8EMioam+xngZ5pCAa2Lkkk oEBU8tlEH5CC9eCqZjSRFLThEhkp9gzS2uKWLnk39QMn8DOLchRHKfgsOwXkYvtw8ibS IqnKlb48VeteCzeMzOExZizHMD4Y01z+QGs/+q8wiWdVF0Nvm1ccY42vjm1Gi+ZVxBH1 K6jBlO6DuTq57aSZ9de3lLMVARlewTnU+hnP+mvNjcCzBpRypxaqzhc5Xb5fP/0YR5ns YDUQ== X-Gm-Message-State: AOAM533nxNFnrx7Lsrh/SDtiR8ovP7E/GVqQAQPlY1Ay6aLaRh9DrywI J6fiK1AqRM7kikWzUZtSrTNB1/o6A190Q3vuxfefTSjwnODiECntDiQpRJJURf8xr7DhshJWmdS RdgS0G15C8Zg= X-Received: by 2002:ac8:5b81:: with SMTP id a1mr24760228qta.303.1622644653347; Wed, 02 Jun 2021 07:37:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxTs6tJ8uqmSORfD2Z3iGZv/0i+S2lgFtI+HHVdQRE2EuDI89DE3ak/M7GvRJs9513WyidOxA== X-Received: by 2002:ac8:5b81:: with SMTP id a1mr24760200qta.303.1622644652986; Wed, 02 Jun 2021 07:37:32 -0700 (PDT) Received: from t490s (bras-base-toroon474qw-grc-61-184-147-118-108.dsl.bell.ca. [184.147.118.108]) by smtp.gmail.com with ESMTPSA id e127sm87950qkf.62.2021.06.02.07.37.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Jun 2021 07:37:32 -0700 (PDT) Date: Wed, 2 Jun 2021 10:37:30 -0400 From: Peter Xu To: Balbir Singh Cc: John Hubbard , Andrew Morton , Alistair Popple , linux-mm@kvack.org, nouveau@lists.freedesktop.org, bskeggs@redhat.com, rcampbell@nvidia.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, hch@infradead.org, jglisse@redhat.com, willy@infradead.org, jgg@nvidia.com, hughd@google.com, Christoph Hellwig Subject: Re: [PATCH v9 07/10] mm: Device exclusive memory access Message-ID: References: <20210524132725.12697-1-apopple@nvidia.com> <20210524132725.12697-8-apopple@nvidia.com> <20210524151157.2dc5d2bb510ff86dc449bf0c@linux-foundation.org> <8844f8c1-d78c-e0f9-c046-592bd75d4c07@nvidia.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PZxBvMZW; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf15.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: pcw741knqys45p99g7x717o96q3mbh14 X-Rspamd-Queue-Id: A83D7A000269 X-Rspamd-Server: rspam02 X-HE-Tag: 1622644650-89068 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 02, 2021 at 06:50:37PM +1000, Balbir Singh wrote: > On Wed, May 26, 2021 at 12:17:18AM -0700, John Hubbard wrote: > > On 5/25/21 4:51 AM, Balbir Singh wrote: > > ... > > > > How beneficial is this code to nouveau users? I see that it permits a > > > > part of OpenCL to be implemented, but how useful/important is this in > > > > the real world? > > > > > > That is a very good question! I've not reviewed the code, but a sample > > > program with the described use case would make things easy to parse. > > > I suspect that is not easy to build at the moment? > > > > > > > The cover letter says this: > > > > This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program > > which checks that GPU atomic accesses to system memory are atomic. Without > > this series the test fails as there is no way of write-protecting the page > > mapping which results in the device clobbering CPU writes. For reference > > the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/ > > > > Further testing has been performed by adding support for testing exclusive > > access to the hmm-tests kselftests. > > > > ...so that seems to cover the "sample program" request, at least. > > Thanks, I'll take a look > > > > > > I wonder how we co-ordinate all the work the mm is doing, page migration, > > > reclaim with device exclusive access? Do we have any numbers for the worst > > > case page fault latency when something is marked away for exclusive access? > > > > CPU page fault latency is approximately "terrible", if a page is resident on > > the GPU. We have to spin up a DMA engine on the GPU and have it copy the page > > over the PCIe bus, after all. > > > > > I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would > > > > Yes, for now. > > > > > only impact the address space of programs using the GPU. Should the exclusively > > > marked range live in the unreclaimable list and recycled back to active/in-active > > > to account for the fact that > > > > > > 1. It is not reclaimable and reclaim will only hurt via page faults? > > > 2. It ages the page correctly or at-least allows for that possibility when the > > > page is used by the GPU. > > > > I'm not sure that that is *necessarily* something we can conclude. It depends upon > > access patterns of each program. For example, a "reduction" parallel program sends > > over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back > > to the CPU. In that case, freeing the physical page on the CPU is actually the > > best decision for the OS to make (if the OS is sufficiently prescient). > > > > With a shared device or a device exclusive range, it would be good to get the device > usage pattern and update the mm with that knowledge, so that the LRU can be better > maintained. With your comment you seem to suggest that a page used by the GPU might > be a good candidate for reclaim based on the CPU's understanding of the age of > the page should not account for use by the device > (are GPU workloads - access once and discard?) Hmm, besides the aging info, this reminded me: do we need to isolate the page from lru too when marking device exclusive access? Afaict the current patch didn't do that so I think it's reclaimable. If we still have the rmap then we'll get a mmu notify CLEAR when unmapping that special pte, so device driver should be able to drop the ownership. However we dropped the rmap when marking exclusive. Now I don't know whether and how it'll work if page reclaim runs with the page being exclusively owned if without isolating the page.. -- Peter Xu