From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1E7FC433E6 for ; Tue, 9 Feb 2021 13:37:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B944564ED5 for ; Tue, 9 Feb 2021 13:37:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B944564ED5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 322FE6B006C; Tue, 9 Feb 2021 08:37:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F9AB6B006E; Tue, 9 Feb 2021 08:37:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2108B6B0070; Tue, 9 Feb 2021 08:37:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id 0A93F6B006C for ; Tue, 9 Feb 2021 08:37:46 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BCDBBA8F8 for ; Tue, 9 Feb 2021 13:37:45 +0000 (UTC) X-FDA: 77798832090.14.mice67_52128ae27607 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 9DA8E1807F366 for ; Tue, 9 Feb 2021 13:37:45 +0000 (UTC) X-HE-Tag: mice67_52128ae27607 X-Filterd-Recvd-Size: 8348 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 13:37:44 +0000 (UTC) Received: by mail-oi1-f176.google.com with SMTP id h6so19446506oie.5 for ; Tue, 09 Feb 2021 05:37:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=N841iQ9kWSoyGbDxnyS2lfPbTFn9GbQBCWGWYptwBjo=; b=dZnwC0EP2oRvQWDzVYjaBz0mpRcOi213Sz/23TmswG07qWiPNm18GTaVBf5tLMQHnN npPa0SIiwwz5hq4z6EPRBu6MenNeP46VigR7K9W+N0XELD7lgOgModNvxP1ianmO1OUC WgV8qTbjUWfRbzNtOTCmMhaBEd+myqyLjIfv8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=N841iQ9kWSoyGbDxnyS2lfPbTFn9GbQBCWGWYptwBjo=; b=eVSBtBmhPYmwJF51WczOam5KPVPEDxi32v7A1ucABa74izHqPGGmf3L4DLEOWyKKMI iEVx+eTnSsxIiWMhIHao57Ij9Kx45l/H8lGxUUt/PQWuyz2pv5cuxb4vmb7UyksiAzWg mGfkS0Bo+nPa18n6dRG6UEN3ON8el0zuZS6On1fS2JMT0mRVC5NE0TsFUclsvMCPcm1/ bTMiK3gTig/yRqxkwm8HQ2KYOtFOeVvWTf6ZJLCa3BpcraPbn0qlKwU/GN+QrcdmmfmM iDd16lOVsSZZ0xLp/mqPUD+IdejD+hggSeX7Ks8SCCMrppsN2/KRYL6MTBtBrcfaTVvP Z7AA== X-Gm-Message-State: AOAM532xgv4MkQAwkfTj99Wg9/JvkaTd9PM+YTy7O8ALM1VnFekuLYIJ OX3QurW7L1fpLP/1N+zTomn3jJXkUxAU78KOCwCO7A== X-Google-Smtp-Source: ABdhPJyy5MepLQLdvLY0EPRLtIDNjgs2IIyBoFeuNQNErnK1iydPhtA2Lf5mM4hgfdSRq9s19tLXbQJR2E7el+1dVPg= X-Received: by 2002:aca:df42:: with SMTP id w63mr2479956oig.128.1612877863700; Tue, 09 Feb 2021 05:37:43 -0800 (PST) MIME-Version: 1.0 References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> In-Reply-To: <3426910.QXTomnrpqD@nvdebian> From: Daniel Vetter Date: Tue, 9 Feb 2021 14:37:32 +0100 Message-ID: Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau To: Alistair Popple Cc: Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , John Hubbard , Ralph Campbell , Jerome Glisse , Jason Gunthorpe Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 9, 2021 at 1:57 PM Alistair Popple wrote: > > On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: > > > > > > Recent changes to pin_user_pages() prevent the creation of pinned pages in > > > ZONE_MOVABLE. This series allows pinned pages to be created in > ZONE_MOVABLE > > > as attempts to migrate may fail which would be fatal to userspace. > > > > > > In this case migration of the pinned page is unnecessary as the page can > be > > > unpinned at anytime by having the driver revoke atomic permission as it > > > does for the migrate_to_ram() callback. However a method of calling this > > > when memory needs to be moved has yet to be resolved so any discussion is > > > welcome. > > > > Why do we need to pin for gpu atomics? You still have the callback for > > cpu faults, so you > > can move the page as needed, and hence a long-term pin sounds like the > > wrong approach. > > Technically a real long term unmoveable pin isn't required, because as you say > the page can be moved as needed at any time. However I needed some way of > stopping the CPU page from being freed once the userspace mappings for it had > been removed. Obviously I could have just used get_page() but from the > perspective of page migration the result is much the same as a pin - a page > which can't be moved because of the extra refcount. long term pin vs short term page reference aren't fully fleshed out. But the rule more or less is: - short term page reference: _must_ get released in finite time for migration and other things, either because you have a callback, or because it's just for direct I/O, which will complete. This means short term pins will delay migration, but not foul it complete - long term pin: the page cannot be moved, all migration must fail. Also this will have an impact on COW behaviour for fork (but not sure where those patches are, John Hubbard will know). So I think for your use case here you want a) short term page reference to make sure it doesn't disappear plus b) callback to make sure migrate isn't blocked. Breaking ZONE_MOVEABLE with either allowing long term pins or failing migrations because you don't release your short term page reference isn't good. > The normal solution of registering an MMU notifier to unpin the page when it > needs to be moved also doesn't work as the CPU page tables now point to the > device-private page and hence the migration code won't call any invalidate > notifiers for the CPU page. Yeah you need some other callback for migration on the page directly. it's a bit awkward since there is one already for struct address_space, but that's own by the address_space/page cache, not HMM. So I think we need something else, maybe something for each ZONE_DEVICE? > > That would avoid all the hacking around long term pin constraints, because > > for real unmoveable long term pinned memory we really want to have all > > these checks. So I think we might be missing some other callbacks to be > > able to move these pages, instead of abusing longterm pins for lack of > > better tools. > > Yes, I would like to avoid the long term pin constraints as well if possible I > just haven't found a solution yet. Are you suggesting it might be possible to > add a callback in the page migration logic to specially deal with moving these > pages? s/possible/need to type some code to address it/ I think. But also I'm not much of an expert on this, I've only just started learning how this all fits together coming from the gpu side. There's a _lot_ of finesse involved. Cheers, Daniel > > Thanks, Alistair > > > Cheers, Daniel > > > > > > > > > > > > Alistair Popple (9): > > > mm/migrate.c: Always allow device private pages to migrate > > > mm/migrate.c: Allow pfn flags to be passed to migrate_vma_setup() > > > mm/migrate: Add a unmap and pin migration mode > > > Documentation: Add unmap and pin to HMM > > > hmm-tests: Add test for unmap and pin > > > nouveau/dmem: Only map migrating pages > > > nouveau/svm: Refactor nouveau_range_fault > > > nouveau/dmem: Add support for multiple page types > > > nouveau/svm: Implement atomic SVM access > > > > > > Documentation/vm/hmm.rst | 22 +- > > > arch/powerpc/kvm/book3s_hv_uvmem.c | 4 +- > > > drivers/gpu/drm/nouveau/include/nvif/if000c.h | 1 + > > > drivers/gpu/drm/nouveau/nouveau_dmem.c | 190 +++++++++++++++--- > > > drivers/gpu/drm/nouveau/nouveau_dmem.h | 9 + > > > drivers/gpu/drm/nouveau/nouveau_svm.c | 148 +++++++++++--- > > > drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h | 1 + > > > .../drm/nouveau/nvkm/subdev/mmu/vmmgp100.c | 6 + > > > include/linux/migrate.h | 2 + > > > include/linux/migrate_mode.h | 1 + > > > lib/test_hmm.c | 109 ++++++++-- > > > lib/test_hmm_uapi.h | 1 + > > > mm/migrate.c | 82 +++++--- > > > tools/testing/selftests/vm/hmm-tests.c | 49 +++++ > > > 14 files changed, 524 insertions(+), 101 deletions(-) > > > > > > -- > > > 2.20.1 > > > > > > > -- > > Daniel Vetter > > Software Engineer, Intel Corporation > > http://blog.ffwll.ch > > > > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch