From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4A93C4360C for ; Wed, 16 Oct 2019 08:58:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5904020872 for ; Wed, 16 Oct 2019 08:58:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vI5EgckJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5904020872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 06DE88E0006; Wed, 16 Oct 2019 04:58:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 045E48E0001; Wed, 16 Oct 2019 04:58:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4F338E0006; Wed, 16 Oct 2019 04:58:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id C5A2E8E0001 for ; Wed, 16 Oct 2019 04:58:06 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 21DE8824556B for ; Wed, 16 Oct 2019 08:58:06 +0000 (UTC) X-FDA: 76049045772.28.bait07_298ffe58e6523 X-HE-Tag: bait07_298ffe58e6523 X-Filterd-Recvd-Size: 8599 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Wed, 16 Oct 2019 08:58:05 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id o28so4429407wro.7 for ; Wed, 16 Oct 2019 01:58:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=reply-to:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=V+4SMbJY+xnY/02Qu1SPKip6GAEd3pyslcFpF7IxHBM=; b=vI5EgckJStxRO73bc+3CwPgKxclRbKnLXPsOtNhcrVURZW63+9T1e2AR+xD4ChmYjb F09Az3ubghdrV9WGNALSeqQTPTWu3DRy4fj4jt/7XpCfcYOXb97srJ+8PG01E8O4sYxe alaKfq+3azM37/Z9sab8NfOx5RuG3cz3D7bOj0LoGEmV03UEbCQ2M8o4eH+iNDAL65kT NUkh3nPZoaRQ4VtHUDdLuoZtEpF2ba26YAvw6u7cdrsWGafqXs3O/NxmX5gnSy0+Nkjz xUjXcoBkOZ9KEXo6In0z6fpRL/TMFSnEc3h3Phoj68Pum7UJui5fB1f08XIfVZboPN+1 5OYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=V+4SMbJY+xnY/02Qu1SPKip6GAEd3pyslcFpF7IxHBM=; b=kYFgnw6nQY700gdnph86Vmv4q4uhAPpBsm5gDJhtAuwwoF/CskbpqqO3tAF7ijXIT9 r71jSHZYZJyINYO4FvLy7H7sRlg+XDyJJXH0/1QhZhatao7OGuZYlu8BuC6itrjdzGlF SeCDfFsVrcN0w7hGM5ULsXhnbY+HVWaKQbCRJnSTercge2ZV2RK3PuMasm2b+xahN5fg 0iEXzhxcyH3nYYrqnwghFHjfpqnD8ccL2cZYmNQl9X+UE/2tjzLM85Xmdoz64b9mutc7 o0aJl17st98yMjORJAJNuYWDD41ra2+BSKw/mD0hMpDHSiec53++jjyI9ZHUT4dUGObt MfIQ== X-Gm-Message-State: APjAAAUzemgwd8Mk0NgXrUN7J4bSLN+F8N6/OnRtjudnQj0c5QGRRVBv yK0HP6IvqatSGnOX2c2axF1n8FBv X-Google-Smtp-Source: APXvYqywHpqS2WwakM+BUQmjn7/0SQxlW6RiaETqeNUjKm43xk/4p1AdnGlHpqoSz8YMsyfharKZLg== X-Received: by 2002:adf:fa92:: with SMTP id h18mr1684319wrr.220.1571216284378; Wed, 16 Oct 2019 01:58:04 -0700 (PDT) Received: from ?IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7? ([2a02:908:1252:fb60:be8a:bd56:1f94:86e7]) by smtp.gmail.com with ESMTPSA id b5sm1610909wmj.18.2019.10.16.01.58.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Oct 2019 01:58:03 -0700 (PDT) Reply-To: christian.koenig@amd.com Subject: Re: [PATCH hmm 00/15] Consolidate the mmu notifier interval_tree and locking To: Jason Gunthorpe , Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: Andrea Arcangeli , linux-rdma@vger.kernel.org, amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Ben Skeggs References: <20191015181242.8343-1-jgg@ziepe.ca> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Wed, 16 Oct 2019 10:58:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20191015181242.8343-1-jgg@ziepe.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Am 15.10.19 um 20:12 schrieb Jason Gunthorpe: > From: Jason Gunthorpe > > 8 of the mmu_notifier using drivers (i915_gem, radeon_mn, umem_odp, hfi1, > scif_dma, vhost, gntdev, hmm) drivers are using a common pattern where > they only use invalidate_range_start/end and immediately check the > invalidating range against some driver data structure to tell if the > driver is interested. Half of them use an interval_tree, the others are > simple linear search lists. > > Of the ones I checked they largely seem to have various kinds of races, > bugs and poor implementation. This is a result of the complexity in how > the notifier interacts with get_user_pages(). It is extremely difficult to > use it correctly. > > Consolidate all of this code together into the core mmu_notifier and > provide a locking scheme similar to hmm_mirror that allows the user to > safely use get_user_pages() and reliably know if the page list still > matches the mm. That sounds really good, but could you outline for a moment how that is archived? Please keep in mind that the page reference get_user_pages() grabs is *NOT* sufficient to guarantee coherency here. Regards, Christian. > > This new arrangment plays nicely with the !blockable mode for > OOM. Scanning the interval tree is done such that the intersection test > will always succeed, and since there is no invalidate_range_end exposed to > drivers the scheme safely allows multiple drivers to be subscribed. > > Four places are converted as an example of how the new API is used. > Four are left for future patches: > - i915_gem has complex locking around destruction of a registration, > needs more study > - hfi1 (2nd user) needs access to the rbtree > - scif_dma has a complicated logic flow > - vhost's mmu notifiers are already being rewritten > > This is still being tested, but I figured to send it to start getting help > from the xen, amd and hfi drivers which I cannot test here. > > It would be intended for the hmm tree. > > Jason Gunthorpe (15): > mm/mmu_notifier: define the header pre-processor parts even if > disabled > mm/mmu_notifier: add an interval tree notifier > mm/hmm: allow hmm_range to be used with a mmu_range_notifier or > hmm_mirror > mm/hmm: define the pre-processor related parts of hmm.h even if > disabled > RDMA/odp: Use mmu_range_notifier_insert() > RDMA/hfi1: Use mmu_range_notifier_inset for user_exp_rcv > drm/radeon: use mmu_range_notifier_insert > xen/gntdev: Use select for DMA_SHARED_BUFFER > xen/gntdev: use mmu_range_notifier_insert > nouveau: use mmu_notifier directly for invalidate_range_start > nouveau: use mmu_range_notifier instead of hmm_mirror > drm/amdgpu: Call find_vma under mmap_sem > drm/amdgpu: Use mmu_range_insert instead of hmm_mirror > drm/amdgpu: Use mmu_range_notifier instead of hmm_mirror > mm/hmm: remove hmm_mirror and related > > Documentation/vm/hmm.rst | 105 +--- > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + > .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 9 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + > drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 445 ++------------ > drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 53 -- > drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 111 ++-- > drivers/gpu/drm/nouveau/nouveau_svm.c | 229 +++++--- > drivers/gpu/drm/radeon/radeon.h | 9 +- > drivers/gpu/drm/radeon/radeon_mn.c | 218 ++----- > drivers/infiniband/core/device.c | 1 - > drivers/infiniband/core/umem_odp.c | 288 +--------- > drivers/infiniband/hw/hfi1/file_ops.c | 2 +- > drivers/infiniband/hw/hfi1/hfi.h | 2 +- > drivers/infiniband/hw/hfi1/user_exp_rcv.c | 144 ++--- > drivers/infiniband/hw/hfi1/user_exp_rcv.h | 3 +- > drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- > drivers/infiniband/hw/mlx5/mr.c | 3 +- > drivers/infiniband/hw/mlx5/odp.c | 48 +- > drivers/xen/Kconfig | 3 +- > drivers/xen/gntdev-common.h | 8 +- > drivers/xen/gntdev.c | 179 ++---- > include/linux/hmm.h | 195 +------ > include/linux/mmu_notifier.h | 124 +++- > include/rdma/ib_umem_odp.h | 65 +-- > include/rdma/ib_verbs.h | 2 - > kernel/fork.c | 1 - > mm/Kconfig | 2 +- > mm/hmm.c | 275 +-------- > mm/mmu_notifier.c | 542 +++++++++++++++++- > 32 files changed, 1180 insertions(+), 1923 deletions(-) >