From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B14FAC2D0A3 for ; Fri, 6 Nov 2020 19:53:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F1F2B2072E for ; Fri, 6 Nov 2020 19:53:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="FudZRGzA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1F2B2072E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 58BBA6B0036; Fri, 6 Nov 2020 14:53:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53C2E6B005C; Fri, 6 Nov 2020 14:53:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4522C6B005D; Fri, 6 Nov 2020 14:53:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 189BD6B0036 for ; Fri, 6 Nov 2020 14:53:44 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AE6E3181AC550 for ; Fri, 6 Nov 2020 19:53:43 +0000 (UTC) X-FDA: 77455043526.07.rail62_3b03af1272d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 8DC521803F9A8 for ; Fri, 6 Nov 2020 19:53:43 +0000 (UTC) X-HE-Tag: rail62_3b03af1272d5 X-Filterd-Recvd-Size: 6278 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 6 Nov 2020 19:53:43 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id v11so1245961qtq.12 for ; Fri, 06 Nov 2020 11:53:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=sU45ASLE3qTNEeDnzXy0mWmjw/B5hdl1YB3C1dRGzkI=; b=FudZRGzAxEFk9lGKJUSPzsyZKArhHRwyzmRlvf+GFC+ATQPjzcBbVOeDLLB7moiVUf ObxxpnS/zZJpTm82iWz+fyi9jGs/I9YeNae0ROJk7xm4ksGu3vErOwXPC3998pjQlR5p 4czYVsKEbRHoddebJV0noju3HQZREanU58l+vhPFj17Ta8rGMwytFISX8XsKNrHR+BX8 lRJNDwJeHT2t31rpqbAdGGp0+akczI7HoAc6Ksjc1BT3ksZah6pY+y3OqCayYTmpzL+D Vcc6mEGa7A07b9rAWqQo7QjWrLXkk8WsSTWPLKnX0XjNAe7EuqN+sDQBd5I1lutR9Gx3 kmvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=sU45ASLE3qTNEeDnzXy0mWmjw/B5hdl1YB3C1dRGzkI=; b=C8axoxIyoOk973P7zeMvnJkripRTh58r9wJCMLiRZ816RnSe1v93w/n4+a3MNh04jj htzSV28pvrWWVtRvDCtDRyzeSxyqqEN8ikpbQi0s1MqW+0ZD2kdOJafSM7fb4A2R2egd +ioxztbNihcwYzhS5iw/X5TBwacblF7ENHX/RBdFPIZBIle0mQ4L+GbnO4PTs8NU7CUt 3jMrszPtIrOFgEX3prAqqUoSY07pqKzfEHbEr0QEiPvSH+XSJIA7s+eVLiUFiAAMdX+r xZyew5gzUpkMaNOmZVKW48D+81FsyMTP9UNVu/O64vSfl4eDwo/mvlnsd/6/HiGeLmIU ww6A== X-Gm-Message-State: AOAM533oGkBo6TZmFTik3uXX60kFoc7XAe+K/rhaEBkj7jGiidbhSl+C yRwkcDEQ5kLCqvcLaKAuuBmrfQ== X-Google-Smtp-Source: ABdhPJwcMytaU5RLKIBqp2abyuS1iVFiLHOqHqGF/QD5vN1zrtKnu5OM6Laul9BgJOP5lCmGBjWjKA== X-Received: by 2002:aed:32c7:: with SMTP id z65mr3146052qtd.266.1604692422443; Fri, 06 Nov 2020 11:53:42 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id r41sm1239041qtj.23.2020.11.06.11.53.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 11:53:41 -0800 (PST) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kb7np-0011j7-5m; Fri, 06 Nov 2020 15:53:41 -0400 Date: Fri, 6 Nov 2020 15:53:41 -0400 From: Jason Gunthorpe To: Logan Gunthorpe Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, Stephen Bates , Christoph Hellwig , Dan Williams , Christian =?utf-8?B?S8O2bmln?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter Subject: Re: [RFC PATCH 14/15] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Message-ID: <20201106195341.GA244516@ziepe.ca> References: <20201106170036.18713-1-logang@deltatee.com> <20201106170036.18713-15-logang@deltatee.com> <20201106172206.GS36674@ziepe.ca> <20201106174223.GU36674@ziepe.ca> <2c2d2815-165e-2ef9-60d6-3ace7ff3aaa5@deltatee.com> <20201106180922.GV36674@ziepe.ca> <09885400-36f8-bc1d-27f0-a8adcf6104d4@deltatee.com> <20201106193024.GW36674@ziepe.ca> <03032637-0826-da76-aec2-121902b1c166@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <03032637-0826-da76-aec2-121902b1c166@deltatee.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Nov 06, 2020 at 12:44:59PM -0700, Logan Gunthorpe wrote: > > > On 2020-11-06 12:30 p.m., Jason Gunthorpe wrote: > >> I certainly can't make decisions for code that isn't currently > >> upstream. > > > > The rdma drivers are all upstream, what are you thinking about? > > Really? I feel like you should know what I mean here... > > I mean upstream code that actually uses the APIs that I'd have to > introduce. I can't say here's an API feature that no code uses but the > already upstream rdma driver might use eventually. It's fairly easy to > send patches that make the necessary changes when someone adds a use of > those changes inside the rdma code. Sure, but that doesn't mean you have to actively design things to be unusable beyond this narrow case. The RDMA drivers are all there, all upstream, if this work is accepted then the changes to insert P2P pages into their existing mmap flows is a reasonable usecase to consider at this point when building core code APIs. You shouldn't add dead code, but at least have a design in mind for what it needs to look like and some allowance. > >> Ultimately, if you aren't using the genpool you will have to implement > >> your own mmap operation that somehow allocates the pages and your own > >> page_free hook. > > > > Sure, the mlx5 driver already has a specialized alloctor for it's BAR > > pages. > > So it *might* make sense to carve out a common helper to setup a VMA for > P2PDMA to do the vm_flags check and set VM_MIXEDMAP... but besides that, > there's no code that would be common to the two cases. I think the whole insertion of P2PDMA pages into a VMA should be similar to io_remap_pfn() so all the VM flags, pgprot_decrypted and other subtle details are all in one place. (also it needs a pgprot_decrypted before doing vmf_insert, I just learned that this month) > >> I also don't expect this to be going upstream in the near term so don't > >> get too excited about using it. > > > > I don't know, it is actually not that horrible, the GUP and IOMMU > > related changes are simpler than I expected > > I think the deal breaker is the SGL hack and the fact that there are > important IOMMU implementations that won't have support. Yes, that is pretty hacky, maybe someone will have a better idea Jason