From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4063BC02193 for ; Wed, 29 Jan 2025 13:48:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE323280063; Wed, 29 Jan 2025 08:48:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A698728005E; Wed, 29 Jan 2025 08:48:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BE31280063; Wed, 29 Jan 2025 08:48:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6547228005E for ; Wed, 29 Jan 2025 08:48:02 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BE862160DA0 for ; Wed, 29 Jan 2025 13:48:01 +0000 (UTC) X-FDA: 83060617962.03.3958E36 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) by imf07.hostedemail.com (Postfix) with ESMTP id CADCF40006 for ; Wed, 29 Jan 2025 13:47:59 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=AWHZXi6w; dmarc=none; spf=pass (imf07.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.172 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738158480; a=rsa-sha256; cv=none; b=SJrQqstnr3uQJxPwJZRxXswHMDSm7OSD3xPY50V19Y109DiIlsp849FJ67DrAII6k3tr4e jpwwEhx6UNJ7mWIjzL+l49GiiVMB9whJMwcnqAgGeUVG1OyEsrpFc6IFbvZCRYUSluymq0 707+f+N6DY43Y+ISpsXrIOVt5Nwqy2k= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=AWHZXi6w; dmarc=none; spf=pass (imf07.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.172 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738158479; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TGxs1D5kHZWfV3TMw8ydKze6LYDZn037JbgBy8rBgVo=; b=nw0YiLCTEQ9/lHhgul5tU1hYD1bGaRPvpgdo35nNVzSdJBnv4ACvV58wQAIF22hePUL/k3 mrsO+zrb8NtAD2os/hOx9n/zBG9zA7burWdhO2jPWEOr1rPmMH5G+HzrDvx0zu/YrCpyyl xOv8MT6x8Ksdn3Ro11+x/KOsA/RXHhE= Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-4679eacf25cso40051091cf.3 for ; Wed, 29 Jan 2025 05:47:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1738158479; x=1738763279; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=TGxs1D5kHZWfV3TMw8ydKze6LYDZn037JbgBy8rBgVo=; b=AWHZXi6wRD/WksBRfIAFjldj4VA4xrTBBSaU8H/c3+USPFskHHimocOzIBZx9Dnrt5 3C/oZidegcmi7JYYd+dgafl/n4In3U0pH+UNumc3KFLopcuqe86VgXcjnYqR2idsn8Mt wrkM0XIxwzolP3K7C/pJbW4Lo2o9watC8A/ncmF4+npkkHajswJFNvkN08Wn7gKkQZBy /ylC//SrkuvNy/ncoHjyhYlN5Se0ql2hJU3B3PJGL0/oiYGtCtXPlZlIBfpgJOote1mu EgCbrE8/BAy2MJ7FBbYJBRwwSaV2xUlloi2hB28leV3L+gbiN3Gl7LJbO8IomvfX4qbo KzDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738158479; x=1738763279; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=TGxs1D5kHZWfV3TMw8ydKze6LYDZn037JbgBy8rBgVo=; b=m7HXty2MwH2+Bsbn40e9mJxLW6su7c+lHuDCOelwZjeyXKn0oBQqXM9iX8BpoQc/Pr Y/TOYzLM1Fj42R+6Ku5O1SwsC8RIhbDg/oCu8JiP3G8pGpqb3/oMHuEfdY4h1q6zX2IH hmHahFHfxK0K0qu0dptpnZe9MyJ9tDXQE4D5zi3Wl4as1br9PYYu7hiE4pcWoVmLLoHh 9gdN/rZ1ZzJka/dbpYz93kycdY/NBjq8rSAnrfAZdNQEIc11Alx6iqBcx8fn+qVSp8ee 9tUOdK3WHAiBiVTcHXIUi0wWwFoH0LPQn2GmXv0z+/tcHuGds2i0amwEojMkUp0qxYeC HgTw== X-Forwarded-Encrypted: i=1; AJvYcCXJzKdvrGb9R5FhZ6FeVvokFgra3ZDdVzr75alzNUsQVILOmYYbufppgn8qAMTFVM3/Xk8yA4otMw==@kvack.org X-Gm-Message-State: AOJu0Yw5zf1uqIxuGS9+pW4tDDYth42G7OQFxZHe9jc2ruMf+O7HpAMZ /oBSQyYxH5h8irNP9MBSq0nvd8izcRAnzP+TZkc+TFy0B4DDa6BAZYf4KG1NMhE= X-Gm-Gg: ASbGnctsyTmNsNBwAem/ciYdb1uG0x64GbHYTjajOpkjf76+/r/AHtebIAoTr0GTrW4 k9k6SzTdF7dBjt2XGIm0b+7A6u9cx7YwSLESSVl1JPfyyxlsse+o9Sbo38/pJkjVFOWxd3i5vFY 25KYoJ96gklJLOWCK1hAoNOZe4FEuy/WNNMsxYwOUE0bH5yvGmm6CWo4MY13Sq2Ra7vQNS8mJ9O DnwiMmlZ5h4MpWqpWoxzgT40di6SRPtAd/jbZlO4amFOEV+Kh3pxYbZ3F0ydCqd9HifZGK0RIi4 c56ncHeLVI/OKc9cDNMEIbOL4YWVi/3mixosO+kx8g6yjhI4b1+ROrlIpcJjQ0Xr X-Google-Smtp-Source: AGHT+IHuYCAk+wYHqjFRDvmI1uuhKYfRXYz+H9jof+ajiAjSs3miFEirWAObJrrIBPKksD6vvP2lVg== X-Received: by 2002:ac8:5a08:0:b0:46c:782f:5f85 with SMTP id d75a77b69052e-46fd0b98596mr53288201cf.52.1738158478735; Wed, 29 Jan 2025 05:47:58 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-46e66b67680sm62281881cf.60.2025.01.29.05.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jan 2025 05:47:58 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1td8Q9-00000009BlF-1by8; Wed, 29 Jan 2025 09:47:57 -0400 Date: Wed, 29 Jan 2025 09:47:57 -0400 From: Jason Gunthorpe To: Thomas =?utf-8?Q?Hellstr=C3=B6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Message-ID: <20250129134757.GA2120662@ziepe.ca> References: <20241201103659.420677-1-ymaman@nvidia.com> <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> <20250128151610.GC1524382@ziepe.ca> <20250128172123.GD1524382@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: CADCF40006 X-Stat-Signature: zuer6d9rxjayhngks1a5qkjem7nd5mys X-HE-Tag: 1738158479-691399 X-HE-Meta: U2FsdGVkX1/hI+cMGRihtlrIiqJyNzWTeGB+iGAJr0qp97zq2D0LrnDKG1r6aM+P8BnF5hEtL0io9OayHkhWzQrDdBZ50Ofrx/Xf7O8ipnlmG3O0PtMXoybri52ABM0QnNdqzqPPuMFvtFPEp2sNFqB1q6FQVtJYi3/i66SXIanmd5CwybK2f0OHRlve2HrH+GJ0BP0syAeh5z6GJNLBplDhHTj9P5k7iqJhenyS39dnNjCctZeGBeVjjLCCSVA6kpWp5uONQMV0A1zezZD2+9LunTpqrboM5URkVSWTN91IMbNlzxym7ZXWSWOW8f/0R2sFYf7/0mQv3jmSCjrMLTZy9dvdBEAom5aESge6ljj2Piw+REjvV/2osexvqoSuNkPgZR4eAvOFlHXCxjPZis2TFyrcZiXEeAlXGnBxKhgShiQWNIaonbkzZY7ysCfgQOPJIyxhpUdArFU41FbemqcGplEhZlSRVSLE+gR5zkJfH34CdMg9Spgy0ZUAL1fa9hWqcyvJKXxeyN+c9NNDs/7r7D7J4LcX0FrpJ+Q/EwGfI6E5JLrDx3qgcifCYgSsuCXBDDvzd5jZOrn1LQEW9AzjY4tXrI2ohux5WOf+Hn8ovvtSjUKGWnRVMR27XcomGRO/mp3WI5EBwEhZnFKDMQl6lRW4bc6pszYpZwmxQ0jYY1Dvi91dvNY6WviBPxgfDd9TjXANe/6H0iU/vvJd5KC/j8+b5m8n5XHLUZ99czobDbSbrRNHhWzfC6uKcObittrTJAbxt7C9LKEr+wKCuaM1d8M8C0zA4VO7Ufw5042awtyEB65ym0IjA/knuM1zV9uT+ozCm1x7gbSVsyFojgH+oyicWwkKelcqYzlh/8ueeoHdsskwUfh6gsFcLEp/1UaJSkzZc54ZyPF/iy/AeFu65qotwIcDII1lTpqUXdye73cPXO4PeNQKwNXjQoauXlK6oRzv2W07r6t6Q94 tb4vLJYR xoieBvX5K31Wf478s1XGuqu9GALYtR3PWvorKDfHFlIplYZWswVtrzvXDrUW9IEDLLx2jYiY2x06PuMk/0JNcV5UH8XWvApGw7gAH83dz8CREoI95RhvBXtWbP4PsS25DXkMDvMPEzx9xC3UenfTwHe2lITiZhxyv7vyyYAeNjcvjC5SFcs8p49HXPV9PMsaIsmeaKFoD8hi3Q0n92ANsw/XR8iprR9tWXEUGmXYMyBPU05fCxJlDIVgOkn15oh3zdW815vvpWkVnAQPA0HcVCppViyWw1FUqK6J8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001065, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 29, 2025 at 02:38:58PM +0100, Simona Vetter wrote: > > The pgmap->owner doesn't *have* to fixed, certainly during early boot before > > you hand out any page references it can be changed. I wouldn't be > > surprised if this is useful to some requirements to build up the > > private interconnect topology? > > The trouble I'm seeing is device probe and the fundemantal issue that you > never know when you're done. And so if we entirely rely on pgmap->owner to > figure out the driver private interconnect topology, that's going to be > messy. That's why I'm also leaning towards both comparing owners and > having an additional check whether the interconnect is actually there or > not yet. Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in hmm_range_fault.. The notion is that owner should represent a contiguous region of connectivity. IMHO you can always create this, but I suppose there could be corner cases where you need to split/merge owners. But again, this isn't set in stone, if someone has a better way to match the private interconnects without going to driver callbacks then try that too. I think driver callbacks inside hmm_range_fault should be the last resort.. > You can fake that by doing these checks after hmm_range_fault returned, > and if you get a bunch of unsuitable pages, toss it back to > hmm_range_fault asking for an unconditional migration to system memory for > those. But that's kinda not great and I think goes at least against the > spirit of how you want to handle pci p2p in step 2 below? Right, hmm_range_fault should return pages that can be used and you should not call it twice. Jason