From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B5C8C0218A for ; Thu, 30 Jan 2025 10:50:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAD05280069; Thu, 30 Jan 2025 05:50:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E5C83280058; Thu, 30 Jan 2025 05:50:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFBD9280069; Thu, 30 Jan 2025 05:50:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AC161280058 for ; Thu, 30 Jan 2025 05:50:34 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 24051120C0F for ; Thu, 30 Jan 2025 10:50:34 +0000 (UTC) X-FDA: 83063799588.17.056B535 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf14.hostedemail.com (Postfix) with ESMTP id 20E5C10000E for ; Thu, 30 Jan 2025 10:50:31 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b=gkHCCz1y; spf=none (imf14.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.128.46) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738234232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WlIOJ9U3QA9sHWVQXA2SUvaDy201Q+tlYK7jAEa09Tg=; b=I0EqOenONUgU1wY/QPM6rX9qXyL/O85VYEADElC5t0TuY6QIMdIfK1FqKLata/qHjycPAr GJBwWw++hc7eNaI7EUsxnhCE+bjnWNoxdZmGpkkFzjPyWNMxyh6645uzPmOZDLl+Y2RgKR C6KK8Y+maX4p78TYIx6JQzrYTkCzIBs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b=gkHCCz1y; spf=none (imf14.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.128.46) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738234232; a=rsa-sha256; cv=none; b=7SU6cW08h4t7uFJ2T5Uzj1gZy7o5UKUyTB4PZ0ZVAM6MRSsoyccPr6HVnZz8OCKl15Faog v8c/pY2R+hX8Vt7dK1Cf2LWZuoQCwJvRMLQvXiXUx9HhcbGv+77qVLnZVNEgM+vVHYZOBC XjCOqqPwWIRdYHwo1i1QnYwWtKEYocs= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4364a37a1d7so6188485e9.3 for ; Thu, 30 Jan 2025 02:50:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; t=1738234230; x=1738839030; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=WlIOJ9U3QA9sHWVQXA2SUvaDy201Q+tlYK7jAEa09Tg=; b=gkHCCz1yMhtePBzTV3ghPmXY+3948BLH9JPge/6rfVoXXTuYysHjVgm1gmDjilkzPl zk9zo0CGzwwsa+LVvLRIsFwcK+bkl106fVQqblzz5UebWxecJ7Ja/TRuKpXabq1EcN2X 97MXDBHipLwiQKH+J/9R2g8Lb5kKZWNju+eYw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738234230; x=1738839030; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WlIOJ9U3QA9sHWVQXA2SUvaDy201Q+tlYK7jAEa09Tg=; b=ndB0wPhHz61wjgYnodlrYZePjLHsJ7awLXJKJEOY3AblTXorrcwlmHS2Ka5xrp1oVL 6oS/Vn6TpMRTzRw442Eqmr4JHZBKwMDUNkjxBZ2+cB1xX3H8BCjptxaF5QLSw4WWmNK4 HRmt+4PNbO0/YOnc+mlG049f4uNY8kDgo4lzhkxh4hOb5T43FvsFj8gKW+Hn0QoPrX/r LNC7ouagLd2wEHAmTe9XWtIMGdyCqx/xtd0M6OCz9skJ2d8D5VLkbFzAkW8FL/SkP0Ve NMSuwZz97SYrduxSHpk3Cta7yLqT1k0RpUGybvivOEGtabqYQX4CTeZb8leW7pPvnr7z +Jfg== X-Forwarded-Encrypted: i=1; AJvYcCXIo65mQCtmlRQioeBnW8SO3dKLlSwALc2sVWw67Kx5j8vhT1RjQbyDus603/6gIQQZVXAVhvQWwQ==@kvack.org X-Gm-Message-State: AOJu0YyPrAACh/QG47QXihF/2fYHUOJvtpYDYr6vK8GRQYfPjdclkudD cPvr6thK2EiVZEqbmhxv4CVFMPK566Lq57vdN+8lgJaL8KGbKhXOBUD9e/NQyx5FoN2f+wpMk1j 1 X-Gm-Gg: ASbGncuDpotUzADOCIatG806itMOCKYmA77AGkxMWqeEHnYgxDBS7deeCqO982YpmQ/ Prf1o7tATtIMIF1xWxfOrQWxQDrdHBScFxoM4UxYoetFpvRevzTq94WgAkIKTqQ6OGY6Ek9Bq2H UkrrtWdcYSIkEHx3busEVSOp0rFtUwSKBRjUnjehFiDns4SmS+3pBwwO5gJvpQEmMGKcCNcDgsi KgiC8AsglGaMKLHN1phO86DY/YsJf28KSlIIDZgHuISTe9XHVZmiAgnqNtD8KaCwNKkaR6QUOry vTXtzvQsXawOXMByACYXl701NdM= X-Google-Smtp-Source: AGHT+IHMw0cYgOTvVvK6HIRKMLy20cgFzE3d4vlaCtti/GNUtg6PxQxKz8d7r2TT9CWFYzLA0t+WOA== X-Received: by 2002:a05:600c:4e0c:b0:434:a386:6cf with SMTP id 5b1f17b1804b1-438dc3a8faemr61859005e9.2.1738234230580; Thu, 30 Jan 2025 02:50:30 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:5485:d4b2:c087:b497]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438dcc81d74sm54599715e9.37.2025.01.30.02.50.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jan 2025 02:50:29 -0800 (PST) Date: Thu, 30 Jan 2025 11:50:27 +0100 From: Simona Vetter To: Jason Gunthorpe Cc: Thomas =?iso-8859-1?Q?Hellstr=F6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Message-ID: Mail-Followup-To: Jason Gunthorpe , Thomas =?iso-8859-1?Q?Hellstr=F6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org References: <20241201103659.420677-1-ymaman@nvidia.com> <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> <20250128151610.GC1524382@ziepe.ca> <20250128172123.GD1524382@ziepe.ca> <20250129134757.GA2120662@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250129134757.GA2120662@ziepe.ca> X-Operating-System: Linux phenom 6.12.11-amd64 X-Rspamd-Queue-Id: 20E5C10000E X-Stat-Signature: 3j3hkanda1c5cummg8ejfy1zaskcomg9 X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738234231-294069 X-HE-Meta: U2FsdGVkX18ZsrLRZbOHlhQrZWE6L/7Tnz2lx4VPAsXSC3/DkhnhKIpyFFmHl8F8larm+OyqP6FBcJsN7U/PogJVAHqIF+znYywGoEifeyVhYXVIzPMX24FUXWzf9LADl9BQuxLcv6SRUYmcFPhXvPLK1DqZdP2Amh5SCSc7gUb/v+q0TSip5SH8NNOE4IyyXvbaHT1GTG6ZsAf2gay1eM3zNK9OLk5BMNA/+gIfY5gJHagaaqp5Hp2l9i7Yoif1Us41hLkQZY+dNRwJgdqwhIlrZd6Ok9zhEMLB6Nf088287XwGjScSgbstJEd4svY4BrzHZQBDEHwqKQIKc09okvoGwbFYyFsw5H7uFCuXuhHUPFKHOZ45pVt7B4LlMakLzk/h9j+jK4INAV/VfzR0fxo2zXH5SGLHOVZ93Icbp228Gu0+BAIgcm38muISoEel5fxGjkBZckk5HuP22hHAhpkUFb1gA+CKsbmUVpd8xR9GZzAOC2QmC8S04NClwqDU/buLv77wkt7L/FPcgz3GWp0TqLOPJ+HBdpQNP7pa6T9wVhTUJE+c2yLsebNVNvho0V06NtZX3wl3Owwnl6Y3wI7SAZhcu207miwb8f52clTs8ljR9Qa9dZ4bDidExd4mWO4sf/9n44wAV/DuEFUUzAH0nPIsteZhvX+6P6+Pf+9SL1j1dGM8r88wde1ts6aPg4wYeNr52Pgyz/mbbWrDqx3S0imSGABR9e0fxj2oQ4g2S6trVkLwCAnCPnn5SRHV8bAnTOCuDjmZWec7+DtQ1+NmxE7RtBy/aVRNYAiNdI0itGRH629TUbyi6KF+HWY8avL9d8vgncbrEkxs0PqSSYjNEzbu/OnJIPJfFUEn8yl/75vJzYqfaxsZraikbmypMfuYNGwIlJJNSrdC3EFiyL1hyp0GNkKh/xtTnzsGMTtmqjS95H1omxuouNiEp8OkVr4N4bUf7H02ZU72Kvr TaYiKn4x 7zTQCcYxK4DmgLGNNmw0cGZP3mvGQ0ypy1m5dwqErH9c4x6OgwUjG2io0zqwK2So790Byi6n5qjmsV+mfdG7Mxd7rebHCZgy2SwIfkjmyvHrK9iKJavZ8HymBXWqm4IsCdrkjso5rreUUuwbn81fck7+c3A+Dno3sV5pJ1APzzW+u3fNMPAhIGebUYbuFmx6ZrqtNCXQIcetd5G8PZMwfldh5+hpRhbp9e7Yc8fb+riS1Rrvb0YAYFmNwChLfVpKqwAWLG66TgZagNQjR/FXeSU+FLXCk0o/hZ1frwxqSSEK4Nho= X-Bogosity: Ham, tests=bogofilter, spamicity=0.155328, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 29, 2025 at 09:47:57AM -0400, Jason Gunthorpe wrote: > On Wed, Jan 29, 2025 at 02:38:58PM +0100, Simona Vetter wrote: > > > > The pgmap->owner doesn't *have* to fixed, certainly during early boot before > > > you hand out any page references it can be changed. I wouldn't be > > > surprised if this is useful to some requirements to build up the > > > private interconnect topology? > > > > The trouble I'm seeing is device probe and the fundemantal issue that you > > never know when you're done. And so if we entirely rely on pgmap->owner to > > figure out the driver private interconnect topology, that's going to be > > messy. That's why I'm also leaning towards both comparing owners and > > having an additional check whether the interconnect is actually there or > > not yet. > > Hoenstely, I'd rather invest more effort into being able to update > owner for those special corner cases than to slow down the fast path > in hmm_range_fault.. I'm not sure how you want to make the owner mutable. The only design that I think is solid is to evict all device private memory, unregister the dev_pagemap and register a new one with the updated owner. I think any other approach boils down to the same issue, except we pretend it's easier and just ignore all the race conditions. And I've looked at the lifetime fun of unregistering a dev_pagemap for device hotunplug and pretty firmly concluded it's unfixable and that I should run away to do something else :-P An optional callback is a lot less scary to me here (or redoing hmm_range_fault or whacking the migration helpers a few times) looks a lot less scary than making pgmap->owner mutable in some fashion. Cheers, Sima > The notion is that owner should represent a contiguous region of > connectivity. IMHO you can always create this, but I suppose there > could be corner cases where you need to split/merge owners. > > But again, this isn't set in stone, if someone has a better way to > match the private interconnects without going to driver callbacks then > try that too. > > I think driver callbacks inside hmm_range_fault should be the last resort.. > > > You can fake that by doing these checks after hmm_range_fault returned, > > and if you get a bunch of unsuitable pages, toss it back to > > hmm_range_fault asking for an unconditional migration to system memory for > > those. But that's kinda not great and I think goes at least against the > > spirit of how you want to handle pci p2p in step 2 below? > > Right, hmm_range_fault should return pages that can be used and you > should not call it twice. > > Jason -- Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch