From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10FB4C02194 for ; Thu, 30 Jan 2025 13:23:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8732C280295; Thu, 30 Jan 2025 08:23:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FC00280293; Thu, 30 Jan 2025 08:23:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6753F280295; Thu, 30 Jan 2025 08:23:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4B707280293 for ; Thu, 30 Jan 2025 08:23:22 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CA8EDA0ED8 for ; Thu, 30 Jan 2025 13:23:21 +0000 (UTC) X-FDA: 83064184602.06.4B7BECE Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf17.hostedemail.com (Postfix) with ESMTP id CD5CA40012 for ; Thu, 30 Jan 2025 13:23:19 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=BWW4tk3y; spf=pass (imf17.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.48 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738243399; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z4xPSZzw/OoNchPOgrFnhgjUTxZA4+QRO6/LNv63tO8=; b=o6Q/DjFDc9PveMgc0iehvR0UMA/6ZC+8CV7vXbTgNaGtkWLBKWyCQTzihJWeQRpc4I/JeV nveXGuxN1jrVxqbZ6C+LU6dRna+BO9XYXktDThiTNL8eVfy9riI649qLYSQUvyf+YQQ1fO F2WbIfB6kYsgOz0kTkTKOF7RCU0hYtc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=BWW4tk3y; spf=pass (imf17.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.48 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738243399; a=rsa-sha256; cv=none; b=OVclHCO7lvxG/YLJyB3Uy/bNbtOe5kOFK1RmdXhbVTM8OkDjOZe4B2YzVj3MMygBH3B47s DIXOVnangMmcXlzxyNQOn3Z8qK43U3cDv27PrA0WEydDht+pLnNfB11vrSD2TmQ7BUNtuZ /H5TI1iA/qASs38R56G1abnq/LuUFno= Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-6e1a41935c3so10323716d6.3 for ; Thu, 30 Jan 2025 05:23:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1738243399; x=1738848199; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Z4xPSZzw/OoNchPOgrFnhgjUTxZA4+QRO6/LNv63tO8=; b=BWW4tk3y3bse4um6vZAQ4tT4CjELM2JZ9TwaXw+iZveTuQX71hkWqH9v90KqUsdED6 +QoupUj6xuRAZBhymdj+WVL6D9TCo1px/cCCgTpphI8F83yMimvRZMPN4WS1V99KsfIR fxpLLOKJYrNNDe/UtS9m/luGcrlfnVW/np4rClMz2acSbmK+TerFja3EnPOQqvRvNLd3 fmbwI20i3XaSLCJ6V7v/JLNB4DFRnTAe2WBkVj4IIsnqxKei7I1kNiVQET4YlvCfSRIO KegwOUigFg0nDAVnwV8xYZbDhJC651/zhSHlaZHWaogKUnrK2FSRh03JcwfQPiI2z25D IxoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738243399; x=1738848199; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Z4xPSZzw/OoNchPOgrFnhgjUTxZA4+QRO6/LNv63tO8=; b=qW5Qn9r0I9Qjk//v2jPYPCmyoq3fdf8gWGbSHd7F9cfXgxIWRopNpsFzMGaAnTcKRO Lce4TPTC1xQ0vlZp6iIY/Io8/VX8VF7OArc6kSMABJLLfTYbmWFyoIsmLaMzQeP2JKFb LyHmp1RSDYBSDnW7UnT3R7DN2vxaebcCxc6LXMorV2f0o8UmF5jb52QfUJd24Znka/4+ 7Zms6l/UFluletP87FmPEyaRRGS7ETqtfbe001dygzIXL8XIejaWxZJIJUkcg99viyMh daZCgdat5jZAt196HULNLZW/1KInrO7IVg53OnHHJfGcWk0RAD+5CSvWUHzX0oJB+wbp FFXQ== X-Forwarded-Encrypted: i=1; AJvYcCXwEMsArL1RLaQFkns+silrY0u5zPMI8IHBUFKrKnjbD5Noyxvl23bVHcRN+IwXMhjdNAGvEmkhqw==@kvack.org X-Gm-Message-State: AOJu0YyjYvMr4cRIT4zL09xlpLs8ICVkU1ep9U5Le2COIUBXvw6aizn5 NWGagvy0St4Bqi2RLmP7LqrZFOd6A4aHiRzHesSIRhlDnbSWBW6dNmq9edHgO8g= X-Gm-Gg: ASbGnct75N5orIZXbfNWhgLfnVEyxV9gEYuTkgB0L6htR+kxhyC6OFV1VtitGSIKniP scf8pyegbTub6iRtWqrRrZTjhBUZMgUUCtJGaFsfP/381Tn+oZ8WjRWqZJ18xgVEiMQIiQcVtGa J/62Ha4cNzEtgTDb3kiftYSzpvdhBpbipeLXHxyuv++zrpRBnW8B4xHtyWtdcwfizJjT2XQwSEt ylAoVEd1TqYRi0TLqVTTbI8QmU+mnzgyEj1iFj3BKeVPrAJkplbPM4mh8yWsSTNWqnJN4fGdDl6 4sjhoebYjO7XAN/AgV3fofjv7OxNuhCUaefwIxiKp21jKDOv+RUjqPY4GQ3leaHm X-Google-Smtp-Source: AGHT+IGu3gG6c418DyCLyq64UOVytvPPjptyLj9HsxmCFyjptBXKcCidLbQRWokyMo3VhHjgkQmzrQ== X-Received: by 2002:ad4:5d63:0:b0:6d8:aa45:a8a2 with SMTP id 6a1803df08f44-6e243bbbb2bmr94974656d6.11.1738243398854; Thu, 30 Jan 2025 05:23:18 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e2549222cesm6056236d6.83.2025.01.30.05.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jan 2025 05:23:18 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1tdUVp-00000009YDe-2eaQ; Thu, 30 Jan 2025 09:23:17 -0400 Date: Thu, 30 Jan 2025 09:23:17 -0400 From: Jason Gunthorpe To: Thomas =?utf-8?Q?Hellstr=C3=B6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Message-ID: <20250130132317.GG2120662@ziepe.ca> References: <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> <20250128151610.GC1524382@ziepe.ca> <20250128172123.GD1524382@ziepe.ca> <20250129134757.GA2120662@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: CD5CA40012 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3xogq57dr7jfrd76oadcxmmaeshtmamn X-HE-Tag: 1738243399-375704 X-HE-Meta: U2FsdGVkX1/QVfr4gExg8SvfUEOu88Nqag3VUqsXMsCEA7Vp8a90SVtGeaVjPNjzSyTDbcLVOxhAdmEJPxxJHu8D4KfQEjf3+q2mUOKAWI6daStSKncvwkq31Zlio9JOaSGE+ybcbzPe5t/x31Sqk0/RnJAZUEUDz7yUFiKm194JZ7NJqMtjZp4kOp0SHOJZWTtV2Y+C6y20LE6O6ssulxsfVWYtbxmRnh3tK3Oe25GTkyH/cyxoZ9CvOd8Dl7RFSN2H68H2vkbaMWF4pwMV3tpCh+sM5OeVI7cKfXSE2M3MnqnxAGUL3iP6M2M5+j3ebyIloTBoyWnSm19c6qB4NSlR+j8NrDD5kDr0TvDEf9fso5NBzDpwbOGdnTgQv2cCJ7AkrfTCgaVLwnBcBBT8zJXJDdfC3yJLu03K1hUWSZpS/ZZXuHmzd545CO/KguS0utktEuwX84z8LbGvykpykgjvW8hUgtYqfUZr5FAL4iGVRgs2G+BZ+F76+CY/q1JY8UsuCMR5XnyyaH1gn9lwXah5eaf/d1eWgtLYgvjssyNk2TZ4eBhOHVBWino9KdvSoGBQ6fpJwcHU3mG7FRYPIt+I9lvnrQlCjVQN0uvkm1AuiG3W1wTGFEZA1czSiRZ3G4VAqkKejtof29GwCvjsteIf98w98L2n7lMjwLFh4SHnnHAxmVCNtqizOaudpusqIEtZOb/kyCUAUc+p6j5GPUU3yH7lBS4+OEOlWT78ELC3+oJna6F1r4IS2Fg7VNzomSOExFTV8H8oW6BSVKuDLaCZh9p9GOKO3q3qmmd+84CH6wPFTdx7jAecBIsEESwpRROWhVmifdowzNZrveNXj0hCbBmUvSJyTLmWm7mUPL3sOqfC7FQa34HgMhBoBu8eClQStUYWYjU+BJU+V04sCJG473EgDGQtmNgAjCmSvuDWaygJHV2r/NDetNm/U1rqahm4xeWsKB4uf7E/yrN NTJgzCqS BcUXEwKjdeww6mEYhlg5mNcMeWmgVM1dWpl4czoEHvAxoJEKHFqUI3W7kOCCT3KdEvdB26dM8ofp6raavveMg7C9/gpFp8r0DwIo+hauclTBGGtunhu4pcN6znyaVhk7DGFSQekycA3ls5qzcp+n6jyRlMJBSV6MRZ9PEj5qxi9BvGZkxA2Uo31Xycyz4Fe4W8cj7IOzpQ7fV22Ieqj857tt2j6QXQ+JtjSbM+xzcqo8kZ3e6PkejgaqQuEZBDRSYpF5+kKgJyThlLbJEAyv9xgAM6/8DSAbDrTiR X-Bogosity: Ham, tests=bogofilter, spamicity=0.003783, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jan 30, 2025 at 11:50:27AM +0100, Simona Vetter wrote: > On Wed, Jan 29, 2025 at 09:47:57AM -0400, Jason Gunthorpe wrote: > > On Wed, Jan 29, 2025 at 02:38:58PM +0100, Simona Vetter wrote: > > > > > > The pgmap->owner doesn't *have* to fixed, certainly during early boot before > > > > you hand out any page references it can be changed. I wouldn't be > > > > surprised if this is useful to some requirements to build up the > > > > private interconnect topology? > > > > > > The trouble I'm seeing is device probe and the fundemantal issue that you > > > never know when you're done. And so if we entirely rely on pgmap->owner to > > > figure out the driver private interconnect topology, that's going to be > > > messy. That's why I'm also leaning towards both comparing owners and > > > having an additional check whether the interconnect is actually there or > > > not yet. > > > > Hoenstely, I'd rather invest more effort into being able to update > > owner for those special corner cases than to slow down the fast path > > in hmm_range_fault.. > > I'm not sure how you want to make the owner mutable. You'd probably have to use a system where you never free them until all the page maps are destroyed. You could also use an integer instead of a pointer to indicate the cluster of interconnect, I think there are many options.. > And I've looked at the lifetime fun of unregistering a dev_pagemap for > device hotunplug and pretty firmly concluded it's unfixable and that I > should run away to do something else :-P ? It is supposed to work, it blocks until all the pages are freed, but AFAIK ther is no fundamental life time issue. The driver is responsible to free all its usage. > An optional callback is a lot less scary to me here (or redoing > hmm_range_fault or whacking the migration helpers a few times) looks a lot > less scary than making pgmap->owner mutable in some fashion. It extra for every single 4k page on every user :\ And what are you going to do better inside this callback? Jason