From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E5DCC02197 for ; Mon, 3 Feb 2025 15:08:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBD68280002; Mon, 3 Feb 2025 10:08:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E6DF6280001; Mon, 3 Feb 2025 10:08:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5C71280002; Mon, 3 Feb 2025 10:08:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B9E6F280001 for ; Mon, 3 Feb 2025 10:08:10 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 789D01C543F for ; Mon, 3 Feb 2025 15:08:10 +0000 (UTC) X-FDA: 83078963940.28.52CEAC4 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) by imf19.hostedemail.com (Postfix) with ESMTP id 8F57C1A0010 for ; Mon, 3 Feb 2025 15:08:08 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=KcI0XMdo; spf=pass (imf19.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.182 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738595288; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VIfqpXWa2saTTZIkeCGgjoPhbUp2hEhSFcQHJJf5WU0=; b=lX+bTkQ7F0kyX44WvAYsqlNtQvUUHaCMAGgn8dKF6xVXVdFb5yaTFzJ+vLVyNTHaTYt0y3 ruTz8j56w82MiJRUYdDSPO3CtPu1CnFI0B7ei0ESiRuu15iFl/YXv9Y/59IDEKprvppRNM DgX8IlNaDmuZPY0jio/EirjIljXU6X0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=KcI0XMdo; spf=pass (imf19.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.182 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738595288; a=rsa-sha256; cv=none; b=X7LPGn4LsLjgxHPHJ+L2R2FFsdhvw57ir4CUWP+ApySfgyuyEer93a+nupWCromqupkoeg utMdhABLIc/9eztVOvk3dDS0P4dQAsj9Ho+11yAaNltB4XHKZgfd/TqW+puo6Z9pgKFq1i 0sWg317elWbFkH6URwMgONUhd02wMB8= Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7be3d681e74so400355885a.0 for ; Mon, 03 Feb 2025 07:08:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1738595287; x=1739200087; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=VIfqpXWa2saTTZIkeCGgjoPhbUp2hEhSFcQHJJf5WU0=; b=KcI0XMdoBvUsZJxsa3HsrGZw4UcU1rfFHcTxoGcwTYSbyQnRz1cN9tS5GSct17SjKM vvvy7bNq35wav0aXkO6ON4ce9/D+7WE5TFvn9G7GYPt5q93E0CVXiJK3nZlDK8H4Z+vM hEYepaNQ7bH6+XPs42O27oG54gQTYy3OmpLZpfokkSX5qmeVmpAve7zazSW9nWXAKgYk ONeP0DvKwOVwsM6srerEncuMKCYnznqrjDc7hxp4B9wyVrbg8TAvlt6NP+4zINCfBdU3 dTkv01djk7bOT0jLC28EBqYHMBOSDUZ090It6FdE7Ui5LKSwADE/6NxKqwmKiLETaPPx VIQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738595287; x=1739200087; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=VIfqpXWa2saTTZIkeCGgjoPhbUp2hEhSFcQHJJf5WU0=; b=QEnaHjjvtJXUDMf5sVhtBSrZFrTziQj85nkRIFVH7XL1NbDs1jUu7DoOzJRk1kHyPb WuCghrmDaP/rt98bSYM9d3UG2pn25Jp5NgHvc3QsRYSfFgYpW48k+cT35P9SWiL7+znO j11iMH+v0zl1yqvZYzipfB6p8TxZAJwhCEQN3HMEx4saDtw4dmpfImSiM8sBXU6gd66p e8HmDk729DROWecMKff6pQUME9CO1kRPUHGEO4+qPeaxi9qsWjTFN7mAGzL5PpIQCyrv IL9qUxe9bQ2Je5up/Oas7w7Z9V1h1JbgmtyKLIQz5wjqroLlk7UuiMmYkbKtHCT01wgL w04Q== X-Forwarded-Encrypted: i=1; AJvYcCXOK3VEJlKPIZy3Tl8FNJGJWi1GyF9BGt/Vf4jAAUlpVV5Ai70tkRwW7rN6bKVz94oHg7Z2Eat6AA==@kvack.org X-Gm-Message-State: AOJu0YzxU3/NQSAevFvZwCGSSS5VwGP6GPBJ9BQspaGbaQH3ovienBHP vTdj1mkk0XV3G/wOLF5MK8ds9IUej0sY9Q++mchYCiN5Eh9OCbLb9VKg3MXeQw4= X-Gm-Gg: ASbGncsKxgc6iLDbV4lHlJPlLn75a0uBeo9+jgqO54eEqLYLWV+XlwLR/20VxriSQEr y2+SkIPbfI2mV8cPTpMtjx8hO0rs7IbHbnnax9Qba7B/L7eBEJ3JF1JowB0clBsNZj7WjHlt8zh OULUwFY4i64FTcrVeC8zxUMRtCjHgStqu4GAi30oPYI4SMI3/Pon7weQtqtt0hGZ2JXpLqDxw2s grWOuzsAZifI3V9+lTDe+GI7zuyOgc7zNBKbjQ8WdpLNhb+4V74PH2zXBz3Djom4w== X-Google-Smtp-Source: AGHT+IH/qa1lrVLcKxozjVl1Axtfg0S5tmpdY0Hz1inHc6vNMg7mQBJDuUSeOKIlGn70ib6W5YOi8A== X-Received: by 2002:a05:620a:19a8:b0:7b6:c540:9531 with SMTP id af79cd13be357-7bffcce8d01mr3518318185a.18.1738595287317; Mon, 03 Feb 2025 07:08:07 -0800 (PST) Received: from ziepe.ca ([130.41.10.206]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c00a8d1280sm534889985a.45.2025.02.03.07.08.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Feb 2025 07:08:06 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1tey3R-0000000As1Z-1rVV; Mon, 03 Feb 2025 11:08:05 -0400 Date: Mon, 3 Feb 2025 11:08:05 -0400 From: Jason Gunthorpe To: Thomas =?utf-8?Q?Hellstr=C3=B6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Message-ID: <20250203150805.GC2296753@ziepe.ca> References: <20250128151610.GC1524382@ziepe.ca> <20250128172123.GD1524382@ziepe.ca> <20250129134757.GA2120662@ziepe.ca> <20250130132317.GG2120662@ziepe.ca> <20250130174217.GA2296753@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8F57C1A0010 X-Stat-Signature: 8neiiwhid5itokcsm74miwrpx5uhrgiy X-Rspam-User: X-HE-Tag: 1738595288-121141 X-HE-Meta: U2FsdGVkX18RidOiE75OSECcpRXzGVgJzGv482DJZHW7Te/ZEvEwCy7Qz5TgNBDMXLBt3FuNKJ493k9D04BsHUzVZgkernFBaX2LbQKdSt7HhomVAmVCWkPkyoUUPtN3NIIY+OfqKKn3bf7PGBJO+SzjH76IZHfQ60AdRNVoUo8h+baNE5VEo/IlUNpFZSZ4sjAuK0nOPcbgRz1VGvLRGPwHfty+5Knp2cCoe0ZHHHGRTUlAj1irrGYWDplh4OEM45bhalpPMZGx54z65UDcgUpdtYds271TXhCvnx+wNK19du1BbDHJ2NWF59wcoFGFU02W2b7MEZdhUSxu4OlAth6GCWRwrdAM3Y9zDWPofsEbiauWHJ307c+eKsV5IdRi6NyTb8WQ90vTmvMRd4AQPwcHAXc6dzGvWHUxc8LgbvvCWyYon/6zFgtktS7gzIk4hvyOImPYSszZFT68qhYdQn6bdRLDswv35R8fUoqQhV4UKZLWa26vkqS+ZfsekEyJ4vJ2RolbNvO8gNveNSi7xb9BhBiKgZeWtpjYaZrj/KeiODW2SavzkfDFYCf9T9sy50c7kaMw6j0cF2p+g84QcT/AX60fJmEbtiM/gaQJGmkctpcmvA310KMYS0rvIgOxCSnFYDV66yfKzfVVrBS2MqPTNC51xkODiKCsXFksx+K3Voy6Etn7rS47KvQv92xKRSgJ1dNbx2EyPVgdhH9SvE0VUptEZ0smggnoWIN3Y2nohGolY7Zm1kNuRA6KN5FcKmDxqpaiPmxw4dvZ/yMmZVyo69rmT1g4UMrXkPJ36roN5UHj9Fxcry93YNTniX+taHeqEjI4Wkwtrqm22WbQLwmsnI8xpwLpFfLJebJzWttCDCGJHpzQ2EU2+eL5UbvNSHhZq0KbFIakyRhcZPM4LCt0NJkZq3VntXTQNEoRJJJ92bJJHMBceoKglCmBPfALfxsiPaNj4nqFYCw2iuc qBOVBZUp fP4ni4UtECfE43LUzyPaglDuabOo7iEg0+/0+E+oHsjiK+ptqHNyzMEJ8RwUHzLjlpg8mG0s8rV6YDafcXG8R9sazY10LRG+KpMkGC6gh/cmcAs+Qzk+C+PrCvxfyeCcMTtfn+JlyajMV97TlFMBb1OpMxxDRyXy6KC7RC8rbuS+mZklrkRtkU0Y1bXp1E03h6bYbyqtAmoxaVUePNePCK8M/Nv8odooC2Mc+eVkqHK+c12TPsZDJXpcVsLINoXdiEYZx9fmrogsEU8ApXQwBFle2G1xlZTwkqmMv X-Bogosity: Ham, tests=bogofilter, spamicity=0.015598, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 31, 2025 at 05:59:26PM +0100, Simona Vetter wrote: > So one aspect where I don't like the pgmap->owner approach much is that > it's a big thing to get right, and it feels a bit to me that we don't yet > know the right questions. Well, I would say it isn't really complete yet. No driver has yet attempted to use a private interconnect with these scheme. Probably it needs more work. > A bit related is that we'll have to do some driver-specific migration > after hmm_range_fault anyway for allocation policies. With coherent > interconnect that'd be up to numactl, but for driver private it's all up > to the driver. And once we have that, we can also migrate memory around > that's misplaced for functional and not just performance reasons. Are you sure? This doesn't seem to what any hmm_range_fault() user should be doing. hmm_range_fault() is to help mirror the page table to a secondary, that is all. Migration policy shouldn't be part of it, just mirroring doesn't necessarily mean any access was performed, for instance. And mirroring doesn't track any access done by non-faulting cases either. > The plan I discussed with Thomas a while back at least for gpus was to > have that as a drm_devpagemap library, I would not be happy to see this. Please improve pagemap directly if you think you need more things. > which would have a common owner (or > maybe per driver or so as Thomas suggested). Neither really match the expected design here. The owner should be entirely based on reachability. Devices that cannot reach each other directly should have different owners. > But upfront speccing all this out doesn't seem like a good idea to, > because I honestly don't know what we all need. This is why it is currently just void *owner :) Jason