From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 872B8C02194 for ; Fri, 31 Jan 2025 17:20:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 234046B008A; Fri, 31 Jan 2025 12:20:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20B74280001; Fri, 31 Jan 2025 12:20:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D3C86B0092; Fri, 31 Jan 2025 12:20:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E324F6B008A for ; Fri, 31 Jan 2025 12:20:17 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8D9EBB1105 for ; Fri, 31 Jan 2025 17:20:17 +0000 (UTC) X-FDA: 83068410474.26.099905F Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by imf01.hostedemail.com (Postfix) with ESMTP id 805CF40013 for ; Fri, 31 Jan 2025 17:20:15 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b="i7MRx9H/"; spf=none (imf01.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.221.50) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738344015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xczt2wiwP9aoq6grPvh88tM/HM9YU6nKxYw7mbHJ51k=; b=I6zxHfVxPvX5px3I66cjGMNeyGOw1pWHjq3ixyjRWVeZmhrPf9JP0KKVefhaTNZZzMhueO EJqdqyLHoJ0GzDqbIhskNJtfiYdCLwHF8pDP4/BT3ml9TpNtLS7oKhCfs17CSZKhVfvaar OBKl9g9kH3MpPf9wp4cVDmJ3/8I5UIA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738344015; a=rsa-sha256; cv=none; b=5E04b+z1akcy6RHTxNCgJQrRG1gegDgNRokvhInQ2Gyr51cZwyLD0bZOupoV/0zYywX2tu 8aSAVaiv7Vj3Gw4u+qoN/4rDzXZq0HnECC+vjPbvmRh5rfLDOD4z3WfSALsXNKyQ8weyTx xBjOPcQWENruRf65e7P6/k7m1osl7PU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b="i7MRx9H/"; spf=none (imf01.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.221.50) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-385de59c1a0so1220929f8f.2 for ; Fri, 31 Jan 2025 09:20:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; t=1738344014; x=1738948814; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=Xczt2wiwP9aoq6grPvh88tM/HM9YU6nKxYw7mbHJ51k=; b=i7MRx9H/l1yzjqroNMnTeeNgdmguxSwXjHpvvBb/jxBLmegWpKHluFz0Of6opFWG6k 77pTf7HNyj+jJ07uiIDtVwkeE9wmy3OJopJi1lZAABSmCAwpxunIBCT1K1NrGB5dFzJe gVkt8lqKNgxYc2sk7YLYdUtIOUfpoPuRo7Cts= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738344014; x=1738948814; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xczt2wiwP9aoq6grPvh88tM/HM9YU6nKxYw7mbHJ51k=; b=UC+jjGs4ZPaLhp06FaBfZNHwa+remCkYyNM6ujJQXhfyXPX/VlVY1qRDIpOz1euCBh LZtM1rgu8Ux5crEdezpTkk0EgJUg2X1AA3XgmOKM8fOzXKuDb9RT4C7tFWnA19eXhTHo VCaXWXOYdIETuhMB/F34w4QACUySjhkfWvlV3sBICZ0yLAL+hYY5G4iI5f4FsJTLieA7 P2N5OPkK8g30ATRjWyMbn5IgCK48ZYSyxgmP6e+yaKIQ4q4YICfUq5diEGf6/3YnSsuA 9iH6IzWBEc6rBEqT2veVwZZO5NC5Re+CwAIm3SsxuwkEJLyhmMZITOLBPEOVe5Ijc/MT Jukw== X-Forwarded-Encrypted: i=1; AJvYcCV5ho5gvPsvH3g2Gj1r0XLzi1Q1fPWmX7mW0UrM7mvPh1518HiMZppddg5wsMMW/uEuAwALaSgWZg==@kvack.org X-Gm-Message-State: AOJu0Yw0aSj3z3tuoyMGaueffS7W8v+1aZ3/KjVLJtqrs+lQhyTtbXpk /Br1g7YTNMWRDMYfucut5Q0+W0LZCgcESAvjr36dz7jaTHFKZLZlBRMkBMTtWG4= X-Gm-Gg: ASbGncu8VWBVeqpqKlS1vQa946nnNV7KB4gJcjtC2mEmIe/VTDvzefeh2MbXPGOOWN9 o9qdZgp2G058pNYO66/9KH2h9EDtv+VcKubJIjCfChTJiprD3Zo9cSy2xejhCVABc65+RepZo9a 6xOXvrNuTHR8VF9TyePH4DXZLZq85qJjuLIuefEVPidMkcExqdF3ggMhQgpInFVdE6r9u8xGv2y VzX4KO2nughyeE4SZeCfdiIn9GCwh7mqeA3Ltj/hRI2XjOjA/cFH6PXwel6T24Y7PuR+VFrlDil Q68mfBjWeyijTxyDOCqXJh5Dr0Q= X-Google-Smtp-Source: AGHT+IEBqHvHKIqdGp0t4Qh+w+AOU1lOYA0A7nG1BCLDh39RGEo9xrVIZ8SI5EuvHfApoC/uo/mR7w== X-Received: by 2002:a05:6000:bd0:b0:385:f64e:f163 with SMTP id ffacd0b85a97d-38c51967e87mr7449467f8f.32.1738344013909; Fri, 31 Jan 2025 09:20:13 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:5485:d4b2:c087:b497]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c5c122707sm5254768f8f.50.2025.01.31.09.20.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jan 2025 09:20:13 -0800 (PST) Date: Fri, 31 Jan 2025 18:20:11 +0100 From: Simona Vetter To: Alistair Popple Cc: David Hildenbrand , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, Andrew Morton , =?iso-8859-1?B?Suly9G1l?= Glisse , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Jason Gunthorpe Subject: Re: [PATCH v1 4/4] mm/memory: document restore_exclusive_pte() Message-ID: Mail-Followup-To: Alistair Popple , David Hildenbrand , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, Andrew Morton , =?iso-8859-1?B?Suly9G1l?= Glisse , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Jason Gunthorpe References: <20250129115803.2084769-1-david@redhat.com> <20250129115803.2084769-5-david@redhat.com> <7vejbjs7btkof4iguvn3nqvozxqpnzbymxbumd7pant4zi4ac4@3ozuzfzsm5tp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: Linux phenom 6.12.11-amd64 X-Stat-Signature: ars9qcnj5xuq6a861djn9bcxcanszyed X-Rspamd-Queue-Id: 805CF40013 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1738344015-365104 X-HE-Meta: U2FsdGVkX18/wElkIj1l/r5umpbnzcQslLZbtYPKhqv0aift2d8fzcwSK2W4XILmcUz6c6PmTj21lQ81uy8xxxCxwVCxKXmXIL5qQHz0RHjr2jyU4urJxgBdS5Phwg8UqRqahlRY7V6ApVGGspwtpfJH/XgZ2L7A6307WISxRz0fEfVt0m/PH5XIMulN3eX8lNM7kv53vtZL4Z7DoxCVlZaRzeq3BDPeiufaMdDzZ8BxzydcWq+3+MANE03vN7iq04YU1xt4f1Kdfb8Chbu0QYb2zLTVOx8d1hdcuF2UAh5eIMEG09GLcWW39d/SpIVtwYYLutICIWYj1xNH4a24OJLE13aLyV8Ox6wgP/1HzUDzOtLaYAtG335t19VObXJDpfOYc3iR/mxzKp6zEaRhOFcOpmOlYjWE5VELvIHHQrUHCKvVm6l4JgmZDmVZrHUFsl858LioZ0l2O1huwFq4gn0QaHwaSlRGobCQG0qZav+vXmCEf7d0EvErho4PHrRhaHrvAyBgwq0u4CKNqoLH9FoW1evjEts74z/QDBNJ9qTrLlOo5ao/dvsTDGaaH418POgQ2jtHkffVWnzF6Zyj2GMKPWLQcqjwMabL7Goa67evC1aH6O4DDAP8NCNfthD2gjTqKPRMJ4i+nIAQg8YCIGd60bDRTt2UuX3rL1po2jrKjHs6lx71K8LwOCVxrt+K/LOqiSQRS0lg0nEOcIJhp1iggTLlEitA9pQrl50mNklJ/RtAoCWDBpjhJ7pe8BHd5u29h25NqM2aa4RDSQTMBuuTomGswm6xdg04ji2YDcNEOq88In/IVQQQDjFA6ZoGAoyEUwWgjBcFCyrJnNH39Q7/w91ojGm3q/sp1v/ZzT7N8asVhipDGnNxChDVT442K4gvMCUpKnQk/h9ffY4FDGPd8mKJYX404iKlUdlEcbqGuDvcRzTs/bUgRH0p33/B0UbBafDQ9wmtmhvh1aL EY/kFRuF A84YanHE8llUQANHviQwQYKklBviMxoGfVIRD3Rwv12isjmijshVZ9VX21rfDQPml0yW91ks5ij/QGKAq2EPlB5Yv3GBbyTYRTKtiv+CwhXFir+2/HcyABmlu8kyLTq0tMfEJZFosFgg/atgxdWJ27ebYoZV84W3c7oLmZVIh+K4E/ns2LZn0ypa6l5ytjMhVBCIzuo0Pqr1jYY2Jkx8ci+h6Ft1KvJwMEPbroAb8ZjTrxYpSZkCgkqbp65Ox63Hbw6kgn+Y3vYDz0OfFUKB6fHJeaxfwg47JHi21dCTzsyZsAfi9CzAJeek8zupYuc/FN/5p7pISDlz+OLlqTOWSVx5BGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000919, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 31, 2025 at 11:14:06AM +1100, Alistair Popple wrote: > On Thu, Jan 30, 2025 at 04:29:33PM +0100, David Hildenbrand wrote: > > On 30.01.25 14:31, Simona Vetter wrote: > > > On Thu, Jan 30, 2025 at 10:37:06AM +0100, David Hildenbrand wrote: > > > > On 30.01.25 01:27, Alistair Popple wrote: > > > > > On Wed, Jan 29, 2025 at 12:58:02PM +0100, David Hildenbrand wrote: > > > > > > Let's document how this function is to be used, and why the requirement > > > > > > for the folio lock might maybe be dropped in the future. > > > > > > > > > > Sorry, only just catching up on your other thread. The folio lock was to ensure > > > > > the GPU got a chance to make forward progress by mapping the page. Without it > > > > > the CPU could immediately invalidate the entry before the GPU had a chance to > > > > > retry the fault. > > > > > > Obviously performance wise having such thrashing is terrible, so should > > > > > really be avoided by userspace, but the lock at least allowed such programs > > > > > to complete. > > > > > > > > Thanks for the clarification. So it's relevant that the MMU notifier in > > > > remove_device_exclusive_entry() is sent after taking the folio lock. > > > > > > > > However, as soon as we drop the folio lock, remove_device_exclusive_entry() > > > > will become active, lock the folio and trigger the MMU notifier. > > > > > > > > So the time it is actually mapped into the device is rather > > > > I meant to say "rather short." :) > > > > > > > > Looks like you cut off a bit here (or mail transport did that somewhere), > > > but see my other reply I don't think this is a legit use-case. So we don't > > > have to worry. > > > > In that case, we would need the folio lock in the future. > > > > > Well beyond documenting that if userspace concurrently thrashes > > > the same page with both device atomics and cpu access it will stall real > > > bad. > > > > I'm curious, is locking between device-cpu or device-device something that > > can happen frequently? In that case, you would get that trashing naturally? > > It results in terrible performance so in practice it isn't something that I've > seen except when stress testing the driver. Those stress tests were useful for > exposing a range of kernel/driver bugs/issues though, and despite the short time > it is mapped the lock was sufficient to allow atomic thrashing tests to complete > vs. having the device fault endlessly. Yeah the practical use-case of device atomics is that they alternate, as the processing shifts between the cpu and the gpu. Classic one is when you generate variable amounts of data per input block that you fill into a big preallocated array, and the atomic counter is your dumb-as-rock malloc for that. The atomic moves as the generating/consuming of that data moves around the system (and needs a fault each time it moves), but you really never want to have concurrent access going on. Any thrashing concurrent access is just broken userspace (or a driver stress test). > So unless it's making things more difficult I'd rather keep the lock. But why do these stress-tests need to complete? We have a lot of these in our gpu test suite, and we just nuke them after a short timeout if nothing blows up. Concurrently hammering the same page from both gpu and cpu really isn't something that should have any forward progress gurantees imo. And I feel like too much locking just risks us hiding some much more nasty (design) bugs underneath - I've had that experience reviewing both amdkfd and the in-work xe code. So my preference for gpu interacting with core mm is that we have the least amount of locking to make sure we really don't have a breaking design impendence mismatch going on. Cheers, Sima -- Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch