From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95385C0218B for ; Fri, 24 Jan 2025 15:29:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21FED280075; Fri, 24 Jan 2025 10:29:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D013280070; Fri, 24 Jan 2025 10:29:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04A92280075; Fri, 24 Jan 2025 10:29:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D7259280070 for ; Fri, 24 Jan 2025 10:29:03 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5236748BF3 for ; Fri, 24 Jan 2025 15:29:03 +0000 (UTC) X-FDA: 83042728566.09.39F6896 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf30.hostedemail.com (Postfix) with ESMTP id F0E318001A for ; Fri, 24 Jan 2025 15:29:00 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b=HzDl6uzc; spf=none (imf30.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.128.47) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737732541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w+AUOleCAQXNWX4R6WEOCPMezKuf6jJkTIyVfG+jJvY=; b=YbqK8gTd1/7sY449dGO/qfIwZhlfrHX6Y98HQQKGT1qG+GRA7Ywl27yJ2T/rzFXU6J0AR7 a86o6CFRTtJ8ifzIQ6TIJdTbgUdnWkXccq666TlN2XrUpwkNzPoljCfqez/YgZAm+/utD8 nbGG0mZ/l8P1bQ0PHcqVq3q2j/zAkb4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=ffwll.ch header.s=google header.b=HzDl6uzc; spf=none (imf30.hostedemail.com: domain of simona.vetter@ffwll.ch has no SPF policy when checking 209.85.128.47) smtp.mailfrom=simona.vetter@ffwll.ch; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737732541; a=rsa-sha256; cv=none; b=mgGUeZW/T6mWxmNehxpzqVYlumQ4volbbhMz/fPFhmBX6vjvvwmEEDe0GNQraB/h9jrX94 JZuocpMHUdgZ9DSgaHxSsl2/0oipoAMSPfRB3n7SBld6r6wYNXuzbREXPpsiMvZgRjPhI7 T35vhPTsU/ZWU+axf5fcIy7TSgpPIUY= Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-4362bae4d7dso16104275e9.1 for ; Fri, 24 Jan 2025 07:29:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; t=1737732539; x=1738337339; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=w+AUOleCAQXNWX4R6WEOCPMezKuf6jJkTIyVfG+jJvY=; b=HzDl6uzcrlya67xDgpQnfko7nPs/n9lB0bBPTcNLSDbXfMWdihyiUwPhDEKQSG5aqn Gq1KzCfUxo0vRNZBKSR7AHzZCIla6ORLwoLqqStel0tGPHBswH9UoQsTuYXYxjAxKPXA rIf+mgrDNNCn3BJmePlWnnu+x/rzqsOpKLwso= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737732539; x=1738337339; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=w+AUOleCAQXNWX4R6WEOCPMezKuf6jJkTIyVfG+jJvY=; b=lgrfOzVXQrBFdy5QQK/zIc+fwZXW0FKghcjPQzmxNQDP4vmVwLzdu5yZlvxmFf/lNj WCIv5cUHQ+EdAgFf1iwSOclSEfzAOALrtUu8c3LhiIJDJP8bKbOpYFXw7mK9/Ko60dAj BUAkx8QqtGELas6wsL8cy3vScnVDwZ4xfF0KvBWZopmrYEGnysKi69t1G+mq+TDvAGK+ znGrCUwIbHLmP/UnATFzl/kA0mP7DS201huK2dI9FkzQJFFjQ/+2Zt0BSdcgimC6gLe6 nQKJaH5IjQDRqj86RV5wf1r1XU3Pnz6TGCr2eVY1wi1efBHe8qwhrjHwVcr3E2/R//bX rKUg== X-Gm-Message-State: AOJu0YwYaqPVyTz96aSCYUjeqVhhk3CC3QzEVWfn4kX/Z+LO0Y0bVBRL YPLtQlrqJKOE4tlVLxACoO/8N8ubhgTVObayKCZhixFhjrqce9/Yg3AJ0qbfX6w= X-Gm-Gg: ASbGncufmYCcfo2ha6ze5C0nrWRjswJnyRcpqYOmUbcXqRIKjYAR966Hc4iV8BIwfyd NwdxUfpTq8QR19iRT2u+Gom+zSEx4SCvsECgRt0fEg6lTLnsnbVFJxcjxDdXz3MRtPxTDNNAsJo JruGxY6GROzwLf3L5C0nbIG4dfYIDd5R8agcfvJQuNxc4tohzGRh8VB+UGj/h2qh+oN8GWYFGGc luryYB9FTdTOm/cxiW9zaYx+ExfJYK93FtQwr0rYGFHSz4rA9g67P1qzr8oiUN8cs93r+vo/cp+ cxZ3L2eBQ49SfQjm X-Google-Smtp-Source: AGHT+IG7AtKSShZbMOeW1UKchLmGsIKlwNKebn/4Phf0AM9NhDt5TUVmFQc7i/i3BJ2+0nT3Qeq81w== X-Received: by 2002:a5d:64e9:0:b0:386:2ebe:7ae2 with SMTP id ffacd0b85a97d-38bf57b8aa7mr28516016f8f.45.1737732539149; Fri, 24 Jan 2025 07:28:59 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:5485:d4b2:c087:b497]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a17662bsm3012012f8f.4.2025.01.24.07.28.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2025 07:28:58 -0800 (PST) Date: Fri, 24 Jan 2025 16:28:56 +0100 From: Simona Vetter To: David Hildenbrand Cc: "linux-mm@kvack.org" , John Hubbard , nouveau@lists.freedesktop.org, Jason Gunthorpe , Alistair Popple , DRI Development , Karol Herbst , Lyude Paul , Danilo Krummrich Subject: Re: [Question] Are "device exclusive non-swap entries" / "SVM atomics in Nouveau" still getting used in practice? Message-ID: Mail-Followup-To: David Hildenbrand , "linux-mm@kvack.org" , John Hubbard , nouveau@lists.freedesktop.org, Jason Gunthorpe , Alistair Popple , DRI Development , Karol Herbst , Lyude Paul , Danilo Krummrich References: <346518a4-a090-4eaa-bc04-634388fd4ca3@redhat.com> <8c6f3838-f194-4a42-845d-10011192a234@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8c6f3838-f194-4a42-845d-10011192a234@redhat.com> X-Operating-System: Linux phenom 6.12.3-amd64 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: F0E318001A X-Stat-Signature: x1m7qrdu78e1kfstj64dqu6i47dutj3f X-Rspam-User: X-HE-Tag: 1737732540-217214 X-HE-Meta: U2FsdGVkX1//OFNdNmzTyBVaRfgoM3oRwMoTEcEw5pBHbsjID36F3Bjo937ytcxSCbauoCaCIGrBfS5qg6wC1w7afSdQZMd/acrCNyPPxx3vEplupbn1d1dd303R/ZQLSndK6aGcZEwFiAKLLUxxZWpooUARx+EgL/+S/1OUmNPNdLoWcT+Wz/EGPbkKfpoW1o5RqVODQCSjfmZhsk41g56gG/K6lCpQiyvi2u9LDthEifPF1ewB9dCfdYzTj56TxV2W0V88HgMDr7YhESHgX3lu19qVqMCKwL97lyKlIHAZy56O2CGgqHE9CkU2O9hKwb+StaiOU7AfzvnoVXb/VNmhhc67ZLO3dGxNXpxcQALl2inXd5U/wiAGVl2Lalhj9il3XzmpEELtlfZSI3I9M0PjZEQ8IH4esFC52Uzp4hLjiByflWgyYCPihQb4r5x37AHiNF0c+SUyvWPGvomhhbYjgCZTkCFE68vG/lEXhRpi1rpLCEYjWZjhwTu4bJtPo0MWX+kd7oE7LGay+WO1LaDhCKwANC6tifyjPaXBb5eVyvKqirO/bbUKKEM/wHe5yujOM8OROgzL/2B01hdNj0h9OdWJmL/dmuzu4RzgVJ1Kayup6GvW4JTBmFwNk/A/txDtpS5gKxhBvgg2jXeqUAxZBZ/8z3v8dX40HGEMu2VNyLziqprTi6r5JGraBkOAOMGLxrM5zwBZThyhbe+H+NuczodmYFEAUlLrDdJ4XldG2004JJfZ880UTvxqPz8TrmzFuOTIqbruUur/sK9p3TZSIZvT9xDIws5MiSU99YqMxskqKKVBIEQXSs4BKwmetiRAqrBeaFOE7aMzff8aIMifQX/wm8MziSjPRFyai5gFw5gTVPNEymvmKMXdVan66MDlK+yD4Fvhb+UX22BhpRy2crALvKU+J4zcWmnyrjNFEI56VuTUsMy3fGFWuha7Mj5hkKpAigawnNdZNNA 4ZMY8OFL WOmyBub3Se+NPlhu4U9SM3qb2lgl8sZwI/3SeD5MY08+IvBtEjKI8mfYNvwrc3xKTNi90xwuqPszHX/1qXal9muD8PMLiCQc4d3KyR5vVuD5XagfcGrQCEYzPStyr9vhbM49hg1DaflzOU/dMvDLbWmEpnWsklcZoCXCO2C43okMUQ/Zi1nepJpbjwMO9NAIHldhxCBJoug8WVFbah4F9iZiexwUsPzH1/pdpCZWJn+pJ7FRP4ziua5ulimPyLN7MXIKcEk/fWbBgjWQdz+Wkr9LNUqFzZT7cHZSklizxZc0tPQ0/20jQcf5rhyt+VeObEUuoIC9ei79VneCdQ/zLUwPy03dpXNjcPxuBMrlcDHU1Mg8Plx16QKrhAosg1QI6ZfO4uGB6xV4cttNjOqPlCUHn0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000135, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 24, 2025 at 11:44:28AM +0100, David Hildenbrand wrote: > On 23.01.25 16:08, Simona Vetter wrote: > > On Thu, Jan 23, 2025 at 11:20:37AM +0100, David Hildenbrand wrote: > > > Hi, > > > > > > I keep finding issues in our implementation of "device exclusive non-swap > > > entries", and the way it messes with mapcounts is disgusting. > > > > > > As a reminder, what we do here is to replace a PTE pointing to an anonymous > > > page by a "device exclusive non-swap entry". > > > > > > As long as the original PTE is in place, the only CPU can access it, as soon > > > as the "device exclusive non-swap entry" is in place, only the device can > > > access it. Conversion back and forth is triggered by CPU / device faults. > > > > > > I have fixes/reworks/simplifications for most things, but as there is only a > > > "real" single user in-tree of make_device_exclusive(): > > > > > > drivers/gpu/drm/nouveau/nouveau_svm.c > > > > > > to "support SVM atomics in Nouveau [1]" > > > > > > naturally I am wondering: is this still a thing on actual hardware, or is it > > > already stale on recent hardware and not really required anymore? > > > > > > > > > [1] https://lore.kernel.org/linux-kernel//6621654.gmDyfcmpjF@nvdebian/T/ > > > > Thanks for your answer! > > Nvidia folks told me on a different channel that it's still getting used. > > > As long as you don't have a coherent interconnect it's needed. On intel > > discrete device atomics require device memory, so they need full hmm > > migration (and hence wont use this function even once we land intel gpu > > svm code in upstream). > > Makes sense. > > > On integrated the gpu is tied into the coherency > > fabric, so there it's not needed. > > > > I think the more fundamental question with both this function here and > > with forced migration to device memory is that there's no guarantee it > > will work out. > > Yes, in particular with device-exclusive, it doesn't really work with THP > and is only limited to anonymous memory. I have patches to at least make it > work reliably with THP. I should have crawled through the implementation first before replying. Since it only looks at folio_mapcount() make_device_exclusive() should at least in theory work reliably on anon memory, and not be impacted by elevated refcounts due to migration/ksm/thp/whatever. This is unlike device atomics that require migration to device memory, which is just fundamentally not a reliable thing. > Then, we seem to give up too easily if we cannot lock the folio when wanting > to convert to device-exclusive, which also looks rather odd. But well, maybe > it just works good enough in the common case, or there is some other retry > logic that makes it fly. I've crawled through the path to migrate pages from device memory back to system memory a few months ago, and found some livelock issues in there. Wouldn't be surprised if m_d_e has some of the same, but I didn't dig through it (least because intel can't use it because not-so-great hw design). > > At least that's my understanding. And for this gpu device > > atomics without coherent interconnect idea to work, we'd need to be able > > to guarantee that we can make any page device exclusive. So from my side I > > have some pretty big question marks on this entire thing overall. > > I don't think other memory (shmem/file/...) is really feasible as soon as > other processes (not the current process) map/write/read file pages. Yeah none of the apis that use this internally in their implementations make any promises beyond memory acquired with libc's malloc() or one of the variants. So this limitation is fine. > We could really only handle if we converted a single PTE and that PTE is > getting converted back again. > > There are other concerns I have (what if the page is pinned and access > outside of the user space page tables?). Maybe there was not need to handle > these cases so far. I think that's also ok, but might be good to document this clearly that concurrent direct i/o or rdma registered buffer or whatever will mess with this. The promise is only between the gpu and the cpu, not anything else, in current apis. At least to my knowledge. > So best I can do is make anonymous memory more reliable with > device-exclusive and fixup some of the problematic parts that I see (e.g., > broken page reclaim, page migration, ...). > > But before starting to cleanup+improve the existing handling of anonymous > memory, I was wondering if this whole thing is getting used at all. Yeah if this can be made reliable (under the limitation of only anon memory and only excluding userspace access) then I expect we'll need this for a very long time. I just had no idea whether even that is possible. What isn't good is if it's only mostly reliable, like the current pgmap->ops->migrate_to_ram() path in do_swap_page() still is. But that one is fixable, the patches should be floating around somewhere. -Sima -- Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch