From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4423DC3601B for ; Thu, 3 Apr 2025 06:07:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73E31280003; Thu, 3 Apr 2025 02:07:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EC2F280001; Thu, 3 Apr 2025 02:07:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B3BD280003; Thu, 3 Apr 2025 02:07:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3E466280001 for ; Thu, 3 Apr 2025 02:07:38 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F0FEC121806 for ; Thu, 3 Apr 2025 06:07:39 +0000 (UTC) X-FDA: 83291701038.03.5255008 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf20.hostedemail.com (Postfix) with ESMTP id 0F3A71C0002 for ; Thu, 3 Apr 2025 06:07:37 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="L9of8//c"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of airlied@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=airlied@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743660458; a=rsa-sha256; cv=none; b=Qd+qlOltXa39XHLyNn4v3SGt2IU/HsLSXmGJ4IfYyllWIv3cASHFtnwZ3+OGN/QRmLvowx cCq3krTcXC2szYrMWkWGQXafgSONcM5rn+0+mvg4j8J6u4rK/tc8aRmdwXw0dlQqBT74bF uikLFwhmpsjMO0tANSqB59ZcdxNKvFk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="L9of8//c"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of airlied@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=airlied@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743660458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0y1chSLTgq4by+/iKzl7cqrc4OIVMXRZ18Ndod/oVA4=; b=jASdjIWbpcd7bUvOkS+aJ8JTPrbanirczgZwf9jK9WWRs0b10ySNVoT4aVwWdI3uxWn8sy cxRW6Xq7PB3VXn/u2Vuhp72Hq0xqsyyPhQRN2IoDxNI9xtUarbVToA4zFOCmutJG2EdJLm tlldwoIG2D0rPs62u1dYXNIREY878fw= Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-ac25520a289so81096266b.3 for ; Wed, 02 Apr 2025 23:07:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743660456; x=1744265256; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=0y1chSLTgq4by+/iKzl7cqrc4OIVMXRZ18Ndod/oVA4=; b=L9of8//c6XRRRYm0bFf964RhrdSyRSHaVSemC3FIqW65qvDp1lORmCW85B7Yhf0kVF AmmmgLj2OFscCQpv5ByEFzTNr6TTrcTA25KhlU766p3lwvaIiTpcipzyUiqa7jupg3Dp 186P1aPMejhEKaVf5/Z2n66DTzIguaOZYz0FuQ1Cl2I7PTQhsmDjNpLeoBREgUuFVrod 8LMIIIiUKubKgWpPnu4tg6wwypYCBUE2pWzoTJkOqUe2KHEQTl2ggpBbHQHefU45bCpU W2ySBO3MRRmqYY0Qs2nuHEfB312QFW7nedX38jRijmh4dJOjlYFZFBPzZMnmpPP3troP r8nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743660456; x=1744265256; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0y1chSLTgq4by+/iKzl7cqrc4OIVMXRZ18Ndod/oVA4=; b=dxKuZsq+jiozUZXWN3smm+tqQlWGAugUpVFzq1nKPGktnWoLO2ZkmgQjMRC83DGwRY SzfX8SNo8PAqX/baUUqCYwYyykZ+aUXmk2+61KVfxl9De0LjS9kwODGHqZjU93llyDNQ 4Rqj25kANcEAU88GYokplXlRTinFtsXSUKXmkATwYfWUMohVf7M63yX1+RQ1yIChzoik arojKPwDNFxph+l1F94M2Fdis+zmbkltnSbhZuxv187PUf0AAD6MFKc9txB0vIXFW/xx RmPfoCkU/rDI5BwlyDi0lgeM9aF77kUzC4GwLZ6l8tpcQLzhlN8KDcY/HqSESJNSptqY ewFw== X-Forwarded-Encrypted: i=1; AJvYcCVARWH3O4BzVb0OTgh5COs1PFMnMDikthoE0P1u+HUbbVSfp16aMaF1HyzIdhqtTy3UON6IszJIRw==@kvack.org X-Gm-Message-State: AOJu0YxTMmOPQH/y52Xwz7uSFDfo8ONKEy3FFpKDpr0PA/1fyQL7gL1X kd9cVCNcKaigAhexO0itH1m8dB0vQkNB++5ymoMfZa2VqzwLmVy3IoodgApPvEDlxlhqEEv9/7c wXmWjlwSfAk9zWKUq87IvuZwGAA4= X-Gm-Gg: ASbGncs8zxbmomU3BkE6g58rWQZGekr6CSFI4k+jJaC+lr86nlcUkZnI92iRYZXr+7C OujZGv5Fw1veeLOZWUKM48i4NffY6xHqGapqr3kb46GPjya14MprPmswMeuP9xDtrLfvYupxUTJ XQXijXirR+w/HpJcpsx2zeC4bNH0r+4n0tB8k= X-Google-Smtp-Source: AGHT+IEf6my0Oevy7HxSEvSBLIVZB6UUtRLQAMtnvT9ULs0ZmHL4HufuJOz3Xwan6rPv7/6qdLH2tQeyB4bomKzEbiI= X-Received: by 2002:a17:906:7955:b0:ac1:e00c:a566 with SMTP id a640c23a62f3a-ac738bbe6e9mr1868178566b.45.1743660456051; Wed, 02 Apr 2025 23:07:36 -0700 (PDT) MIME-Version: 1.0 References: <20250310-dmem-cgroups-v1-0-2984c1bc9312@kernel.org> <20250310-eccentric-wonderful-puffin-ddbb26@houat> In-Reply-To: From: Dave Airlie Date: Thu, 3 Apr 2025 16:07:24 +1000 X-Gm-Features: AQ5f1JqIs6baj7zESBDpkv7pR6DzQLKEPOII2QRNAbnnHufJyGkPQs5rKDbOWPQ Message-ID: Subject: Re: [PATCH RFC 00/12] dma: Enable dmem cgroup tracking To: =?UTF-8?Q?Christian_K=C3=B6nig?= Cc: Maxime Ripard , Andrew Morton , Marek Szyprowski , Robin Murphy , Sumit Semwal , Benjamin Gaignard , Brian Starkey , John Stultz , "T.J. Mercier" , Maarten Lankhorst , Thomas Zimmermann , Simona Vetter , Tomasz Figa , Mauro Carvalho Chehab , Ben Woodard , Hans Verkuil , Laurent Pinchart , linux-mm@kvack.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 0F3A71C0002 X-Stat-Signature: 3c9cwc8jpues83axbg4wis4cmwpzjn7r X-HE-Tag: 1743660457-10294 X-HE-Meta: U2FsdGVkX1+/zqPOuzhCYfZgbMWbrfMwR+qZbbfdyk1J1XXRNPFpXqpcQL81a2zemjhWxSZtEt/OMbAeR8gr1btvz//DNOUjDmjEHtKm+W3bF/CkLC/3EqtQHXnicRMC5m3jM58SUaJ+cn8rcZZA66nDEPhPi2Cvj3otFenun52/3bGMfIxhQ15CEJoqk8/KysrTC7+52xJBG1EJu0sNU8xRb5MVDXYtVOdGaqfs2fe/K2Up6/jX/43zqip5byMioP50kWhdNbErMkXEYJJvLRb9H60+X3zWPd0Mh7aOLX7dMLqP0h0/GB0u2lBHpUwtq1By7wgY7dDR/yIJuYDWHggXLtCUbCWxmUsVmJ0WlM7+n6b62KIproydXVCZZhU+ec5Zcfbr48DdcyK9woadyrrv0q/VZHZ+1WnvVjnRpGRrZpICgEtqqTFvkX2SLbY3kKeoTyaSGF0rzH/RWc+RtwISaU+0d4ULjSTy6LWqBl3bdtBAzPDEr71uwfkb9btqwTE1UCQn8JUyQwRlgt5NZtTU3xOgy3oEtEynR9Knbe1Yw6jXmxgNWqdFyJ4aLmRNx1XlotdYL8P3w7EBeYvvry2Gb8jozYRBm27KeW2dYxvpM979ltK2rh7DaXyce459g5R9TNBezy5dMo9KM7QSGJygQR6286qcJti438LR32xjrkEoDI3fuhf4XA+t04JytJN87FUuQ03LtZIa/Fvsonn3FwtYw4OIAd62tSUvX2sACX7m7h1uYetJ4HE2cpv1Xasi0BdxqAJfQYhTO/N5WM43jpy0IPyyaP0LYCMn/UIa02tZ11PZha7mz9wRbV/dAcghwh1IQPgqzgpS1w3Zz4lBEJUXH7/BYEYbY5eeeEbTC7zMMPgRqo6Yw/PJVseiPnLPVfF1bDR1irVedasTT1KpTPRETFMtp4iwFSIp66D2oICV3fSBAyU6nlHY0E8LZhkGhXdHd6ypzeMK715 pggemdwF 2dsZEwA5EZ9MwGUwWxehtjsgpNAY0wU+4/kKYuIrtbRtAXIwNkaRbJBi+NmcuQbnf2m+epFH796jtfh8LOXRiNd0GBOv6cxfL9JR17vgoc1pcom3sKS++9Fn4MKvEV8BzdjHMNJs/jK0IfZVJyJcK1CF5Hn74pNqjyQTetZE5ay856IV5jHyPRoEtSAZVFD3p35+1bbdJFpCLB36g07+sCOmBfJhauTtvrB8DOnq1Q/sT4z/888m5WraOgxOgs8PAUa50HdzdstauBwBqfC7ctW1FxpYTTydb0mfsq33BA9e+SG7HdPidnqTo0bHrHgdfFm+9p793AHztML6aqgQoog0INLoLhuWenQUJXy8w2BqvW67kOEioNTpM7NyfaHdoeW5VHhrcy6zLRMsPh8+1aurq2dVY5YEpmGT18cpSegPO7WEAeCBTWRjhNAlokamcNo/H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 1 Apr 2025 at 21:03, Christian K=C3=B6nig wrote: > > Am 31.03.25 um 22:43 schrieb Dave Airlie: > > On Tue, 11 Mar 2025 at 00:26, Maxime Ripard wrote: > >> Hi, > >> > >> On Mon, Mar 10, 2025 at 03:16:53PM +0100, Christian K=C3=B6nig wrote: > >>> [Adding Ben since we are currently in the middle of a discussion > >>> regarding exactly that problem] > >>> > >>> Just for my understanding before I deep dive into the code: This uses > >>> a separate dmem cgroup and does not account against memcg, don't it? > >> Yes. The main rationale being that it doesn't always make sense to > >> register against memcg: a lot of devices are going to allocate from > >> dedicated chunks of memory that are either carved out from the main > >> memory allocator, or not under Linux supervision at all. > >> > >> And if there's no way to make it consistent across drivers, it's not t= he > >> right tool. > >> > > While I agree on that, if a user can cause a device driver to allocate > > memory that is also memory that memcg accounts, then we have to > > interface with memcg to account that memory. > > This assumes that memcg should be in control of device driver allocated m= emory. Which in some cases is intentionally not done. > > E.g. a server application which allocates buffers on behalves of clients = gets a nice deny of service problem if we suddenly start to account those b= uffers. Yes we definitely need the ability to transfer an allocation between cgroups for this case. > > That was one of the reasons why my OOM killer improvement patches never l= anded (e.g. you could trivially kill X/Wayland or systemd with that). > > > The pathological case would be a single application wanting to use 90% > > of RAM for device allocations, freeing it all, then using 90% of RAM > > for normal usage. How to create a policy that would allow that with > > dmem and memcg is difficult, since if you say you can do 90% on both > > then the user can easily OOM the system. > > Yeah, completely agree. > > That's why the GTT size limit we already have per device and the global 5= 0% TTM limit doesn't work as expected. People also didn't liked those limit= s and because of that we even have flags to circumvent them, see AMDGPU_GEM= _CREATE_PREEMPTIBLE and TTM_TT_FLAG_EXTERNAL. > > Another problem is when and to which process we account things when evict= ion happens? For example process A wants to use VRAM that process B current= ly occupies. In this case we would give both processes a mix of VRAM and sy= stem memory, but how do we account that? > > If we account to process B then it can be that process A fails because of= process Bs memcg limit. This creates a situation which is absolutely not t= raceable for a system administrator. > > But process A never asked for system memory in the first place, so we can= 't account the memory to it either or otherwise we make the process respons= ible for things it didn't do. > > There are good argument for all solutions and there are a couple of block= s which rule out one solution or another for a certain use case. To summari= ze I think the whole situation is a complete mess. > > Maybe there is not this one solution and we need to make it somehow confi= gurable? My feeling is that we can't solve the VRAM eviction problem super effectively, but it's also probably not going to be a major common case, I don't think we should double account memcg/dmem just in case we have to evict all of a users dmem at some point, maybe if there was some kind of soft memcg limit we could add as an accounting but not enforced overhead it might be useful to track evictions, but yes we can't have A allocating memory causing B to fall over because we evict memory into it's memcg space and it fails to allocate the next time it tries, or having A fail in that case. For the UMA GPU case where there is no device memory or eviction problem, perhaps a configurable option to just say account memory in memcg for all allocations done by this process, and state yes you can work around it with allocation servers or whatever but the behaviour for well behaved things is at least somewhat defined. Dave.