From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F30E5C4167B for ; Tue, 28 Nov 2023 21:34:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78CF16B034F; Tue, 28 Nov 2023 16:34:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 715AA6B0355; Tue, 28 Nov 2023 16:34:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 590216B0358; Tue, 28 Nov 2023 16:34:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 436DB6B034F for ; Tue, 28 Nov 2023 16:34:38 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0E261A00B7 for ; Tue, 28 Nov 2023 21:34:37 +0000 (UTC) X-FDA: 81508667436.13.9E72FE1 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by imf20.hostedemail.com (Postfix) with ESMTP id 295AC1C0004 for ; Tue, 28 Nov 2023 21:34:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NLuf0x1a; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.48 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701207276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8NXFzGfhAzOfJO06Gf5RsXFoYKrXmtsSvTLzk7MbcCk=; b=8QDQrOqcp9twAjtSH1qCQ2Q2CCK8W1b7QYJIr94E9IMfHXhb1VEkxRiuXKML/EWYri95RD CjdlkUH8vausV0bK9ZSb4NA4SisdZsq0mkwcu3UTh0hCjJghSXPp5WkJeFR7uDP+KmG3+I t5zPT9bdoENpm73G9qoBl304QVO+xxg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NLuf0x1a; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.48 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701207276; a=rsa-sha256; cv=none; b=rjvu1N3AGvKM281Wyl9fmGpizbxff8cVVKFtuF9XYd0CGi7ln/RlFCkJctfFtuxB+Upt9N /2Po6uMUnC/glEtd8kBFjdAWpCQIhYOS0m1G7Ht7OkAJBmJhV3ODn4FsdGDq3xdVVVQ29L qtHvHk01CYtNp1G0JWfakrdjZX3kx6g= Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a0b7f793b8aso488840666b.2 for ; Tue, 28 Nov 2023 13:34:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701207274; x=1701812074; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8NXFzGfhAzOfJO06Gf5RsXFoYKrXmtsSvTLzk7MbcCk=; b=NLuf0x1aSxsYz9sACSW4P0EqwN/QS5W8PayieHJiq3T9Hh5nOCf1uPrPomuWIMwY9N Ni0/T3pTCeDFRnqhtfcbgOGLTUv8BRwSi+BqoASwdVkI3CxK9ZwKHMdJkebhAp8LybkM XiIsIyRqRasWsBnAUC8uWnT5EzCdkuc9zdN6KbKt6LX7FIIi8HB30pG0WfulCoEHruIk iJS/J45p6D6oYSoP9LYqPhwhtepu3J563WgCw7RkDZwFPFaIprreFBHI03HytXenheQ4 kVe74buJYBW7bvsIzAdvqnMa5hKT/MRPw4E5a7IP9bMrcLWra5+8ri36hNdzZBqS/DwF 3FVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701207274; x=1701812074; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8NXFzGfhAzOfJO06Gf5RsXFoYKrXmtsSvTLzk7MbcCk=; b=TbC2Grm4elvbPCBwevZlAB8Tgd9dd3PWVOy8J9KBvc4b3txfp2fB6EsOOhlF4YjMYD yGXVldwwPxhxJzByv43H8qmbTWCiovq4zJ2knTb/c/waYZTSivjjGSmD+r/b+dmlzpXy UJsv25W7vWTb6PELkID7r54871/ORlTQxzDtN7Tf/7n0cqUekTtJ7p86JJa3KbXd2O2E eXTxV0x7Ie+U42lZUOLGUaD2NF2NQ28ZsMMe+a63R2C6UuiaB4aALlmUzEwmtBwyLEXr CZKx6F00Ca/qaVjtQSkR90sEZEy+Sr+eqfNAG5n4ZvPK2Yc2sSNQC5oEE30jzF3jvK4P nKTQ== X-Gm-Message-State: AOJu0YxLD40GDuj6vo4/rjCpI+8hxSZpeMQVG4T0uaPMQseIYBlrWfnb wD79IVixXbm6Iditrd9KDsbUys7KrO1sg25IB14Rqw== X-Google-Smtp-Source: AGHT+IFrrdaJl+8CDzGqIseL+lJ7M7KDt9GszVwJUOFBD/DHsSLfgFwP8+NKKfkfsTtyJyGUtnD8OJJylXlFpNkSb5g= X-Received: by 2002:a17:906:b248:b0:a04:cc0e:ff3b with SMTP id ce8-20020a170906b24800b00a04cc0eff3bmr12465188ejb.27.1701207274446; Tue, 28 Nov 2023 13:34:34 -0800 (PST) MIME-Version: 1.0 References: <20231128204938.1453583-1-pasha.tatashin@soleen.com> In-Reply-To: <20231128204938.1453583-1-pasha.tatashin@soleen.com> From: Yosry Ahmed Date: Tue, 28 Nov 2023 13:33:55 -0800 Message-ID: Subject: Re: [PATCH 00/16] IOMMU memory observability To: Pasha Tatashin Cc: akpm@linux-foundation.org, alex.williamson@redhat.com, alim.akhtar@samsung.com, alyssa@rosenzweig.io, asahi@lists.linux.dev, baolu.lu@linux.intel.com, bhelgaas@google.com, cgroups@vger.kernel.org, corbet@lwn.net, david@redhat.com, dwmw2@infradead.org, hannes@cmpxchg.org, heiko@sntech.de, iommu@lists.linux.dev, jasowang@redhat.com, jernej.skrabec@gmail.com, jgg@ziepe.ca, jonathanh@nvidia.com, joro@8bytes.org, kevin.tian@intel.com, krzysztof.kozlowski@linaro.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-rockchip@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, lizefan.x@bytedance.com, marcan@marcan.st, mhiramat@kernel.org, mst@redhat.com, m.szyprowski@samsung.com, netdev@vger.kernel.org, paulmck@kernel.org, rdunlap@infradead.org, robin.murphy@arm.com, samuel@sholland.org, suravee.suthikulpanit@amd.com, sven@svenpeter.dev, thierry.reding@gmail.com, tj@kernel.org, tomas.mudrunka@gmail.com, vdumpa@nvidia.com, virtualization@lists.linux.dev, wens@csie.org, will@kernel.org, yu-cheng.yu@intel.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 295AC1C0004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ud1yddp4d3u39wgas4ewkrgbduoip88c X-HE-Tag: 1701207275-350229 X-HE-Meta: U2FsdGVkX1+uGqq+R5sJa32YED+niOQc+l6mvKZEsY8LOiZYtohP2TKN+VXEPOZ5FchqL4qaVs0vuEGr2FwxG/xqBjfLF0aDY5+wav195WB8GFgCmLlWYVClJdaESUuxISPXGAC3LcXJ8a2zVQ7RF1oPShL6AHownOizGUL2uJeFfRmS5rJxhCzQD/6aFpAdnYMev7HoiR3r+dO/SlnaVqXz39iHnvjd43qFA0rVlXaQgtDKB9HN6NLnqmbVFMCPy1FaKFa1puqZNOdQeDiIH5DR4xYmC8DA3BR7E9z+owNy9FIDjsAB/3slEAeZBFTQT9uV8/2W25k0f72EvuZXJJtZQtbJtgAwkzVWPiXLARl0WJJNeTthKwNGTb4bqzdrW1OQ+U+vT/jf+tm4REsWe1w5SURESbvu81TjfFOdx4fZwdmXITYWACOEf8CGy1ZTGfBI/6XzLup9FalvEdimYu+G6ul+I54CUdoFKxb0dnv8t3NU5I1ZLPd0kq/Q99fhP5x+8tYmBIrsTYXtVP48ij+OF+/XyuIHF22DMJo83IihNAUrymLNWBHerCmon0WQZBvHKEdoInXvBj3I0eJNK6IQYDIfzjhdnT4OSEVOa+HS6Qoh7JAWr23bqxdhZsyw5Gb+3qKuIVqtqsqQBigZ3kPmXLQuKGzrg7AxbciUKMLHw3UnXSoIF3sUPyoxHGprSg3QbwNSUZFXDkDZ9GMkR5hePXnJ0YZQiC9APawQyq7zQuXE+WxUvRyxkIDBQtoGt9b36uJ3TQJ0ImiMpJz1Qvembva1xIo6iRiv48m0az1Ghcb+ZCV1x8v/sBk5FzsBI0JRcTYewUVAsQYy9TdqcT9v/IQkSGqcAtGyez+vqZhgYzPsx1l1D0LnEt6sPIikV4wOAtbskyid8w0TRd0ZEEaWRDf0lc0A7Q+Sn1Kd6H2tbQj9Lcw/PCFLQyTggfyrnKRUg9d7Z2tKEd2Brrl mLajPoYB FuIAiYNwj3TA1VC0kfhjsovAh3HMMfqiruDTygj5mvvbnzjC9wM+2HMpFkOn7ipVrFIQNe4l2OpVAC+1lplklZIMqJO28YkL76Ih9UkZsRNEwlftBSsMDk5R4xSXi1zPOOQfb7TKuGk8JAc5DGVQ/cLWuiWeYt2gLQtNxRPdldUNxDftf+5PXU333uZ0DuDGOfYNYaqCgg6ewSfbUKm3J5897gOZVRtAHKlLwlqr7+X9Mh8XiHDWwTWLs9ssuD80UB26tpuKiBKxEei2rYBCnR7jMi/i9lY0fVkqBojDhdhoN4W78HpTDaQ0spvpXCKEqCzsXY2ZAwx3XMVvU3YaVTVITN/Kk8Dv9LYqb3Mic6ba3mk8lXQ2HeF/ULpY7NtO85M0f7EKFoTORJ40= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Nov 28, 2023 at 12:49=E2=80=AFPM Pasha Tatashin wrote: > > From: Pasha Tatashin > > IOMMU subsystem may contain state that is in gigabytes. Majority of that > state is iommu page tables. Yet, there is currently, no way to observe > how much memory is actually used by the iommu subsystem. > > This patch series solves this problem by adding both observability to > all pages that are allocated by IOMMU, and also accountability, so > admins can limit the amount if via cgroups. > > The system-wide observability is using /proc/meminfo: > SecPageTables: 438176 kB > > Contains IOMMU and KVM memory. > > Per-node observability: > /sys/devices/system/node/nodeN/meminfo > Node N SecPageTables: 422204 kB > > Contains IOMMU and KVM memory memory in the given NUMA node. > > Per-node IOMMU only observability: > /sys/devices/system/node/nodeN/vmstat > nr_iommu_pages 105555 > > Contains number of pages IOMMU allocated in the given node. Does it make sense to have a KVM-only entry there as well? In that case, if SecPageTables in /proc/meminfo is found to be suspiciously high, it should be easy to tell which component is contributing most usage through vmstat. I understand that users can do the subtraction, but we wouldn't want userspace depending on that, in case a third class of "secondary" page tables emerges that we want to add to SecPageTables. The in-kernel implementation can do the subtraction for now if it makes sense though. > > Accountability: using sec_pagetables cgroup-v2 memory.stat entry. > > With the change, iova_stress[1] stops as limit is reached: > > # ./iova_stress > iova space: 0T free memory: 497G > iova space: 1T free memory: 495G > iova space: 2T free memory: 493G > iova space: 3T free memory: 491G > > stops as limit is reached. > > This series encorporates suggestions that came from the discussion > at LPC [2]. > > [1] https://github.com/soleen/iova_stress > [2] https://lpc.events/event/17/contributions/1466 > > Pasha Tatashin (16): > iommu/vt-d: add wrapper functions for page allocations > iommu/amd: use page allocation function provided by iommu-pages.h > iommu/io-pgtable-arm: use page allocation function provided by > iommu-pages.h > iommu/io-pgtable-dart: use page allocation function provided by > iommu-pages.h > iommu/io-pgtable-arm-v7s: use page allocation function provided by > iommu-pages.h > iommu/dma: use page allocation function provided by iommu-pages.h > iommu/exynos: use page allocation function provided by iommu-pages.h > iommu/fsl: use page allocation function provided by iommu-pages.h > iommu/iommufd: use page allocation function provided by iommu-pages.h > iommu/rockchip: use page allocation function provided by iommu-pages.h > iommu/sun50i: use page allocation function provided by iommu-pages.h > iommu/tegra-smmu: use page allocation function provided by > iommu-pages.h > iommu: observability of the IOMMU allocations > iommu: account IOMMU allocated memory > vhost-vdpa: account iommu allocations > vfio: account iommu allocations > > Documentation/admin-guide/cgroup-v2.rst | 2 +- > Documentation/filesystems/proc.rst | 4 +- > drivers/iommu/amd/amd_iommu.h | 8 - > drivers/iommu/amd/init.c | 91 +++++----- > drivers/iommu/amd/io_pgtable.c | 13 +- > drivers/iommu/amd/io_pgtable_v2.c | 20 +- > drivers/iommu/amd/iommu.c | 13 +- > drivers/iommu/dma-iommu.c | 8 +- > drivers/iommu/exynos-iommu.c | 14 +- > drivers/iommu/fsl_pamu.c | 5 +- > drivers/iommu/intel/dmar.c | 10 +- > drivers/iommu/intel/iommu.c | 47 ++--- > drivers/iommu/intel/iommu.h | 2 - > drivers/iommu/intel/irq_remapping.c | 10 +- > drivers/iommu/intel/pasid.c | 12 +- > drivers/iommu/intel/svm.c | 7 +- > drivers/iommu/io-pgtable-arm-v7s.c | 9 +- > drivers/iommu/io-pgtable-arm.c | 7 +- > drivers/iommu/io-pgtable-dart.c | 37 ++-- > drivers/iommu/iommu-pages.h | 231 ++++++++++++++++++++++++ > drivers/iommu/iommufd/iova_bitmap.c | 6 +- > drivers/iommu/rockchip-iommu.c | 14 +- > drivers/iommu/sun50i-iommu.c | 7 +- > drivers/iommu/tegra-smmu.c | 18 +- > drivers/vfio/vfio_iommu_type1.c | 8 +- > drivers/vhost/vdpa.c | 3 +- > include/linux/mmzone.h | 5 +- > mm/vmstat.c | 3 + > 28 files changed, 415 insertions(+), 199 deletions(-) > create mode 100644 drivers/iommu/iommu-pages.h > > -- > 2.43.0.rc2.451.g8631bc7472-goog > >