From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BFB6C4828D for ; Thu, 1 Feb 2024 21:23:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1269C6B007E; Thu, 1 Feb 2024 16:23:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D7116B0080; Thu, 1 Feb 2024 16:23:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F10876B0081; Thu, 1 Feb 2024 16:23:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DEBBA6B007E for ; Thu, 1 Feb 2024 16:23:48 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5E2C5A20AA for ; Thu, 1 Feb 2024 21:23:48 +0000 (UTC) X-FDA: 81744512136.14.4C11560 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 949E81C0012 for ; Thu, 1 Feb 2024 21:23:46 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706822626; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KxQlnh0jYq7Lajb11kmnVhJocOnz71q97DEBawJfyPE=; b=G8paKz+whY4zkqxpE3p8Ao/wSkjeRzAiLSu1JYeoOGUGsD8r6/OkxhFC7L2BIRguReX9Oo Iwx2MhtO303Iz7EQtMnLVyj0Wc6O9gz/vCk5/660/WKJVFJqNbU77A57gJmGcsDEIx1z4v 0vgVuNT59MXZzbuoIiqac0g/DMJwD8c= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706822626; a=rsa-sha256; cv=none; b=w73bL5K7YwGteEqUzPb21PVm04V5jF3kplBKtmt1L/YOwIdtSKrxbnX8ukndauLtLACOa2 3L1n1Ok8SFcUu5qZPQV4OJuObL6b2soUxLFx9Bnt8RWEDSfTGY/YnDpz9KflDfI5wGUY/I O5bpbxUF3Ih2RvLGzReu6bKinmdHFzQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1AE03DA7; Thu, 1 Feb 2024 13:24:28 -0800 (PST) Received: from [10.57.47.101] (unknown [10.57.47.101]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 311563F762; Thu, 1 Feb 2024 13:23:44 -0800 (PST) Message-ID: <84c7e816-f749-48d8-a429-8b0ef799cdbb@arm.com> Date: Thu, 1 Feb 2024 21:23:42 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] iommu/iova: use named kmem_cache for iova magazines Content-Language: en-GB To: Pasha Tatashin Cc: joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com References: <20240201193014.2785570-1-tatashin@google.com> <02610629-05ef-4956-a122-36b6ac98fbc2@arm.com> From: Robin Murphy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: a1bxijjwjs7kd639tgokochjs7bcupux X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 949E81C0012 X-HE-Tag: 1706822626-317363 X-HE-Meta: U2FsdGVkX1/Ysx6D8dQAioM5EUEXFawoTe4QU+HIK1gQ69X3Ymsl+BkdLsoaTs+C5ZVPiLzsQJn6Wr75d7beiBdRzBTnJ1pPef+DSg59qlftrUbPynEuvUK/jcGRLnqnUrf3FXam7iE2c+oKwazNQ+pfGnvqZtDC2DK11yqOsDr9kdlcn0Ob9f9BBzRmhYt6O+YYtY+B90fuFkFR0fBA1r1iwvSZuDg6iSaLG0REqFHtYdhdtrzTrMKvHuha6ooURQBpvzlWSjvtotjsktXgWTGdzqPs5WvXq31O1ZuO4mwRxZq9UlyXaYuaaV0TMsc592fcFxmc9/Td+eCp+/tUp6PEEE4RPMKd0TMnKgruyjBcpRkI0CFqIqTzyBx1ClZpp1Mnnysue4PWNxdcXmwrbndOUQRbDLGL7XeJ5Wf+ZVccdQCQO2HIrw2DKvUD9McWj6OoAwPnUZZ0MraUzHtX6+0LcHnwYBp+Q7dKm0kEjVu9IyPFFMky+F/1PEuDCONxuRhTxjvQA1woiMmL3MP1uRyXWqU1gXMVK1W7/e59g7vnYFHaUiJVxdL6AiHPFI3rbSwGgg4qQcz9Q3Sg02pDMxgDCaHieJ5SEqFGe3CNhtn1hrjg+2+pnmDbgX8ppciudX6TYHF6fX2rDmhsZIubjjSTROK/BSxPsWzMrKiKijZyrVfxAG7qHPT7lgvUjkRLjYkmmPEQdJxuMpi6YrlsDjsymiWezaQQe5LcsxJbSqr8ShhwY+rByaw6l8DWMlKGoAhtdyigq/QwEvb/BYSc17TB4bdQy03jnZZKXMR+18moly28m96zXLVsXA3H0/EyW5YqNwztPyiLmvHkiQDJqSzQBXBJLUW1FnsrXucvoBE0dB12N2szSTerA+cWd2XK/OZUxJRVPyqz0PAZN9Fyd9hWbT5DPjwgfbY7quZvwfgkfWybqz247Aewm2wgw9Jq3LjZXG3+Ae/DN9sy4Np sq0VxuWY l/Wy4qaAApaaIVkwpft1uWgAPr9t7o70wYI3NC7ukWXOtG27KrTEqPrA0W3V8xa6i2sDAQEgjfHV3zHRPh/OfqqaPJ3RizaQLRArqmf1LtEaeXZoxlqmvSnVpXfld0Lml6KHi6vC0ouIK0XxmY/Ui91mn9efJBjs4Sf09bgiLV1gyeSkq8LNpYKrO5cgbynXKYD4Bb9KbQ9NiHbg3xPtYwLWeqMzjVpdX+90p2ln6EBOzl5FkJQ1gdkbSX+rfDA0cI1hKxNHgB1cCSS6K5L2SGfLom7BFSEhVZVLCvCYu6lJPi2LaGhfq9EGKNv/L3rMZemlnf7O5SpoYJ7Fxwqb3FV73ng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-02-01 9:06 pm, Pasha Tatashin wrote: > On Thu, Feb 1, 2024 at 3:56 PM Robin Murphy wrote: >> >> On 2024-02-01 7:30 pm, Pasha Tatashin wrote: >>> From: Pasha Tatashin >>> >>> The magazine buffers can take gigabytes of kmem memory, dominating all >>> other allocations. For observability prurpose create named slab cache so >>> the iova magazine memory overhead can be clearly observed. >>> >>> With this change: >>> >>>> slabtop -o | head >>> Active / Total Objects (% used) : 869731 / 952904 (91.3%) >>> Active / Total Slabs (% used) : 103411 / 103974 (99.5%) >>> Active / Total Caches (% used) : 135 / 211 (64.0%) >>> Active / Total Size (% used) : 395389.68K / 411430.20K (96.1%) >>> Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K >>> >>> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME >>> 244412 244239 99% 1.00K 61103 4 244412K iommu_iova_magazine >>> 91636 88343 96% 0.03K 739 124 2956K kmalloc-32 >>> 75744 74844 98% 0.12K 2367 32 9468K kernfs_node_cache >>> >>> On this machine it is now clear that magazine use 242M of kmem memory. >> >> Hmm, something smells there... >> >> In the "worst" case there should be a maximum of 6 * 2 * >> num_online_cpus() empty magazines in the iova_cpu_rcache structures, >> i.e., 12KB per CPU. Under normal use those will contain at least some >> PFNs, but mainly every additional magazine stored in a depot is full >> with 127 PFNs, and each one of those PFNs is backed by a 40-byte struct >> iova, i.e. ~5KB per 1KB magazine. Unless that machine has many thousands >> of CPUs, if iova_magazine allocations are the top consumer of memory >> then something's gone wrong. > > This is an upstream kernel + few drivers that is booted on AMD EPYC, > with 128 CPUs. > > It has allocations stacks like these: > init_iova_domain+0x1ed/0x230 iommu_setup_dma_ops+0xf8/0x4b0 > amd_iommu_probe_finalize. > And also init_iova_domain() calls for Google's TPU drivers 242M is > actually not that much, compared to the size of the system. Hmm, I did misspeak slightly (it's late and I really should have left this for tomorrow...) - that's 12KB per CPU *per domain*, but still that would seem to imply well over 100 domains if you have 242MB of magazine allocations while the iommu_iova cache isn't even on the charts... what the heck is that driver doing? (I don't necessarily disagree with the spirit of the patch BTW, I just really want to understand the situation that prompted it, and make sure we don't actually have a subtle leak somewhere.) Thanks, Robin. > > Pasha > >> >> Thanks, >> Robin. >> >>> Signed-off-by: Pasha Tatashin >>> --- >>> drivers/iommu/iova.c | 57 +++++++++++++++++++++++++++++++++++++++++--- >>> 1 file changed, 54 insertions(+), 3 deletions(-) >>> >>> diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c >>> index d30e453d0fb4..617bbc2b79f5 100644 >>> --- a/drivers/iommu/iova.c >>> +++ b/drivers/iommu/iova.c >>> @@ -630,6 +630,10 @@ EXPORT_SYMBOL_GPL(reserve_iova); >>> >>> #define IOVA_DEPOT_DELAY msecs_to_jiffies(100) >>> >>> +static struct kmem_cache *iova_magazine_cache; >>> +static unsigned int iova_magazine_cache_users; >>> +static DEFINE_MUTEX(iova_magazine_cache_mutex); >>> + >>> struct iova_magazine { >>> union { >>> unsigned long size; >>> @@ -654,11 +658,51 @@ struct iova_rcache { >>> struct delayed_work work; >>> }; >>> >>> +static int iova_magazine_cache_init(void) >>> +{ >>> + int ret = 0; >>> + >>> + mutex_lock(&iova_magazine_cache_mutex); >>> + >>> + iova_magazine_cache_users++; >>> + if (iova_magazine_cache_users > 1) >>> + goto out_unlock; >>> + >>> + iova_magazine_cache = kmem_cache_create("iommu_iova_magazine", >>> + sizeof(struct iova_magazine), >>> + 0, SLAB_HWCACHE_ALIGN, NULL); >>> + >>> + if (!iova_magazine_cache) { >>> + pr_err("Couldn't create iova magazine cache\n"); >>> + ret = -ENOMEM; >>> + } >>> + >>> +out_unlock: >>> + mutex_unlock(&iova_magazine_cache_mutex); >>> + >>> + return ret; >>> +} >>> + >>> +static void iova_magazine_cache_fini(void) >>> +{ >>> + mutex_lock(&iova_magazine_cache_mutex); >>> + >>> + if (WARN_ON(!iova_magazine_cache_users)) >>> + goto out_unlock; >>> + >>> + iova_magazine_cache_users--; >>> + if (!iova_magazine_cache_users) >>> + kmem_cache_destroy(iova_magazine_cache); >>> + >>> +out_unlock: >>> + mutex_unlock(&iova_magazine_cache_mutex); >>> +} >>> + >>> static struct iova_magazine *iova_magazine_alloc(gfp_t flags) >>> { >>> struct iova_magazine *mag; >>> >>> - mag = kmalloc(sizeof(*mag), flags); >>> + mag = kmem_cache_alloc(iova_magazine_cache, flags); >>> if (mag) >>> mag->size = 0; >>> >>> @@ -667,7 +711,7 @@ static struct iova_magazine *iova_magazine_alloc(gfp_t flags) >>> >>> static void iova_magazine_free(struct iova_magazine *mag) >>> { >>> - kfree(mag); >>> + kmem_cache_free(iova_magazine_cache, mag); >>> } >>> >>> static void >>> @@ -766,11 +810,17 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) >>> unsigned int cpu; >>> int i, ret; >>> >>> + ret = iova_magazine_cache_init(); >>> + if (ret) >>> + return -ENOMEM; >>> + >>> iovad->rcaches = kcalloc(IOVA_RANGE_CACHE_MAX_SIZE, >>> sizeof(struct iova_rcache), >>> GFP_KERNEL); >>> - if (!iovad->rcaches) >>> + if (!iovad->rcaches) { >>> + iova_magazine_cache_fini(); >>> return -ENOMEM; >>> + } >>> >>> for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { >>> struct iova_cpu_rcache *cpu_rcache; >>> @@ -948,6 +998,7 @@ static void free_iova_rcaches(struct iova_domain *iovad) >>> >>> kfree(iovad->rcaches); >>> iovad->rcaches = NULL; >>> + iova_magazine_cache_fini(); >>> } >>> >>> /*