From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E3C4C48286 for ; Thu, 1 Feb 2024 22:29:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BDF46B007B; Thu, 1 Feb 2024 17:29:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06BB56B007D; Thu, 1 Feb 2024 17:29:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E75446B007E; Thu, 1 Feb 2024 17:29:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D49A26B007B for ; Thu, 1 Feb 2024 17:29:39 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9F6C61C10BB for ; Thu, 1 Feb 2024 22:29:39 +0000 (UTC) X-FDA: 81744678078.07.E2FF3ED Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by imf28.hostedemail.com (Postfix) with ESMTP id D17F1C000D for ; Thu, 1 Feb 2024 22:29:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Oq4FY3iz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706826577; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fGDKw/IQ4kdSzFRMSnxN+nzueWWpVC66gEX7pOJuy80=; b=DCExpiHeKhfVkPA2SzfhJsyNBpgep2IYVnDmC4svNvbVVItrzGjd3dxzpJ80XcXMVHqIEx osByvNMcQYgTZeqcfWd4IQOzyICaZT5XbvWpLPZiFhuSeXPh7tUMlnZFnYCz6V8qW/QDPG t/aGj1DbPgLhRjKaJEty/0SxhqyPT1Y= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Oq4FY3iz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706826577; a=rsa-sha256; cv=none; b=GUjoUBF84Dcv9vyN7jkonRLQGMNXLEElXOx/fQ1dbnD6cnqm3MkyVeCSl9NOsDrWO6by21 fZUJ+tNf3SdUz1epN28j0MQzgxFlRWQBbNB/8uR0BNq+Mt2CgeCgj9xzEfQjhIDVaxmIGr ZzwWpikhPLZo2sTWC8VwU1e8WunXUZQ= Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a354408e6bfso444360566b.1 for ; Thu, 01 Feb 2024 14:29:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706826576; x=1707431376; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=fGDKw/IQ4kdSzFRMSnxN+nzueWWpVC66gEX7pOJuy80=; b=Oq4FY3izgyYi8hOHSfWwWwvg0iTECGgO/7YEF5CODJeveCYER649HgsGYgqFrmMDgO qpp7NgBR2n8iElvd2lhg9xRz7ipZ3MY/AKjdPxrtchBB+amO4fHEZCispihPV6yB7RDy NZd3R6vc45fGYy7ljimnZlrPVnpEZpSLdxZmp+x0GwVVcr0EcLVI1Xmd9pqAT6f8Tiaq QTMNFTfwdA97cT08mEVS630VLUZ4P97eZsb9V+uBjkyFEVGENs9goyBgQ8QLd/5nAMCN wFN/VohM3D3OwgX8DIMqaHx3IYMceWI9PbncGmQhO/CIlwOlQnWSpRB26UvD1Jb/fToB wDDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706826576; x=1707431376; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fGDKw/IQ4kdSzFRMSnxN+nzueWWpVC66gEX7pOJuy80=; b=ZHXV24VjmxMoMJzqbAxVENFQRgps6kFznC180DXRIVrIdF3pzrS0BTz8+3uPg6+0UF Uuw9v4lyEm62qLfw8bvLrVwSgOcqnrIKNixZMvS/qXDSt3INVMo/VtcYBIoLxjC5pBtc 8kpH858a4Ab7moj4Pz0DFMJGbMsjK3Vr6/o4Wc0W9z+B6Hp131AVAyd3KMMsDYOuou+e pOPMTHqhHhM5sgsavEUfLOloTHvRVKBC5n3m+UjmTOwZ/f4Qw5DcMYfI3QDLdND49lAE nezrvWH2IcFnzTUJ70O/E1nJRrpICsumxlwEc9wOg0hkGM2mhxS/Bz51Ub1ZBt7kUgit jn2A== X-Gm-Message-State: AOJu0YyKyXF6JSCsSK/2mViPihQBZ9XCtL3+eb9Jj7doPUEpK7uPDEu+ WuQLiAVeskMBlGkBTZ8mHwp1s+MKRFiCwzxOKSp/5kTiqJTH139KtPncStD9iuPwzH7lD61iAU+ 2HHDA2UpNaZb3XNbPIJ/GQjC/drjqzqfG4t0p X-Google-Smtp-Source: AGHT+IEuhiZvixWFGymGW7m4qj1dTBRZUXAhAcduL9snVjgqkWJ4xtPr92++LvjpCa9nwtPiSV2wXnbN3Xjfiu/tP6M= X-Received: by 2002:a17:906:43:b0:a36:c8fa:45a2 with SMTP id 3-20020a170906004300b00a36c8fa45a2mr320163ejg.1.1706826576073; Thu, 01 Feb 2024 14:29:36 -0800 (PST) MIME-Version: 1.0 References: <20240201193014.2785570-1-tatashin@google.com> In-Reply-To: <20240201193014.2785570-1-tatashin@google.com> From: Yosry Ahmed Date: Thu, 1 Feb 2024 14:28:58 -0800 Message-ID: Subject: Re: [PATCH] iommu/iova: use named kmem_cache for iova magazines To: Pasha Tatashin Cc: robin.murphy@arm.com, joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: fy5ersii5okzu3oht4yo486tfpijt84m X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D17F1C000D X-HE-Tag: 1706826577-424604 X-HE-Meta: U2FsdGVkX19QezQhzg4JCMv6morGq2ooDCpxF0VRuDC3QAgsmOokIyeRvReuskX5rh/aZaMWbzaDvz3pZWhX9+MOob9staRgPU/yl1KFXEYwsSaKr7TgkOYcI5FaexHxmADhmBRedlVgwX8Ohl1a1Y/Diga9bKli+HqR8RT/S5q4l0naFCfjZwBFCdLTtvJ4pr7xb2mumJl9EBTfQuNKti3plTuvek0D6DIug2H4CxdF8FPM5m/umOVX8hFP3ZEpV9JSWkiZIl+/IfcOcqCi9GZqtMrRukDfd7Nj6qsfG5Sm1GdBQywk8STccHmiieIU8jttiHwyTg9x2VdqT+EjOTUwGlfKL2AKBEZ30VuNG3mhTRaDKh2xl6EMWQcQrO3vvBQNFT3Es9fNjl/v8IR8rASU1cY8404GxDkgoRiqRClKwKU4yL19ae5k5KrlJqDq5XziRKK5hZvdP/tbAaQyqhB1oQC5qESIzX/a1zuEivoISQX3M4OfvveXPQGnmZc67uP9VmlXK55RiH49h8Jk73/OtUiP83unR3SkIPCFIilBFEYGRl8C2FBqYr7RtKxOTHpBCFovd8tbnblJTt4Up++Zj/xsQH5gCm9FkdGLDzkbA/shfWgCGn3bA5frC6rr1uM0g3n38xF1davkDJWIFXd86jtMav/LLmOH7nVfGi6h8BLbOOleRO46OozbwyfYp9B2iQ7jBKVjFhIORmsUsko4hcbqL0YdqjsMAe5hPYxMBOHIrdE0utQOLAGFzfkNWHN/YQLk0p/PjOjDT4JY61w0CmqlpUE8H15lDQZIeBHXfnRZuBBxPIsdj6hhgw1yIhqUbispl3uH9pTUePT5yj1Z4MIRUI7UXy0BDkj0i/Z804eSFTTE8SZFfexlEaGPN9eBUqQV43zrV2aBDJjKWZ9lUQeB0Gjs8URR5nRRczE5/xRTeOUKqi4bHNb0YkZVFdeL02CkZZile8XlLHr J910iQr8 f3aTVo9SYd/LDNOLmCH1lZF4vRE0+KM2W7Ye3kJt1KnfBM41lYNQMF0QhZLtmgJW7fZMRmq+wGrhO+O7NiNTYNx9uoxqlyR8cDJca9azpYVCbdA2raqCuszEamdg/5FE1GMDnYTYwCjCIInwHXKbSk1+BvHoH0CW3U8J8FbLqqcn5z/ZwzF+bwS9pRNv/oJdXqJeFC4tuXWb7tbVnjOHSK/BR4fiCK/mdtiSRFBgNadAoHZMl5EaQ3qfjbNcDfIVl7M6SND66Uop/G9WwXL0H8aCJR9ARjwh8lTF5G96Shj5EOeSJpm3g5n/XTXjBiaBNpbW48a1foUU1Use1KUReOe6UcpBMMLzYSjpd0dIgb6AK6jU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Feb 1, 2024 at 11:30=E2=80=AFAM Pasha Tatashin wrote: > > From: Pasha Tatashin > > The magazine buffers can take gigabytes of kmem memory, dominating all > other allocations. For observability prurpose create named slab cache so > the iova magazine memory overhead can be clearly observed. > > With this change: > > > slabtop -o | head > Active / Total Objects (% used) : 869731 / 952904 (91.3%) > Active / Total Slabs (% used) : 103411 / 103974 (99.5%) > Active / Total Caches (% used) : 135 / 211 (64.0%) > Active / Total Size (% used) : 395389.68K / 411430.20K (96.1%) > Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > 244412 244239 99% 1.00K 61103 4 244412K iommu_iova_magazine > 91636 88343 96% 0.03K 739 124 2956K kmalloc-32 > 75744 74844 98% 0.12K 2367 32 9468K kernfs_node_cache > > On this machine it is now clear that magazine use 242M of kmem memory. > > Signed-off-by: Pasha Tatashin > --- > drivers/iommu/iova.c | 57 +++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 54 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c > index d30e453d0fb4..617bbc2b79f5 100644 > --- a/drivers/iommu/iova.c > +++ b/drivers/iommu/iova.c > @@ -630,6 +630,10 @@ EXPORT_SYMBOL_GPL(reserve_iova); > > #define IOVA_DEPOT_DELAY msecs_to_jiffies(100) > > +static struct kmem_cache *iova_magazine_cache; > +static unsigned int iova_magazine_cache_users; > +static DEFINE_MUTEX(iova_magazine_cache_mutex); > + > struct iova_magazine { > union { > unsigned long size; > @@ -654,11 +658,51 @@ struct iova_rcache { > struct delayed_work work; > }; > > +static int iova_magazine_cache_init(void) > +{ > + int ret =3D 0; > + > + mutex_lock(&iova_magazine_cache_mutex); > + > + iova_magazine_cache_users++; > + if (iova_magazine_cache_users > 1) > + goto out_unlock; > + > + iova_magazine_cache =3D kmem_cache_create("iommu_iova_magazine", > + sizeof(struct iova_magazi= ne), > + 0, SLAB_HWCACHE_ALIGN, NU= LL); Could this slab cache be merged with a compatible one in the slab code? If this happens, do we still get a separate entry in /proc/slabinfo? It may be useful to use SLAB_NO_MERGE if the purpose is to specifically have observability into this slab cache, but the comments above the flag make me think I may be misunderstanding it.