From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F36B9C48297 for ; Tue, 6 Feb 2024 11:32:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BCD56B006E; Tue, 6 Feb 2024 06:32:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66CAF6B0071; Tue, 6 Feb 2024 06:32:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55C846B0072; Tue, 6 Feb 2024 06:32:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 48A3F6B006E for ; Tue, 6 Feb 2024 06:32:23 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 10BFB1C159A for ; Tue, 6 Feb 2024 11:32:23 +0000 (UTC) X-FDA: 81761165766.27.29905F1 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 41E18A0014 for ; Tue, 6 Feb 2024 11:32:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707219141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qvng+wUUd4munMOPMeJKr7FDLMK3w1fkevuqxeojIuI=; b=zq+wUpQu5V74OU20yNcGSD0Kdch0RYNuWk2jKjIoa0CzE3WvSh4Nc67eYjJFaJTWzRGDOS GkGjFJ7crvzDg6/7Vd2+bQH94a1AxL17v6U0fium1iLJNHj27MJzcHkQimWcyOshg+b8J1 HQq3RTAXxaThEvbPhSd370tlLRpTboI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707219141; a=rsa-sha256; cv=none; b=WUoAUiaH5pu8IHBw5yI6OpJ15mppQOF0u270BzPvR5U68BQZ8Izc0y7fWvyJp88qecjqNM wkukMsDC61C4wLHaKprsq7dm1THedlRSybQhpICdGoIBuQfm32ovIxOqRUmAg+dNY2d+Sr 70UjQheL3jN/AMMohEMool208yASB8M= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C2CE51FB; Tue, 6 Feb 2024 03:33:02 -0800 (PST) Received: from [10.1.196.40] (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 306B03F762; Tue, 6 Feb 2024 03:32:19 -0800 (PST) Message-ID: <0d1f81a8-999e-459d-9e5c-7efff665646e@arm.com> Date: Tue, 6 Feb 2024 11:32:17 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 3/3] iommu/iova: use named kmem_cache for iova magazines Content-Language: en-GB To: John Garry , joro@8bytes.org Cc: will@kernel.org, pasha.tatashin@soleen.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com, yosryahmed@google.com References: <83a0aafc-8a9c-48ce-a203-96b901489d41@oracle.com> From: Robin Murphy In-Reply-To: <83a0aafc-8a9c-48ce-a203-96b901489d41@oracle.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 41E18A0014 X-Rspam-User: X-Stat-Signature: dukaerns7qr6e5w8zaum7g1ojko8zhqx X-Rspamd-Server: rspam01 X-HE-Tag: 1707219141-91231 X-HE-Meta: U2FsdGVkX19Lvt0y68yQWw78QJgzGcjNlgl+uGnYMoIXmWXYmcRYZ+jNxiETyg1eYsWEl9ks2PD+PF1wV4gZZZEZWdV31AxRlP+f+MvWu/gsG9XJ75+iylQCOQBRPtBmM90ZDd78k5ZWukAzDTo7IW1KP6TEPv6XY3VnzTuXw/smZ0lryApDVJy5MA6FmQ2UPnyuc6bbDJFiRonregJVLVA8m3V7EbUclHfXNpgaYTtk7ozSYmtC8jiDXEpZhtoYcUWHKFu5cZ7lu2QdYGEyXQaqy8Jm+AXuCAfHj1HAYc6RkvHdXW/fWLmtN9qsWFqZFoT9vdqkT4BdqvEnuXTZiSc8JrvdOETrF3W9Rc5vAyQMGJge6s1ObPlLa3lfTW/cXDdFK8g5k3FmyFNRiUUY9O9VpeQk6jMrMic7GSAV/b+E8EqNuxbsjlqdJLg4HXU46+Cg8AL9ZR8eNoEjFU24rEMIN7WxJkTnZsOYG0m/YngcfPdWABzfrabtxaW9NtFMk0pWHOMVwD9qE0f9T6NCvdNKlHJX27S5AE4xyhDVAuZY976SXoTklL/PbsaewM+un3iOKs72sLecHvIMqYUm4Dj5+MQKFHnsFJPZuLPl0HZlderzRE4HSX+A0iH+YtzWHGUZUmXRTGfLE3Gry5u/GPra+y4RPL3TnS7wEyeyAptkzImBbWEaaJAQehd5rBtHMDsgSDSsGRHM05P3M6gFZtQ32uDY3EPw1UA35Lmk1wOfniOzN1yJ5Tit/26/iMMFrclKBW+CE8xxle0r8Z7zaNSGm/EyxffNQa8WD7mLxunu7PvSIp4m6j3pZXml1K1Fuj7VuZuDz77WRoMWQ/NXz/LmJGkP5Ccf+OmoD2saIvorpBQXIepMTMKYknymOliN/40eN/dItWWP9ewSlEmIh+iox+qNs5ctG0kPS9ZMdDwOfJB3wbVGdSRuLu5xW03BJi62GZq/aXVngvEPlEI s+SawlZb xz4CZAnG6+emR41iTFlH9zMtq5LcYJ0cyj/rFCMf2TF9bsX/2pCOT4ogUZnaQVTIsKP50gIaklunVSw3hK5/wQRbsavrgde5SSZVhPCd8/m+pr4l7HtmAy1E3rUc9+VoxiNApsDJ8GqZXsHw1lBxkhaHMRGx/jepI471wIr9vAlAQyL/j1YJqV3CeXs1ELhLEH+s07mnmsyXNxc03ToJdLb0NRw+g74FHIs7zkpj2oeXM57r5dson2xd8sjwAusMzvkybHxSaPLNRorHB9Dlaefnk36q5VLvBWqJQVCzxbFIFDNEUGW0elmBaFwoaAriVV5OHRg4+R+tpHFXPUCgSE1ITECILcoaobbFf76DlavknvbXFc8TxHTCU0DxBLcAZurEkInvcJFYeO855arZjo89fnozHLObmwMLP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 06/02/2024 11:24 am, John Garry wrote: > On 05/02/2024 15:32, Robin Murphy wrote: >> From: Pasha Tatashin >> >> The magazine buffers can take gigabytes of kmem memory, dominating all >> other allocations. For observability purpose create named slab cache so >> the iova magazine memory overhead can be clearly observed. >> >> With this change: >> >>> slabtop -o | head >>   Active / Total Objects (% used)    : 869731 / 952904 (91.3%) >>   Active / Total Slabs (% used)      : 103411 / 103974 (99.5%) >>   Active / Total Caches (% used)     : 135 / 211 (64.0%) >>   Active / Total Size (% used)       : 395389.68K / 411430.20K (96.1%) >>   Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K >> >> OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME >> 244412 244239 99%    1.00K  61103       4    244412K iommu_iova_magazine >>   91636  88343 96%    0.03K    739     124      2956K kmalloc-32 >>   75744  74844 98%    0.12K   2367      32      9468K kernfs_node_cache >> >> On this machine it is now clear that magazine use 242M of kmem memory. > > Those caches could do with a trimming ... See the discussion on v1 for more details: https://lore.kernel.org/linux-iommu/20240201193014.2785570-1-tatashin@google.com/ but it seems this really is the idle baseline for lots of CPUs * lots of domains - if all those devices get going in anger it's likely that combined iova + iova_magazine usage truly will blow up into gigabytes. Cheers, Robin. > >> >> Acked-by: David Rientjes >> Signed-off-by: Pasha Tatashin >> [ rm: adjust to rework of iova_cache_{get,put} ] >> Signed-off-by: Robin Murphy >> --- > > FWIW: > Reviewed-by: John Garry