From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89A42C48291 for ; Fri, 2 Feb 2024 18:27:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E74606B007D; Fri, 2 Feb 2024 13:27:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E25706B007E; Fri, 2 Feb 2024 13:27:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D13466B0080; Fri, 2 Feb 2024 13:27:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BF1FA6B007D for ; Fri, 2 Feb 2024 13:27:56 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 925A1406CE for ; Fri, 2 Feb 2024 18:27:55 +0000 (UTC) X-FDA: 81747697710.14.29A1A1F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf18.hostedemail.com (Postfix) with ESMTP id CEDC21C001A for ; Fri, 2 Feb 2024 18:27:53 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706898474; a=rsa-sha256; cv=none; b=d14ZdAR7jRlwyrbD/mkrfNx8tCVEGW7dd0Kczfx3ccDConDmucaPn6lFjh3UkVf2LPqu6W woH8OY3wB9xeHxxx0y4/K/xvuPffpnkSjb4nalIHhmrRTp8ns7kri76FENRx49Ho9FwEYJ XvcJ4l7c4RrCdxjI72Y0EK6OYiAsNK4= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706898474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SxZJr2PAVBagnsZVWxwOs/B16EOYBY5Zxf2VWIcsCqE=; b=FGddMsz0HACkE/4ctrB94CDRAy2wM58c92lpXkYjy866ByXWk3b+nemM/qIUne/EB2FrOc LdVWlH+Ay9xs2b8osBRal+B5sP9XLn846zeMkeRnfg+rEiTJU4WpQ//1C35/lTZeDAOPBW i59biKSN5tcU/Pu5+vQ1Uv5l7nMhkUo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4313FDA7; Fri, 2 Feb 2024 10:28:35 -0800 (PST) Received: from [10.1.196.40] (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D24AD3F762; Fri, 2 Feb 2024 10:27:51 -0800 (PST) Message-ID: Date: Fri, 2 Feb 2024 18:27:50 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] iommu/iova: use named kmem_cache for iova magazines Content-Language: en-GB To: Pasha Tatashin Cc: joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com References: <20240201193014.2785570-1-tatashin@google.com> <02610629-05ef-4956-a122-36b6ac98fbc2@arm.com> <84c7e816-f749-48d8-a429-8b0ef799cdbb@arm.com> From: Robin Murphy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CEDC21C001A X-Stat-Signature: f8wirnnsfpwwb11s6wxfxaiziz4ho4hf X-HE-Tag: 1706898473-51829 X-HE-Meta: U2FsdGVkX1823JH1KP+butGc9iXz33e2wVLp0gJYszEfyN0Kj3sE30vpirSWBVukkmSIMojOji9jyuAKS8G+zAweSLLBPbm0gSCamQLvK3q1SG9qEioI0jXmFh0uCL5OkpzVeGLZGTtuld7XniUNI2b5jF4dMvNtmOHSN/eY/8vTdzSpSBnFIh/V8WmyQtSVJGSuT4ZdSQ1vaqvrBO+WTJWRa8l6fjOx2KjoSwfqV9mlHokYm+1RcVvE1qRfGdo74HFaW3EsOYy6ymfk8Q4Fr4QIqPx3zdnwbrqp2GGDrl7S1FbzpwnQ7ozhl9ADkmxn6H1olXvK4KZJfodJp5WMEJgU9m8vpUiXI0CDK+2/CxXI4MNZHQtxbqbHYl98zbV9p6g0pmNqqzyIJZcfKS5r5SqbDq4zj/eXbJrSV4GTWL6iYVD3f7hdDf+/XcN1Pq5rVnvz0IuDw9sTWa5OXq2j4U0ooo69cJOkixkqLBf7VVIbzplwLECTAAPQGIvM2XwybRY7XlX79qVtO2mggygMLkYMptcyzAOOMdDOzy8Y1gn1th43Hqoyg4qSHEUJQQ7H97yTtD3MvXQxFpWBAkKIb8K11PS8XAtW15jc9LF0kkvOg+H26L6bjEfDhWrWNQirtkkFDPArCuOWUiuPBW9/ZNq6TkaI+vlpj/CVz9UluMdAS6ROlc69EZfb0f1oflFy26wPsRogIEM73uAWq4jBIVnAhDRMpihgWdHxUHkcOJjDV2V9QMndbIvya3DSHvgr8FoWWSCgv3ZCpllXXoXW6BMkIwR3io9a75Ude7viO3sOx70to6Uv1adVj6HstPvKOK8iVkWiw6/ZcQlV1Y3nJ/Cmaa368BnBC5trdfVBtFz1U7zRLzIkDIRfhibiLHMNdOguW1sOB2qTe5pz2Z29S7ngG+RxWqw3bTW8HyKW3ldm5KYgXNX+31UZOcvmV6N8+FXY+34lu6qqSvP9Z6u rcZ9oLzd yfIaEEJFWD7w9xVrj3HOPFWGU6CLch42+y4y2p9FonKwhhXVb4agcmnkhb/b+nq3sS6o2SvXYaGM2YaD1qytb3rXGy4yWBipApf6t8IJBYU/Bwpwpv0ZxFGCHnIRc57YS8lTS2acczYlQFGRnspxutmf8/dtLxEULlIZ9wT3pulw4j0DxIgfr/PaRXCXPgRso/R5y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 02/02/2024 6:04 pm, Pasha Tatashin wrote: >> Hmm, I did misspeak slightly (it's late and I really should have left >> this for tomorrow...) - that's 12KB per CPU *per domain*, but still that >> would seem to imply well over 100 domains if you have 242MB of magazine >> allocations while the iommu_iova cache isn't even on the charts... what >> the heck is that driver doing? >> >> (I don't necessarily disagree with the spirit of the patch BTW, I just >> really want to understand the situation that prompted it, and make sure >> we don't actually have a subtle leak somewhere.) > > Hi Robin, > > The following tracing is without Google TPU, simply upstream kernel: > > The iova_domain_init_rcaches is called 159 with the following stack: > > iova_domain_init_rcaches > iommu_setup_dma_ops > amd_iommu_probe_finalize > bus_iommu_probe > iommu_device_register > iommu_init_pci > amd_iommu_init_pci > state_next > iommu_go_to_state > amd_iommu_init > pci_iommu_init > do_one_initcall > > Each time 1536K is allocated: in total 159 * 1536K = 238.5M Yikes, so it really does just have that many IOMMU groups? OK, fair enough, call me convinced :) On balance though, I think I'd prefer to just stick the lifecycle management into iova_cache_{get,put} for simplicity - spending ~256 bytes on another kmem_cache we might not use can hardly be significantly more than the extra code and static data necessary to track its usage separately anyway. Thanks, Robin. > The allocation happens like this: > for (IOVA_RANGE_CACHE_MAX_SIZE) > for_each_possible_cpu() > iova_magazine_alloc > iova_magazine_alloc > > IOVA_RANGE_CACHE_MAX_SIZE = 6 > ncpu = 128 > sizeof (struct iova_magazine) = 1K > > 6 * 128 * (1K + 1K) = 1536K > > Pasha