From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 434A0C4828F for ; Fri, 2 Feb 2024 18:04:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD56B6B007D; Fri, 2 Feb 2024 13:04:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C850D6B007E; Fri, 2 Feb 2024 13:04:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B73D36B0080; Fri, 2 Feb 2024 13:04:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id ABF2F6B007D for ; Fri, 2 Feb 2024 13:04:47 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 324201C1986 for ; Fri, 2 Feb 2024 18:04:47 +0000 (UTC) X-FDA: 81747639414.09.5B261B2 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf28.hostedemail.com (Postfix) with ESMTP id 43F83C0019 for ; Fri, 2 Feb 2024 18:04:45 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=nEJQdTyJ; spf=none (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.160.170) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706897085; a=rsa-sha256; cv=none; b=n1ePwxACCtjoEynRgLB3kh0imzcFb5+VRl1vEBXw36KxemroDM7Qj+0IMdf3ebAoSdqirY kSHjpR9UmANe1j2u6eDstYK3ds4AD9GBmnBiKKB9FPtLq/emZ6chkJz85tW35LuudUmUt8 4oHo0Nt2be9syvUXDJL5vEDRaXx57ss= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=nEJQdTyJ; spf=none (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.160.170) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706897085; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9bijSb9yEaTmlnOsnr5FnwiHhV7CJXfrStAeun3CCBg=; b=k7qAwdTnwfQ3zq3qCr9CKnuLUJI09QESnfntJHZJ7gfR38Y1gTj1DZjFEAH5MlAKQHhtC7 zYFkfjnoKT0zfpXuawWAh9ddLY5+Jd+Vi0lcwfeysEB7pGRRZRo1Lx+7IPT1IhKp34MDdX cWq++haB5KZbmA3txvc1BvS0WNgu4L0= Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-42a35c720b8so9991021cf.3 for ; Fri, 02 Feb 2024 10:04:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; t=1706897084; x=1707501884; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9bijSb9yEaTmlnOsnr5FnwiHhV7CJXfrStAeun3CCBg=; b=nEJQdTyJUO/0NMRRTVda4+eyuX+iMlGAaOFy3uTnHvk0WEFDLHjBQvFD2vEuC4DjdM ndDPdxrvyAAqeogYTaICpkEtZrfjRsmHw/CmGOyrB2chY+On2QHLARjlEjj4AayL8roR IX3zH/pxOdpQBpB1zRZt+CTvP4OVIf2osnrgFBAq+OJ1QZ4OqChoub8El7lkSS+qq+zw T3jwuIXFzb7lN+IygGyLM8X0zL9huhElazh/pwr5NgoGZDyE7PKLHfvClhrFznLI82j6 UGCKPtF4WOeDjFZR4gGrfUF/Vff+yfO9TcNPR4xrBHlVvZKiagkJArC2dYL4D+7wt8oS k9FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706897084; x=1707501884; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9bijSb9yEaTmlnOsnr5FnwiHhV7CJXfrStAeun3CCBg=; b=GIsldhuEOUlA21yyBbedqHc12ql/y27G5/Hg78TfRro0hnsQphr7Mc/PA2h66FlSUS ipA8vetOS604bu9XpSp/o+gpfy89faiAn4gk5M4hq4p6Li+EfNX+ceOB0Rj8waw75jPT bP3BpmHPBURspwImXSIa5O9fMIekd0sy/5uG1z4IpQVAF7CGZOkps97vHMZNFnOl47lM I7w/2fQCaPf48B+FuoQkyfI9EqHKhnv4dF/Mhvjxmu/VmYYmRsCkB+t8lYIJP3ePPsS/ sU61FNh/mcFW8XvcpND9XTlKkTBSjwIXpNvMokPni1t+oZOWjqzw11zSZjJWIMCPoSFD ChXw== X-Gm-Message-State: AOJu0YxqQOM5nb21QSryn59TRrZ39pEP2L8gTlud5IT2fyVxJFygPgXl KXg9SqbBOmNRtYqwTFRv/JXViqUk2i1DSQ86Ahsy+t7zwt61iRDQ5YbFh1lBzpKaDjW3TiWkks2 fqECrKCaXsw2juSkAKXV6MeD/VKtmc8ypz5iw4amaP0KqVoLIAyg= X-Google-Smtp-Source: AGHT+IHZP6Un/HfjT028jnx0QVAGxq1NUEXIMQe8jhyWY9SpyTJFVDbuNtWHifT6v8EukAWp/hKqGTYNjWvGHCDW5lY= X-Received: by 2002:a05:622a:343:b0:42b:eee7:5f5c with SMTP id r3-20020a05622a034300b0042beee75f5cmr9136031qtw.63.1706897084294; Fri, 02 Feb 2024 10:04:44 -0800 (PST) MIME-Version: 1.0 References: <20240201193014.2785570-1-tatashin@google.com> <02610629-05ef-4956-a122-36b6ac98fbc2@arm.com> <84c7e816-f749-48d8-a429-8b0ef799cdbb@arm.com> In-Reply-To: <84c7e816-f749-48d8-a429-8b0ef799cdbb@arm.com> From: Pasha Tatashin Date: Fri, 2 Feb 2024 13:04:08 -0500 Message-ID: Subject: Re: [PATCH] iommu/iova: use named kmem_cache for iova magazines To: Robin Murphy Cc: joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 43F83C0019 X-Stat-Signature: 153r8wz55ryzeeq7swsz9c83xwzsru6z X-Rspam-User: X-HE-Tag: 1706897085-306958 X-HE-Meta: U2FsdGVkX1/Ri3jciEOkrTlA60ovhtkLWEh7lfZ1pRJ9hWgmkY0MbeHqg6uXJh2nUz+5OpSwk1q1kiIK9aNrXb8CjVzlGQmNLgxgOSviKSJhGQ2hyz6Ojk3pvO/b8+5A1DPNtpuuDNj9fshX3Ikf6A0UER1obA5mqh2zqUt8YiigTxDQFAaXQH5xhhqzNanXo9BLK+OhkV1adHxMGVXzz2VgIVZNVl6Pp3rqokwoqvJ9vgucNAtCKQ0UUDWRxKr76k3iP/EpIJBI9d1+ngnRgjPGobxQiGkde7yZTjYUjjoWVlR/FxdQ5UbKwsoWWhrW4HzVq6ih9mko6QhWndPlbK6zuoePV5ka0gaak5MFq8KJRyJh8sTuWs9t0dKrg7PxfwAw8ERAjmaklVTTcGJqsaaZyiqoxSn0KTywUMc0WESEkIy0akuPxPer7/PFoFyw2hZW4hD5Fe0MAG8urXEl5nqd3vttHtqTDpmxPgUcSxl89rxFgKP2IctTJ5HAOkXfyxWCD9cVH2pTu81LxfZTsr1EJRQZBqcc/ELfqmIA0ReNDPIFoO08kRByGMDRtUT/B66Fy51XqeHBrwIHEuseKWsvVKc92dN7wAGoq5ewFu9Es+PdUMCkBB0G5IxaTaH7IRmEGNC0m0motn1NGRg2EmKM06hhvl5gLTY9vDWZf6C5qRt8wIMoJ8vl84z1EdsODVpAW21XE/4AJUa0dDr/cL7V+QtJQR4xHwHIiQ+EmifGdU4yO0YzKSLLt5VzoDOluG72fVwM4F6WE29cUS9ZAr7Vvtx4Old+o82ViY3eqYcy/D/MEFZwIj/IJlC52g6hxUsdzBNAs2UETPLhjIebzpfev0acOQRIABqmmtjctKuT1Rd9qr58suwZMqJM8vOGn5+ln5YPOlMOLufsBWJPH8x5TaX4UqIH+IuetR0IIdX+kAsvOYNcuqTjWdqtRp+xe39nBYYRTgzd2M7QG8+ yVN+d9/S vuEazBbSd8Iy5BQ777mJ/AW0JWWmHd20m+EjXSR/kSlpaQGa8RHTSj1R0Nyw2Oqp4HBFjM1H98RFBRoUGshIcaK0koDob+j+cTKK9h7EInZL0/m7bvB4PfmfYDTy7diNa+7row0hsEueF3Ep5hVABuVGqc2zcLfz9PFYr7shubc6m2CLAMjv5DiiBTN499xG/8KElflPoaX9fGzQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > Hmm, I did misspeak slightly (it's late and I really should have left > this for tomorrow...) - that's 12KB per CPU *per domain*, but still that > would seem to imply well over 100 domains if you have 242MB of magazine > allocations while the iommu_iova cache isn't even on the charts... what > the heck is that driver doing? > > (I don't necessarily disagree with the spirit of the patch BTW, I just > really want to understand the situation that prompted it, and make sure > we don't actually have a subtle leak somewhere.) Hi Robin, The following tracing is without Google TPU, simply upstream kernel: The iova_domain_init_rcaches is called 159 with the following stack: iova_domain_init_rcaches iommu_setup_dma_ops amd_iommu_probe_finalize bus_iommu_probe iommu_device_register iommu_init_pci amd_iommu_init_pci state_next iommu_go_to_state amd_iommu_init pci_iommu_init do_one_initcall Each time 1536K is allocated: in total 159 * 1536K = 238.5M The allocation happens like this: for (IOVA_RANGE_CACHE_MAX_SIZE) for_each_possible_cpu() iova_magazine_alloc iova_magazine_alloc IOVA_RANGE_CACHE_MAX_SIZE = 6 ncpu = 128 sizeof (struct iova_magazine) = 1K 6 * 128 * (1K + 1K) = 1536K Pasha