From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 164EEC43334 for ; Thu, 30 Jun 2022 08:52:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8F158E0001; Thu, 30 Jun 2022 04:52:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3EF96B0073; Thu, 30 Jun 2022 04:52:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92F638E0001; Thu, 30 Jun 2022 04:52:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8002A6B0072 for ; Thu, 30 Jun 2022 04:52:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4FE412147C for ; Thu, 30 Jun 2022 08:52:22 +0000 (UTC) X-FDA: 79634285724.07.AD793AD Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 9EF7610001D for ; Thu, 30 Jun 2022 08:52:21 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EBE561BF7; Thu, 30 Jun 2022 01:52:20 -0700 (PDT) Received: from [10.57.85.25] (unknown [10.57.85.25]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF03E3F5A1; Thu, 30 Jun 2022 01:52:18 -0700 (PDT) Message-ID: <2cf27cd6-5cb9-57e7-dc52-e39f37945343@arm.com> Date: Thu, 30 Jun 2022 09:52:12 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [RFC PATCH] mm/slub: enable debugging memory wasting of kmalloc Content-Language: en-GB To: Andrew Morton , Feng Tang Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dave.hansen@intel.com, Joerg Roedel References: <20220630014715.73330-1-feng.tang@intel.com> <20220629193006.77e9f071a5940e882c459cdd@linux-foundation.org> <20220630023844.GA4668@shbuild999.sh.intel.com> <20220629194747.62effc10a994f67e26fe96af@linux-foundation.org> From: Robin Murphy In-Reply-To: <20220629194747.62effc10a994f67e26fe96af@linux-foundation.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656579142; a=rsa-sha256; cv=none; b=zgsucCQ9Lw/qMXMyG0XbFpc7UFWpfLXCa6rsOjI0huxhesncH3LIqyiPHz4uIlIt13thv6 RLdByp7Ddhn/ucBf30C4xEJWTH7E4dlm1QZOAHH86kU9XFwwLiu6/OOI5rP6RdnTrYqG0x SxNf034ojNn8yAuFNCzkjR67rmzGAzk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656579142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DugCVQMRBR9H4FZJggXTVQiDVYhh6Bme3er9LaiGRXc=; b=TwjJML2UCwyUzMDm4vrVm0cMtZ0Vm40BjPUH3ljJMpYkpEf+jEeXwmgQcYKwMiV1JMoZVx btFZUaXpRIGujDRga0sa6HIu4FN8owKgraEvMESSBqznUARzpHOA9m0gjee0SOtFks12jf wUUi+AOXQ0g9nUCpQerWHRwmm8U1D7Y= X-Stat-Signature: o8mnorpqh5gqrehw8tfdpyrkdfk8rq8e X-Rspamd-Queue-Id: 9EF7610001D Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1656579141-552183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-06-30 03:47, Andrew Morton wrote: > On Thu, 30 Jun 2022 10:38:44 +0800 Feng Tang wrote: > >> Hi Andrew, >> >> Thanks for the review! >> >> On Wed, Jun 29, 2022 at 07:30:06PM -0700, Andrew Morton wrote: >>> On Thu, 30 Jun 2022 09:47:15 +0800 Feng Tang wrote: >>> >>>> kmalloc's API family is critical for mm, with one shortcoming that >>>> its object size is fixed to be power of 2. When user requests memory >>>> for '2^n + 1' bytes, actually 2^(n+1) bytes will be allocated, so >>>> in worst case, there is around 50% memory space waste. >>>> >>>> We've met a kernel boot OOM panic, and from the dumped slab info: >>>> >>>> [ 26.062145] kmalloc-2k 814056KB 814056KB >>>> >>>> >From debug we found there are huge number of 'struct iova_magazine', >>>> whose size is 1032 bytes (1024 + 8), so each allocation will waste >>>> 1016 bytes. Though the issue is solved by giving the right(bigger) >>>> size of RAM, it is still better to optimize the size (either use >>>> a kmalloc friendly size or create a dedicated slab for it). >>> >>> Well that's nice, and additional visibility is presumably a good thing. >>> >>> But what the heck is going on with iova_magazine? Is anyone looking at >>> moderating its impact? >> >> Yes, I have a very simple patch at hand >> >> --- a/drivers/iommu/iova.c >> +++ b/drivers/iommu/iova.c >> @@ -614,7 +614,7 @@ EXPORT_SYMBOL_GPL(reserve_iova); >> * dynamic size tuning described in the paper. >> */ >> >> -#define IOVA_MAG_SIZE 128 >> +#define IOVA_MAG_SIZE 127 > > Well OK. Would benefit from a comment explaining the reasoning. > > But we still have eleventy squillion of these things in flight. Why? They're storage for a per-CPU caching scheme - for n CPUs, there should currently be (2n + 32) * 6 in flight, since there's one set for each of 6 sizes. The 32 really should be n or 2n as well since it's needlessly large for small systems and a bottleneck for large ones, but it needs some unpicking to allow for dynamic allocations. Thanks, Robin.