From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E126C43334 for ; Thu, 30 Jun 2022 09:37:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDA3E6B0072; Thu, 30 Jun 2022 05:37:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B89496B0073; Thu, 30 Jun 2022 05:37:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A77A98E0001; Thu, 30 Jun 2022 05:37:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9654A6B0072 for ; Thu, 30 Jun 2022 05:37:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 7056881286 for ; Thu, 30 Jun 2022 09:37:30 +0000 (UTC) X-FDA: 79634399460.16.7B90972 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf18.hostedemail.com (Postfix) with ESMTP id 5CBF81C0038 for ; Thu, 30 Jun 2022 09:37:28 +0000 (UTC) Received: from fraeml705-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4LYY8N6qV4z686VP; Thu, 30 Jun 2022 17:35:00 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml705-chm.china.huawei.com (10.206.15.54) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2375.24; Thu, 30 Jun 2022 11:37:23 +0200 Received: from [10.126.174.156] (10.126.174.156) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 30 Jun 2022 10:37:22 +0100 Message-ID: <7c29d01d-d90c-58d3-a6e0-0b6c404173ac@huawei.com> Date: Thu, 30 Jun 2022 10:37:25 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Subject: Re: [PATCH] iommu/iova: change IOVA_MAG_SIZE to 127 to save memory To: Robin Murphy , Feng Tang , Joerg Roedel , Will Deacon , , , Andrew Morton , Christoph Lameter , "Vlastimil Babka" CC: , , Paul Menzel References: <20220630073304.26945-1-feng.tang@intel.com> <13db50bb-57c7-0d54-3857-84b8a4591d9e@arm.com> From: John Garry In-Reply-To: <13db50bb-57c7-0d54-3857-84b8a4591d9e@arm.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.174.156] X-ClientProxiedBy: lhreml727-chm.china.huawei.com (10.201.108.78) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of john.garry@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=john.garry@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656581850; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MveCOVSg0FI4qV60vc+RgYkqLcL1lFLAijm58ZHEvJA=; b=Lon5dQcFm/xNGlzQ33Oukdyj5OC/ust5NGRA+FtDOkHahg7qyR/ZypWw9S3JUs8P1J1FnI PJ7QcTz5YZLT/69I+0JN4CA7CYXUmYr/vTNbbtBsjvHU08J5ZWGWMPcMdOVzIQEdbgOURP Kn9+d+2m7Yp9XMOBiGl93GvNOKN3f8w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656581850; a=rsa-sha256; cv=none; b=qWsDdO72j61/waS6lLrs/LBQ7q9MyqA6YNAPXrbxWJOAtq3/fBqpMI7FpXPVbk/XWJszd+ FAdnNvL6U/gBoLEm/1WBjlkvRA58jvzI86Wr4bN38dQRm3eZGVoTKLrYW+9ulFJB07IkmZ OGZAfUf5vyfhAIEYDIu9sAe5wZxbtWw= X-Stat-Signature: eye38izsj37zwxgfp8kb19jtbmis718z X-Rspamd-Queue-Id: 5CBF81C0038 Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of john.garry@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=john.garry@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1656581848-922034 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 30/06/2022 10:02, Robin Murphy wrote: > On 2022-06-30 08:33, Feng Tang wrote: >> kmalloc will round up the request size to power of 2, and current >> iova_magazine's size is 1032 (1024+8) bytes, so each instance >> allocated will get 2048 bytes from kmalloc, causing around 1KB >> waste. >> >> And in some exstreme case, the memory wasted can trigger OOM as >> reported in 2019 on a crash kernel with 256 MB memory [1]. > > I don't think it really needs pointing out that excessive memory > consumption can cause OOM. Especially not in the particularly silly > context of a system with only 2MB of RAM per CPU - that's pretty much > guaranteed to be doomed one way or another. > >>    [    4.319253] iommu: Adding device 0000:06:00.2 to group 5 >>    [    4.325869] iommu: Adding device 0000:20:01.0 to group 15 >>    [    4.332648] iommu: Adding device 0000:20:02.0 to group 16 >>    [    4.338946] swapper/0 invoked oom-killer: >> gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=0, >> oom_score_adj=0 >>    [    4.350251] swapper/0 cpuset=/ mems_allowed=0 >>    [    4.354618] CPU: 0 PID: 1 Comm: swapper/0 Not tainted >> 4.19.57.mx64.282 #1 >>    [    4.355612] Hardware name: Dell Inc. PowerEdge R7425/08V001, >> BIOS 1.9.3 06/25/2019 >>    [    4.355612] Call Trace: >>    [    4.355612]  dump_stack+0x46/0x5b >>    [    4.355612]  dump_header+0x6b/0x289 >>    [    4.355612]  out_of_memory+0x470/0x4c0 >>    [    4.355612]  __alloc_pages_nodemask+0x970/0x1030 >>    [    4.355612]  cache_grow_begin+0x7d/0x520 >>    [    4.355612]  fallback_alloc+0x148/0x200 >>    [    4.355612]  kmem_cache_alloc_trace+0xac/0x1f0 >>    [    4.355612]  init_iova_domain+0x112/0x170 >>    [    4.355612]  amd_iommu_domain_alloc+0x138/0x1a0 >>    [    4.355612]  iommu_group_get_for_dev+0xc4/0x1a0 >>    [    4.355612]  amd_iommu_add_device+0x13a/0x610 >>    [    4.355612]  add_iommu_group+0x20/0x30 >>    [    4.355612]  bus_for_each_dev+0x76/0xc0 >>    [    4.355612]  bus_set_iommu+0xb6/0xf0 >>    [    4.355612]  amd_iommu_init_api+0x112/0x132 >>    [    4.355612]  state_next+0xfb1/0x1165 >>    [    4.355612]  amd_iommu_init+0x1f/0x67 >>    [    4.355612]  pci_iommu_init+0x16/0x3f >>    ... >>    [    4.670295] Unreclaimable slab info: >>    ... >>    [    4.857565] kmalloc-2048           59164KB      59164KB >> >> Change IOVA_MAG_SIZE from 128 to 127 to make size of 'iova_magazine' >> 1024 bytes so that no memory will be wasted. >> >> [1]. https://lkml.org/lkml/2019/8/12/266 >> >> Signed-off-by: Feng Tang >> --- >>   drivers/iommu/iova.c | 7 ++++++- >>   1 file changed, 6 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c >> index db77aa675145b..27634ddd9b904 100644 >> --- a/drivers/iommu/iova.c >> +++ b/drivers/iommu/iova.c >> @@ -614,7 +614,12 @@ EXPORT_SYMBOL_GPL(reserve_iova); >>    * dynamic size tuning described in the paper. >>    */ >> -#define IOVA_MAG_SIZE 128 >> +/* >> + * As kmalloc's buffer size is fixed to power of 2, 127 is chosen to >> + * assure size of 'iova_magzine' to be 1024 bytes, so that no memory > > Typo: iova_magazine > >> + * will be wasted. >> + */ >> +#define IOVA_MAG_SIZE 127 I do wonder if we will see some strange new behaviour since IOVA_FQ_SIZE % IOVA_MAG_SIZE != 0 now... > > The change itself seems perfectly reasonable, though. > > Acked-by: Robin Murphy