From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 025C7C43334 for ; Thu, 30 Jun 2022 03:23:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50D496B0072; Wed, 29 Jun 2022 23:23:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4BAAC6B0073; Wed, 29 Jun 2022 23:23:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3840A8E0001; Wed, 29 Jun 2022 23:23:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 22B646B0072 for ; Wed, 29 Jun 2022 23:23:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E0CDFDBA for ; Thu, 30 Jun 2022 03:23:22 +0000 (UTC) X-FDA: 79633456644.19.CBC0899 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf31.hostedemail.com (Postfix) with ESMTP id 5A18E20018 for ; Thu, 30 Jun 2022 03:23:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1656559401; x=1688095401; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=Kt86ua8WgIQ6xNJV4KsVEmxVoqhINuYbDD56u/ebnzM=; b=d9n+Wj9w3ayuegOK/8ArQ8ITvY3sFcqtjFkbPrjC9oREtQ15TYLdq5T7 LDE7gV0UJ0CRsLs7uv/aC/RaTLglE1VohzavuVI3HebPfjTRDkeIcfZgA ac00lVW38r5VqYhcyXvgV3C8zksKN8ttFNqjiMjao6PFDOaJAywCgwG3Z tMXLDz55MdtQkkY50EHCc44uTDGHyYV+g1rIcLLHnFjN4FZFHN2WQ8BFv jriL6cXy52hfpedQuRizgk4wQdxx27Um0F46/L7WlMvM86NNQb7qg+PU9 jnIM7wetyws5WZbXoUdSmKUCTNtSGpiJ4mQqSbju2uAY1Bjr63uN7jjRZ g==; X-IronPort-AV: E=McAfee;i="6400,9594,10393"; a="270995055" X-IronPort-AV: E=Sophos;i="5.92,232,1650956400"; d="scan'208";a="270995055" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jun 2022 20:23:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,232,1650956400"; d="scan'208";a="647690885" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.146.138]) by fmsmga008.fm.intel.com with ESMTP; 29 Jun 2022 20:23:16 -0700 Date: Thu, 30 Jun 2022 11:23:15 +0800 From: Feng Tang To: Andrew Morton Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dave.hansen@intel.com, Joerg Roedel , Robin Murphy Subject: Re: [RFC PATCH] mm/slub: enable debugging memory wasting of kmalloc Message-ID: <20220630032315.GB4668@shbuild999.sh.intel.com> References: <20220630014715.73330-1-feng.tang@intel.com> <20220629193006.77e9f071a5940e882c459cdd@linux-foundation.org> <20220630023844.GA4668@shbuild999.sh.intel.com> <20220629194747.62effc10a994f67e26fe96af@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220629194747.62effc10a994f67e26fe96af@linux-foundation.org> ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d9n+Wj9w; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.20) smtp.mailfrom=feng.tang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656559402; a=rsa-sha256; cv=none; b=KNSBtPr79dcF4U7fcoZ6wVjT8HIbTbY21i5V+HQEtUtS1NvZ+YHUreOk1b3s4SwHPHyVps Xl8t/V1grxNgw0fbEfdQvdrHKVx+S571qViL+C7XBIZYYQRcN1PHhXlPoTFdjMhh/xSRKJ UfI6GwP2lc18JIBKWZJjmm/cJnKYOnU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656559402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7x2roU8NMHUYts0J/LAQR6fIWEsubADf0PJnvvQx/08=; b=TG3h4uP6n5N3e1t/kuO3qsIPbPQ3p2AYOMblrENtyNCS3Pcj+p467rq2MA0Em7GDR/V+9h 3Ux60lu0d/I4DF1uAS5RDZ7DCdzk1JggTuK/JNvdWf8p33fNRFjuS3eqNgK37uIWEfZ2Uv FTaNgf+qysT96akl7Ms7rK7gFjlcwXo= X-Rspam-User: Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d9n+Wj9w; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.20) smtp.mailfrom=feng.tang@intel.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5A18E20018 X-Stat-Signature: md4p5aidhshyhatrasth94y8exascn5q X-HE-Tag: 1656559401-66895 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 29, 2022 at 07:47:47PM -0700, Andrew Morton wrote: > On Thu, 30 Jun 2022 10:38:44 +0800 Feng Tang wrote: > > > Hi Andrew, > > > > Thanks for the review! > > > > On Wed, Jun 29, 2022 at 07:30:06PM -0700, Andrew Morton wrote: > > > On Thu, 30 Jun 2022 09:47:15 +0800 Feng Tang wrote: > > > > > > > kmalloc's API family is critical for mm, with one shortcoming that > > > > its object size is fixed to be power of 2. When user requests memory > > > > for '2^n + 1' bytes, actually 2^(n+1) bytes will be allocated, so > > > > in worst case, there is around 50% memory space waste. > > > > > > > > We've met a kernel boot OOM panic, and from the dumped slab info: > > > > > > > > [ 26.062145] kmalloc-2k 814056KB 814056KB > > > > > > > > >From debug we found there are huge number of 'struct iova_magazine', > > > > whose size is 1032 bytes (1024 + 8), so each allocation will waste > > > > 1016 bytes. Though the issue is solved by giving the right(bigger) > > > > size of RAM, it is still better to optimize the size (either use > > > > a kmalloc friendly size or create a dedicated slab for it). > > > > > > Well that's nice, and additional visibility is presumably a good thing. > > > > > > But what the heck is going on with iova_magazine? Is anyone looking at > > > moderating its impact? > > > > Yes, I have a very simple patch at hand > > > > --- a/drivers/iommu/iova.c > > +++ b/drivers/iommu/iova.c > > @@ -614,7 +614,7 @@ EXPORT_SYMBOL_GPL(reserve_iova); > > * dynamic size tuning described in the paper. > > */ > > > > -#define IOVA_MAG_SIZE 128 > > +#define IOVA_MAG_SIZE 127 > > Well OK. Would benefit from a comment explaining the reasoning. Sure, will try to give the full context. > But we still have eleventy squillion of these things in flight. Why? I've checked the waste info right after boot for desktop/server, the waste is not severe generally, and I didn't even find 'iova_magzine' (could be due to it's virtulization related). When dockers are started to run workload, more kmalloc is invoked and the waste increases accordingly. Another case that can benefit is budget devices with limited memory, which wants to squeeze the wasted memory. Thanks, Feng > > #define MAX_GLOBAL_MAGS 32 /* magazines per bin */ > > > > struct iova_magazine { > > > > I guess changing it from 128 to 127 will not hurt much, and plan to > > send it out soon. >