From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 716E5C7EE2E for ; Tue, 13 Jun 2023 12:56:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85E448E0003; Tue, 13 Jun 2023 08:56:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E8378E0002; Tue, 13 Jun 2023 08:56:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 687FB8E0003; Tue, 13 Jun 2023 08:56:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 52E578E0002 for ; Tue, 13 Jun 2023 08:56:11 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1F255AF9D4 for ; Tue, 13 Jun 2023 12:56:11 +0000 (UTC) X-FDA: 80897722542.22.D1B93CC Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf03.hostedemail.com (Postfix) with ESMTP id 7DD0620017 for ; Tue, 13 Jun 2023 12:56:07 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=jgk16kRP; spf=pass (imf03.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686660967; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QbKnGKMYsWuxmR4BukqDGTQxre+6IXc9rdlwaue5plE=; b=HMBD/1mvCtaMFVdrJIBfnW1rfME0b+FYF2VNU+vBUO+YPCW+FqbyXS/VrcxkrvrmuvpvqK bRhRgZDVlVx2wWCpqCUITsvvCcJzcGcDtGDPbUy+NxLZlvK5Yaz2ZlzjIzD6pRZbJKaPZx 38IoG7KvH3BMNoO33yD4abI50/W6sNw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686660967; a=rsa-sha256; cv=none; b=hGETUl3RArr7IzDWMg7qg2uOvQEzIf7K70wLYTBX8Y6hksEY5g6IIEdsar6E0A8ruQ/WS1 LRh87UUeqgj3ygDMJzJCnX1l1/U5l+/ufGvSPe8WoaX/nyjLpL4HEOlB19LS5MSc62I+Yy BRYdNJv1lSN0f7YiaJ0XlPdGA/YoaRA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=jgk16kRP; spf=pass (imf03.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35DCl60V014774; Tue, 13 Jun 2023 12:55:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : reply-to : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=QbKnGKMYsWuxmR4BukqDGTQxre+6IXc9rdlwaue5plE=; b=jgk16kRPpg5lyGM+b5O3sCJNYywMpgqqS5f9GLSpJHRwylYU5DrgJR31gsIreS3BC/Mq tsM1K+PZzzx70miboYzdPgnO+pplKC7IHug+3gEKGPXDEKxHvfxEk6wH0RgvoepQXe+9 tPgn8b0YiVdi0ezGIVyEutNBDkLAh2Ka5xMre1G9sWVF8zqCxcVNVXcZVndySx+RowQN lk1aQJq6jXwsU99L6HWNfDsV5l88Gre2P3gMy2vFUjvbF/18DnAgHOnvEmfUnlzaB7gJ 074MLA/dj8Em7geAzAoetvniAXLzR+SjOUqVL8sTXYBmifojM9rlUkyCZrqnbevDwjSi 2g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6rrug6j9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:55:56 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35DCo7s0022697; Tue, 13 Jun 2023 12:55:56 GMT Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6rrug6hp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:55:56 +0000 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35D99SSq002823; Tue, 13 Jun 2023 12:55:54 GMT Received: from smtprelay07.dal12v.mail.ibm.com ([9.208.130.99]) by ppma03wdc.us.ibm.com (PPS) with ESMTPS id 3r4gt5gd51-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 12:55:54 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay07.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35DCtri620840868 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 12:55:53 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A807C58067; Tue, 13 Jun 2023 12:55:53 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2BF4C58063; Tue, 13 Jun 2023 12:55:50 +0000 (GMT) Received: from patel.in.ibm.com (unknown [9.109.195.231]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 12:55:49 +0000 (GMT) Message-ID: <51045addfd3ec7804bcc36daed5827bf51095b68.camel@linux.ibm.com> Subject: Re: [RFC PATCH ] mm/slub: Reducing slub memory wastage From: Jay Patel Reply-To: jaypatel@linux.ibm.com To: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, aneesh.kumar@linux.ibm.com, tsahu@linux.ibm.com, piyushs@linux.ibm.com Date: Tue, 13 Jun 2023 18:25:48 +0530 In-Reply-To: References: <20230612085535.275206-1-jaypatel@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 (3.28.5-18.el8) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: RdH9Eue48JKGqXxZ-1q0wUdheCrHpO2k X-Proofpoint-GUID: vVkS-kLG0M2U9XJed2QssakfKkN24znT X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-13_04,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 adultscore=0 mlxscore=0 priorityscore=1501 spamscore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 malwarescore=0 lowpriorityscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130110 X-Stat-Signature: q7ptq5wcgyxysqez4hbt4dyqjbaqo95c X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7DD0620017 X-Rspam-User: X-HE-Tag: 1686660967-783773 X-HE-Meta: U2FsdGVkX1+GyQfV2A4/bvEc0kGUilSOiUFJinONYl+OewkM6rxORreI+1/XkEbK65ZEJ+1X6keSfSpb4AIwZDFWAXq4bN+LnntEt9sq8zeefgWvqqA4TNPnGARpat1qMbk4Jr3IcUrZEtP3Lo1Y3kwSoix7H1jZkHCDMZA9Lb77zEnM2GZ94FrHkXCN7h/Y+pHjziYUSwxfKILwRiEOpJ3xYiSNVwcL9sppm2fjffKqAxi4wgSDl+qEb8gLoLTrJZM76kIl9cRUKYl4UNe8aPehNftHA9NlYz1qcJ4Zmi+dUgalkhIO6PLPe02z+oFB1lIQr9Jjsx52n+7fxrRsN9XB6AhHbQ6Nks11wYEX7ZKYukz1wc0sTLUbHq4n1LMGDq+nnXbc2RMPHK+TBT3pnPPLiWx91NcWUDBRn3iYeoIsI7GV4YjYN3cJEOUsO8Q2blASCcFBHunN7yTWPMx/qUndqmyhl0lYam7FxfFiFgaD2dnw38VexG8QczSAM1lM9WexLlogQWhumDOlO82D/ZotdtzAPRTBFXwFFadqjkbh+SSHibDn24+nkVQJPTI02eYn4/8sHlw1j4bOpFPoOVLkEviTIxp9yNvh9ojpjhlgDgBZPjJYW97rwMH2FoTQU8FU12ETVqDU+WSLVv867wikw0EIMfeYr9CoyinZNRCEzGUC18FPRiqJEqx00t0EP75WhXciEO2n+Y/Z5KNtsrXikj264yOnUuU+mVokvVpVb8m86GwwxVehgkDWvBo+eDCYdKeKxpWabDqAyOhHFbqx+IaiSPUk+iNwPcsRuJYMVxsacaIYxt4vT0A3kKt5TVkhVB8EOBc+0fCqtYowm19H0mWIpf8JbUpwiy6GBTv5n3EPL3mW2lWmRTT20PJDzlAklkNpx0x/wElJAU4nwjruH0+a133OaC9DRoS/dU3U1IybVhldQHDWziMq3lyuZRRuZwfl2tRSy7Q84iX XPRvMWwB V2EBeGPWcePTSqjh+KrZ11SYin7z5ubmwIt3VTn6CQnL2xFJWfL19qnqIb/K/G+lYRN25sBI47ZQ7g0xZ+43hgpjGvAcXvard6nf/tJqpxsUNwUToQMLcOpiSxS/V2hO7aXbTqcz+Du//+MWZ3va3wIF0yN/bqzsYmG0hk6hkHrBbQ1sqiv5Cr/5eiXSTfD1GuKJ0sd6VP3SB4f84LRzXGAJsMaMNT2kPx0xEbudXhE+sR6wrPlKA1CyGV8NSbUZ7CAGN2GemyBSAKG87rctvRq7+/Xy6JM5tV5mwUU/e+EZvTZ0E3bl2xzlVO+531j8t56H/rkPrAy/WvHXz7kJE5zwMyU/k2I/wYWuY9wpGxZI+OAc5NZYvW9nHujdz2YcbtHqK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 2023-06-12 at 19:17 +0900, Hyeonggon Yoo wrote: > On Mon, Jun 12, 2023 at 02:25:35PM +0530, Jay Patel wrote: > > 3) If the minimum order is less than the slub_max_order, iterate > > through > > a loop from minimum order to slub_max_order and check if the > > condition > > (rem <= slab_size / fract_leftover) holds true. Here, slab_size is > > calculated as (PAGE_SIZE << order), rem is (slab_size % > > object_size), > > and fract_leftover can have values of 16, 8, or 4. If the condition > > is > > true, select that order for the slab. > > > > However, in point 3, when calculating the fraction left over, it > > can > > result in a large range of values (like 1 Kb to 256 bytes on 4K > > page > > size & 4 Kb to 16 Kb on 64K page size with order 0 and goes on > > increasing with higher order) when compared to the remainder (rem). > > This > > can lead to the selection of an order that results in more memory > > wastage. To mitigate such wastage, we have modified point 3 as > > follows: > > instead of selecting the first order that satisfies the condition > > (rem > > <= slab_size / fract_leftover), we iterate through the loop from > > min_order to slub_max_order and choose the order that minimizes > > memory > > wastage for the slab. > > Hi Jay, > > If understand correctly, slub currently chooses an order if it > does not waste too much memory, but the order could be sub-optimal > because there can be an order that wastes less memory. right? > > Hmm, the new code might choose larger order than before, as SLUB > previously > wasted more memory instead of increasing order. > > BUT the maximum slub order is still bound by slub_max_order, > so that looks fine to me. If using high order for less fragmentation > becomes a problem, slub_max_order should be changed. > > Hi Hyeonggon Based on my understanding, the slub_max_order parameter is derived from the PAGE_ALLOC_COSTLY_ORDER configuration option. Any changes made to can have an impact on system performance. If you reduce the value of slub_max_order, it means that smaller contiguous memory will be used for certain slab caches. However, this can lead to a situation where the minimum number of objects required (min_objects) cannot fit within those smaller memory. As a result, performance issues may arise. > <...snip...> > > > I conducted tests on systems with 160 CPUs and 16 CPUs, using 4K > > and > > 64K page sizes. Through these tests, it was observed that the patch > > successfully reduces the wastage of slab memory without any > > noticeable > > performance degradation in the hackbench test report. However, it > > should > > be noted that the patch also increases the total number of objects, > > leading to an overall increase in total slab memory usage. > > <...snip...> > > Then my question is that, why is this a useful change if total memory > usage is increased? > This patch aimed in reducing memory wastage can potentially lead to an increase in the slab order for a slab cache. Consequently, this increase in page order can result in a higher number of objects per slab, reducing wastage and leading to a more efficient utilization of memory. This enhancement is advantageous since the presence of unused objects can be leveraged in the future, depending on varying workloads. > > > Test results are as follows: > > 3) On 16 CPUs with 4K Page size > > > > +-----------------+----------------+------------------+ > > > Total wastage in slub memory | > > +-----------------+----------------+------------------+ > > > | After Boot | After Hackbench | > > > Normal | 666 Kb | 902 Kb | > > > With Patch | 533 Kb | 694 Kb | > > > Wastage reduce | ~20% | ~23% | > > +-----------------+----------------+------------------+ > > > > +-----------------+----------------+----------------+ > > > Total slub memory | > > +-----------------+----------------+----------------+ > > > | After Boot | After Hackbench| > > > Normal | 82360 | 122532 | > > > With Patch | 87372 | 129180 | > > > Memory increase | ~6% | ~5% | > > +-----------------+----------------+----------------+ > > > > How should we understand this data? > reducing amount of memory wastage by increasing slab order > might not reduce total SLUB memory usage? > Indeed, the total slub memory is increase with this patch. However, the memory utilization is improved. The slub memory comprises both active objects and unused objects, along with memory wastage. With this patch, the memory wastage is reduced, leading to a higher number of unused objects (the numbers of active objects mostly remains the same). The presence of these unused objects provides the opportunity for their utilization, which can vary depending on different workloads. > > hackbench-process-sockets > > +-------+----+---------+---------+-----------+ > > > Amean | 1 | 1.4983 | 1.4867 | ( 0.78%) | > > > Amean | 4 | 5.6613 | 5.6793 | ( -0.32%) | > > > Amean | 7 | 9.9813 | 9.9873 | ( -0.06%) | > > > Amean | 12 | 17.6963 | 17.8527 | ( -0.88%) | > > > Amean | 21 | 31.2017 | 31.2060 | ( -0.01%) | > > > Amean | 30 | 44.0297 | 44.1750 | ( -0.33%) | > > > Amean | 48 | 70.2073 | 69.6210 | ( 0.84%) | > > > Amean | 64 | 92.3257 | 93.7410 | ( -1.53%) | > > +-------+----+---------+---------+-----------+ -- Jay Patel