From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95A04C7EE23 for ; Mon, 12 Jun 2023 10:17:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 364366B0072; Mon, 12 Jun 2023 06:17:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 312426B0074; Mon, 12 Jun 2023 06:17:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DA206B0075; Mon, 12 Jun 2023 06:17:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 077316B0072 for ; Mon, 12 Jun 2023 06:17:50 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C57C71C762B for ; Mon, 12 Jun 2023 10:17:49 +0000 (UTC) X-FDA: 80893694658.25.9F07B68 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) by imf27.hostedemail.com (Postfix) with ESMTP id E7EFB40012 for ; Mon, 12 Jun 2023 10:17:47 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=nvflBZb7; spf=pass (imf27.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686565067; a=rsa-sha256; cv=none; b=mYKR1cOYDKUjTCZUmtyXYjGn3AXr6aZ/lI6fuzbYZd2dZIkCkseHjJon1cPF5zrwlMlrT7 kiW5a79yu+nVSAJmEL4jv7qevH3VhbFloPghfVYXqWtRvCmEriYZOInPigJPDubiSewgV2 T4jKz9x6XU6JHYsi6rr8OlZ9Nt36N1g= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=nvflBZb7; spf=pass (imf27.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.167.170 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686565067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4AtxGVc0OBPhiwj5JHg1PHaNUpIRo2QX4yO19rFeqLk=; b=y0LEFlFHxhtYnUtR8i+96fFuCI4wJtH1MIE7zruz4mOh44jMeDZ/xPocUGLB5qLeg/k6qb CMcisIM5dUObOEYbCoTMM0TjiYUmukbzLa15Ys4P0pyOTWGZuz2dB2MxrCANk00HBmTI0v NA+TITlEHIbOV9b71kRWjQvAs+AvrpI= Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3942c6584f0so1401662b6e.3 for ; Mon, 12 Jun 2023 03:17:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686565067; x=1689157067; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=4AtxGVc0OBPhiwj5JHg1PHaNUpIRo2QX4yO19rFeqLk=; b=nvflBZb7dPXfHPMGRcEyfklAcBJQk2Zx4vQFYK/215N2Wy99P3RAvJnGbfc5pTNpi/ smoZQTbsbSz0T8vuO6y7NK3YH+IU6SdP2Ag+J/XpoALS/yNtXymThqeCT/36798E8yc5 h1CxhjFHYTSSgiMDDlEvJZRZxjI1i6CqYXgBxqhy/vZ2NPYSgJAOXGOA2CJrS14dXFOg JG5ICT/y/iku4wfl/WHiaqVh4S/IM4idbLZqAIZJ91OTZh5TCfhiOGQyt568EPaH2AKZ FQSAu6JybNGoitB3cIRPyT4bbbhbKb53gteHZHMWv79D1s4/j+h+WFwEVJZM3u7rj0J3 D0Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686565067; x=1689157067; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=4AtxGVc0OBPhiwj5JHg1PHaNUpIRo2QX4yO19rFeqLk=; b=SFH1oEhUIGUCKwLvr+cO9K5uKIQmnBBTJ1e4QtxCbg50GtzzqtaExBz5E5U8MrFSr6 X2lsf92UnhJAVwcbdiMLcRXY+ReP3DULUTHEu4K2igFosRWn6UN/Nw2lf8Bb8zpore8N F5LcE/ve/coboiPMB+1e97vJuxtmqD4vhIzyo7LAG8vaukMr3gFpeOl3ICnZjuVeTYir dOJv7eR0XSUaDW0pwQuAnsEJT2l8/o8jJVg0rQD6BI+5D4VCSni6hCwFsKFu3+F0aPdB gPGofV/nOXOqNE8KiBxpXWnwvxvyXIPUoyv59XGWcEUzitNgV3jsyiyIA2YiylSbzKLc OwAg== X-Gm-Message-State: AC+VfDz/BZLnSIJL+2IXbmo+7tEmrFnpTstyxPta80HoUqXX6ICZm8lz CF0MsVBvFb2FEGWRuNirZB8= X-Google-Smtp-Source: ACHHUZ4vwKC1LmhmQXet01AbKZtwKNs2t28PYozLoyLWmmn1NBnfUak0fgrYDs/BtJHd/IJx2VaEIw== X-Received: by 2002:a05:6808:1895:b0:39c:786c:c8cf with SMTP id bi21-20020a056808189500b0039c786cc8cfmr4549545oib.20.1686565066792; Mon, 12 Jun 2023 03:17:46 -0700 (PDT) Received: from debian-BULLSEYE-live-builder-AMD64 ([118.42.100.119]) by smtp.gmail.com with ESMTPSA id mq16-20020a17090b381000b00259980d373dsm9045127pjb.1.2023.06.12.03.17.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 03:17:46 -0700 (PDT) Date: Mon, 12 Jun 2023 19:17:37 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Jay Patel Cc: linux-mm@kvack.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, aneesh.kumar@linux.ibm.com, tsahu@linux.ibm.com, piyushs@linux.ibm.com Subject: Re: [RFC PATCH ] mm/slub: Reducing slub memory wastage Message-ID: References: <20230612085535.275206-1-jaypatel@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230612085535.275206-1-jaypatel@linux.ibm.com> X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E7EFB40012 X-Stat-Signature: nqc61rk77z48s684j3jceztt4adyumzz X-Rspam-User: X-HE-Tag: 1686565067-733276 X-HE-Meta: U2FsdGVkX19l++ACCyILVvnmY362MsmJN/bBqR0qG8o6gMbkSBZdtXBYngWk2QPj1FlFGwD7ralY0Aa7bg6E8BxPSxzWlc6mlqt/CI0qGdHtpBWwCf/mxo+1eEawtw+J1zjs5ShZAH2d0/y0PWHYJLzOJpUTisaT4/0CWthSd2F6vDovFJj1RkzytqSiRq3tZRpQec/mK8/LhYNe2Kjh/yWFQ5CSDvUmSAiVWYTAVtS2Udw+TwmAehuoqigwL4UIZjgMkn2R9gcCVZ/WYHFS0Ztep3UK8rkkRK5iW2JBSbW39+Vr2xlSRjbBZ+t3COUzoXFamhNUGxF5KwNjEnTzuzDKLq3bA2zE+pkYdh3tsSVnVyMjPHmIT97u8n7Y1EEf2N1ZB+kAmsnH/DYoBPLPIJIH/UZ/LMsDgoOOKmQNEI+0WhJPd/D7kYUvBD3xR6S/hYkCEFOERCYBkdKuRwfiwBr2aUcgTPeiUKndgPbuzJzysKVvmUBo8Cnzr4lHEiQx80W6AAMuR0iSFfWtMN9ntBlRgIgHH3jXj+08wVxRcEwWIR6t08IfC2LH/imcfIr0y5XumeoTvwdE8iKcIDx7SDJWDu31Std2zg+5xST/HeLjWAezsYlDhxUgdYH2Xy76BP+5YIK6lSG9Y7A7ShHiNfn5xqcgxwNhnvEDiLyfJloPBWMIyykS+/6OLAbO4kLyBgc+OATHEHBwo9wEqLqAsU1G0fNiL0UBDp0Lj4+d5/4B8Fn9Ei6ZlCI6L/FEiiH5B/KsStQjM6wvCnqeFp+sCvdRGsWVNAq/Hjbpqn/WT9o2a1hh6N2PugeQx0McjwVC6SqBIdR0UBfq6z69qLF9D7xVuL8Yh7PLCjHH5epwzfJ9AlLq0I9hvLPa72Jibwe+l2vj3UzRBmhR8uOOCfyhGCubxOPQ8bxRbmRnvH3xzj+cq1p1Zd9KSsU5RpZgHIZOPM8LoPmLzihUdnuGZDw DuLf55/p zB0PAJOIXWPKnR9D/51GHZ5w8pry1vEiwZ0U1x8DEPGNhvFFRU4VdHbH9VxgpG1B4xY7rhoWWYVl44wnpSwvGMjNJQbnG3m1QlvC4OWDXKxHPUeym1S87G+a5Fo8g7BumcRBcIfcoVnj3C0z5hrUl71WA4etuQQ398HQsNXjsYS1lczM3iNrWnUpsLt8Mx38BcYyfJdvIhXX7fVgyhwhM7kq/Yfnhwx1+q+MwJrFmUJT27rOQ5DKP/2UOLfb9TYOVDDVPM1XOtBPiOQLoQP6kNX0C3k/kIUsEdGPqWhZM4Fu5DKU6jKh3NxirFodR2ToGxepALcvq3mb+NSHzZSSakfnPj+vaufnXCKjLiXdWDksA/j9OAH3e7DQgUibrqVIgjK2MTxAk05rUsyNR05ekn8Koc+fgQGvF5fuqNH+CO7AYTVjjLN0pBrP0juLt0+mOayrY2k6kZ2JXtSx7zzqedQ4ykJLvKKoDbCQuh4UBW+urfWMEDI7nd1FN2rITImYIvWrq+MEl9rNnBBbobcYSuXRyCM7RIQp4PfmLYVN2Xfhz29I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 12, 2023 at 02:25:35PM +0530, Jay Patel wrote: > 3) If the minimum order is less than the slub_max_order, iterate through > a loop from minimum order to slub_max_order and check if the condition > (rem <= slab_size / fract_leftover) holds true. Here, slab_size is > calculated as (PAGE_SIZE << order), rem is (slab_size % object_size), > and fract_leftover can have values of 16, 8, or 4. If the condition is > true, select that order for the slab. > > However, in point 3, when calculating the fraction left over, it can > result in a large range of values (like 1 Kb to 256 bytes on 4K page > size & 4 Kb to 16 Kb on 64K page size with order 0 and goes on > increasing with higher order) when compared to the remainder (rem). This > can lead to the selection of an order that results in more memory > wastage. To mitigate such wastage, we have modified point 3 as follows: > instead of selecting the first order that satisfies the condition (rem > <= slab_size / fract_leftover), we iterate through the loop from > min_order to slub_max_order and choose the order that minimizes memory > wastage for the slab. Hi Jay, If understand correctly, slub currently chooses an order if it does not waste too much memory, but the order could be sub-optimal because there can be an order that wastes less memory. right? Hmm, the new code might choose larger order than before, as SLUB previously wasted more memory instead of increasing order. BUT the maximum slub order is still bound by slub_max_order, so that looks fine to me. If using high order for less fragmentation becomes a problem, slub_max_order should be changed. <...snip...> > I conducted tests on systems with 160 CPUs and 16 CPUs, using 4K and > 64K page sizes. Through these tests, it was observed that the patch > successfully reduces the wastage of slab memory without any noticeable > performance degradation in the hackbench test report. However, it should > be noted that the patch also increases the total number of objects, > leading to an overall increase in total slab memory usage. <...snip...> Then my question is that, why is this a useful change if total memory usage is increased? > Test results are as follows: > 3) On 16 CPUs with 4K Page size > > +-----------------+----------------+------------------+ > | Total wastage in slub memory | > +-----------------+----------------+------------------+ > | | After Boot | After Hackbench | > | Normal | 666 Kb | 902 Kb | > | With Patch | 533 Kb | 694 Kb | > | Wastage reduce | ~20% | ~23% | > +-----------------+----------------+------------------+ > > +-----------------+----------------+----------------+ > | Total slub memory | > +-----------------+----------------+----------------+ > | | After Boot | After Hackbench| > | Normal | 82360 | 122532 | > | With Patch | 87372 | 129180 | > | Memory increase | ~6% | ~5% | > +-----------------+----------------+----------------+ > How should we understand this data? reducing amount of memory wastage by increasing slab order might not reduce total SLUB memory usage? > hackbench-process-sockets > +-------+----+---------+---------+-----------+ > | Amean | 1 | 1.4983 | 1.4867 | ( 0.78%) | > | Amean | 4 | 5.6613 | 5.6793 | ( -0.32%) | > | Amean | 7 | 9.9813 | 9.9873 | ( -0.06%) | > | Amean | 12 | 17.6963 | 17.8527 | ( -0.88%) | > | Amean | 21 | 31.2017 | 31.2060 | ( -0.01%) | > | Amean | 30 | 44.0297 | 44.1750 | ( -0.33%) | > | Amean | 48 | 70.2073 | 69.6210 | ( 0.84%) | > | Amean | 64 | 92.3257 | 93.7410 | ( -1.53%) | > +-------+----+---------+---------+-----------+ -- Hyeonggon Yoo Undergraduate | Chungnam National University Dept. Computer Science & Engineering