From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99EF0EDE9BC for ; Thu, 14 Sep 2023 12:43:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9CC46B02A8; Thu, 14 Sep 2023 08:43:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4CA36B02AA; Thu, 14 Sep 2023 08:43:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC71B6B02B1; Thu, 14 Sep 2023 08:43:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 927266B02A8 for ; Thu, 14 Sep 2023 08:43:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 514561CA11A for ; Thu, 14 Sep 2023 12:43:42 +0000 (UTC) X-FDA: 81235169484.07.9EB9C2A Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf19.hostedemail.com (Postfix) with ESMTP id 69F591A0022 for ; Thu, 14 Sep 2023 12:43:39 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Y09TuMTw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf19.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694695419; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sc+GW4A9jQOyrZbvvyhgSWtGSaeYP59oPFDHcKhFLE4=; b=4cTGG99spM4GLsgoYm4ZbC9pN0eGO61kOleOcj0V0X1HP5yl/9nZAXigXhG3v+6k835w7e /i3m+woF5IC/ah7XQ5nAatH4Fas95Rs86VUxsxtFoFoQdQ5F2ZKy2FomOevi7N/Qda3Uzd q2STeLjqqJZ85vjZnQJuaut30UTP9HI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Y09TuMTw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf19.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694695419; a=rsa-sha256; cv=none; b=kzmjjYHz+wSB45cUq7+c/BLCgjsyTELtgyWgi8qfxaFV5do1MrNuxtU7ddnLF/+KINynLV 3/rtbRKceeHLZu30YKPkKdzFsSlGM6/zwgDi+4o1r1SbAUX9VWprQZO+i0jh+YibgE3Y1l xhlcTS0HKz+iDitrwcqbN6O8MP23ykA= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38ECcg2B017412; Thu, 14 Sep 2023 12:43:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : reply-to : to : cc : date : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=pp1; bh=sc+GW4A9jQOyrZbvvyhgSWtGSaeYP59oPFDHcKhFLE4=; b=Y09TuMTw9ogJC/jbDYBbC95s5+/qNjEJu5a3eF4BflUlBRxE1XQAlS3AEDyE2UgNOUvB 3Ycilhpas/MhOQ2NwB9a3Iu0IoxX/0xBRSk6ZL6OAZs/hdz9oFX/Tj0NKHOW5IkqefTZ v6gc33MoiJ8lZc4x6pbMC3cuDXF9VtQ6wIsFqIkLI8TJba7ZchRnXE0mrowfJB4qRqoT EbdPCCmTDCQ7ZP5Lmd9pTfsT4ulNvUCWF7lUvw+XkMCVVGyPzzhpt14uK57rWxTCDhIh QJ1jv9vkpGRiDc7Qn2YRqlk0+DLEBDPSVQl2beHkd5waRJn89y2zO9iCA5YAPXHFPnb6 Yg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t42758a6m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 12:43:29 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 38ECcpxP017754; Thu, 14 Sep 2023 12:43:28 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t42758a65-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 12:43:28 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 38EAeV39023099; Thu, 14 Sep 2023 12:43:26 GMT Received: from smtprelay04.dal12v.mail.ibm.com ([172.16.1.6]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3t141p2wbf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 12:43:26 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay04.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 38EChP3212386900 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Sep 2023 12:43:25 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C47FA58057; Thu, 14 Sep 2023 12:43:25 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0244C58058; Thu, 14 Sep 2023 12:43:22 +0000 (GMT) Received: from patel.in.ibm.com (unknown [9.109.195.146]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 14 Sep 2023 12:43:21 +0000 (GMT) Message-ID: <1094067efad01704b32ac8b1e4d6e142ea7fcb5a.camel@linux.ibm.com> Subject: Re: [RFC PATCH v4] mm/slub: Optimize slub memory usage From: Jay Patel Reply-To: jaypatel@linux.ibm.com To: Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, tsahu@linux.ibm.com, piyushs@linux.ibm.com Date: Thu, 14 Sep 2023 18:13:20 +0530 In-Reply-To: References: <20230720102337.2069722-1-jaypatel@linux.ibm.com> <7fdf3f5dfd9fa1b5e210cc4176cac58a9992ecc0.camel@linux.ibm.com> <2e257eb4b3cc76f78619f5b8c9f95462421762d4.camel@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 (3.28.5-22.el8) X-TM-AS-GCONF: 00 X-Proofpoint-GUID: KWi0vj46FHCmlaHk51d6nWAlfzBNjJ7m X-Proofpoint-ORIG-GUID: qzvBt6Ph8OBH0CI7bS1KTAc8OJn_tjVh Content-Transfer-Encoding: 8bit X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-14_09,2023-09-14_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 lowpriorityscore=0 bulkscore=0 impostorscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 priorityscore=1501 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309140108 X-Rspam-User: X-Stat-Signature: eeob5kf8rhxogz4pktihkeeinn4g86gh X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 69F591A0022 X-HE-Tag: 1694695419-146677 X-HE-Meta: U2FsdGVkX195hn/YGbe1BuxBLi/cIK1ezyAIbt5yrSMXC3voB70VkykvcBnel3XFHeWUJm2vbwgXH8ZVEh7ssAWCUoDkCqrrP3WFOaHyqhkIjgSBdpm2g9ashCs0PingThZ7OG0j5lvpJLLlVrgqT9Cl8FlNcGFIRmlKHw83R84xv4e70z9RdsYJo8ArkGPLujfmJVPK0fGO7OXt28jHIESVHsAQUZ0lfrGF7qY1zcE7n/B3JYRoP6e3i2loB/UR8tU13OtSDbthHvKTUczI40QwYC9WN5OBI+Km8ubIIyFPwE1QhBAiyZQq1wN2ZwM4b5OzEy77l72+cxyHYhavFzq04j3ZWglS5rSsAtALNOtUr+uiLyzTNt7826mi+4OgZFe7i1Yxa1PisgAAmc1PjYrZT+tF1RQuGFSuGX0AhSVMpFC9GAyXYuPg5vjH7CUTC1v8va6F6LCSQJeMVk9iJIaVk/hx7gyMAnkxDAZSrmVQA19ja5CTM3i6FRZ/GGqk3Vk21rFpYenVdZAmTGBH1hJikXa+AZ/i5YZcBIWSL4RxE3E+5HXDTX+97VTHzcr7N5hwP3w6ZJhGSadSXLcnX4QN1zfEWT176Z6hymc29phCz3ZNf8pjdSaTrCps0CFAgEI+DDbUQKJfT7kd2+SoIGRGOxRHoRszCkLuYYwyYC8tvQesj4SelS8gprKLAn2LYtIA2O7XecGEZr5WQmMUd4tNPp+Enl+IJMxdsVlDAT+Kz+cCIgKXmNPVZ60503wC/AmGaR+85UinucJaUeiHx/WiXYR8nYOgEyj/54UbHh2U5QV918yS9nlex2gatTRNZGozstKO/tvY3VwtP+X1jK52NUjFEuxP1xstc+myxPE/U7ko4UVS1tV3KY06/Ac3ISIufrgGkuBX9fbrd9i77dVvr8uArNAZ5R8cQT0bUvtVfoKbza9qBYJJADC8NKqQ0ki5wzGh1etNoam2YKQ TDDxK3e5 mJe3JJP8iAxzpfhdiPzhAopQACMogXy70yLetQMv7kY5a0j3EiK0tbiXNoJDMZa/FA8Xmnjoi1ht7fRJGr+y9NrBotiXYQOmz9T6SD3Gl+QspAio23WpXoTMVFqBX6+AOfItnRYuBIOYDQBgLtPpMCwl/UyA2xQVdwHkUqjwqEsY+9HQptyc3GY9GCMndcM7QcM41oD+7tsoNnwCJM1xNOBOlsn4Il05Kam2X6cyClsmMmMebWykLGsTITx+TXNw+uartdk/fCxFL4PAHfOP7TlZN+JdyhWbsQgWh6JnQ6U6i/wVfpTf9Nva7prEMFaJKMOnnuXGEu+mCBJrrjbWepiSJD95jw+7EfPHMOvln/WtjXQ8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 2023-09-14 at 08:38 +0200, Vlastimil Babka wrote: > On 9/14/23 07:40, Jay Patel wrote: > > On Thu, 2023-09-07 at 15:42 +0200, Vlastimil Babka wrote: > > > On 8/24/23 12:52, Jay Patel wrote: > > > How can increased fraction_size ever result in a lower order? I > > > think > > > it can > > > only result in increased order (or same order). And the > > > simulations > > > with my > > > hack patch don't seem to counter example that. Note previously I > > > did > > > expect > > > the order to be lower (or same) and was surprised by my results, > > > but > > > now I > > > realized I misunderstood the v4 patch. > > > > Hi, Sorry for late reply as i was on vacation :) > > > > You're absolutely > > right. Increasing the fraction size won't reduce the order, and I > > apologize for any confusion in my previous response. > > No problem, glad that it's cleared :) > > > > > 2) Have also seen reduction in overall slab cache numbers as > > > > because of > > > > increasing page order > > > > > > I think your results might be just due to randomness and could > > > turn > > > out > > > different with repeating the test, or converge to be the same if > > > you > > > average > > > multiple runs. You posted them for "160 CPUs with 64K Page size" > > > and > > > if I > > > add that combination to my hack print, I see the same result > > > before > > > and > > > after your patch: > > > > > > Calculated slab orders for page_shift 16 nr_cpus 160: > > > 8 0 > > > 1824 1 > > > 3648 2 > > > 7288 3 > > > 174768 2 > > > 196608 3 > > > 524296 4 > > > > > > Still, I might have a bug there. Can you confirm there are actual > > > differences with a /proc/slabinfo before/after your patch? If > > > there > > > are > > > none, any differences observed have to be due to randomness, not > > > differences > > > in order. > > > > Indeed, to eliminate randomness, I've consistently gathered data > > from > > /proc/slabinfo, and I can confirm a decrease in the total number of > > slab caches. > > > > Values as on 160 cpu system with 64k page size > > Without > > patch 24892 slab caches > > with patch 23891 slab caches > > I would like to see why exactly they decreased, given what the patch > does it > has to be due to getting a higher order slab pages. So the values of > " " columns should increase for some caches > - > which ones and what is their ? yes correct, increase in page order for a slab cache will result in increasing values of " " I just check total numbers of slab cache, so let me check this values in details and will get back with objsize :) > > > > Going back to the idea behind your patch, I don't think it makes > > > sense to > > > try increase the fraction only for higher-orders. Yes, with 1/16 > > > fraction, > > > the waste with 64kB page can be 4kB, while with 1/32 it will be > > > just > > > 2kB, > > > and with 4kB this is only 256 vs 128bytes. However the object > > > sizes > > > and > > > counts don't differ with page size, so with 4kB pages we'll have > > > more > > > slabs > > > to host the same number of objects, and the waste will accumulate > > > accordingly - i.e. the fraction metric should be independent of > > > page > > > size > > > wrt resulting total kilobytes of waste. > > > > > > So maybe the only thing we need to do is to try setting it to 32 > > > initial > > > value instead of 16 regardless of page size. That should > > > hopefully > > > again > > > show a good tradeoff for 4kB as one of the earlier versions, > > > while on > > > 64kB > > > it shouldn't cause much difference (again, none at all with 160 > > > cpus, > > > some > > > difference with less than 128 cpus, if my simulations were > > > correct). > > > > > Yes, We can modify the default fraction size to 32 for all page > > sizes. > > I've noticed that on a 160 CPU system with a 64K page size, there's > > a > > noticeable change in the total memory allocated for slabs – it > > decreases. > > > > Alright, I'll make the necessary changes to the patch, setting the > > fraction size default to 32, and I'll post v5 along with some > > performance metrics. > > Could you please also check my cleanup series at > > https://lore.kernel.org/all/20230908145302.30320-6-vbabka@suse.cz/ > > (I did Cc you there). If it makes sense, I'd like to apply the > further > optimization on top of those cleanups, not the other way around. > > Thanks! > I've just gone through that patch series,and yes we can adjust the fraction size related change within that series :) > > > > > > > > Anyway my point here is that this evaluation approach might > > > > > be > > > > > useful, even > > > > > if it's a non-upstreamable hack, and some postprocessing of > > > > > the > > > > > output is > > > > > needed for easier comparison of before/after, so feel free to > > > > > try > > > > > that out. > > > > > > > > Thank you for this details test :) > > > > > BTW I'll be away for 2 weeks from now, so further feedback > > > > > will > > > > > have > > > > > to come > > > > > from others in that time... > > > > > > > > > Do we have any additional feedback from others on the same > > > > matter? > > > > > > > > Thank > > > > > > > > Jay Patel > > > > > > Thanks! > > > > > > -- > > > > > > Hyeonggon