From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7003ECA0FEB for ; Fri, 1 Sep 2023 17:29:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8CA12440149; Fri, 1 Sep 2023 13:29:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 851998D0002; Fri, 1 Sep 2023 13:29:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7187B440149; Fri, 1 Sep 2023 13:29:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5EA808D0002 for ; Fri, 1 Sep 2023 13:29:19 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2BAF31402B3 for ; Fri, 1 Sep 2023 17:29:19 +0000 (UTC) X-FDA: 81188714838.09.D1C689F Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf15.hostedemail.com (Postfix) with ESMTP id 53826A0016 for ; Fri, 1 Sep 2023 17:29:17 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=o0h8Fspl; spf=pass (imf15.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693589357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NouKehUZ1/ja07DRWOckLtQmRtHkQt8ow3jpBfelpLc=; b=wLFBJNMInhgmUjtvmeI/JAXMvg8t3dLConpFfibiqdrhKXi5e44N7KkEnBBe3qUXslWlkJ MRKb6Zu2BU1b+A1Iza4L29NC523kHAgd0NZkKJJGhknFlTKQGDM8ckHJMfTwMJOgOHrIej 4k9alrFAtFytr2bcwLhOcsMthQTqnmI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693589357; a=rsa-sha256; cv=none; b=oomJbOVITKGphKafJbwVpuxGtY8mBU7v1B1qIZ/2fAggdbhITLHsNGnqVwrYXR3hSl4/Si HyWGLcPVMADD5ZDog6AGc4WoZdrRaqy7MewOKnjOh+Wi8eArDkTdAEEPGzwPQkp9ayzS4P 87rj0YlVwwAe02mFyio2WTGMzx0ezmc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=o0h8Fspl; spf=pass (imf15.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-68c3ec0578bso1895457b3a.2 for ; Fri, 01 Sep 2023 10:29:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693589356; x=1694194156; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=NouKehUZ1/ja07DRWOckLtQmRtHkQt8ow3jpBfelpLc=; b=o0h8FsplH/u9Ju0XJx2xZTDa8K5odPrfQ91hnfVPfhUH/639OeT4POCeNz3a5MI78s q1paN+gcTbiCmwsl5iFzona6Eo+iHlwQECkCYlbtSV8MZJRnzgi1YIqvHtxYXQ4wja0R O0L70J1Nle6Psh2bNCWumqSL4HutEJVfq6SoZ+R5BIsCKQTTsioTFElmgnYkOUBJAnWs YJErvSOG39KaqLvmImxxA7F7d2OK/ah3eYgX0u1pJ/UZT1OsHYq6C5eYJCHC6R2P0Um+ zWvHn3BcfV0p6gUxyCoogZvvQ49jt0R3mytho1BAAGtV+BsqR87Qv+WnkF3J2zzLG2ta kTcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693589356; x=1694194156; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NouKehUZ1/ja07DRWOckLtQmRtHkQt8ow3jpBfelpLc=; b=PGGchjIOofkYKXtPtEvn7Q4QXhgiZnX0lvTy5SGTOCvQrBUeibO1ifdbWBsDuXHQti jY77VefwaKoZ7U58g0vvP3EjmUAXTRnJ+piTZqjgyYqf7EZKit3/LNcRjWay9zdJe6Bm 9YlnQ/szIhe053u//xslm8vJtCWtdeC4X+zI/k2qx3ybFiYcApcHBHzTyad9o4udEgNF 2jNbTbyXx4m2BozdJ1Shfe6LLi6bzL5OzjqCiianXJi9eYPVZB/X7OQtKrxk1xtwcejy cPk9o/UPONa3HPrq5wKrfSZ8d462l+t0t1wsrSXFGTi0Q8ZEd+MiNFRc8wuU/Vc5q2Ec XPOA== X-Gm-Message-State: AOJu0YwFEskXIQfRF7gac5od0wLtfUTH7+NmUb15xG0bQWtm7bOimA2w 1TFghONFX6Be5VVzhoSa+Ek200qra7Hwg6x2WLs= X-Google-Smtp-Source: AGHT+IHCD/btXitk61S80sfG4nrcWxOI8Bma2QDgeMeK2+90bnvuVn1s+/5lUQf8Q+C6mRfhybCuIzBe05vJ5jmnlds= X-Received: by 2002:a17:90a:e015:b0:26d:1f85:556 with SMTP id u21-20020a17090ae01500b0026d1f850556mr2997550pjy.45.1693589356045; Fri, 01 Sep 2023 10:29:16 -0700 (PDT) MIME-Version: 1.0 References: <20230817035155.84230-1-liusong@linux.alibaba.com> In-Reply-To: From: Yang Shi Date: Fri, 1 Sep 2023 10:29:04 -0700 Message-ID: Subject: Re: [PATCH] mm/khugepaged: increase transparent_hugepage_recommend_disable parameter to disable active modification of min_free_kbytes To: Liu Song Cc: corbet@lwn.net, akpm@linux-foundation.org, paulmck@kernel.org, rdunlap@infradead.org, catalin.marinas@arm.com, dave.hansen@linux.intel.com, rostedt@goodmis.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 53826A0016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ckxm13zjffagxrpb3c3ctyqftjb7zb9q X-HE-Tag: 1693589357-18538 X-HE-Meta: U2FsdGVkX19Yw0HRM+8qAcRBMnMvcW5wsJpiM0UcOMGnzUNcb19hmqIA5WYtHjhDAlTudH1LTJCaR3ibhmv5NDDl9U064/m1K4KIn1BFWprhcdRF9yzPJR4hkA2G0upMu2N2fDJN3ElNDJSD1+zYsRamiALRPuyx66a4n6990ZOq0giN/HLqm+bbBHVq3RlRdu09oncRlejUbAQ/i6NF1pD5QVTZm/zCN1W100Yll1Jtml5DopPq0upEbD5uIBGFzv8q/AcZIvDlPV21Og66KAXSkcOqABNw+zTdMGhR+nEMWnVwN7eDqyfRffCZ+FfxKhx4JzeS0zE2oxuNCEK31Lz0eDTknnlxH85+hafy5FI6JIbSL9J+GVz901W2279QL5q1h+mGTMbc+h9jPL8ykO6ZXsgpnoqUh8iF7PkB95vyYv5MWw8SeoqeQxFt2JPENVhqfjADbqCe3iMoeqNEGhi2Nma7utGFpuB17UcVcdtGUfHO7WEfj6oJIYc6HwaDO2uqNtPvUqCWIMw9JYjjSslQEce3lZCtHI+qsKA0DJjn3JE1fim0gUkYU63f4i+Igs44A8+3WOyuzqdZpXXEUNlBI6vBKQcwqgz+1Z2aW4NGwrgP3O/X6Nfo4KTuUFjpSnH+lhQHpA8rjsaI8hAqnemWR5r35l7bPdEwf2XL8EGJmNgwcS3SIjls5a0Y4/0jqZI8q1XFe42R5Vw7AXsscfgY29dCd7HZbJY28RBEm7lHUsPI+dSeNb5RDrJSFNCYoNtkfS0JDB62/zVpT4NZFXIwcWIAHo5vXcEOBcgHbd/OjE5qF0jqBS24hb7DG6WoXp8/5+DEEYMcIin7zIyY3N/Eg0zfcvK9SaKvII5r4t5yr///gRJfgfIVeU9gtUHBvSvFmnjFZLtwFnJpTh55okYb0pPtH/S4PP5xp5kpflHjNRL0Kkyv6IfpLLy8ngWkSbONWXGrC7ftouBMB1Y xT4JlAte qKilwulgMaUMAjGOW296LRwIHpl1J5OwAv/0MUYmXZ6WOblRnN7Q2BG0Bj0EdLGY9ji4qEbyaTyGHn235U85kymE0FoP4K72MdcwCLNnXttBIOBPEwRh4vMlLa5+h7gKAq5d+8AAPNKPEXGJn+f5/K+007Xmw8LLBLaWNozr0wp3Xh0amdtgyMhwwq0V+MduZ40ztv+93Qw1VKTct1j19fiSyvf3zkwsR6xJA133+ctKkPVL75TtyrHgGaW6Duju1m0knF8+XO0FPS72ZJ/Q8zwzI3nN7kZO1vddNUEQvBF81xddmLjTo0+lFKJaJC29BTzM1DSwrJ/DVGGG3irT+cA6OMUjVfZIBd8I2ulsDLOIOvT3odOXe9D3AG26QkzJXrixm3ZsPBgXQIyc+HUiHLin8UbjUqGXS/Umc8ZbIENPaz7UGxXc9qE0lVq+UnJULsG7/Bhc4PheLi/o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 31, 2023 at 7:30=E2=80=AFAM Liu Song wrote: > > =E5=9C=A8 2023/8/30 04:04, Yang Shi =E5=86=99=E9=81=93: > > > On Wed, Aug 16, 2023 at 8:52=E2=80=AFPM Liu Song wrote: > >> In the arm64 environment, when PAGESIZE is 4K, the "pageblock_nr_pages= " > >> value is 512, and the recommended min_free_kbytes in > >> "set_recommended_min_free_kbytes" usually does not exceed 44MB. > >> > >> However, when PAGESIZE is 64K, the "pageblock_nr_pages" value is 8192, > >> and the recommended min_free_kbytes in "set_recommended_min_free_kbyte= s" > >> is 8192 * 2 * (2 + 9) * 64K, which directly increases to 11GB. > >> > >> According to this calculation method, due to the modification of min_f= ree_kbytes, > >> the reserved memory in my 128GB memory environment reaches 10GB, and M= emAvailable > >> is correspondingly reduced by 10GB. > >> > >> In the case of PAGESIZE 64K, transparent hugepages are 512MB, and we o= nly > >> need them to be used on demand. If transparent hugepages cannot be all= ocated, > >> falling back to regular 64K pages is completely acceptable. > >> > >> Therefore, we added the transparent_hugepage_recommend_disable paramet= er > >> to disable active modification of min_free_kbytes, thereby meeting our > >> requirements for transparent hugepages in the 64K scenario, and it wil= l > >> not excessively reduce the available memory. > > Thanks for debugging this. I agree 11GB for min_free_kbytes is too > > much. But a kernel parameter sounds overkilling to me either. IMHO we > > just need to have a better scaling for bigger base page size. For > > example, we just keep one or two pageblock for min_free_kbytes when > > the base page size is bigger than 4K. > > > Thank you very much for your advice, but how do we determine the number > of pageblocks? TBH, I can't tell. I don't have a magic number... > This is a difficult number to determine. When PAGESIZE is 64K, arm64 > supports hugepages > of 2M, 512M, and 16G, which can meet the requirements of scenarios that > require hugepages. > > However, transparent huge pages can only support 512M, and 512M is a > very large number, so > enabling transparent huge pages should be carefully considered, not to > mention whether it makes > sense to reserve such a large amount of memory. > > Therefore, I think that in the scenario of 64K PAGESIZE, it might also > be a good choice to directly > cancel set_recommended_min_free_kbytes? It should be ok too. There shouldn't be too many real life workloads running with this configuration other than some Java users may run some workloads with 64K base page + THP AFAIK. But it should be not very common. > > Thanks > > > >> Signed-off-by: Liu Song > >> --- > >> .../admin-guide/kernel-parameters.txt | 5 +++++ > >> mm/khugepaged.c | 20 +++++++++++++++++= +- > >> 2 files changed, 24 insertions(+), 1 deletion(-) > >> > >> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documen= tation/admin-guide/kernel-parameters.txt > >> index 654d0d921101..612bdf601cce 100644 > >> --- a/Documentation/admin-guide/kernel-parameters.txt > >> +++ b/Documentation/admin-guide/kernel-parameters.txt > >> @@ -6553,6 +6553,11 @@ > >> See Documentation/admin-guide/mm/transhuge.rs= t > >> for more details. > >> > >> + transparent_hugepage_recommend_disable > >> + [KNL,THP] > >> + Can be used to disable transparent hugepage to= actively modify > >> + /proc/sys/vm/min_free_kbytes during enablement= process. > >> + > >> trusted.source=3D [KEYS] > >> Format: > >> This parameter identifies the trust source as= a backend > >> diff --git a/mm/khugepaged.c b/mm/khugepaged.c > >> index 78fc1a24a1cc..ac40c618f4f6 100644 > >> --- a/mm/khugepaged.c > >> +++ b/mm/khugepaged.c > >> @@ -88,6 +88,9 @@ static unsigned int khugepaged_max_ptes_none __read_= mostly; > >> static unsigned int khugepaged_max_ptes_swap __read_mostly; > >> static unsigned int khugepaged_max_ptes_shared __read_mostly; > >> > >> +/* default enable recommended */ > >> +static unsigned int transparent_hugepage_recommend __read_mostly =3D = 1; > >> + > >> #define MM_SLOTS_HASH_BITS 10 > >> static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BIT= S); > >> > >> @@ -2561,6 +2564,11 @@ static void set_recommended_min_free_kbytes(voi= d) > >> goto update_wmarks; > >> } > >> > >> + if (!transparent_hugepage_recommend) { > >> + pr_info("do not allow to recommend modify min_free_kby= tes\n"); > >> + return; > >> + } > >> + > >> for_each_populated_zone(zone) { > >> /* > >> * We don't need to worry about fragmentation of > >> @@ -2591,7 +2599,10 @@ static void set_recommended_min_free_kbytes(voi= d) > >> > >> if (recommended_min > min_free_kbytes) { > >> if (user_min_free_kbytes >=3D 0) > >> - pr_info("raising min_free_kbytes from %d to %l= u to help transparent hugepage allocations\n", > >> + pr_info("raising user specified min_free_kbyte= s from %d to %lu to help transparent hugepage allocations\n", > >> + min_free_kbytes, recommended_min); > >> + else > >> + pr_info("raising default min_free_kbytes from = %d to %lu to help transparent hugepage allocations\n", > >> min_free_kbytes, recommended_min); > >> > >> min_free_kbytes =3D recommended_min; > >> @@ -2601,6 +2612,13 @@ static void set_recommended_min_free_kbytes(voi= d) > >> setup_per_zone_wmarks(); > >> } > >> > >> +static int __init setup_transparent_hugepage_recommend_disable(char *= str) > >> +{ > >> + transparent_hugepage_recommend =3D 0; > >> + return 1; > >> +} > >> +__setup("transparent_hugepage_recommend_disable", setup_transparent_h= ugepage_recommend_disable); > >> + > >> int start_stop_khugepaged(void) > >> { > >> int err =3D 0; > >> -- > >> 2.19.1.6.gb485710b > >> > >>