From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44D26C531DF for ; Mon, 19 Aug 2024 02:36:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA9436B0096; Sun, 18 Aug 2024 22:36:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B57D86B0099; Sun, 18 Aug 2024 22:36:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F8EB6B009A; Sun, 18 Aug 2024 22:36:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 817CA6B0096 for ; Sun, 18 Aug 2024 22:36:46 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3E180160EFA for ; Mon, 19 Aug 2024 02:36:46 +0000 (UTC) X-FDA: 82467432012.14.3F5CE12 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf04.hostedemail.com (Postfix) with ESMTP id 4FD3840012 for ; Mon, 19 Aug 2024 02:36:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=E3wyRMXr; spf=pass (imf04.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.170 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724034989; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W7DhXcrCBAYkpAcK/K1Vjg1bkBS0JcnGfWq1mQe7MK4=; b=l+kSmjQCaMlWKmZCe2WjXm75z71+rnvo8CKfdfC+7h/Fe1ymWSiRDcdiEyIxuDY/ffLuHx vRbWMmtb7sYLu2G/zNLPgFskQtRsesM+neFJLWFHav3HtceySm1r7gPk9rI9XfWwsN+FDp TFnA0xrdeoK3DwY5DSNJJt7N3E0p1rs= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=E3wyRMXr; spf=pass (imf04.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.170 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724034989; a=rsa-sha256; cv=none; b=CGVD4IgfqUryS1xRGbDBOS2W7w2cuk1sf7QrXaqGt3FOZZHEZ2GZQU02GtzeWC+mSCx7Fd EngulYXwvQVxyyt68iy+mLjvPqy4fii94KXEvVwwf5zoHvV5+r1o/V76diYikOblrAPglg QMlPA0m4FsVcbUNbQgJ3zDC+uhfPqho= Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7a1d3874c1eso222176985a.2 for ; Sun, 18 Aug 2024 19:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724035003; x=1724639803; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=W7DhXcrCBAYkpAcK/K1Vjg1bkBS0JcnGfWq1mQe7MK4=; b=E3wyRMXr3qPxyDE0KiBszUADT4ub+KN6xKdy0Atvdy+XBprWS7mQKQamB0aO4zpYAK QeYAMRW5afsKlsm3Rnru08o9iFxrT0t4dOi+qbM90C33efYM3aSkHrRjnPKQSY9250/e gV+feaD0AmesiCVoy85XNpUXZvrxNmq0Zk9UU6Mv1LjqnGxPIREQ/X3ph5Cw8RYut5MB SRG4jNQL7WAeuNV3LuQ0ryH0AFKCvyy5yTfhpatNF9CKeV7QoGSt6Ne19G7yyoU4yDMd uqvC5bdD/UMiqlV38ObFKtwXh5dBIax1MmqQA1qUy8ZX9/CW1UoqZSUGGclBO75BC9fI 8AxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724035003; x=1724639803; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W7DhXcrCBAYkpAcK/K1Vjg1bkBS0JcnGfWq1mQe7MK4=; b=L6uLq/p2mkBE+YQyo4BtKC/H9YYReHtHP3YEQ1VYF8Z1fWK/+8PbT20J0brOba12TW VQNBZ9Dq5yIGmfktzD8hApp0o27wV8sDaYBRjPQjVPsEnc+MemAcP74KIJH4x7CCQ5qa 05ZiFLFSck2djN8O8HZoi1FvpXXqS01SwKJHmuKrybMVxn1Adm0xFAxgXZQvDXBw+HHV LLs3WBBJ0GsRHFbQYuvMGAp6YKtEWZmkUQgHrcVOlpt/ogOyTIZ7xd1FJl0IMNtK2Nvt LEkDVA5lSo914t6Hm3a95LA4Zf6m3+XOhXSTuJS8QygZ9jWVXnYDpySbgeoqjbsQPQJz 5muw== X-Forwarded-Encrypted: i=1; AJvYcCUaoGhlJtp8HJ0fPh+goj1yPoayLejwfJOT7WEHtnfMd+KdUn9SJ+XdWXu73b1MMFXF3DtIugc7V+CXr6MeVMkmTA4= X-Gm-Message-State: AOJu0YwHwDfQn9NEvg4N1KJyszkAXd5SEQz2u0HofRudTqCsAvnUqlLj OozPpG2etVhtNRh1c56171hT669svh7NNQwx7tT7jM7CeXQmJxao X-Google-Smtp-Source: AGHT+IHvMXHyn46aiUdvzUCbod4ZY+UpH2bw6xWEnCGIPdBvEutX8p5dNQKvex4DDpzO5RSgrfyzCw== X-Received: by 2002:a05:620a:40d0:b0:79f:dc6:e611 with SMTP id af79cd13be357-7a5069c5b27mr999121585a.53.1724035003106; Sun, 18 Aug 2024 19:36:43 -0700 (PDT) Received: from [172.20.6.203] ([64.20.177.247]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4ff051635sm397254785a.31.2024.08.18.19.36.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 18 Aug 2024 19:36:42 -0700 (PDT) Message-ID: <66241708-1294-496b-a8f9-316b2b20398d@gmail.com> Date: Sun, 18 Aug 2024 22:36:41 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 0/6] mm: split underutilized THPs To: Hugh Dickins Cc: akpm@linux-foundation.org, linux-mm@kvack.org, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, ryncsn@gmail.com, ak@linux.intel.com, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com References: <20240813120328.1275952-1-usamaarif642@gmail.com> <1e6f3b38-d309-e63f-bca0-5093e152f7d7@google.com> Content-Language: en-US From: Usama Arif In-Reply-To: <1e6f3b38-d309-e63f-bca0-5093e152f7d7@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Stat-Signature: 3gnaacpgzsnqbys77pe3tz46pjjtwerr X-Rspamd-Queue-Id: 4FD3840012 X-Rspamd-Server: rspam11 X-HE-Tag: 1724035004-905146 X-HE-Meta: U2FsdGVkX1//IwtHU5YgpKezvI8hWDz+FJAoMm1ICTPEDX8e2TDfEbWt5jRXe3fTcxNSf+KreJkyqG6T2RKTRmj5GlJfN4lyCppjCYJLAqUwPb82x44O4Xc5b8cCBP7FvGBv1xbwDFQ9XziKMIa+0NgSf375H5mh6H1YteMb/PkWxDke9P3uJn7ZDSiIlb+RkIPfCTzvO4j8wPb3bWjfKrVz8IbvW/HQt/YekH/WrMSqAO1TKxAKJM7wSZ0VhRCw4lR5uYfFIB4madcWb0z4KJZE/7k1PvhtRIf1Bx3PfXIYK7Lpu2GHZ++9m9WWcMAy8WX5sb9/Pxhx7pg6pso6LWGLJXTIfVxMN3nE+kc0F8/+mdri0n1CMGE8uKAezJ2UJRJXwIuqC0KcFNf3CXisCGL49h+/0OkWy+quMswZ2XIlveSDCWtdQ/JNiH+/i+FC5DQWfGKJ/IdapyRRjYVX2JeabEZ11dPZwWoz0j/nKUFjQ1bU9Nm+qJCn0UD55wCc3GJrGvMHDA3qgxyxgRxTkbCOEVtzP1vV5H1UkfuuXiqrUWyQqRdeqqO4rLk//P+P3vYRmBL5DnpKgGfCGEEIbuAwdXVQadXbqKjWum19vAsdDk+bKXX2mO+7RpPz+dfKLDb9NVSbNCrIm7M96pF3lBAMAtr6kRcwoGcY/Ovl2B62i31lELDT93ROk/NCMiiWzUe92ZapF+uSXG9PPYr3lupGekpli60AQeN7kSb176HyISMEtf4oYX/Hc6SHd+ISZXIsRYeOykNyP7cgh+LFROcfxsFoCb1Ga/qJ98p55bLNo1DfQ0i2Ydml2eQof2Yf57QwGdx6JWx5++YwVPFLHM44Ps0UrooTLuVIE46rxjc7AkRrraJ3Gk4hZ/WjKD6/DW7krX95WoQiIg967c6pDGcB/0XDqJGZJc5ckvWJMi4cGZYWreAtYoE0xn1vdzDTpU+jxBgTkQCaTN1MAUY JO/eLoQn 8TIfwkpX8DSohStrjbyEOawyDUkXk3qOR9Rws8DrVEUgiUzbpQ8eFC3RXFMpUtj+EAJqTcIM5nDVzPIWU7a/XK3olHA2vcNcAAzN6MOzubn7RyvDzP7dw4OHaBQQgrYLSjlu2FiReRSzl8HSkzYDdoDk/JvMKs2Q8KFa00bD5obVtIAv1kRnjRcmZ0vieY9R3C9vnvetpUhMAfFctGmLRhArmOt48VzcDFAUs0VhoABwAvjeUCAx7oodEf+/opxL5i9Rxl9YLWIk9T+MjSTJQGlo7hPgu/TRrkzjD5T9SRGh3r22bJ00371m+2B2cb9BPyYpIB7U6txaq9a+VvECvE0Wqg4CTMBdcJWZnEyXQDA1Rc7wLc0ALC+/Q1B1Pa6/cen2G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 18/08/2024 06:13, Hugh Dickins wrote: > On Tue, 13 Aug 2024, Usama Arif wrote: > >> The current upstream default policy for THP is always. However, Meta >> uses madvise in production as the current THP=always policy vastly >> overprovisions THPs in sparsely accessed memory areas, resulting in >> excessive memory pressure and premature OOM killing. >> Using madvise + relying on khugepaged has certain drawbacks over >> THP=always. Using madvise hints mean THPs aren't "transparent" and >> require userspace changes. Waiting for khugepaged to scan memory and >> collapse pages into THP can be slow and unpredictable in terms of performance >> (i.e. you dont know when the collapse will happen), while production >> environments require predictable performance. If there is enough memory >> available, its better for both performance and predictability to have >> a THP from fault time, i.e. THP=always rather than wait for khugepaged >> to collapse it, and deal with sparsely populated THPs when the system is >> running out of memory. >> >> This patch-series is an attempt to mitigate the issue of running out of >> memory when THP is always enabled. During runtime whenever a THP is being >> faulted in or collapsed by khugepaged, the THP is added to a list. >> Whenever memory reclaim happens, the kernel runs the deferred_split >> shrinker which goes through the list and checks if the THP was underutilized, >> i.e. how many of the base 4K pages of the entire THP were zero-filled. >> If this number goes above a certain threshold, the shrinker will attempt >> to split that THP. Then at remap time, the pages that were zero-filled are >> mapped to the shared zeropage, hence saving memory. This method avoids the >> downside of wasting memory in areas where THP is sparsely filled when THP >> is always enabled, while still providing the upside THPs like reduced TLB >> misses without having to use madvise. >> >> Meta production workloads that were CPU bound (>99% CPU utilzation) were >> tested with THP shrinker. The results after 2 hours are as follows: >> >> | THP=madvise | THP=always | THP=always >> | | | + shrinker series >> | | | + max_ptes_none=409 >> ----------------------------------------------------------------------------- >> Performance improvement | - | +1.8% | +1.7% >> (over THP=madvise) | | | >> ----------------------------------------------------------------------------- >> Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%) >> ----------------------------------------------------------------------------- >> max_ptes_none=409 means that any THP that has more than 409 out of 512 >> (80%) zero filled filled pages will be split. >> >> To test out the patches, the below commands without the shrinker will >> invoke OOM killer immediately and kill stress, but will not fail with >> the shrinker: >> >> echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none >> mkdir /sys/fs/cgroup/test >> echo $$ > /sys/fs/cgroup/test/cgroup.procs >> echo 20M > /sys/fs/cgroup/test/memory.max >> echo 0 > /sys/fs/cgroup/test/memory.swap.max >> # allocate twice memory.max for each stress worker and touch 40/512 of >> # each THP, i.e. vm-stride 50K. >> # With the shrinker, max_ptes_none of 470 and below won't invoke OOM >> # killer. >> # Without the shrinker, OOM killer is invoked immediately irrespective >> # of max_ptes_none value and kills stress. >> stress --vm 1 --vm-bytes 40M --vm-stride 50K >> >> v2 -> v3: >> - Use my_zero_pfn instead of page_to_pfn(ZERO_PAGE(..)) (Johannes) >> - Use flags argument instead of bools in remove_migration_ptes (Johannes) >> - Use a new flag in folio->_flags_1 instead of folio->_partially_mapped >> (David Hildenbrand). >> - Split out the last patch of v2 into 3, one for introducing the flag, >> one for splitting underutilized THPs on _deferred_list and one for adding >> sysfs entry to disable splitting (David Hildenbrand). >> >> v1 -> v2: >> - Turn page checks and operations to folio versions in __split_huge_page. >> This means patches 1 and 2 from v1 are no longer needed. >> (David Hildenbrand) >> - Map to shared zeropage in all cases if the base page is zero-filled. >> The uffd selftest was removed. >> (David Hildenbrand). >> - rename 'dirty' to 'contains_data' in try_to_map_unused_to_zeropage >> (Rik van Riel). >> - Use unsigned long instead of uint64_t (kernel test robot). >> >> Alexander Zhu (1): >> mm: selftest to verify zero-filled pages are mapped to zeropage >> >> Usama Arif (3): >> mm: Introduce a pageflag for partially mapped folios >> mm: split underutilized THPs >> mm: add sysfs entry to disable splitting underutilized THPs >> >> Yu Zhao (2): >> mm: free zapped tail pages when splitting isolated thp >> mm: remap unused subpages to shared zeropage when splitting isolated >> thp > > Sorry, I don't have time to review this, but notice you're intending > a v4 of the series, so want to bring up four points quickly before that. > > 1. Even with the two fixes to 1/6 in __split_huge_page(), under load > this series causes system lockup, with interrupts disabled on most CPUs. > > The error is in deferred_split_scan(), where the old code just did > a list_splice_tail() under split_queue_lock, but this series ends up > doing more there, including a folio_put(): deadlock when racing, and > that is the final folio_put() which brings refcount down to 0, which > then wants to take split_queue_lock. > > The patch I've been using successfully on 6.11-rc3-next-20240816 below: > I do have other problems with current mm commits, so have not been able > to sustain a load for very long, but suspect those problems unrelated > to this series. Please fold this fix, or your own equivalent, into > your next version. > > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -3270,7 +3270,8 @@ static void __split_huge_page(struct pag > > folio_clear_active(new_folio); > folio_clear_unevictable(new_folio); > - if (!folio_batch_add(&free_folios, folio)) { > + list_del(&new_folio->lru); > + if (!folio_batch_add(&free_folios, new_folio)) { > mem_cgroup_uncharge_folios(&free_folios); > free_unref_folios(&free_folios); > } > @@ -3706,42 +3707,37 @@ static unsigned long deferred_split_scan > bool did_split = false; > bool underutilized = false; > > - if (folio_test_partially_mapped(folio)) > - goto split; > - underutilized = thp_underutilized(folio); > - if (underutilized) > - goto split; > - continue; > -split: > + if (!folio_test_partially_mapped(folio)) { > + underutilized = thp_underutilized(folio); > + if (!underutilized) > + goto next; > + } > if (!folio_trylock(folio)) > - continue; > - did_split = !split_folio(folio); > - folio_unlock(folio); > - if (did_split) { > - /* Splitting removed folio from the list, drop reference here */ > - folio_put(folio); > + goto next; > + if (!split_folio(folio)) { > + did_split = true; > if (underutilized) > count_vm_event(THP_UNDERUTILIZED_SPLIT_PAGE); > split++; > } > - } > - > - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); > - /* > - * Only add back to the queue if folio is partially mapped. > - * If thp_underutilized returns false, or if split_folio fails in > - * the case it was underutilized, then consider it used and don't > - * add it back to split_queue. > - */ > - list_for_each_entry_safe(folio, next, &list, _deferred_list) { > - if (folio_test_partially_mapped(folio)) > - list_move(&folio->_deferred_list, &ds_queue->split_queue); > - else { > + folio_unlock(folio); > +next: > + /* > + * split_folio() removes folio from list on success. > + * Only add back to the queue if folio is partially mapped. > + * If thp_underutilized returns false, or if split_folio fails > + * in the case it was underutilized, then consider it used and > + * don't add it back to split_queue. > + */ > + if (!did_split && !folio_test_partially_mapped(folio)) { > list_del_init(&folio->_deferred_list); > ds_queue->split_queue_len--; > } > folio_put(folio); > } > + > + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); > + list_splice_tail(&list, &ds_queue->split_queue); > spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); > > /* > > 2. I don't understand why there needs to be a new PG_partially_mapped > flag, with all its attendant sets and tests and clears all over. Why > can't deferred_split_scan() detect that case for itself, using the > criteria from __folio_remove_rmap()? I see folio->_nr_pages_mapped > is commented "Do not use outside of rmap and debug code", and > folio_nr_pages_mapped() is currently only used from mm/debug.c; but > using the info already maintained is preferable to adding a PG_flag > (and perhaps more efficient - skips splitting when _nr_pages_mapped > already fell to 0 and folio will soon be freed). > > 3. Everything in /sys/kernel/mm/transparent_hugepage/ is about THPs, > so please remove the "thp_" from "thp_low_util_shrinker" - > "shrink_underused" perhaps. And it needs a brief description in > Documentation/admin-guide/mm/transhuge.rst. > > 4. Wouldn't "underused" be better than "underutilized" throughout? > > Hugh Thanks for the review, especially for pointing out the deadlock. I have addressed points 1, 3 and 4 in v4, point 2 was addressed by David.