From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45788C52D7C for ; Sun, 18 Aug 2024 05:13:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 501968D00D8; Sun, 18 Aug 2024 01:13:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B17C8D00B8; Sun, 18 Aug 2024 01:13:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 305F88D00D8; Sun, 18 Aug 2024 01:13:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F2D6E8D00B8 for ; Sun, 18 Aug 2024 01:13:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 45C55A6FB3 for ; Sun, 18 Aug 2024 05:13:43 +0000 (UTC) X-FDA: 82464198726.14.92A04EC Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) by imf23.hostedemail.com (Postfix) with ESMTP id 7C6E9140003 for ; Sun, 18 Aug 2024 05:13:41 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=25aoiHZU; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.161.54 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723958007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KVvU9xxK5dTeMKAjFl5rkhtM+aF2FRQXOAf4rQRLTtc=; b=dkrZ8+CXwffLttigs/zxY4JqxlKoGunfeOIjXgC5UyGSznC5nTkPhpWrXHZYp4tyszFzGu MxsrntxVFInnbXS+YJuTXSyBKG7oceQH98Rwt+cgnyyAR28nwu/SSj1/JYKh6MG34aHLbD 9qNAMjb+HRfnpTwTpzNixNdOB4iJL9Y= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=25aoiHZU; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.161.54 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723958007; a=rsa-sha256; cv=none; b=25tmVIK6otnJHOYeuFvBwimg/RA72QkRXUE5wAo74O6c+eD4d25QNKkvOv0CSRYJP4V1RG ks9zovk+0hfejz40Ugwgp5+NFPo/H8ynuExU/tPLxf+l3Yi8v5p44kK3M4WVzR22xnwAQh JhQkI1QPQ/Zg15RWoQ3bkL0ORQ/tsTY= Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5da775a15e3so1680143eaf.1 for ; Sat, 17 Aug 2024 22:13:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723958020; x=1724562820; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=KVvU9xxK5dTeMKAjFl5rkhtM+aF2FRQXOAf4rQRLTtc=; b=25aoiHZUqBNR/6dE0fTSHo5tbqCzaW2hGPg3wSe3y6CQDM/j+0f7pSYCKULapaJ5Io gHjCQ/4TDQDyrYa+L7WvRm2dbkQLFeJx8GD/qmrPbLEbEqCXJg3e4Af35SxfBpPVLwfi i8IAr8V14Dx0I+d/Vqp5RfZF44QsuvUIIUAcgP/jh8qSqjoUr+YSfyt52GQoRttvEb6A dgHaPiUOiQlJM+DBKE/Hx6sVWATJ2rXUNQuewKcIx8OgGXFcs5Ry19uKOrWYeVHU5XRY wsRrGwYEy9Gni/BiWmV33GT1d/WsS33H3s1ZH1josqn7ZY6Vxj6KG1ex4HUkni/pmkoB 0ALA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723958020; x=1724562820; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KVvU9xxK5dTeMKAjFl5rkhtM+aF2FRQXOAf4rQRLTtc=; b=mEa4znNrxiesSRUCSPH8PVwSye9LrKa08lBJatkLVXENz8/jiJBvHtjAuLhX4lX9cr trBHZH/9KoSoenFag6LTEp83dmVlMWq86G+Do3A+aUpSxa0dYZPqf+IrkSF6zqR0fDgc XiGCKV415mh1TowjJOztIiAFiNpBrWPF5HlHlFkAnCwdsBiEvRYbbRYug+mrmB6hezT1 us8Z9HcDY2SuACr4Kjd+kgP4j42l24vr08A8AeArmHAvdCh9UwQ7CYSmt6iSzBrxMsjJ Pnx91veR0fY+SWkKuzHVYxGgDjbj3THQICzcSQCvCzF3xK0gpNMYQbBpOPAStyWVRDtD 7hrQ== X-Forwarded-Encrypted: i=1; AJvYcCU/gfUxNFNEEoIrW5WiytEoHbrIQYI+9SsQpMUea3KbbGE0GiiZULtympolpPpoKSR7DfQJ43iqOi/8vtjkghlQz80= X-Gm-Message-State: AOJu0Yx4IrFHyJXlpv3d1KSoHA3QUq43Q5Ix1sv+4zy3P5F8bgnCc0FF H7FJasEMHRKeat87tZ82xqUvmdcF17Ec2adxHt8IMu+Hvo56Cl2D1ENKu0DGUA== X-Google-Smtp-Source: AGHT+IEnSew4mblQUWx6K7ns+fH3kd0Pjo6bjaib28/wf6243qJPhvm301wZLvvVUMfJCB8msFnOPg== X-Received: by 2002:a05:6871:e706:b0:25e:b839:cf06 with SMTP id 586e51a60fabf-26fff4b3114mr5701300fac.11.1723958020054; Sat, 17 Aug 2024 22:13:40 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5da8cd69811sm1457955eaf.4.2024.08.17.22.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 22:13:38 -0700 (PDT) Date: Sat, 17 Aug 2024 22:13:27 -0700 (PDT) From: Hugh Dickins To: Usama Arif cc: akpm@linux-foundation.org, linux-mm@kvack.org, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, ryncsn@gmail.com, ak@linux.intel.com, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com Subject: Re: [PATCH v3 0/6] mm: split underutilized THPs In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> Message-ID: <1e6f3b38-d309-e63f-bca0-5093e152f7d7@google.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Stat-Signature: g43kirhjjo5ns3ug9hchpdqe437ydxoo X-Rspamd-Queue-Id: 7C6E9140003 X-Rspamd-Server: rspam11 X-HE-Tag: 1723958021-754325 X-HE-Meta: U2FsdGVkX18/KbcxRljzGETrmDenhudJV50K7zmknJ1SWi/S+n3s9U1V3tOIJ8Y88uBdHzBynsOAYuUim8rx2ZntvC86ocF2xIV1Q8T6E4H8XiGOcUNn1GkdhC6YwqFDT5wGOfhLQQA/mqnNvLSGfHGOfwKuj22dzSy7FVEOsJPVXLoaFkkuFC1PCHNdPaJEfcnsW9iqCJk0hcVvfi4F2Hm9L3XY0qEPUmpMpbqxrxnu8S9wSxC/wpeNFCaj4MoEsxIyhzczjYJc/dYQDHPZH6fL2qy7tbjFVjIzXxLQEcI2o6vM+K6HOeRlVjrgBTzhFMV/NWPSWXo5xxlFH4tw8kX0sgVU9cj1pitv/WEglLjHZhuMJ/HP4rkz+jnVcPhEMFVa5PbBrmfUH8UFq5s07hI7Y9kJQ/32Qap2RQpm78jqh4/rLYQ1r5UReL6OZaMaJnKYgJn/tz83SVMjYoInAclE4wqjESrYqHSrS5bycQaGrFGPBKSUugHdZ7UioAqquFEywsKjAGqG/jXquMQOMxyYKTBg1zRsDadzPg/7KVarcO0mlvCWTJ6X2QwrAIigB8sYGYTn8M9vT6jmkvOao582cv1k1W19yLyGwJhZ4DA6KZDXoasDZVFGcDV5hgpyOqet7Sw48xEDHO5burBKVr/NqCCMJ0IimF5ELc8Sd7blOFdud5XPuq7/1mhcV+WJR8dEy3YVsCuu2uRC6wxzwTn+Z5LoCWrWP8mOzL9bLqPkZmX3JaTf9qfyBaZZYLcdGHR4pGZeDtWMsIcMlxp9tzFvjSTIqI43uhcPWwBUJgsyAU+LFs+xh3zs0dV78+MXagT7WKl4AblhQBYRrfF3MBo9I09D61qN5z0DSJnzTE82AfAAGRdHLaHszI0fMDTp9Uf9FUHa8T0Cruy3eagqdljSYbWXkfFdRzGFNn63ddv1Dn78sGzp9WzOn+h4UvVrvAQx45WZqQIl7IDKdXT 0rjXK5IK PYa3R6IPYMfyiTt+aWfD2JbezuNVk3u3Dsn19o8Tc/JkEkyz/frnt0nDpS0cNrrxac6bkgnhQKG0qjN44vWqKWjUlMYgMLSUkGf4pKjUsRFRT/onGv66d9nwJm1QKChZkQvadLx047MyK+qfGZuQdCw8sy/IRSE3ou6TVchX7YQEpFdnyzUeIaZkvOt98f0/S6pXEgDV/XBVfzrxaxna5D5wsPdvYU3dtNxRG0bTiWSs+zdUVoVRyZbuUMA4XG/a9rmUiZJcBUOmuFFE8lVOEHtaspFVh/6zIse8TvF44qAZ4U6KGW0c69vUCGZOIrnqFjA8UyXBs96+k7Urukz+mkYESlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 13 Aug 2024, Usama Arif wrote: > The current upstream default policy for THP is always. However, Meta > uses madvise in production as the current THP=always policy vastly > overprovisions THPs in sparsely accessed memory areas, resulting in > excessive memory pressure and premature OOM killing. > Using madvise + relying on khugepaged has certain drawbacks over > THP=always. Using madvise hints mean THPs aren't "transparent" and > require userspace changes. Waiting for khugepaged to scan memory and > collapse pages into THP can be slow and unpredictable in terms of performance > (i.e. you dont know when the collapse will happen), while production > environments require predictable performance. If there is enough memory > available, its better for both performance and predictability to have > a THP from fault time, i.e. THP=always rather than wait for khugepaged > to collapse it, and deal with sparsely populated THPs when the system is > running out of memory. > > This patch-series is an attempt to mitigate the issue of running out of > memory when THP is always enabled. During runtime whenever a THP is being > faulted in or collapsed by khugepaged, the THP is added to a list. > Whenever memory reclaim happens, the kernel runs the deferred_split > shrinker which goes through the list and checks if the THP was underutilized, > i.e. how many of the base 4K pages of the entire THP were zero-filled. > If this number goes above a certain threshold, the shrinker will attempt > to split that THP. Then at remap time, the pages that were zero-filled are > mapped to the shared zeropage, hence saving memory. This method avoids the > downside of wasting memory in areas where THP is sparsely filled when THP > is always enabled, while still providing the upside THPs like reduced TLB > misses without having to use madvise. > > Meta production workloads that were CPU bound (>99% CPU utilzation) were > tested with THP shrinker. The results after 2 hours are as follows: > > | THP=madvise | THP=always | THP=always > | | | + shrinker series > | | | + max_ptes_none=409 > ----------------------------------------------------------------------------- > Performance improvement | - | +1.8% | +1.7% > (over THP=madvise) | | | > ----------------------------------------------------------------------------- > Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%) > ----------------------------------------------------------------------------- > max_ptes_none=409 means that any THP that has more than 409 out of 512 > (80%) zero filled filled pages will be split. > > To test out the patches, the below commands without the shrinker will > invoke OOM killer immediately and kill stress, but will not fail with > the shrinker: > > echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none > mkdir /sys/fs/cgroup/test > echo $$ > /sys/fs/cgroup/test/cgroup.procs > echo 20M > /sys/fs/cgroup/test/memory.max > echo 0 > /sys/fs/cgroup/test/memory.swap.max > # allocate twice memory.max for each stress worker and touch 40/512 of > # each THP, i.e. vm-stride 50K. > # With the shrinker, max_ptes_none of 470 and below won't invoke OOM > # killer. > # Without the shrinker, OOM killer is invoked immediately irrespective > # of max_ptes_none value and kills stress. > stress --vm 1 --vm-bytes 40M --vm-stride 50K > > v2 -> v3: > - Use my_zero_pfn instead of page_to_pfn(ZERO_PAGE(..)) (Johannes) > - Use flags argument instead of bools in remove_migration_ptes (Johannes) > - Use a new flag in folio->_flags_1 instead of folio->_partially_mapped > (David Hildenbrand). > - Split out the last patch of v2 into 3, one for introducing the flag, > one for splitting underutilized THPs on _deferred_list and one for adding > sysfs entry to disable splitting (David Hildenbrand). > > v1 -> v2: > - Turn page checks and operations to folio versions in __split_huge_page. > This means patches 1 and 2 from v1 are no longer needed. > (David Hildenbrand) > - Map to shared zeropage in all cases if the base page is zero-filled. > The uffd selftest was removed. > (David Hildenbrand). > - rename 'dirty' to 'contains_data' in try_to_map_unused_to_zeropage > (Rik van Riel). > - Use unsigned long instead of uint64_t (kernel test robot). > > Alexander Zhu (1): > mm: selftest to verify zero-filled pages are mapped to zeropage > > Usama Arif (3): > mm: Introduce a pageflag for partially mapped folios > mm: split underutilized THPs > mm: add sysfs entry to disable splitting underutilized THPs > > Yu Zhao (2): > mm: free zapped tail pages when splitting isolated thp > mm: remap unused subpages to shared zeropage when splitting isolated > thp Sorry, I don't have time to review this, but notice you're intending a v4 of the series, so want to bring up four points quickly before that. 1. Even with the two fixes to 1/6 in __split_huge_page(), under load this series causes system lockup, with interrupts disabled on most CPUs. The error is in deferred_split_scan(), where the old code just did a list_splice_tail() under split_queue_lock, but this series ends up doing more there, including a folio_put(): deadlock when racing, and that is the final folio_put() which brings refcount down to 0, which then wants to take split_queue_lock. The patch I've been using successfully on 6.11-rc3-next-20240816 below: I do have other problems with current mm commits, so have not been able to sustain a load for very long, but suspect those problems unrelated to this series. Please fold this fix, or your own equivalent, into your next version. --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3270,7 +3270,8 @@ static void __split_huge_page(struct pag folio_clear_active(new_folio); folio_clear_unevictable(new_folio); - if (!folio_batch_add(&free_folios, folio)) { + list_del(&new_folio->lru); + if (!folio_batch_add(&free_folios, new_folio)) { mem_cgroup_uncharge_folios(&free_folios); free_unref_folios(&free_folios); } @@ -3706,42 +3707,37 @@ static unsigned long deferred_split_scan bool did_split = false; bool underutilized = false; - if (folio_test_partially_mapped(folio)) - goto split; - underutilized = thp_underutilized(folio); - if (underutilized) - goto split; - continue; -split: + if (!folio_test_partially_mapped(folio)) { + underutilized = thp_underutilized(folio); + if (!underutilized) + goto next; + } if (!folio_trylock(folio)) - continue; - did_split = !split_folio(folio); - folio_unlock(folio); - if (did_split) { - /* Splitting removed folio from the list, drop reference here */ - folio_put(folio); + goto next; + if (!split_folio(folio)) { + did_split = true; if (underutilized) count_vm_event(THP_UNDERUTILIZED_SPLIT_PAGE); split++; } - } - - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - /* - * Only add back to the queue if folio is partially mapped. - * If thp_underutilized returns false, or if split_folio fails in - * the case it was underutilized, then consider it used and don't - * add it back to split_queue. - */ - list_for_each_entry_safe(folio, next, &list, _deferred_list) { - if (folio_test_partially_mapped(folio)) - list_move(&folio->_deferred_list, &ds_queue->split_queue); - else { + folio_unlock(folio); +next: + /* + * split_folio() removes folio from list on success. + * Only add back to the queue if folio is partially mapped. + * If thp_underutilized returns false, or if split_folio fails + * in the case it was underutilized, then consider it used and + * don't add it back to split_queue. + */ + if (!did_split && !folio_test_partially_mapped(folio)) { list_del_init(&folio->_deferred_list); ds_queue->split_queue_len--; } folio_put(folio); } + + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + list_splice_tail(&list, &ds_queue->split_queue); spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); /* 2. I don't understand why there needs to be a new PG_partially_mapped flag, with all its attendant sets and tests and clears all over. Why can't deferred_split_scan() detect that case for itself, using the criteria from __folio_remove_rmap()? I see folio->_nr_pages_mapped is commented "Do not use outside of rmap and debug code", and folio_nr_pages_mapped() is currently only used from mm/debug.c; but using the info already maintained is preferable to adding a PG_flag (and perhaps more efficient - skips splitting when _nr_pages_mapped already fell to 0 and folio will soon be freed). 3. Everything in /sys/kernel/mm/transparent_hugepage/ is about THPs, so please remove the "thp_" from "thp_low_util_shrinker" - "shrink_underused" perhaps. And it needs a brief description in Documentation/admin-guide/mm/transhuge.rst. 4. Wouldn't "underused" be better than "underutilized" throughout? Hugh