From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBE6CC54E67 for ; Wed, 27 Mar 2024 09:30:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E7F76B007B; Wed, 27 Mar 2024 05:30:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 647AA6B008A; Wed, 27 Mar 2024 05:30:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 474006B0092; Wed, 27 Mar 2024 05:30:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 25F3B6B007B for ; Wed, 27 Mar 2024 05:30:37 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B224A403DD for ; Wed, 27 Mar 2024 09:30:36 +0000 (UTC) X-FDA: 81942298872.17.FFA4E95 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf18.hostedemail.com (Postfix) with ESMTP id 43D9A1C0009 for ; Wed, 27 Mar 2024 09:30:34 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UU2scdN6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Txe5EZeh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=n4sX2sre; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=FagpEbYi; dmarc=none; spf=pass (imf18.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711531834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pVtsAoUH5IH+I7nzQ90FtRSD5xnb1Cluk9Zp6s7yUWo=; b=PBg9EFUm+Osku6tPNgM1thT3mkkN0FCPE+9Y1nKooV93gMT2Lu8T9muYFms7o7ulkOvzfY 1DlRFcrFgKacUUumdioEdfs7WMG8MRB6WWkQeiKhYOYxXcqQlkBYBSZ48iFXwUPoL/W+mx bl7hYO11c1kzDte4+wkCeC18b+RkCKM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UU2scdN6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Txe5EZeh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=n4sX2sre; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=FagpEbYi; dmarc=none; spf=pass (imf18.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711531834; a=rsa-sha256; cv=none; b=lj0vQnK8aqQs7k0PZHiM1AtAGjDOaYf140EYqatTl3zMk25FqYwR6SWdMStjbZWr8aiGsW dUfWbUx0V4OeNu50zBOoN8B1O/lfg2+WBjHWRvzit9Zl6gN1D4h9vBfJ88Q/p9RL0G3LUC NEFkk6GC3QayE3YDcNco5RpWaThBE7o= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7F2CC60190; Wed, 27 Mar 2024 09:30:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711531832; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pVtsAoUH5IH+I7nzQ90FtRSD5xnb1Cluk9Zp6s7yUWo=; b=UU2scdN6Oun/dlVmEZsumb48WftOvPMZyqkgegdn7rFoYZ4JuQuRvzpl2vqlqKsUAIcMVH jBK6WLjHtn4yIg/WO8s5i8/60jF7AG9QjE0RJXVyOrZOcimfOO8cd45emzfm+zRBoM4KcN a+iyTFqkW6uHMjeyXtlE1/X6XXszWbE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711531832; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pVtsAoUH5IH+I7nzQ90FtRSD5xnb1Cluk9Zp6s7yUWo=; b=Txe5EZehw/jccoHVxJFWGUZoWoQRHSI5inevyDFqR6c3AbVFxBkK3yP/cJxKqsbYXHRqcd Yx1hLHoK8l3U1CCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711531830; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pVtsAoUH5IH+I7nzQ90FtRSD5xnb1Cluk9Zp6s7yUWo=; b=n4sX2sredVIOadRqUJ1Jt4CUbA8/AcnWT0qVmrb5wEa+/mW/thnzhLt9zwZ+bZhAo1R9Ad s5m9xAgm+0sLVvBfyUw6oviG8cbCd5g94iug4yEhx+wEabuVp4GmG2vVDYdWqfoDXrqBWR FgVnNqQkPWDdk+hEELS40VTaTQqLvrc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711531830; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pVtsAoUH5IH+I7nzQ90FtRSD5xnb1Cluk9Zp6s7yUWo=; b=FagpEbYi6DMhvU7Rip8/13QgoK/7e6ub3lzPuuCEOh65Bw1GbecPezbOp8W5cGLBkwAFM1 u/MXmKZJHaRfpcAA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 6B7DC13688; Wed, 27 Mar 2024 09:30:30 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id FGLNGTbnA2YEGAAAD6G6ig (envelope-from ); Wed, 27 Mar 2024 09:30:30 +0000 Message-ID: <81b1d642-2ec0-49f5-89fc-19a3828419ff@suse.cz> Date: Wed, 27 Mar 2024 10:30:30 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH V4 00/10] mm: page_alloc: freelist migratetype hygiene Content-Language: en-US To: Johannes Weiner , Andrew Morton Cc: Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240320180429.678181-1-hannes@cmpxchg.org> From: Vlastimil Babka In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 43D9A1C0009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: aw6i8abk5kur986gttswri9ypurattbz X-HE-Tag: 1711531834-593361 X-HE-Meta: U2FsdGVkX18V5v/mD5OaZjvA6hFL+6BSQEkljoA3OtXcLTAKkLLfeKQlzVLvHbF+1tGthNGTi9xO0Dr7+zbs0XgZzKKln7cIvYK2IOo1ePAIafOno23TdevtIlF3yPBdfa5D0lPPt4fv9eFqeQt9zdFH6gBKHGkPlos/+CsIiCtKahtOBQ9iF+QoDzmw7lRLn1PadNwND8W+ERQ4VrMAcCwyoFNpu5uXsDJ8mJUHiUqtSOUWwsmrL4P2Rkvc9BhRi94bWKKPaw/E4Lndqg5GOd7+AdjHaRC3OATF0SsKs0WRTaGnP6tBke9mn8/6XOYaEyicHtWo1ZbgRkIBgaGoJaFvGtzYqksEdrYZUoxDGwfLUcRhhOZhfwPtWeL+Rb//azfC+G5IHA3610fzorjLiElKWi5bQehL7Cjt4AaYuV9+QEb/aSFqGGrhh7xYiusGCv3aLKUY26mvxz+1HWvlcTv1B7TzuBnH96x54SECl0jBbma9TXcvdT9ux/5/ZeHM14ufZhM0mot6KSXXa3dcXpfToYVXmbWmhxSPK/bpruKDme6rbsZElSiPuGyPPu+hZ44MCKqyMFy9TzQCOa9OkmC/8ZphjZOAvk6z7OybpgIdxem+F5tIujGh9OE7VGCBUOc4FD6ZNALDb8PXKo8C6yoLCcpKg8NMPTUuwRSzajJPpwPC2WlA46CBjZwcKKRC1hJjC8XnEJ8HsTi5KFHdrmxUU50dKbxoRinYLuXN2Ujur4w4zJZIOCrid/g4MpJ7Mpx2ZJQtqJFDtsibL0xC9EmuZ8ierhbuz4fAjavHAwcPWDCm56Rq7ckH7AcSWzirYnELJds3qVjCRqyUIG4Bea5CevYRU/ajMjd12gLI94hH0lrFp4OZmYSPoFg8stcKL/47t5yuNVM07QltufgOiaC+z+JsDWVz2/nzcI9HOZhFhCaOdu7RhQ817fFXl1cW5+dJ242JG6Sj9xNx7pM pjonlUUT VHNhsklvMyfW8xBQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 3/20/24 7:02 PM, Johannes Weiner wrote: > V4: > - fixed !pcp_order_allowed() case in free_unref_folios() > - reworded the patch 0 changelog a bit for the git log > - rebased to mm-everything-2024-03-19-23-01 > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > --- > > The page allocator's mobility grouping is intended to keep unmovable > pages separate from reclaimable/compactable ones to allow on-demand > defragmentation for higher-order allocations and huge pages. > > Currently, there are several places where accidental type mixing > occurs: an allocation asks for a page of a certain migratetype and > receives another. This ruins pageblocks for compaction, which in turn > makes allocating huge pages more expensive and less reliable. > > The series addresses those causes. The last patch adds type checks on > all freelist movements to prevent new violations being introduced. > > The benefits can be seen in a mixed workload that stresses the machine > with a memcache-type workload and a kernel build job while > periodically attempting to allocate batches of THP. The following data > is aggregated over 50 consecutive defconfig builds: Great stuff. What would you say to the following on top? ----8<---- >From 84f8a6d3a9e34c7ed8b438c3152d56e359a4ffb4 Mon Sep 17 00:00:00 2001 From: Vlastimil Babka Date: Wed, 27 Mar 2024 10:19:47 +0100 Subject: [PATCH] mm: page_alloc: change move_freepages() to __move_freepages_block() The function is now supposed to be called only on a single pageblock and checks start_pfn and end_pfn accordingly. Rename it to make this more obvious and drop the end_pfn parameter which can be determined trivially and none of the callers use it for anything else. Also make the (now internal) end_pfn exclusive, which is more common. Signed-off-by: Vlastimil Babka --- mm/page_alloc.c | 43 ++++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 34c84ef16b66..75aefbd52ef9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1566,18 +1566,18 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * Change the type of a block and move all its free pages to that * type's freelist. */ -static int move_freepages(struct zone *zone, unsigned long start_pfn, - unsigned long end_pfn, int old_mt, int new_mt) +static int __move_freepages_block(struct zone *zone, unsigned long start_pfn, + int old_mt, int new_mt) { struct page *page; - unsigned long pfn; + unsigned long pfn, end_pfn; unsigned int order; int pages_moved = 0; VM_WARN_ON(start_pfn & (pageblock_nr_pages - 1)); - VM_WARN_ON(start_pfn + pageblock_nr_pages - 1 != end_pfn); + end_pfn = pageblock_end_pfn(start_pfn); - for (pfn = start_pfn; pfn <= end_pfn;) { + for (pfn = start_pfn; pfn < end_pfn;) { page = pfn_to_page(pfn); if (!PageBuddy(page)) { pfn++; @@ -1603,14 +1603,13 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, static bool prep_move_freepages_block(struct zone *zone, struct page *page, unsigned long *start_pfn, - unsigned long *end_pfn, int *num_free, int *num_movable) { unsigned long pfn, start, end; pfn = page_to_pfn(page); start = pageblock_start_pfn(pfn); - end = pageblock_end_pfn(pfn) - 1; + end = pageblock_end_pfn(pfn); /* * The caller only has the lock for @zone, don't touch ranges @@ -1621,16 +1620,15 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, */ if (!zone_spans_pfn(zone, start)) return false; - if (!zone_spans_pfn(zone, end)) + if (!zone_spans_pfn(zone, end - 1)) return false; *start_pfn = start; - *end_pfn = end; if (num_free) { *num_free = 0; *num_movable = 0; - for (pfn = start; pfn <= end;) { + for (pfn = start; pfn < end;) { page = pfn_to_page(pfn); if (PageBuddy(page)) { int nr = 1 << buddy_order(page); @@ -1656,13 +1654,12 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, static int move_freepages_block(struct zone *zone, struct page *page, int old_mt, int new_mt) { - unsigned long start_pfn, end_pfn; + unsigned long start_pfn; - if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, - NULL, NULL)) + if (!prep_move_freepages_block(zone, page, &start_pfn, NULL, NULL)) return -1; - return move_freepages(zone, start_pfn, end_pfn, old_mt, new_mt); + return __move_freepages_block(zone, start_pfn, old_mt, new_mt); } #ifdef CONFIG_MEMORY_ISOLATION @@ -1733,10 +1730,9 @@ static void split_large_buddy(struct zone *zone, struct page *page, bool move_freepages_block_isolate(struct zone *zone, struct page *page, int migratetype) { - unsigned long start_pfn, end_pfn, pfn; + unsigned long start_pfn, pfn; - if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, - NULL, NULL)) + if (!prep_move_freepages_block(zone, page, &start_pfn, NULL, NULL)) return false; /* No splits needed if buddies can't span multiple blocks */ @@ -1767,8 +1763,9 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, return true; } move: - move_freepages(zone, start_pfn, end_pfn, - get_pfnblock_migratetype(page, start_pfn), migratetype); + __move_freepages_block(zone, start_pfn, + get_pfnblock_migratetype(page, start_pfn), + migratetype); return true; } #endif /* CONFIG_MEMORY_ISOLATION */ @@ -1868,7 +1865,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page, unsigned int alloc_flags, bool whole_block) { int free_pages, movable_pages, alike_pages; - unsigned long start_pfn, end_pfn; + unsigned long start_pfn; int block_type; block_type = get_pageblock_migratetype(page); @@ -1901,8 +1898,8 @@ steal_suitable_fallback(struct zone *zone, struct page *page, goto single_page; /* moving whole block can fail due to zone boundary conditions */ - if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, - &free_pages, &movable_pages)) + if (!prep_move_freepages_block(zone, page, &start_pfn, &free_pages, + &movable_pages)) goto single_page; /* @@ -1932,7 +1929,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page, */ if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { - move_freepages(zone, start_pfn, end_pfn, block_type, start_type); + __move_freepages_block(zone, start_pfn, block_type, start_type); return __rmqueue_smallest(zone, order, start_type); } -- 2.44.0