From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF043C3601E for ; Thu, 10 Apr 2025 08:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0810A6B029D; Thu, 10 Apr 2025 04:12:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00B666B02A0; Thu, 10 Apr 2025 04:12:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC5EF6B02A4; Thu, 10 Apr 2025 04:12:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B9F666B029D for ; Thu, 10 Apr 2025 04:12:47 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 60F63C055F for ; Thu, 10 Apr 2025 08:12:49 +0000 (UTC) X-FDA: 83317418058.22.C4A1EB1 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf19.hostedemail.com (Postfix) with ESMTP id D63271A000D for ; Thu, 10 Apr 2025 08:12:46 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=dk8qpuyw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=svQQQJ+M; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=dk8qpuyw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=svQQQJ+M; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744272767; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bhnrev0ra4PwSkpyWtVkGoW6BWZCtaecoXBOEh9FMgo=; b=IcPZYOXtA1GhVuyQEnh+YsiHfkmJKbpC5NaP7iSNgI8v3HmTRe1FlrBBF1W1xSgV9iLwQ5 KZJgrBw5V/a4vRfywhSbzg1QY6rOI5BOpAi6LEZ64ckEddxPlEYtJSsDXww5xRKgEVY7qp jtqN0VomY46VuuzONg6nBsAjKweKRxU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744272767; a=rsa-sha256; cv=none; b=MZG29IIzN43tV58qbra8S+PVrDX1ugeN7X5S5Es6dXYqdBk0VcM/3SB3ETs6bQq1J/8bDD b86grLugmU9HpG3L6ADP4pJiPp4E8eLDAwzsRQ54hOwjAvA3+SkPGp7bR21W/XUQrUoCNX TFbVKj7ua7Ts+gXlTz8alhxC3j3qSSQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=dk8qpuyw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=svQQQJ+M; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=dk8qpuyw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=svQQQJ+M; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 0382F2118B; Thu, 10 Apr 2025 08:12:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1744272765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bhnrev0ra4PwSkpyWtVkGoW6BWZCtaecoXBOEh9FMgo=; b=dk8qpuyw27rmjYdfkloqA4RH91rSbf/EEfazfHTiFw88VwkZMhSg79oDsu/saFEmM67X34 KYn1kL/M/tTkrqiYU0ZCFYVHpZBLjrAu6sfSCKhUS3/UzIdBaurF2flLvOIHRVMmC+B9DD fuQaEHLVgsjUT+tLHUmFpJvKrNSjV7Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1744272765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bhnrev0ra4PwSkpyWtVkGoW6BWZCtaecoXBOEh9FMgo=; b=svQQQJ+Mv3at12FsHgpdGsUm369PuMtGX7NoIuoTIFdCJDO2+QC9mVj9WfoRSifQGOsg00 W1y0yfavdFIPtQCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1744272765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bhnrev0ra4PwSkpyWtVkGoW6BWZCtaecoXBOEh9FMgo=; b=dk8qpuyw27rmjYdfkloqA4RH91rSbf/EEfazfHTiFw88VwkZMhSg79oDsu/saFEmM67X34 KYn1kL/M/tTkrqiYU0ZCFYVHpZBLjrAu6sfSCKhUS3/UzIdBaurF2flLvOIHRVMmC+B9DD fuQaEHLVgsjUT+tLHUmFpJvKrNSjV7Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1744272765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bhnrev0ra4PwSkpyWtVkGoW6BWZCtaecoXBOEh9FMgo=; b=svQQQJ+Mv3at12FsHgpdGsUm369PuMtGX7NoIuoTIFdCJDO2+QC9mVj9WfoRSifQGOsg00 W1y0yfavdFIPtQCw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id DD0E6132D8; Thu, 10 Apr 2025 08:12:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id VhiYNXx992d0BQAAD6G6ig (envelope-from ); Thu, 10 Apr 2025 08:12:44 +0000 Message-ID: <18217ade-4ac4-4654-9c25-d6b65e6c23e5@suse.cz> Date: Thu, 10 Apr 2025 10:12:44 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/2] mm: page_alloc: speed up fallbacks in rmqueue_bulk() To: Johannes Weiner , Andrew Morton Cc: Brendan Jackman , Mel Gorman , Carlos Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel test robot , stable@vger.kernel.org References: <20250407180154.63348-1-hannes@cmpxchg.org> Content-Language: en-US From: Vlastimil Babka In-Reply-To: <20250407180154.63348-1-hannes@cmpxchg.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Action: no action X-Stat-Signature: pxwypnx1dyeqmy8tskjhywtakxejangh X-Rspam-User: X-Rspamd-Queue-Id: D63271A000D X-Rspamd-Server: rspam08 X-HE-Tag: 1744272766-529752 X-HE-Meta: U2FsdGVkX19XE0OhtANGRU7QbXGypckhFXARKOlu5PRjZ0vGwROLFwP6ezKubp/IupYbhX4FzXmYbOzon98sKzONBDy1kzvu2fTl7jN4JdgboTzdVSpwCPz6qgjFpzWEzJQq7rrZvLyYKA7AZA5qUpAtF2ol2zzlkG7plH/YqMjYEGtQyBQo1V6yQuc/V/HLfOK6XPNJw66CRjcO19jtEIhCD6q/cWwnFItDIKbEBh0vWocsXqHbTgEDOxIrOYGz/UgWivtAb4shmW30cteUEWqvMLkZs8kC2EpkmEz91613H0miJndkO+Zxx28RVpuC8qZ8S4q6oPD2UM6Hd7LnFnw45T6+MtSVmUtBj0Lg7WPxG4aOMaSeklHxQOZvSzCLCYn91S7otOhaRoYZmgfCde4YUX/lnAMi8x15BjwHeXK84DE2D6TWLM81nv5n16IV6raLqUWGxcXK8wFZUdDP5uZIwU8wBrNv8RDbb8u5aSxlVRIQTMsSHDfPXP9TTrqdkLQrabOjUdqztsEJbAJwGebl9z+ygfX4RE8klWpfNrVzbxNDD54/sYdD8PhEGE4QS5e4f0mS9KJLSGaakFdjhP1dfPrCoPTU+DfWOKAPumBTzgp9JAS4JA1dtBHPSd65g6R/fMrRwunFM2WmDWnU+jqTG09J1Q3chfROL8NEQqUDhC37LyxDik/C9ixoxonnAlhV5ZvByUNJV+MJmCDEJvqepwKGXNnqUOfceuoylyXaRi4DGI0ZFcwwwSesrSfSyD4abWdv5IXQYVZfV186u4oSX3hhK+AGNYgu3qmsWBdVC0nsgx/5zNoIt9yDgITa2OdWZF25C12eglaq5UkpmQnZ29KOODbKNsdJWIF2UKcS1k3xZ4hxJh+ylbvZwiVy0xOBK+K8G4g0UQyzma01KwQLvJiUl2v+qvw/MZr/TCK02DzlukNKTjvMFkTVGHAPFU0tb72/ESUYXToKYtz pxJQaDft 4heMPQsrQnY6l4NbTUNQ0Xl2D+PR740aGDdbXSBvwi//gxlCZedFYgrRvVYRzTW853cDE9QIIGM+AzFO68t+FAIjsv9iq2XWR/0aJHxNJ6gC8rBm2crMGAcSrdrTLP99favieOLxAHlMzQwp99WFdx2D1jEewDeJywYOSfZXRmB+eYJuFQ3ZgSuclzO10nxbFgt/yVF9m+W70iArhEQdWssoWfZdtMcojPA28WYEPk0CpE7qze6yM//yyI32lxhlcX+xPgeScmMiIvmhYC/H2483YnjqHaucn8LW0ACcAeqeqB0b2uvvogPUq0beu0xNdPhaP64vHfAmkubUZU2x4KKqq7QKC84hC/1TX1NF1BpyIxTOWes6y4uGel5pMgAh6KQoKWjok18GpQvQYXNO49cOy1O50vk4Bimv6EVxqDTmVIXWobMt2GDLSPuLia9VuAPZCLRSoFz2EDZc0JTd+zDwgHLfnZV2yY/BB6dOOdWrZl1uZjEOaqnEbWsm9rlnnzaC/FnG0jR7CPf8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/7/25 20:01, Johannes Weiner wrote: > The test robot identified c2f6ea38fc1b ("mm: page_alloc: don't steal > single pages from biggest buddy") as the root cause of a 56.4% > regression in vm-scalability::lru-file-mmap-read. > > Carlos reports an earlier patch, c0cd6f557b90 ("mm: page_alloc: fix > freelist movement during block conversion"), as the root cause for a > regression in worst-case zone->lock+irqoff hold times. > > Both of these patches modify the page allocator's fallback path to be > less greedy in an effort to stave off fragmentation. The flip side of > this is that fallbacks are also less productive each time around, > which means the fallback search can run much more frequently. > > Carlos' traces point to rmqueue_bulk() specifically, which tries to > refill the percpu cache by allocating a large batch of pages in a > loop. It highlights how once the native freelists are exhausted, the > fallback code first scans orders top-down for whole blocks to claim, > then falls back to a bottom-up search for the smallest buddy to steal. > For the next batch page, it goes through the same thing again. > > This can be made more efficient. Since rmqueue_bulk() holds the > zone->lock over the entire batch, the freelists are not subject to > outside changes; when the search for a block to claim has already > failed, there is no point in trying again for the next page. > > Modify __rmqueue() to remember the last successful fallback mode, and > restart directly from there on the next rmqueue_bulk() iteration. > > Oliver confirms that this improves beyond the regression that the test > robot reported against c2f6ea38fc1b: > > commit: > f3b92176f4 ("tools/selftests: add guard region test for /proc/$pid/pagemap") > c2f6ea38fc ("mm: page_alloc: don't steal single pages from biggest buddy") > acc4d5ff0b ("Merge tag 'net-6.15-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net") > 2c847f27c3 ("mm: page_alloc: speed up fallbacks in rmqueue_bulk()") <--- your patch > > f3b92176f4f7100f c2f6ea38fc1b640aa7a2e155cc1 acc4d5ff0b61eb1715c498b6536 2c847f27c37da65a93d23c237c5 > ---------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ > 25525364 ± 3% -56.4% 11135467 -57.8% 10779336 +31.6% 33581409 vm-scalability.throughput > > Carlos confirms that worst-case times are almost fully recovered > compared to before the earlier culprit patch: > > 2dd482ba627d (before freelist hygiene): 1ms > c0cd6f557b90 (after freelist hygiene): 90ms > next-20250319 (steal smallest buddy): 280ms > this patch : 8ms > > Reported-by: kernel test robot > Reported-by: Carlos Song > Tested-by: kernel test robot > Fixes: c0cd6f557b90 ("mm: page_alloc: fix freelist movement during block conversion") > Fixes: c2f6ea38fc1b ("mm: page_alloc: don't steal single pages from biggest buddy") > Closes: https://lore.kernel.org/oe-lkp/202503271547.fc08b188-lkp@intel.com > Cc: stable@vger.kernel.org # 6.10+ Might be going to be fun with dependency commits. > Signed-off-by: Johannes Weiner Cool stuff. Reviewed-by: Vlastimil Babka