From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23854C54FB3 for ; Thu, 29 May 2025 08:57:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B116A6B0121; Thu, 29 May 2025 04:56:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC1D76B0122; Thu, 29 May 2025 04:56:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B1176B0123; Thu, 29 May 2025 04:56:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 782A36B0121 for ; Thu, 29 May 2025 04:56:59 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CBE9C140D8C for ; Thu, 29 May 2025 08:56:58 +0000 (UTC) X-FDA: 83495340516.19.14A8776 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf22.hostedemail.com (Postfix) with ESMTP id 72D9EC000E for ; Thu, 29 May 2025 08:56:56 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=L6dREbmX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=MTzlF9gJ; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=L6dREbmX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=MTzlF9gJ; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748509016; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JQZAew8msvOctj5ZD0eHM2URNiVKphy+25eIpuwJzoA=; b=rOCKrkmFf2g1JycJwe6XaENm03y1b6fxrtxu/j+d1vLz0xKXsgqeXU1uUrTW4O6at5638G O19gGaJPS881TuBYj8jpSQl9lJwPJjnHLx11RYg8kaRp0Sg29JqxtM4gPPjxD+6e52aCEE 26oIaxFJVE1eXcbe4k0hxaT8XH1HknM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=L6dREbmX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=MTzlF9gJ; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=L6dREbmX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=MTzlF9gJ; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748509016; a=rsa-sha256; cv=none; b=TnexOwR5LTQA0wRkXfNlBn/mrTe9UUowHtrCEJe/Vdn7GM6tJNdMT2qh1nh+b1f/cTzuCn wIK9XoBrGARao/ydHmC31UBh0mZ7buqyGJuHqch/wbQKZJF+J7YYfGNlZMfIBcP7OvIVkV geYhz7cPJDDousTtUF/bBBNLmthSBf4= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 17D651F829; Thu, 29 May 2025 08:56:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748509009; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JQZAew8msvOctj5ZD0eHM2URNiVKphy+25eIpuwJzoA=; b=L6dREbmXVTJPyT4Oqd7BYqALscmAPIiQb325Q3YAEwc1i+xa9dOix3AZpHWIHDfzTvLlBE WYrE/baTXFYA3+tWq8aA4YA1pXGJWBQPRmn4OQlImR9QUxmUeioZZLfYu9v8e/o0lb7gtR L4e6n3iNZ2DRhyJtFDfoz0EREgAo9vE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748509009; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JQZAew8msvOctj5ZD0eHM2URNiVKphy+25eIpuwJzoA=; b=MTzlF9gJgL35ffV7RPYQ5xkQaZ+aS564sgCfmjR263SGnr4UgIzy4glEvauESUq1b1DL/g 093h+z7oHGl4WaCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748509009; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JQZAew8msvOctj5ZD0eHM2URNiVKphy+25eIpuwJzoA=; b=L6dREbmXVTJPyT4Oqd7BYqALscmAPIiQb325Q3YAEwc1i+xa9dOix3AZpHWIHDfzTvLlBE WYrE/baTXFYA3+tWq8aA4YA1pXGJWBQPRmn4OQlImR9QUxmUeioZZLfYu9v8e/o0lb7gtR L4e6n3iNZ2DRhyJtFDfoz0EREgAo9vE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748509009; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JQZAew8msvOctj5ZD0eHM2URNiVKphy+25eIpuwJzoA=; b=MTzlF9gJgL35ffV7RPYQ5xkQaZ+aS564sgCfmjR263SGnr4UgIzy4glEvauESUq1b1DL/g 093h+z7oHGl4WaCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 0569613AAD; Thu, 29 May 2025 08:56:49 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6CjfAFEhOGj5aQAAD6G6ig (envelope-from ); Thu, 29 May 2025 08:56:49 +0000 From: Vlastimil Babka Date: Thu, 29 May 2025 10:56:26 +0200 Subject: [PATCH 1/2] mm, slab: use frozen pages for large kmalloc MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20250529-frozen-pages-for-large-kmalloc-v1-1-b3aa52a8fa17@suse.cz> References: <20250529-frozen-pages-for-large-kmalloc-v1-0-b3aa52a8fa17@suse.cz> In-Reply-To: <20250529-frozen-pages-for-large-kmalloc-v1-0-b3aa52a8fa17@suse.cz> To: Christoph Lameter , David Rientjes Cc: Andrew Morton , Roman Gushchin , Harry Yoo , Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=2857; i=vbabka@suse.cz; h=from:subject:message-id; bh=tU3YQTxDcX04GkM22H8V6kixtDjvJKZbimMVydYqw8w=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBoOCFKCc/ViB6f/SoeXyF+/fKvb++yRgCg4LjTx Va1P5eYtFKJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCaDghSgAKCRC74LB10kWI mpNRB/0T0/yu6UiTDmgQb/r2nr2QvGx5EFeRfY3aWA046YhrKch528CGFXHYjuIQqsgAiZ5blWw WgNclnB9saI7S7yPpD891NKHLq2gc3D6AyrOpIKTxQDmZdkX4j4eWdoADldTbCObSkwXd+OnUIx 2AneNgESjvjUK3Y4DeCA8y0z+NvdaHf2zRGW1YuP4fv5NBkL+hWJFs5ggyh56Br5c0Uz9YUKS+4 M5NRx8ZlVzyshF8/hVRpvRlJJ3WV2HUAYjxczrQx0tmXNLi3+s4oWHCgDX9FCio3l0iinEZTCes ub70DEn9o/IEwCujVSZMFQDQFfxBe+LvjkX5WWbRch6IDVHF X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Action: no action X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 72D9EC000E X-Stat-Signature: 93xpwfz3pmotms69ctmqrpj1qxewmtcy X-Rspam-User: X-HE-Tag: 1748509016-751339 X-HE-Meta: U2FsdGVkX18MPpTgwTH4MMllBcVNMa84gBeO6+VicXs+8YI0rQK9J7OGO4jDZKgDvFjPYR+c8uBwSHch7uiKCj+I3nqq+hSNIYxL7mypYmIyUbcNzTYMEXVewKen5OCCnvssRwbQxDXAEPPLqDK+Lderm4XFbIIQFFfhV53/Po8dJkgeJR7cETidmz2tdsqbEZKNferUMpoxuXzqxFRHQSWVVkaJ3+w0qOvtH/LQ3EVGHMYo3E2UNoskrlKi8qbcjujnAWhBmFsFCFkFOzpjt8ePblIH4/0mW3sMZfnyhjC9HgcxaUKCItktccibd0YGus1UUYCsd4TUEY4lCCG2blwukd7db7/Z/BjLJyA4Xwj+FN3P7uSPZQS2PGmMrLegHiPnaykAMSB1uWLswnZmREwI1melVHS7XI5hnYWzEe2tRLfR7KFv//90WSgKVyxDEi5ZFRb73K94hQdN9x6CWWCPGHG0Ck9C6WoncOJxwH3vkW1MucMTeyiKUE3CegzsKqWkkv6YfKczFYWNcuo3sYrbX3dcjgYBQMab0XlBlDKMtyrc7Xk8IKWKdXeFkE7TSUjThYR+S/cKWcwrrr6SAo76n0EiliyRv0/QnzTzYXBegNoCSfGQuQlnnBoJGUGtME0ToSjkxBqKi+V3LayX+HulyC+1Qf55aVff585TtEWdDjCFYSKxdrJcQPPeIDTHEW9kAKSkLwCgk/o+DXWKJa4bLwQkNuqwnv23KZ9BrAxZnmwbmM3+NrDHrlHM1PgZYC0FALBHXRHSw9R/9qVhtB57zv1wKBq6kRZTsnaDuJKuUCAZI0VU6CS9E6RDyl/oW2hQf058NaqqGWMJmnJWUVx2xHyuRaAkQrA40mUhn9QXLpN5yBm1gvbZ9kmHza1xV54FEQlE634/67qc96eDmsYKe8UhTRGTXecii00j+r97uKz+XVYJIboF6WQSY0KrDnEw1rFVfpH441r43vb an0yxwpQ WsybKInck6HeNAfPTD+p/9TH3ZLVPNxcvi4jHEYqQa28kuEPC9pNJVjTcvoEI2BQYOovhgd3C5ntX8ksQkzoG1F+Z/FX/vZbYYGM3LwJ4XuXTX8PLArPnJucSmwHQHMK5APRc62nXFGkXALMg9X9gpgebWC/rTVL9sWFKcB5sElGTw6F1Di1N6Y922h1HWwUqxKYBmOZZ5zIrDek+2hsOuQSgS/4BJFNaLlVtgKp1COFXjDcHc1Ahxji5SQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since slab pages are now frozen, it makes sense to have large kmalloc() objects behave same as small kmalloc(), as the choice between the two is an implementation detail depending on allocation size. Notably, increasing refcount on a slab page containing kmalloc() object is not possible anymore, so it should be consistent for large kmalloc pages. Therefore, change large kmalloc to use the frozen pages API. Because of some unexpected fallout in the slab pages case (see commit b9c0e49abfca ("mm: decline to manipulate the refcount on a slab page"), implement the same kind of checks and warnings as part of this change. Notably, networking code using sendpage_ok() to determine whether the page refcount can be manipulated in the network stack should continue behaving correctly. Before this change, the function returns true for large kmalloc pages and page refcount can be manipulated. After this change, the function will return false. Signed-off-by: Vlastimil Babka --- include/linux/mm.h | 4 +++- mm/slub.c | 7 +++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bf55206935c467f7508e863332063bb15f904a24..d3eb6adf9fa949fbd611470182a03c743b16aac7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1549,6 +1549,8 @@ static inline void get_page(struct page *page) struct folio *folio = page_folio(page); if (WARN_ON_ONCE(folio_test_slab(folio))) return; + if (WARN_ON_ONCE(folio_test_large_kmalloc(folio))) + return; folio_get(folio); } @@ -1643,7 +1645,7 @@ static inline void put_page(struct page *page) { struct folio *folio = page_folio(page); - if (folio_test_slab(folio)) + if (folio_test_slab(folio) || folio_test_large_kmalloc(folio)) return; folio_put(folio); diff --git a/mm/slub.c b/mm/slub.c index dc9e729e1d269b5d362cb5bc44f824640ffd00f3..d7a62063a1676a327e13536bf724f0160f1fc8dc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4281,8 +4281,11 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node) if (unlikely(flags & GFP_SLAB_BUG_MASK)) flags = kmalloc_fix_flags(flags); + if (node == NUMA_NO_NODE) + node = numa_mem_id(); + flags |= __GFP_COMP; - folio = (struct folio *)alloc_pages_node_noprof(node, flags, order); + folio = (struct folio *)__alloc_frozen_pages_noprof(flags, order, node, NULL); if (folio) { ptr = folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, @@ -4778,7 +4781,7 @@ static void free_large_kmalloc(struct folio *folio, void *object) lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); __folio_clear_large_kmalloc(folio); - folio_put(folio); + free_frozen_pages(&folio->page, order); } /* -- 2.49.0