From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A0F5C25B07 for ; Tue, 9 Aug 2022 17:19:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFC008E0001; Tue, 9 Aug 2022 13:19:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13BFC8E0010; Tue, 9 Aug 2022 13:19:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC11F8E000E; Tue, 9 Aug 2022 13:19:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7BA258E0008 for ; Tue, 9 Aug 2022 13:19:02 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4F5AA141104 for ; Tue, 9 Aug 2022 17:19:02 +0000 (UTC) X-FDA: 79780714524.20.EA07A49 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 9DFF210002E for ; Tue, 9 Aug 2022 17:19:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=iEsoM2GbZpv7BfN0naqBZLr1B1bQ9PAqHFTrd0G4rA4=; b=MuLOefjlRQDmeylJ430eeGje17 HJvkC+fPAFdJqYAdDwx0XzOm791zYeKPSmPFVyVQ5Ro4WbPyvd4u3lR1eXLpaohycSyW9iK0I4Pk+ APUXsJFacEThZlreaay988wDeplt29c5owyZ5zfZ/+dX0Elnd/h7VPN/qVR8vHpJPRuN80Ig07zvc 2FnqCGesFh96z06LziicBRHyKdqxABhyL/TAV5A2WPtrPo+BkcXyspN3zmWJ10v1WSCGoCCMatLGa mq1r78Ob5Tvt/Bcfs77Z65GgRogUytlYgOzmN4xJlx1aincPftgKRg5uMyGq9AbZ+VOMs3Fa6WSqX b2/e4ZmQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oLSse-00FdFq-Ab; Tue, 09 Aug 2022 17:19:00 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , William Kucharski Subject: [PATCH v2 15/16] slab: Allocate frozen pages Date: Tue, 9 Aug 2022 18:18:53 +0100 Message-Id: <20220809171854.3725722-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220809171854.3725722-1-willy@infradead.org> References: <20220809171854.3725722-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660065541; a=rsa-sha256; cv=none; b=M8Js9NBebjtEqfL1hohBLtPjLgFKkP/9w3YzUO+BgpvNQtd5KkjlCZw0Eo7ACd5AwflsuW QUaaGDO3P7BdTD8bQqBPyLAMyb1gYo2axGAMhrdpm4TaWufZ9Gwt6sGi8Q3Mth1AbXToBy p1RXl3if48gEkQOvlzamIopAuKwBBF8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MuLOefjl; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660065541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iEsoM2GbZpv7BfN0naqBZLr1B1bQ9PAqHFTrd0G4rA4=; b=b3q+vbgmqdjlqv/X2FpKqJ3vpQCoA2m0NHvShcvyQq8OCxctNOTnuPa55VeWio4sHMv+ym fB7QLwI9PP8b3ShIfkWD9CjG4KR+8eWOc6xBrd4e5rD5olPf2ia8hqOqV6UvwZrS8yqdJL 3hMrf8KGkOXp1zPVlqzbUGUtIinKS/U= X-Stat-Signature: 3omiix3aesgmy4i99iaoz9e95qipowyt X-Rspamd-Queue-Id: 9DFF210002E Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MuLOefjl; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1660065541-340973 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since slab does not use the page refcount, it can allocate and free frozen pages, saving one atomic operation per free. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: William Kucharski --- mm/slab.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 10e96137b44f..e7603d23c6c9 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1355,23 +1355,23 @@ slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nodeid) static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { - struct folio *folio; + struct page *page; struct slab *slab; flags |= cachep->allocflags; - folio = (struct folio *) __alloc_pages_node(nodeid, flags, cachep->gfporder); - if (!folio) { + page = __alloc_frozen_pages(flags, cachep->gfporder, nodeid, NULL); + if (!page) { slab_out_of_memory(cachep, flags, nodeid); return NULL; } - slab = folio_slab(folio); + __SetPageSlab(page); + slab = (struct slab *)page; account_slab(slab, cachep->gfporder, cachep, flags); - __folio_set_slab(folio); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ - if (sk_memalloc_socks() && page_is_pfmemalloc(folio_page(folio, 0))) + if (sk_memalloc_socks() && page_is_pfmemalloc(page)) slab_set_pfmemalloc(slab); return slab; @@ -1383,18 +1383,17 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab) { int order = cachep->gfporder; - struct folio *folio = slab_folio(slab); + struct page *page = (struct page *)slab; - BUG_ON(!folio_test_slab(folio)); __slab_clear_pfmemalloc(slab); - __folio_clear_slab(folio); - page_mapcount_reset(folio_page(folio, 0)); - folio->mapping = NULL; + __ClearPageSlab(page); + page_mapcount_reset(page); + page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += 1 << order; unaccount_slab(slab, order, cachep); - __free_pages(folio_page(folio, 0), order); + free_frozen_pages(page, order); } static void kmem_rcu_free(struct rcu_head *head) -- 2.35.1