From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7621FC5479D for ; Wed, 11 Jan 2023 04:22:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06C9494000B; Tue, 10 Jan 2023 23:22:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0F94940007; Tue, 10 Jan 2023 23:22:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9EE4940007; Tue, 10 Jan 2023 23:22:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AC7C094000B for ; Tue, 10 Jan 2023 23:22:41 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7ED8A120140 for ; Wed, 11 Jan 2023 04:22:41 +0000 (UTC) X-FDA: 80341222122.30.8223AC6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 0834140004 for ; Wed, 11 Jan 2023 04:22:39 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=krfKGVAQ; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673410960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=00FqBCzZySWB3TyzUQFGpi0m9pANOQoVzxBJXXHFCBo=; b=pK1bL0vqVhcByz6z/6SOI04GjS6HhSOaIcpJoXoRy+zt3BS3f/CVAzNg2GsINHsWVOESlm eBRdG4z1bmozREy9sw92p72eksWfU7z5OCR7KXiGXG9cDI2yHYKPqLFSmTNfrzaxUKBFXz DsTfhVZDK3D8b60v8TnAHB9zQD2kyXo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=krfKGVAQ; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673410960; a=rsa-sha256; cv=none; b=wnAK3Vs7GhUw2NLMXaviqYYsNgvF6Zl9gASzEl9MxmJhrEMJYMsIvzrn4Xzjjo+qmbnfee aSxGUrM8OuH5+7kFM5WHgOuf3C9GK2qv3qzIDlToucqT412RZf8Sb0xqxGhb23C/a+kh94 LQPvj8+EvbOjQn6Yir51tltJ8YfQyIM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=00FqBCzZySWB3TyzUQFGpi0m9pANOQoVzxBJXXHFCBo=; b=krfKGVAQH/1m9UUn3g85XFE4ly gSm8bAkQCTgGArdKd/rqgZamAhyuHx1rZzTSOK0ajtPu9YItY4aVr/gsqtPohhZcPTOfP0bhrwGiV pNg0ykc3MiJlvhjoTxQvMTrJWsjj4/wAdhZuu98HxAR3e54aYY1tmym6kSO9mNzMZ47PsJLK4kHot JoirFZ5FqB2Z7F1vPE2znaNIC/kONHWxMDsei/OuVYuc2dulZ0LhCEw28Qu2y/9+qoWQi1lHsj7qn Ypj1Mm/V7Sv0ptgQc8QbZwpKVlFoKssNjC7mzK7YVyT40zp6kjMbyBcPi9PXkp84joNOcvRqq1MLA zEwSMCCA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFScy-003nxq-Ji; Wed, 11 Jan 2023 04:22:16 +0000 From: "Matthew Wilcox (Oracle)" To: Jesper Dangaard Brouer , Ilias Apalodimas Cc: "Matthew Wilcox (Oracle)" , netdev@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Jesper Dangaard Brouer , Jesse Brandeburg Subject: [PATCH v3 05/26] page_pool: Start using netmem in allocation path. Date: Wed, 11 Jan 2023 04:21:53 +0000 Message-Id: <20230111042214.907030-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111042214.907030-1-willy@infradead.org> References: <20230111042214.907030-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0834140004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 7z5obndd8a7p63kd6ajnpxy5ctmanppm X-HE-Tag: 1673410959-460128 X-HE-Meta: U2FsdGVkX19XEVmYxKTaNbI6P0EnYnq4gSG/9D1qdWxrBcq/+txNEPilXEMi7Yij4k7ynqO4Mym0AMjKFAZx2/qCBEAyamzgG03VFRI80czGIKjaCLFoWzd5WjjEJs9InOf7Lk0rcTQ8zdaMc8XTXM7oGvZFBGransRBmnld2DlKPRZ5PnNSb2fHOp8VH7HRVan9WSmh9KWPsrRX7Cx7PBkWA5BNVkN75H1ZZ9CTQLnsLG+XoR/Z/LtQT5TW2UVa1jfA3k7H79curDN2p5bCd7qg+dTmOfqVUUyOhPD91Z6L7cPOuLEg1X1rm9rbcasM4t/68Mfvnc29wZruD2UytuXPTwgM+yw4CgNKbHh5jstodzlX/S3KJ4GGf2PcgWOhx74OlGfPsaLJVvcQvG3YRw2j9CwgeosKmQOUUeXCcvtuQM0M+uydoWuo+5uvDHJmORw4ZT6XK3bHbF2HYNOSD9OzNIaEo0VJtdLXK9gZLPWUcLYC8nmjmJBvraiMfTWpOPy3a3cGxAbvJRiTWgXehqd0NV7jYhdtgIb/jk8WkKKC0srjSVcgEhd7NpJwiBMPEW1upy6+A/l4BWncnO6XvzJu2yXBMMt/uWdxq9FtPPeCgd8sG43JmMEUUJSEWTh9TaqiH4XEOCC6gJS5SpRDtJ3usibcxI8js1eyO4BseHEHveqs8/YBBVWdlQFCcAzQAbMR04zIn4qo4Hi+bgXnXoJYsKawnM+/NApzFk0U2I9FtvypoTZMqmoE1hGWJeOR+S0FdJZ/gd1NPmkZvmKI0rBp7/PfdYii2mXeH4c8gz+cuoJJ0lzHR1HTSJRxMppxEsEQ6Jd5Uns7XhlyW83F9dkyHlf2pxUXec9OnZRaF//Wn4gpYhuj7fah4QRn3F10r3G0U/rAfwB4P+/5UK9ugc0wiRl3WwPAhzJ/5Z51K+FfTOrG46hx1g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert __page_pool_alloc_page_order() and __page_pool_alloc_pages_slow() to use netmem internally. This removes a couple of calls to compound_head() that are hidden inside put_page(). Convert trace_page_pool_state_hold(), page_pool_dma_map() and page_pool_set_pp_info() to take a netmem argument. Saves 83 bytes of text in __page_pool_alloc_page_order() and 98 in __page_pool_alloc_pages_slow() for a total of 181 bytes. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas Reviewed-by: Jesse Brandeburg --- include/trace/events/page_pool.h | 14 +++++------ net/core/page_pool.c | 42 +++++++++++++++++--------------- 2 files changed, 29 insertions(+), 27 deletions(-) diff --git a/include/trace/events/page_pool.h b/include/trace/events/page_pool.h index 113aad0c9e5b..d1237a7ce481 100644 --- a/include/trace/events/page_pool.h +++ b/include/trace/events/page_pool.h @@ -67,26 +67,26 @@ TRACE_EVENT(page_pool_state_release, TRACE_EVENT(page_pool_state_hold, TP_PROTO(const struct page_pool *pool, - const struct page *page, u32 hold), + const struct netmem *nmem, u32 hold), - TP_ARGS(pool, page, hold), + TP_ARGS(pool, nmem, hold), TP_STRUCT__entry( __field(const struct page_pool *, pool) - __field(const struct page *, page) + __field(const struct netmem *, nmem) __field(u32, hold) __field(unsigned long, pfn) ), TP_fast_assign( __entry->pool = pool; - __entry->page = page; + __entry->nmem = nmem; __entry->hold = hold; - __entry->pfn = page_to_pfn(page); + __entry->pfn = netmem_pfn(nmem); ), - TP_printk("page_pool=%p page=%p pfn=0x%lx hold=%u", - __entry->pool, __entry->page, __entry->pfn, __entry->hold) + TP_printk("page_pool=%p netmem=%p pfn=0x%lx hold=%u", + __entry->pool, __entry->nmem, __entry->pfn, __entry->hold) ); TRACE_EVENT(page_pool_update_nid, diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 437241aba5a7..4e985502c569 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -304,8 +304,9 @@ static void page_pool_dma_sync_for_device(struct page_pool *pool, pool->p.dma_dir); } -static bool page_pool_dma_map(struct page_pool *pool, struct page *page) +static bool page_pool_dma_map(struct page_pool *pool, struct netmem *nmem) { + struct page *page = netmem_page(nmem); dma_addr_t dma; /* Setup DMA mapping: use 'struct page' area for storing DMA-addr @@ -328,12 +329,12 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) } static void page_pool_set_pp_info(struct page_pool *pool, - struct page *page) + struct netmem *nmem) { - page->pp = pool; - page->pp_magic |= PP_SIGNATURE; + nmem->pp = pool; + nmem->pp_magic |= PP_SIGNATURE; if (pool->p.init_callback) - pool->p.init_callback(page, pool->p.init_arg); + pool->p.init_callback(netmem_page(nmem), pool->p.init_arg); } static void page_pool_clear_pp_info(struct netmem *nmem) @@ -345,26 +346,26 @@ static void page_pool_clear_pp_info(struct netmem *nmem) static struct page *__page_pool_alloc_page_order(struct page_pool *pool, gfp_t gfp) { - struct page *page; + struct netmem *nmem; gfp |= __GFP_COMP; - page = alloc_pages_node(pool->p.nid, gfp, pool->p.order); - if (unlikely(!page)) + nmem = page_netmem(alloc_pages_node(pool->p.nid, gfp, pool->p.order)); + if (unlikely(!nmem)) return NULL; if ((pool->p.flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { - put_page(page); + unlikely(!page_pool_dma_map(pool, nmem))) { + netmem_put(nmem); return NULL; } alloc_stat_inc(pool, slow_high_order); - page_pool_set_pp_info(pool, page); + page_pool_set_pp_info(pool, nmem); /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; - trace_page_pool_state_hold(pool, page, pool->pages_state_hold_cnt); - return page; + trace_page_pool_state_hold(pool, nmem, pool->pages_state_hold_cnt); + return netmem_page(nmem); } /* slow path */ @@ -398,18 +399,18 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, * page element have not been (possibly) DMA mapped. */ for (i = 0; i < nr_pages; i++) { - page = pool->alloc.cache[i]; + struct netmem *nmem = page_netmem(pool->alloc.cache[i]); if ((pp_flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { - put_page(page); + unlikely(!page_pool_dma_map(pool, nmem))) { + netmem_put(nmem); continue; } - page_pool_set_pp_info(pool, page); - pool->alloc.cache[pool->alloc.count++] = page; + page_pool_set_pp_info(pool, nmem); + pool->alloc.cache[pool->alloc.count++] = netmem_page(nmem); /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; - trace_page_pool_state_hold(pool, page, + trace_page_pool_state_hold(pool, nmem, pool->pages_state_hold_cnt); } @@ -421,7 +422,8 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, page = NULL; } - /* When page just alloc'ed is should/must have refcnt 1. */ + /* When page just allocated it should have refcnt 1 (but may have + * speculative references) */ return page; } -- 2.35.1