From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F416AC54ED0 for ; Fri, 23 May 2025 07:47:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BE116B00C9; Fri, 23 May 2025 03:47:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 56E866B00CA; Fri, 23 May 2025 03:47:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45D5D6B00CB; Fri, 23 May 2025 03:47:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 273276B00C9 for ; Fri, 23 May 2025 03:47:51 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C34B51A1B8B for ; Fri, 23 May 2025 07:47:50 +0000 (UTC) X-FDA: 83473393500.21.090985A Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf16.hostedemail.com (Postfix) with ESMTP id 00BE2180008 for ; Fri, 23 May 2025 07:47:47 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747986469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z2ConCB4cw/IAZhlLuG3yZ0VSPHzBnhusy9kguaWByI=; b=hnEmHIzLgUVWhUdqcULk3QfBxxIBGYIisSrO+BY4rjop3mNVDVJ91v0kA+i29xVhoUqIM8 0O3DhxUz5o/6CCq3wEprx+NRjQpyXqDiqqV4LaSqIF3dNsyDZBzZl416pZC0/++dqab8D/ afJwU7+RBVSDHfqHxJmIv2rZf9UJSdM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747986469; a=rsa-sha256; cv=none; b=TGJ0Z5wqYC0Z01QRmNzlRqXJH4KABlKNV0d9+a/fCkKKYTX+G6LooSktPZZgmYtUT52eWk PXNEhHEDHfFEBvKvWZ7JsXMhmvUmTbWOL1YWrXOQCuXGUrvwKXLBgXXWfDa+nRodnwDd8o SVJIzvmfnuk/I7tkadioXUhdzunFGCo= X-AuditID: a67dfc5b-669ff7000002311f-83-6830282111e6 Date: Fri, 23 May 2025 16:47:40 +0900 From: Byungchul Park To: Taehee Yoo Cc: willy@infradead.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, kuba@kernel.org, almasrymina@google.com, ilias.apalodimas@linaro.org, harry.yoo@oracle.com, hawk@kernel.org, akpm@linux-foundation.org, davem@davemloft.net, john.fastabend@gmail.com, andrew+netdev@lunn.ch, asml.silence@gmail.com, toke@redhat.com, tariqt@nvidia.com, edumazet@google.com, pabeni@redhat.com, saeedm@nvidia.com, leon@kernel.org, ast@kernel.org, daniel@iogearbox.net, david@redhat.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, horms@kernel.org, linux-rdma@vger.kernel.org, bpf@vger.kernel.org, vishal.moola@gmail.com Subject: Re: [PATCH 00/18] Split netmem from struct page Message-ID: <20250523074740.GA8205@system.software.com> References: <20250523032609.16334-1-byungchul@sk.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTYRTHe97bXoeLt2X6qJA4u2GkVkIHUjP68kJfiqCggrbypS2vbGoa FaaiNNMiLW1NmZZmJg6WuRlDaplTDLSVsbw0MW9FF2pqznVzSuS3H//z5/zOh8OS0gY6hFWl ZQrqNEWKjBFT4s/+tdvCt8QoYxoqIkBvbGbgwXwO3Bu10OBpniJA39SGYMYzJAJ3p52BO7Vz JOj7CimYNS6QMNE1JgJXwyQF1mIzCWNXuxkoLfSSkG9pJKC/rYyGioV6Esx5oyJ49VjPwLvm PzRM2kop6NHdp8BVlghdhkCY6/2EoNNoJmDuSjUD5Q4DA+8LXQgcz8YouH2pDIGxw0mDd17P JMr41vtvCb5dNyLiDaYs/mFjJK91Okje1HSZ4U3fr4v44TdWhu+u8lJ8u8VN8KUFXxj+28Qg xX/tGGB4Y+sAxb8wdIp4t2n9Ae6oOC5JSFFlC+roBLlY6R0eoDMcu3OK7Y1UHqqO1iKWxVws 9pRu0CK/Jeyvd5E+priNuEVXi3zMcJux0+khffUALgJrO/ZqkZgluR80/uQuJnydtRzgKWs7 5WMJtwvXWT+IfCzlzuO82RJyOV+De26NL3XIxZ0/axxLO0kuFN/7zS7HYbjg0e2luh93EFcZ jLSP1y1qn7TZCZ8Xc09YbHrtYpZvDsZPG53UNbRGt0KhW6HQ/VfoVigMiGpCUlVadqpClRIb pcxNU+VEnUpPNaHF72m48POYBX3vP2RDHItk/hKLOFoppRXZmtxUG8IsKQuQPJ+MUkolSYrc c4I6/YQ6K0XQ2FAoS8mCJDvmziZJudOKTCFZEDIE9b8pwfqF5KHYmcB8r51I2qcJrTsQcWPP lcqx2LjKhfjAnjDn4LydCC8y+Wsjras/juOy2a3xblKekNqdOTM/2DyOHW2D0xNBQ8pf5qxN 0lUT+2+2vCo+GdBbdeaY/O5spSggudx2ePry0J3gTm/By/7w+J3y3otMSQ038rzo+Lr9IdVH +jokMkqjVGyPJNUaxV+dWULZOQMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02SbUhTYRiGe895z9nZaHGcVicNqoVpX2pR9IBaRj966UfUjygiylWnNpym m8pMg9Ukazr7JGxtMbP8ppGaH2UW01IrKCbGNGtmaRr2aVqmWM6I/HdxPzfX/efhaIWZCeQ0 CcmiLkGlVbIyLNsaaVq5KDRCHfHOxYPNWc5C2U8DFHXXMjBa/p4CW2k1gu+jLyUw1NTMQkH+ CA22Z5kYhp2/aOh91CMBb2EfhvqsGhp6zrSwYMkco+FEbTEFjfZWBp5X5zJw8dcNGmqM3RJo u2Nj4XX5bwb6XBYMrdYSDN7cGHjkmAMjTwYRNDlrKBjJsbNwwe1g4W2mF4G7sQfDleO5CJwN HgbGftrYmMWkqqSDInXWVxLiqEghlcXLiNnjpklF6WmWVHw7LyFdL+pZ0pI3hkld7RBFLKZP LPna24nJ54Z2lhT0f6GIs6odk6eOJsk2v92yqIOiVpMq6sLXx8rUY13tTKI70pDVXIyNyB5u RlJO4NcIz294aR9jPli4ac1HPmb5EMHjGZ3MOS6AXyyYGzaakYyj+R+MMDiURfk6/jwI7+vr sI/l/DrhWv2AxMcKPkMwDmfTf3M/ofXyu6kOPekcv+qectJ8kFA0wf2NFwim21em6lJ+u5Dn cDI+nj05+6C6mTqLZlmnmazTTNb/Jus0kwPhUhSgSUiNV2m0a8P0ceq0BI0h7MCR+Ao0+SGF x8bP1aLvbZtdiOeQcqY8ND5crWBUqfq0eBcSOFoZIH/YF6ZWyA+q0o6KuiP7dClaUe9CQRxW zpVv2SnGKvjDqmQxThQTRd2/K8VJA40o5eOKSwGne1+E8tr0oUrTXVP21ZPSxjP3Ooi3sPdZ UnrZdf/5OePj9KE7ndLPu66nhxurFprc+YXRm+8vnGcoKdsxmBF17tbEqX7Pkv2Rb/zW22Nc ewvagpenGCJsM+y/UzeU4uGkvO7Vi8pzB3YIluhNDz0fDuNTiY8ztoZYlu6ZocR6tWrVMlqn V/0B0MxHWB0DAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 00BE2180008 X-Stat-Signature: d7j5ew7bisjguo1ksmnmzmrqo9qu5kcm X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1747986467-407045 X-HE-Meta: U2FsdGVkX1+4TD1iYDAG1y6NFMxpU1rL/Gd6xZveVU3UhQg+MOWKSJM07RPxUMuO+5cs5hQg+jT46Fgnxu/1km7Dv4J043beoCvuzp9o8Yc4FmkRL1tGjYdwvcLempcmlL3Ddabj1uiU4k3bx9e9dOo2KRk2C95AljdhCNh3LGc5zd5OFfHxkRsMiq1x45WDB6PH1KrECiivwyHlTc2Lmz2DwWhiJoQtcRY8I67g8k7GbVjKjWfqFoGD7HYnuOUq/NzpU4Fm171FmQKmmKUmA9K84dG6jHS69uQbBW9jxhdZID8Y9l0h/lHrVnnkyDF7NysO8E71vx01P05chnIMSduVUTHyng7dm/l1f8Rb51cBMfkZpx9cXtyCQACC8+Pgi9wV1FeA9hj+Qo8iwAUyl+eWauKmgGSaxOd7Hsuxj4YVyAxeJQ3aWTjkX94ouweWRzaCE4rP3WuRKKJWGK3IlqSnMN3KnMhVY9ZuHQ3AzuPMQawASaxkw7zqPI9kOLY5EgChjp203v2xse7noT2fVeT7nCeyuNhyqursKcL3bITc+NTFWKvfuV6uE1DjmRFlDi4lagqCc+w/NUiiByxWZsfF9f8mufuNdNuF7IAW2PpgDjGcLqDDoLwDY1OaQRM9KbR+ay3y18YuYCw8qGw0Ca98J4C9+m2/QKS8Ttf+btmnK6hkISAJFyYQbdMvieTSXBQJsSMDwQAZPgQwiwObSVikGvMQwiOz3duL+aCZFMy91clHp66WyefxWQZkwAp1Wic7UEEyZo8px6HVwrJF8jtCGRQabVxMzYNadO3DkuIr80KhvJa905x+hqoLpIk7kLDbAaDx79u1NgdA4hTNnrxVUuzOc1uE/QBEU4p3mTnCizp+zHXc4tGhgYhz7/ijgS1MHthUoHw0w9r9baJ1bQHYfIUZeCaNBJgi6ihdOcln1lyD8qUWuvCqdf3Wd1Z9hJEFzdCv7qUhV7FKv+z ogfAnBfI nUMUD8XzZHcOhnCB5r4UHfbIgk6IST5MrZj8AHwjgjmHJLyWZfdAa7yMuT16Nf9pUc31beqZkgqF2LopnHYn9ioMup/kzlwUFsEjr+auZtXwvHEs44iU6+51mko1hHzaOnphiirU35qnuZzs51/Pw7g1xkOepjOAeFCQgvAK/RGatHwGPm2qvZvD+s7kqlaZGlmGBlK2Hd8BXK1jRoYAGn1fc0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, May 23, 2025 at 03:20:27PM +0900, Taehee Yoo wrote: > On Fri, May 23, 2025 at 12:36 PM Byungchul Park wrote: > > > > Hi Byungchul, > Thanks a lot for this work! > > > The MM subsystem is trying to reduce struct page to a single pointer. > > The first step towards that is splitting struct page by its individual > > users, as has already been done with folio and slab. This patchset does > > that for netmem which is used for page pools. > > > > Matthew Wilcox tried and stopped the same work, you can see in: > > > > https://lore.kernel.org/linux-mm/20230111042214.907030-1-willy@infradead.org/ > > > > Mina Almasry already has done a lot fo prerequisite works by luck, he > > said :). I stacked my patches on the top of his work e.i. netmem. > > > > I focused on removing the page pool members in struct page this time, > > not moving the allocation code of page pool from net to mm. It can be > > done later if needed. > > > > My rfc version of this work is: > > > > https://lore.kernel.org/all/20250509115126.63190-1-byungchul@sk.com/ > > > > There are still a lot of works to do, to remove the dependency on struct > > page in the network subsystem. I will continue to work on this after > > this base patchset is merged. > > There is a compile failure. Thanks a lot. I will fix it. Byungchul > > In file included from drivers/net/ethernet/intel/libeth/rx.c:4: > ./include/net/libeth/rx.h: In function ‘libeth_rx_sync_for_cpu’: > ./include/net/libeth/rx.h:140:40: error: ‘struct page’ has no member named ‘pp’ > 140 | page_pool_dma_sync_for_cpu(page->pp, page, fqe->offset, len); > | ^~ > drivers/net/ethernet/intel/libeth/rx.c: In function ‘libeth_rx_recycle_slow’: > drivers/net/ethernet/intel/libeth/rx.c:210:38: error: ‘struct page’ > has no member named ‘pp’ > 210 | page_pool_recycle_direct(page->pp, page); > | ^~ > make[7]: *** [scripts/Makefile.build:203: > drivers/net/ethernet/intel/libeth/rx.o] Error 1 > make[6]: *** [scripts/Makefile.build:461: > drivers/net/ethernet/intel/libeth] Error 2 > make[5]: *** [scripts/Makefile.build:461: drivers/net/ethernet/intel] Error 2 > make[5]: *** Waiting for unfinished jobs.... > > There are page->pp usecases in drivers/net > ./drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c:1574: > } else if (page->pp) { > ./drivers/net/ethernet/freescale/fec_main.c:1046: > page_pool_put_page(page->pp, page, 0, false); > ./drivers/net/ethernet/freescale/fec_main.c:1584: > page_pool_put_page(page->pp, page, 0, true); > ./drivers/net/ethernet/freescale/fec_main.c:3351: > page_pool_put_page(page->pp, page, 0, false); > ./drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c:370: > page_pool_recycle_direct(page->pp, page); > ./drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c:395: > page_pool_recycle_direct(page->pp, page); > ./drivers/net/ethernet/ti/icssg/icssg_common.c:111: > page_pool_recycle_direct(page->pp, swdata->data.page); > ./drivers/net/ethernet/intel/idpf/idpf_txrx.c:389: > page_pool_put_full_page(rx_buf->page->pp, rx_buf->page, false); > ./drivers/net/ethernet/intel/idpf/idpf_txrx.c:3254: u32 hr = > rx_buf->page->pp->p.offset; > ./drivers/net/ethernet/intel/idpf/idpf_txrx.c:3286: dst = > page_address(hdr->page) + hdr->offset + hdr->page->pp->p.offset; > ./drivers/net/ethernet/intel/idpf/idpf_txrx.c:3287: src = > page_address(buf->page) + buf->offset + buf->page->pp->p.offset; > ./drivers/net/ethernet/intel/idpf/idpf_txrx.c:3305: u32 hr = > buf->page->pp->p.offset; > ./drivers/net/ethernet/intel/libeth/rx.c:210: > page_pool_recycle_direct(page->pp, page); > ./drivers/net/ethernet/intel/iavf/iavf_txrx.c:1200: u32 hr = > rx_buffer->page->pp->p.offset; > ./drivers/net/ethernet/intel/iavf/iavf_txrx.c:1217: u32 hr = > rx_buffer->page->pp->p.offset; > ./drivers/net/wireless/mediatek/mt76/mt76.h:1800: > page_pool_put_full_page(page->pp, page, allow_direct); > ./include/net/libeth/rx.h:140: page_pool_dma_sync_for_cpu(page->pp, > page, fqe->offset, len); > > Thanks a lot! > Taehee Yoo > > > > > --- > > > > Changes from rfc: > > 1. Rebase on net-next's main branch > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/ > > 2. Fix a build error reported by kernel test robot > > https://lore.kernel.org/all/202505100932.uzAMBW1y-lkp@intel.com/ > > 3. Add given 'Reviewed-by's, thanks to Mina and Ilias > > 4. Do static_assert() on the size of struct netmem_desc instead > > of placing place-holder in struct page, feedbacked by Matthew > > 5. Do struct_group_tagged(netmem_desc) on struct net_iov instead > > of wholly renaming it to strcut netmem_desc, feedbacked by > > Mina and Pavel > > > > Byungchul Park (18): > > netmem: introduce struct netmem_desc struct_group_tagged()'ed on > > struct net_iov > > netmem: introduce netmem alloc APIs to wrap page alloc APIs > > page_pool: use netmem alloc/put APIs in __page_pool_alloc_page_order() > > page_pool: rename __page_pool_alloc_page_order() to > > __page_pool_alloc_large_netmem() > > page_pool: use netmem alloc/put APIs in __page_pool_alloc_pages_slow() > > page_pool: rename page_pool_return_page() to page_pool_return_netmem() > > page_pool: use netmem put API in page_pool_return_netmem() > > page_pool: rename __page_pool_release_page_dma() to > > __page_pool_release_netmem_dma() > > page_pool: rename __page_pool_put_page() to __page_pool_put_netmem() > > page_pool: rename __page_pool_alloc_pages_slow() to > > __page_pool_alloc_netmems_slow() > > mlx4: use netmem descriptor and APIs for page pool > > page_pool: use netmem APIs to access page->pp_magic in > > page_pool_page_is_pp() > > mlx5: use netmem descriptor and APIs for page pool > > netmem: use _Generic to cover const casting for page_to_netmem() > > netmem: remove __netmem_get_pp() > > page_pool: make page_pool_get_dma_addr() just wrap > > page_pool_get_dma_addr_netmem() > > netdevsim: use netmem descriptor and APIs for page pool > > mm, netmem: remove the page pool members in struct page > > > > drivers/net/ethernet/mellanox/mlx4/en_rx.c | 46 ++++---- > > drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +- > > drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 4 +- > > drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +- > > .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 18 ++-- > > .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 2 +- > > .../net/ethernet/mellanox/mlx5/core/en_main.c | 15 ++- > > .../net/ethernet/mellanox/mlx5/core/en_rx.c | 66 ++++++------ > > drivers/net/netdevsim/netdev.c | 18 ++-- > > drivers/net/netdevsim/netdevsim.h | 2 +- > > include/linux/mm.h | 5 +- > > include/linux/mm_types.h | 11 -- > > include/linux/skbuff.h | 14 +++ > > include/net/netmem.h | 101 ++++++++++-------- > > include/net/page_pool/helpers.h | 11 +- > > net/core/page_pool.c | 97 +++++++++-------- > > 16 files changed, 221 insertions(+), 201 deletions(-) > > > > > > base-commit: f44092606a3f153bb7e6b277006b1f4a5b914cfc > > -- > > 2.17.1 > > > >