From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E3DCE8D78 for ; Thu, 19 Sep 2024 10:54:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32D696B008A; Thu, 19 Sep 2024 06:54:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B5DB6B0095; Thu, 19 Sep 2024 06:54:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 156F86B0098; Thu, 19 Sep 2024 06:54:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E99CA6B008A for ; Thu, 19 Sep 2024 06:54:35 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6ABACAB0C6 for ; Thu, 19 Sep 2024 10:54:35 +0000 (UTC) X-FDA: 82581179310.03.8B111AB Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf19.hostedemail.com (Postfix) with ESMTP id 1CFB11A000A for ; Thu, 19 Sep 2024 10:54:31 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726743160; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eJPKc6WpQJVo+cZJ0F30D2qYLx9MeNdOoR7H0Px8Ma4=; b=UoG3pt1V3DPYS2x2RFiqftPC76KehvVFZUf/d9g5Ydy/Eu1gCrvTA61LkzQv8PxKAw9xNv N4VWnRPc9ZoTZ8719gYOER6I69djMTLNfUe/kQFInQkClALMwgBzZUiGgBHp4yFhNo7+7V l2jQ6s3jFgvscnMUtLIL+lERBKjtSN4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726743160; a=rsa-sha256; cv=none; b=HakurZFWyTAd9iAsdwKJNOJDNd2a1wuv+osnVYc3Pq/9gO7RUSWhRGmAC2Q0/u8KO3Oxdv N69VT4by9Sq43ZPu/uE2kwII/6utrFOC0oZ0sTRhJ+4e1LlcwgX/vGwOxKcQDR+Mxb4J5W fGzcH1/a4PCp35mT32djL/m55VpQVGE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4X8XQv3YptzmYlT; Thu, 19 Sep 2024 18:52:23 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 5515E140390; Thu, 19 Sep 2024 18:54:27 +0800 (CST) Received: from [10.67.120.129] (10.67.120.129) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Thu, 19 Sep 2024 18:54:26 +0800 Message-ID: <50a463d5-a5a1-422f-a4f7-d3587b12c265@huawei.com> Date: Thu, 19 Sep 2024 18:54:26 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH net 2/2] page_pool: fix IOMMU crash when driver has already unbound To: Ilias Apalodimas CC: , , , , , , Robin Murphy , Alexander Duyck , IOMMU , Wei Fang , Shenwei Wang , Clark Wang , Eric Dumazet , Tony Nguyen , Przemek Kitszel , Alexander Lobakin , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Felix Fietkau , Lorenzo Bianconi , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Andrew Morton , , , , , , , , , , References: <20240918111826.863596-1-linyunsheng@huawei.com> <20240918111826.863596-3-linyunsheng@huawei.com> Content-Language: en-US From: Yunsheng Lin In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.67.120.129] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf200006.china.huawei.com (7.185.36.61) X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1CFB11A000A X-Stat-Signature: fdxy3w97yb3o614t4o683ht6yzpckbe1 X-HE-Tag: 1726743271-21077 X-HE-Meta: U2FsdGVkX1/O3/N4v774R/dplm/gZYpKKOAMloUzyoCVEvnswy5Bu3eYl2YgPUSCtSigqJbqV1r1d70L9yI4QeZiJGOIjSYRcVgT5FUa9Hd2vkLY+XSZFItX81vBqepBVZNmrENtm6c+7dbUndgyOVBmOYSMLiWptJbifiJtteWZxfe8j6h5w173bRuvD0yFMF3hzqUG5Ou5UfWyRuCBmS5TfAe1pFn9ekkIzploDdzdU2F9310Tgc7r9QHrRKaD5+yVwEsGO28DCZXiZlTjYuw+qJe3WaAQ4gxFwm7fqEHVQUO80VDZIUv4i/wLOk66N80J4rXmXpdpbKc2s2aEo2mniy7m/z2ELs+qVzR80iavi4jTsw6XnHzNUdU2qMu7PshxWqaw+nSBe7T2W0Teo0zvYXgDOUX8NBU4lfAK4bYcQstIg7hkPZeid5Kui3iZgFTqFkp7VYwuiqck8mkfkk7r6phtKSHb0wXH47JcA9mD97RNJtk1+1/OIIWtursdHoFgKGi+yYKolNP3ydNBYiD5PA0ueMh6Fn1xpUGqw3tR6U8oHpul7pdeyUciz3zhTg5KrdHpcBW3khkW40KJyYfL15CH5WEDI+OLBxCjUCNjGx53ME3SDFQRjfvKcohMs8Iq3mnMge6axGe4DQwesn9cruvzQc33IBn/WRBG7TqVKGgWDGbUH7uXUfI1/zCMMXFQ8P38NhDqZKqRoRekojtDid89XYR0tByaPbEOODPWruOIv3rl1K7O9HAlbEPWxHxUM97QfkSAiGxA7RX8C6YRkbg3IMM9McHMpcyq+fYquhCpwVHitc1qq1oj09g2Br6xK+Km0XPYqR4WguNAAcS4p0u6FKqAsx5foMNDIkMJzHDD9hKduGb07FRuBcxD+LVqQr3pqNRHX2eMqFthJs0yTT+PCRMOSSsgWW+NsmF1z/YAraIYnfmC+h8fnlrvRDP6QtzR9RyZ0iwVIwI kLzHD/Dt 8UtctudmB4pSihH3skn0gY7KoQK3fPfBM/u0PLetilC1qyx7BVXEMHXWRSnFUDpjjMLjvbIIst2w1DpeuV3KX0Ym3C3BO2/ntlTTObp8OkZMQDw3Jnqnx1RU3QlGV6CgVFmV07JjATxO3tbt9GgvDjm3No2uW5fs0H11rb/Ec8IkhCM4NVstoDZL/sHsxAWT8wtaomMQTlD+GgqxTP2q6DHlp3WrEnKzYtbL4jQOz7G4UfyJa4N1PYwJXVTK5Of6ZGzpjz9NkHs5qKlTU6ODbRkjh+WYJ6D2on2eJIVXuQFGb5oAT8XmEN45e4JSWVgusa2Xt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/9/19 1:06, Ilias Apalodimas wrote: > Hi Yunsheng, > > Thanks for looking into this! > > On Wed, 18 Sept 2024 at 14:24, Yunsheng Lin wrote: >> >> Networking driver with page_pool support may hand over page >> still with dma mapping to network stack and try to reuse that >> page after network stack is done with it and passes it back >> to page_pool to avoid the penalty of dma mapping/unmapping. > > I think you can shorten this to "If recycling and DMA mapping are > enabled during the pool creation" I am not sure if I understand the 'recycling' part here. Is the 'recycling' part referring to whether skb_mark_for_recycle() is called to enable recycling for the skb? Is there still any driver with page_pool support but doesn't call skb_mark_for_recycle() when handing over page to network stack? For the 'DMA mapping' part, as there is no space in 'struct page' to track the inflight pages, so 'pp' in 'struct page' is renamed to 'pp_item' to enable the tracking of inflight page. I tried shortening this for 'pool->dma_map being false' when coding, but it seems differentiating the same field in 'struct page' doesn't make much sense according to 'pool->dma_map' as it means we might need to add an union in 'struct page' for that to work and add additional checking to decide if it is 'pp' or 'pp_item'. > >> With all the caching in the network stack, some pages may be >> held in the network stack without returning to the page_pool >> soon enough, and with VF disable causing the driver unbound, >> the page_pool does not stop the driver from doing it's >> unbounding work, instead page_pool uses workqueue to check >> if there is some pages coming back from the network stack >> periodically, if there is any, it will do the dma unmmapping >> related cleanup work. >> >> As mentioned in [1], attempting DMA unmaps after the driver >> has already unbound may leak resources or at worst corrupt >> memory. Fundamentally, the page pool code cannot allow DMA >> mappings to outlive the driver they belong to. >> >> Currently it seems there are at least two cases that the page >> is not released fast enough causing dma unmmapping done after >> driver has already unbound: >> 1. ipv4 packet defragmentation timeout: this seems to cause >> delay up to 30 secs: >> >> 2. skb_defer_free_flush(): this may cause infinite delay if >> there is no triggering for net_rx_action(). >> >> In order not to do the dma unmmapping after driver has already >> unbound and stall the unloading of the networking driver, add >> the pool->items array to record all the pages including the ones >> which are handed over to network stack, so the page_pool can >> do the dma unmmapping for those pages when page_pool_destroy() >> is called. > > So, I was thinking of a very similar idea. But what do you mean by > "all"? The pages that are still in caches (slow or fast) of the pool > will be unmapped during page_pool_destroy(). Yes, it includes the one in pool->alloc and pool->ring. > Don't we 'just' need a list of the inflight packets and their pages or > fragments? What we could do is go through that list and unmap these > pages during page_pool_destroy(). The main reason for that is to avoid the overhead of page_pool_item_del() and page_pool_item_add() when allocing/freeing page from/to pool->alloc and pool->ring. Yes, including the pages in pool->ring seems to make the pool->ring somewhat duplicated, maybe we can remove pool->ring if we can make and prove 'pool->items' is performing better than pool->ring in the future? > > I'll have a closer look at the patch tomorrow Thanks for the reviewing. > > Thanks! > /Ilias >