From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4102ED75BCE for ; Thu, 21 Nov 2024 08:04:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5958F6B007B; Thu, 21 Nov 2024 03:04:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 545A16B0082; Thu, 21 Nov 2024 03:04:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 435166B0083; Thu, 21 Nov 2024 03:04:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 286516B007B for ; Thu, 21 Nov 2024 03:04:18 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AD0F0160A14 for ; Thu, 21 Nov 2024 08:04:17 +0000 (UTC) X-FDA: 82809360900.20.3507A75 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf14.hostedemail.com (Postfix) with ESMTP id EB0CF100004 for ; Thu, 21 Nov 2024 08:03:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732176193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w1hkRnD91m5+GPeZtjvmrxK32K61Xk/KmepA9QQhO8M=; b=SG/JZwS+8Kvd7zchhCec1KJWhVlNBfCC66O7eqynjg7nYB8xZ9oU06u6NLN9MKkq9xu5Ae Uu9/MNcv5bAPnkPNHabJjIu3LyBtCZnWF0RoGTsd1llGQ0MNLIskMUnoBMtXjp+BmHj4k8 xQ/yg1zDbiK7INkWh1dM0QQU9vr0hTU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732176193; a=rsa-sha256; cv=none; b=ESH4kLYGhN6by+3vWjdHhxuZIazeg7uyiG8jjz+h9ti8ZzOkW6HLImIhYHf8pWiw+j61dA a1ifOX7F/n70bbgTUBIHYmBybp9feaDMBRIFXLV1/upfe4CidWR/sLmeUpsrwWGfLCEHJT AWrkcl0ydFtj4a105SEJ0IdUwGRXoVw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4Xv9fd0mMvz1T9JH; Thu, 21 Nov 2024 16:01:29 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 8530D1800F2; Thu, 21 Nov 2024 16:04:08 +0800 (CST) Received: from [10.67.120.129] (10.67.120.129) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Thu, 21 Nov 2024 16:04:07 +0800 Message-ID: <57dea3ef-6982-4946-bf96-8ebadf6f883c@huawei.com> Date: Thu, 21 Nov 2024 16:04:07 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC v4 3/3] page_pool: skip dma sync operation for inflight pages To: Robin Murphy , , , CC: , , , Alexander Duyck , Andrew Morton , IOMMU , MM , Jesper Dangaard Brouer , Ilias Apalodimas , Eric Dumazet , Simon Horman , , References: <20241120103456.396577-1-linyunsheng@huawei.com> <20241120103456.396577-4-linyunsheng@huawei.com> <15f937d7-7ac1-4a7f-abcd-abfede191b51@arm.com> Content-Language: en-US From: Yunsheng Lin In-Reply-To: <15f937d7-7ac1-4a7f-abcd-abfede191b51@arm.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.67.120.129] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemf200006.china.huawei.com (7.185.36.61) X-Stat-Signature: du5yiz6tn8g3sbrgyg747w6bnd6auzjy X-Rspam-User: X-Rspamd-Queue-Id: EB0CF100004 X-Rspamd-Server: rspam02 X-HE-Tag: 1732176195-563593 X-HE-Meta: U2FsdGVkX18F6IBXV13p6hLYmP9hKRWqwFA4ZUgmoqZ88RFCxdkgNKAy/ZPsr4nMtiAvc1qKepwSB2ym5pKQa3LAsLwjF5s4VPFsC2OvOKa/Hb66vrBmzVK5qWZYb5KzbRMMUBGv/00gIOUjoQzfNsoA7GcvcE+n8ZQtlQ+D7tKfSWiRwzIrA24GRQx1MA1rjuYUNNt4y+gX7DzNbO1uDNcsbnF8sHaIQEXkekOkp0X+ghBTSx87lU3t3p/oYF2MIiAc/r/U/ZQEsyzTcN04/4/qjVLnOdyoBgCDzcCtizhKx+6MRQCaH/M8U3zGiufa4HquAItZgHfeSJOdg0yq0polOpLyLp4WYVIXJBlOgnBsltoMT8+J8QVXhKwUcLfJ7DWjaEXKHpvcghCpIaPKBwWtsNEj5NS2DwFNyMqudvqX1g6kOvtnfaR8JWgXqi7NfZJiXDbfRxET0ux03eBZRNpg9v/2XL06AEJsjermM6oEYagGaBQgmbkVSE7NmPHLyl7fH57sxy0+unXGNo/+lEwonFqAqTawHvdpfvRIVxHFALd5KxaID11VAPTOI4TrQv5z/EQWF932uLOin8U14TBqwcI8YE8YhXmNQOErJ5laGsMx9qC7i4+DLfCkpFOanPJv+/GzIQ9SPlHa8o+AQSdZ+9w6zckr4KMiwYcUldg3OxBvA0vKC7zpIOejhJ+pMnyWFksay9dAxhQEwEzdAOvNsjhNGjWfJMjcjcyNlelwXDn2GLs1vfhvDEo2yXDDmELtWg2AH2xOuAt8qt0/7A289NW1Msdt3bsZz8Lrz+tYter2SCX8eO8Dw2jQQy1gWPvKkEwYGwbWEPcAQccfIR0evlqJeJ4JGD8E9EApFOYc13rSaW7QDFm8azeLhEq7HcZnH+egC5iBStHEreo/91qsst2IH6q0686u5Y7kPN36TGcdZGogbPtttYj5DahX/CE3ASDUUUzoE505k/I VIH8zQDK FHtTgmGF67wrbuR3PMa1xq/MXQ3bOJeCxke4Y2p/wQ+x++WQRqu99RC+arTJX7iHx4bdzMp30/8+30UEIpsvRTlO3XqoUxkuPcvLCA/NChTropFpiAGOoK8+tPuL0lhf+jqg86tz3vxRfH8Mxion/jru6Y2sYyoRxWFMs0/uZ9X3bDNa2A6N48Eg+oj2LvFytAhDrIFqB5IC/rjtJQQghxD99X7l576LUQE7/DR38ekgPYSARxyfjV29KuzdBsI5einBZxtiovJlIKFxks4z1P4PIxx4S/doaJCwe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/11/21 0:17, Robin Murphy wrote: > On 20/11/2024 10:34 am, Yunsheng Lin wrote: >> Skip dma sync operation for inflight pages before the >> page_pool_destroy() returns to the driver as DMA API >> expects to be called with a valid device bound to a >> driver as mentioned in [1]. >> >> After page_pool_destroy() is called, the page is not >> expected to be recycled back to pool->alloc cache and >> dma sync operation is not needed when the page is not >> recyclable or pool->ring is full, so only skip the dma >> sync operation for the infilght pages by clearing the >> pool->dma_sync under protection of rcu lock when page >> is recycled to pool->ring to ensure that there is no >> dma sync operation called after page_pool_destroy() is >> returned. > > Something feels off here - either this is a micro-optimisation which I wouldn't really expect to be meaningful, or it means patch #2 doesn't actually do what it claims. If it really is possible to attempt to dma_sync a page *after* page_pool_inflight_unmap() has already reclaimed and unmapped it, that represents yet another DMA API lifecycle issue, which as well as being even more obviously incorrect usage-wise, could also still lead to the same crash (if the device is non-coherent). For a page_pool owned page, it mostly goes through the below steps: 1. page_pool calls buddy allocator API to allocate a page, call DMA mapping and sync_for_device API for it if its pool is empty. Or reuse the page in pool. 2. Driver calls the page_pool API to allocate the page, and pass the page to network stack after packet is dma'ed into the page and the sync_for_cpu API is called. 3. Network stack is done with page and called page_pool API to free the page. 4. page_pool releases the page back to buddy allocator if the page is not recyclable before doing the dma unmaping. Or do the sync_for_device and put the page in the its pool, the page might go through step 1 again if the driver calls the page_pool allocate API. The calling of dma mapping and dma sync API is controlled by pool->dma_map and pool->dma_sync respectively, the previous patch only clear pool->dma_map after doing the dma unmapping. This patch ensures that there is no dma_sync for recycle case of step 4 by clearing pool->dma_sync. The dma_sync skipping should also happen before page_pool_inflight_unmap() is called too because all the caller will see the clearing of pool->dma_sync after synchronize_rcu() and page_pool_inflight_unmap() is called after the same synchronize_rcu() in page_pool_destroy(). > > Otherwise, I don't imagine it's really worth worrying about optimising out syncs for any pages which happen to get naturally returned after page_pool_destroy() starts but before they're explicitly reclaimed. Realistically, the kinds of big server systems where reclaim takes an appreciable amount of time are going to be coherent and skipping syncs anyway. The skipping is about skipping the dma sync for those inflight pages, I should make it clearer that the skipping happens before the calling of page_pool_inflight_unmap() instead of page_pool_destroy() in the commit log. >