From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 725A3C07548 for ; Wed, 15 Nov 2023 09:21:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E58316B0320; Wed, 15 Nov 2023 04:21:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E07086B032E; Wed, 15 Nov 2023 04:21:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCEAC6B0331; Wed, 15 Nov 2023 04:21:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BDDC16B0320 for ; Wed, 15 Nov 2023 04:21:25 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 92C2D120364 for ; Wed, 15 Nov 2023 09:21:25 +0000 (UTC) X-FDA: 81459645330.29.C7C9E3E Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf17.hostedemail.com (Postfix) with ESMTP id C7CDA40012 for ; Wed, 15 Nov 2023 09:21:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf17.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700040083; a=rsa-sha256; cv=none; b=7QQ85I41gevQcwzNDrfs2TSByayDwjUcpaZ3wPpKBVHrw1IOcw+9sFGl4bDr8qRrMHNn6R qZ+uwGgQoaz3pEuuCIIhhQdSCYzL0rtsN6IR87+HLvKwoUP2yfVOuwfMA39MSLtGhGeuPr 0gu8iBPBJyN/mjE1kyZTg63ngAvuWT0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf17.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700040083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fcFUZa+OxVcMqVssDVCLudOoaSxMr168F1svsxSBhqw=; b=uYXsHzYCZytszowI0VvpUNzsObjpsk5fmsBM1ibF5ZaNcRHN6PGqWlV1CGC14bQV5H/cvN Y8f+/yoIblBH+Orgb/kL1p5NG48IdEUESrC1vts5QXUm4aT9nTnr+M5rvVvxeoej1jgNqm kh0HGdfqGtxNnVVzg2mRYUgzX2E5Uks= Received: from dggpemm500005.china.huawei.com (unknown [172.30.72.53]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4SVcyG45DXzmXDl; Wed, 15 Nov 2023 17:17:42 +0800 (CST) Received: from [10.69.30.204] (10.69.30.204) by dggpemm500005.china.huawei.com (7.185.36.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 15 Nov 2023 17:21:02 +0800 Subject: Re: [PATCH RFC 3/8] memory-provider: dmabuf devmem memory provider To: Jason Gunthorpe , Mina Almasry CC: Jakub Kicinski , , , , , Willem de Bruijn , Kaiyuan Zhang , Jesper Dangaard Brouer , Ilias Apalodimas , Eric Dumazet , =?UTF-8?Q?Christian_K=c3=b6nig?= , Matthew Wilcox , Linux-MM References: <20231113130041.58124-1-linyunsheng@huawei.com> <20231113130041.58124-4-linyunsheng@huawei.com> <20231113180554.1d1c6b1a@kernel.org> <0c39bd57-5d67-3255-9da2-3f3194ee5a66@huawei.com> From: Yunsheng Lin Message-ID: Date: Wed, 15 Nov 2023 17:21:02 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.69.30.204] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm500005.china.huawei.com (7.185.36.74) X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C7CDA40012 X-Stat-Signature: mn8b1a5iq17dfxq5dnst5czis4tojqhb X-Rspam-User: X-HE-Tag: 1700040081-883707 X-HE-Meta: U2FsdGVkX1/gtLr7DUbHZOZvMUyLQbtceflhUsGTrUrii70EVhXjIICHRwnqf4h5SIDbJEnQ3gML13HSJskgGwyYP3vGQJoUaad+UlHXEDq1OAjloyLn/y4hXZsaFzsVRnrXBacFfjJur55VzipRCW+5pJ6FYWGoKAQgpv+Gs6KXxUo8MwcI0SJFQVBVRqpPGmXxytLLz2/khpbvxiz+mxuPyK31TM2THyKscohQ4UhEdEj5Q9gX2uc9gDbqRh7uaCOu0SXxRYxoyUEhxu266QkFVARadp/31VosMFEdNV8My2MpbAq7FYbnIiSE9oKwXmM31K0SWvcQiZi1qS4cqc9T6HWJkilBklsD/slyo86NzvAIwg6fW1dHEWbuNI80JeJi9qVeGGnTfZz8NyGm8e13FpsG0M03hbqc46ZFuHPbdh8ztSS7pR+SUbwZPh5y0q8op6ic3rfMWcV/h+GHhufZDSVbBGhjhgtkX9YTsp6ZrrOTmFOG/wvEijHyhYPNe/UbXEdDo0lOEt2h5powr1PIxB/0e6tduJ71nZ3MnK+COHKg0Xfz78jq00Mb7t4hx0yuTDoQ/57CRWoFXgIUPa0/mgAQSJZqq4ZFUe28oECsiLMbSqi6OCWHfwL+A45TqXPHn20EM9Yt1+lkL9uZ8gDXUxbMfQQqBeLdVVbzEEFaFATmG9HSlokmaXB6n/gzVcVyQfVoLQ77DHOfia0Im3vcd5aIjYMW8qc9mwmDh7lSLO97d2qSxoPF99iDVDMErdvsP6ZNTRbLupnNK31gQFD09yLJ/C8F7pvsRKfGxBD1R43Aw6FMWbjqRMoEGCiKhRFVFh8WPlyqvLb80CJfve3DGbMEVT0Y5bmlSYTRNQU2/WfYxHNCRVBlvlKBjq47xIAvBVRQ0Jrkm3/3kHD7SSJWZBIlh5StSsIswY4jtkXwhu5XCccLO9Ay3fQsUE91OtPZ2vXPmJtc3IEkvXT v3ePECKN XTUzfwOaLzgTeHWzGXW9A3+opi/Me1y0t1G1Ikn8NwR3/vs2PwPdhHejJ+Yvd+htKCBVTnt2pej6FIg6nATgBGAt9ApshnzMlwFAUjDqbbhcwLh5B6ldZF/YSvjmAmGO2L6xGTz9OWcvQyZA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2023/11/14 21:16, Jason Gunthorpe wrote: > On Tue, Nov 14, 2023 at 04:21:26AM -0800, Mina Almasry wrote: > >> Actually because you put the 'strtuct page for devmem' in >> skb->bv_frag, the net stack will grab the 'struct page' for devmem >> using skb_frag_page() then call things like page_address(), kmap, >> get_page, put_page, etc, etc, etc. > > Yikes, please no. If net has its own struct page look alike it has to > stay entirely inside net. A non-mm owned struct page should not be > passed into mm calls. It is just way too hacky to be seriously > considered :( Yes, that is something this patchset is trying to do, defining its own struct page look alike for page pool to support devmem. struct page for devmem will not be called into the mm subsystem, so most of the mm calls is avoided by calling into the devmem memory provider' ops instead of calling mm calls. As far as I see for now, only page_ref_count(), page_is_pfmemalloc() and PageTail() is called for devmem page, which should be easy to ensure that those call for devmem page is consistent with the struct page owned by mm. I am not sure if we can use some kind of compile/runtime checking to ensure those kinds of consistency? > >>> I would expect net stack, page pool, driver still see the 'struct page', >>> only memory provider see the specific struct for itself, for the above, >>> devmem memory provider sees the 'struct page_pool_iov'. >>> >>> The reason I still expect driver to see the 'struct page' is that driver >>> will still need to support normal memory besides devmem. > > I wouldn't say this approach is unreasonable, but it does have to be > done carefully to isolate the mm. Keeping the struct page in the API > is going to make this very hard. I would expect that most of the isolation is done in page pool, as far as I can see: 1. For control part: the driver may need to tell the page pool which memory provider it want to use. Or the administrator specifies which memory provider to use by some netlink-based cmd. 2. For data part: I am thinking that driver should only call page_pool_alloc(), page_pool_free() and page_pool_get_dma_addr related function. Of course the driver may need to be aware of that if it can call kmap() or page_address() on the page returned from page_pool_alloc(), and maybe tell net stack that those pages is not kmap()'able and page_address()'able. > > Jason > . >