From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A82BDE64A93 for ; Tue, 3 Dec 2024 13:30:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E3616B0085; Tue, 3 Dec 2024 08:30:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 392D76B0088; Tue, 3 Dec 2024 08:30:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2827D6B0089; Tue, 3 Dec 2024 08:30:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 083056B0085 for ; Tue, 3 Dec 2024 08:30:18 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B4A238093E for ; Tue, 3 Dec 2024 13:30:17 +0000 (UTC) X-FDA: 82853731422.16.7DDF52C Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by imf28.hostedemail.com (Postfix) with ESMTP id A10A3C0032 for ; Tue, 3 Dec 2024 13:29:58 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733232606; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KTyM5YpeX+vR4KNvtmqP4+zs34jr26gEbc54i9sM4io=; b=n0xYoAydKTtNyyqmasTcnF6MDDBmmABwa1q0Vvfa/5MhV4sIulhduCPJ8tcaq5yckDHAew 4KtFMY2L8jFz9BHndVrGmsK6NiAS9aFlPlJc9S+SjnT+5h03mSp4xDGtTWbMcueTO/woaV mYRxaofyULPrPkykmUJ5VaQaVK87gf0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733232606; a=rsa-sha256; cv=none; b=1qdi9HHQ0uWtni9eRtdjZDxeP3iPbYKDqD+ehWeLLuqn08aqxk30QlbjufnDtyRgPoCB5P SlMFz1ZO/gMS1GjWJD5QnldHbin0WzgMV2eCihpA+ppi+H8jzom6cNVY5c4NZ7icsh7bX6 ZIAUmPVfInc3aCfab6U9lwubxoOKO3o= Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4Y2hNC67SHz1JD6m; Tue, 3 Dec 2024 21:30:03 +0800 (CST) Received: from dggpemf100008.china.huawei.com (unknown [7.185.36.138]) by mail.maildlp.com (Postfix) with ESMTPS id 771F11A0188; Tue, 3 Dec 2024 21:30:10 +0800 (CST) Received: from [10.174.177.243] (10.174.177.243) by dggpemf100008.china.huawei.com (7.185.36.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 3 Dec 2024 21:30:09 +0800 Message-ID: Date: Tue, 3 Dec 2024 21:30:09 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH -next] mm: usercopy: add a debugfs interface to bypass the vmalloc check. To: zuoze , Uladzislau Rezki CC: Matthew Wilcox , , , , , References: <20241203023159.219355-1-zuoze1@huawei.com> <57f9eca2-effc-3a9f-932b-fd37ae6d0f87@huawei.com> <92768fc4-4fe0-f74a-d61c-dde0eb64e2c0@huawei.com> Content-Language: en-US From: Kefeng Wang In-Reply-To: <92768fc4-4fe0-f74a-d61c-dde0eb64e2c0@huawei.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.177.243] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf100008.china.huawei.com (7.185.36.138) X-Rspamd-Server: rspam05 X-Stat-Signature: gjms9jccphb3egf3hjc3u7zpx8dp1zkb X-Rspamd-Queue-Id: A10A3C0032 X-Rspam-User: X-HE-Tag: 1733232598-846119 X-HE-Meta: U2FsdGVkX19LnAlRv59avf8Q627QzOXkl9EM3lZeAKuQkQ370tX5Xsp9lJOe4G2QikVQaf455vwOvP1bRclirwlTG9V3byO97lLrJPOJXX9CLUYOpVMA2d3HwBtd0LMBN1U2XlBku0yUjnUq8gmVtyrGdjMjfZVNH/K8g2/P6voQm2QYxu0jFa807iZpbljr/HTSsW5r61FuAfNp8Asv2wM2uEJyXuykNHoSJd72JANwxMCilY6K8omtGQyPV4UzXfOmWHV0uToMPAKaxmNvY9ZSoFgf3mfol3JEsxaV4yTn9XezGuB4YUg3R6wSGiNxEyJBOfJXeUS+yNogWNcq1b/egybTRaYxHR4YDXVaoRA6KhgSPwf/uzWQr5hTP/Op29bfUONdFCpyi02hWIOBZooVyp0iFphyRBqBizSTKEK5CKt0y6vIg5KHJ9cxP7mf8hqYtyMdkmBSSI1HkIgNVX15VUrn5DBLVwdkqL4Bh5jW+OSAPlYJS4vudexXqxK1yGQzdFfvruY2h5ZOGNBLP4Ykvz8OMCbj9mrkygz245VOsRheYyFoLVpcSaKOMFT48R1GCXq8PKLNfYCVXMVfJ5OWMC6678XREEsH9xdrlK8uP1HsA1Elq4xXUDAPBrX+LoXqNt7zgdDbPIcXZgK+sscl/mzY2mN+GXeLySQqbukknWJoPo4mkdvbj9kcRevAJgVWKebKFRbO1UN60wHHgfCoaYeTHyXJ6x90+Anlh3hE+0SsOfJGostzP+OF4llHSC4sYgyWJiA+diJQ+HrE5TnH1fIqLdEW+Pn63stxKFWqWkGcdjZK+Riz4GWUXdxCSZW7cEAWfe9rlh4D4JNB+FtXdv1lrbjwavZM1Q6Z32taSYngzInLnFkRgZ5fj2Cm2ltf+ITS2/WdooX0/VKuuH/RDtycbMjin/sQsiEGwyI5CNRNdj/nu8URXns7ltYHYmYZrxTKXKjH6dzskM/ 8yn2yuy1 1Aw2r5Bke54galE33l6pj63tyv9XuMnr7QKYcdHAmMJi/Gq2n1bUtudpXZzZuqG/1z7GwO/IGVZBhlEVx8xM/skNdXOnFrU3D1Bsmlu9waFb1xIryT85dMrE0vRlepNiwQnF2HzesShh4jpVdsBJAyGxPIhhb70lxV6wArcVOjHtdDyOar0EOfhb5XZspZlX1vB8E8tqFOeQuxGxyZe2eDapycTtRUHDC3wkE9sTXYH5dMhMsMKswxHgoa+fUIEgAPJ2yfpCX9jLVpPnJP4vwkW/bPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000029, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/12/3 21:10, zuoze wrote: > > > 在 2024/12/3 20:39, Uladzislau Rezki 写道: >> On Tue, Dec 03, 2024 at 07:23:44PM +0800, zuoze wrote: >>> We have implemented host-guest communication based on the TUN device >>> using XSK[1]. The hardware is a Kunpeng 920 machine (ARM architecture), >>> and the operating system is based on the 6.6 LTS version with kernel >>> version 6.6. The specific stack for hotspot collection is as follows: >>> >>> -  100.00%     0.00%  vhost-12384  [unknown]      [k] 0000000000000000 >>>     - ret_from_fork >>>        - 99.99% vhost_task_fn >>>           - 99.98% 0xffffdc59f619876c >>>              - 98.99% handle_rx_kick >>>                 - 98.94% handle_rx >>>                    - 94.92% tun_recvmsg >>>                       - 94.76% tun_do_read >>>                          - 94.62% tun_put_user_xdp_zc >>>                             - 63.53% __check_object_size >>>                                - 63.49% __check_object_size.part.0 >>>                                     find_vmap_area >>>                             - 30.02% _copy_to_iter >>>                                  __arch_copy_to_user >>>                    - 2.27% get_rx_bufs >>>                       - 2.12% vhost_get_vq_desc >>>                            1.49% __arch_copy_from_user >>>                    - 0.89% peek_head_len >>>                         0.54% xsk_tx_peek_desc >>>                    - 0.68% vhost_add_used_and_signal_n >>>                       - 0.53% eventfd_signal >>>                            eventfd_signal_mask >>>              - 0.94% handle_tx_kick >>>                 - 0.94% handle_tx >>>                    - handle_tx_copy >>>                       - 0.59% vhost_tx_batch.constprop.0 >>>                            0.52% tun_sendmsg >>> >>> It can be observed that most of the overhead is concentrated in the >>> find_vmap_area function. >>> >> I see. Yes, it is pretty contented, since you run the v6.6 kernel. There >> was a work that tends to improve it to mitigate a vmap lock contention. >> See it here: https://lwn.net/Articles/956590/ >> >> The work was taken in the v6.9 kernel: >> >> >> commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc >> Author: Uladzislau Rezki (Sony) >> Date:   Tue Jan 2 19:46:23 2024 +0100 >> >>      mm: vmalloc: add va_alloc() helper >> >>      Patch series "Mitigate a vmap lock contention", v3. >> >>      1. Motivation >> ... >> >> >> Could you please try the v6.9 kernel on your setup? >> >> How to solve it, probably, it can be back-ported to the v6.6 kernel. > > All the vmalloc-related optimizations have already been merged into 6.6, > including the set of optimization patches you suggested. Thank you very > much for your input. > It is unclear, we have backported the vmalloc optimization into our 6.6 kernel before, so the above stack already with those patches and even with those optimization, the find_vmap_area() is still the hotpots.