From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EB56E64AA3 for ; Tue, 3 Dec 2024 13:40:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8329D6B0085; Tue, 3 Dec 2024 08:40:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E2DA6B0088; Tue, 3 Dec 2024 08:40:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 683A46B008C; Tue, 3 Dec 2024 08:40:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 43A006B0085 for ; Tue, 3 Dec 2024 08:40:23 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 40515160B04 for ; Tue, 3 Dec 2024 13:39:21 +0000 (UTC) X-FDA: 82853754396.29.976CEE1 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf23.hostedemail.com (Postfix) with ESMTP id 62F1D14001B for ; Tue, 3 Dec 2024 13:39:10 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VQ396aYs; spf=pass (imf23.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733233150; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nsXC+tBl68LL521BM7Yb2saeVBRmerDcwFzutaDm/o4=; b=e0M0odBQolFpuC0jyE9GRSYEvpJYRcmTKoShpXqiJGWV/7JPheHS/6lTf4NDh6+V5pGGM5 EtPoCvITSOgZ5dv41QbkgUBnND+/EGT+jjEkUKMLzv7G0zKRT2aq/ztQU1eAojgL+FEKoB BIZR+IHafYXtwkfpIlTITTpeEt2ohbU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733233150; a=rsa-sha256; cv=none; b=raLSy95+KWBW5cqeulJyYu5D1oQCkqHHV7fAi0UV/P3SSl4EbILhEBeThAgTitWA00gU8C ha9aCfwDTi9x4kX72XgRIzDSRiESN2i/zdoRv1D9/LPM7cTS/6+8C0BhfDJpEp5GiEoBp5 9K7JhcqoCkIJPWOBBVFTjTUgp+L6cY4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VQ396aYs; spf=pass (imf23.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-53df119675dso6748240e87.0 for ; Tue, 03 Dec 2024 05:39:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733233157; x=1733837957; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from:from:to :cc:subject:date:message-id:reply-to; bh=nsXC+tBl68LL521BM7Yb2saeVBRmerDcwFzutaDm/o4=; b=VQ396aYsbWHq3RgomFJZA+tBDDdWFwsD5zszkVdhU7gvwTGKbtZba6DApJWnDhYG6a ftrWRSEIo06SBU/y4z74G+AM8tsjPevBcIur5tPMZbmHDAjgtLzVmO1MXoiuq+u4w98U X38MO29ROH7o+XRDF358JsngGcWHO6WfD/a2lhkvcX3XS16kZ3Tuc8VmS7sdrgzrZjvt HP2UXaGBnpeSzYbuJBeeMc/jP46VOALqkq9MnAQHq8qzh4ukyX3mYxYBa6xBTZo43ePT pFs0RbtfjJ/N7PUCFX2f6ng/6dieURRUC3oWeyeMag2W2tR9ifKl3rdyM2AHUokj+42N u5VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733233157; x=1733837957; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nsXC+tBl68LL521BM7Yb2saeVBRmerDcwFzutaDm/o4=; b=HtQBnv0FmOFb109ijyIs+kH4uAmTAnNE5I6ks3nTOXZmXkNafAwhNtHNLJ/C0X8Ok+ midYnXJXA2KPMuXYllwhWUhKsbNSFf7/VkaoXVw/2VFXyuxiN5+uk4KZUKyTdm9/LXdR KqwM8/ZxBCalWK+CJb3boeg2SJxQwbpQLzq/SS3EJXY0j8NroX3iZ3mzme5sJawdx7MS 88BcQW4m9wly/dbfWwVJbQHIBH0avLFp/Wq9O3SV7GSD1+/N7rXSEJKg/xmooYwnRV+c Ib/n1PLkZDRgtFjZWcqRal+ivxGTMRsRGH1ixuAjZh8I3kfr/H/X6g+pIlm8hLJUIvha 4L4Q== X-Forwarded-Encrypted: i=1; AJvYcCVr5kgaKB1DRVV+PVLeneBpe8WT+rRND5OEsrWObg/b+Vhj2WXhvXqlWb8Un9M3Gu/fh3n5h0z1mg==@kvack.org X-Gm-Message-State: AOJu0YyV/IpqkHuaEbY5xqJh3Pe4GIuVS0s44IR8gybqLLBW9YSYcQMr cxl+STBrvQ/3jXUJttAi+Qv6B1MzDr+hk9MNKcMNtzUDRt/z0+eo X-Gm-Gg: ASbGncueSeTCY9BWAfMRBfizxlCo4X7k+pjprDcCkFVs0647CCrm+T5Hbq5T9LR/EE7 NnTN3Upnsw/mSuiXUHRT3QYCmDn6IT3K1UC4Uig39nWhDGKAfbFvey5e+91aqkv8NAWE83cbWTz tYtS38yTZQdk14JgAuJ7fau12jisjYacJWCazeIlkT9clfUCTCSvasBRrdVafbiP9ujd2oh/ef5 8SypvgAZZLB3NEGRLfCF7Vbzh66eytOspftJ+d+dSu8ZzXMAF7vDTOO+xtfT0HEUnZaTUBNaoQ= X-Google-Smtp-Source: AGHT+IFiNea8noRxLx0ybm43E6g5REpoL8P/YyW8t9MOua+uWQAdXbMKpaDuFKTwtpdgb8e32gaI0g== X-Received: by 2002:a05:6512:6d6:b0:53d:ed2f:fab0 with SMTP id 2adb3069b0e04-53e129ef19bmr1789639e87.9.1733233157095; Tue, 03 Dec 2024 05:39:17 -0800 (PST) Received: from pc636 (host-95-203-13-87.mobileonline.telia.com. [95.203.13.87]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53df649f6dcsm1826140e87.243.2024.12.03.05.39.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 05:39:16 -0800 (PST) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Tue, 3 Dec 2024 14:39:14 +0100 To: Kefeng Wang Cc: zuoze , Uladzislau Rezki , Matthew Wilcox , gustavoars@kernel.org, akpm@linux-foundation.org, linux-hardening@vger.kernel.org, linux-mm@kvack.org, keescook@chromium.org Subject: Re: [PATCH -next] mm: usercopy: add a debugfs interface to bypass the vmalloc check. Message-ID: References: <20241203023159.219355-1-zuoze1@huawei.com> <57f9eca2-effc-3a9f-932b-fd37ae6d0f87@huawei.com> <92768fc4-4fe0-f74a-d61c-dde0eb64e2c0@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 62F1D14001B X-Stat-Signature: g3kc84kcfkrqnbabicby6qy5p661copq X-Rspam-User: X-HE-Tag: 1733233150-775655 X-HE-Meta: U2FsdGVkX19oMNk903k7bizPuGxJP10c4dHQX0AhX1TmeF+PS1ZCRr9tMxy7GpfCv2GU27GOFMNc26gJsOnwP+RsQZTY3yjbuiZaffOlsRPBYu298HD0wlUKwM95GE0nCL7JQFsVh6cmYnvAmSUSMirdqJNdXu5//uKHw+rYm1pdxbxLkj+VKu05rcAZ+f21vgVJz94n77kKHBoLmbGxKcEKkO5z0eHtN5yH579JoTogQr72oCy1HACj/9m2x/3N0k0IaIQ0YvtwejqrR4zeGDB7cUUMrAziDzm3XMN/3pdmxKblRIAesGlwNXsjBx+5U6TBpF6f+eAlLv/278QP0zPT85p6jyUAXs3a5KvRVxDng3KU64mNuoPnChZ7sFgFrEQc8qgxL76g3F3B3eoQN0yPdQj4+YEadfgmYLaCylsd8sZxUOe0l5JZogJufp3NYokOWVFyYcZN8fl07u446azywZsAzkB3cGqNE36SRfhiFkA9QKIs2SBsvx+8tROJhKDQLKc135k/Uc71Tmduj1TXX2BI2KPlLk1CpJrgdyI4mLRqkyIoEcoHqUpk9lKAOwlasSKXm8A4hlWmVBOOC3IEoz5oWbeIHbNGKOIl4h6L8wPbfJAr8a73llac7UhciE2/KKQVfivkzlLNnvcYBu+64fJw6UHtKSko/HM6NqeGsolfGd3Fitn1Sj+SDjMxCPsOw7/r/HI/gzYBr/Z7bBB/QB1UeaUCIvzvUQkw9hS9aKDGVLF/SmNXGDT/utwJrjUrx5a6GM/sLGYmYoxaBUPUanMhjvaWGZjJ06laur+Bn0Jh8YAVY/izvJxevN1D8BKNAebyNrz+b+dZ9W8itCpUKOqo19L+ExCh3f1xqZbvylpvKzIhyfjrBJ1IbYTXxNSSnRUtnhcCGoXbE6LT3hJ4youSx8YBwD7QXHLmqrI7nG8tGBKbrlD7vQ1cA0yn5GSdWtjPBK/9r7MelK+ dpgp/jUi 3yrwo2Mgu69Sti+4eCPlgFwjjjEhtZD+6F244KqxC9mlnsYqoGbCKlGq41rhEFCVbma/dZII4i6rb9HHBkRRnlriuwDQo2mpGHXrodRrsc0DcbaSixWv9JR4ucoGlNWysGLdxG2Oxno82Im03Ptjq2elcpOW/DeokED1je8Uem49FdFasEPYx3BAikl8zC0C+O6brSQAlF8lmL96Yi/tC2GjksZv+juP6XQqUbBS4GwUOg/Ko3jDQtevxIGKrrvCT25wt9KhaBXDEDofc2EnwJZsPHVrWbYRdgu09sOI4fRnwJfqyRmklTwOx9BnKhL7viCpFsQjciQDC5kjSGiuZWULnFlZMfib3c0j0wie7bn25gV60TWkunUIIbWrtdBIuszPvohZ6ZK9zLTUns9ikbYdyKE7dqd98l4jhEByTBSKQ2VMGp8BPTmEqyvarML8l+Mtb1+8/oZt3eoBZAsORAWrHoI1jlcRCTxks2M82o9OmL8h9nxIhNjLzA4WWxkmKTeHq4AezVx7uYCB6X/vX/olLFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002751, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Dec 03, 2024 at 09:30:09PM +0800, Kefeng Wang wrote: > > > On 2024/12/3 21:10, zuoze wrote: > > > > > > 在 2024/12/3 20:39, Uladzislau Rezki 写道: > > > On Tue, Dec 03, 2024 at 07:23:44PM +0800, zuoze wrote: > > > > We have implemented host-guest communication based on the TUN device > > > > using XSK[1]. The hardware is a Kunpeng 920 machine (ARM architecture), > > > > and the operating system is based on the 6.6 LTS version with kernel > > > > version 6.6. The specific stack for hotspot collection is as follows: > > > > > > > > -  100.00%     0.00%  vhost-12384  [unknown]      [k] 0000000000000000 > > > >     - ret_from_fork > > > >        - 99.99% vhost_task_fn > > > >           - 99.98% 0xffffdc59f619876c > > > >              - 98.99% handle_rx_kick > > > >                 - 98.94% handle_rx > > > >                    - 94.92% tun_recvmsg > > > >                       - 94.76% tun_do_read > > > >                          - 94.62% tun_put_user_xdp_zc > > > >                             - 63.53% __check_object_size > > > >                                - 63.49% __check_object_size.part.0 > > > >                                     find_vmap_area > > > >                             - 30.02% _copy_to_iter > > > >                                  __arch_copy_to_user > > > >                    - 2.27% get_rx_bufs > > > >                       - 2.12% vhost_get_vq_desc > > > >                            1.49% __arch_copy_from_user > > > >                    - 0.89% peek_head_len > > > >                         0.54% xsk_tx_peek_desc > > > >                    - 0.68% vhost_add_used_and_signal_n > > > >                       - 0.53% eventfd_signal > > > >                            eventfd_signal_mask > > > >              - 0.94% handle_tx_kick > > > >                 - 0.94% handle_tx > > > >                    - handle_tx_copy > > > >                       - 0.59% vhost_tx_batch.constprop.0 > > > >                            0.52% tun_sendmsg > > > > > > > > It can be observed that most of the overhead is concentrated in the > > > > find_vmap_area function. > > > > > > > I see. Yes, it is pretty contented, since you run the v6.6 kernel. There > > > was a work that tends to improve it to mitigate a vmap lock contention. > > > See it here: https://lwn.net/Articles/956590/ > > > > > > The work was taken in the v6.9 kernel: > > > > > > > > > commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc > > > Author: Uladzislau Rezki (Sony) > > > Date:   Tue Jan 2 19:46:23 2024 +0100 > > > > > >      mm: vmalloc: add va_alloc() helper > > > > > >      Patch series "Mitigate a vmap lock contention", v3. > > > > > >      1. Motivation > > > ... > > > > > > > > > Could you please try the v6.9 kernel on your setup? > > > > > > How to solve it, probably, it can be back-ported to the v6.6 kernel. > > > > All the vmalloc-related optimizations have already been merged into 6.6, > > including the set of optimization patches you suggested. Thank you very > > much for your input. > > > > It is unclear, we have backported the vmalloc optimization into our 6.6 > kernel before, so the above stack already with those patches and even > with those optimization, the find_vmap_area() is still the hotpots. > > Could you please check that all below patches are in your v6.6 kernel? commit 8be4d46e12af32342569840d958272dbb3be3f4c Author: Uladzislau Rezki (Sony) Date: Wed Jan 24 19:09:20 2024 +0100 mm: vmalloc: refactor vmalloc_dump_obj() function commit 15e02a39fb6b43f37100563c6a246252d5d1e6da Author: Uladzislau Rezki (Sony) Date: Wed Jan 24 19:09:19 2024 +0100 mm: vmalloc: improve description of vmap node layer commit 7679ba6b36dbb300b757b672d6a32a606499e14b Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:33 2024 +0100 mm: vmalloc: add a shrinker to drain vmap pools commit 8f33a2ff307248c3e55a7696f60b3658b28edb57 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:32 2024 +0100 mm: vmalloc: set nr_nodes based on CPUs in a system commit 8e1d743f2c2671aa54f6f91a2b33823f92512870 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:31 2024 +0100 mm: vmalloc: support multiple nodes in vmallocinfo commit 53becf32aec1c8049b854f0c31a11df5ed75df6f Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:30 2024 +0100 mm: vmalloc: support multiple nodes in vread_iter commit 96aa8437d169b8e030a98e2b74fd9a8ee9d3be7e Author: Uladzislau Rezki (Sony) Date: Fri Feb 2 20:06:28 2024 +0100 mm: vmalloc: add a scan area of VA only once commit 72210662c5a2b6005f6daea7fe293a0dc573e1a5 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:29 2024 +0100 mm: vmalloc: offload free_vmap_area_lock lock commit 282631cb2447318e2a55b41a665dbe8571c46d70 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:28 2024 +0100 mm: vmalloc: remove global purge_vmap_area_root rb-tree commit 55c49fee57af99f3c663e69dedc5b85e691bbe50 Author: Baoquan He Date: Tue Jan 2 19:46:27 2024 +0100 mm/vmalloc: remove vmap_area_list commit d093602919ad5908532142a048539800fa94a0d1 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:26 2024 +0100 mm: vmalloc: remove global vmap_area_root rb-tree commit 7fa8cee003166ef6db0bba70d610dbf173543811 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:25 2024 +0100 mm: vmalloc: move vmap_init_free_space() down in vmalloc.c commit 5b75b8e1b9040b34f43809b1948eaa4e83e39112 Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:24 2024 +0100 mm: vmalloc: rename adjust_va_to_fit_type() function commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc Author: Uladzislau Rezki (Sony) Date: Tue Jan 2 19:46:23 2024 +0100 mm: vmalloc: add va_alloc() helper Thank you! -- Uladzislau Rezki