From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09685C77B75 for ; Tue, 23 May 2023 11:59:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F993900004; Tue, 23 May 2023 07:59:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 682BE900002; Tue, 23 May 2023 07:59:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FBBF900004; Tue, 23 May 2023 07:59:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 37231900002 for ; Tue, 23 May 2023 07:59:03 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EEDD91C6E5F for ; Tue, 23 May 2023 11:59:02 +0000 (UTC) X-FDA: 80821373724.27.0582860 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf04.hostedemail.com (Postfix) with ESMTP id 1C68840014 for ; Tue, 23 May 2023 11:59:00 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=oBuMY1mw; spf=pass (imf04.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684843141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n4th3iXOfa5qIG2MobVS/IDr8buOOQSispksL/hzEfA=; b=6gtURJoa5BWW9HgsnK4VJpmKal5vKRnCHByW4WY3IoaE/Gafey6vG51IpTLY6CbNGKGCUK YIkXhZEJZWuMWCVbg+E502a6BQj0TspdkjEb2bLNla6WW2c8uSMJ4QnLd3bzFI5a0kWZXD 7lYqFfAexqXLGHwWvdxLveouLfCz2aU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684843141; a=rsa-sha256; cv=none; b=Bdcl1aGuv+/R0mdbUbPEkTorIlFzcX3ubbrFwUopVb7wzWin11eX/j2i5VnP+Iabpfl+pj fpg0zmjpJ6uUsrjMp+t2daua0bo+v1B1zVm251GcDZrek+ichFJucvdJOsWWRXZsrLSSTL jLQgs9i0hpYzb0M1mGFifKTR8hJ+p3I= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=oBuMY1mw; spf=pass (imf04.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-52cb8e5e9f5so3270422a12.0 for ; Tue, 23 May 2023 04:59:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684843140; x=1687435140; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=n4th3iXOfa5qIG2MobVS/IDr8buOOQSispksL/hzEfA=; b=oBuMY1mwpDRyQ/c9mrNJt6P4ahBppKTLrL1gS6q7FTOkPweEwSDwIMF6kpRz8gpVAE /zr6l2BFroCbmEhqu23/LI6Cu9W2/+8TztbaLnQqteEU0caVwds1TkJQt9aLss4OGhDb OjpPgbVOomylzCfu+gJPYBf1JXjC0shXmIgh0vO3powFQMvn/+rZ5x7935z+VFlNWWzX A6xlMpLYHDIbUVPnd0F8qEevcgibG4FsZdlBzdcAh76BUcvCbWuftm3RuX2AsX6rfGK2 aPN6X5kr339zlfvg/RvbpYrseq0eAadjAFsZR6t7XXit5w1C5wrCmgZL3qiRH39OGaDV 0NVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684843140; x=1687435140; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=n4th3iXOfa5qIG2MobVS/IDr8buOOQSispksL/hzEfA=; b=gUG7GZP3GzmJUdaILRKbH3AQoBTCUbhctpaDS5BlEXyz0yy81is+ZHwpyp9wQjiNX3 6djKtFnGoATxCwd9zi3w3Bc0HtWTDQ5dRBlxYyv9zvnGvNJmX9X73GB1AqGDsnBwdszY HzBZC2CpD90Wn/VZ3po+5n2WuIpD3EDoR1y+5V7/uhHnsgxHZWoHxKJSxqNa4GY+pVI6 g9iLgCC2TOjAROJ+5khtc1MLq5HwJPuWEpR/7xDJUOnkdNFowq90PdrFy3eqxjYX1961 vVMtKTeuBTQr8TYS1/YoMZWuCUWPUaBAUqkONWRfdhv0WD/Q8M+gaImpG3gJ5Xw+cByz LeUw== X-Gm-Message-State: AC+VfDxomdDz36afGm+tBN/7OqlTxBmelfHSGf3fiY4HHQiYGVFHi2b0 xPMv202pGQ12MM3fbJ4Ih5I= X-Google-Smtp-Source: ACHHUZ6pTcFC6bhAYSWxCL+9RgiqZwcBn1Swkgkp+WkLW+9AgtOk1VjAOPYjwpciAhT574UEhmE/og== X-Received: by 2002:a17:902:c403:b0:1a9:4cd5:e7e0 with SMTP id k3-20020a170902c40300b001a94cd5e7e0mr21664656plk.17.1684843139548; Tue, 23 May 2023 04:58:59 -0700 (PDT) Received: from debian-BULLSEYE-live-builder-AMD64 ([121.183.137.144]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b001a943c41c37sm6659206plc.7.2023.05.23.04.58.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 04:58:58 -0700 (PDT) Date: Tue, 23 May 2023 20:59:05 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: "Uladzislau Rezki (Sony)" Cc: linux-mm@kvack.org, Andrew Morton , LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Oleksiy Avramchenko Subject: Re: [PATCH 0/9] Mitigate a vmap lock contention Message-ID: References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> X-Rspamd-Queue-Id: 1C68840014 X-Rspam-User: X-Stat-Signature: 4bf3wp8d8uhxuwe15uypz8x5p8oy86r1 X-Rspamd-Server: rspam03 X-HE-Tag: 1684843140-528130 X-HE-Meta: U2FsdGVkX18j5sInDYejOKoMjes1dBhS/xS+Ol3qOn8jrl1YHkT/hrOIJtfQsOlEEKZjBY+enajdTSMGI9ql7aPfMYRlzERUjnXSx5LzD27N26vvwsD5D/GziMIszsBwo/bUdT7c4EVIDqkAEWDPW8AZGQphc81aaZpuyW1tl8WxNvnj1r7QDzS/cMUNwS1TOzg5+hldtCtsKvkrG6qIGrz2c/xycL4cQToE7GABRY9lYKhnVGlqLK5pkw+BUtqhDvvFqQvcBWJaVn9ZKWfzfYSP221ijo7xty70HOlrpF/y0PqytWc966hsrELtxFt83HA6GaOtL1xDlkQYlORSjSf55NsjZ84fccz1nuiqh526Shi5mfZixwtAzFbnUjqbNmrokKUOEPhuH7mQ3apYOcHQgR9IN7ubPevdVkigqWFeG/EA5vR7Ue0Ic2Y9wexlwbKIimNL8yN2ld89GELL2uh72ojmu74duh97HiAcn04DydkNmxZHR4JNj4Pdv0c3KCDg7IH2Ki+KShzF8ReuXDZv8fPQGDq9+WRJTj94cNERB2Lldbupolqy7Czevtz2/7r8akFJCpQL39IKoc9LeC6p3XIH1X/I6dJwPbIGjyTIOTG/LtelzaXYbG29xUMVnaj8nJKHBuB21jFrW/zx48u/5kkDXbTyF9bJvp2Hv5QSKci8fX+M5ht/jZc2Z7rBZA0lMTkYx0HvClEdYOqGRmjqJqzZa+m7nrN1sm2xPAjtJ8FpVaLE7i/aRjMHz6lLwfR1HZJRVXEH4N5Bt7dKFBTT7r+GSfrCgRR58raO0Xoc3dgdSdw2z9rdB4HtE6BmvqyhpxQBQZmZWBeVz/qx84DUSPG4b57Pu4RY3v/uCL2Yw1Yq8yh0rgx+XbZmO5tmPPKj411gaROmRvMM/oLS7lG1rkO+pRAPPk9360dqY5ysgKE6jvSsLoYussSThWBa7kZ49ZZDu83njDDaRyx 48Ui18St ZSnc9tqbtaWrCHO7u/MfwLxljKwGP/7js7vFzaF9lywgQZ6tY0t09BzpGYYkvJcsjX3WzjsLvPKoyweJW3z4mCK0O1zeNcno6qmfAMbEoHjPa4wB4lbE6fOhenQgzHdWmS88+8Z1nRDQHafVJrb/h4czFaipR2MTBEFUly0NuhopmT95yBjHjU8a/qoR/xF+WlgvtLhtVworgW/T1/vwzvCu/TcFHnQGiQiscb8kbl4rEsN7yd1n/UMsPiFbXUmMvpFL8E/1h9Z3jmi5s3xkauvptfJEvZyHkmwK+S5XuYp0iwid7zuBaH8OVr0SqWcntHHKCDFWyqzAkrGbaid0yHa9Lc9txFxSaTA507XZ/t2YOO7vZLkr8fJ6l0BfMKXgDfVMRvUYA52F4XSxFfginpUhWnbSPzKL8fuqDcyNOpRlnOLz4lJfEOA06WA7RSLaNAM4loaSOqaU6/vGT0B/knTpyiOok5ee3Xs0FvSLpqqA7ut/1q+8Z3F1vhoM4MYXdwebeLqqvNR5xZhRRyVvtJomQ3N5MS/4FBGTq5qXgkPajm/PB7XoXdfPXjt3rPDoHgphbSp7Oi3liTcf7VT8lRqZ9E+cpqyGq8xxYipJmAC/u9fymqVims65NZT9PdaphvgChNwz+NR/XkZMdflfP3ViQTklmg79RkNYD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, May 22, 2023 at 01:08:40PM +0200, Uladzislau Rezki (Sony) wrote: > Hello, folk. > > 1. This is a followup of the vmap topic that was highlighted at the LSFMMBPF-2023 > conference. This small serial attempts to mitigate the contention across the > vmap/vmalloc code. The problem is described here: > Hello Uladzislau, thank you for the work! > wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf I ran the exactly same command but couldn't download the file, did I miss something? $ wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf [...] ==> PASV ... done. ==> RETR Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf ... No such file `Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf'. > The material is tagged as a v2 version. It contains extra slides about testing > the throughput, steps and comparison with a current approach. > > 2. Motivation. > > - The vmap code is not scalled to number of CPUs and this should be fixed; > - XFS folk has complained several times that vmalloc might be contented on > their workloads: > > > commit 8dc9384b7d75012856b02ff44c37566a55fc2abf > Author: Dave Chinner > Date: Tue Jan 4 17:22:18 2022 -0800 > > xfs: reduce kvmalloc overhead for CIL shadow buffers > > Oh, let me count the ways that the kvmalloc API sucks dog eggs. > > The problem is when we are logging lots of large objects, we hit > kvmalloc really damn hard with costly order allocations, and > behaviour utterly sucks: based on the commit I guess xfs should use vmalloc/kvmalloc is because it allocates large buffers, how large could it be? > 3. Test > > On my: AMD Ryzen Threadripper 3970X 32-Core Processor, i have below figures: > > 1-page 1-page-this-patch > 1 0.576131 vs 0.555889 > 2 2.68376 vs 1.07895 > 3 4.26502 vs 1.01739 > 4 6.04306 vs 1.28924 > 5 8.04786 vs 1.57616 > 6 9.38844 vs 1.78142 > 29 20.06 vs 3.59869 > 30 20.4353 vs 3.6991 > 31 20.9082 vs 3.73028 > 32 21.0865 vs 3.82904 > > 1..32 - is a number of jobs. The results are in usec and is a vmallco()/vfree() > pair throughput. I would be more interested in real numbers than synthetic benchmarks, Maybe XFS folks could help performing profiling similar to commit 8dc9384b7d750 with and without this patchset? By the way looking at the commit, teaching __p?d_alloc() about gfp context (that I'm _slowly_ working on...) could be nice for allowing non-GFP_KERNEL kvmalloc allocations, as Matthew mentioned. [1] Thanks! [1] https://lore.kernel.org/linux-mm/Y%2FOHC33YLedMXTlD@casper.infradead.org -- Hyeonggon Yoo Doing kernel stuff as a hobby Undergraduate | Chungnam National University Dept. Computer Science & Engineering