From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B76F0C7EE31 for ; Wed, 24 May 2023 09:50:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 188BA6B0074; Wed, 24 May 2023 05:50:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13A2D900003; Wed, 24 May 2023 05:50:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0003D900002; Wed, 24 May 2023 05:50:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E637C6B0074 for ; Wed, 24 May 2023 05:50:19 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BADC1120956 for ; Wed, 24 May 2023 09:50:19 +0000 (UTC) X-FDA: 80824678158.07.D09574F Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf14.hostedemail.com (Postfix) with ESMTP id B077E100012 for ; Wed, 24 May 2023 09:50:17 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=erPEwsZP; spf=pass (imf14.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684921817; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bIFS61j7iDPSL85BcFHlWNxY6AjWNgDQmUG8g1ETRCk=; b=EKUbDlrKUGyN2eIl3SPdLUzSd825d2zE8mGibgGjusFddtCgNAXRLrDz7Xm8qDXxG5+UCb x46fwEXS2Xf94xkhTjikwMbToKld3grsaEXcdppjXl+G6Vdyb/cxU9PWrU8jQFaXgVedsZ meEyCvwfHLh1SUuk6R+YUIkFEqF1sf8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684921817; a=rsa-sha256; cv=none; b=F9LJPjcGe5gtA2jRdZu78KQ6c703PpT4vSWWmrGmVjmtivxRxqrGVjHcSUhBWZVcd8bpPi tcsWTPcrgHdbyLMz4ig35TKh13JTX27Pqs0UDsmjuEn+ee1cOfEdAirCrFB0s8DLDOLhTC JV9jYkLPlJ7Jq18B47K1T1B5zfApBKo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=erPEwsZP; spf=pass (imf14.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2af278ca45eso9934821fa.1 for ; Wed, 24 May 2023 02:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684921816; x=1687513816; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=bIFS61j7iDPSL85BcFHlWNxY6AjWNgDQmUG8g1ETRCk=; b=erPEwsZP3SMwCMrsS/EXircokdB5BtnQdxFnUmKYDYxLt6lJa7p/JIUni1uEzHPenh mg4H4avg9/ZtIhi4jj3T+p+zg4fk6VvGlAlKYkDHPKL2uQFJvv3aONqDF0Dm3cAwMIfL gOHNKlzfqUd12Ypidot26YVTfqu2ebvxzpSRpcBI+BoWMQ9t6nmw2M6Pm9N+ILcrf98J bygtUEqMSZ005BBb5opzPqq9qL6xMtorByvFRZOZrEfICCsVlOyqDZfjVp2n66oeIsN8 MXcIsQ4eSCKtoD+OMMedMguyxF47QN+qltyxxfjKoWsnlFVKsLvDehn7NhO3LnqYBKMa zA+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684921816; x=1687513816; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bIFS61j7iDPSL85BcFHlWNxY6AjWNgDQmUG8g1ETRCk=; b=flBEMlwxcTENVsW3d93EsCjpfP1RL3iKO5pj+DVxqrceCcat7THCwFcJPhtWsW1trD o6Ns9IsoNJTjbP/3Kp/OYgMqQgEQp+fbtQ1hnxqkWlG9s6PlA0AtuJvXAA4mVEkjxhmS ijLg4fX213DhzUh9OUlBMaE8DC9BoPMVNMCyt4j+VJDB4veTCnj75vqyY/OAWlcAore0 GGX24ePwoavP07XMFJq/vRyDYbi+cHmx27KAnIEdjkpZa2ZsZ4CS4TqPsDuHFR7ML7DY Br+PuqR3OdDAOcHfl43tUCm25c6C1LYXnNzZdHLUJnBYRKlIPbVUWTOuuRaHYw2teES3 tpuA== X-Gm-Message-State: AC+VfDy6jVZH4eQbPwvMIWIu1Sc9rXMxzLBD81HURP5m9+AYPZYgm6EO 0GeNZCqS1RWFyPrfAFFhGls= X-Google-Smtp-Source: ACHHUZ4EM5PUh0Z6UbF6nimxtZU9JpqNcBDMDxV2BFsVhit4pH91VVFq7DCBYIckGfm0xuta2qhLBg== X-Received: by 2002:ac2:4ac2:0:b0:4f3:b1ae:311e with SMTP id m2-20020ac24ac2000000b004f3b1ae311emr4371885lfp.53.1684921815706; Wed, 24 May 2023 02:50:15 -0700 (PDT) Received: from pc636 (host-90-235-19-70.mobileonline.telia.com. [90.235.19.70]) by smtp.gmail.com with ESMTPSA id s6-20020a19ad46000000b004eff530efe7sm228459lfd.93.2023.05.24.02.50.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 May 2023 02:50:15 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Wed, 24 May 2023 11:50:12 +0200 To: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Uladzislau Rezki , linux-mm@kvack.org, Andrew Morton , LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Oleksiy Avramchenko , linux-xfs@vger.kernel.org Subject: Re: [PATCH 0/9] Mitigate a vmap lock contention Message-ID: References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: B077E100012 X-Rspam-User: X-Stat-Signature: ho8ms3gzt44jnwcs4dchtahwk37gjexh X-Rspamd-Server: rspam03 X-HE-Tag: 1684921817-629552 X-HE-Meta: U2FsdGVkX18eUcD3qAJXO1cvlqjEI5o/n0OIT8oQkkG7cLPTr2JXTo2cQB3PQ1LkQPICO6UotTywpJbxiecpHIeyqyKBoEacr2ybdFHhSuszFCc2aXBDel64q71sLtYenMwu09R0+390J7U80XH5KIYrDvxeyT+fiNvCovlXhzRJmXHXprK4DrzZ/5JHA+OR2wR1hIq7sRAjALuU4RZid/7YpoE3x/Laeq6UgWytoS4DH+ipeRmbzJoer69jqlGx3XP5T8iDsWAkUUq41DoQYfhrHb4sntZ/TOF5oxUqKYL7WjQKjYakfO7w8IBt+oGIC1LNO19nOGmEJVWQi3Ins6ASXXtoNv/FA0cpjCjcoKC3iJ9SaH2n6juqYxstU1cdTJycJ1vdoEURsOW1VxgSZSbrxUH7ituBW/CBRiGHI9tob3fLdYLMWJtLFNBPRvHa/gl5vGe4G7E1Yn6jVKuf6pXnh9hgexzpPU9jSiSDTMEImklzyBQbweiFpA6MQkP4F24ssL7XnDvOaP99YvtAT9QaEmEZ/oURjvFyP5r40HaOPlax/qq4bqTGZhjPXuk//TsGo4BHX989Asbe/y99SnzT+V/up2Bj6GhORFlpJR2BKrGStzgQvJ54mYhYdKW89zUY02b6c4ZsmJlY/dTZo+vJvweujoHgIG0FXyUuZ0btRrPM4rURUJrKMCtpl6F/2COcXA9+Axz+XCHkJJrS5slByXcq6uAWtdJ76J4sUpv3MtT47Ngm7sl9VhksPYNVjXNHbb804G/oCYLf7k6mszGx2OvEQj1SQuoYK/krdVOVcCAaCx3tB4H9fYi0o7LKlG2iYa0P2ZNdA+DS6vqDcn7XmL2c0MJzLrhVV0WiBzxnPfzzx4QYqsEoYD6v2V4QGeiPshBS7ZXZbtinnmuYkrwvUy0pXgWwmz2wujzn3arZOAXeIrL3qfKigDRtl+ItD2IYJeGtOBH9PwwkJvl tTnGgcGu 3nxoRfQZXodtw1PGAez6C2XSA2+CJ/2xbaP1uVUh7GjwWzD1Rx0z9+ahqG3QvwUr/w1SifFCgLjU9+sqJWevSAIRacnmR6IitbATA6/F9Ih2HaNJEFJ1NfciCo1Beyej9jJ7SHRy45p8x4n7hbiKm+ttazyarxh5CSfBh4QShMdc3R+BUH9UTTCfOUApg5wiY6eiu1eR9OVcq1BOgXlTLA39wpkud14n2/jFmEwuaLtLJmjmEpHbHkBi/Vsiyy7iqo0tF4MaNaQcP5wTqB+KfxDMFenbB6OiBXreSBqWaUnZbrsErSqFfHTMc1lKMZ5QbEP1AKYZfppG4e/oJz/OU7frWoN6RM6+HeORlVIOGVYzyW83qkdHYMT2YpvccPDvyO9fXCv1hUEbPeFbtDzurwzjW8sgKWn44rFsvA7cpfWH6T9ciWtIRf2XUvIANZ8v3HS8t3JjWtaBXr4UfKik9d8Ku8q64PQj4aJ/6De+qmMV7ICiF11AUrqZYleICfc9RjogwKF0IUKA30rLytcgAblOqjzrT8EhZPUzKG9cTSALEzxG91CHjFKAK4KniHePHl2uwHSw88LS11mQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 24, 2023 at 03:04:28AM +0900, Hyeonggon Yoo wrote: > On Tue, May 23, 2023 at 05:12:30PM +0200, Uladzislau Rezki wrote: > > > > 2. Motivation. > > > > > > > > - The vmap code is not scalled to number of CPUs and this should be fixed; > > > > - XFS folk has complained several times that vmalloc might be contented on > > > > their workloads: > > > > > > > > > > > > commit 8dc9384b7d75012856b02ff44c37566a55fc2abf > > > > Author: Dave Chinner > > > > Date: Tue Jan 4 17:22:18 2022 -0800 > > > > > > > > xfs: reduce kvmalloc overhead for CIL shadow buffers > > > > > > > > Oh, let me count the ways that the kvmalloc API sucks dog eggs. > > > > > > > > The problem is when we are logging lots of large objects, we hit > > > > kvmalloc really damn hard with costly order allocations, and > > > > behaviour utterly sucks: > > > > > > based on the commit I guess xfs should use vmalloc/kvmalloc is because > > > it allocates large buffers, how large could it be? > > > > > They use kvmalloc(). When the page allocator is not able to serve a > > request they fallback to vmalloc. At least what i see, the sizes are: > > > > from 73728 up to 1048576, i.e. 18 pages up to 256 pages. > > > > > > 3. Test > > > > > > > > On my: AMD Ryzen Threadripper 3970X 32-Core Processor, i have below figures: > > > > > > > > 1-page 1-page-this-patch > > > > 1 0.576131 vs 0.555889 > > > > 2 2.68376 vs 1.07895 > > > > 3 4.26502 vs 1.01739 > > > > 4 6.04306 vs 1.28924 > > > > 5 8.04786 vs 1.57616 > > > > 6 9.38844 vs 1.78142 > > > > > > > > > > > > > 29 20.06 vs 3.59869 > > > > 30 20.4353 vs 3.6991 > > > > 31 20.9082 vs 3.73028 > > > > 32 21.0865 vs 3.82904 > > > > > > > > 1..32 - is a number of jobs. The results are in usec and is a vmallco()/vfree() > > > > pair throughput. > > > > > > I would be more interested in real numbers than synthetic benchmarks, > > > Maybe XFS folks could help performing profiling similar to commit 8dc9384b7d750 > > > with and without this patchset? > > > > > I added Dave Chinner to this thread. > > Oh, I missed that, and it would be better to [+Cc linux-xfs] > > > But. The contention exists. > > I think "theoretically can be contended" doesn't necessarily mean it's actually > contended in the real world. > > Also I find it difficult to imagine vmalloc being highly contended because it was > historically considered slow and thus discouraged when performance is important. > > IOW vmalloc would not be contended when allocation size is small because we have > kmalloc/buddy API, and therefore I wonder which workloads are allocating very large > buffers and at the same time allocating very frequently, thus performance-sensitive. > > I am not against this series, but wondering which workloads would benefit ;) > > > Apart of that per-cpu-KVA allocator can go away if we make it generic instead. > > Not sure I understand your point, can you elaborate please? > > And I would like to ask some side questions: > > 1. Is vm_[un]map_ram() API still worth with this patchset? > It is up to community to decide. As i see XFS needs it also. Maybe in the future it can be removed(who knows). If the vmalloc code itself can deliver such performance as vm_map* APIs. > > 2. How does this patchset deals with 32-bit machines where > vmalloc address space is limited? > It can deal without any problems. Though i am not sure it is needed for 32-bit systems. The reason is, the vmalloc code was a bit slow when it comes to lookup time, it used to be O(n). After that it was improved to O(logn). vm_map_ram() and friends interface was added because of vmalloc drawbacks. I am not sure that there are 32-bit systems with 10/20/30... CPUs on board. In that case it is worth care about contention. -- Uladzislau Rezki