From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C6F7C77B75 for ; Tue, 23 May 2023 18:05:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BECA3900003; Tue, 23 May 2023 14:05:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9C8F900002; Tue, 23 May 2023 14:05:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3DAF900003; Tue, 23 May 2023 14:05:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 93502900002 for ; Tue, 23 May 2023 14:05:55 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 657824079B for ; Tue, 23 May 2023 18:05:55 +0000 (UTC) X-FDA: 80822298270.16.306DE48 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf25.hostedemail.com (Postfix) with ESMTP id C4172A0187 for ; Tue, 23 May 2023 18:04:22 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=OjRBHmKy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684865062; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7G+6JAe83A/18uddV8Wr/e+zu8FEvMqf/FTgHFCweHM=; b=lPw3jBv9KB4AybjBJsVwuMkMFBAlRR1tWsQQpMQEMi/aTuyxqJeHSSAb1fMpHleAnMFBtk KrYjk0eOZ6yP7zD5lGwCkj5PlAZjUSRXOM/j61blggclH1amhpwRKdbd+ARQgjKtP3mQ8Q /6F3dYVz3g5Fz2KzhxF59ne84mA3vAY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=OjRBHmKy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684865062; a=rsa-sha256; cv=none; b=P7qpozdUgvPqbD+8OZleGv4QMRJpotgBwqyREVwZkn64D0dulOnfrJ6mO9OpE8KDoVYQTp S49EMlvKLQpkGhBN9BXMLpIdM3MN8FjnO3c/ugSevFgAqm1InuFciaM2O8bnrEkfC29NhH F8TSVgKv5TimbGWeShPIGNmqDlS+Jy0= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1ae5dc9eac4so49719535ad.1 for ; Tue, 23 May 2023 11:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684865061; x=1687457061; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=7G+6JAe83A/18uddV8Wr/e+zu8FEvMqf/FTgHFCweHM=; b=OjRBHmKyZVkTv8ipCSSjMC979Aa2g8Vef/qdvxWXNK7NfpzJNVJzYzTbPwBqOJYnIM vwYPxKbp7kt4/uzWVCrugRgi2qAog0S5e0N/D4KORZxxCcleybetuj0ZoVe1zrvhJuHs NPimC3YHLcXmNFW845z/8XZ99tVVoyiyvn0aNHlu7cELROsQq5Ejt0Mkg5qTC4q8O1P/ RLk34aD+U5QHietxZJhXe+22UN+3njLRq4RA6mjnCw5zYbssPk5+vRPk9RiS2LDaRzdw TmjTW23dM/3zhjSzXiKomSO4Z95x93uk/fiaoRI2DOFOpAN7I0mtdv09iiB9T1/tOxaB NoCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684865061; x=1687457061; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7G+6JAe83A/18uddV8Wr/e+zu8FEvMqf/FTgHFCweHM=; b=f3yZn1gTdg1XWMEdQjMCKZHgxUeSf3iCyeumPeqxyKv0E9sfeF9ano8xSCJEmuqK3r Mn92OvqXU6clguprzCMpHyjJao32oy9gsuw700mZn88hOJEsnJ4RkjrJ0OXzERyQJReQ MczttmTxGSwkIFIWHOvlqv8+x0Om4tKEDZCrxW845gXJjBhjk6lXpLE8EpGdHw/Yo6w7 072JUjl+5EqfZT8Z2i3Li0GcRjf/YcaVF9xYjX04tZ5DK6HatCHW9cgYwubedvrxhOgK uQk7gvon0iJLmluSSu9sHMA7vOSieJ6a4pvvkYkmbGBkW4z24XQK1tPI/xc57reI0sVX AxqA== X-Gm-Message-State: AC+VfDzWG8THG+2KtbIT38vSHQ+vkWAs5tA6OmwGyjww/a6QVLeiispF d222r/hgWricHmCSEQNfb2A= X-Google-Smtp-Source: ACHHUZ7Qj/je6b3i19kqNy7D3SzNF2pLpsy4V4UegPxSa9h9jg3U20AWBdyQoww8RgK0muN41BIKXA== X-Received: by 2002:a17:902:ce91:b0:1af:c599:6a88 with SMTP id f17-20020a170902ce9100b001afc5996a88mr3866632plg.49.1684865060999; Tue, 23 May 2023 11:04:20 -0700 (PDT) Received: from debian-BULLSEYE-live-builder-AMD64 ([211.108.101.96]) by smtp.gmail.com with ESMTPSA id bj6-20020a170902850600b001a183ade911sm7080795plb.56.2023.05.23.11.04.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 11:04:20 -0700 (PDT) Date: Wed, 24 May 2023 03:04:28 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Uladzislau Rezki Cc: linux-mm@kvack.org, Andrew Morton , LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Oleksiy Avramchenko , linux-xfs@vger.kernel.org Subject: Re: [PATCH 0/9] Mitigate a vmap lock contention Message-ID: References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: C4172A0187 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: iowdsgnq4k51s1z4tp55zkx76cok3k8u X-HE-Tag: 1684865062-522035 X-HE-Meta: U2FsdGVkX1+gPAkYXgBZOSlYd0UteGhnu/8JI6eurfRzsKFKYHdUX8cFG7j6p6Qho6UBArNCZ+ukjCnyREmvrApXZ5wQqlDkVq8xFbw6tBiqCrpg6Bw2D3ebIjXdxEWqSl9YLlrCK0COYPlGvpK2khLFNg1ciSACYz5aqzmg/y7lnuXWMK/O8CfWjA9w0whOQWCutHfbg3nnDTBSiXPv0uR+W/AwIN8/v6/QZmVxUSRlqbCx9uCAlKE4YqCPi70Hd9qDV0LIqILr+BeCXEBJRGHFi1UGUwt0h3laJ4uMT288bLkIgokDahRMbHzm64+OfbiAIe89g20QMpy8T+98xjL1nRa7IivrSlJlOVzsx8Bj3k6rC+KoCEoP9+7Yto1NvnYiWpN02n6QMnnTL1rWUCf95OgPUN9NlEJbLd8vMGHPT2k1aUfrCAZ+P4Gt7AAgsK+maG/BVEVg+9U5TilA2kFRpVvGWJT5uMh65auzc+/Y90ppbvWUWqoc5lzukoGbu0+cTiFrb9L78frEOJMfy0/tbZpOT8ZtvEmIdB8Jt0iYtAE5ChNb70bj3rA90di/Yn52XmvuXHUHc9sJR4FhOkAei/05LE3zzdOKKiwsNcfv8hjctCSWzaSosbImCP9w0rdjJE+2BU7b5OKZAdWOtUnrcUgZ1dzVBcxKvpDLsz7Ozw5WpbmTkNW9Mc686xfaourwZ5bzo/y5dkKiPetfMV5IlgdO55StpUC1NeE6PdIzoO1Rg4Ud16tKm6lzlWKLwDBu6Pk+oA0u0aHL7ZNPVl2f7B0dPEcCvm4XxCBfqRlG3Lqrm9oLwfQghD5PzwFbLnseB4LQ7jGKpoxG2a3oYmxdLY6bC4cHgK/paC7R8Wj+rme7ublPheSq/TN1lAxL1lJTG0aMvm4fEsEIIoQS647bFmTLplojMZERlj+HePEO9p69P0DiHDHRBR+4/aAB91Sj5gAiu+L0k5oCCgh Iq375DCF 0HrapHL0ue2IfC04Vhp0l4+h1E9Xtd10uOuFM6xO5LLnjab+EwBzGK135EvpyntStDxkffTEt+wuM6hZb7/tWy9suGgzAB1xeCHRnFQOQ9OFNDkaxdrbyG8j8T1HasaEktBo0PnMe6C+lrya/QVkGidf9VWwDSsmzlm90tKS4B5S7jW8+xDe69nH1RoDRq/c4Dt+ypcf7XHReySkRgGBDFCVuivgNZm/uuS5UcCzswf3hP1OaZs9my8cBUSRpZuBtGwSnIbXIQ7J6yWBW5T/9ZFEm7xGY27lrT4l/QWaR8bd34nc7OykmUW4xuVDYIHBYzquk0eKRf+jzn6VcsWBiNs/Iz15tL2sY2vw3biF1IscPPHPhx/oIiIbXOMOwOYKrZbUh4i9vIRJ09vjyh0iEc5go4TP44Lsou65wfKkOpxeK91AMHaRqyp0pWe4LnGEdXxc+dwRmDso9V+ey7jJdMdU2hIoQ/QbiPR8gDZD3NgrpvYSOKGmz8h8bWkbGGGW8TdMDbBZA2nSgcCcebqMn+GFyxpnqB1bUNXdyb8hslLqAEdOhfxhbdQaWK6LzJZWv3vUebHUZsy9yAAuHcRock6NWMYbblrLtDzK+iVq6DDD19dfAVJ8n84B/dd9DgIMZ5hhmmLxPAnw+0/0Og6KOABI1ZJctu3QZZdBcbt7rVUk+xl7iB8v3SS0lNTOCi1oYH4LpVwqEL4yl1HU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 23, 2023 at 05:12:30PM +0200, Uladzislau Rezki wrote: > > > 2. Motivation. > > > > > > - The vmap code is not scalled to number of CPUs and this should be fixed; > > > - XFS folk has complained several times that vmalloc might be contented on > > > their workloads: > > > > > > > > > commit 8dc9384b7d75012856b02ff44c37566a55fc2abf > > > Author: Dave Chinner > > > Date: Tue Jan 4 17:22:18 2022 -0800 > > > > > > xfs: reduce kvmalloc overhead for CIL shadow buffers > > > > > > Oh, let me count the ways that the kvmalloc API sucks dog eggs. > > > > > > The problem is when we are logging lots of large objects, we hit > > > kvmalloc really damn hard with costly order allocations, and > > > behaviour utterly sucks: > > > > based on the commit I guess xfs should use vmalloc/kvmalloc is because > > it allocates large buffers, how large could it be? > > > They use kvmalloc(). When the page allocator is not able to serve a > request they fallback to vmalloc. At least what i see, the sizes are: > > from 73728 up to 1048576, i.e. 18 pages up to 256 pages. > > > > 3. Test > > > > > > On my: AMD Ryzen Threadripper 3970X 32-Core Processor, i have below figures: > > > > > > 1-page 1-page-this-patch > > > 1 0.576131 vs 0.555889 > > > 2 2.68376 vs 1.07895 > > > 3 4.26502 vs 1.01739 > > > 4 6.04306 vs 1.28924 > > > 5 8.04786 vs 1.57616 > > > 6 9.38844 vs 1.78142 > > > > > > > > > 29 20.06 vs 3.59869 > > > 30 20.4353 vs 3.6991 > > > 31 20.9082 vs 3.73028 > > > 32 21.0865 vs 3.82904 > > > > > > 1..32 - is a number of jobs. The results are in usec and is a vmallco()/vfree() > > > pair throughput. > > > > I would be more interested in real numbers than synthetic benchmarks, > > Maybe XFS folks could help performing profiling similar to commit 8dc9384b7d750 > > with and without this patchset? > > > I added Dave Chinner to this thread. Oh, I missed that, and it would be better to [+Cc linux-xfs] > But. The contention exists. I think "theoretically can be contended" doesn't necessarily mean it's actually contended in the real world. Also I find it difficult to imagine vmalloc being highly contended because it was historically considered slow and thus discouraged when performance is important. IOW vmalloc would not be contended when allocation size is small because we have kmalloc/buddy API, and therefore I wonder which workloads are allocating very large buffers and at the same time allocating very frequently, thus performance-sensitive. I am not against this series, but wondering which workloads would benefit ;) > Apart of that per-cpu-KVA allocator can go away if we make it generic instead. Not sure I understand your point, can you elaborate please? And I would like to ask some side questions: 1. Is vm_[un]map_ram() API still worth with this patchset? 2. How does this patchset deals with 32-bit machines where vmalloc address space is limited? Thanks! > > By the way looking at the commit, teaching __p?d_alloc() about gfp > > context (that I'm _slowly_ working on...) could be nice for allowing > > non-GFP_KERNEL kvmalloc allocations, as Matthew mentioned. [1] > > > > Thanks! > > > > [1] https://lore.kernel.org/linux-mm/Y%2FOHC33YLedMXTlD@casper.infradead.org > > -- Hyeonggon Yoo Doing kernel stuff as a hobby Undergraduate | Chungnam National University Dept. Computer Science & Engineering