From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92815C433F5 for ; Fri, 27 May 2022 21:01:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E13E68D0003; Fri, 27 May 2022 17:01:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC5BB8D0002; Fri, 27 May 2022 17:01:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAFFD8D0003; Fri, 27 May 2022 17:01:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BDC248D0002 for ; Fri, 27 May 2022 17:01:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 9C4E660BB3 for ; Fri, 27 May 2022 21:01:34 +0000 (UTC) X-FDA: 79512744108.06.62B1FB2 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf26.hostedemail.com (Postfix) with ESMTP id 6B849140036 for ; Fri, 27 May 2022 21:01:29 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 74E37B824FD; Fri, 27 May 2022 21:01:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5164C385A9; Fri, 27 May 2022 21:01:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1653685290; bh=b1REJzW0O2tqVTt711WSZgo7eiYCCpX58Hv7sbbE0tw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=oExo7SPI3jcDj+dU4oC3F5/k6QyytX+pQIzjuetvwOUlqQ3ZABt6GEieb82uXewIY RSwGyx1DW+jLmQp2kN2npOCaCyMzuOhTvAguW9DOV5/X5s1cn+DE5jX1wlhCoZ6k9F LbuwtHdUSj0IIjGzk0u+2vw/ycIZjlj5nfo7seiUS7+fcklWBByWq5QmI5mCxSZOfa qek1J+nvnNCNFFdTNNavebjSw1fhqUeWZ9cGvLe4U3vSJXNiUnwL6nC2Mo8eItmXKy BvzsG4gzDHIF4QbQKanLB+lYuuJGPj2Pq3gmZvNRkOQ2o+r8+kgMGWh+cY9arr6+// 5DKv08hjwFyMw== Date: Fri, 27 May 2022 15:01:27 -0600 From: Keith Busch To: Tony Battersby Cc: kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org Subject: Re: [PATCH 0/2] dmapool performance enhancements Message-ID: References: <20220428202714.17630-1-kbusch@kernel.org> <156da4ae-20de-a40f-5173-3b02c779b43c@cybernetics.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <156da4ae-20de-a40f-5173-3b02c779b43c@cybernetics.com> X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 6B849140036 X-Stat-Signature: ujpp7rup5o75xath737c73sbw638xprk Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oExo7SPI; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of kbusch@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=kbusch@kernel.org X-Rspam-User: X-HE-Tag: 1653685289-939035 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, May 27, 2022 at 03:35:47PM -0400, Tony Battersby wrote: > I posted a similar patch series back in 2018: > > https://lore.kernel.org/linux-mm/73ec1f52-d758-05df-fb6a-41d269e910d0@cybernetics.com/ > https://lore.kernel.org/linux-mm/15ff502d-d840-1003-6c45-bc17f0d81262@cybernetics.com/ > https://lore.kernel.org/linux-mm/1288e597-a67a-25b3-b7c6-db883ca67a25@cybernetics.com/ > > > I initially used a red-black tree keyed by the DMA address, but then for > v2 of the patchset I put the dma pool info directly into struct page and > used virt_to_page() to get at it.  But it turned out that was a bad idea > because not all architectures have struct page backing > dma_alloc_coherent(): > > https://lore.kernel.org/linux-kernel/20181206013054.GI6707@atomide.com/ > > I intended to go back and resubmit the red-black tree version, but I was > too busy at the time and forgot about it.  A few days ago I finally > decided to update the patches and submit them upstream.  I found your > recent dmapool xarray patches by searching the mailing list archive to > see if anyone else was working on something similar. > > Using the following as a benchmark: > > modprobe mpt3sas > drivers/scsi/mpt3sas/mpt3sas_base.c > _base_allocate_chain_dma_pool > loop dma_pool_alloc(ioc->chain_dma_pool) > > rmmod mpt3sas > drivers/scsi/mpt3sas/mpt3sas_base.c > _base_release_memory_pools() > loop dma_pool_free(ioc->chain_dma_pool) > > Here are the benchmark results showing the speedup from the patchsets: > > modprobe rmmod > orig 1x 1x > xarray 5.2x 186x > rbtree 9.3x 269x > > It looks like my red-black tree version is faster than the v1 of the > xarray patch on this benchmark at least, although the mpt3sas usage of > dmapool is hardly typical.  I will try to get some testing done on my > patchset and post it next week. Thanks for the info. Just comparing with xarray, I actually found that the list was still faster until you get >100 pages in the pool, after which xarray becomes the clear winner. But it turns out I don't often see that many pages allocated for a lot of real use cases, so I'm trying to take this in a different direction by replacing the lookup structures with an intrusive stack. That is safe to do since pages are never freed for the lifetime of the pool, and it's by far faster than anything else. The downside is that I'd need to increase the size of the smallest allowable pool block, but I think that's okay. Anyway I was planning to post this new idea soon. My reasons for wanting a faster dma pool are still in the works, though, so I'm just trying to sort out those patches before returning to this one.