From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF657C4332F for ; Thu, 13 Oct 2022 16:57:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A0286B0071; Thu, 13 Oct 2022 12:57:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 650866B0073; Thu, 13 Oct 2022 12:57:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F09A8E0001; Thu, 13 Oct 2022 12:57:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3B92A6B0071 for ; Thu, 13 Oct 2022 12:57:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 109DA1C6089 for ; Thu, 13 Oct 2022 16:57:36 +0000 (UTC) X-FDA: 80016532512.21.7619476 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf02.hostedemail.com (Postfix) with ESMTP id 7AEF380026 for ; Thu, 13 Oct 2022 16:57:34 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8731661821; Thu, 13 Oct 2022 16:57:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C30FEC433D6; Thu, 13 Oct 2022 16:57:29 +0000 (UTC) Date: Thu, 13 Oct 2022 17:57:26 +0100 From: Catalin Marinas To: Isaac Manjarres Cc: Herbert Xu , Ard Biesheuvel , Will Deacon , Marc Zyngier , Arnd Bergmann , Greg Kroah-Hartman , Andrew Morton , Linus Torvalds , Linux Memory Management List , Linux ARM , Linux Kernel Mailing List , "David S. Miller" , Saravana Kannan , kernel-team@android.com Subject: Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665680254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UhICjJGuy5ivRglRCtwAvB9Ha21GebQ0ZP8eNSNGBBA=; b=QFoMsLdgCADiuiuKuC3sy5bsR94x8L3Ii8c7m2vNd0OK1Bgv0lIByDUHjVli3fecx8wfG1 jUnD5oPBQGn/Yl2EWxjM+sVg0P5GLifEg3nHwOZu/1I4GLzCfGc3F0C6FMD83ZSQ+3TGiB /81D67/4nRHe+IDsYXm+sMowF66iLec= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665680254; a=rsa-sha256; cv=none; b=pAL/T9ycO/AeU1xJ3dKbP8YST1W8LB2kC3fiXLEhPapCg+AvVDej8XrOnhPOzlcxJD0duK PJJzu8PNm+7Iu9/LnLAN2ryGgGutJ8UW7vTLBHlccfiXsB43nfIiYjBBfYo+lae//TMUsG JhA86JlKFp3/uCpzHZQTqBtxfmsHTd4= X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) X-Stat-Signature: bebfnnoyuttxbej4pfntjg3qa5mrab1d X-Rspamd-Queue-Id: 7AEF380026 X-HE-Tag: 1665680254-675206 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 12, 2022 at 10:45:45AM -0700, Isaac Manjarres wrote: > On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote: > > I started refreshing the series but I got stuck on having to do bouncing > > for small buffers even if when they go through the iommu (and I don't > > have the set up to test it yet). > > For devices that go through the IOMMU, are you planning on adding > similar logic as you did in the direct-DMA path to bounce the buffer > prior to calling into whatever DMA ops are registered for the device? Yes. > Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because > none of the peripherals that they engage in DMA with need bounce buffering, > and also to reclaim the default 64 MB of memory that SWIOTLB uses. With > this approach, SWIOTLB usage will become mandatory if those devices need > to perform non-coherent DMA transactions that may not necessarily be DMA > aligned (e.g. small buffers), correct? Correct. I've been thinking about this and a way around is to combine the original series (dynamic kmalloc_minalign) with the new one so that the arch code can lower the minimum alignment either to 8 if swiotlb is available (usually in server space with more RAM) or the cache line size if there is no bounce buffer. > If so, would there be concerns that the memory savings we get back from > reducing the memory footprint of kmalloc might be defeated by how much > memory is needed for bounce buffering? It's not necessarily about the saved memory but also locality of the small buffer allocations, less cache and TLB pressure. > I understand that we can use the > "swiotlb=num_slabs" command line parameter to minimize the amount of > memory allocated for bounce buffering. If this is the only way to > minimize this impact, how much memory would you recommend to allocate > for bounce buffering on a system that will only use bounce buffers for > non-DMA-aligned buffers? It's hard to tell, it would need to be guessed by trial and error on specific hardware if you want to lower it. Another issue is that IIRC the swiotlb is allocated in 2K slots, so you may need a lot more bounce buffers than the actual memory allocated. I wonder whether swiotlb is actually the best option for bouncing unaligned buffers. We could use something like mempool_alloc() instead if we stick to small buffers rather than any (even large) buffer that's not aligned to a cache line. Or just go for kmem_cache_alloc() directly. A downside is that we may need GFP_ATOMIC for such allocations, so higher risk of failure. -- Catalin