From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97152C43334 for ; Fri, 1 Jul 2022 12:01:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8E6A6B0071; Fri, 1 Jul 2022 08:01:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3EE86B0073; Fri, 1 Jul 2022 08:01:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C06366B0074; Fri, 1 Jul 2022 08:01:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AE5ED6B0071 for ; Fri, 1 Jul 2022 08:01:40 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 75D7B80DC9 for ; Fri, 1 Jul 2022 12:01:40 +0000 (UTC) X-FDA: 79638391560.22.F0B1F54 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id CDB871200A6 for ; Fri, 1 Jul 2022 12:01:39 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C4AB113E; Fri, 1 Jul 2022 05:01:39 -0700 (PDT) Received: from [10.57.85.162] (unknown [10.57.85.162]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E838D3F792; Fri, 1 Jul 2022 05:01:36 -0700 (PDT) Message-ID: <1eeeec76-5271-f915-e3fd-f15095efb981@arm.com> Date: Fri, 1 Jul 2022 13:01:31 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [PATCH] iommu/iova: change IOVA_MAG_SIZE to 127 to save memory Content-Language: en-GB To: John Garry , Feng Tang Cc: Joerg Roedel , Will Deacon , iommu@lists.linux-foundation.org, iommu@lists.linux.dev, Andrew Morton , Christoph Lameter , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Paul Menzel References: <20220630073304.26945-1-feng.tang@intel.com> <13db50bb-57c7-0d54-3857-84b8a4591d9e@arm.com> <7c29d01d-d90c-58d3-a6e0-0b6c404173ac@huawei.com> <117b31b5-8d06-0af4-7f1c-231d86becf1d@arm.com> <2920df89-9975-5785-f79b-257d3052dfaf@huawei.com> <20220701035622.GB14806@shbuild999.sh.intel.com> <51af869a-83d4-631a-2d91-edb8b066bf4d@huawei.com> From: Robin Murphy In-Reply-To: <51af869a-83d4-631a-2d91-edb8b066bf4d@huawei.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656676899; a=rsa-sha256; cv=none; b=WabwBq/pwH+GuDL3+0OsPWVV69j/rFLZTpb3QNWdbwD8EPyfCzOx3y5Wf2W/7H86+cYtHE rWaepFYnAxxKT9OjgUacptVquKb7r6JtVWaizZsxtSMg8T0/lPLHd+FbsbF54JL0TgU+C7 UnR/0GmMo6+3xgkrYB9OI9mhzgd8C9o= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656676899; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2g9fSCa6kW+2G0EWF+D4LK/1cp5VRievk8PPx3pw1Aw=; b=SbubmjQE2Sa7EGdFEMHh9Ro32NFFzBNrhAYax3svHsJoaN/8yZdcU6mFGRjb5DyQPkwP6N 65WkJYel4qbFWK9K1AqgKrQu2ZqQ7uXFR5/DhR8T2bMcR8DSE8+A12qyzzovIvkGszkEFe XjowHWUcYluBH4sCelnOMq7ijwh3kk8= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CDB871200A6 Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com X-Rspam-User: X-Stat-Signature: npiy3fkiuz8ppdd64516c6x7npm9z139 X-HE-Tag: 1656676899-968900 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-07-01 12:33, John Garry wrote: > On 01/07/2022 04:56, Feng Tang wrote: >>>> inclination. >>>> >>> ok, what you are saying sounds reasonable. I just remember that when we >>> analyzed the longterm aging issue that we concluded that the FQ size >>> and its >>> relation to the magazine size was a factor and this change makes me a >>> little >>> worried about new issues. Better the devil you know and all that... >>> >>> Anyway, if I get some time I might do some testing to see if this >>> change has >>> any influence. >>> >>> Another thought is if we need even store the size in the >>> iova_magazine? mags >>> in the depot are always full. As such, we only need worry about mags >>> loaded >>> in the cpu rcache and their sizes, so maybe we could have something like >>> this: >>> >>> struct iova_magazine { >>> -       unsigned long size; >>>         unsigned long pfns[IOVA_MAG_SIZE]; >>> }; >>> >>> @@ -631,6 +630,8 @@ struct iova_cpu_rcache { >>>         spinlock_t lock; >>>         struct iova_magazine *loaded; >>>         struct iova_magazine *prev; >>> +       int loaded_size; >>> +       int prev_size; >>> }; >>> >>> I haven't tried to implement it though.. >> I have very few knowledge of iova, so you can chose what's the better >> solution. I just wanted to raise the problem and will be happy to see >> it solved:) > > I quickly tested your patch for performance and saw no noticeable > difference, which is no surprise. > > But I'll defer to Robin if he thinks that your patch is a better > solution - I would guess that he does. For me personally I would prefer > that this value was not changed, as I mentioned before. This idea is interesting, but it would mean a bit more fiddly work to keep things in sync when magazines are allocated, freed and swapped around. It seems like the kind of non-obvious thing that might make sense if it gave a significant improvement in cache locality or something like that, but for simply fixing an allocation size it feels a bit too wacky. From my perspective, indeed I'd rather do the simple thing for now to address the memory wastage issue directly, then we can do the deeper performance analysis on top to see if further tweaking of magazine sizes and/or design is justified. Cheers, Robin.