From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97E96C02198 for ; Mon, 10 Feb 2025 07:11:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E28696B007B; Mon, 10 Feb 2025 02:11:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB1586B0083; Mon, 10 Feb 2025 02:11:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA01C6B0085; Mon, 10 Feb 2025 02:11:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AAC7F6B007B for ; Mon, 10 Feb 2025 02:11:24 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 61F7EA055B for ; Mon, 10 Feb 2025 07:11:24 +0000 (UTC) X-FDA: 83103164088.27.903A838 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 43B2540013 for ; Mon, 10 Feb 2025 07:11:22 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739171482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h9do5zv6WIomCOMNIvZNZA7b88Z82prAKLGB7Owb+yw=; b=tELq+tHS3afY1Nz2xEWYeRP7/ruZ9W/De9v+pnSt6cifsIfvhsR0LHxwypHITbULWh9hEa YWg93L452RRaIu7Xfy9Vmkt4g1s1473lkMErLyKqaAIiQE8wHHzUMzsOnolPuwuQIuWLRl q2XdEXapBx446o2FheHh4dteXUwDAY4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739171482; a=rsa-sha256; cv=none; b=M7gs6RPweFsg9i/aaGI7Wdvzyh6spHSpjGvgByObwnKagyf8bd9723Izz7lWaKeQbuq8Pu uQIm8IqKijsg8eX5EIlVF80XqNgQP4ZqDvrnNqMV9R01vx6TMCRfuWehFbzKwDLHcldazS +K4MYoJMM+ux2K8rLdHG0aSigDp/srA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4217A1BA8; Sun, 9 Feb 2025 23:11:43 -0800 (PST) Received: from [10.163.35.99] (unknown [10.163.35.99]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3504B3F58B; Sun, 9 Feb 2025 23:11:12 -0800 (PST) Message-ID: <7e184caf-2447-48d4-8d7c-1b63deb0f418@arm.com> Date: Mon, 10 Feb 2025 12:41:13 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 14/16] mm/vmalloc: Batch arch_sync_kernel_mappings() more efficiently To: Ryan Roberts , Catalin Marinas , Will Deacon , Muchun Song , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Mark Rutland , Ard Biesheuvel , Dev Jain , Alexandre Ghiti , Steve Capper , Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20250205151003.88959-1-ryan.roberts@arm.com> <20250205151003.88959-15-ryan.roberts@arm.com> Content-Language: en-US From: Anshuman Khandual In-Reply-To: <20250205151003.88959-15-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Stat-Signature: 8qfty8thipbjocnb7k8infg1nx4p15ai X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 43B2540013 X-HE-Tag: 1739171482-278801 X-HE-Meta: U2FsdGVkX19heIMhxverrgJZSa+49g1lecwhWjV7d9GlS0FnVpVcUqRKZ21znRk8mRbWPPECPfixPb2chMxqdy00mAh6QL8HyepudClQaScMxtkoMRGpVaO3XSRyLlpj0iTSzptcwCw2Bc/WT44XhdAgn2r7HJ8zQ7IWdmDnNDHCFO8AebxiUt9A3Zbd/AakNp9lYFof3V/jVfzZNzGybLDPzTuwkGftWXaY5IsUOuM/blAtexh8qzNrE0v50cxzseDW6Cl7+G6j/Yofb44ra95PiV3WeKweiwz8bF4jm3HMvcn2QpnU6Jq5yLaDXEfrKpZQDGjKhppEyUzi+cQOTL3ZQ0QNLZwgT4eEV2/eg6aM4Cl5xQeVdt+czOWtC57yaPHgRdHFIy/XFIbZ/Wv2QJjaXMD0RH5fJFK08/WNXvpnQw6G5YBdpf0GORPoD9rZye1x9Krx+loK8dSf5heIafH/SJJj6BaXR6fC4noiAU+wLAbyacMqduwk6wzyP4ey0bmEMwkH9T8ilAG3Qs2p2402UynmlctZAqmjlIbUc7M82r2xLETBkkQYLI2uh6yV4TzQHtnWIl+6RotGkOzCiUQg2bJS1nrQFbRPSgG3MHX5JdaWueje7EUOLunvfp5lWczPQl2zyi0Jk3EBHHCXCtLtFH4Z9k429n6QALMGGN09A2SS3ZnkAz+tnkDBWgYhbKJmQPMC7Is34zbjeM58uf5kyUFXrR8RQcEttnu8en4fd9OCHpzha2EsR0FYFCRb78IZQeoDOExFY2CMQXzdXp+1qweT5KY6zAlsjMNqVze0/8krt5tLnoPpJYPWbFrBEBOtm+OGgiei4y+yMaT5XwF6uNjSH/UJD7yFk0gDc+9yuRuS7v85eV+ps/vlPLDgR/t5McJVOEgsYzgQbh2DNSa4O5kqxJ1J8A1uRExjvo4v0gc0oYAeR+urKJ/yBMonJTJCGSq/8jDduXAem4Y m6Yd542M /LbvN0cJNd7ObS9jgzGthUgWjeKevQYztU0dAmqbGO5J1fbc/f98g7z1rZ+HZ+hg44ai+RsFGMeK07UmWNnnC2CAR3cSR2q8xqyaebt+micHveGlGOAiQ019KEYMvSusirIl1JGMk8lJOikB7/47UO1K6AYh+e0sxlSSw+rDm1LXJVBmH5FcCQMH2ooNkyxkPu3tHmFDGe2LFBP9rVsLg1Ei4Q6Nq678AtlbaHbwNjF+XDzh31yjvU0zVkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/5/25 20:39, Ryan Roberts wrote: > When page_shift is greater than PAGE_SIZE, __vmap_pages_range_noflush() > will call vmap_range_noflush() for each individual huge page. But > vmap_range_noflush() would previously call arch_sync_kernel_mappings() > directly so this would end up being called for every huge page. > > We can do better than this; refactor the call into the outer > __vmap_pages_range_noflush() so that it is only called once for the > entire batch operation. This makes sense. > > This will benefit performance for arm64 which is about to opt-in to > using the hook. > > Signed-off-by: Ryan Roberts > --- > mm/vmalloc.c | 60 ++++++++++++++++++++++++++-------------------------- > 1 file changed, 30 insertions(+), 30 deletions(-) > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > index 68950b1824d0..50fd44439875 100644 > --- a/mm/vmalloc.c > +++ b/mm/vmalloc.c > @@ -285,40 +285,38 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, > > static int vmap_range_noflush(unsigned long addr, unsigned long end, > phys_addr_t phys_addr, pgprot_t prot, > - unsigned int max_page_shift) > + unsigned int max_page_shift, pgtbl_mod_mask *mask) > { > pgd_t *pgd; > - unsigned long start; > unsigned long next; > int err; > - pgtbl_mod_mask mask = 0; > > might_sleep(); > BUG_ON(addr >= end); > > - start = addr; > pgd = pgd_offset_k(addr); > do { > next = pgd_addr_end(addr, end); > err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, > - max_page_shift, &mask); > + max_page_shift, mask); > if (err) > break; > } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); > > - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) > - arch_sync_kernel_mappings(start, end); > - > return err; > } arch_sync_kernel_mappings() gets dropped here and moved to existing vmap_range_noflush() callers instead. > > int vmap_page_range(unsigned long addr, unsigned long end, > phys_addr_t phys_addr, pgprot_t prot) > { > + pgtbl_mod_mask mask = 0; > int err; > > err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), > - ioremap_max_page_shift); > + ioremap_max_page_shift, &mask); > + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) > + arch_sync_kernel_mappings(addr, end); > + arch_sync_kernel_mappings() gets moved here. > flush_cache_vmap(addr, end); > if (!err) > err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, > @@ -587,29 +585,24 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, > } > > static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, > - pgprot_t prot, struct page **pages) > + pgprot_t prot, struct page **pages, pgtbl_mod_mask *mask) > { > - unsigned long start = addr; > pgd_t *pgd; > unsigned long next; > int err = 0; > int nr = 0; > - pgtbl_mod_mask mask = 0; > > BUG_ON(addr >= end); > pgd = pgd_offset_k(addr); > do { > next = pgd_addr_end(addr, end); > if (pgd_bad(*pgd)) > - mask |= PGTBL_PGD_MODIFIED; > - err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); > + *mask |= PGTBL_PGD_MODIFIED; > + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, mask); > if (err) > break; > } while (pgd++, addr = next, addr != end); > > - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) > - arch_sync_kernel_mappings(start, end); > - > return err; > } > > @@ -626,26 +619,33 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, > pgprot_t prot, struct page **pages, unsigned int page_shift) > { > unsigned int i, nr = (end - addr) >> PAGE_SHIFT; > + unsigned long start = addr; > + pgtbl_mod_mask mask = 0; > + int err = 0; > > WARN_ON(page_shift < PAGE_SHIFT); > > if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || > - page_shift == PAGE_SHIFT) > - return vmap_small_pages_range_noflush(addr, end, prot, pages); > - > - for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { > - int err; > - > - err = vmap_range_noflush(addr, addr + (1UL << page_shift), > - page_to_phys(pages[i]), prot, > - page_shift); > - if (err) > - return err; > + page_shift == PAGE_SHIFT) { > + err = vmap_small_pages_range_noflush(addr, end, prot, pages, > + &mask); Unlike earlier don't return here until arch_sync_kernel_mappings() gets covered later. > + } else { > + for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { > + err = vmap_range_noflush(addr, > + addr + (1UL << page_shift), > + page_to_phys(pages[i]), prot, > + page_shift, &mask); > + if (err) > + break; > > - addr += 1UL << page_shift; > + addr += 1UL << page_shift; > + } > } > > - return 0; > + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) > + arch_sync_kernel_mappings(start, end); arch_sync_kernel_mappings() gets moved here after getting dropped from both vmap_range_noflush() and vmap_small_pages_range_noflush(). > + > + return err; > } > > int vmap_pages_range_noflush(unsigned long addr, unsigned long end, LGTM, and this can stand on its own as well. Reviewed-by: Anshuman Khandual