From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA6DAC54ED1 for ; Tue, 27 May 2025 07:51:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C8CE6B0085; Tue, 27 May 2025 03:51:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A1366B008A; Tue, 27 May 2025 03:51:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DE2C6B008C; Tue, 27 May 2025 03:51:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1D9E36B0085 for ; Tue, 27 May 2025 03:51:06 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C3EC61A01F0 for ; Tue, 27 May 2025 07:51:05 +0000 (UTC) X-FDA: 83487916890.05.0AA0B08 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id E51FE140011 for ; Tue, 27 May 2025 07:51:03 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf26.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748332264; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=cVK1L1KRz/Nd0TOO2/Dq5iM86olJ3ji0qjvu6Ea9L5o=; b=3cOUuLRPX3mtxzk3niE9a0cahXwSwK5tXel7zUteB/OEIP/cmVwGa3u0PJcXiDa5Ew3iJF dy+a9Ktwm0VO3d2kfwJR2/XLgywThOLqxHH2W+eH5H+MfN4wK6c+JPAMF70cp+rhgtLNnc 3XW90+7DAWIYTPez1QNCCP5hZPtFBlY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748332264; a=rsa-sha256; cv=none; b=A8O4yxa2PfBFSRg0ePL6sHXYYzWlxA1+gjphBHGbzJclg7u+MhYEdwSKd7pX6H7X4SeGQm R+4+7+K51rz/xYs9/pN8aJLg8PMQPrOnFwnzKKAl3xZtk4Bi7JHoDRzXHeL9lygye4k3GV cwhnKM06nrEJuf3l5kwel7Sf4wNlgRk= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf26.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1EA4614BF; Tue, 27 May 2025 00:50:47 -0700 (PDT) Received: from localhost.localdomain (unknown [10.163.85.29]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E66073F694; Tue, 27 May 2025 00:50:55 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, peterx@redhat.com, ryan.roberts@arm.com, mingo@kernel.org, libang.li@antgroup.com, maobibo@loongson.cn, zhengqi.arch@bytedance.com, baohua@kernel.org, anshuman.khandual@arm.com, willy@infradead.org, ioworker0@gmail.com, yang@os.amperecomputing.com, baolin.wang@linux.alibaba.com, ziy@nvidia.com, hughd@google.com, Dev Jain Subject: [PATCH v3 0/2] Optimize mremap() for large folios Date: Tue, 27 May 2025 13:20:47 +0530 Message-Id: <20250527075049.60215-1-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E51FE140011 X-Stat-Signature: zzmtuwh8rb1pfztay77a6zauiabpzkjr X-Rspam-User: X-HE-Tag: 1748332263-445846 X-HE-Meta: U2FsdGVkX1+G5i3la4oVprwc9s0aypdqYhQ6JDMepDNAoJKSLHcsmFfuA1+Y5HxQMQSjOTrNygbdKEZLnYlDZNit/meYpooTMVIbbCbzbl6HHUUpieh7Rz/qPkBbSh6+GNr68znFO/kJ5R0CZEwQ9rj/WQAhRtwohdIoaVajspm4oFysAEDDaGUknMgwl2NDwYe0WSxrWVQSWpA8pV/E9RlZVfuJRu2juvxJ5cgDwm9PrDvb7E6/TyF2MSwZW71br7owBPIRwL2k8zUGfxewn93NBqRA47/2Im4vY0yS5Fm0VO6HlPPVNdIqbkDzvUCtcUPBO7/ReMWZi9WZeoY64Ft2L/BPLIwdf0vCsIZEoaFRLPZjT3ocpAAe+TzNwTN5StT7itqAHBCEJIFju91QdCT0Vsit5lrQqzNvZ7pqYj8BNFXi2RaEPuDyEIsYm2MusUoA5WgKidlkPDE7MOhF/F6RPbu4JC21mTSw8bAR3hkBWN2da5joeiolgmSUjxSapuNNaiKLE8w8mclY6Q4YS9vYpdxlz81w8t0tRbB6DbkDQbGRmKVCFLB8nztJswbd1heDTm8NOjjoXntU/PgMpUp6S5bqrvUDLtc7Zxglwzsp/dvTR0CAtlJ7gFVTqWV490e8DuO6JHKDRw5/hl16kD4VBcoRXTlH2pe90SRqSF86a5fkzzEVj04M8lxiEtpAC1V94TzGSOEJVF7p4Purtj3qV2Sd3Qqv/EyYuScEvWllyQoQQOvp1qwLXxSr4MgLyfV/npvI8eTYJToIbzz/BC0rC2Y1ESr2qO51jhnOi6HLxikDPS2KRzaMT/Vejg7efyibaT2KzY13C4JbMDfZEH8lj9IIxjuvQ/+V84HZcM2Nt7c/N8E8l0ZVMPg+8w5VivyGteNvw9II+EPepyEga4zTZIfLwHjgzG9W0CZreHfHJ5DfLUSzgcBliEh6Q69wkpDuBQJKe3Pc6PaTaGE XMW3e4Xr lBbF4PueTN3cj1MBjX6rVNWglrkdSbcd59gik1TH5f3iRNBZXRvpSMWYmfR+sP+AewoMlT+Smma1rKCH2t74mXAOhnnkYNgf2Ji3F X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently move_ptes() iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch using folio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number of ptep_get() calls (since on a contig block, ptep_get() on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs through get_and_clear_full_ptes, replacing ptep_get_and_clear. Mapping 1M of memory with 64K folios, memsetting it, remapping it to src + 1M, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64). No regression is observed for small folios. The patchset is based on mm-unstable (6ebffe676fcf). Test program for reference: #define _GNU_SOURCE #include #include #include #include #include #include #define SIZE (1UL << 20) // 1M int main(void) { void *new_addr, *addr; for (int i = 0; i < 10000; ++i) { addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 0xAA, SIZE); new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE); if (new_addr != (addr + SIZE)) { perror("mremap"); return 1; } munmap(new_addr, SIZE); } } v2->v3: - Refactor mremap_folio_pte_batch, drop maybe_contiguous_pte_pfns, fix indentation (Lorenzo), fix cover letter description (512K -> 1M) v1->v2: - Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducing mremap_folio_pte_batch(), fix loop iteration (Lorenzo) - Merge patch 2 and 3 (Anshuman, Lorenzo) - Fix maybe_contiguous_pte_pfns (Willy) Dev Jain (2): mm: Call pointers to ptes as ptep mm: Optimize mremap() by PTE batching mm/mremap.c | 57 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 15 deletions(-) -- 2.30.2