From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A514C4345F for ; Fri, 3 May 2024 00:51:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CA6B6B009C; Thu, 2 May 2024 20:51:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2796A6B009D; Thu, 2 May 2024 20:51:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11B186B009E; Thu, 2 May 2024 20:51:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E689C6B009C for ; Thu, 2 May 2024 20:51:29 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6726D1A0DED for ; Fri, 3 May 2024 00:51:29 +0000 (UTC) X-FDA: 82075256298.25.AC2753C Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf09.hostedemail.com (Postfix) with ESMTP id 91D2314001B for ; Fri, 3 May 2024 00:51:27 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="J5bRGjP/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=4sWuIFZeyA3912pgd7+ZHRc6Yn/7I8b2k9gyqj2zG3HFCtPgj+a4cg0DmQST9YPViOgatf 9mqPKSohiYgL7TXWW7OSFt6ce0J4QZWY4VDfHGZVz/NQJ2NRFt9fkNXgXLg1MVj2dKzMSB H6ZJGhnxjYWjobUn3twbJOupHu8uPPA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697487; a=rsa-sha256; cv=none; b=edWYrQ/Wjr5Hp97vIGdSbU9/edVK3ZzvCepdP9/Fp+XcqHgquFOPDv/p2ld42fi2v82yH8 Cmw2gMchhe52Cj9EXMtxM+kYV0BSVIbzETXOsf/sY8IRqC9JZSyvgeyH0iVhWx6dObGqv8 wsoUEsg6zN6pt5t1XhFmZmf35Asoq1I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="J5bRGjP/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1ed012c1afbso9547425ad.1 for ; Thu, 02 May 2024 17:51:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697486; x=1715302286; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=J5bRGjP/hoSoGrAgIG0BmmlJA7DXN8LdIkFpK+cxklevrrmCfWWmp0wIwV8xM9h84w iPdxZmWqxJN8SsWVi+xR2/6Tj1C/ilk92YukiOMzJbhb0/6+yM7VU5IUpnZSsyGXftqD kal12BMfeJQkuT5lPVN2RKm8Ep4+0xxXQU4Lm9krjLjhFNxt6vV/PGwjBC92wmDPdril aqWP6y1gx86pWYabKH3LwzzlqyaOq14vI4kF+Q0q+crV80GOY/XEBLLxAenQ32rz5VJp tFGHG+h029EhwCwYLn09u2suP56FKo05HO1HZ3spBQcclBN9z/TpdTTr6HpGa0LU1cT9 SDHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697486; x=1715302286; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=nrTe3KfeW50n65GLn3GlLfXvEv7R16Bi1TfoE79dcBb9iUqzyMfmHPdk8o0tzc11Sa jmObPkJrK4tvGEkPy8MDyqp/9Lbx6Htv6no4m4C3F+OiySbHPUnuMtvcOnubRAxTPD9r KwBVLSNf0/3Xs/I+rACjL13m60vXEbKCqGiOJCJjAqoMY74AUv9Mh0YFbHCxqbO37SYf vq1hg/o0JnA3YuiNwfkML5M+j1N3k8r03XU/6fVqqFbj5UatiIdzH2KkR1Z6BLMFJiFP d0QwIv/x4O9yATj2HF41Y04n5iUf7aMwgZ7YIt26j0VD5OsUgmWvBAv2zalDfw2eStDZ VyZg== X-Forwarded-Encrypted: i=1; AJvYcCUxiLrRfet9glYbFwmLNsFndigA1GlK3saUPSZOMLMei4i5+YnwLCeqtlRLKnV601KKhEOMHE6THSBG/FTHMg4IWO0= X-Gm-Message-State: AOJu0Ywr04zk0OPEaltcZDAV4dR5gN3Xzu9zcl4gf1fqIMfn0CGFAnF/ 4NwIdNPipOB+xd2lyNWSxx6PNagxaEf8oeltJAXWJcZBVANzgew8 X-Google-Smtp-Source: AGHT+IHl/5rfXtgmTPFxtFxlvOgwDmCzyiDZXrRPvhqN/z1hpeuCBJnRPssnBcPqlIhHPeY6Yn+CcQ== X-Received: by 2002:a17:902:c407:b0:1e5:e676:4b09 with SMTP id k7-20020a170902c40700b001e5e6764b09mr1726427plk.25.1714697486404; Thu, 02 May 2024 17:51:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:26 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache Date: Fri, 3 May 2024 12:50:23 +1200 Message-Id: <20240503005023.174597-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 91D2314001B X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: abhmekcrrx66r7uyebc1tane75qdk5bq X-HE-Tag: 1714697487-151288 X-HE-Meta: U2FsdGVkX1+t+YHZVvS1dFPg8vHFWDxijRJnD0/ane0idqNhMKVYvPam9lTKWKV70Wgy2pmipUL0/9EFa+V7qF2es7BFbhTXDgDiXRPvzcBQ9xE/g6eNVAxnLPAF2DSR/AJL343EcqYSnLrYCah3keu87o/jsjlGir1tWqH0N+AItG11JwHOd/aQMNwwGadWDjFehPHMbNVdX9V3j1bcG6C80bXxHCg3YUHbC1KHU0nWdzPk6NOhCr5kaFDNNm91+/yLznB48CPZCHxqnnrHS3kl1oZeVLA/ku2CDq6uIOBQH1/bJ7OZ7knptUcO+e54tcPoNXkQnvxjgfJ/UuRaF5GjVUMsATOOhDl24brScGaow8zb+6bhCYHpVLvLOaAvdu3qBAKkltNhxd2sYUiNsqSPFjDDakNKqRur092hpkvr5u6F3OlMMrjL6UaLJI90YNmjxFafCs02p5UGxbORXLA9N4z/pYY2aGRRnzEMMirVYCdGrOAftR5cbK4YSxyufh1iAp+l5HPNZctbFzIZKC44IFwdTPcz8lB3BJfxtyvg0nGptYf1iwWgxXBtgKhhjvp3GLNdMNk0GwMil7GLTvUHcvVWhN5qOjJa/CpYJEuyvSwciCaBzRPz7D1l41dtdxYE1KWGWtogeVz2AT4GT2FXaKt0eCxNo9dk5wNpineWuqjgyym8wTBLElb2ppQOVi6IH6kZ67dzhF8aDRYoAtzMJHz6kqsJWZUNHv2JElpuCk8eB0izXsLN3vEjAiFBaeIAzO+pHEBKhqQ/jQ33DRxMp9QQH2HojjeHsZkkjW6fM5fJP0UtXw5eBuJW9Hysnwe95iTaea8blTbRx2o9dOHo1Qrt39rGXVFjXzAGtiaQOOPwa25NeGNLwOFaXwIc3hgeAi+ZQaOtyqSq1wlO7qnzXmXMypFu/SZhL51s+pCeWQ8C4Oqh88efNtyAuVw4Mbqf9pempwOqyvk/BOC uObNGXe4 3I+IfTSG+6Zzyc6KG5q2KS1fFuJ9KzQNx16l6vw8Aio74vOFqAIeDZFASX3oFJtNx+wmiNLEoH1P34hosp0sf3UeeGzj5on/FOA2rgDU6MQn5BtqqNEDggF+/yQ0PtS+jQqDUtwkNYyr0BkQQPcl15gx2rkvFkrCyRKhjIblp/pRFbEMfjXamaJt4VyaFJWkiXGLU+gAasxhSRji0A5yXLI4JAGwsYsE/vvUcyhX8PnFUC1p2yV7jLaLtfXg5hVQakcoB6NpRYe3bjNlSRhjYqZNJBaCoO6apXQAmX+J7x2CkeZrmtyfOz2zu4BBIK6fpn0UhTyJ0GSXjYFwQOALmXAUhBV+63w87/k0TkwbRrf9dqacOQ9ltObzGC5bkuglYWC/Ufx/J/8WXOT0Uq5wMiVdltd+SWaaZzdtH/z/XKQmNliChNOoD8TSpVUBF68ktDc/AOL5RCChWdmoKchCKfG+VGiSj/qQsRcKA2z7PgRh+SzY2MgIpmyqNQQSwekykb1KfxtgIBRoEYU5U41IWAwBufb+Ayl7hzvNEslGj7iKAZK2Mi0HZ4qO2ew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 22e7c33cc747..940fdbe69fa1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long page_idx = 0; + unsigned long address = vmf->address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = vmf->address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: + /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free_nr(entry, 1); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); -- 2.34.1