From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FEF9CD1284 for ; Tue, 9 Apr 2024 08:27:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE8346B0082; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A96A56B009C; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9373D6B009D; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 763CC6B0082 for ; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0D0E31602F6 for ; Tue, 9 Apr 2024 08:27:20 +0000 (UTC) X-FDA: 81989313840.13.0E4FA90 Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by imf18.hostedemail.com (Postfix) with ESMTP id 5F7191C0011 for ; Tue, 9 Apr 2024 08:27:18 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dC2jeEj6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=KxJFmjBhmYw+/E1Wm0OrTl9ZnFxrmtZv13iroaXEa5023Jfi0jHOBx+9tMDTYnHZxG6xyk JV0UQND3L7IjLEs92+u5HKufeUpObkFc7lPDLbgfOkkWdsj0IoIer/LT4cSEie8o4ujDW3 wMZmvn5vtdzySDouy09dAe61BMIy+tM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dC2jeEj6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651238; a=rsa-sha256; cv=none; b=gJeO69qYwG7CkiRRucnUHiYh3dhOE5emNsKPCjpHRa1sT/Bn37Ge9oHdJza7ojEdzvu4bA ZWv6loHH6DxnLxwpnDV51Y353wZxjv6voB/VuMNEV1o82i8kYyoLM6B0Tlpntp3mZR6pP1 ODOp7/N6IADayoiJXvEVww5295uIpho= Received: by mail-oo1-f52.google.com with SMTP id 006d021491bc7-5aa1b7a37b5so1725093eaf.2 for ; Tue, 09 Apr 2024 01:27:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651237; x=1713256037; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=dC2jeEj6X12s/S26d9UZEFdjciYeT2anEKQ8bQAGwXwFxv1tdYgGJ3zaddvutpH3k6 wdVd+hke2a+su6z9TM2wbB1wnnIdxkPaci5sDk9SH/nW98X0/ivnzpMVuM7RpBwUKKLR bahcZvgkdnJS16yQ9LBf56xq//CWXG+EDo2xfInts1FdOUg1MR/pRLuUtgqKQtcW47Uz V0vx6U03VtYAJf5H8BySemLza6VtgaSRiKWhSyNzgyZ4tx7CvVCXd4XPAB9H16O5r2+V yhrFTTMtdgrJKHUcAhh9RhcrL5mepkb+l+AauNDl2KuQKY2uX2aq3WvP+47aARxPqSKc j6GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651237; x=1713256037; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=TqpWCwqctCaD4buS6fUcYudMV2cXMtbDxwRRY8bmpK6eNC06k9SmwrLQZdcYLy9nLP 5+m+F7b3S+SRzNbZRRhPS91OMSB2HwLJQfNsgiWCdsy7bYdfGqmfRyxAGHSi3eV4nkLG LfUdxcyz4m3qjRGoDJJx0NwnVAeHCcxmUJw72TUlzCHVYI3f5fhm1c3nHuDCov2WWROt 8heGP7V2a6Y/jMKVB9QvKAtwSdp1D4AZgZK3WcXGFpNmHN/Ysbfooh1KU+LoIM0Sdl7t yeJ8gEu3djEoCufqL/XDEf+S/tGIwGe9LbfA51I2NXmpIU0TYlPcWnSCtEqi6v+/piMl Av5A== X-Forwarded-Encrypted: i=1; AJvYcCX0+XGwwFcKP66zaAAIUo56corWk4bFzYQY95ugsrIA9cUQ+cM0Pi6HD94V16a594rvxDfBWK1TWDyeSjZHp2eyhcQ= X-Gm-Message-State: AOJu0YxO6oj+ubbb9OVOvkYZQ1LNhjugCVlNJGv45RTIdltUVukIovY6 bCDWmfUdHP+oSM4IdYK0ckIye48jHgZYpy1twbkH8YVO4FDk7O74 X-Google-Smtp-Source: AGHT+IFvN5ohj5Cxouk2/0fAxnzfDvT6AqBlF178427aMryNUnuyIkM8ZO3KggJQq1sFBYMpf6zx2g== X-Received: by 2002:a05:6358:d59b:b0:186:45c:d6de with SMTP id ms27-20020a056358d59b00b00186045cd6demr6322611rwb.26.1712651237158; Tue, 09 Apr 2024 01:27:17 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:16 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm: swap: entirely map large folios found in swapcache Date: Tue, 9 Apr 2024 20:26:30 +1200 Message-Id: <20240409082631.187483-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5F7191C0011 X-Stat-Signature: 3d33umup4mzgwryq9d55onctscjq9qie X-Rspam-User: X-HE-Tag: 1712651238-671154 X-HE-Meta: U2FsdGVkX1+7JDQBwQkhOMkRGxCDQE3cVoopAtmmaZpjOAiWy6ZbiO+ukQCok1U679q2hdxTXARMJvHjRJJlrhPvvDMRcU6Ybc0kc+KROFDyhSHuLkJndORid8a3iIdCMGVGnYdE0MjTH/dFHf4QyzcnmDW8r9UxnD7rc1Db+MADaxnypbXKOrZs5bnU/n0liF2TGI7Lmo8q9nhFyr0MtlCM9X/85DiqHD/S3TEDQAJcMPDbJkaay1GQst6YMYN211TSFtM7JAFtHxYouFm7iMZfo+ctlvpB9lWAL3KnpOMt0ZFfrfIqzMghHDtytE/0HL4n2u2bC4DWW/RGtt7gGcLenHgepiFWuFkglUdfOMK+Y53JwqUi3ULVK4x1Ds1BOI3HCRUSiKay7eYpv+1y2mGgc8lr6RWJCJ7+eBeOLMdAmXoT9Vou9VdBVsTkYCYwISGzdNaBNJWHX4HUv4tV4q+7sK9gRj97Xw6fDUyfbNVV8IBwRb8mhlQqKp7tGwdX10sq0GsgZEnbyvpv9zoXdnngbmSSsiO1afpcvsofAzKWmOyGz4Vfrm1H7BiA1nCcOQfvZV8knetC7hN+WfpKX0ly45h+HL1VxcKx3bs0lAkwLNp+Pwu5GVgMN8W9s/jaINAcYVc4CAqO6dTzMMiiWRc1J2KQTDbYMtmCQTDJqF9stpKFS752XBWGvjENzOAVWFKbu18MpUYLQR8YMdaP6GH0VZVOWdIQgVu3KrNWdFmeCgvALpUvc7mmYWkTeqmmuiiTLIJ4EA0qxtjAJREX60RZ8wEwYsPb2F4uBl2kQheH7uB/87f1KNwgd6tzxUyrh6nwY2aJZ9RSZVeJI0C4xHWFTOREH8pLFtSNnlqVBC+HKQwXhzIPH2V+Z7boJThIPfySZEa9c4kmWLj4AX8XHa/dww7L4hFJrjFrvRXWZbOXo1CfeocTTttYL2HnWLh5fC/gahCKdtTRWcyxhbY 6P9AVvpS TnLTjHEQ0MEhIbFV0ViE3f7Ix7nm9V2zFz4b6wqOGiV9M2LBk0UPrsy4EqhMD7/NJt7ENR19ZANK22f8h84fJUvay5sUofeQiJ3snxca6J/C35T0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 64 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c4a52e8d740a..9818dc1893c8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3947,6 +3947,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long start_address = vmf->address; + pte_t *start_pte = vmf->pte; + bool any_swap_shared = false; if (!pte_unmap_same(vmf)) goto out; @@ -4137,6 +4141,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + /* We hit large folios in swapcache */ + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + int idx = folio_page_idx(folio, page); + unsigned long folio_start = vmf->address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_pte; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_pte; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!is_swap_pte(folio_pte) || non_swap_entry(pte_to_swp_entry(folio_pte)) || + swap_pte_batch(folio_ptep, nr, folio_pte, &any_swap_shared) != nr) + goto check_pte; + + start_address = folio_start; + start_pte = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; @@ -4190,6 +4223,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive = false; } + + /* Reuse the whole large folio iff all entries are exclusive */ + if (nr_pages > 1 && any_swap_shared) + exclusive = false; } /* @@ -4204,12 +4241,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte = mk_pte(page, vma->vm_page_prot); /* @@ -4219,33 +4258,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, - rmap_flags); + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + vmf->orig_pte = ptep_get(vmf->pte); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4269,7 +4309,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); -- 2.34.1