From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C552C4345F for ; Mon, 15 Apr 2024 08:40:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F14D26B0089; Mon, 15 Apr 2024 04:40:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC5046B008A; Mon, 15 Apr 2024 04:40:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8BAE6B008C; Mon, 15 Apr 2024 04:40:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B66836B0089 for ; Mon, 15 Apr 2024 04:40:00 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3E6F2A0535 for ; Mon, 15 Apr 2024 08:40:00 +0000 (UTC) X-FDA: 82011118560.07.254B44E Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) by imf09.hostedemail.com (Postfix) with ESMTP id F2E8C140011 for ; Mon, 15 Apr 2024 08:39:56 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=G1Eu25sS; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.13 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713170398; a=rsa-sha256; cv=none; b=IwKoadj3Pi4bG2lCWXkqCABgQRPMcw13KfTavWAipWX5XjnILSf3hJWMqk9FjtSzhfGdes FgJDf/XEMCppWw7uZp/mu4W95hOvV+7z4YeuoXe7W+ktnc2x9uKv+g+/zOAn/WqhCup7AQ rdUF4A7EcWu74umL8oz+tEmEFOI/C0Y= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=G1Eu25sS; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.13 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713170398; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hs84+YgdhbhsnsQBAKo7iR+liw5k8DkubojRy7aNc/0=; b=BhyeKRZ4T6ImxhWIuiS/mQdwabeYEEw//+jvsDhODzD5EDFhRxNJoiU1CbXGungMvRWyS3 NM4GKv2wao2PDE1zPrZsHRG8sxMnWOITWIyRb9bUKugARtm/Z0xfGZ3iLHCe77yAVLsho5 n1V4YsO0PYi13baRKW24ZA6sKzWdNfs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713170397; x=1744706397; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=Vj4AFOE+O6c1sBlimkPSxbt9T/kes+LkocA0fd9Khgw=; b=G1Eu25sS6aTq3bdUez3Uar0dOPeh/yQEzIDs9HfgxbXIOzR/gvvrfqBE +Ye2Z3SX0+SH+Rghv6/qAuqdmRz9C+8wkk3GjIBqN4f/U2FJNzBtVy88b bMNy8hlaqJpVCsfjskR4cQfANbzbdpBM3hjDhzenD81nu7v+CkBKdSfKI zRQj1oE2vmE9xDPMAIJLKezgIO13YJ9FR8h6TxPdpx1Rfa1ZuUC78C5g9 4q4ke10Gga+Ut5WX/9XFuBSZ3ROqnQJu9ZkCszan41X+zms/Q0y9dSfkJ Qy6XxRTDu4EnGGPhcsk6YQe9scr2w1+uTq9/LCE9QYsnpgm/Zt0vbVOIW w==; X-CSE-ConnectionGUID: /R/jZuNNSPSj2lJcLZC+dQ== X-CSE-MsgGUID: vsaE6wMgSeiWQ8gkeSsmtg== X-IronPort-AV: E=McAfee;i="6600,9927,11044"; a="19695390" X-IronPort-AV: E=Sophos;i="6.07,202,1708416000"; d="scan'208";a="19695390" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Apr 2024 01:39:56 -0700 X-CSE-ConnectionGUID: /KkjZGMNSh6TZTU57YptGQ== X-CSE-MsgGUID: 7yCVsFnoQByTSOmVj/WfiQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,202,1708416000"; d="scan'208";a="52796101" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Apr 2024 01:39:51 -0700 From: "Huang, Ying" To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, linux-mm@kvack.org, baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 4/5] mm: swap: entirely map large folios found in swapcache In-Reply-To: <20240409082631.187483-5-21cnbao@gmail.com> (Barry Song's message of "Tue, 9 Apr 2024 20:26:30 +1200") References: <20240409082631.187483-1-21cnbao@gmail.com> <20240409082631.187483-5-21cnbao@gmail.com> Date: Mon, 15 Apr 2024 16:37:58 +0800 Message-ID: <87frvn2f89.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: F2E8C140011 X-Stat-Signature: zzxas5f8zhde5mhdadscuj31p4rs4j88 X-Rspam-User: X-HE-Tag: 1713170396-168871 X-HE-Meta: U2FsdGVkX18MBfKGfVkPYtUriFGeD4RSerbyDeb3IyL5/iLJJFCi660qwkzMhWsn+aly4qhIcaIe3Q19ct+gv0IQRaHcXs2TYfmdLzHP0e0hRy9MaVdUjYXIgV/sfMBOJWGGcUgmB5kCSlyAsVJIK4f6pPK3qff78K+7VGO56VkAkI8WO6iOQa5LgvmzhpC6zkDnFFL7a1W8tT7ewFFX6vVXAK/73J/jMILHltSd/NIlvUmBX9icsR84aR0WFwp7nWw5MxfY5xUklQFZ47eQEX4yESKa4lD8cFvi8GebtDnnyTXrSyfdHlxQ3cV03ZPl1sJpCOnl2MXhsy9QArvyMBtCOY2ifVTpxTf17XYnf+cQrctc/SOeBtPEj6oYhxBlK8hh4kQVOySE4iWY+D/bSrW1Q7I3MzASyvbJGjH5+DnvTGht+YZNEtUGd8lVDDnRduBthIFT7/Dz0j/m2T6HjyMlFYQAxcJQaeguKkFCRtSQEsH0zCSt1j33DEzKViKHmqFS52ugssKGDcmPIdHVTa4AuRa4QZwO+bjZnR9dFZduXkFyYD4W0LS+uOpOhGCSaEvvKVuZtlAj4OUnJ2y/XJM/2ny5Gr9zridz++t99QWisshcKa/WDW58wxkR+Nlj1BpDDj01Oln1bMlnUBRTscf7PilzFhHQyhyY3MlrJai5FogBy69b3E65YFBABkwZaP5d3kZ3WhCO6C8n4hSlr8lsWr/bZicpRlGzMMSPmF9s2sZKgR3EbY4gOH4aMZVcaJ8UwNW3/DoHwW5PoKHYmG1t+WtPBpmRpkaYaDXCs0rh0XEi8PXYHPtyvdHWEZii527PIO1O59XQFvRCowDcdNTQJZndPeztt94qPRsBfB8XYE2Vmel3lQCJ2BatZ7XooXp3dTDxyF6ZL7p3O37/VeFlpJakFFZhsosQEJEZt2sCusbVv7G1QgYTLe+B2y3HEEqd9aXo5gKoi47RZvy alnJ5E7P JvHRwJnMUAQzhFP0eLcl6KDZE87MFK6B0B4bIspx1pK9voKgowDVAOrN0lraO3rD7qK6ay4wkMqaem1pgC1vkFIFr4BgXf0/UosAHdosOou0knKK9sGle2+TI7ENMgdwgNcFZQkQ6hkjHf1mhseWYdw9wvPjeMVq4PpFzTZIn+CnGoQLv/IVa0Qd8EOnLsX07Qr8Wcwf2M45xKO7fmjkrMJl8srNQmTR86tOqeDrTW9+uGofPgmuC0eOW+AmLzRQGA6ur X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Barry Song <21cnbao@gmail.com> writes: > From: Chuanhua Han > > When a large folio is found in the swapcache, the current implementation > requires calling do_swap_page() nr_pages times, resulting in nr_pages > page faults. This patch opts to map the entire large folio at once to > minimize page faults. Additionally, redundant checks and early exits > for ARM64 MTE restoring are removed. > > Signed-off-by: Chuanhua Han > Co-developed-by: Barry Song > Signed-off-by: Barry Song > --- > mm/memory.c | 64 +++++++++++++++++++++++++++++++++++++++++++---------- > 1 file changed, 52 insertions(+), 12 deletions(-) > > diff --git a/mm/memory.c b/mm/memory.c > index c4a52e8d740a..9818dc1893c8 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -3947,6 +3947,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > pte_t pte; > vm_fault_t ret = 0; > void *shadow = NULL; > + int nr_pages = 1; > + unsigned long start_address = vmf->address; > + pte_t *start_pte = vmf->pte; IMHO, it's better to rename the above 2 local variables to "address" and "ptep". Just my personal opinion. Feel free to ignore the comments. > + bool any_swap_shared = false; > > if (!pte_unmap_same(vmf)) > goto out; > @@ -4137,6 +4141,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, > &vmf->ptl); We should move pte check here. That is, if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; This will simplify the situation for large folio. > + > + /* We hit large folios in swapcache */ The comments seems unnecessary because the code tells that already. > + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { > + int nr = folio_nr_pages(folio); > + int idx = folio_page_idx(folio, page); > + unsigned long folio_start = vmf->address - idx * PAGE_SIZE; > + unsigned long folio_end = folio_start + nr * PAGE_SIZE; > + pte_t *folio_ptep; > + pte_t folio_pte; > + > + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) > + goto check_pte; > + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) > + goto check_pte; > + > + folio_ptep = vmf->pte - idx; > + folio_pte = ptep_get(folio_ptep); It's better to construct pte based on fault PTE via generalizing pte_next_swp_offset() (may be pte_move_swp_offset()). Then we can find inconsistent PTEs quicker. > + if (!is_swap_pte(folio_pte) || non_swap_entry(pte_to_swp_entry(folio_pte)) || > + swap_pte_batch(folio_ptep, nr, folio_pte, &any_swap_shared) != nr) > + goto check_pte; > + > + start_address = folio_start; > + start_pte = folio_ptep; > + nr_pages = nr; > + entry = folio->swap; > + page = &folio->page; > + } > + > +check_pte: > if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) > goto out_nomap; > > @@ -4190,6 +4223,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > exclusive = false; > } > + > + /* Reuse the whole large folio iff all entries are exclusive */ > + if (nr_pages > 1 && any_swap_shared) > + exclusive = false; > } > > /* > @@ -4204,12 +4241,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > * We're already holding a reference on the page but haven't mapped it > * yet. > */ > - swap_free(entry); > + swap_free_nr(entry, nr_pages); > if (should_try_to_free_swap(folio, vma, vmf->flags)) > folio_free_swap(folio); > > - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); > - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); > + folio_ref_add(folio, nr_pages - 1); > + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); > + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); > + > pte = mk_pte(page, vma->vm_page_prot); > > /* > @@ -4219,33 +4258,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > * exclusivity. > */ > if (!folio_test_ksm(folio) && > - (exclusive || folio_ref_count(folio) == 1)) { > + (exclusive || (folio_ref_count(folio) == nr_pages && > + folio_nr_pages(folio) == nr_pages))) { > if (vmf->flags & FAULT_FLAG_WRITE) { > pte = maybe_mkwrite(pte_mkdirty(pte), vma); > vmf->flags &= ~FAULT_FLAG_WRITE; > } > rmap_flags |= RMAP_EXCLUSIVE; > } > - flush_icache_page(vma, page); > + flush_icache_pages(vma, page, nr_pages); > if (pte_swp_soft_dirty(vmf->orig_pte)) > pte = pte_mksoft_dirty(pte); > if (pte_swp_uffd_wp(vmf->orig_pte)) > pte = pte_mkuffd_wp(pte); > - vmf->orig_pte = pte; > > /* ksm created a completely new copy */ > if (unlikely(folio != swapcache && swapcache)) { > - folio_add_new_anon_rmap(folio, vma, vmf->address); > + folio_add_new_anon_rmap(folio, vma, start_address); > folio_add_lru_vma(folio, vma); > } else { > - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, > - rmap_flags); > + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, > + rmap_flags); > } > > VM_BUG_ON(!folio_test_anon(folio) || > (pte_write(pte) && !PageAnonExclusive(page))); > - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); > - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); > + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); > + vmf->orig_pte = ptep_get(vmf->pte); > + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); Do we need to call arch_do_swap_page() for each subpage? IIUC, the corresponding arch_unmap_one() will be called for each subpage. > folio_unlock(folio); > if (folio != swapcache && swapcache) { > @@ -4269,7 +4309,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > } > > /* No need to invalidate - it was non-present before */ > - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); > + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); > unlock: > if (vmf->pte) > pte_unmap_unlock(vmf->pte, vmf->ptl); -- Best Regards, Huang, Ying