From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0300C67861 for ; Sun, 7 Apr 2024 02:24:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8B7E6B0082; Sat, 6 Apr 2024 22:24:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3BBB6B0083; Sat, 6 Apr 2024 22:24:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D028E6B0085; Sat, 6 Apr 2024 22:24:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AD7616B0082 for ; Sat, 6 Apr 2024 22:24:43 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 355AA1A0454 for ; Sun, 7 Apr 2024 02:24:43 +0000 (UTC) X-FDA: 81981142446.18.BF7B4A2 Received: from mail-vk1-f178.google.com (mail-vk1-f178.google.com [209.85.221.178]) by imf13.hostedemail.com (Postfix) with ESMTP id 720A520005 for ; Sun, 7 Apr 2024 02:24:41 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DM+AZdgI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712456681; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LWlN2lISbX21Oq2e3itHZr4X69/Hs7x1fXROUCd3H1A=; b=04HQeKAxvHXh/JPqfPDcPJ9sX29AfLRsprWvoSXtMUXMBGJiKI+/ZUDnamGjR2bgaQrB46 /UR+0x5SYE87m8qQOHkTPDeidxlAu1/fQetFzzG5A9mRvC6o1Dr+ZY8VaoI9WBPa6d7pog HQa8RVY9QcFSxoi03SCam6Waa5aFsl4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DM+AZdgI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712456681; a=rsa-sha256; cv=none; b=Bo+DWbaScngeFyl2A5mEiilR+rD95RpombFAw4mi8vmBRk2aooPjndLelS97sBmpXh9GC9 EYgMz/zdlpyY60c20AMMVAj7M9Bg9vqIl1lDxXPMe3dFNuZMeaZWPKeXypiFgiKGppc0bm YZt43c1/hpHpUkTgmIf1WVvHhGqWCtg= Received: by mail-vk1-f178.google.com with SMTP id 71dfb90a1353d-4da99a22b18so1181670e0c.3 for ; Sat, 06 Apr 2024 19:24:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712456680; x=1713061480; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=LWlN2lISbX21Oq2e3itHZr4X69/Hs7x1fXROUCd3H1A=; b=DM+AZdgIOYhzAq9eGCZVMlCcRBOBD+4FL4ng2kNSi+x0xFJkUhSVLJEXxkMMGCZlvr ujEs5QBx0vmZRpaSK2l5hXFPRNLDd9QnGuwiCdmiMKbLSA4v6vfF8uzaX+4GzKpIe/CO rd+p3WvDdtlkv8dqQYste0fmcK/R+8fW2Rh/pL/6/ObwSEw+bnHVl/lXCNna1vbN5U4n gs2EI5YKbgp2uhtozIaQOzrC57RO+puB8w2WIIEHqWMyTNev0fYzC7kYlVrpHsWA+3hP mjvHEGpuF+XeRX9VJ/V4eKpyz+DQgaao1CYK8e/lu7sgkYcEa9d9xjO00s4wYBNNmrHL 0G8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712456680; x=1713061480; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LWlN2lISbX21Oq2e3itHZr4X69/Hs7x1fXROUCd3H1A=; b=Z5Dksw1E5ystKWeLTvLBgfTCTjHcIcQuP3U5nHMEIlERQLwWd4/yzlfF3LkEj6CzUY XHIdgjsLvM8JWS9IRyyDP9IAE/ZFZFxncb7d3i+RDHnXt/qcaNC6VOFDMGofu4U276zN 4J8NaALKj0yusQMuOCqUK7GwNOyP7Svg36o7WGvFj9NsxIhbY8FMh06vsuT+TRaQW5BA gsNs7UjObxQWnXJc6gDeAtZichz9pTVPFvgoWX1NJWTLC3cqk4eFkE5vn9xE/dxg8vJH xSEuUFT7HcelbDSVUz4mbu0OxkRI9gR1xwJQuCOe6TmSwKIadAfuxTeikTanFxWtdoKh C2tg== X-Forwarded-Encrypted: i=1; AJvYcCUaYOaTOKBjuLKFsHuTCfhNbwhBoeuHuOiRmI6qlZqgyam1b97PhBLizxGgJ0eoRk0n5fnSvxJmRNZT92W97BK3Mx0= X-Gm-Message-State: AOJu0YzcBppOxiixqM3ub8fpj3WGarNixzV4XUMm++QEO9RxmhbX1uum q/Ukdk/ihhVZoyFwEkhyHQjxVtg0kjsA6uM7CSbPIHhJIRu1cfGoggv6oMgPnNUt2LaB8MtkKN4 hm0Jnc6d0lxuPAgp3J/iedNu44zw= X-Google-Smtp-Source: AGHT+IFWBFbGiMcPCaYGXf3PYyeL9yX7vS+aUvX8vGvt8WXH2V75n0PhuiEI0+T+zJehLVnxuMMOAwaNgCMmuZ13AW8= X-Received: by 2002:a05:6102:3f0e:b0:478:799b:a426 with SMTP id k14-20020a0561023f0e00b00478799ba426mr4970379vsv.26.1712456680406; Sat, 06 Apr 2024 19:24:40 -0700 (PDT) MIME-Version: 1.0 References: <20240402073237.240995-1-21cnbao@gmail.com> <20240402073237.240995-5-21cnbao@gmail.com> In-Reply-To: <20240402073237.240995-5-21cnbao@gmail.com> From: Barry Song <21cnbao@gmail.com> Date: Sun, 7 Apr 2024 14:24:29 +1200 Message-ID: Subject: Re: [PATCH 4/4] mm: swap: entirely map large folios found in swapcache To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 720A520005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cbcxdji1fa9e8s3ph5ydjo6cg3t5c44x X-HE-Tag: 1712456681-167172 X-HE-Meta: U2FsdGVkX1/4u717iKJXIdN0iNf2T4+y94R/sTNybq5sbd1+zeVPRhPexA7K0Vy0dkLCLHqXLJOjWUuc2BITWrqWUV6Df4MFuDRXO6G6Wstr7CrGhaHh/i/QyYh2qSSsv9JIU9rfedjwsg/X/SrWDX0IfzX7O/AAiwjEqyChi5ordQ8gcEwQRQQ0Lyz/YFrEvrrTWlFKXpINcQIEIZo7Ub0rg1k4C0QvSaz/hVWTG/qidCyOQKEcP59KrcsO5XzFYRG8j/ZOz9+d3s+ADtH0QH0hahIjOxHjr3XGN+jRaVSpE4MxM83zU2h3yjaSvQT8ZgtWHWElb3yaaK/KD4z2gPcS5OiKyNXGbWu8KQMMXRD7CYEtujpCfJ0j1scTKL+vJ7klUlNI3RiyNd+ACYViWjjZlTeogHVD60OXtPFQvnmH+owFPDMR+Qhi4+A2kT4QS6+hVsEaYQCg5MY1hIQKFwDTKNAAUeS2Cxyv9X+yxqbO4CvMRYwmLO+bxFyVD4aqLreRW6T/G2zZ65mzDNzynFP/ixdJqV15PODhiDGZVBvNQCW5zZ2KGX/TPEvM7yx5G16qBCbnsbOaqZ7PnivdpP4FnTpcjKxxyWoPakBoqiLCso0jnFI+Uu5mYA8lxW3k7O27JG1yugdxxUoVrn2vwkarp0MlwG9/1pLUr9z8ozZZQXWbzBw/kWR+X6bApZCTdUVA9Ba/tY90AYriQPf1zu1cJyJ612RFqCjDAzT4IjhEYORBnBqnRvU41aQTOhnbC/SEgJxNHJ5pONFT0j2XGmUTMNb3C9pazLg6QTE7rsJTe4hhTM4nkxPzMkWdnK9U2oR2KqzPgWZqFajlidrn35xL9+AXrcmJ4Q/TYV4QxgOnY6uTNtB57YQ7i3UMF0vuZd93sYcrXwOCFK4YHFIoG/tRfplvmVYA0CBF1B3dBRQN+RYaXItsgEKuVNI8iyn6LtF2NWxsrpII9HCszCb pEhr8Shh rzqzSq3IoZQas/IlgKlBPl1+E4ZOiA9EM9ts37KjYycNh4vWuH20cL68zw4p1QlqwjJQbfR/rE4Syo0Z+jwCs9FUkcPD5Tm0nehb4tFUWhdaH6+oPsVnjHGmzG1QHWk3d6VPrFvl2EawR+oCfOcGL4+rC5oPOca3vq9XRDjhzD4Q2LzP9GYqnWD9Rxx/dq3FbuBpmDpf9pxKqsKyGFbIyBlAmyFRub9KFH8XrFT2s9tYi+Zlubjb0o9VfslPHTYnoqwuAm+/9qSEOr0Mt+Z5OXWd8eYaeemcJiI0BFeK0eKklf4iWWjqa+0lXxR6ZAvsu+SNt4APY3t0MrcV9P+/gUsvHpzfWrHg2Hi1+c7Sb/VUFnYvsJnETMV7gV0Lf0rrI2TIjR72qT0NVKyP9T3TJbxwgM4aizCHzkQ3IKtkZRnGn6OMXe2GfmhHn72juvaw43Q+rpitF/jL4GOBVMc35OQUwHQWQOI+NxXKUQRpaktFifQLmr2+CUtlrHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 2, 2024 at 8:33=E2=80=AFPM Barry Song <21cnbao@gmail.com> wrote= : > > From: Chuanhua Han > > When a large folio is found in the swapcache, the current implementation > requires calling do_swap_page() nr_pages times, resulting in nr_pages > page faults. This patch opts to map the entire large folio at once to > minimize page faults. Additionally, redundant checks and early exits > for ARM64 MTE restoring are removed. > > Signed-off-by: Chuanhua Han > Co-developed-by: Barry Song > Signed-off-by: Barry Song > --- > mm/memory.c | 61 ++++++++++++++++++++++++++++++++++++++++++----------- > 1 file changed, 49 insertions(+), 12 deletions(-) > > diff --git a/mm/memory.c b/mm/memory.c > index 0a80e75af22c..5f52db6eb494 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -3941,6 +3941,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > pte_t pte; > vm_fault_t ret =3D 0; > void *shadow =3D NULL; > + int nr_pages =3D 1; > + unsigned long start_address =3D vmf->address; > + pte_t *start_pte =3D vmf->pte; > + bool any_swap_shared =3D false; > > if (!pte_unmap_same(vmf)) > goto out; > @@ -4131,6 +4135,30 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->addre= ss, > &vmf->ptl); > + > + /* We hit large folios in swapcache */ > + if (start_pte && folio_test_large(folio) && folio_test_swapcache(= folio)) { > + unsigned long folio_start =3D vmf->address - folio_page_i= dx(folio, page) * PAGE_SIZE; > + unsigned long folio_end =3D folio_start + folio_nr_pages(= folio) * PAGE_SIZE; > + pte_t *folio_pte =3D vmf->pte - folio_page_idx(folio, pag= e); > + int nr =3D folio_nr_pages(folio); > + > + if (unlikely(folio_start < max(vmf->address & PMD_MASK, v= ma->vm_start))) > + goto check_pte; > + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->= vm_end))) > + goto check_pte; > + > + if (swap_pte_batch(folio_pte, nr, folio->swap, &any_swap_= shared) !=3D nr) > + goto check_pte; > + > + start_address =3D folio_start; > + start_pte =3D folio_pte; > + nr_pages =3D nr; > + entry =3D folio->swap; > + page =3D &folio->page; > + } > + > +check_pte: > if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig= _pte))) > goto out_nomap; > > @@ -4184,6 +4212,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > exclusive =3D false; > } > + > + /* Reuse the whole large folio iff all entries are exclus= ive */ > + if (nr_pages > 1 && any_swap_shared) > + exclusive =3D false; > } > > /* > @@ -4198,12 +4230,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > * We're already holding a reference on the page but haven't mapp= ed it > * yet. > */ > - swap_free(entry); > + swap_free_nr(entry, nr_pages); > if (should_try_to_free_swap(folio, vma, vmf->flags)) > folio_free_swap(folio); > > - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); > - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); > + folio_ref_add(folio, nr_pages - 1); > + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); > + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); > + > pte =3D mk_pte(page, vma->vm_page_prot); > > /* > @@ -4213,33 +4247,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > * exclusivity. > */ > if (!folio_test_ksm(folio) && > - (exclusive || folio_ref_count(folio) =3D=3D 1)) { > + (exclusive || (folio_ref_count(folio) =3D=3D nr_pages && > + folio_nr_pages(folio) =3D=3D nr_pages))) { > if (vmf->flags & FAULT_FLAG_WRITE) { > pte =3D maybe_mkwrite(pte_mkdirty(pte), vma); > vmf->flags &=3D ~FAULT_FLAG_WRITE; > } > rmap_flags |=3D RMAP_EXCLUSIVE; > } > - flush_icache_page(vma, page); > + flush_icache_pages(vma, page, nr_pages); > if (pte_swp_soft_dirty(vmf->orig_pte)) > pte =3D pte_mksoft_dirty(pte); > if (pte_swp_uffd_wp(vmf->orig_pte)) > pte =3D pte_mkuffd_wp(pte); > - vmf->orig_pte =3D pte; > > /* ksm created a completely new copy */ > if (unlikely(folio !=3D swapcache && swapcache)) { > - folio_add_new_anon_rmap(folio, vma, vmf->address); > + folio_add_new_anon_rmap(folio, vma, start_address); > folio_add_lru_vma(folio, vma); > + } else if (!folio_test_anon(folio)) { > + folio_add_new_anon_rmap(folio, vma, start_address); The above two lines of code should be removed. Since we're solely addressin= g refault cases of large folios in this patchset. We're constantly dealing with anonymous mappings of large folios now. Howev= er, as we prepare to address non-refault cases of large folios swap-in, per Dav= id's suggestion in a separate thread, we'll need to extend a wrapper function, folio_add_shared_new_anon_rmap(), to accommodate non-exclusive new anonymous folios[1]. [1] https://lore.kernel.org/linux-mm/CAGsJ_4xKTj1PwmJAAZAzAvEN53kze5wSPHb01= pVg9LBy80axGA@mail.gmail.com/ > } else { > - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, > - rmap_flags); > + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, star= t_address, > + rmap_flags); > } > > VM_BUG_ON(!folio_test_anon(folio) || > (pte_write(pte) && !PageAnonExclusive(page))); > - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); > - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_p= te); > + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); > + vmf->orig_pte =3D ptep_get(vmf->pte); > + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); > > folio_unlock(folio); > if (folio !=3D swapcache && swapcache) { > @@ -4263,7 +4300,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > } > > /* No need to invalidate - it was non-present before */ > - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); > + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pag= es); > unlock: > if (vmf->pte) > pte_unmap_unlock(vmf->pte, vmf->ptl); > -- > 2.34.1 > Thanks Barry