From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89C9BC001B0 for ; Wed, 26 Jul 2023 16:41:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B40EE6B0072; Wed, 26 Jul 2023 12:41:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC9BC6B0078; Wed, 26 Jul 2023 12:41:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96B3A8D0001; Wed, 26 Jul 2023 12:41:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 83EC36B0072 for ; Wed, 26 Jul 2023 12:41:47 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4A4FA140F84 for ; Wed, 26 Jul 2023 16:41:47 +0000 (UTC) X-FDA: 81054329454.13.09694C2 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf27.hostedemail.com (Postfix) with ESMTP id 58ACD40008 for ; Wed, 26 Jul 2023 16:41:45 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="WRndJCr/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690389705; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hKDmdFZl24E6t2k90aurDDVVQw5NfuveAfUgdE78zMY=; b=CY+cG4HlXzOeKsfnZB+MAUB1LK6fDz2vwkxDfWcB1h7Vub/K6h6wPwXg4Jx/iH5kl/771Y Ia3sapBdJ9gs4PCPOaHCGAktEuR3sfbSZO0NFnUcAwnXZP3DiDU2PyDfFURiqH/gc2umLM uh69t/qydd6jjC7uFM3EhtrIY48jzNQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="WRndJCr/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690389705; a=rsa-sha256; cv=none; b=DgdcHYhKi8Ncqh1ok1LLHNjLRhszGZvsgl7aAc7oZhOyz4tIe3Dpka3nIv5kZeHTkIACJZ fUtCtOIMrCpq8Z/fmkgFNE4+RXDa2iF1U3CJjNEpgpo29ZOr/Wb+qBlKyThLoPOjDDID16 4UBwvtpIKJgZyFPXGZFDsgx2As93A5M= Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-4036bd4fff1so5291cf.0 for ; Wed, 26 Jul 2023 09:41:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690389704; x=1690994504; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=hKDmdFZl24E6t2k90aurDDVVQw5NfuveAfUgdE78zMY=; b=WRndJCr/3a7whMQmH0MlY8yT617iIDdCmZTTEkAsNFAABHQixWPn3U6wxBr+E6gZHG 5aogzcVq3gUTHNbZ4+ardyPrmNC6ldWFHhd9LCjnWxnQsOTh5q0aIfbgz3b11kbrhATK j8xewnETONG1ejwTxJ5aQT7pRepGVr1W/bTmmec58wYF/5nICQ59kiRCbdvlSG5PnlPe H6PzXbUSYxChAtriNqUATHBcbupT+Fj1HpYIuFAZt4JoUIg+Y5wsUPBfHiXEV0jLTP+b 6Z2fc5BSB9nveT46/TBIDRXesNVz2um6ezKuWo+ijsznx9CpaGqT3QxfvWFXG2rJ1LLI v5ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690389704; x=1690994504; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hKDmdFZl24E6t2k90aurDDVVQw5NfuveAfUgdE78zMY=; b=CswKW5zBv9ktgybRJSvhu+L42+NFoIW3Byr/6c3neaOZ/ploaInf6EWJY20wH/uDCL 6mxyLIz3bKkK37/HyNBH7U7BM+cAhWlokcIFtrrURK1DtVSVLL5TdDTV6c/bDEUlEAYQ brIlIMxGLuLUw5p9FkAYrUTEvTaqt6XpoG2jQqiHF8rYMtbTxRI9Ox4e74aiX4qryqHN ZBBjOJt14qACSumFYZnA8UpQqAS6+zV5OUdydJDxz9QReOH0CRyeTrBC8UxxrJRtVie6 tJslOaS/8ArX723t3/dqb7EkMQbTqJEfdRPN81dj0cFScYbRksww+xnsuZtCfwniHD3X EGYg== X-Gm-Message-State: ABy/qLb2AiI7WbGpnuquWzU99oRdyvkEuEsqf0EFc68z9pK2SaZZam4a JFHWgv+FxzG1zkgro0uBtYoq7Mb0rG5g29MDJ0VZlQ== X-Google-Smtp-Source: APBJJlGZwBdw2fRBhstZqLkoktLqTYSLAfKSGe75PQ77m8UDQ2vdgvi6bSlj4FIxTr3E69hARiUE42g1WoxbzBhNjL0= X-Received: by 2002:ac8:5805:0:b0:403:e1d1:8b63 with SMTP id g5-20020ac85805000000b00403e1d18b63mr447403qtg.24.1690389704256; Wed, 26 Jul 2023 09:41:44 -0700 (PDT) MIME-Version: 1.0 References: <20230726095146.2826796-1-ryan.roberts@arm.com> <20230726095146.2826796-3-ryan.roberts@arm.com> In-Reply-To: <20230726095146.2826796-3-ryan.roberts@arm.com> From: Yu Zhao Date: Wed, 26 Jul 2023 10:41:07 -0600 Message-ID: Subject: Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance To: Ryan Roberts Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 58ACD40008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 665f9h1r39adocguudzux6zyqskmh86g X-HE-Tag: 1690389705-652742 X-HE-Meta: U2FsdGVkX19xdK4tLcePFhwpXv6n+vS34HmWxfgtJuns9EDzI/LCE36PHb1xJPPtnwCeptbUfGrgVoWIp3JeQ0w03eDgFw71zoo1Z3mVxwx+y75LMA4xj9jGOFwvhklutqR4/JXLSAgJn+8dwXJwN7oQAnmEPeJOTrai7Z/Hcv9ny35TyBozGRxoVFA1ddoeMu5fhiaWsJ0XIDpYu/mqyOc9PfPu1o4F8Px7+9j0fEzwUxi6bx3D4plOac81cGt+NPp/2S3HbgkJ/7H5eYFksRH4iOKkkor9wsi0i1b8uTgFk+SOWkTd46hYRnOGdjVzEOT/BxkzAfN2JyErxhyRdz8aod4WvDM5uARQwXw0dG9mlCxbo8BWi/vVCiCS6kINf5yboiDjQvwdUb2xid7rigr8LKElQjInK9y0hHn5JzaWYJhx4PNLSUKYagQj0nH9xYOaHDxTeZiN5OwdJVRV7e/BqpVzAOZssp2SEL1bAjqrQ4r6v2mon3GqtkiNd+tzl/xkJ6ohcXT5H6QK4TNb0nU2bWw1SE+R2/y9R0XL2Xr0O9LI/k2ZAccdyjOUG9mxFM7j9Gzg/5xRiKdJ4ukT+AGELmS0LDf8cVgtYd/gTz2qhv5DDJ/NARu9sYYDxNUkUHw1/xhWwGqMity0wrg/2PQ215R8gotP9O9Y2RrFIdEqrg8mJHgPldqDo+TJgKIlHJlwWJJ6H35Dj/pus4VJ5tEByCm2L8MY6loF0pyPfUmrgJzAkahvhFkT8lo/Srr+TIsa2AVI9IHSAhtAZuuV++ttp4UQiIphFh3S/1xy/VS2p0Oxf5Lvk12yvHMZpe7DSsLFK35ZLd2clLEBeOeBODLKUeU9ZCe5QI/bF2LIixKzmm4ngwLHKCgym7KDf44AV992dV04gmFSxWkxeSV9F2yAvkP9QXcLT15jdlpEYGvV6Q1KdiBv7d70gqrFRuH5qKBMCmNgOfTLuI9q91E EJHV7+AZ EWHOsKVotlFTIYM4vDSXfdnLBQ+zzlfeC18g2sjX4lkqrkrBS8wdRmH9a8j05ASL1IPxGTKzDfy6UH3RdV9x0sM3ZPQ5BMvnWMH39JwxDiti6/pQWJWglEcFHyE8LHRoaf9JMoHnp2PcwQPb39HeZpv0hldPvoGmL+c8xvMjqj3m90owEEgvFwqJb0k0122H6Qshb1k4KskO0UpQhrse5huAY1kTjZcYmVGjROEALUZR8JquXzPlRP11rdrmGONhe0g91gwMMcGPAlYUk+UCmRjE++3v7Rg0H+hZuuvdA0lIJKkEAWxqx3OI1kWGqks/zSzvtqihTXAB9PRymmmdMI3eqLCdjZyrbkWFW8oWjcvetSmQ1fpXbBo/1d2E1NErWVwFF+GBVojpKy11bIS5bIQ9mfg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 26, 2023 at 3:52=E2=80=AFAM Ryan Roberts = wrote: > > Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be > allocated in large folios of a determined order. All pages of the large > folio are pte-mapped during the same page fault, significantly reducing > the number of page faults. The number of per-page operations (e.g. ref > counting, rmap management lru list management) are also significantly > reduced since those ops now become per-folio. > > The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, > which defaults to disabled for now; The long term aim is for this to > defaut to enabled, but there are some risks around internal > fragmentation that need to be better understood first. > > When enabled, the folio order is determined as such: For a vma, process > or system that has explicitly disabled THP, we continue to allocate > order-0. THP is most likely disabled to avoid any possible internal > fragmentation so we honour that request. > > Otherwise, the return value of arch_wants_pte_order() is used. For vmas > that have not explicitly opted-in to use transparent hugepages (e.g. > where thp=3Dmadvise and the vma does not have MADV_HUGEPAGE), then > arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is > bigger). This allows for a performance boost without requiring any > explicit opt-in from the workload while limitting internal > fragmentation. > > If the preferred order can't be used (e.g. because the folio would > breach the bounds of the vma, or because ptes in the region are already > mapped) then we fall back to a suitable lower order; first > PAGE_ALLOC_COSTLY_ORDER, then order-0. > > arch_wants_pte_order() can be overridden by the architecture if desired. > Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous > set of ptes map physically contigious, naturally aligned memory, so this > mechanism allows the architecture to optimize as required. > > Here we add the default implementation of arch_wants_pte_order(), used > when the architecture does not define it, which returns -1, implying > that the HW has no preference. In this case, mm will choose it's own > default order. > > Signed-off-by: Ryan Roberts > --- > include/linux/pgtable.h | 13 ++++ > mm/Kconfig | 10 +++ > mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- > 3 files changed, 172 insertions(+), 17 deletions(-) > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index 5063b482e34f..2a1d83775837 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) > } > #endif > > +#ifndef arch_wants_pte_order > +/* > + * Returns preferred folio order for pte-mapped memory. Must be in range= [0, > + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires larg= e folios > + * to be at least order-2. Negative value implies that the HW has no pre= ference > + * and mm will choose it's own default order. > + */ > +static inline int arch_wants_pte_order(void) > +{ > + return -1; > +} > +#endif > + > #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR > static inline pte_t ptep_get_and_clear(struct mm_struct *mm, > unsigned long address, > diff --git a/mm/Kconfig b/mm/Kconfig > index 09130434e30d..fa61ea160447 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA > > source "mm/damon/Kconfig" > > +config LARGE_ANON_FOLIO > + bool "Allocate large folios for anonymous memory" > + depends on TRANSPARENT_HUGEPAGE > + default n > + help > + Use large (bigger than order-0) folios to back anonymous memory= where > + possible, even for pte-mapped memory. This reduces the number o= f page > + faults, as well as other per-page overheads to improve performa= nce for > + many workloads. > + > endmenu > diff --git a/mm/memory.c b/mm/memory.c > index 01f39e8144ef..64c3f242c49a 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > return ret; > } > > +static bool vmf_pte_range_changed(struct vm_fault *vmf, int nr_pages) > +{ > + int i; > + > + if (nr_pages =3D=3D 1) > + return vmf_pte_changed(vmf); > + > + for (i =3D 0; i < nr_pages; i++) { > + if (!pte_none(ptep_get_lockless(vmf->pte + i))) > + return true; > + } > + > + return false; > +} > + > +#ifdef CONFIG_LARGE_ANON_FOLIO > +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ > + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SH= IFT) > + > +static int anon_folio_order(struct vm_area_struct *vma) > +{ > + int order; > + > + /* > + * If THP is explicitly disabled for either the vma, the process = or the > + * system, then this is very likely intended to limit internal > + * fragmentation; in this case, don't attempt to allocate a large > + * anonymous folio. > + * > + * Else, if the vma is eligible for thp, allocate a large folio o= f the > + * size preferred by the arch. Or if the arch requested a very sm= all > + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDE= R, > + * which still meets the arch's requirements but means we still t= ake > + * advantage of SW optimizations (e.g. fewer page faults). > + * > + * Finally if thp is enabled but the vma isn't eligible, take the > + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHIN= TED. > + * This ensures workloads that have not explicitly opted-in take = benefit > + * while capping the potential for internal fragmentation. > + */ What empirical evidence is SZ_64K based on? What workloads would benefit from it? How much would they benefit from it? Would they benefit more or less from different values? How much internal fragmentation would it cause? What cost function was used to arrive at the conclusion that its benefits outweigh its costs? > + if ((vma->vm_flags & VM_NOHUGEPAGE) || > + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || > + !hugepage_flags_enabled()) > + order =3D 0; > + else { > + order =3D max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_O= RDER); > + > + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, = true)) > + order =3D min(order, ANON_FOLIO_MAX_ORDER_UNHINTE= D); > + } > + > + return order; > +} > + > +static int alloc_anon_folio(struct vm_fault *vmf, struct folio **folio) static struct folio *alloc_anon_folio(struct vm_fault *vmf) and use ERR_PTR() and its friends. > +{ > + int i; > + gfp_t gfp; > + pte_t *pte; > + unsigned long addr; > + struct vm_area_struct *vma =3D vmf->vma; > + int prefer =3D anon_folio_order(vma); > + int orders[] =3D { > + prefer, > + prefer > PAGE_ALLOC_COSTLY_ORDER ? PAGE_ALLOC_COSTLY_ORDE= R : 0, > + 0, > + }; > + > + *folio =3D NULL; > + > + if (vmf_orig_pte_uffd_wp(vmf)) > + goto fallback; > + > + for (i =3D 0; orders[i]; i++) { > + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i])= ; > + if (addr >=3D vma->vm_start && > + addr + (PAGE_SIZE << orders[i]) <=3D vma->vm_end) > + break; > + } > + > + if (!orders[i]) > + goto fallback; > + > + pte =3D pte_offset_map(vmf->pmd, vmf->address & PMD_MASK); > + if (!pte) > + return -EAGAIN; > + > + for (; orders[i]; i++) { > + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i])= ; > + vmf->pte =3D pte + pte_index(addr); > + if (!vmf_pte_range_changed(vmf, 1 << orders[i])) > + break; > + } > + > + vmf->pte =3D NULL; > + pte_unmap(pte); > + > + gfp =3D vma_thp_gfp_mask(vma); > + > + for (; orders[i]; i++) { > + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << orders[i])= ; > + *folio =3D vma_alloc_folio(gfp, orders[i], vma, addr, tru= e); > + if (*folio) { > + clear_huge_page(&(*folio)->page, addr, 1 << order= s[i]); > + return 0; > + } > + } > + > +fallback: > + *folio =3D vma_alloc_zeroed_movable_folio(vma, vmf->address); > + return *folio ? 0 : -ENOMEM; > +} > +#else > +static inline int alloc_anon_folio(struct vm_fault *vmf, struct folio **= folio) > +{ > + *folio =3D vma_alloc_zeroed_movable_folio(vmf->vma, vmf->address)= ; > + return *folio ? 0 : -ENOMEM; > +} > +#endif > + > /* > * We enter with non-exclusive mmap_lock (to exclude vma changes, > * but allow concurrent faults), and pte mapped but not yet locked. > @@ -4057,6 +4178,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > { > + int i =3D 0; > + int nr_pages =3D 1; > + unsigned long addr =3D vmf->address; > bool uffd_wp =3D vmf_orig_pte_uffd_wp(vmf); > struct vm_area_struct *vma =3D vmf->vma; > struct folio *folio; > @@ -4101,10 +4225,15 @@ static vm_fault_t do_anonymous_page(struct vm_fau= lt *vmf) > /* Allocate our own private page. */ > if (unlikely(anon_vma_prepare(vma))) > goto oom; > - folio =3D vma_alloc_zeroed_movable_folio(vma, vmf->address); > + ret =3D alloc_anon_folio(vmf, &folio); > + if (unlikely(ret =3D=3D -EAGAIN)) > + return 0; > if (!folio) > goto oom; > > + nr_pages =3D folio_nr_pages(folio); > + addr =3D ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); > + > if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) > goto oom_free_page; > folio_throttle_swaprate(folio, GFP_KERNEL); > @@ -4116,17 +4245,12 @@ static vm_fault_t do_anonymous_page(struct vm_fau= lt *vmf) > */ > __folio_mark_uptodate(folio); > > - entry =3D mk_pte(&folio->page, vma->vm_page_prot); > - entry =3D pte_sw_mkyoung(entry); > - if (vma->vm_flags & VM_WRITE) > - entry =3D pte_mkwrite(pte_mkdirty(entry)); > - > - vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->addre= ss, > - &vmf->ptl); > + vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf= ->ptl); > if (!vmf->pte) > goto release; > - if (vmf_pte_changed(vmf)) { > - update_mmu_tlb(vma, vmf->address, vmf->pte); > + if (vmf_pte_range_changed(vmf, nr_pages)) { > + for (i =3D 0; i < nr_pages; i++) > + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pt= e + i); > goto release; > } > > @@ -4141,16 +4265,24 @@ static vm_fault_t do_anonymous_page(struct vm_fau= lt *vmf) > return handle_userfault(vmf, VM_UFFD_MISSING); > } > > - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); > - folio_add_new_anon_rmap(folio, vma, vmf->address); > + folio_ref_add(folio, nr_pages - 1); > + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); > + folio_add_new_anon_rmap(folio, vma, addr); > folio_add_lru_vma(folio, vma); > + > + for (i =3D 0; i < nr_pages; i++) { > + entry =3D mk_pte(folio_page(folio, i), vma->vm_page_prot)= ; > + entry =3D pte_sw_mkyoung(entry); > + if (vma->vm_flags & VM_WRITE) > + entry =3D pte_mkwrite(pte_mkdirty(entry)); > setpte: > - if (uffd_wp) > - entry =3D pte_mkuffd_wp(entry); > - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); > + if (uffd_wp) > + entry =3D pte_mkuffd_wp(entry); > + set_pte_at(vma->vm_mm, addr + PAGE_SIZE * i, vmf->pte + i= , entry); > > - /* No need to invalidate - it was non-present before */ > - update_mmu_cache(vma, vmf->address, vmf->pte); > + /* No need to invalidate - it was non-present before */ > + update_mmu_cache(vma, addr + PAGE_SIZE * i, vmf->pte + i)= ; > + } > unlock: > if (vmf->pte) > pte_unmap_unlock(vmf->pte, vmf->ptl); The rest looks good to me.