From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f198.google.com (mail-qk0-f198.google.com [209.85.220.198]) by kanga.kvack.org (Postfix) with ESMTP id 2ACEE6B0038 for ; Tue, 16 May 2017 12:16:10 -0400 (EDT) Received: by mail-qk0-f198.google.com with SMTP id v195so65266105qka.1 for ; Tue, 16 May 2017 09:16:10 -0700 (PDT) Received: from omr2.cc.vt.edu (omr2.cc.ipv6.vt.edu. [2607:b400:92:8400:0:33:fb76:806e]) by mx.google.com with ESMTPS id m2si2338606qkc.8.2017.05.16.09.16.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 May 2017 09:16:08 -0700 (PDT) Received: from mr1.cc.vt.edu (mr1.cc.ipv6.vt.edu [IPv6:2607:b400:92:8300:0:31:1732:8aa4]) by omr2.cc.vt.edu (8.14.4/8.14.4) with ESMTP id v4GGG7QI030366 for ; Tue, 16 May 2017 12:16:07 -0400 Received: from mail-vk0-f69.google.com (mail-vk0-f69.google.com [209.85.213.69]) by mr1.cc.vt.edu (8.14.7/8.14.7) with ESMTP id v4GGG2jv005093 for ; Tue, 16 May 2017 12:16:07 -0400 Received: by mail-vk0-f69.google.com with SMTP id y190so37371500vkc.12 for ; Tue, 16 May 2017 09:16:07 -0700 (PDT) MIME-Version: 1.0 From: Sarunya Pumma Date: Tue, 16 May 2017 12:16:00 -0400 Message-ID: Subject: [PATCH] Patch for remapping pages around the fault page Content-Type: multipart/alternative; boundary="001a1143a92ad0e36d054fa67ca3" Sender: owner-linux-mm@kvack.org List-ID: To: akpm@linux-foundation.org, kirill.shutemov@linux.intel.com, jack@suse.cz, ross.zwisler@linux.intel.com, mhocko@suse.com, aneesh.kumar@linux.vnet.ibm.com, lstoakes@gmail.com, dave.jiang@intel.com Cc: linux-mm@kvack.org --001a1143a92ad0e36d054fa67ca3 Content-Type: text/plain; charset="UTF-8" After the fault handler performs the __do_fault function to read a fault page when a page fault occurs, it does not map other pages that have been read together with the fault page. This can cause a number of minor page faults to be large. Therefore, this patch is developed to remap pages around the fault page by aiming to map the pages that have been read with the fault page. The major function of this patch is the redo_fault_around function. This function computes the start and end offsets of the pages to be mapped, determines whether to do the page remapping, remaps pages using the map_pages function, and returns. In the redo_fault_around function, the start and end offsets are computed the same way as the do_fault_around function. To determine whether to do the remapping, we determine if the pages around the fault page are already mapped. If they are, the remapping will not be performed. As checking every page can be inefficient if a number of pages to be mapped is large, we have added a threshold called "vm_nr_rempping" to consider whether to check the status of every page around the fault page or just some pages. Note that the vm_nr_rempping parameter can be adjusted via the Sysctl interface. In the case that a number of pages to be mapped is smaller than the vm_nr_rempping threshold, we check all pages around the fault page (within the start and end offsets). Otherwise, we check only the adjacent pages (left and right). The page remapping is beneficial when performing the "almost sequential" page accesses, where pages are accessed in order but some pages are skipped. The following is one example scenario that we can reduce one page fault every 16 page: Assume that we want to access pages sequentially and skip every page that marked as PG_readahead. Assume that the read-ahead size is 32 pages and the number of pages to be mapped each time (fault_around_pages) is 16. When accessing a page at offset 0, a major page fault occurs, so pages from page 0 to page 31 is read from the disk to the page cache. With this, page 24 is marked as a read-ahead page (PG_readahead). Then only page 0 is mapped to the virtual memory space. When accessing a page at offset 1, a minor page fault occurs, pages from page 0 to page 15 will be mapped. We keep accessing pages until page 31. Note that we skip page 24. When accessing a page at offset 32, a major page fault occurs. The same process will be repeated. The other 32 pages will be read from the disk. Only page 32 is mapped. Then a minor page fault at the next page (page 33) will occur. >>From this example, two page faults occur every 16 page. With this patch, we can eliminate the minor page fault in every 16 page. Thank you very much for your time for reviewing the patch. Signed-off-by: Sarunya Pumma --- include/linux/mm.h | 2 ++ kernel/sysctl.c | 8 +++++ mm/memory.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7cb17c6..2d533a3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -34,6 +34,8 @@ struct bdi_writeback; void init_mm_internals(void); +extern unsigned long vm_nr_remapping; + #ifndef CONFIG_NEED_MULTIPLE_NODES /* Don't use mapnrs, do it properly */ extern unsigned long max_mapnr; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 4dfba1a..16c7efe 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1332,6 +1332,14 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, .extra2 = &one_hundred, }, + { + .procname = "nr_remapping", + .data = &vm_nr_remapping, + .maxlen = sizeof(vm_nr_remapping), + .mode = 0644, + .proc_handler = proc_doulongvec_minmax, + .extra1 = &zero, + }, #ifdef CONFIG_HUGETLB_PAGE { .procname = "nr_hugepages", diff --git a/mm/memory.c b/mm/memory.c index 6ff5d72..3d0dca9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -83,6 +83,9 @@ #warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid. #endif +/* A preset threshold for considering page remapping */ +unsigned long vm_nr_remapping = 32; + #ifndef CONFIG_NEED_MULTIPLE_NODES /* use the per-pgdat data instead for discontigmem - mbligh */ unsigned long max_mapnr; @@ -3374,6 +3377,82 @@ static int do_fault_around(struct vm_fault *vmf) return ret; } +static int redo_fault_around(struct vm_fault *vmf) +{ + unsigned long address = vmf->address, nr_pages, mask; + pgoff_t start_pgoff = vmf->pgoff; + pgoff_t end_pgoff; + pte_t *lpte, *rpte; + int off, ret = 0, is_mapped = 0; + + nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; + mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK; + + vmf->address = max(address & mask, vmf->vma->vm_start); + off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1); + start_pgoff -= off; + + /* + * end_pgoff is either end of page table or end of vma + * or fault_around_pages() from start_pgoff, depending what is nearest. + */ + end_pgoff = start_pgoff - + ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) + + PTRS_PER_PTE - 1; + end_pgoff = min3(end_pgoff, vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1, + start_pgoff + nr_pages - 1); + + if (nr_pages < vm_nr_remapping) { + int i, start_off = 0, end_off = 0; + + lpte = vmf->pte - off; + for (i = 0; i < nr_pages; i++) { + if (!pte_none(*lpte)) { + is_mapped++; + } else { + if (!start_off) + start_off = i; + end_off = i; + } + lpte++; + } + if (is_mapped != nr_pages) { + is_mapped = 0; + end_pgoff = start_pgoff + end_off; + start_pgoff += start_off; + vmf->pte += start_off; + } + lpte = NULL; + } else { + lpte = vmf->pte - 1; + rpte = vmf->pte + 1; + if (!pte_none(*lpte) && !pte_none(*rpte)) + is_mapped = 1; + lpte = NULL; + rpte = NULL; + } + + if (!is_mapped) { + vmf->pte -= off; + vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); + vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT); + } + + /* Huge page is mapped? Page fault is solved */ + if (pmd_trans_huge(*vmf->pmd)) { + ret = VM_FAULT_NOPAGE; + goto out; + } + + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); + +out: + vmf->address = address; + vmf->pte = NULL; + return ret; +} + static int do_read_fault(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -3394,6 +3473,17 @@ static int do_read_fault(struct vm_fault *vmf) if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; + /* + * Remap pages after read + */ + if (!(vma->vm_flags & VM_RAND_READ) && vma->vm_ops->map_pages + && fault_around_bytes >> PAGE_SHIFT > 1) { + ret |= alloc_set_pte(vmf, vmf->memcg, vmf->page); + unlock_page(vmf->page); + redo_fault_around(vmf); + return ret; + } + ret |= finish_fault(vmf); unlock_page(vmf->page); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) -- 2.7.4 --001a1143a92ad0e36d054fa67ca3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
After the fault handler performs the __do_fault funct= ion to read a fault
page when a page fault occurs, it do= es not map other pages that have been
read together with the faul= t page. This can cause a number of minor page
faults to be large.= Therefore, this patch is developed to remap pages
around the fau= lt page by aiming to map the pages that have been read
with the f= ault page.

The major function of this patch is the= redo_fault_around function. This
function computes the start and= end offsets of the pages to be mapped,
determines whether to do = the page remapping, remaps pages using the
map_pages function, an= d returns. In the redo_fault_around function, the
start and end o= ffsets are computed the same way as the do_fault_around
function.= To determine whether to do the remapping, we determine if the
pa= ges around the fault page are already mapped. If they are, the remapping
will not be performed.=C2=A0

As checking e= very page can be inefficient if a number of pages to be mapped
is= large, we have added a threshold called "vm_nr_rempping" to cons= ider
whether to check the status of every page around the fault p= age or just
some pages. Note that the vm_nr_rempping parameter ca= n be adjusted via the
Sysctl interface. In the case that a number= of pages to be mapped is
smaller than the vm_nr_rempping thresho= ld, we check all pages around the
fault page (within the start an= d end offsets). Otherwise, we check only the
adjacent pages (left= and right).=C2=A0

The page remapping is beneficia= l when performing the "almost sequential"
page accesses= , where pages are accessed in order but some pages are
skipped.

The following is one example scenario that we can r= educe one page fault
every 16 page:

Assu= me that we want to access pages sequentially and skip every page that
=
marked as PG_readahead. Assume that the read-ahead size is 32 pages an= d the
number of pages to be mapped each time (fault_around_pages)= is 16.

When accessing a page at offset 0, a major= page fault occurs, so pages from
page 0 to page 31 is read from = the disk to the page cache. With this, page
24 is marked as a rea= d-ahead page (PG_readahead). Then only page 0 is
mapped to the vi= rtual memory space.

When accessing a page at offse= t 1, a minor page fault occurs, pages from
page 0 to page 15 will= be mapped.=C2=A0

We keep accessing pages until pa= ge 31. Note that we skip page 24.

When accessing a= page at offset 32, a major page fault occurs.=C2=A0 The same
pro= cess will be repeated. The other 32 pages will be read from the disk.
=
Only page 32 is mapped. Then a minor page fault at the next page (page=
33) will occur.=C2=A0

From this example= , two page faults occur every 16 page. With this patch, we=C2=A0
= can eliminate the minor page fault in every 16 page.=C2=A0

Thank you very much for your time for reviewing the patch.

Signed-off-by: Sarunya Pumma <sarunya@vt.edu>
---
=C2=A0i= nclude/linux/mm.h | =C2=A02 ++
=C2=A0kernel/sysctl.c =C2=A0 =C2= =A0| =C2=A08 +++++
=C2=A0mm/memory.c =C2=A0 =C2=A0 =C2=A0 =C2=A0|= 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
=C2=A0= 3 files changed, 100 insertions(+)

diff --git a/in= clude/linux/mm.h b/include/linux/mm.h
index 7cb17c6..2d533a3 1006= 44
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -34,6 +34,8 @@ struct bdi_writeback;
=C2=A0
= =C2=A0void init_mm_internals(void);
=C2=A0
+extern unsi= gned long vm_nr_remapping;
+
=C2=A0#ifndef CONFIG_NEED_= MULTIPLE_NODES /* Don't use mapnrs, do it properly */
=C2=A0extern= unsigned long max_mapnr;
=C2=A0
diff --git a/kernel/sy= sctl.c b/kernel/sysctl.c
index 4dfba1a..16c7efe 100644
= --- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1332,6 = +1332,14 @@ static struct ctl_table vm_table[] =3D {
=C2=A0 .extra1 =3D &= amp;zero,
=C2=A0 .extra2 =3D &one_hundred,
=C2=A0 },
+= {
+ <= /span>.procname =3D "nr_remapping",
+ .data =3D &vm_nr_remapping= ,
+ .maxlen =3D sizeof(vm_nr_remapping),
+ .mode =3D 0644,
+<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> .pro= c_handler =3D proc_doulongvec_minmax,
+ .extra1 =3D &zero,
+ },
=
=C2=A0#ifdef CONFIG_HUGETLB_PAGE
=C2=A0 {
=C2=A0 .procname<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> =3D &= quot;nr_hugepages",
diff --git a/mm/memory.c b/mm/memory.c
index 6ff5d72..3d0dca9 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -83,6 +83,9 @@
=C2=A0#warning Un= fortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupi= d.
=C2=A0#endif
=C2=A0
+/* A preset threshold= for considering page remapping */
+unsigned long vm_nr_remapping= =3D 32;
+
=C2=A0#ifndef CONFIG_NEED_MULTIPLE_NODES
=C2=A0/* use the per-pgdat data instead for discontigmem - mbligh */=
=C2=A0unsigned long max_mapnr;
@@ -3374,6 +3377,82 @@ = static int do_fault_around(struct vm_fault *vmf)
=C2=A0 return ret;
=C2=A0}
=C2=A0
+static int redo_fault_around(st= ruct vm_fault *vmf)
+{
+ unsigned long address =3D vmf->a= ddress, nr_pages, mask;
+ pgoff_t start_pgoff =3D vmf->pgoff;
=
+ pgoff_t end_pgoff;
+ pte_t *lpte, *rpte;
+ int off, ret =3D 0,= is_mapped =3D 0;
+
+ nr_pages =3D READ_ONCE(fault_around_by= tes) >> PAGE_SHIFT;
+ mask =3D ~(nr_pages * PAGE_SIZE - 1) &= PAGE_MASK;
+
+ vmf->address =3D max(address & mask, = vmf->vma->vm_start);
+ off =3D ((address - vmf->address) >= > PAGE_SHIFT) & (PTRS_PER_PTE - 1);
+ start_pgoff -=3D off;
+
+ /*
+ * =C2=A0end_pgoff is either end of page table or= end of vma
+ * =C2=A0or fault_around_pages() from start_pgoff, depen= ding what is nearest.
+ */
+ end_pgoff =3D start_pgoff -
+ ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
+ PTRS_PER_PTE - 1;
+ end_pgoff =3D min3(end_pgoff, vma_pages(vmf->v= ma) + vmf->vma->vm_pgoff - 1,
+ start_pgoff + nr_pages - 1);
+
+ if (nr_pages < vm_nr_remapping) {
+ int i, start= _off =3D 0, end_off =3D 0;
+
+ lpte =3D vmf->pte - off;<= /div>
+ = for (i =3D 0; i < nr_pages; i++) {
+ if (!pte_none(*lpte)= ) {
+ is_mapped++;
+ } else {
+ if (!start_off)
+ = start_off =3D i;
+ end_off =3D i;
+ }
+ lpte++= ;
+ }
+ if (is_mapped !=3D nr_pages) {
+ is_mapped =3D= 0;
+ end_pgoff =3D start_pgoff + end_off;
+ start_pgoff +=3D= start_off;
+ vmf->pte +=3D start_off;
+ }
+ lpte =3D= NULL;
+ } else {
+ lpte =3D vmf->pte - 1;
+ rpte =3D v= mf->pte + 1;
+ if (!pte_none(*lpte) && !pte_none(*rpte))
+ = is_mapped =3D 1;
+ lpte =3D NULL;
+ rpte =3D NULL;
+ }
+
+ if (!is_mapped) {
+ vmf->pte -=3D off;
+ vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
+ vmf->pte -=3D (vmf->address >> PAGE_SHIFT) - (address >= > PAGE_SHIFT);
+ }
+
+ /* Huge page is mapped? Page = fault is solved */
+ if (pmd_trans_huge(*vmf->pmd)) {
+ ret = =3D VM_FAULT_NOPAGE;
+ goto out;
+ }
+
+ if (vmf= ->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+
+out:
+ vmf->address =3D address;
+ vmf->pte =3D= NULL;
+ return ret;
+}
+
=C2=A0static i= nt do_read_fault(struct vm_fault *vmf)
=C2=A0{
=C2=A0 struct= vm_area_struct *vma =3D vmf->vma;
@@ -3394,6 +3473,17 @@ stat= ic int do_read_fault(struct vm_fault *vmf)
=C2=A0 if (unlikely(ret &am= p; (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
=C2=A0 retur= n ret;
=C2=A0
+ /*
+ * Remap pages after read
+ = */
+ if (!(vma->vm_flags & VM_RAND_READ) && vma->vm_= ops->map_pages
+ && fault_around_bytes >> PAGE_SHIF= T > 1) {
+ ret |=3D alloc_set_pte(vmf, vmf->memcg, vmf->page= );
+ unlock_page(vmf->page);
+ redo_fault_around(vmf);
=
+ return ret;
+ }
+
=C2=A0 ret |=3D finish_fault(vmf);=
=C2=A0 unlock_page(vmf->page);
=C2=A0 if (unlikely(ret & = (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
--=C2=A0
2.7.4

--001a1143a92ad0e36d054fa67ca3-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org