mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Naresh Kamboju]

While running LTP mm thp01 test case on i386 kernel running on x86_64 device
the following kernel warning was noticed multiple times.

This issue is not new,
we have noticed on stable-rc 5.4, stable-rc 5.5 and stable-rc 5.6 branches.
FYI, CONFIG_HAVE_MOVE_PMD=y is set on
and total memory 2.2G as per free output.

steps to reproduce:
--------------------
boot i386 kernel on x86_64 device,
cd /opt/ltp
./runltp -f mm
thp01.c:98: PASS: system didn't crash.
thp01.c:98: PASS: system didn't crash.
thp01.c:98: PASS: system didn't crash.

[  207.317499] ------------[ cut here ]------------
[  207.322153] WARNING: CPU: 0 PID: 18963 at mm/mremap.c:211
move_page_tables+0x5b0/0x5d0
[  207.330061] Modules linked in: x86_pkg_temp_thermal
[  207.334940] CPU: 0 PID: 18963 Comm: true Tainted: G        W
 5.6.2-rc1+ #1
[  207.342498] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.2 05/23/2018
[  207.349881] EIP: move_page_tables+0x5b0/0x5d0
[  207.354233] Code: 00 00 c0 ff 2b 45 08 39 c3 0f 46 c3 89 45 d4 01
f8 89 45 cc e9 7e fb ff ff 8d 45 d8 83 4d e8 01 e8 65 b0 01 00 e9 b2
fa ff ff <0f> 0b 80 7d be 00 0f 84 7e fd ff ff 31 db e9 74 fe ff ff 31
db e9
[  207.372969] EAX: 7ce5f067 EBX: 00400000 ECX: e2cc8000 EDX: 00000000
[  207.379225] ESI: e2cc8bfc EDI: bfc00000 EBP: f3273e18 ESP: f3273dc0
[  207.385484] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010202
[  207.392261] CR0: 80050033 CR2: b7d02f50 CR3: 22cc8000 CR4: 003406d0
[  207.398517] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  207.404774] DR6: fffe0ff0 DR7: 00000400
[  207.408605] Call Trace:
[  207.411053]  setup_arg_pages+0x22b/0x310
[  207.414977]  ? security_bprm_committed_creds+0x22/0x30
[  207.420107]  load_elf_binary+0x2fb/0x10a0
[  207.424110]  ? selinux_inode_permission+0xfb/0x1d0
[  207.428894]  ? bm_status_write+0x61/0xa0
[  207.432811]  ? security_inode_permission+0x2c/0x50
[  207.437597]  ? writenote+0xb0/0xb0
[  207.440992]  search_binary_handler+0x77/0x190
[  207.445356]  __do_execve_file+0x429/0x760
[  207.449375]  sys_execve+0x21/0x30
[  207.452693]  do_fast_syscall_32+0x7a/0x280
[  207.456784]  entry_SYSENTER_32+0xa5/0xf8
[  207.460702] EIP: 0xb7ee7c5d
[  207.463491] Code: Bad RIP value.
[  207.466716] EAX: ffffffda EBX: bfff9ed0 ECX: 08069420 EDX: bfffa134
[  207.472971] ESI: 080599d4 EDI: bfff9ed9 EBP: bfff9f78 ESP: bfff9ea8
[  207.479230] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000296
[  207.486008] ---[ end trace d49b75932d5396d5 ]---

full test log,
https://lkft.validation.linaro.org/scheduler/job/1328795#L14498
https://lkft.validation.linaro.org/scheduler/job/1331455#L8923
https://lkft.validation.linaro.org/scheduler/job/1331632#L17251

Kernel config:
https://builds.tuxbuild.com/RJ9BGpsgfPfj3Sfje8oLSw/kernel.config

Test case source:
https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/mem/thp/thp01.c
-- 
Linaro LKFT
https://lkft.linaro.org

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Kirill A. Shutemov]

On Thu, Apr 02, 2020 at 04:49:02PM +0530, Naresh Kamboju wrote:
> While running LTP mm thp01 test case on i386 kernel running on x86_64 device
> the following kernel warning was noticed multiple times.
> 
> This issue is not new,
> we have noticed on stable-rc 5.4, stable-rc 5.5 and stable-rc 5.6 branches.
> FYI, CONFIG_HAVE_MOVE_PMD=y is set on
> and total memory 2.2G as per free output.
> 
> steps to reproduce:
> --------------------
> boot i386 kernel on x86_64 device,
> cd /opt/ltp
> ./runltp -f mm
> thp01.c:98: PASS: system didn't crash.
> thp01.c:98: PASS: system didn't crash.
> thp01.c:98: PASS: system didn't crash.
> 
> [  207.317499] ------------[ cut here ]------------
> [  207.322153] WARNING: CPU: 0 PID: 18963 at mm/mremap.c:211
> move_page_tables+0x5b0/0x5d0


> Kernel config:
> https://builds.tuxbuild.com/RJ9BGpsgfPfj3Sfje8oLSw/kernel.config

Interesting. I suspect it's related to 2-level page tables in this
configuration. But I cannot immediately see how.

Could you test if enabling HIGHMEM64 fixes the issue?

Below is patch that prints some additional info:

diff --git a/mm/mremap.c b/mm/mremap.c
index d28f08a36b96..065d5ec3614a 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -208,8 +208,14 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 	 * The destination pmd shouldn't be established, free_pgtables()
 	 * should have release it.
 	 */
-	if (WARN_ON(!pmd_none(*new_pmd)))
+	if (WARN_ON(!pmd_none(*new_pmd))) {
+		dump_vma(vma);
+		printk("old_addr: %#lx, new_addr: %#lx, old_end: %#lx\n",
+				old_addr, new_addr, old_end);
+		printk("old_pmd: %#lx", pmd_val(*old_pmd));
+		printk("new_pmd: %#lx", pmd_val(*new_pmd));
 		return false;
+	}
 
 	/*
 	 * We don't have to worry about the ordering of src and dst
-- 
 Kirill A. Shutemov

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Naresh Kamboju]

On Thu, 2 Apr 2020 at 19:08, Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> On Thu, Apr 02, 2020 at 04:49:02PM +0530, Naresh Kamboju wrote:
> > While running LTP mm thp01 test case on i386 kernel running on x86_64 device
> > the following kernel warning was noticed multiple times.
> >
>
> Interesting. I suspect it's related to 2-level page tables in this
> configuration. But I cannot immediately see how.
>
> Could you test if enabling HIGHMEM64 fixes the issue?

CONFIG_HIGHMEM64G=y

>
> Below is patch that prints some additional info:

Applied your patch and reproduced the problem,
The boot log show this warning,

[    0.261879] ************************************************************
[    0.261880] ** WARNING! WARNING! WARNING! WARNING! WARNING! WARNING!  **
[    0.261881] **                                                        **
[    0.261881] ** You are using 32-bit PTI on a 64-bit PCID-capable CPU. **
[    0.261882] ** Your performance will increase dramatically if you     **
[    0.261882] ** switch to a 64-bit kernel!                             **
[    0.261883] **                                                        **
[    0.261884] ** WARNING! WARNING! WARNING! WARNING! WARNING! WARNING!  **
[    0.261884] ************************************************************
...

Reproducing steps:
------------------------
cd /opt/ltp
./runltp -f mm

[  734.485672] ------------[ cut here ]------------
[  734.490306] WARNING: CPU: 3 PID: 32321 at mm/mremap.c:212
move_page_tables+0x7c3/0x830
[  734.498212] Modules linked in: x86_pkg_temp_thermal
[  734.503084] CPU: 3 PID: 32321 Comm: true Tainted: G        W
 5.6.2-rc1+ #14
[  734.510729] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.2 05/23/2018
[  734.518110] EIP: move_page_tables+0x7c3/0x830
[  734.522463] Code: 0c eb a7 8d 45 d8 83 4d e8 01 e8 c8 e6 01 00 e9
be f8 ff ff 8d 45 d8 31 d2 e8 59 e8 01 00 e9 a5 fc ff ff 31 db e9 81
fc ff ff <0f> 0b 8b 45 b8 e8 43 e0 fe ff ff 75 b0 ff 75 08 ff 75 cc 68
e4 50
[  734.541200] EAX: eb5f1fe8 EBX: 00400000 ECX: 2b5f1001 EDX: 2b5f1000
[  734.547456] ESI: ea5d6010 EDI: eb5f1ff0 EBP: da381e14 ESP: da381d84
[  734.553714] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010202
[  734.560492] CR0: 80050033 CR2: b7d11f50 CR3: 2a5d6000 CR4: 003406f0
[  734.566748] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  734.573006] DR6: fffe0ff0 DR7: 00000400
[  734.576835] Call Trace:
[  734.579282]  setup_arg_pages+0x22c/0x350
[  734.583207]  ? strlcpy+0x33/0x50
[  734.586459]  load_elf_binary+0x352/0x1010
[  734.590468]  ? selinux_inode_permission+0xe5/0x1f0
[  734.595254]  search_binary_handler+0x77/0x1a0
[  734.599614]  __do_execve_file+0x5aa/0x710
[  734.603615]  sys_execve+0x21/0x30
[  734.606926]  do_fast_syscall_32+0x75/0x260
[  734.611019]  entry_SYSENTER_32+0xa5/0xf8
[  734.614942] EIP: 0xb7ef6c11
[  734.617735] Code: Bad RIP value.
[  734.620956] EAX: ffffffda EBX: bfb8dcb0 ECX: 08069420 EDX: bfb8df14
[  734.627238] ESI: 080599d4 EDI: bfb8dcb9 EBP: bfb8dd58 ESP: bfb8dc88
[  734.633499] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000296
[  734.640276] ---[ end trace e625d4d55b8380f3 ]---
[  734.644934] vma dbc4c180 start bf801000 end c0000000
[  734.644934] next 00000000 prev 00000000 mm d0d75d40
[  734.644934] prot 25 anon_vma d0dfddc8 vm_ops 00000000
[  734.644934] pgoff bfa01 file 00000000 private_data 00000000
[  734.644934] flags:
0x118173(read|write|mayread|maywrite|mayexec|growsdown|seqread|randread|account)
[  734.674346] old_addr: 0xbfc00000, new_addr: 0xbfa00000, old_end: 0xc0000000
[  734.681322] old_pmd: 0x77923067
[  734.681322] new_pmd: 0x7796d067
[  734.684510] ------------[ cut here ]------------
[  734.692295] WARNING: CPU: 2 PID: 32321 at mm/mremap.c:212
move_page_tables+0x7c3/0x830
[  734.700241] Modules linked in: x86_pkg_temp_thermal
[  734.705128] CPU: 2 PID: 32321 Comm: true Tainted: G        W
 5.6.2-rc1+ #14
[  734.712770] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.2 05/23/2018
[  734.720156] EIP: move_page_tables+0x7c3/0x830
[  734.724506] Code: 0c eb a7 8d 45 d8 83 4d e8 01 e8 c8 e6 01 00 e9
be f8 ff ff 8d 45 d8 31 d2 e8 59 e8 01 00 e9 a5 fc ff ff 31 db e9 81
fc ff ff <0f> 0b 8b 45 b8 e8 43 e0 fe ff ff 75 b0 ff 75 08 ff 75 cc 68
e4 50
[  734.743256] EAX: eb5f1ff0 EBX: 00200000 ECX: 2b5f1001 EDX: 2b5f1000
[  734.749517] ESI: ea5d6010 EDI: eb5f1ff8 EBP: da381e14 ESP: da381d84
[  734.755776] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010202
[  734.762553] CR0: 80050033 CR2: b7f1f4e0 CR3: 2a5d6000 CR4: 003406f0
[  734.768808] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  734.775067] DR6: fffe0ff0 DR7: 00000400
[  734.778898] Call Trace:
[  734.781344]  setup_arg_pages+0x22c/0x350
[  734.785269]  ? strlcpy+0x33/0x50
[  734.788499]  load_elf_binary+0x352/0x1010
[  734.792503]  ? selinux_inode_permission+0xe5/0x1f0
[  734.797287]  search_binary_handler+0x77/0x1a0
[  734.801639]  __do_execve_file+0x5aa/0x710
[  734.805642]  sys_execve+0x21/0x30
[  734.808953]  do_fast_syscall_32+0x75/0x260
[  734.813045]  entry_SYSENTER_32+0xa5/0xf8
[  734.816959] EIP: 0xb7ef6c11
[  734.819752] Code: Bad RIP value.
[  734.822976] EAX: ffffffda EBX: bfb8dcb0 ECX: 08069420 EDX: bfb8df14
[  734.829261] ESI: 080599d4 EDI: bfb8dcb9 EBP: bfb8dd58 ESP: bfb8dc88
[  734.835525] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000296
[  734.842302] ---[ end trace e625d4d55b8380f4 ]---
[  734.846940] vma dbc4c180 start bf801000 end c0000000
[  734.846940] next 00000000 prev 00000000 mm d0d75d40
[  734.846940] prot 25 anon_vma d0dfddc8 vm_ops 00000000
[  734.846940] pgoff bfa01 file 00000000 private_data 00000000
[  734.846940] flags:
0x118173(read|write|mayread|maywrite|mayexec|growsdown|seqread|randread|account)
[  734.876355] old_addr: 0xbfe00000, new_addr: 0xbfc00000, old_end: 0xc0000000
[  734.883316] old_pmd: 0x77a08067


Full test log,
https://lkft.validation.linaro.org/scheduler/job/1334357#L478

- Naresh

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Kirill A. Shutemov]

On Fri, Apr 03, 2020 at 12:56:57AM +0530, Naresh Kamboju wrote:
> [  734.876355] old_addr: 0xbfe00000, new_addr: 0xbfc00000, old_end: 0xc0000000

The ranges are overlapping. We don't expect it. mremap(2) never does this.

shift_arg_pages() only moves range downwards. It should be safe.

Could you try this:

diff --git a/mm/mremap.c b/mm/mremap.c
index af363063ea23..ddd5337de395 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -205,10 +205,14 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		return false;
 
 	/*
-	 * The destination pmd shouldn't be established, free_pgtables()
-	 * should have release it.
+	 * mremap(2) clears the new place, so the new_pmd is expected to be
+	 * clear.
+	 *
+	 * But move_page_tables() is also called from shift_arg_pages() that
+	 * allows for overlapping address ranges. The shift_arg_pages() case
+	 * is also safe as we only move page tables downwards.
 	 */
-	if (WARN_ON(!pmd_none(*new_pmd)))
+	if (WARN_ON(!pmd_none(*new_pmd) && old_addr > new_addr))
 		return false;
 
 	/*
-- 
 Kirill A. Shutemov

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Naresh Kamboju]

On Fri, 3 Apr 2020 at 19:02, Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> On Fri, Apr 03, 2020 at 12:56:57AM +0530, Naresh Kamboju wrote:
> > [  734.876355] old_addr: 0xbfe00000, new_addr: 0xbfc00000, old_end: 0xc0000000
>
> The ranges are overlapping. We don't expect it. mremap(2) never does this.
>
> shift_arg_pages() only moves range downwards. It should be safe.
>
> Could you try this:

Applied the patch and tested and still getting kernel warning.
CONFIG_HIGHMEM64G=y is still enabled.

[  790.041040] ------------[ cut here ]------------
[  790.045664] WARNING: CPU: 3 PID: 3195 at mm/mremap.c:212
move_page_tables+0x7a7/0x840
[  790.053486] Modules linked in: x86_pkg_temp_thermal
[  790.058358] CPU: 3 PID: 3195 Comm: true Tainted: G        W
5.6.2-rc1+ #15
[  790.065915] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.0b 07/27/2017
[  790.073386] EIP: move_page_tables+0x7a7/0x840
[  790.077737] Code: 9f 84 c0 0f 84 b7 fc ff ff 89 c3 e9 ba fe ff ff
8b 40 54 8b 40 10 8b 40 1c 8b 80 20 02 00 00 8b 40 0c 8b 50 08 83 c2
0c eb a7 <0f> 0b e9 55 fd ff ff 8d 45 d8 83 4d e8 01 e8 c6 e6 01 00 e9
ac f8
[  790.096475] EAX: bfe00000 EBX: 00200000 ECX: 07606001 EDX: 07606000
[  790.102732] ESI: c64c0010 EDI: c7606ff8 EBP: c845de14 ESP: c845dd7c
[  790.108989] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010206
[  790.115764] CR0: 80050033 CR2: b7e13b50 CR3: 064c0000 CR4: 003406f0
[  790.122024] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  790.128281] DR6: fffe0ff0 DR7: 00000400
[  790.132111] Call Trace:
[  790.134558]  setup_arg_pages+0x22c/0x350
[  790.138514]  ? strlcpy+0x33/0x50
[  790.141776]  load_elf_binary+0x352/0x1010
[  790.145788]  ? selinux_inode_permission+0xe5/0x1f0
[  790.150573]  search_binary_handler+0x77/0x1a0
[  790.154931]  __do_execve_file+0x5aa/0x710
[  790.158935]  sys_execve+0x21/0x30
[  790.162246]  do_fast_syscall_32+0x75/0x260
[  790.166336]  entry_SYSENTER_32+0xa5/0xf8
[  790.170254] EIP: 0xb7f12c11
[  790.173045] Code: Bad RIP value.
[  790.176266] EAX: ffffffda EBX: bfc687d0 ECX: 08069420 EDX: bfc68a34
[  790.182548] ESI: 080599d4 EDI: bfc687d9 EBP: bfc68878 ESP: bfc687a8
[  790.188808] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000296
[  790.195585] ---[ end trace e8f9014a5c1de460 ]---

full test log,
https://lkft.validation.linaro.org/scheduler/job/1339582#L9858

- Naresh

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Kirill A. Shutemov]

On Sat, Apr 04, 2020 at 08:10:42PM +0530, Naresh Kamboju wrote:
> On Fri, 3 Apr 2020 at 19:02, Kirill A. Shutemov <kirill@shutemov.name> wrote:
> >
> > On Fri, Apr 03, 2020 at 12:56:57AM +0530, Naresh Kamboju wrote:
> > > [  734.876355] old_addr: 0xbfe00000, new_addr: 0xbfc00000, old_end: 0xc0000000
> >
> > The ranges are overlapping. We don't expect it. mremap(2) never does this.
> >
> > shift_arg_pages() only moves range downwards. It should be safe.
> >
> > Could you try this:
> 
> Applied the patch and tested and still getting kernel warning.
> CONFIG_HIGHMEM64G=y is still enabled.
> 
> [  790.041040] ------------[ cut here ]------------
> [  790.045664] WARNING: CPU: 3 PID: 3195 at mm/mremap.c:212
> move_page_tables+0x7a7/0x840

Are you sure the patch is applied? The line number in the warning supposed
to change.

-- 
 Kirill A. Shutemov

Re: mm/mremap.c : WARNING: at mm/mremap.c:211 move_page_tables+0x5b0/0x5d0

[Naresh Kamboju]

On Sat, 4 Apr 2020 at 21:36, Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> On Sat, Apr 04, 2020 at 08:10:42PM +0530, Naresh Kamboju wrote:
> > On Fri, 3 Apr 2020 at 19:02, Kirill A. Shutemov <kirill@shutemov.name> wrote:
> > >
> > > On Fri, Apr 03, 2020 at 12:56:57AM +0530, Naresh Kamboju wrote:
> > > > [  734.876355] old_addr: 0xbfe00000, new_addr: 0xbfc00000, old_end: 0xc0000000
> > >
> > > The ranges are overlapping. We don't expect it. mremap(2) never does this.
> > >
> > > shift_arg_pages() only moves range downwards. It should be safe.
> > >
> > > Could you try this:
> >
> > Applied the patch and tested and still getting kernel warning.
> > CONFIG_HIGHMEM64G=y is still enabled.
> >
> > [  790.041040] ------------[ cut here ]------------
> > [  790.045664] WARNING: CPU: 3 PID: 3195 at mm/mremap.c:212
> > move_page_tables+0x7a7/0x840
>
> Are you sure the patch is applied? The line number in the warning supposed
> to change.

Yes. The patch was applied and tested.
The reason for line number change is due to linux/mmdebug.h included
because an earlier patch "dump_vma(vma);" needed this.

diff --git a/mm/mremap.c b/mm/mremap.c
index af363063ea23..cf02d4244e83 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -24,6 +24,7 @@
 #include <linux/uaccess.h>
 #include <linux/mm-arch-hooks.h>
 #include <linux/userfaultfd_k.h>
+#include <linux/mmdebug.h>

 #include <asm/cacheflush.h>
 #include <asm/tlbflush.h>
@@ -208,7 +209,7 @@ static bool move_normal_pmd(struct vm_area_struct
*vma, unsigned long old_addr,
         * The destination pmd shouldn't be established, free_pgtables()
         * should have release it.
         */
-       if (WARN_ON(!pmd_none(*new_pmd)))
+       if (WARN_ON(!pmd_none(*new_pmd) && old_addr > new_addr))
                return false;

        /*



- Naresh