Kernel panic - not syncing: corrupted stack end detected inside scheduler

Kernel panic - not syncing: corrupted stack end detected inside scheduler

[Qian Cai]

Running LTP oom01 [1] test triggered kernel panic on an aarch64 server with the latest mainline (rc2).

[1] https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/mem/oom/oom01.c

[ 3433.338741] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[ 3433.347644] CPU: 49 PID: 2189 Comm: in:imjournal Kdump: loaded Tainted: G        W         4.20.0-rc2+ #15
[ 3433.357298] Hardware name: Huawei TaiShan 2280 /BC11SPCD, BIOS 1.50 06/01/2018
[ 3433.364523] Call trace:
[ 3433.366993]  dump_backtrace+0x0/0x2c8
[ 3433.370659]  show_stack+0x24/0x30
[ 3433.373980]  dump_stack+0x118/0x19c
[ 3433.377473]  panic+0x1b8/0x31c
[ 3433.380530]  schedule+0x0/0x240
[ 3433.383672]  schedule+0xdc/0x240
[ 3433.386905]  io_schedule+0x24/0x48
[ 3433.390313]  get_request+0x3b0/0xb68
[ 3433.393891]  blk_queue_bio+0x3a4/0xcd8
[ 3433.397642]  generic_make_request+0x440/0x7d8
[ 3433.402000]  submit_bio+0xbc/0x300
[ 3433.405409]  __swap_writepage+0xa54/0xd00
[ 3433.409420]  swap_writepage+0x44/0xb0
[ 3433.413086]  pageout.isra.12+0x580/0xd80
[ 3433.417011]  shrink_page_list+0x2480/0x36f0
[ 3433.421196]  shrink_inactive_list+0x388/0xb98
[ 3433.425555]  shrink_node_memcg+0x344/0x9c0
[ 3433.429653]  shrink_node+0x200/0x940
[ 3433.433231]  do_try_to_free_pages+0x234/0x7d0
[ 3433.437589]  try_to_free_pages+0x228/0x6b0
[ 3433.441689]  __alloc_pages_nodemask+0xcbc/0x2028
[ 3433.446309]  alloc_pages_vma+0x1a4/0x208
[ 3433.450235]  __read_swap_cache_async+0x4fc/0x858
[ 3433.454855]  read_swap_cache_async+0xa4/0x100
[ 3433.459214]  swap_cluster_readahead+0x598/0x650
[ 3433.463746]  shmem_swapin+0xd4/0x150
[ 3433.467324]  shmem_getpage_gfp+0xf50/0x1c48
[ 3433.471509]  shmem_fault+0x140/0x340
[ 3433.475086]  __do_fault+0xd0/0x440
[ 3433.478490]  do_fault+0x54c/0xf48
[ 3433.481807]  __handle_mm_fault+0x4c0/0x928
[ 3433.485905]  handle_mm_fault+0x30c/0x4b8
[ 3433.489832]  do_page_fault+0x294/0x658
[ 3433.493584]  do_translation_fault+0x98/0xa8
[ 3433.497769]  do_mem_abort+0x64/0xf0
[ 3433.501258]  el0_da+0x24/0x28

[PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

If the kernel is configured with KASAN_EXTRA, the stack size is
increasted significantly due to enable this option will set
-fstack-reuse to "none" in GCC [1]. As the results, it could trigger
stack overrun quite often with 32k stack size compiled using GCC 8. For
example, this reproducer

https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/\
syscalls/madvise/madvise06.c

could trigger a "corrupted stack end detected inside scheduler" very
reliably with CONFIG_SCHED_STACK_END_CHECK enabled. Also, See other bug
reports,

https://lore.kernel.org/lkml/1542144497.12945.29.camel@gmx.us/
https://lore.kernel.org/lkml/721E7B42-2D55-4866-9C1A-3E8D64F33F9C@gmx.us/

There are just too many functions that could have a large stack with
KASAN_EXTRA due to large local variables that have been called over and
over again without being able to reuse the stacks. Some noticiable ones
are,

size
7536 shrink_inactive_list
7440 shrink_page_list
6560 fscache_stats_show
3920 jbd2_journal_commit_transaction
3216 try_to_unmap_one
3072 migrate_page_move_mapping
3584 migrate_misplaced_transhuge_page
3920 ip_vs_lblcr_schedule
4304 lpfc_nvme_info_show
3888 lpfc_debugfs_nvmestat_data.constprop

There are other 49 functions are over 2k in size while compiling kernel
with "-Wframe-larger-than=" on this machine. Hence, it is too much work
to change Makefiles for each object to compile without
-fsanitize-address-use-after-scope individually.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Signed-off-by: Qian Cai <cai@lca.pw>
---
 arch/arm64/include/asm/memory.h | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index b96442960aea..56562ff01076 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -76,12 +76,17 @@
 /*
  * KASAN requires 1/8th of the kernel virtual address space for the shadow
  * region. KASAN can bloat the stack significantly, so double the (minimum)
- * stack size when KASAN is in use.
+ * stack size when KASAN is in use, and then double it again if KASAN_EXTRA is
+ * on.
  */
 #ifdef CONFIG_KASAN
 #define KASAN_SHADOW_SCALE_SHIFT 3
 #define KASAN_SHADOW_SIZE	(UL(1) << (VA_BITS - KASAN_SHADOW_SCALE_SHIFT))
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_THREAD_SHIFT	2
+#else
 #define KASAN_THREAD_SHIFT	1
+#endif /* CONFIG_KASAN_EXTRA */
 #else
 #define KASAN_SHADOW_SIZE	(0)
 #define KASAN_THREAD_SHIFT	0
-- 
2.17.2 (Apple Git-113)

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Arnd Bergmann]

On Fri, Dec 7, 2018 at 11:35 PM Qian Cai <cai@lca.pw> wrote:
>
> If the kernel is configured with KASAN_EXTRA, the stack size is
> increasted significantly due to enable this option will set
> -fstack-reuse to "none" in GCC [1]. As the results, it could trigger
> stack overrun quite often with 32k stack size compiled using GCC 8. For
> example, this reproducer

>
> size
> 7536 shrink_inactive_list
> 7440 shrink_page_list
> 6560 fscache_stats_show
> 3920 jbd2_journal_commit_transaction
> 3216 try_to_unmap_one
> 3072 migrate_page_move_mapping
> 3584 migrate_misplaced_transhuge_page
> 3920 ip_vs_lblcr_schedule
> 4304 lpfc_nvme_info_show
> 3888 lpfc_debugfs_nvmestat_data.constprop
>
> There are other 49 functions are over 2k in size while compiling kernel
> with "-Wframe-larger-than=" on this machine. Hence, it is too much work
> to change Makefiles for each object to compile without
> -fsanitize-address-use-after-scope individually.
>
> [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Could you clarify: are the numbers you see with or without the bugfix
from that bugzilla?

       Arnd

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

On Tue, 2018-12-11 at 13:42 +0100, Arnd Bergmann wrote:
> On Fri, Dec 7, 2018 at 11:35 PM Qian Cai <cai@lca.pw> wrote:
> > 
> > If the kernel is configured with KASAN_EXTRA, the stack size is
> > increasted significantly due to enable this option will set
> > -fstack-reuse to "none" in GCC [1]. As the results, it could trigger
> > stack overrun quite often with 32k stack size compiled using GCC 8. For
> > example, this reproducer
> > 
> > size
> > 7536 shrink_inactive_list
> > 7440 shrink_page_list
> > 6560 fscache_stats_show
> > 3920 jbd2_journal_commit_transaction
> > 3216 try_to_unmap_one
> > 3072 migrate_page_move_mapping
> > 3584 migrate_misplaced_transhuge_page
> > 3920 ip_vs_lblcr_schedule
> > 4304 lpfc_nvme_info_show
> > 3888 lpfc_debugfs_nvmestat_data.constprop
> > 
> > There are other 49 functions are over 2k in size while compiling kernel
> > with "-Wframe-larger-than=" on this machine. Hence, it is too much work
> > to change Makefiles for each object to compile without
> > -fsanitize-address-use-after-scope individually.
> > 
> > [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23
> 
> Could you clarify: are the numbers you see with or without the bugfix
> from that bugzilla?
> 

The numbers were from GCC8 which does NOT contain this patch [1].

GCC9 is awesome which reduced the numbers in half even for KASAN_EXTRA. Only
thing is that GCC9 has not been officially released yet, so it is a bit
inconvenient for users need to compile the compiler by themselves first.

I am fine either way to drop this patch or keep it until GCC9 is GA.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c35

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Arnd Bergmann]

On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
>
> On Tue, 2018-12-11 at 13:42 +0100, Arnd Bergmann wrote:
> > On Fri, Dec 7, 2018 at 11:35 PM Qian Cai <cai@lca.pw> wrote:
> > >
> > > If the kernel is configured with KASAN_EXTRA, the stack size is
> > > increasted significantly due to enable this option will set
> > > -fstack-reuse to "none" in GCC [1]. As the results, it could trigger
> > > stack overrun quite often with 32k stack size compiled using GCC 8. For
> > > example, this reproducer
> > >
> > > size
> > > 7536 shrink_inactive_list
> > > 7440 shrink_page_list
> > > 6560 fscache_stats_show
> > > 3920 jbd2_journal_commit_transaction
> > > 3216 try_to_unmap_one
> > > 3072 migrate_page_move_mapping
> > > 3584 migrate_misplaced_transhuge_page
> > > 3920 ip_vs_lblcr_schedule
> > > 4304 lpfc_nvme_info_show
> > > 3888 lpfc_debugfs_nvmestat_data.constprop
> > >
> > > There are other 49 functions are over 2k in size while compiling kernel
> > > with "-Wframe-larger-than=" on this machine. Hence, it is too much work
> > > to change Makefiles for each object to compile without
> > > -fsanitize-address-use-after-scope individually.
> > >
> > > [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23
> >
> > Could you clarify: are the numbers you see with or without the bugfix
> > from that bugzilla?
> >
>
> The numbers were from GCC8 which does NOT contain this patch [1].
>
> GCC9 is awesome which reduced the numbers in half even for KASAN_EXTRA. Only
> thing is that GCC9 has not been officially released yet, so it is a bit
> inconvenient for users need to compile the compiler by themselves first.
>
> I am fine either way to drop this patch or keep it until GCC9 is GA.
>
> [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c35

Maybe we can make the constant depend on the compiler version?
It may also be possible to reduce the KASAN_THREAD_SHIFT
constant for the normal case with gcc-9 and go back to the
default frame size then.

       Arnd

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
> > 
> > On Tue, 2018-12-11 at 13:42 +0100, Arnd Bergmann wrote:
> > > On Fri, Dec 7, 2018 at 11:35 PM Qian Cai <cai@lca.pw> wrote:
> > > > 
> > > > If the kernel is configured with KASAN_EXTRA, the stack size is
> > > > increasted significantly due to enable this option will set
> > > > -fstack-reuse to "none" in GCC [1]. As the results, it could trigger
> > > > stack overrun quite often with 32k stack size compiled using GCC 8. For
> > > > example, this reproducer
> > > > 
> > > > size
> > > > 7536 shrink_inactive_list
> > > > 7440 shrink_page_list
> > > > 6560 fscache_stats_show
> > > > 3920 jbd2_journal_commit_transaction
> > > > 3216 try_to_unmap_one
> > > > 3072 migrate_page_move_mapping
> > > > 3584 migrate_misplaced_transhuge_page
> > > > 3920 ip_vs_lblcr_schedule
> > > > 4304 lpfc_nvme_info_show
> > > > 3888 lpfc_debugfs_nvmestat_data.constprop
> > > > 
> > > > There are other 49 functions are over 2k in size while compiling kernel
> > > > with "-Wframe-larger-than=" on this machine. Hence, it is too much work
> > > > to change Makefiles for each object to compile without
> > > > -fsanitize-address-use-after-scope individually.
> > > > 
> > > > [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23
> > > 
> > > Could you clarify: are the numbers you see with or without the bugfix
> > > from that bugzilla?
> > > 
> > 
> > The numbers were from GCC8 which does NOT contain this patch [1].
> > 
> > GCC9 is awesome which reduced the numbers in half even for KASAN_EXTRA. Only
> > thing is that GCC9 has not been officially released yet, so it is a bit
> > inconvenient for users need to compile the compiler by themselves first.
> > 
> > I am fine either way to drop this patch or keep it until GCC9 is GA.
> > 
> > [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c35
> 
> Maybe we can make the constant depend on the compiler version?

I am not too keen to do the version-check considering some LTS versions could
just back-port those patches and the render the version-check incorrectly.

> It may also be possible to reduce the KASAN_THREAD_SHIFT
> constant for the normal case with gcc-9 and go back to the
> default frame size then.

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Arnd Bergmann]

On Tue, Dec 11, 2018 at 10:52 PM Qian Cai <cai@lca.pw> wrote:
> On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> > On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:

> I am not too keen to do the version-check considering some LTS versions could
> just back-port those patches and the render the version-check incorrectly.

I'm not following what the problem is. Do you mean distro versions gcc
with the compiler bugfix, or LTS kernel versions?

For backported kernel fixes, this doesn't seem to be any different
from other kernel changes that might be incorrect, but a straight
backport of that kernel patch should still do the right thing in older
kernels, and set the frame size higher for old compilers but not
new ones.

    Arnd

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

On Tue, 2018-12-11 at 22:56 +0100, Arnd Bergmann wrote:
> On Tue, Dec 11, 2018 at 10:52 PM Qian Cai <cai@lca.pw> wrote:
> > On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> > > On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
> > I am not too keen to do the version-check considering some LTS versions
> > could
> > just back-port those patches and the render the version-check incorrectly.
> 
> I'm not following what the problem is. Do you mean distro versions gcc
> with the compiler bugfix, or LTS kernel versions?
> 

I mean distro versions of GCC where the version is still 8 but keep back-porting 
tons of patches.

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Arnd Bergmann]

On Tue, Dec 11, 2018 at 10:59 PM Qian Cai <cai@lca.pw> wrote:
>
> On Tue, 2018-12-11 at 22:56 +0100, Arnd Bergmann wrote:
> > On Tue, Dec 11, 2018 at 10:52 PM Qian Cai <cai@lca.pw> wrote:
> > > On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> > > > On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
> > > I am not too keen to do the version-check considering some LTS versions
> > > could
> > > just back-port those patches and the render the version-check incorrectly.
> >
> > I'm not following what the problem is. Do you mean distro versions gcc
> > with the compiler bugfix, or LTS kernel versions?
> >
>
> I mean distro versions of GCC where the version is still 8 but keep back-porting
> tons of patches.

Ok, but in that case, checking the version would still be no worse
than your current patch, the only difference is that for users of a
fixed older gcc, the kernel would use more stack than it needs.

        Arnd

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

On Tue, 2018-12-11 at 23:12 +0100, Arnd Bergmann wrote:
> On Tue, Dec 11, 2018 at 10:59 PM Qian Cai <cai@lca.pw> wrote:
> > 
> > On Tue, 2018-12-11 at 22:56 +0100, Arnd Bergmann wrote:
> > > On Tue, Dec 11, 2018 at 10:52 PM Qian Cai <cai@lca.pw> wrote:
> > > > On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> > > > > On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
> > > > 
> > > > I am not too keen to do the version-check considering some LTS versions
> > > > could
> > > > just back-port those patches and the render the version-check
> > > > incorrectly.
> > > 
> > > I'm not following what the problem is. Do you mean distro versions gcc
> > > with the compiler bugfix, or LTS kernel versions?
> > > 
> > 
> > I mean distro versions of GCC where the version is still 8 but keep back-
> > porting
> > tons of patches.
> 
> Ok, but in that case, checking the version would still be no worse
> than your current patch, the only difference is that for users of a
> fixed older gcc, the kernel would use more stack than it needs.
> 

I am thinking about something it is probably best just waiting for those major
distors to complete upgrading to GCC9 or back-porting those stack reduction
patches first. Then, it is good time to tie up loose ends for those default
stack sizes in all combinations.

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Arnd Bergmann]

On Tue, Dec 11, 2018 at 11:22 PM Qian Cai <cai@lca.pw> wrote:
>
> On Tue, 2018-12-11 at 23:12 +0100, Arnd Bergmann wrote:
> > On Tue, Dec 11, 2018 at 10:59 PM Qian Cai <cai@lca.pw> wrote:
> > >
> > > On Tue, 2018-12-11 at 22:56 +0100, Arnd Bergmann wrote:
> > > > On Tue, Dec 11, 2018 at 10:52 PM Qian Cai <cai@lca.pw> wrote:
> > > > > On Tue, 2018-12-11 at 22:43 +0100, Arnd Bergmann wrote:
> > > > > > On Tue, Dec 11, 2018 at 6:18 PM Qian Cai <cai@lca.pw> wrote:
> > > > >
> > > > > I am not too keen to do the version-check considering some LTS versions
> > > > > could
> > > > > just back-port those patches and the render the version-check
> > > > > incorrectly.
> > > >
> > > > I'm not following what the problem is. Do you mean distro versions gcc
> > > > with the compiler bugfix, or LTS kernel versions?
> > > >
> > >
> > > I mean distro versions of GCC where the version is still 8 but keep back-
> > > porting
> > > tons of patches.
> >
> > Ok, but in that case, checking the version would still be no worse
> > than your current patch, the only difference is that for users of a
> > fixed older gcc, the kernel would use more stack than it needs.
> >
>
> I am thinking about something it is probably best just waiting for those major
> distors to complete upgrading to GCC9 or back-porting those stack reduction
> patches first. Then, it is good time to tie up loose ends for those default
> stack sizes in all combinations.

I was basically trying to make sure we don't forget it when it gets to that.

Another alternative would be to just disable KASAN_EXTRA now
for gcc versions before 9, which essentially means for everyone,
but then we get it back once a working version gets released. As
I understand, this kasan option is actually fairly useless given its
cost, so very few people would miss it.

On a related note, I think we have to turn off asan-stack entirely
on all released clang versions. asan-stack in general is much more
useful than the use-after-scope check, but we clang produces some
very large stack frames with it and we probably can't even work
around it with KASAN_THREAD_SHIFT=2 but would need even
more than that otherwise.

         Arnd

Re: [PATCH] arm64: increase stack size for KASAN_EXTRA

[Qian Cai]

On 12/11/18 6:06 PM, Arnd Bergmann wrote:
>> I am thinking about something it is probably best just waiting for those major
>> distors to complete upgrading to GCC9 or back-porting those stack reduction
>> patches first. Then, it is good time to tie up loose ends for those default
>> stack sizes in all combinations.
> 
> I was basically trying to make sure we don't forget it when it gets to that.

I added a reminder in my calendar to check the GCC9 adoption in Q2 FY19.

> 
> Another alternative would be to just disable KASAN_EXTRA now
> for gcc versions before 9, which essentially means for everyone,
> but then we get it back once a working version gets released. As
> I understand, this kasan option is actually fairly useless given its
> cost, so very few people would miss it.
> 
> On a related note, I think we have to turn off asan-stack entirely
> on all released clang versions. asan-stack in general is much more
> useful than the use-after-scope check, but we clang produces some
> very large stack frames with it and we probably can't even work
> around it with KASAN_THREAD_SHIFT=2 but would need even
> more than that otherwise.
> 
>          Arnd
>