mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Michal Hocko]

Hi,
I am getting tons of splats like this:
[  187.593291] BUG: using smp_processor_id() in preemptible [00000000] code: kworker/u4:1/24
[  187.593293] caller is debug_smp_processor_id+0x17/0x19
[  187.599127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[  187.601997]  ffffffff8181786e ffff8800068a3d78 ffffffff8151a4b8 ffffffff81078e0e
[  187.604373]  0000000000000000 ffff8800068a3da8 ffffffff81280ac0 ffff8800068acac8
[  187.606619]  00000000000029b2 ffff880007401500 00000000000000d0 ffff8800068a3db8
[  187.608869] Call Trace:
[  187.609602]  [<ffffffff8151a4b8>] dump_stack+0x4f/0x7c
[  187.615881]  [<ffffffff81078e0e>] ? down_trylock+0x2d/0x37
[  187.617468]  [<ffffffff81280ac0>] check_preemption_disabled+0xe7/0xf9
[  187.619295]  [<ffffffff81280ae9>] debug_smp_processor_id+0x17/0x19
[  187.621055]  [<ffffffff8113b2f6>] kmem_cache_alloc_trace+0x78/0x22e
[  187.622854]  [<ffffffff81088d36>] ? do_syslog+0xb2/0x452
[  187.624376]  [<ffffffff81088d36>] do_syslog+0xb2/0x452
[  187.626030]  [<ffffffff811675b4>] ? __fdget_pos+0x3d/0x43
[  187.627557]  [<ffffffff810745ac>] ? wait_woken+0x5d/0x5d
[  187.629073]  [<ffffffff811675b4>] ? __fdget_pos+0x3d/0x43
[  187.630611]  [<ffffffff811a787b>] kmsg_read+0x2d/0x54
[  187.632069]  [<ffffffff8119c4ff>] proc_reg_read+0x4a/0x6c
[  187.633775]  [<ffffffff8114e4d8>] vfs_read+0xa5/0x141
[  187.635242]  [<ffffffff8114ee35>] SyS_read+0x51/0x8f
[  187.636666]  [<ffffffff81522892>] system_call_fastpath+0x12/0x17

$ grep "BUG: using smp_processor_id" mmap.qcow_serial.log | wc -l
660
after few minutes of runtime when running the current mmotm tree 
(https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/log/?h=since-3.18).

The warning seems to come from this_cpu_ptr which is using my_cpu_offset
which is
#define my_cpu_offset per_cpu_offset(smp_processor_id())
if CONFIG_DEBUG_PREEMPT

So it matches the following code in slab_alloc_node resp. slab_free:
        do {
                tid = this_cpu_read(s->cpu_slab->tid);
                c = this_cpu_ptr(s->cpu_slab); <<< HERE
        } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));

This is with:
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_DEBUG_PREEMPT=y

I am not sure how to fix this but it sounds like this_cpu_ptr should
offer the same preempt expectations as other this_cpu_* functions.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Krzysztof Kozłowski]

2015-01-21 14:23 GMT+01:00 Michal Hocko <mhocko@suse.cz>:
> Hi,
> I am getting tons of splats like this:
> [ 187.593291] BUG: using smp_processor_id() in preemptible [00000000] code: kworker/u4:1/24
> [ 187.593293] caller is debug_smp_processor_id+0x17/0x19
> [ 187.599127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
> [  187.601997]  ffffffff8181786e ffff8800068a3d78 ffffffff8151a4b8 ffffffff81078e0e
> [  187.604373]  0000000000000000 ffff8800068a3da8 ffffffff81280ac0 ffff8800068acac8
> [  187.606619]  00000000000029b2 ffff880007401500 00000000000000d0 ffff8800068a3db8
> [  187.608869] Call Trace:
> [  187.609602]  [<ffffffff8151a4b8>] dump_stack+0x4f/0x7c
> [  187.615881]  [<ffffffff81078e0e>] ? down_trylock+0x2d/0x37
> [  187.617468]  [<ffffffff81280ac0>] check_preemption_disabled+0xe7/0xf9
> [  187.619295]  [<ffffffff81280ae9>] debug_smp_processor_id+0x17/0x19
> [  187.621055]  [<ffffffff8113b2f6>] kmem_cache_alloc_trace+0x78/0x22e
> [  187.622854]  [<ffffffff81088d36>] ? do_syslog+0xb2/0x452
> [  187.624376]  [<ffffffff81088d36>] do_syslog+0xb2/0x452
> [  187.626030]  [<ffffffff811675b4>] ? __fdget_pos+0x3d/0x43
> [  187.627557]  [<ffffffff810745ac>] ? wait_woken+0x5d/0x5d
> [  187.629073]  [<ffffffff811675b4>] ? __fdget_pos+0x3d/0x43
> [  187.630611]  [<ffffffff811a787b>] kmsg_read+0x2d/0x54
> [  187.632069]  [<ffffffff8119c4ff>] proc_reg_read+0x4a/0x6c
> [  187.633775]  [<ffffffff8114e4d8>] vfs_read+0xa5/0x141
> [  187.635242]  [<ffffffff8114ee35>] SyS_read+0x51/0x8f
> [  187.636666]  [<ffffffff81522892>] system_call_fastpath+0x12/0x17

Same here :) [1] . So actually only ARM seems affected (both armv7 and
armv8) because it is the only one which uses smp_processor_id() in
my_cpu_offset.

[1] https://lkml.org/lkml/2015/1/20/162

Best regards,
Krzysztof

>
> $ grep "BUG: using smp_processor_id" mmap.qcow_serial.log | wc -l
> 660
> after few minutes of runtime when running the current mmotm tree
> (https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/log/?h=since-3.18).
>
> The warning seems to come from this_cpu_ptr which is using my_cpu_offset
> which is
> #define my_cpu_offset per_cpu_offset(smp_processor_id())
> if CONFIG_DEBUG_PREEMPT
>
> So it matches the following code in slab_alloc_node resp. slab_free:
>         do {
>                 tid = this_cpu_read(s->cpu_slab->tid);
>                 c = this_cpu_ptr(s->cpu_slab); <<< HERE
>         } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));
>
> This is with:
> CONFIG_PREEMPT=y
> CONFIG_PREEMPT_COUNT=y
> CONFIG_DEBUG_PREEMPT=y
>
> I am not sure how to fix this but it sounds like this_cpu_ptr should
> offer the same preempt expectations as other this_cpu_* functions.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Michal Hocko]

On Wed 21-01-15 15:06:03, Krzysztof KozA?owski wrote:
[...]
> Same here :) [1] . So actually only ARM seems affected (both armv7 and
> armv8) because it is the only one which uses smp_processor_id() in
> my_cpu_offset.

This was on x86_64 with CONFIG_DEBUG_PREEMPT so it is not only ARM
specific.
 
> [1] https://lkml.org/lkml/2015/1/20/162

Sorry, have missed this post.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Andrew Morton]

On Wed, 21 Jan 2015 15:11:38 +0100 Michal Hocko <mhocko@suse.cz> wrote:

> On Wed 21-01-15 15:06:03, Krzysztof Koz__owski wrote:
> [...]
> > Same here :) [1] . So actually only ARM seems affected (both armv7 and
> > armv8) because it is the only one which uses smp_processor_id() in
> > my_cpu_offset.
> 
> This was on x86_64 with CONFIG_DEBUG_PREEMPT so it is not only ARM
> specific.
>  

Hopefully
mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off-v3.patch
will fix this.

The most recent -mmotm was a bit of a trainwreck.  I'm scrambling to
get the holes plugged so I can get another mmotm out today.



From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Subject: mm/slub: optimize alloc/free fastpath by removing preemption on/off

Change from v2:
- use raw_cpu_ptr() rather than this_cpu_ptr() to avoid warning from
 preemption debug check since this is intended behaviour
- fix typo alogorithm -> algorithm

Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Acked-by: Christoph Lameter <cl@linux.com>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
Tested-by: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/slub.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff -puN mm/slub.c~mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off-v3 mm/slub.c
--- a/mm/slub.c~mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off-v3
+++ a/mm/slub.c
@@ -2404,11 +2404,11 @@ redo:
 	 */
 	do {
 		tid = this_cpu_read(s->cpu_slab->tid);
-		c = this_cpu_ptr(s->cpu_slab);
+		c = raw_cpu_ptr(s->cpu_slab);
 	} while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));
 
 	/*
-	 * Irqless object alloc/free alogorithm used here depends on sequence
+	 * Irqless object alloc/free algorithm used here depends on sequence
 	 * of fetching cpu_slab's data. tid should be fetched before anything
 	 * on c to guarantee that object and page associated with previous tid
 	 * won't be used with current tid. If we fetch tid first, object and
@@ -2670,7 +2670,7 @@ redo:
 	 */
 	do {
 		tid = this_cpu_read(s->cpu_slab->tid);
-		c = this_cpu_ptr(s->cpu_slab);
+		c = raw_cpu_ptr(s->cpu_slab);
 	} while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));
 
 	/* Same with comment on barrier() in slab_alloc_node() */
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Joonsoo Kim]

On Wed, Jan 21, 2015 at 02:21:07PM -0800, Andrew Morton wrote:
> On Wed, 21 Jan 2015 15:11:38 +0100 Michal Hocko <mhocko@suse.cz> wrote:
> 
> > On Wed 21-01-15 15:06:03, Krzysztof Koz__owski wrote:
> > [...]
> > > Same here :) [1] . So actually only ARM seems affected (both armv7 and
> > > armv8) because it is the only one which uses smp_processor_id() in
> > > my_cpu_offset.
> > 
> > This was on x86_64 with CONFIG_DEBUG_PREEMPT so it is not only ARM
> > specific.
> >  
> 
> Hopefully
> mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off-v3.patch
> will fix this.

Yes, it will fix this error.

> The most recent -mmotm was a bit of a trainwreck.  I'm scrambling to
> get the holes plugged so I can get another mmotm out today.

Another mmotm will fix many issues from me. :/

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Andrew Morton]

On Thu, 22 Jan 2015 10:51:23 +0900 Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:

> > The most recent -mmotm was a bit of a trainwreck.  I'm scrambling to
> > get the holes plugged so I can get another mmotm out today.
> 
> Another mmotm will fix many issues from me. :/

I hit a wont-boot-cant-find-init in linux-next so I get to spend
tomorrow bisecting that :(

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

Hi Andrew,

On Wed, 21 Jan 2015 19:34:11 -0800 Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Thu, 22 Jan 2015 10:51:23 +0900 Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> 
> > > The most recent -mmotm was a bit of a trainwreck.  I'm scrambling to
> > > get the holes plugged so I can get another mmotm out today.
> > 
> > Another mmotm will fix many issues from me. :/
> 
> I hit a wont-boot-cant-find-init in linux-next so I get to spend
> tomorrow bisecting that :(

There has been a long discussion about something like that already.
Subject "Re: linux-next: Tree for Jan 20 -- Kernel panic - Unable to
mount root fs"

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Andrew Morton]

On Thu, 22 Jan 2015 16:14:02 +1100 Stephen Rothwell <sfr@canb.auug.org.au> wrote:

> Hi Andrew,
> 
> On Wed, 21 Jan 2015 19:34:11 -0800 Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Thu, 22 Jan 2015 10:51:23 +0900 Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > 
> > > > The most recent -mmotm was a bit of a trainwreck.  I'm scrambling to
> > > > get the holes plugged so I can get another mmotm out today.
> > > 
> > > Another mmotm will fix many issues from me. :/
> > 
> > I hit a wont-boot-cant-find-init in linux-next so I get to spend
> > tomorrow bisecting that :(
> 
> There has been a long discussion about something like that already.
> Subject "Re: linux-next: Tree for Jan 20 -- Kernel panic - Unable to
> mount root fs"

ah, thanks.  You might have saved me a ton of effort there.

And oolookitat, a fix.  I love it when viro says "Oh, fuck".

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: mmotm: mm-slub-optimize-alloc-free-fastpath-by-removing-preemption-on-off.patch is causing preemptible splats

[Christoph Lameter]

> I am not sure how to fix this but it sounds like this_cpu_ptr should
> offer the same preempt expectations as other this_cpu_* functions.

One returns a pointer that may not be useful if the context is switched
and the other completes a operationo on an opject.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>