linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Harry Yoo <harry.yoo@oracle.com>
To: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
	Carlos Maiolino <cem@kernel.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Michal Hocko <mhocko@kernel.org>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Shakeel Butt <shakeel.butt@linux.dev>,
	linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	LKML <linux-kernel@vger.kernel.org>,
	Madhavan Srinivasan <maddy@linux.ibm.com>,
	Ritesh Harjani <riteshh@linux.ibm.com>,
	ojaswin@linux.ibm.com, Muchun Song <muchun.song@linux.dev>,
	Cgroups <cgroups@vger.kernel.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	Hao Li <hao.li@linux.dev>
Subject: Re: [next-20260216]NULL pointer dereference in drain_obj_stock() (RCU free path)
Date: Sun, 22 Feb 2026 20:47:03 +0900	[thread overview]
Message-ID: <aZrstwhqX6bSpjtz@hyeyoo> (raw)
In-Reply-To: <ccdcd672-b5e1-45bc-86f3-791af553f0d8@linux.ibm.com>

On Sun, Feb 22, 2026 at 03:38:57PM +0530, Venkat Rao Bagalkote wrote:
> 
> On 18/02/26 5:06 pm, Vlastimil Babka wrote:
> > On 2/17/26 13:40, Carlos Maiolino wrote:
> > > On Tue, Feb 17, 2026 at 04:59:12PM +0530, Venkat Rao Bagalkote wrote:
> > > > Greetings!!!
> > > > 
> > > > I am observing below OOPs, while running xfstests generic/428 test case. But
> > > > I am not able to reproduce this consistently.
> > > > 
> > > > 
> > > > Platform: IBM Power11 (pSeries LPAR), Radix MMU, LE, 64K pages
> > > > Kernel: 6.19.0-next-20260216
> > > > Tests: generic/428
> > > > 
> > > > local.config >>>
> > > > [xfs_4k]
> > > > export RECREATE_TEST_DEV=true
> > > > export TEST_DEV=/dev/loop0
> > > > export TEST_DIR=/mnt/test
> > > > export SCRATCH_DEV=/dev/loop1
> > > > export SCRATCH_MNT=/mnt/scratch
> > > > export MKFS_OPTIONS="-b size=4096"
> > > > export FSTYP=xfs
> > > > export MOUNT_OPTIONS=""-
> > > > 
> > > > 
> > > > 
> > > > Attached is .config file used.
> > > > 
> > > > Traces:
> > > > 
> > > /me fixing trace's indentation
> > CCing memcg and slab folks.
> > Would be nice to figure out where in drain_obj_stock things got wrong. Any
> > change for e.g. ./scripts/faddr2line ?
> > 
> > I wonder if we have either some bogus objext pointer, or maybe the
> > rcu_free_sheaf() context is new (or previously rare) for memcg and we have
> > some locking issues being exposed in refill/drain.
> 
> 
> This issue also got reproduced on mainline repo.
> 
> Traces:
> 
> [ 8058.036083] Kernel attempted to read user page (0) - exploit attempt?
> (uid: 0)
> [ 8058.036116] BUG: Kernel NULL pointer dereference on read at 0x00000000
> [ 8058.036127] Faulting instruction address: 0xc0000000008b018c
> [ 8058.036137] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 8058.036147] LE PAGE_SIZE=64K MMU=Radix  SMP NR_CPUS=8192 NUMA pSeries
> [ 8058.036159] Modules linked in: overlay dm_zero dm_thin_pool
> dm_persistent_data dm_bio_prison dm_snapshot dm_bufio dm_flakey xfs loop
> dm_mod nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
> nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set bonding nf_tables tls
> rfkill sunrpc nfnetlink pseries_rng vmx_crypto dax_pmem fuse ext4 crc16
> mbcache jbd2 nd_pmem papr_scm sd_mod libnvdimm sg ibmvscsi ibmveth
> scsi_transport_srp pseries_wdt [last unloaded: scsi_debug]
> [ 8058.036339] CPU: 19 UID: 0 PID: 115 Comm: ksoftirqd/19 Kdump: loaded Not
> tainted 6.19.0+ #1 PREEMPTLAZY
> [ 8058.036361] Hardware name: IBM,9080-HEX Power11 (architected) 0x820200
> 0xf000007 of:IBM,FW1110.01 (NH1110_069) hv:phyp pSeries
> [ 8058.036379] NIP:  c0000000008b018c LR: c0000000008b0180 CTR:
> c00000000036d680
> [ 8058.036395] REGS: c00000000b5976c0 TRAP: 0300   Not tainted (6.19.0+)
> [ 8058.036411] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR:
> 84042002  XER: 20040000
> [ 8058.036482] CFAR: c000000000862cf4 DAR: 0000000000000000 DSISR: 40000000
> IRQMASK: 0
> [ 8058.036482] GPR00: c0000000008b0180 c00000000b597960 c00000000243a500
> 0000000000000001
> [ 8058.036482] GPR04: 0000000000000008 0000000000000001 c0000000008b0180
> 0000000000000001
> [ 8058.036482] GPR08: a80e000000000000 0000000000000001 0000000000000007
> a80e000000000000
> [ 8058.036482] GPR12: c00e00000120f8d5 c000000d0ddf0b00 c000000073567780
> 0000000000000006
> [ 8058.036482] GPR16: c000000007012fa0 c000000007012fa4 c000000005160980
> c000000007012f88
> [ 8058.036482] GPR20: c00c000001c3daac c000000d0d10f008 0000000000000001
> ffffffffffffff78
> [ 8058.036482] GPR24: 0000000000000005 c000000d0d58f180 c00000000cd6f580
> c000000d0d10f01c
> [ 8058.036482] GPR28: c000000d0d10f008 c000000d0d10f010 c00000000cd6f588
> 0000000000000000
> [ 8058.036628] NIP [c0000000008b018c] drain_obj_stock+0x620/0xa48
> [ 8058.036646] LR [c0000000008b0180] drain_obj_stock+0x614/0xa48
> [ 8058.036659] Call Trace:
> [ 8058.036665] [c00000000b597960] [c0000000008b0180]
> drain_obj_stock+0x614/0xa48 (unreliable)
> [ 8058.036688] [c00000000b597a10] [c0000000008b2a64]
> refill_obj_stock+0x104/0x680
> [ 8058.036715] [c00000000b597a90] [c0000000008b94b8]
> __memcg_slab_free_hook+0x238/0x3ec
> [ 8058.036738] [c00000000b597b60] [c0000000007f3c10]
> __rcu_free_sheaf_prepare+0x314/0x3e8
> [ 8058.036763] [c00000000b597c10] [c0000000007fbf70]
> rcu_free_sheaf_nobarn+0x38/0x78
> [ 8058.036788] [c00000000b597c40] [c000000000334550]
> rcu_do_batch+0x2ec/0xfa8
> [ 8058.036812] [c00000000b597d40] [c0000000003399e8] rcu_core+0x22c/0x48c
> [ 8058.036835] [c00000000b597db0] [c0000000001cfe6c]
> handle_softirqs+0x1f4/0x74c
> [ 8058.036862] [c00000000b597ed0] [c0000000001d0458] run_ksoftirqd+0x94/0xb8
> [ 8058.036885] [c00000000b597f00] [c00000000022a130]
> smpboot_thread_fn+0x450/0x648
> [ 8058.036912] [c00000000b597f80] [c000000000218408] kthread+0x244/0x28c
> [ 8058.036927] [c00000000b597fe0] [c00000000000ded8]
> start_kernel_thread+0x14/0x18
> [ 8058.036943] Code: 60000000 3bda0008 7fc3f378 4bfb148d 60000000 ebfa0008
> 38800008 7fe3fb78 4bfb2b51 60000000 7c0004ac 39200001 <7d40f8a8> 7d495050
> 7d40f9ad 40c2fff4
> [ 8058.037000] ---[ end trace 0000000000000000 ]---
> 
> 
> And below is the corresponding o/p from faddr2line.

Thanks!

> drain_obj_stock+0x620/0xa48:
> arch_atomic64_sub_return_relaxed at arch/powerpc/include/asm/atomic.h:272
> (inlined by) raw_atomic64_sub_return at
> include/linux/atomic/atomic-arch-fallback.h:2917
> (inlined by) raw_atomic64_sub_and_test at
> include/linux/atomic/atomic-arch-fallback.h:4386
> (inlined by) raw_atomic_long_sub_and_test at
> include/linux/atomic/atomic-long.h:1551
> (inlined by) atomic_long_sub_and_test at
> include/linux/atomic/atomic-instrumented.h:4522
> (inlined by) percpu_ref_put_many at include/linux/percpu-refcount.h:334
> (inlined by) percpu_ref_put at include/linux/percpu-refcount.h:351
> (inlined by) obj_cgroup_put at include/linux/memcontrol.h:794
> (inlined by) drain_obj_stock at mm/memcontrol.c:3059

It seems it crashed while dereferencing objcg->ref->data->count.
I think that implies that obj_cgroup_release()->percpu_ref_exit()
is already called due to the refcount reaching zero and set
ref->data = NULL.

Wait, was the stock->objcg ever a valid objcg?
I think it should be valid when refilling the obj stock, otherwise
it should have crashed in refill_obj_stock() -> obj_cgroup_get() path
in the first place, rather than crashing when draining.

And that sounds like we're somehow calling obj_cgroup_put() more times
than obj_cgroup_get().

Anyway, this is my theory that it may be due to mis-refcounting of objcgs.

> drain_obj_stock+0x614/0xa48:
> instrument_atomic_read_write at include/linux/instrumented.h:112
> (inlined by) atomic_long_sub_and_test at
> include/linux/atomic/atomic-instrumented.h:4521
> (inlined by) percpu_ref_put_many at include/linux/percpu-refcount.h:334
> (inlined by) percpu_ref_put at include/linux/percpu-refcount.h:351
> (inlined by) obj_cgroup_put at include/linux/memcontrol.h:794
> (inlined by) drain_obj_stock at mm/memcontrol.c:3059
> refill_obj_stock+0x104/0x680:

-- 
Cheers,
Harry / Hyeonggon


      reply	other threads:[~2026-02-22 11:47 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <ca241daa-e7e7-4604-a48d-de91ec9184a5@linux.ibm.com>
     [not found] ` <aZReMzl-S9KM_snh@nidhogg.toxiclabs.cc>
2026-02-18 11:36   ` Vlastimil Babka
2026-02-18 21:25     ` Shakeel Butt
2026-02-22 10:08     ` Venkat Rao Bagalkote
2026-02-22 11:47       ` Harry Yoo [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aZrstwhqX6bSpjtz@hyeyoo \
    --to=harry.yoo@oracle.com \
    --cc=cem@kernel.org \
    --cc=cgroups@vger.kernel.org \
    --cc=hannes@cmpxchg.org \
    --cc=hao.li@linux.dev \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-xfs@vger.kernel.org \
    --cc=maddy@linux.ibm.com \
    --cc=mhocko@kernel.org \
    --cc=muchun.song@linux.dev \
    --cc=ojaswin@linux.ibm.com \
    --cc=riteshh@linux.ibm.com \
    --cc=roman.gushchin@linux.dev \
    --cc=shakeel.butt@linux.dev \
    --cc=vbabka@suse.cz \
    --cc=venkat88@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox