From: "John Thomson" <lists@johnthomson.fastmail.com.au>
To: "Feng Tang" <feng.tang@intel.com>, "Vlastimil Babka" <vbabka@suse.cz>
Cc: "Andrew Morton" <akpm@linux-foundation.org>,
"Christoph Lameter" <cl@linux.com>,
"Pekka Enberg" <penberg@kernel.org>,
"David Rientjes" <rientjes@google.com>,
"Joonsoo Kim" <iamjoonsoo.kim@lge.com>,
"Roman Gushchin" <roman.gushchin@linux.dev>,
"Hyeonggon Yoo" <42.hyeyoo@gmail.com>,
"Dmitry Vyukov" <dvyukov@google.com>,
"Jonathan Corbet" <corbet@lwn.net>,
"Andrey Konovalov" <andreyknvl@gmail.com>,
"Hansen, Dave" <dave.hansen@intel.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"kasan-dev@googlegroups.com" <kasan-dev@googlegroups.com>,
"Robin Murphy" <robin.murphy@arm.com>,
"John Garry" <john.garry@huawei.com>,
"Kefeng Wang" <wangkefeng.wang@huawei.com>
Subject: Re: [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc
Date: Mon, 31 Oct 2022 10:05:58 +0000 [thread overview]
Message-ID: <c4285caf-277c-45fd-8fc7-8a1d61685ce8@app.fastmail.com> (raw)
In-Reply-To: <Y180l6zUnNjdCoaE@feng-clx>
[-- Attachment #1: Type: text/plain, Size: 6532 bytes --]
On Mon, 31 Oct 2022, at 02:36, Feng Tang wrote:
> Hi John,
>
> Thanks for the bisecting and reporting!
>
> On Mon, Oct 31, 2022 at 05:30:24AM +0800, Vlastimil Babka wrote:
>> On 10/30/22 20:23, John Thomson wrote:
>> > On Tue, 13 Sep 2022, at 06:54, Feng Tang wrote:
>> >> kmalloc's API family is critical for mm, with one nature that it will
>> >> round up the request size to a fixed one (mostly power of 2). Say
>> >> when user requests memory for '2^n + 1' bytes, actually 2^(n+1) bytes
>> >> could be allocated, so in worst case, there is around 50% memory
>> >> space waste.
>> >
>> >
>> > I have a ralink mt7621 router running Openwrt, using the mips ZBOOT kernel, and appear to have bisected
>> > a very-nearly-clean kernel v6.1rc-2 boot issue to this commit.
>> > I have 3 commits atop 6.1-rc2: fix a ZBOOT compile error, use the Openwrt LZMA options,
>> > and enable DEBUG_ZBOOT for my platform. I am compiling my kernel within the Openwrt build system.
>> > No guarantees this is not due to something I am doing wrong, but any insight would be greatly appreciated.
>> >
>> >
>> > On UART, No indication of the (once extracted) kernel booting:
>> >
>> > transfer started ......................................... transfer ok, time=2.01s
>> > setting up elf image... OK
>> > jumping to kernel code
>> > zimage at: 80BA4100 810D4720
>> > Uncompressing Linux at load address 80001000
>> > Copy device tree to address 80B96EE0
>> > Now, booting the kernel...
>>
>> It's weird that the commit would cause no output so early, SLUB code is
>> run only later.
>
> I noticed your cmdline has console setting, could you enable the
> earlyprintk in cmdline like "earlyprintk=ttyS0,115200" etc to see
> if there is more message printed out.
Still nothing from vmlinux with earlykprint on UART unless revert.
>
> Also I want to confirm this is a boot failure and not only a boot
> message missing.
Yes, boot failure.
Network comes up automatically on successful boot. Not happening when no kernel UART
>
>> > Nothing follows
>> >
>> > 6edf2576a6cc ("mm/slub: enable debugging memory wasting of kmalloc") reverted, normal boot:
>> > transfer started ......................................... transfer ok, time=2.01s
>> > setting up elf image... OK
>> > jumping to kernel code
>> > zimage at: 80BA4100 810D47A4
>> > Uncompressing Linux at load address 80001000
>> > Copy device tree to address 80B96EE0
>> > Now, booting the kernel...
>> >
>> > [ 0.000000] Linux version 6.1.0-rc2 (john@john) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.3.0 r19724+16-1521d5f453) 11.3.0, GNU ld (GNU Binutils) 2.37) #0 SMP Fri Oct 28 03:48:10 2022
>> > [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
>> > [ 0.000000] printk: bootconsole [early0] enabled
>> > [ 0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
>> > [ 0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS
>> > [ 0.000000] Initrd not found or empty - disabling initrd
>> > [ 0.000000] VPE topology {2,2} total 4
>> > [ 0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
>> > [ 0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
>> > [ 0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
>> > [ 0.000000] Zone ranges:
>> > [ 0.000000] Normal [mem 0x0000000000000000-0x000000000fffffff]
>> > [ 0.000000] HighMem empty
>> > [ 0.000000] Movable zone start for each node
>> > [ 0.000000] Early memory node ranges
>> > [ 0.000000] node 0: [mem 0x0000000000000000-0x000000000fffffff]
>> > [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
>> > [ 0.000000] percpu: Embedded 11 pages/cpu s16064 r8192 d20800 u45056
>> > [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 64960
>> > [ 0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
>> > [ 0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
>> > [ 0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
>> > [ 0.000000] Writing ErrCtl register=00019146
>> > [ 0.000000] Readback ErrCtl register=00019146
>> > [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
>> > [ 0.000000] Memory: 246220K/262144K available (7455K kernel code, 628K rwdata, 1308K rodata, 3524K init, 245K bss, 15924K reserved, 0K cma-reserved, 0K highmem)
>> > [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>> > [ 0.000000] rcu: Hierarchical RCU implementation.
>> >
>> >
>> > boot continues as expected
>> >
>> >
>> > possibly relevant config options:
>> > grep -E '(SLUB|SLAB)' .config
>> > # SLAB allocator options
>> > # CONFIG_SLAB is not set
>> > CONFIG_SLUB=y
>> > CONFIG_SLAB_MERGE_DEFAULT=y
>> > # CONFIG_SLAB_FREELIST_RANDOM is not set
>> > # CONFIG_SLAB_FREELIST_HARDENED is not set
>> > # CONFIG_SLUB_STATS is not set
>> > CONFIG_SLUB_CPU_PARTIAL=y
>> > # end of SLAB allocator options
>> > # CONFIG_SLUB_DEBUG is not set
>>
>> Also not having CONFIG_SLUB_DEBUG enabled means most of the code the
>> patch/commit touches is not even active.
>> Could this be some miscompile or code layout change exposing some
>> different bug, hmm.
Yes, it could be.
>> Is it any different if you do enable CONFIG_SLUB_DEBUG ?
No change
>> Or change to CONFIG_SLAB? (that would be really weird if not)
This boots fine
> I haven't found any clue from the code either, and I compiled
> kernel with the config above and tested booting on an Alder-lake
> desktop and a QEMU, which boot fine.
>
> Could you provide the full kernel config and demsg (in compressed
> format if you think it's too big), so we can check more?
Attached
> Thanks,
> Feng
vmlinux is bigger, and entry point is larger (0x8074081c vs 0x807407dc revert vs 0x8073fcbc),
so that may be it? Or not?
revert + SLUB_DEBUG + SLUB_DEBUG_ON is bigger still, but does successfully boot.
vmlinux entry point is 0x8074705c
transfer started ......................................... transfer ok, time=2.01s
setting up elf image... OK
jumping to kernel code
zimage at: 80BA4100 810D6FA0
Uncompressing Linux at load address 80001000
Copy device tree to address 80B9EEE0
Now, booting the kernel...
[ 0.000000] Linux version 6.1.0-rc2 (john@john) (mipsel-openwrt-linux-musl-gc
c (OpenWrt GCC 11.3.0 r19724+16-1521d5f453) 11.3.0, GNU ld (GNU Binutils) 2.37)
#0 SMP Fri Oct 28 03:48:10 2022
I will keep looking.
Thank you,
--
John Thomson
[-- Attachment #2: mt7621-mm-waste-debug.tar.zstd --]
[-- Type: application/octet-stream, Size: 29758 bytes --]
next prev parent reply other threads:[~2022-10-31 10:06 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-13 6:54 [PATCH v6 0/4] mm/slub: some debug enhancements for kmalloc Feng Tang
2022-09-13 6:54 ` [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc Feng Tang
2022-09-23 11:43 ` Vlastimil Babka
2022-09-24 7:08 ` Feng Tang
2022-10-30 19:23 ` John Thomson
2022-10-30 21:30 ` Vlastimil Babka
2022-10-31 2:36 ` Feng Tang
2022-10-31 10:05 ` John Thomson [this message]
2022-10-31 11:36 ` Hyeonggon Yoo
2022-10-31 11:42 ` Feng Tang
2022-11-01 0:18 ` John Thomson
2022-11-01 2:41 ` John Thomson
2022-11-01 7:57 ` Feng Tang
2022-11-01 9:20 ` John Thomson
2022-11-01 9:31 ` Hyeonggon Yoo
2022-11-01 10:33 ` John Thomson
2022-11-01 10:42 ` Hyeonggon Yoo
2022-11-01 13:55 ` Feng Tang
2022-11-01 19:39 ` John Thomson
2022-11-02 6:08 ` Feng Tang
2022-11-02 7:16 ` Hyeonggon Yoo
2022-11-03 7:18 ` Feng Tang
2022-11-03 7:45 ` John Thomson
2022-11-03 8:16 ` Feng Tang
2022-11-02 8:22 ` Vlastimil Babka
2022-11-03 5:54 ` Feng Tang
2022-11-03 8:33 ` Vlastimil Babka
2022-11-03 14:16 ` Feng Tang
2022-11-03 14:36 ` Hyeonggon Yoo
2022-11-03 16:57 ` Vlastimil Babka
2022-11-03 17:35 ` Vlastimil Babka
2022-11-04 3:52 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 2/4] mm/slub: only zero the requested size of buffer for kzalloc Feng Tang
2022-09-26 19:11 ` Andrey Konovalov
2022-09-26 20:15 ` Kees Cook
2022-09-27 1:22 ` Feng Tang
2022-09-27 2:42 ` Feng Tang
2022-10-13 14:00 ` Andrey Konovalov
2022-10-14 5:59 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 3/4] mm: kasan: Add free_meta size info in struct kasan_cache Feng Tang
2022-09-20 19:20 ` Andrey Konovalov
2022-09-21 12:02 ` Feng Tang
2022-09-24 18:05 ` Andrey Konovalov
2022-09-25 11:26 ` Feng Tang
2022-09-25 16:31 ` Andrey Konovalov
2022-09-27 3:03 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 4/4] mm/slub: extend redzone check to extra allocated kmalloc space than requested Feng Tang
2022-09-13 8:53 ` Hyeonggon Yoo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c4285caf-277c-45fd-8fc7-8a1d61685ce8@app.fastmail.com \
--to=lists@johnthomson.fastmail.com.au \
--cc=42.hyeyoo@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=andreyknvl@gmail.com \
--cc=cl@linux.com \
--cc=corbet@lwn.net \
--cc=dave.hansen@intel.com \
--cc=dvyukov@google.com \
--cc=feng.tang@intel.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=john.garry@huawei.com \
--cc=kasan-dev@googlegroups.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=robin.murphy@arm.com \
--cc=roman.gushchin@linux.dev \
--cc=vbabka@suse.cz \
--cc=wangkefeng.wang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox