From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 571FCC00140 for ; Tue, 2 Aug 2022 06:59:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D0E96B0071; Tue, 2 Aug 2022 02:59:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 981136B0072; Tue, 2 Aug 2022 02:59:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84A2A8E0001; Tue, 2 Aug 2022 02:59:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7591D6B0071 for ; Tue, 2 Aug 2022 02:59:15 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 40D13160DAB for ; Tue, 2 Aug 2022 06:59:15 +0000 (UTC) X-FDA: 79753751070.03.635EEF3 Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by imf11.hostedemail.com (Postfix) with ESMTP id C4CBB40119 for ; Tue, 2 Aug 2022 06:59:14 +0000 (UTC) Received: by mail-lf1-f52.google.com with SMTP id w15so20587517lft.11 for ; Mon, 01 Aug 2022 23:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=O6Pyr8Fl9p6t8WEJ0ab3NlyF43bwbAF4/cHexI37Gdo=; b=aCdRHTiQ8U0jx6qDc190LOoAw5sHDUHzciS9Upg01AyVmQ3HR6FSRfA/Ko9v6jVNPP eK3zTUtJUlr3RDx1VxFqZbWM5wnbD6+QQXRyRr8bjEBtb6G/r7rr8uD/XARPSv3xvz/W 8TG+KK9QMQutxlVUcaijJTJrZo9nvw23r8T3aNdW1fCfkrXjFeioYMAGl5FPDgECc4YK gx0AlEcA4nexTHSfk4UnfJ2uSZe+6OlXM/Y2n35ltywITs9PS8vS79RwE9vTOX8nDaKE aslI3rd0drLelOpceo+6vd7aBWsF8BwSuEJHyBT4hssQltycmU7YWHfcUO0LuZqXRdh/ NUUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=O6Pyr8Fl9p6t8WEJ0ab3NlyF43bwbAF4/cHexI37Gdo=; b=KU7ATX+oDPrQP9hZxbWayTw3YJEWBDKq3VkA7wKBPZp8fJZ8jHGoytTv2O3Fnk10Pz Fa86m+9HfrureAquB2mwflLhvQbg0jf/ccYeMuZp4YBBV97SyN9jF9dpQZgQJawbK8SC IQiNTvkPXMABXu1F18re/BmgTlAxMjn/oadUupVL/5MJKBJCfTlqkD3ONql2jxexBu/X 9DHcvlIjGF1D2SL4xu8g8CnrIzgKPqmndHPVix3R3ej/mEvHbdEuHbR5bBYy1uAflO0g dze77VXUiYIg3qwiWd9YSJmP01BK/CrlKpbScT3sM8H/C7HZJuTue5cjJMam/fKCET0b zAHg== X-Gm-Message-State: ACgBeo38hxjmNsZlNgpzbPD9PGwnNqtaL44JuBUTDNNs/3gUgb0tSQby tx0m/sDwOeGyUrY6FO4leKfsmhga8HDxc4TbOc49pg== X-Google-Smtp-Source: AA6agR5xPkwqFqAX2DyQ0b+8dG0j9wX07SPTEbTX2MtGtP6oDYYKx01Gr7vN/0E7iru0YEDldU8RrsycwCIljAveEbM= X-Received: by 2002:a05:6512:c0b:b0:48a:f3e5:4b3e with SMTP id z11-20020a0565120c0b00b0048af3e54b3emr3439438lfu.410.1659423552972; Mon, 01 Aug 2022 23:59:12 -0700 (PDT) MIME-Version: 1.0 References: <20220727071042.8796-4-feng.tang@intel.com> <0e545088-d140-4c84-bbb2-a3be669740b2@suse.cz> In-Reply-To: <0e545088-d140-4c84-bbb2-a3be669740b2@suse.cz> From: Dmitry Vyukov Date: Tue, 2 Aug 2022 08:59:01 +0200 Message-ID: Subject: Re: [mm/slub] 3616799128: BUG_kmalloc-#(Not_tainted):kmalloc_Redzone_overwritten To: Vlastimil Babka Cc: Feng Tang , "Sang, Oliver" , lkp , LKML , "linux-mm@kvack.org" , "lkp@lists.01.org" , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Hansen, Dave" , Robin Murphy , John Garry , Kefeng Wang , Andrey Konovalov , Andrey Ryabinin , Alexander Potapenko , kasan-dev@googlegroups.com Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659423554; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O6Pyr8Fl9p6t8WEJ0ab3NlyF43bwbAF4/cHexI37Gdo=; b=2TVLjCaHlHQQw3GaXh8TvRa1iZzh0f1fx+2inCiCqNxnYv7aqonzGggNmB74n//9IDHDoD jiQHNoB7lrOo/GjKuFFZaKWilcM7gSp2+4Y1JSvHXMUirtWH+MozV6Xcnc/bumRAodR4mF VljFRrkyejenLRjw5UyJWn9DDlAQsws= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aCdRHTiQ; spf=pass (imf11.hostedemail.com: domain of dvyukov@google.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=dvyukov@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659423554; a=rsa-sha256; cv=none; b=NZQfCtFru07sZny29fAWx3KswHs8B2W8FglVwkRRTvw+NYqEYNhHszEcxnA6HN497hSj0n 8tYUy2dryWruk9dJ4dPzp+8FrQyPQUfSWd80z+3Da40fEKKioRSVF+rSnkvVUzSTeTCt2f 3bWvnXIWZ+wluuO098ozUneFs3gBfUA= X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aCdRHTiQ; spf=pass (imf11.hostedemail.com: domain of dvyukov@google.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=dvyukov@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: pin7d8p9etdgnhwwhn5z66s7f8r7eu6k X-Rspamd-Queue-Id: C4CBB40119 X-Rspamd-Server: rspam10 X-HE-Tag: 1659423554-185758 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 1 Aug 2022 at 16:23, Vlastimil Babka wrote: > > On 8/1/22 08:21, Feng Tang wrote: > > On Sun, Jul 31, 2022 at 04:16:53PM +0800, Tang, Feng wrote: > >> Hi Oliver, > >> > >> On Sun, Jul 31, 2022 at 02:53:17PM +0800, Sang, Oliver wrote: > >> > > >> > > >> > Greeting, > >> > > >> > FYI, we noticed the following commit (built with gcc-11): > >> > > >> > commit: 3616799128612e04ed919579e2c7b0dccf6bcb00 ("[PATCH v3 3/3] mm/slub: extend redzone check to cover extra allocated kmalloc space than requested") > >> > url: https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-slub-some-debug-enhancements/20220727-151318 > >> > base: git://git.kernel.org/cgit/linux/kernel/git/vbabka/slab.git for-next > >> > patch link: https://lore.kernel.org/linux-mm/20220727071042.8796-4-feng.tang@intel.com > >> > > >> > in testcase: boot > >> > > >> > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G > >> > > >> > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): > >> > > >> > > >> > If you fix the issue, kindly add following tag > >> > Reported-by: kernel test robot > >> > > >> > > >> > [ 50.637839][ T154] ============================================================================= > >> > [ 50.639937][ T154] BUG kmalloc-16 (Not tainted): kmalloc Redzone overwritten > >> > [ 50.641291][ T154] ----------------------------------------------------------------------------- > >> > [ 50.641291][ T154] > >> > [ 50.643617][ T154] 0xffff88810018464c-0xffff88810018464f @offset=1612. First byte 0x7 instead of 0xcc > >> > [ 50.645311][ T154] Allocated in __sdt_alloc+0x258/0x457 age=14287 cpu=0 pid=1 > >> > [ 50.646584][ T154] ___slab_alloc+0x52b/0x5b6 > >> > [ 50.647411][ T154] __slab_alloc+0x1a/0x22 > >> > [ 50.648374][ T154] __kmalloc_node+0x10c/0x1e1 > >> > [ 50.649237][ T154] __sdt_alloc+0x258/0x457 > >> > [ 50.650060][ T154] build_sched_domains+0xae/0x10e8 > >> > [ 50.650981][ T154] sched_init_smp+0x30/0xa5 > >> > [ 50.651805][ T154] kernel_init_freeable+0x1c6/0x23b > >> > [ 50.652767][ T154] kernel_init+0x14/0x127 > >> > [ 50.653594][ T154] ret_from_fork+0x1f/0x30 > >> > [ 50.654414][ T154] Slab 0xffffea0004006100 objects=28 used=28 fp=0x0000000000000000 flags=0x1fffc0000000201(locked|slab|node=0|zone=1|lastcpupid=0x3fff) > >> > [ 50.656866][ T154] Object 0xffff888100184640 @offset=1600 fp=0xffff888100184520 > >> > [ 50.656866][ T154] > >> > [ 50.658410][ T154] Redzone ffff888100184630: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ > >> > [ 50.660047][ T154] Object ffff888100184640: 00 32 80 00 81 88 ff ff 01 00 00 00 07 00 80 8a .2.............. > >> > [ 50.661837][ T154] Redzone ffff888100184650: cc cc cc cc cc cc cc cc ........ > >> > [ 50.663454][ T154] Padding ffff8881001846b4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ > >> > [ 50.665225][ T154] CPU: 0 PID: 154 Comm: systemd-udevd Not tainted 5.19.0-rc5-00010-g361679912861 #1 > >> > [ 50.666861][ T154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-4 04/01/2014 > >> > [ 50.668694][ T154] Call Trace: > >> > [ 50.669331][ T154] > >> > [ 50.669832][ T154] dump_stack_lvl+0x57/0x7d > >> > [ 50.670601][ T154] check_bytes_and_report+0xca/0xfe > >> > [ 50.671436][ T154] check_object+0xdc/0x24d > >> > [ 50.672163][ T154] free_debug_processing+0x98/0x210 > >> > [ 50.673904][ T154] __slab_free+0x46/0x198 > >> > [ 50.675746][ T154] qlist_free_all+0xae/0xde > >> > [ 50.676552][ T154] kasan_quarantine_reduce+0x10d/0x145 > >> > [ 50.677507][ T154] __kasan_slab_alloc+0x1c/0x5a > >> > [ 50.678327][ T154] slab_post_alloc_hook+0x5a/0xa2 > >> > [ 50.680069][ T154] kmem_cache_alloc+0x102/0x135 > >> > [ 50.680938][ T154] getname_flags+0x4b/0x314 > >> > [ 50.681781][ T154] do_sys_openat2+0x7a/0x15c > >> > [ 50.706848][ T154] Disabling lock debugging due to kernel taint > >> > [ 50.707913][ T154] FIX kmalloc-16: Restoring kmalloc Redzone 0xffff88810018464c-0xffff88810018464f=0xcc > >> > >> Thanks for the report! > >> > >> From the log it happened when kasan is enabled, and my first guess is > >> the data processing from kmalloc redzone handling had some conflict > >> with kasan's in allocation path (though I tested some kernel config > >> with KASAN enabled) > >> > >> Will study more about kasan and reproduce/debug this. thanks > > > > Cc kansan mail list. > > > > This is really related with KASAN debug, that in free path, some > > kmalloc redzone ([orig_size+1, object_size]) area is written by > > kasan to save free meta info. > > > > The callstack is: > > > > kfree > > slab_free > > slab_free_freelist_hook > > slab_free_hook > > __kasan_slab_free > > ____kasan_slab_free > > kasan_set_free_info > > kasan_set_track > > > > And this issue only happens with "kmalloc-16" slab. Kasan has 2 > > tracks: alloc_track and free_track, for x86_64 test platform, most > > of the slabs will reserve space for alloc_track, and reuse the > > 'object' area for free_track. The kasan free_track is 16 bytes > > large, that it will occupy the whole 'kmalloc-16's object area, > > so when kmalloc-redzone is enabled by this patch, the 'overwritten' > > error is triggered. > > > > But it won't hurt other kmalloc slabs, as kasan's free meta won't > > conflict with kmalloc-redzone which stay in the latter part of > > kmalloc area. > > > > So the solution I can think of is: > > * skip the kmalloc-redzone for kmalloc-16 only, or > > * skip kmalloc-redzone if kasan is enabled, or > > * let kasan reserve the free meta (16 bytes) outside of object > > just like for alloc meta > > Maybe we could add some hack that if both kasan and SLAB_STORE_USER is > enabled, we bump the stored orig_size from <16 to 16? Similar to what > __ksize() does. This looks like the simplest workaround. And with a proper comment I think it's fine. > > I don't have way to test kasan's SW/HW tag configuration, which > > is only enabled on arm64 now. And I don't know if there will > > also be some conflict. > > > > Thanks, > > Feng > > >