From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF7B4C433B4 for ; Tue, 4 May 2021 12:01:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E486611C0 for ; Tue, 4 May 2021 12:01:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E486611C0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 451226B0036; Tue, 4 May 2021 08:01:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FE0D6B006E; Tue, 4 May 2021 08:01:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF6AC6B0070; Tue, 4 May 2021 08:01:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id C4F806B0036 for ; Tue, 4 May 2021 08:01:48 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 74DC9824999B for ; Tue, 4 May 2021 12:01:48 +0000 (UTC) X-FDA: 78103409496.37.F036B39 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf07.hostedemail.com (Postfix) with ESMTP id F390EA0002FE for ; Tue, 4 May 2021 12:01:45 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 9D21CB29A; Tue, 4 May 2021 12:01:46 +0000 (UTC) Subject: Re: [PATCH 5.13] mm, slub: move slub_debug static key enabling outside slab_mutex To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, David Rientjes , Joonsoo Kim , Christoph Lameter , Pekka Enberg , "Paul E . McKenney" References: <20210504120019.26791-1-vbabka@suse.cz> From: Vlastimil Babka Message-ID: Date: Tue, 4 May 2021 14:01:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210504120019.26791-1-vbabka@suse.cz> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: F390EA0002FE X-Stat-Signature: dbgbxrn6hr58ojaafr7uexgu1nd5z1fu Received-SPF: none (suse.cz>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1620129705-283124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/4/21 2:00 PM, Vlastimil Babka wrote: > Paul E. McKenney reported [1] that commit 1f0723a4c0df ("mm, slub: enable > slub_debug static key when creating cache with explicit debug flags") results > in the lockdep complaint: > > ====================================================== > WARNING: possible circular locking dependency detected > 5.12.0+ #15 Not tainted > ------------------------------------------------------ > rcu_torture_sta/109 is trying to acquire lock: > ffffffff96063cd0 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_enable+0x9/0x20 > > but task is already holding lock: > ffffffff96173c28 (slab_mutex){+.+.}-{3:3}, at: kmem_cache_create_usercopy+0x2d/0x250 > > which lock already depends on the new lock. > > the existing dependency chain (in reverse order) is: > > -> #1 (slab_mutex){+.+.}-{3:3}: > lock_acquire+0xb9/0x3a0 > __mutex_lock+0x8d/0x920 > slub_cpu_dead+0x15/0xf0 > cpuhp_invoke_callback+0x17a/0x7c0 > cpuhp_invoke_callback_range+0x3b/0x80 > _cpu_down+0xdf/0x2a0 > cpu_down+0x2c/0x50 > device_offline+0x82/0xb0 > remove_cpu+0x1a/0x30 > torture_offline+0x80/0x140 > torture_onoff+0x147/0x260 > kthread+0x10a/0x140 > ret_from_fork+0x22/0x30 > > -> #0 (cpu_hotplug_lock){++++}-{0:0}: > check_prev_add+0x8f/0xbf0 > __lock_acquire+0x13f0/0x1d80 > lock_acquire+0xb9/0x3a0 > cpus_read_lock+0x21/0xa0 > static_key_enable+0x9/0x20 > __kmem_cache_create+0x38d/0x430 > kmem_cache_create_usercopy+0x146/0x250 > kmem_cache_create+0xd/0x10 > rcu_torture_stats+0x79/0x280 > kthread+0x10a/0x140 > ret_from_fork+0x22/0x30 > > other info that might help us debug this: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(slab_mutex); > lock(cpu_hotplug_lock); > lock(slab_mutex); > lock(cpu_hotplug_lock); > > *** DEADLOCK *** > > 1 lock held by rcu_torture_sta/109: > #0: ffffffff96173c28 (slab_mutex){+.+.}-{3:3}, at: kmem_cache_create_usercopy+0x2d/0x250 > > stack backtrace: > CPU: 3 PID: 109 Comm: rcu_torture_sta Not tainted 5.12.0+ #15 > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1.1 04/01/2014 > Call Trace: > dump_stack+0x6d/0x89 > check_noncircular+0xfe/0x110 > ? lock_is_held_type+0x98/0x110 > check_prev_add+0x8f/0xbf0 > __lock_acquire+0x13f0/0x1d80 > lock_acquire+0xb9/0x3a0 > ? static_key_enable+0x9/0x20 > ? mark_held_locks+0x49/0x70 > cpus_read_lock+0x21/0xa0 > ? static_key_enable+0x9/0x20 > static_key_enable+0x9/0x20 > __kmem_cache_create+0x38d/0x430 > kmem_cache_create_usercopy+0x146/0x250 > ? rcu_torture_stats_print+0xd0/0xd0 > kmem_cache_create+0xd/0x10 > rcu_torture_stats+0x79/0x280 > ? rcu_torture_stats_print+0xd0/0xd0 > kthread+0x10a/0x140 > ? kthread_park+0x80/0x80 > ret_from_fork+0x22/0x30 > > This is because there's one order of locking from the hotplug callbacks: > > lock(cpu_hotplug_lock); // from hotplug machinery itself > lock(slab_mutex); // in e.g. slab_mem_going_offline_callback() > > And commit 1f0723a4c0df made the reverse sequence possible: > lock(slab_mutex); // in kmem_cache_create_usercopy() > lock(cpu_hotplug_lock); // kmem_cache_open() -> static_key_enable() > > The simplest fix is to move static_key_enable() to a place before slab_mutex is > taken. That means kmem_cache_create_usercopy() in mm/slab_common.c which is not > ideal for SLUB-specific code, but the #ifdef CONFIG_SLUB_DEBUG makes it > at least self-contained and obvious. > > [1] https://lore.kernel.org/lkml/20210502171827.GA3670492@paulmck-ThinkPad-P17-Gen-1/ > > Reported-and-tested-by: Paul E. McKenney Forgot: Fixes: 1f0723a4c0df ("mm, slub: enable slub_debug static key when creating cache with explicit debug flags")