From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DF7FC636CC for ; Wed, 15 Feb 2023 10:36:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5521B6B0072; Wed, 15 Feb 2023 05:36:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 502C26B0073; Wed, 15 Feb 2023 05:36:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CB8E6B0074; Wed, 15 Feb 2023 05:36:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2A2EF6B0072 for ; Wed, 15 Feb 2023 05:36:13 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CE604121519 for ; Wed, 15 Feb 2023 10:36:12 +0000 (UTC) X-FDA: 80469171384.18.725558F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 5DAEC160018 for ; Wed, 15 Feb 2023 10:36:10 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CgZ+cYfj; spf=pass (imf08.hostedemail.com: domain of xiubli@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=xiubli@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676457370; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0DULdTVfIZoN5gfe54AIlB0FglUr8CaQBCm49yRWhdM=; b=tUvq+yglHhxX9rUCDxGFucfhhg5ytjeHKYcOipR/qycld12KWA3Us+Dr+1qZWosiypt4VY 9XDwZwzHdHp3+aLDVyNgYSXe8z7F9D10WXZlo2GrbhioDiqqcwdZRDRFGbo1mDptSWtEv8 F0K0aY3Rahneu9hxAcqNZdN9Em59gvY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CgZ+cYfj; spf=pass (imf08.hostedemail.com: domain of xiubli@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=xiubli@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676457370; a=rsa-sha256; cv=none; b=HJqbGId+us1nS6c6WSn2+rFm4inEY2TGWWPr6kr8UybXNXfDd/1fnPHTRTae++uS71J1bM 7UxgmGK8VGifdZGxs4eOTAIMvVZ2vn+85PQK2uX9/vM83zOWRkb0YflLufxTQ34lIUS06l Tp8ol/5fdhmHQJPjTQAd0nHiRW9goOI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676457369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0DULdTVfIZoN5gfe54AIlB0FglUr8CaQBCm49yRWhdM=; b=CgZ+cYfjvhwVCBJ8SgIa1eNrfT6zqK72UN/T4nPtWYtu+4zIjF+/lHu1fK/4iMUjUC3fyC c+FJI30UogSeYIKiacMetXAtVFzK59LuDiHkoKaBqwln1W0RqnAcaaU7EGCknm+A1hQ3lL NSS9yEAQJcsZEqQMPlTZQuT19GGwtac= Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-529-fyUPaH7HMSCNn2n76Aqn5w-1; Wed, 15 Feb 2023 05:36:08 -0500 X-MC-Unique: fyUPaH7HMSCNn2n76Aqn5w-1 Received: by mail-pf1-f200.google.com with SMTP id e9-20020a62ee09000000b0058db55a8d7aso9595457pfi.21 for ; Wed, 15 Feb 2023 02:36:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0DULdTVfIZoN5gfe54AIlB0FglUr8CaQBCm49yRWhdM=; b=tLeYefCIYd7YSfBNNLQXSvB5UJ4WoKobsD9hQIsC4CrM7V8EXTaHsuhdhFQkw8JMyU bPFJtYln29A9OOzA1Lv9NsTsNEud+U4mV2VWyRfnIRx4lqy1L5TqFnNjmEWbqptELsia LmMcbzdMPSogGzhQA8AeY40tduQBVdgcVXK4Vm0SYagbExUMJcbcG80U83eQgqYLiSI1 0B4fj4+XP821BS1D/eST6VFomN+FSZX4TEB1JSfBwpHj0RFMT0GEmJclMnCbLfYRTcR2 DfwEEImwn1SxaEodxMH/+BADf8moYeUjHkvy5QQxDHmjFJPCtI9QhG4n4fYgMNYK0kNU 8DOw== X-Gm-Message-State: AO0yUKWnSKyKtrRg7zqGtA+32Q/difypJi/Z1hihy+C9BEn6mhGdA5Ac HhH2IbgwwbIOQC32uW1KVRwi+lGjxyzHPUV+ZvcXBwYalbTMSMBcPkKV2Ya4mVzOfvKG/ZE3YNi Bt+8Hua8SigM= X-Received: by 2002:a62:6386:0:b0:5a8:d774:b1bb with SMTP id x128-20020a626386000000b005a8d774b1bbmr1242718pfb.17.1676457367428; Wed, 15 Feb 2023 02:36:07 -0800 (PST) X-Google-Smtp-Source: AK7set9LIdiV+BZC34EN4XzgfYunSTZ8ydX5eez16+K7fnvylsGll/d6Y1RhpFXInG2c+mL+K9q5gA== X-Received: by 2002:a62:6386:0:b0:5a8:d774:b1bb with SMTP id x128-20020a626386000000b005a8d774b1bbmr1242709pfb.17.1676457367110; Wed, 15 Feb 2023 02:36:07 -0800 (PST) Received: from [10.72.12.170] ([43.228.180.230]) by smtp.gmail.com with ESMTPSA id j21-20020aa78d15000000b00581c741f95csm11322638pfe.46.2023.02.15.02.36.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Feb 2023 02:36:06 -0800 (PST) Message-ID: <90147a2b-982e-ae57-9b7c-062bee0fab07@redhat.com> Date: Wed, 15 Feb 2023 18:36:02 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Subject: Re: cgroup: deadlock between cpu_hotplug_lock and freezer_mutex To: Hillf Danton Cc: tj@kernel.org, hannes@cmpxchg.org, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20230215072501.3764-1-hdanton@sina.com> From: Xiubo Li In-Reply-To: <20230215072501.3764-1-hdanton@sina.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5DAEC160018 X-Stat-Signature: 361ejt6cowbspa33j7bxsxgrxzc4zqbs X-Rspam-User: X-HE-Tag: 1676457370-807644 X-HE-Meta: U2FsdGVkX184l2hKaIxTr7qIS+sKZP1a/hAlzxwznojLtvIkXXcpGhABpGQiaMLv9GjC8112XigiwwBNA891UNtq3REkKp0lgR55ki/v5+slkkDa25Y+//VFE/xbKW4K1k3WBQdudph8i8zesccoBfjYDsfp/hMZxr4/4mupAM89Aa9Mu4UWUlJ1/DGfCBqIzP1zPqdZXGpBJhwMu4PTZxInS8ttQhC/hWioSrddxixPIDNXnAB3te9wL+mG5OswHHju13kwRPupERrvtFtrLTXmPQrN+OuT6z/e4qrPJyVVqjzIOCLhWS8vVtqiTETPKSXTOtYh6JV8+8R1UIdy7mnSr9Vssk2HnanLDvxSOhupXyghe76RyPPRxSmkILmvtowkxgpODOoYHxQCb5G4g9gM2i+ippYnPhC40cBHsmwahoGa4CBaT/E0uE/86sKm3b0USObwcqElD/bOcM2rImlUSS68YtiLEdaRugVos/Br+JRwZyOiYXcQI6D2hqNdJXo4xfPgVI8UPHSdZUvQUgw8ThbKTUNR4CvirfmyVeXWSRBa4AM6c0wCLKSsBOU8HQ2oh2yUQW37z1nx/c30CwH4h9dR0ADz77/TAYPqXKDXnpKV7YyHjBiG8IMT/2UVJ6+RRSrIK7T501Dgmsw6mEebtznxkb29nUkn54g87vm9st+TqgTGK1x8HRPBghrH30yHaJj/i1ZyULV06qX0iaO64aStLswJuL7vlWcd0xG4ATzSX3bzD3vXqKMc6H/d3rGW/fFRcd4ljt8CJdE62qbSjiBcE8MlrJ20ISTC+zbNMfknAbCrH1dHdK4oPSdYdk8BEjblpzL4PhjOp8fhBcBfGq00TGCeeyCyz5Gw8xg3TCawot1GX0DeqIafbw5Zax+OAe1dK/Nhdq2iV9Zq+kjnhC+KILvZBgZKWVfQisN5/2rG7PY8bclPS/2YbtiMxvtD+ALAtrYqTx2RE6T jb9czt8N kpdet33ECvygdnGGac8EujPiPKdSG6DEV4xwnFCJX2vceACiEmogEi2dQgGofpho63B4WI22aRB7um7U8oRw8MFCkVMr7TwNM9cSEJGstjhCGGRxUukEfUz4J6fSZzhlhw0ZGi0GqoKCYT87P+xyE5fZakFQ+6KF1oh6NSPh3DKJhywwZ8aolE4dVry/spcKSV3SBVsQ9S7x0c3y7/q7Ftk1Ik9aLx4QzLA2O6OUJqnYXHWZn66axPwgtt7xTBgyW5zpJ3gvKm1ptZwyEEgx3YS7B5PubjquwRPoYbsRHDPby1g3WrX9LEZrE8Pm6PAb+chLfokPLCK46m4ZBtmIiAD0EQ5267YTIb/OW7RjqV+SSy1YASjsFZE83AHOkUOd4FyVG3dNp31Se7UiFtPrHSYKGPaOcgjrdlCxAqehPFe+TAChCfOFcqsAhntdsc6uL7sh406jfvq/Pq4Xtno3Y7i8sjBGHWBkbRIrPny3/A3mXAvW8n8r6WvE+FjU6E4qbzGi4tbw/83E99ppyzgxTzrGcX4ijdIByNCBJKUJ6rTuJfCLp7ygQtiI5XQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Hillf, On 15/02/2023 15:25, Hillf Danton wrote: > On Wed, 15 Feb 2023 10:07:23 +0800 Xiubo Li >> Hi >> >> Recently when running some test cases for ceph we hit the following >> deadlock issue in cgroup code. Has this been fixed ? I have checked the >> latest code and it seems no any commit is fixing this. >> >> This call trace could also be found in >> https://tracker.ceph.com/issues/58564#note-4, which is more friendly to >> read. >> >>  ====================================================== >>  WARNING: possible circular locking dependency detected >>  6.1.0-rc5-ceph-gc90f64b588ff #1 Tainted: G S >>  ------------------------------------------------------ >>  runc/90769 is trying to acquire lock: >>  ffffffff82664cb0 (cpu_hotplug_lock){++++}-{0:0}, at: >> static_key_slow_inc+0xe/0x20 >>  #012but task is already holding lock: >>  ffffffff8276e468 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0x89/0x530 >>  #012which lock already depends on the new lock. >>  #012the existing dependency chain (in reverse order) is: >>  #012-> #2 (freezer_mutex){+.+.}-{3:3}: >>        __mutex_lock+0x9c/0xf20 >>        freezer_attach+0x2c/0xf0 >>        cgroup_migrate_execute+0x3f3/0x4c0 >>        cgroup_attach_task+0x22e/0x3e0 >>        __cgroup1_procs_write.constprop.12+0xfb/0x140 >>        cgroup_file_write+0x91/0x230 >>        kernfs_fop_write_iter+0x137/0x1d0 >>        vfs_write+0x344/0x4d0 >>        ksys_write+0x5c/0xd0 >>        do_syscall_64+0x34/0x80 >>        entry_SYSCALL_64_after_hwframe+0x63/0xcd >>  #012-> #1 (cgroup_threadgroup_rwsem){++++}-{0:0}: >>        percpu_down_write+0x45/0x2c0 >>        cgroup_procs_write_start+0x84/0x270 >>        __cgroup1_procs_write.constprop.12+0x57/0x140 >>        cgroup_file_write+0x91/0x230 >>        kernfs_fop_write_iter+0x137/0x1d0 >>        vfs_write+0x344/0x4d0 >>        ksys_write+0x5c/0xd0 >>        do_syscall_64+0x34/0x80 >>        entry_SYSCALL_64_after_hwframe+0x63/0xcd >>  #012-> #0 (cpu_hotplug_lock){++++}-{0:0}: >>        __lock_acquire+0x103f/0x1de0 >>        lock_acquire+0xd4/0x2f0 >>        cpus_read_lock+0x3c/0xd0 >>        static_key_slow_inc+0xe/0x20 >>        freezer_apply_state+0x98/0xb0 >>        freezer_write+0x307/0x530 >>        cgroup_file_write+0x91/0x230 >>        kernfs_fop_write_iter+0x137/0x1d0 >>        vfs_write+0x344/0x4d0 >>        ksys_write+0x5c/0xd0 >>        do_syscall_64+0x34/0x80 >>        entry_SYSCALL_64_after_hwframe+0x63/0xcd >>  #012other info that might help us debug this: >>  Chain exists of:#012  cpu_hotplug_lock --> cgroup_threadgroup_rwsem >> --> freezer_mutex >>  Possible unsafe locking scenario: >>        CPU0                    CPU1 >>        ----                    ---- >>   lock(freezer_mutex); >>                                lock(cgroup_threadgroup_rwsem); >>                                lock(freezer_mutex); >>   lock(cpu_hotplug_lock); >>  #012 *** DEADLOCK *** > Thanks for your report. > > Change locking order if it is impossible to update freezer_active in atomic manner. > > Only for thoughts. Sure, I will test this. Thanks > > Hillf > +++ linux-6.1.3/kernel/cgroup/legacy_freezer.c > @@ -350,7 +350,7 @@ static void freezer_apply_state(struct f > > if (freeze) { > if (!(freezer->state & CGROUP_FREEZING)) > - static_branch_inc(&freezer_active); > + static_branch_inc_cpuslocked(&freezer_active); > freezer->state |= state; > freeze_cgroup(freezer); > } else { > @@ -361,7 +361,7 @@ static void freezer_apply_state(struct f > if (!(freezer->state & CGROUP_FREEZING)) { > freezer->state &= ~CGROUP_FROZEN; > if (was_freezing) > - static_branch_dec(&freezer_active); > + static_branch_dec_cpuslocked(&freezer_active); > unfreeze_cgroup(freezer); > } > } > @@ -379,6 +379,7 @@ static void freezer_change_state(struct > { > struct cgroup_subsys_state *pos; > > + cpus_read_lock(); > /* > * Update all its descendants in pre-order traversal. Each > * descendant will try to inherit its parent's FREEZING state as > @@ -407,6 +408,7 @@ static void freezer_change_state(struct > } > rcu_read_unlock(); > mutex_unlock(&freezer_mutex); > + cpus_read_unlock(); > } > > static ssize_t freezer_write(struct kernfs_open_file *of, > -- Best Regards, Xiubo Li (李秀波) Email: xiubli@redhat.com/xiubli@ibm.com Slack: @Xiubo Li