From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDD6DCCA468 for ; Wed, 1 Jun 2022 14:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62B408D000E; Wed, 1 Jun 2022 10:36:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D8F08D001C; Wed, 1 Jun 2022 10:36:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F8808D000E; Wed, 1 Jun 2022 10:36:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 42E838D000E for ; Wed, 1 Jun 2022 10:36:26 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1D93432307 for ; Wed, 1 Jun 2022 14:36:26 +0000 (UTC) X-FDA: 79529917572.30.76D7D60 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id A54F94005D for ; Wed, 1 Jun 2022 14:36:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654094180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ndVFt2OYBW8cSRXpIjiijVDbpZaQ/wTXyFWQt9Vv3wA=; b=NUs4K+sujqm0Hp5wL9cp92l9FzTX7FxnIZTwyoAhtaxDZFNodE6lNQYDQptucIfKqoOGgB d0TKwG5k53Xj9GCylOM4dcZqUy648OXGxeOVSUFO/EgG8Rwr9aM+FRCutSkl2BHKbjyGbW RDTUeK0LwpWQYOtQllJdolxQHb8FZ+k= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-65-w4erR0F1P7i3TUONlg_KjA-1; Wed, 01 Jun 2022 10:36:19 -0400 X-MC-Unique: w4erR0F1P7i3TUONlg_KjA-1 Received: by mail-ed1-f72.google.com with SMTP id g3-20020a056402320300b0042dc956d80eso1390610eda.14 for ; Wed, 01 Jun 2022 07:36:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=ndVFt2OYBW8cSRXpIjiijVDbpZaQ/wTXyFWQt9Vv3wA=; b=PFNZ7lE5QqETSd9EM46cn8+eItH/sfeSKi2oi9I/qmwAWY5QUKntVlPtwwzacxDkBS raotsDcqdrEfgdJoZayjKLCwBR4YcZPt98Q0kXRCTIyZmSRJU6rBTtqiJNl/0VY0Eo49 0jAZGsSzwPPyksG4oIXf3OcGqEnLTiOr2r3+Mf3obvRonT88IOqYOe5lRwu3yTgB+g2F hlgowMNSBoIwnm5IjUqIKupQmj2ULTfJDbz2jk4UoX76/ppWGJSLRdOjNzrQepDiaLt7 Qph4U0RvUm10bTnPHZz9UOxSS0USHDFcd51qv8K8adj0kSQ4CcJQDxts5ZiEt7MgXk2S JTqA== X-Gm-Message-State: AOAM533gaTh78raUDhuxwLvS5cxzwVXbwQv0fOPuqE95lPSHzvGh+x9l /oqiiOKJ4jfLOgvm3oWoej5n3gXPaf3acxzja9JwMvAq9kMBc0vBT0TPcliFZO9KdoJ4pWGVPda T3HasCRvpEM4WS6vHABdJbv4o+50= X-Received: by 2002:a05:6402:6cc:b0:42d:bd2d:9f82 with SMTP id n12-20020a05640206cc00b0042dbd2d9f82mr237696edy.59.1654094177795; Wed, 01 Jun 2022 07:36:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyFh8H56y/l8Ke6nyDbJQA0j9RiyCmE3CPbXmuv5QhyDCmnY54KlRZlTjdvlq4rdEAuPxbDDSvggkS7C+zAZow= X-Received: by 2002:a05:6402:6cc:b0:42d:bd2d:9f82 with SMTP id n12-20020a05640206cc00b0042dbd2d9f82mr237650edy.59.1654094177475; Wed, 01 Jun 2022 07:36:17 -0700 (PDT) MIME-Version: 1.0 References: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com> In-Reply-To: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com> From: Maurizio Lombardi Date: Wed, 1 Jun 2022 16:36:06 +0200 Message-ID: Subject: Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy To: Xin Long Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Sebastian Andrzej Siewior , Antoine Tenart X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NUs4K+su; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf27.hostedemail.com: domain of mlombard@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=mlombard@redhat.com X-Stat-Signature: n4bmhxseuj37zbqjaxdkbjgtnnyqnawt X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A54F94005D X-HE-Tag: 1654094176-114289 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: p=C3=A1 14. 1. 2022 v 15:23 odes=C3=ADlatel Xin Long = napsal: > > cpus_read_lock() is introduced into kmem_cache_destroy() by > commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations > __free_slab() invocations out of IRQ context"), and it could cause > a deadlock. FYI, I received a bug report from one of our customers, he complains that his system (with nvmefc boot from SAN) hangs when rebooting. He runs a RHEL-9 kernel based on version 5.14.0. What is interesting is that, according to him, after reverting commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context") the reboot operation doesn't hang anymore. The call trace seems to point to a possible problem due to the fact that nvme_delete_ctrl_work is allocated with the WQ_MEM_RECLAIM bit set. [ 453.012078] ------------[ cut here ]------------ [ 453.016744] workqueue: WQ_MEM_RECLAIM nvme-delete-wq:nvme_delete_ctrl_work [nvme_core] is flushing !WQ_MEM_RECLAIM events:flush_cpu_slab [ 453.016789] WARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637 check_flush_dependency+0x10a/0x120 [...] [ 453.262125] Call Trace: [ 453.264582] __flush_work.isra.0+0xbf/0x220 [ 453.268775] ? __queue_work+0x1dc/0x420 [ 453.272623] flush_all_cpus_locked+0xfb/0x120 [ 453.276992] __kmem_cache_shutdown+0x2b/0x320 [ 453.281361] kmem_cache_destroy+0x49/0x100 [ 453.285465] bioset_exit+0x143/0x190 [ 453.289052] blk_release_queue+0xb9/0x100 [ 453.293075] kobject_cleanup+0x37/0x130 [ 453.296922] nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc] [ 453.302397] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core] [ 453.307818] device_release+0x31/0x90 [ 453.312005] kobject_cleanup+0x37/0x130 [ 453.316369] process_one_work+0x1e5/0x3c0 [ 453.320895] worker_thread+0x50/0x3b0 [ 453.325074] ? rescuer_thread+0x370/0x370 [ 453.329592] kthread+0x146/0x170 [ 453.333322] ? set_kthread_struct+0x40/0x40 [ 453.338027] ret_from_fork+0x1f/0x30 [ 453.342082] ---[ end trace 8c9cdd85adbbfc4f ]--- Maurizio Lombardi