From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90B26C021A4 for ; Mon, 24 Feb 2025 11:45:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 065476B007B; Mon, 24 Feb 2025 06:45:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0123D6B0083; Mon, 24 Feb 2025 06:45:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF5306B0085; Mon, 24 Feb 2025 06:45:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C20686B007B for ; Mon, 24 Feb 2025 06:45:19 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0A340121300 for ; Mon, 24 Feb 2025 11:44:55 +0000 (UTC) X-FDA: 83154656550.15.24794A6 Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by imf30.hostedemail.com (Postfix) with ESMTP id 08F438000F for ; Mon, 24 Feb 2025 11:44:52 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j6XMa04z; spf=pass (imf30.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740397493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GKmie4ql2EMQXkPV0dHeKHuPylk+abYCTsODvoG58iw=; b=e1n0EntiLXkKoev0FXzmab1ocsUu7nhU+p0AkbT5SLfrpV4ymx+B68ghu+rhIjig2igK0w wE9jAgwLHgDCOTf/YY6OylMAzRgytup+umt2yj9G2JFflVWFTkFIy8T9AjXhEfcR4eVbPb zI4SBd+3F5BRUZfHVEAEfZw7kbbJ6xc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j6XMa04z; spf=pass (imf30.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740397493; a=rsa-sha256; cv=none; b=v7xHJ5J0xPcdXNSmN/4tj3PBaSHMFnbcEYNP+3EGKqDwl9ZAVNZAQIfoB8qUcQ1exCsDss 4cRqmHGXkn/ebmXHUvG8SpuTTvHE0grgnHIHdsPkHrKlqQ74Dd8hjQyfRYFxDKHAKP/kte rH0xS2hzSutdxyYdS+qH8+Vdqi88zHE= Received: by mail-lf1-f50.google.com with SMTP id 2adb3069b0e04-545284eac3bso4262768e87.0 for ; Mon, 24 Feb 2025 03:44:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740397491; x=1741002291; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=GKmie4ql2EMQXkPV0dHeKHuPylk+abYCTsODvoG58iw=; b=j6XMa04zuwy88uuEeOSK2vaz/D4eD+BKFLLkBM5RX4GVzqKF1aeIsXldI2NbthIc7U baefWGmDiEYW5goueIEAiiRCF3dWj45FhivUrcND1wxjgOEBles82+5oiS9yw5xXQGUQ DadD/OLXet/Xm6hMisj4t+It0yJ2PcklH5ZeEcx/ZwHeWS1LcryBOAMD20o/JA09tOPy r95mfJFimsDDGU793aPGfM0D23C1jBbgyUtZ1TuCLRicglTF5F9RaL8UtJBsmPsRd7mK cKKr4JVQ6juRz+L6E2SK1nRLqGPK+jjA0vb1AD2t0B3rUBWPGeWnxFobPHUrJTmcPYi4 zE2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740397491; x=1741002291; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=GKmie4ql2EMQXkPV0dHeKHuPylk+abYCTsODvoG58iw=; b=rtDn6eKnS2l1jGwB5D6OGUHu4qSWINXOUwCJWHBU2ogfVZ/0zqDkWm4XO2j1L6mSSO 1+XGv6OJhR74R3wCze1Ni3ljatZG2wSKVyif6SMsX1RCUe7RBN6nDmgQijv8T0kYbeSm I5dOUR6qiWBZsvSJ4nzFzPCFPu8i+2R22kzYpHGGt8428yq3PmiavaWmbA34hX5gVxvy qxVOiG/YwSanN9ylPXhiJJRSdkheVJuurupipVkIK2N6dimdaEd8U5eKbDDfKd+9cI1C omsHM2YKmVLMVS1Vcii8Ho6y68yNnwE9N5nYz+DYeR2kBXqkynxTNT/knhGb6DJ34Cug gNaA== X-Forwarded-Encrypted: i=1; AJvYcCVan3sCSfh71YL3caFgQgAbyn+QIN7NmFURVCeuttkBruBEZoaH2MncAYbAUfL8G4cSbA6ubmiHHQ==@kvack.org X-Gm-Message-State: AOJu0YxH17B7gTNK49Km35JB+EMwVyXvuW9d5lsbAY4D5FzcdFkjFWBW 2A9Xot6Owy+UsboyNQ0PvA6kdZKtagvzz+y246WI4tIbbitUsmhs X-Gm-Gg: ASbGncvcr2/76M4iQBJCWeSKBjr1sMu+a1ljzd7FT2tYdiSltCdA1NdQV6oRclNXuK6 Sd3V3/P1fVq4VzrmRSuY5v+ZiPojC7HGBWgxJZ5jzT3c0dUVeJqVYtApll1K5UecB2ycgmSn1kZ muXvUojLEN7hWnXBIrjGtdxC8ArEiVE0WTGz7LwD5+27zci/4GYrMZNSQnfeUJoNrmIEvW8r7fc Dt9189AS56bt3XKSxM75UIUUsr5rdxZZ1W4VtA6k8TH+5usDX7RPS4LUaYor2MOdNb6rTaZQTWd 57HxJdAC5G4UShyor1HEk1WBL/pSB9t0FE071pGvqkZs7Nrk X-Google-Smtp-Source: AGHT+IGjbXqoMdV+6yP1nHJ/PGwFmTUOZnYyR49KFY2a21Z1wZNQdeD93xJxOmEV3IxFMucNPB3mfQ== X-Received: by 2002:a05:6512:1244:b0:545:1104:617d with SMTP id 2adb3069b0e04-54838eddea5mr5695115e87.11.1740397490753; Mon, 24 Feb 2025 03:44:50 -0800 (PST) Received: from pc636 (host-95-203-6-24.mobileonline.telia.com. [95.203.6.24]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5461f541653sm2376875e87.156.2025.02.24.03.44.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Feb 2025 03:44:50 -0800 (PST) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Mon, 24 Feb 2025 12:44:46 +0100 To: Vlastimil Babka , Keith Busch Cc: Keith Busch , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik , linux-nvme@lists.infradead.org, leitao@debian.org Subject: Re: [PATCH v2 6/7] mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy() Message-ID: References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> <20240807-b4-slab-kfree_rcu-destroy-v2-6-ea79102f428c@suse.cz> <2811463a-751f-4443-9125-02628dc315d9@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2811463a-751f-4443-9125-02628dc315d9@suse.cz> X-Rspamd-Queue-Id: 08F438000F X-Stat-Signature: xxyupg843eoxwz4dnr7eqatu31do4ef1 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740397492-995365 X-HE-Meta: U2FsdGVkX18BVX8eoVpJ73ligtwoVlZlGXJEm9qZ3dXC8tkhcZLWASrQBvG6B+aFg10TRG2amxXhw4UPTReCLKy1n0V4ATzF5Dz4sDlP5Ds0z/ADwo3hHVaO08W+1uL00HOgSPRUorz3+Pe5cc13PsnfngWjxVV8egiU9nFXasNbJIBdzydJnj5ZSI2KC9k/0UEBNHLmh+NJoikwkCM9vSXju/cNIICysZYOoj1qUQ+3aTNQ7+zgATQiH4SdYgg2W6Ff6Lz5QyxVDcC4vabeTwNhh/Y4YfqjwMdgpIk+x6eU88Uk984GA4p14boQ1Q/Rxi08TapcNOjAcTDdb59BdP5ds/SOZaeNybOhO8v+oiI2l0tLgtLZX+ohXjiQJhtfuGYaC9ejXzoftYoIy//Y3a2B/DrF7XF6eEYqAi0waxS4ibJI+mBHvl0A2JLfowszqnJVGz0gLw5lxZB/GtkTdqSz5Dgmvpq9m4i8aOXNuShu7xTtLMMl/kFQlCZ6wX2oTQPqYizMBHH0qlQw2ovnStjGfbDhtj4icJANA00+0daB7Sldw3lRxOJusMlvUjreZLlPNV2SPmpN4bVI6L+hwr0sOlO8CTJl9fnNT27x9qAdBwRCSATLtH3PNJ6NtQTvu3aIVfj4aeVYyXE1cJczMlaWehuNaJ7/O2jmbZzCIhiGHBCSlzECfHGZMZDxkvK+eRQjh66Llj03ee+4YnNfyDxv19fWQZDUQhlHnnwntsgECZQMn3YibXNnhvIVNHNaqAZoBMnmr3VDKMrYLohUDR9jyVpnW4Nov2A1Kn2dSnrFYB2hoVClfV9Q8hp8wb/dRXnnwJhNq24ht6yUAwpHW1HBCV+eydNUtCkngBkql74/k2NzoDcMlJrYdhyfzMwAyhMF0ghiz6qY6c8Yxk8BjOBVui9oBVuobeLs4lOi4m39qYe44b4h9cAvfDqgowfgxd5SD59emksyOEGGH96 j8+ZnXim ADlE90/t1brI7XuxiPPTRqz3woOPMT6cK7DZz7zS8t7PlFYbSUPg8PeaZIYWDh19N8So8TN2RciOrxdMOduP7EZe7M5rHkRYBYkKG78XmOTYQ4ZlDCYWB4Zqh/kQSVm2PqnsZkrhlIS5cVMquH1QXGVjwd1lKFp+7TEbTI9uucsW2SpnQafvUUGzgFmQVlLtORDx87ial4+O1BwGr7CoY3G/CgseOxHblReoHX5iI8nePPRWUsvyeXDTtbk1mumHqmc2XZpKbYB7WGWUrmKZ/vS+yYDxY78dVM9Z+I9wnV/G8HD/3DfCaa4lKfP3g5OU8W/YU1gCkpGrT8uJA5GNaYe7OPo7E0f6prnGeMUjXDu1n6obEca0BjyT5UzZ+O2Tf9WYMvjegwT8zjLE5bGJht7/KqgQMNrLJlIDcNRhDSjx8W+gBcOxdT8uIqX4qmh9K8pASFQ1Z1YCAytB+MJDK/AtGEBmPt1NDkoN/m7tdt5Qt2bdUbZYH5WxNnldY875KL3zgnrxudZ6t8TI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 21, 2025 at 06:28:49PM +0100, Vlastimil Babka wrote: > On 2/21/25 17:30, Keith Busch wrote: > > On Wed, Aug 07, 2024 at 12:31:19PM +0200, Vlastimil Babka wrote: > >> We would like to replace call_rcu() users with kfree_rcu() where the > >> existing callback is just a kmem_cache_free(). However this causes > >> issues when the cache can be destroyed (such as due to module unload). > >> > >> Currently such modules should be issuing rcu_barrier() before > >> kmem_cache_destroy() to have their call_rcu() callbacks processed first. > >> This barrier is however not sufficient for kfree_rcu() in flight due > >> to the batching introduced by a35d16905efc ("rcu: Add basic support for > >> kfree_rcu() batching"). > >> > >> This is not a problem for kmalloc caches which are never destroyed, but > >> since removing SLOB, kfree_rcu() is allowed also for any other cache, > >> that might be destroyed. > >> > >> In order not to complicate the API, put the responsibility for handling > >> outstanding kfree_rcu() in kmem_cache_destroy() itself. Use the newly > >> introduced kvfree_rcu_barrier() to wait before destroying the cache. > >> This is similar to how we issue rcu_barrier() for SLAB_TYPESAFE_BY_RCU > >> caches, but has to be done earlier, as the latter only needs to wait for > >> the empty slab pages to finish freeing, and not objects from the slab. > >> > >> Users of call_rcu() with arbitrary callbacks should still issue > >> rcu_barrier() before destroying the cache and unloading the module, as > >> kvfree_rcu_barrier() is not a superset of rcu_barrier() and the > >> callbacks may be invoking module code or performing other actions that > >> are necessary for a successful unload. > >> > >> Signed-off-by: Vlastimil Babka > >> --- > >> mm/slab_common.c | 3 +++ > >> 1 file changed, 3 insertions(+) > >> > >> diff --git a/mm/slab_common.c b/mm/slab_common.c > >> index c40227d5fa07..1a2873293f5d 100644 > >> --- a/mm/slab_common.c > >> +++ b/mm/slab_common.c > >> @@ -508,6 +508,9 @@ void kmem_cache_destroy(struct kmem_cache *s) > >> if (unlikely(!s) || !kasan_check_byte(s)) > >> return; > >> > >> + /* in-flight kfree_rcu()'s may include objects from our cache */ > >> + kvfree_rcu_barrier(); > >> + > >> cpus_read_lock(); > >> mutex_lock(&slab_mutex); > > > > This patch appears to be triggering a new warning in certain conditions > > when tearing down an nvme namespace's block device. Stack trace is at > > the end. > > > > The warning indicates that this shouldn't be called from a > > WQ_MEM_RECLAIM workqueue. This workqueue is responsible for bringing up > > and tearing down block devices, so this is a memory reclaim use AIUI. > > I'm a bit confused why we can't tear down a disk from within a memory > > reclaim workqueue. Is the recommended solution to simply remove the WQ > > flag when creating the workqueue? > > I think it's reasonable to expect a memory reclaim related action would > destroy a kmem cache. Mateusz's suggestion would work around the issue, but > then we could get another surprising warning elsewhere. Also making the > kmem_cache destroys async can be tricky when a recreation happens > immediately under the same name (implications with sysfs/debugfs etc). We > managed to make the destroying synchronous as part of this series and it > would be great to keep it that way. > > > ------------[ cut here ]------------ > > workqueue: WQ_MEM_RECLAIM nvme-wq:nvme_scan_work is flushing !WQ_MEM_RECLAIM events_unbound:kfree_rcu_work > > Maybe instead kfree_rcu_work should be using a WQ_MEM_RECLAIM workqueue? It > is after all freeing memory. Ulad, what do you think? > We reclaim memory, therefore WQ_MEM_RECLAIM seems what we need. AFAIR, there is an extra rescue worker, which can really help under a low memory condition in a way that we do a progress. Do we have a reproducer of mentioned splat? -- Uladzislau Rezki