From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01DAAC52D7C for ; Fri, 9 Aug 2024 16:26:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D7026B00AB; Fri, 9 Aug 2024 12:26:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 887076B00AC; Fri, 9 Aug 2024 12:26:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74E286B00AE; Fri, 9 Aug 2024 12:26:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 560C16B00AB for ; Fri, 9 Aug 2024 12:26:46 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 00E7E1211A4 for ; Fri, 9 Aug 2024 16:26:45 +0000 (UTC) X-FDA: 82433235612.23.3196380 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf19.hostedemail.com (Postfix) with ESMTP id F17351A0009 for ; Fri, 9 Aug 2024 16:26:43 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L3ysYhqz; spf=pass (imf19.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723220737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8GhuZXBHrLQFERoBhWAzGEBHzpGEHA+J+lL0pNJNCos=; b=TPQTXejuym9vx6y50f1cw1EOw5ag1WMW/JGlZj3IJD0vUrCAAJZ3ZJJgvYWw9fZW5CSnnG BFN7/W7rAb3TymO7uNQnJC3F5yYuDMQKMqEhfO0gg6dQt9WVwP+MT2Sp15zl3CezEmRtQ7 cHalmPuj8nDNk2SJPHOW6FmrvZohHhk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723220737; a=rsa-sha256; cv=none; b=UuWNX0Y/A/6HGCV1HOxoBrnaR/EvIrv0uEZMx8Qbs9ez+bOaeX6DKXxKt69gVFJHICr9lc cHJIum5Juepx5zQnz7+yvoKRaMBERZijj5JR535n/ZbvyCZKAhNkv9maGdChzsqb8RdKiO zCFkjDicjgjmC883BHQZIawymDE+in4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L3ysYhqz; spf=pass (imf19.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2f0271b0ae9so22825341fa.1 for ; Fri, 09 Aug 2024 09:26:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723220802; x=1723825602; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=8GhuZXBHrLQFERoBhWAzGEBHzpGEHA+J+lL0pNJNCos=; b=L3ysYhqzx84pd43+HXgy9PQjNEnyN2nVe9dBlnGRbGI3+2UwUczU2Q6cbxxgtGUail HC+biTaqn4I11X5XWRm/yl31zz9tLiY4VOfAN7VFndDhK3In5KJJr/mryK2GoayNdy3c E3ceR3EaFiK5XAgybP12TvNLrgVQyTQlEePfiVa6VrrEpg5mDuTf9ORRTyc1PWXKjjYm 6+DD7QfoEcbc3ES7ygYApvSKPuDkQfWNEyaf8xCxPS0GPJx4Dl1N+82DGp4uBiWgN6Ql tUi4LCs578gsG6hfg7AmMYWAsNt6PZOARTvnHnSeHuzzySBUjbWP6L8hJe+oh99e7oAQ MImg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723220802; x=1723825602; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=8GhuZXBHrLQFERoBhWAzGEBHzpGEHA+J+lL0pNJNCos=; b=aVU8Yx1HGrqoPBRChWSRFMqyIiSfH0dtdYlJvXh0YWOFY2uy2LKrMzGsh50XSKyvLd 7kTz6qoorM/C3/F281xxvMc53mrADtiojyrhgabFElqAYozPjahNzoFj+2p7XFa1FoZu zw5xaAPE+bxJtYk82VXCDvJrfG3ht/FnB1Cf2bkgSHXZ/cReW3jJFBEtYv0UNjOjQl8J hzkBM5IzOVwYCYeLzAJ3TwA84/KVFK1+CCskKQEGBQjp3WAzhu1jG0jIORiSy1N+BOCe qz3TtLpKe4jZ6vwR77868hlcL0oBMKOkWsTDHl4uhdNwOFjWU57LwB5k21hUttZjTFLK dGvg== X-Forwarded-Encrypted: i=1; AJvYcCWSaTBl+b/LR4l395QWuVRLgHg8jMPqETBsXovGyYOnoDe0kLnW2muYJcKgJptuImZ5jBSOl4SDtqR4eFosYfkr2gY= X-Gm-Message-State: AOJu0YwfbwKXd2T427tQDVfa4daCDuaeu4ErWdRHszrZSOeNZWGT0Ehm FV3n6X04N2ujpxLCkNsidL0ls3ecF43TdtEuvG8PSEm3aLqpo+IL X-Google-Smtp-Source: AGHT+IHIcKoF3Ie/2OsybzOIBFmvl+sf2xsm7hrxBYfGa1D+ay/Ymjy0D/SiESw+BOi2PbJsSYyuEw== X-Received: by 2002:a2e:a581:0:b0:2ec:500c:b2e0 with SMTP id 38308e7fff4ca-2f1a6d1d1e9mr19917041fa.22.1723220801209; Fri, 09 Aug 2024 09:26:41 -0700 (PDT) Received: from pc636 (host-90-233-216-8.mobileonline.telia.com. [90.233.216.8]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f15e26038bsm25892501fa.117.2024.08.09.09.26.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:26:40 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Fri, 9 Aug 2024 18:26:36 +0200 To: Vlastimil Babka Cc: "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Christoph Lameter , David Rientjes , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Julia Lawall , Jakub Kicinski , "Jason A. Donenfeld" , "Uladzislau Rezki (Sony)" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, Alexander Potapenko , Marco Elver , Dmitry Vyukov , kasan-dev@googlegroups.com, Jann Horn , Mateusz Guzik Subject: Re: [PATCH v2 5/7] rcu/kvfree: Add kvfree_rcu_barrier() API Message-ID: References: <20240807-b4-slab-kfree_rcu-destroy-v2-0-ea79102f428c@suse.cz> <20240807-b4-slab-kfree_rcu-destroy-v2-5-ea79102f428c@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240807-b4-slab-kfree_rcu-destroy-v2-5-ea79102f428c@suse.cz> X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F17351A0009 X-Stat-Signature: wixbqiownbeetynzatwdbpqj8ag9rddq X-HE-Tag: 1723220803-978088 X-HE-Meta: U2FsdGVkX18LHnlYRJIP+xTkRchO8m5yCbTpGn58qxVA2Ex5p9Ro/GU5Y/elrrYVIJj51FvJgSTRiPgWt/TjF80SA8PufMln3gYQA82qCtrwcnk1kcWbqFYFnA03Jakcm75pNKnKNaRpNPATbyJtmbRPmkObFhi0gnYaFxHfd+RBiH0u8xpzFgnYAq8+/awYclBA1h7rgApaHWN23mrUoOBXV5QjOOb5Wx9au7swyYO3g9+vM5FG82a35QcbeHiG+K6jt7axaS4I2dmtybxOLfftCfHZAqSs0+B/oYxtVEHW3kJajLu0mn1yyTCVwvpX0pQKXZB/Cz76bvpjznR4BW/G8xmevPBSUJfsslI08mQw98Ltjq5JmSt+AQNcLax1ezTcZII5WQx6O38bitRinuhVP+r5yQFEPr030ZzBDb69ZkkgTQgyOOg+6fsaKPwflkwBAzdH/wXSv3pCrz6jvI3zmxxhmpVuszL3exehND7A8h53B0LwaK2QEBiifnGq280nvILvemsA/F+7aHq+TgbOWgf1LDk8ujsCKHhX/P0sUoKs83TyLWA366CstOOag/VBSTA289qZZ4sSciWjFd5fs/tPIB+HgVlW4YpNcfOrGjgSbnAf8u2+6Xv+4g9ZHS195toxvmzbjzHQA7LMXAcAgFPKyKlegUaEea5oJKR6eCTFGZMEyXktso1Ceq+fJYo5xWDjYLWGzuAhM+bPwH1P7ZVf2gq2kL0VOmw4Q24OeeQKuQUZMX2Wwpr0jXd92XqXskgbJib7kjYS4KWdQHKGcMi9nKR+6QZwXoF6xxDClDfecLj630QT9j5JzuqKrWkIgEo+YNPrlBl/X8hnNjmIrlfOhu/NB9XGc6gb8b5QP2CxwRrB3ekoaAyoyjqLc6PP1EIpIPQzVjdAB2fLbl+gsLF9BSmeb69UHJ1QR6ZZMf6uJ/Ch2mKUDiQ0vOHIls2a1goH4ixqFg8mSqm e8QsZI+g MNzQPIJwBM/pkQWMxsDzhVVlcsGBlsDuN1ctgoztq+1L0ZZlBjts036xvmK4yGtAHjUsvDOVvSq0ZS4LVn2nB1RwhTd4ePkSk5x/MGuAAHRV78EDpOgl4FMvJAI/xuJAOggJzNNFo8mgpADfJ3Nc21cS5KOPXzXHDo78/u2qUtQubHVH/bgqn4k1euFI+bWwxxf3QKUdgBJRt3sdMMXEErzGJvysJC/9tuIbpCMSTOhpjjoMDSc5T1f7aHYUqdXtWhVJqn6sz57dRne64T+ELE1IqozSRgc9pYyFdXKutSSbacTFPlUH9Z/F7Su6g4bXO8g+ZKDHyH+MvMYZaQxxrdDDOflv7bnH+2VRqhItDRlS64f/7XCG/5PjBjs6wWx4ph47fsFR0EK2rQQbK7Bb2i3Xz8lQevI4M9Ys72CfWnwwlSwBH85JZvACuJDjfxEosJWDqby7rPBvO+5vaw9ctn9HMVfJMWuvWk/DBkS7hMyKLKX7704JH2biaaQEII1i3tS10iF9yWO8McE8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, Vlastimil! > From: "Uladzislau Rezki (Sony)" > > Add a kvfree_rcu_barrier() function. It waits until all > in-flight pointers are freed over RCU machinery. It does > not wait any GP completion and it is within its right to > return immediately if there are no outstanding pointers. > > This function is useful when there is a need to guarantee > that a memory is fully freed before destroying memory caches. > For example, during unloading a kernel module. > > Signed-off-by: Uladzislau Rezki (Sony) > Signed-off-by: Vlastimil Babka > --- > include/linux/rcutiny.h | 5 +++ > include/linux/rcutree.h | 1 + > kernel/rcu/tree.c | 103 ++++++++++++++++++++++++++++++++++++++++++++---- > 3 files changed, 101 insertions(+), 8 deletions(-) > > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h > index d9ac7b136aea..522123050ff8 100644 > --- a/include/linux/rcutiny.h > +++ b/include/linux/rcutiny.h > @@ -111,6 +111,11 @@ static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) > kvfree(ptr); > } > > +static inline void kvfree_rcu_barrier(void) > +{ > + rcu_barrier(); > +} > + > #ifdef CONFIG_KASAN_GENERIC > void kvfree_call_rcu(struct rcu_head *head, void *ptr); > #else > diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h > index 254244202ea9..58e7db80f3a8 100644 > --- a/include/linux/rcutree.h > +++ b/include/linux/rcutree.h > @@ -35,6 +35,7 @@ static inline void rcu_virt_note_context_switch(void) > > void synchronize_rcu_expedited(void); > void kvfree_call_rcu(struct rcu_head *head, void *ptr); > +void kvfree_rcu_barrier(void); > > void rcu_barrier(void); > void rcu_momentary_dyntick_idle(void); > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c > index e641cc681901..ebcfed9b570e 100644 > --- a/kernel/rcu/tree.c > +++ b/kernel/rcu/tree.c > @@ -3584,18 +3584,15 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp) > } > > /* > - * This function is invoked after the KFREE_DRAIN_JIFFIES timeout. > + * Return: %true if a work is queued, %false otherwise. > */ > -static void kfree_rcu_monitor(struct work_struct *work) > +static bool > +kvfree_rcu_queue_batch(struct kfree_rcu_cpu *krcp) > { > - struct kfree_rcu_cpu *krcp = container_of(work, > - struct kfree_rcu_cpu, monitor_work.work); > unsigned long flags; > + bool queued = false; > int i, j; > > - // Drain ready for reclaim. > - kvfree_rcu_drain_ready(krcp); > - > raw_spin_lock_irqsave(&krcp->lock, flags); > > // Attempt to start a new batch. > @@ -3634,11 +3631,27 @@ static void kfree_rcu_monitor(struct work_struct *work) > // be that the work is in the pending state when > // channels have been detached following by each > // other. > - queue_rcu_work(system_wq, &krwp->rcu_work); > + queued = queue_rcu_work(system_wq, &krwp->rcu_work); > } > } > > raw_spin_unlock_irqrestore(&krcp->lock, flags); > + return queued; > +} > + > +/* > + * This function is invoked after the KFREE_DRAIN_JIFFIES timeout. > + */ > +static void kfree_rcu_monitor(struct work_struct *work) > +{ > + struct kfree_rcu_cpu *krcp = container_of(work, > + struct kfree_rcu_cpu, monitor_work.work); > + > + // Drain ready for reclaim. > + kvfree_rcu_drain_ready(krcp); > + > + // Queue a batch for a rest. > + kvfree_rcu_queue_batch(krcp); > > // If there is nothing to detach, it means that our job is > // successfully done here. In case of having at least one > @@ -3859,6 +3872,80 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) > } > EXPORT_SYMBOL_GPL(kvfree_call_rcu); > > +/** > + * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete. > + * > + * Note that a single argument of kvfree_rcu() call has a slow path that > + * triggers synchronize_rcu() following by freeing a pointer. It is done > + * before the return from the function. Therefore for any single-argument > + * call that will result in a kfree() to a cache that is to be destroyed > + * during module exit, it is developer's responsibility to ensure that all > + * such calls have returned before the call to kmem_cache_destroy(). > + */ > +void kvfree_rcu_barrier(void) > +{ > + struct kfree_rcu_cpu_work *krwp; > + struct kfree_rcu_cpu *krcp; > + bool queued; > + int i, cpu; > + > + /* > + * Firstly we detach objects and queue them over an RCU-batch > + * for all CPUs. Finally queued works are flushed for each CPU. > + * > + * Please note. If there are outstanding batches for a particular > + * CPU, those have to be finished first following by queuing a new. > + */ > + for_each_possible_cpu(cpu) { > + krcp = per_cpu_ptr(&krc, cpu); > + > + /* > + * Check if this CPU has any objects which have been queued for a > + * new GP completion. If not(means nothing to detach), we are done > + * with it. If any batch is pending/running for this "krcp", below > + * per-cpu flush_rcu_work() waits its completion(see last step). > + */ > + if (!need_offload_krc(krcp)) > + continue; > + > + while (1) { > + /* > + * If we are not able to queue a new RCU work it means: > + * - batches for this CPU are still in flight which should > + * be flushed first and then repeat; > + * - no objects to detach, because of concurrency. > + */ > + queued = kvfree_rcu_queue_batch(krcp); > + > + /* > + * Bail out, if there is no need to offload this "krcp" > + * anymore. As noted earlier it can run concurrently. > + */ > + if (queued || !need_offload_krc(krcp)) > + break; > + > + /* There are ongoing batches. */ > + for (i = 0; i < KFREE_N_BATCHES; i++) { > + krwp = &(krcp->krw_arr[i]); > + flush_rcu_work(&krwp->rcu_work); > + } > + } > + } > + > + /* > + * Now we guarantee that all objects are flushed. > + */ > + for_each_possible_cpu(cpu) { > + krcp = per_cpu_ptr(&krc, cpu); > + > + for (i = 0; i < KFREE_N_BATCHES; i++) { > + krwp = &(krcp->krw_arr[i]); > + flush_rcu_work(&krwp->rcu_work); > + } > + } > +} > +EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); > + > static unsigned long > kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) > { > > -- > 2.46.0 > I need to send out a v2. What is a best way? Please let me know. I have not checked where this series already landed. Thank you! -- Uladzislau Rezki