From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
To: davem@davemloft.net
Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org,
memxor@gmail.com, delyank@fb.com, linux-mm@kvack.org,
bpf@vger.kernel.org, kernel-team@fb.com
Subject: [PATCH v6 bpf-next 13/16] bpf: Prepare bpf_mem_alloc to be used by sleepable bpf programs.
Date: Fri, 2 Sep 2022 14:10:55 -0700 [thread overview]
Message-ID: <20220902211058.60789-14-alexei.starovoitov@gmail.com> (raw)
In-Reply-To: <20220902211058.60789-1-alexei.starovoitov@gmail.com>
From: Alexei Starovoitov <ast@kernel.org>
Use call_rcu_tasks_trace() to wait for sleepable progs to finish.
Then use call_rcu() to wait for normal progs to finish
and finally do free_one() on each element when freeing objects
into global memory pool.
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/memalloc.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
index f7b07787581b..8895c016dcdb 100644
--- a/kernel/bpf/memalloc.c
+++ b/kernel/bpf/memalloc.c
@@ -230,6 +230,13 @@ static void __free_rcu(struct rcu_head *head)
atomic_set(&c->call_rcu_in_progress, 0);
}
+static void __free_rcu_tasks_trace(struct rcu_head *head)
+{
+ struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu);
+
+ call_rcu(&c->rcu, __free_rcu);
+}
+
static void enque_to_free(struct bpf_mem_cache *c, void *obj)
{
struct llist_node *llnode = obj;
@@ -255,7 +262,11 @@ static void do_call_rcu(struct bpf_mem_cache *c)
* from __free_rcu() and from drain_mem_cache().
*/
__llist_add(llnode, &c->waiting_for_gp);
- call_rcu(&c->rcu, __free_rcu);
+ /* Use call_rcu_tasks_trace() to wait for sleepable progs to finish.
+ * Then use call_rcu() to wait for normal progs to finish
+ * and finally do free_one() on each element.
+ */
+ call_rcu_tasks_trace(&c->rcu, __free_rcu_tasks_trace);
}
static void free_bulk(struct bpf_mem_cache *c)
@@ -457,6 +468,7 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma)
/* c->waiting_for_gp list was drained, but __free_rcu might
* still execute. Wait for it now before we free 'c'.
*/
+ rcu_barrier_tasks_trace();
rcu_barrier();
free_percpu(ma->cache);
ma->cache = NULL;
@@ -471,6 +483,7 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma)
}
if (c->objcg)
obj_cgroup_put(c->objcg);
+ rcu_barrier_tasks_trace();
rcu_barrier();
free_percpu(ma->caches);
ma->caches = NULL;
--
2.30.2
next prev parent reply other threads:[~2022-09-02 21:11 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-02 21:10 [PATCH v6 bpf-next 00/16] bpf: BPF specific memory allocator Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 01/16] bpf: Introduce any context " Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 02/16] bpf: Convert hash map to bpf_mem_alloc Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 03/16] selftests/bpf: Improve test coverage of test_maps Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 04/16] samples/bpf: Reduce syscall overhead in map_perf_test Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 05/16] bpf: Relax the requirement to use preallocated hash maps in tracing progs Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 06/16] bpf: Optimize element count in non-preallocated hash map Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 07/16] bpf: Optimize call_rcu " Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 08/16] bpf: Adjust low/high watermarks in bpf_mem_cache Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 09/16] bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 10/16] bpf: Add percpu allocation support to bpf_mem_alloc Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 11/16] bpf: Convert percpu hash map to per-cpu bpf_mem_alloc Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 12/16] bpf: Remove tracing program restriction on map types Alexei Starovoitov
2022-09-02 21:10 ` Alexei Starovoitov [this message]
2022-09-02 21:10 ` [PATCH v6 bpf-next 14/16] bpf: Remove prealloc-only restriction for sleepable bpf programs Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 15/16] bpf: Remove usage of kmem_cache from bpf_mem_cache Alexei Starovoitov
2022-09-02 21:10 ` [PATCH v6 bpf-next 16/16] bpf: Optimize rcu_barrier usage between hash map and bpf_mem_alloc Alexei Starovoitov
2022-09-05 13:50 ` [PATCH v6 bpf-next 00/16] bpf: BPF specific memory allocator patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220902211058.60789-14-alexei.starovoitov@gmail.com \
--to=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=delyank@fb.com \
--cc=kernel-team@fb.com \
--cc=linux-mm@kvack.org \
--cc=memxor@gmail.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox