From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51C54CDB47E for ; Wed, 18 Oct 2023 11:32:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE9CD8D0154; Wed, 18 Oct 2023 07:32:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C70308D0151; Wed, 18 Oct 2023 07:32:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B387D8D0154; Wed, 18 Oct 2023 07:32:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9F5738D0151 for ; Wed, 18 Oct 2023 07:32:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7100AC01FC for ; Wed, 18 Oct 2023 11:32:47 +0000 (UTC) X-FDA: 81358369974.06.BAA8D34 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf01.hostedemail.com (Postfix) with ESMTP id B63CB40012 for ; Wed, 18 Oct 2023 11:32:44 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=none; spf=none (imf01.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697628765; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=624PohsT+i5wWYZdJgRn5EHS4SOahN5eGhk4igVlWUg=; b=hvFvZAkQ7DJ+0SQwWRB443tvNTD9gTi5u/zqGfattnZpqV3xclsCyjNQMfbUZR2tF575k9 bSLodETE84I0pUBLuS/EeIPxCZBCeGSewbje23E6dvKGQK5mFJN9Vat1c9BFrCF0eUnbog 9t6Eig7UWtgFiVAWuB5T2UzD0N4tqQM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=none; spf=none (imf01.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697628765; a=rsa-sha256; cv=none; b=MwDisU7jOUmpTZg+QFwlqg5Tx2WgMfdpMHThgMyfp74umcu5zzgXjhnF4HvRttbGbWCRNj gZY8V9uujdpcCYFfyBi3TpfSeghO6kpdB524KaoK6/xsyTvdo6KhY/UVAGlCCvecxwqMuD PjLo8FlG+zBbO/x3A8v7zNwDcRWpmN8= Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4S9TGm6SFmz4f3tpQ for ; Wed, 18 Oct 2023 19:32:32 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP4 (Coremail) with SMTP id gCh0CgDnfd1Mwi9l9jYmDQ--.41845S10; Wed, 18 Oct 2023 19:32:35 +0800 (CST) From: Hou Tao To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: Martin KaFai Lau , Alexei Starovoitov , Andrii Nakryiko , Song Liu , Hao Luo , Yonghong Song , Daniel Borkmann , KP Singh , Stanislav Fomichev , Jiri Olsa , John Fastabend , houtao1@huawei.com, Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton Subject: [PATCH bpf-next v2 6/7] bpf: Use bpf_global_percpu_ma for per-cpu kptr in __bpf_obj_drop_impl() Date: Wed, 18 Oct 2023 19:33:42 +0800 Message-Id: <20231018113343.2446300-7-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20231018113343.2446300-1-houtao@huaweicloud.com> References: <20231018113343.2446300-1-houtao@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:gCh0CgDnfd1Mwi9l9jYmDQ--.41845S10 X-Coremail-Antispam: 1UD129KBjvJXoW3Jr1xGryxGrWkZrWkCrW8Crg_yoWxJry3pF 4ftr12yr4kJFs7Z3s8Gw4I934FqrW3Xa47C34kG34Svr4avryDZw1kCF1xZa45JrW0kw1I y3Wq9r9xA3y8AaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBSb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4UJVWxJr1lIxAI cVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2js IEc7CjxVAFwI0_Gr1j6F4UJbIYCTnIWIevJa73UjIFyTuYvjxUFgAwUUUUU X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: B63CB40012 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: crsexsr4r67shb933zx57ouzr8nginyj X-HE-Tag: 1697628764-214860 X-HE-Meta: U2FsdGVkX1/Q3JMtdIDYN6J6NFm+Ox3j0lzGmR8PSJ3W8ERG1LrZx5V606LSsUTnfKlb76jUhWP4gCrvxDCo4EiHPFjxfA4IvER0O9JbYmLZ7+yjKali85z5KM8wZyLM2UKq1Ad3IsQs4i8KGih+HTJBf0AtbbBDYk6XIDdzWn2lMKjNvI+K/oHTOGVxD1KiV6BUlbt7IVZdeqFe5j04A56VHn9v4Njp4/b1q/Ihv2xcaARupwL6cBGrMG9JcAEC8GvPEO3iXDBOoNJ6boMoH9i05QjjVlEPUAYZrELFWFn4N/UcArgr48doQAJrirMAF/yx3e1tLxJRkmY94Q6RbfZjHu61JDM+yWJ9aWhFxO/nPuTynGGF7wXXVwmDmouZk9I+cs03QzngYuoz4XYcYQ4a5PXgVLMFdDXvsiFdE/m7PzsAbc9+dhTRH+GACfN6tfiYijr/aoXaGyCVxAIWXatFYM1HgWtsSSy5YiAXzGQ8BalXlNSy7KjUgZ2FvKuQReyB89mz1vDjLK6/qWcUIfZKs2WRkY8HAZsI7LTxkwo/91KWYdJ8C0dT0X8limn/QRVYgDEilAJogTlQ4rFg78/ZcqHyQccqMQAeGfHfgYdlWggS9zT/IQSP/RtyfsmxlNLUH6LA67uMITVcTfiJn46c37fU17jiIbIR5vmTgXzJji4Lp5pqjDVTYticPWA6EcZrrz6bEaT9tkM23M0hyfhY9EQQ6ZJ7cZjknRx+QlzBp4dMosjx3AAL9fjFjeJ22hVQFsQwISUEgiwdZb3U2i19M0uppvQ4XCJL7TbnmkMu/KzfWGvnpoE2nrUzPCjYRWkC2FQxfePH37Tq8gpQuNkUhRrU4C/l3+ocXbn83yEnp5RpS87XKkp/w8XXgm4+OX47E3P0KEt5cAVj4rUGWvG4vMkZNhpQm4MvHaH9iskIf4NsR//Slp+f2Xh+O3rj5oZp4NciqKZJoVlNeyD D2eHJanj hUt6Jy/rQpqR4LSR71p1T+u3TV6RvQDcGTOUcjez2XaQ+g9lBfQ3uQmIB0lvPX5zr90mWCiCg6D9tJfvExyjofkry9f3S9ws4vmzwcrehGizsiBB50UUr/3D8qzrhyUfI1R7GHsji+FM02mhGdgnM/Uv1VNQEzwR5K+zrEXGik6RtLTsaXQ9wFvvp/elmSr9dloNVKzhYVgj2HCeXhk0ViIwMSPfY73Ql2UAOHT1UoUxaqB2/Dj+sLzePXYiZ3G2mJRqaWDmafgMbZHE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hou Tao The following warning was reported when running "./test_progs -t test_bpf_ma/percpu_free_through_map_free": ------------[ cut here ]------------ WARNING: CPU: 1 PID: 68 at kernel/bpf/memalloc.c:342 CPU: 1 PID: 68 Comm: kworker/u16:2 Not tainted 6.6.0-rc2+ #222 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996) Workqueue: events_unbound bpf_map_free_deferred RIP: 0010:bpf_mem_refill+0x21c/0x2a0 ...... Call Trace: ? bpf_mem_refill+0x21c/0x2a0 irq_work_single+0x27/0x70 irq_work_run_list+0x2a/0x40 irq_work_run+0x18/0x40 __sysvec_irq_work+0x1c/0xc0 sysvec_irq_work+0x73/0x90 asm_sysvec_irq_work+0x1b/0x20 RIP: 0010:unit_free+0x50/0x80 ...... bpf_mem_free+0x46/0x60 __bpf_obj_drop_impl+0x40/0x90 bpf_obj_free_fields+0x17d/0x1a0 array_map_free+0x6b/0x170 bpf_map_free_deferred+0x54/0xa0 process_scheduled_works+0xba/0x370 worker_thread+0x16d/0x2e0 kthread+0x105/0x140 ret_from_fork+0x39/0x60 ret_from_fork_asm+0x1b/0x30 ---[ end trace 0000000000000000 ]--- The reason is simple: __bpf_obj_drop_impl() does not know the freeing field is a per-cpu pointer and it uses bpf_global_ma to free the pointer. Because bpf_global_ma is not a per-cpu allocator, so ksize() is used to select the corresponding cache. The bpf_mem_cache with 16-bytes unit_size will always be selected to do the unmatched free and it will trigger the warning in free_bulk() eventually. Because per-cpu kptr doesn't support list or rb-tree now, so fix the problem by only checking whether or not the type of kptr is per-cpu in bpf_obj_free_fields(), and using bpf_global_percpu_ma to these kptrs. Signed-off-by: Hou Tao --- include/linux/bpf.h | 2 +- kernel/bpf/helpers.c | 22 ++++++++++++++-------- kernel/bpf/syscall.c | 4 ++-- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ebd412179771..b4825d3cdb29 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2058,7 +2058,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec); bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *rec_b); void bpf_obj_free_timer(const struct btf_record *rec, void *obj); void bpf_obj_free_fields(const struct btf_record *rec, void *obj); -void __bpf_obj_drop_impl(void *p, const struct btf_record *rec); +void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); struct bpf_map *bpf_map_get(u32 ufd); struct bpf_map *bpf_map_get_with_uref(u32 ufd); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index c67012d28e52..da878d957911 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1842,7 +1842,7 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, * bpf_list_head which needs to be freed. */ migrate_disable(); - __bpf_obj_drop_impl(obj, field->graph_root.value_rec); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); migrate_enable(); } } @@ -1881,7 +1881,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, migrate_disable(); - __bpf_obj_drop_impl(obj, field->graph_root.value_rec); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); migrate_enable(); } } @@ -1913,8 +1913,10 @@ __bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign) } /* Must be called under migrate_disable(), as required by bpf_mem_free */ -void __bpf_obj_drop_impl(void *p, const struct btf_record *rec) +void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu) { + struct bpf_mem_alloc *ma; + if (rec && rec->refcount_off >= 0 && !refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) { /* Object is refcounted and refcount_dec didn't result in 0 @@ -1926,10 +1928,14 @@ void __bpf_obj_drop_impl(void *p, const struct btf_record *rec) if (rec) bpf_obj_free_fields(rec, p); + if (percpu) + ma = &bpf_global_percpu_ma; + else + ma = &bpf_global_ma; if (rec && rec->refcount_off >= 0) - bpf_mem_free_rcu(&bpf_global_ma, p); + bpf_mem_free_rcu(ma, p); else - bpf_mem_free(&bpf_global_ma, p); + bpf_mem_free(ma, p); } __bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) @@ -1937,7 +1943,7 @@ __bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) struct btf_struct_meta *meta = meta__ign; void *p = p__alloc; - __bpf_obj_drop_impl(p, meta ? meta->record : NULL); + __bpf_obj_drop_impl(p, meta ? meta->record : NULL, false); } __bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign) @@ -1981,7 +1987,7 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, */ if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { /* Only called from BPF prog, no need to migrate_disable */ - __bpf_obj_drop_impl((void *)n - off, rec); + __bpf_obj_drop_impl((void *)n - off, rec, false); return -EINVAL; } @@ -2080,7 +2086,7 @@ static int __bpf_rbtree_add(struct bpf_rb_root *root, */ if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { /* Only called from BPF prog, no need to migrate_disable */ - __bpf_obj_drop_impl((void *)n - off, rec); + __bpf_obj_drop_impl((void *)n - off, rec, false); return -EINVAL; } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 69998f84f7c8..a9b2cb500bf7 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -660,8 +660,8 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) field->kptr.btf_id); migrate_disable(); __bpf_obj_drop_impl(xchgd_field, pointee_struct_meta ? - pointee_struct_meta->record : - NULL); + pointee_struct_meta->record : NULL, + fields[i].type == BPF_KPTR_PERCPU); migrate_enable(); } else { field->kptr.dtor(xchgd_field); -- 2.29.2