From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1E61E95A66 for ; Sat, 7 Oct 2023 13:50:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C3F48D0015; Sat, 7 Oct 2023 09:50:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 070C18D0005; Sat, 7 Oct 2023 09:50:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2C838D0015; Sat, 7 Oct 2023 09:50:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D66CD8D0005 for ; Sat, 7 Oct 2023 09:50:11 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B308D16023B for ; Sat, 7 Oct 2023 13:50:11 +0000 (UTC) X-FDA: 81318799422.07.2D20F75 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) by imf20.hostedemail.com (Postfix) with ESMTP id 278481C0003 for ; Sat, 7 Oct 2023 13:50:08 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=none (imf20.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696686610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uIklnj7C6XSUTiEw1Q9BH2KnAsXVXIQd/t8QU4DinK8=; b=A0yYYK58vSjQTgAPIxMXVsmMVa9D1cZCPjYTviqdy5sGMrLNmW/56oU9Z6PmkTOhCez5u2 qOEWWde1CGOSROTZvPy4IX9/GUpOnjLbu4p9326DnLv0j6EiINADQ/3rfzRjzCc5JKSggT vjawmr9sThjfkZGNguw3I9HJtsy4Atc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696686610; a=rsa-sha256; cv=none; b=iIPnGu9lvEI0cqutJgwlmJ9/HYPqZ81ELeKWP6GBv7zJyW+EmOgQTfgW24T+Y7FkMPmJ30 E96m3SJGy4K8+rJyHE+625ZQucNDhZ4nPvEmTfWhoWWxjhHTUVF5L2BuCbe/ZrcvkRH7go 5MiZiDyeH0UAcOmThsHj/Ib8+X1+/Uk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=none (imf20.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com; dmarc=none Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4S2mrN1Hxkz4f3kKg for ; Sat, 7 Oct 2023 21:49:56 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP4 (Coremail) with SMTP id gCh0CgAnvdz9YSFl4YF2CQ--.30763S9; Sat, 07 Oct 2023 21:49:57 +0800 (CST) From: Hou Tao To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: Martin KaFai Lau , Alexei Starovoitov , Andrii Nakryiko , Song Liu , Hao Luo , Yonghong Song , Daniel Borkmann , KP Singh , Stanislav Fomichev , Jiri Olsa , John Fastabend , houtao1@huawei.com, Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton Subject: [PATCH bpf-next 5/6] bpf: Use bpf_global_percpu_ma for per-cpu kptr in __bpf_obj_drop_impl() Date: Sat, 7 Oct 2023 21:51:05 +0800 Message-Id: <20231007135106.3031284-6-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20231007135106.3031284-1-houtao@huaweicloud.com> References: <20231007135106.3031284-1-houtao@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:gCh0CgAnvdz9YSFl4YF2CQ--.30763S9 X-Coremail-Antispam: 1UD129KBjvJXoW3Jr1xGryxGrWkZrWkCrW8Crg_yoW7uF1UpF 4ftr12yr4kJFs2q3s8Gw4I9ryFqrW5Xa47G34kG34Fyr4avryDZw18CFyxua45trW8Kr1I y3Z09ry3A3y8AaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBSb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4UJVWxJr1lIxAI cVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2js IEc7CjxVAFwI0_Gr1j6F4UJbIYCTnIWIevJa73UjIFyTuYvjxUFgAwUUUUU X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 278481C0003 X-Rspam-User: X-Stat-Signature: qc7r7qd7bief7cis8mmzyqpcw8b16qen X-Rspamd-Server: rspam03 X-HE-Tag: 1696686608-793441 X-HE-Meta: U2FsdGVkX1/F/UyoAnmm61UwM16jFCxGGYv1H9AAHR9s9QsDIKH3N/VN72bwGTRk+Bs4wEqAOY3gMoNXHMUJ3ED/ywSRoWC1WqleFz6st1KuqULZCfihoe57M5VO7hawQlbbRlaJ4yJREKTHVxvs+mBDcwPaTxp/E93Xm8C0rxxkrq5WtvsoJzhhEI2MiQrCvpfbQ56UTVZ1mZGO51tH7lqUeXZUgvolb+moZespDcOFK7YRk+WdPR0saxyN+7EsGSjC5iIGIWGAqAL20KACNNiZyD7XGHHyYGNcrlY/6CoYUhC4ASb+8JNZ+X3W9Zqq7Q8dbg/Aa1Tu5tG5bzrv7bBXp//Nw7R/gEmvs3AE/m+ut0v72Q2S/YCy47SvTtVqbiNXTb4BVDd6/fgGUCXsOvwjE/GV2gvVutgaa3YfnwP7bq6HX3sH1hu4CLkfxu/KGdskmD7kjlqKyzYFtcoRNoRYW8b37GnMndn4VqKpqxlqkARJZfSjyNJdycOiKC/bXim6dQQkBeAmNRUkGqndtoDmHQYleJ8KtI4G0rIp0l28XVcAfgLnsTRGAfXPr4BZoORYNonStk1ipVoXwcZpuHwJ7maGJuLsJZN8EEly9/V+WW70uN0mf6tTTvDWWqYnEbDKVO25tm+PsuQ+LcxYpzRaBaAItzEvd4jkj1E+wwfd9o6fa4UYYU0tsMR7lVC3sycEFODyZtl7URJHTCusXDspRSRtLsh5pQu3lbNawTt2IMA5MySLe9GXzlXeF5Ta0M9t0K+CptGiXL6M54G2wORObHcMO2GpDGb4K67qslKy9+vouYpEIXXA+S3RAvEdZiH4MRW2SZdbLIptXjGK7EwL/PsuRsO5UcyPOVM1t4PEklKG2jeskj2SjLd7xQC4DJktoXsqGOaf90NVQOnDFZ4Q4eXR0RqI80SmRYFOhVWvAmmWqZE+TGU0yGlH9tWQTG3brmFadK/a1HJljCQ JEVu1Ypp WJHAmUtjM5Yg1JfH7PHnD+RAA88/Tv+mCAOJlNVsWl3PaR4XQd6O0yDI0sWfRFapzY8rjXBOAdewPgt5XKJuXCeeaJQjKumQcIHYhpEepMFNWGaXwuTmBpLr9IT+z/LTFB5tYT9HtQPMUKJIagBRjq+si8bzV8gbZqJs/5PgAJGljxvwpALlGWBnPPtnfrbvCqXp0Vdfjah6zP53sc+5k/mby+ri8/noL9/7YW6oSzSQ8hFlTgmqD1DiOwTgfO5Fqalc9ovDtqw3YINof5S/HZ6V9rUkD93EomW90mwJcxoTWU6dp1hWJYsOChboPAmQt1txf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hou Tao The following warning was reported when running "./test_progs -t test_bpf_ma/percpu_free_through_map_free": ------------[ cut here ]------------ WARNING: CPU: 1 PID: 68 at kernel/bpf/memalloc.c:342 CPU: 1 PID: 68 Comm: kworker/u16:2 Not tainted 6.6.0-rc2+ #222 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996) Workqueue: events_unbound bpf_map_free_deferred RIP: 0010:bpf_mem_refill+0x21c/0x2a0 ...... Call Trace: ? bpf_mem_refill+0x21c/0x2a0 irq_work_single+0x27/0x70 irq_work_run_list+0x2a/0x40 irq_work_run+0x18/0x40 __sysvec_irq_work+0x1c/0xc0 sysvec_irq_work+0x73/0x90 asm_sysvec_irq_work+0x1b/0x20 RIP: 0010:unit_free+0x50/0x80 ...... bpf_mem_free+0x46/0x60 __bpf_obj_drop_impl+0x40/0x90 bpf_obj_free_fields+0x17d/0x1a0 array_map_free+0x6b/0x170 bpf_map_free_deferred+0x54/0xa0 process_scheduled_works+0xba/0x370 worker_thread+0x16d/0x2e0 kthread+0x105/0x140 ret_from_fork+0x39/0x60 ret_from_fork_asm+0x1b/0x30 ---[ end trace 0000000000000000 ]--- The reason is simple: __bpf_obj_drop_impl() does not know the freeing field is a per-cpu pointer and it uses bpf_global_ma to free the pointer. Because bpf_global_ma is not a per-cpu allocator, so ksize() is used to select the corresponding cache. The bpf_mem_cache with 16-bytes unit_size will always be selected to do the unmatched free and it will trigger the warning in free_bulk() eventually. Because per-cpu kptr doesn't support list or rb-tree now, so fix the problem by only checking whether or not the type of kptr is per-cpu in bpf_obj_free_fields(), and using bpf_global_percpu_ma to these kptrs. Signed-off-by: Hou Tao --- kernel/bpf/helpers.c | 22 ++++++++++++++-------- kernel/bpf/internal.h | 2 +- kernel/bpf/syscall.c | 4 ++-- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 07f49f8831c0..078217c921e8 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1840,7 +1840,7 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, * bpf_list_head which needs to be freed. */ migrate_disable(); - __bpf_obj_drop_impl(obj, field->graph_root.value_rec); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); migrate_enable(); } } @@ -1879,7 +1879,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, migrate_disable(); - __bpf_obj_drop_impl(obj, field->graph_root.value_rec); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); migrate_enable(); } } @@ -1911,8 +1911,10 @@ __bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign) } /* Must be called under migrate_disable(), as required by bpf_mem_free */ -void __bpf_obj_drop_impl(void *p, const struct btf_record *rec) +void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu) { + struct bpf_mem_alloc *ma; + if (rec && rec->refcount_off >= 0 && !refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) { /* Object is refcounted and refcount_dec didn't result in 0 @@ -1924,10 +1926,14 @@ void __bpf_obj_drop_impl(void *p, const struct btf_record *rec) if (rec) bpf_obj_free_fields(rec, p); + if (percpu) + ma = &bpf_global_percpu_ma; + else + ma = &bpf_global_ma; if (rec && rec->refcount_off >= 0) - bpf_mem_free_rcu(&bpf_global_ma, p); + bpf_mem_free_rcu(ma, p); else - bpf_mem_free(&bpf_global_ma, p); + bpf_mem_free(ma, p); } __bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) @@ -1935,7 +1941,7 @@ __bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) struct btf_struct_meta *meta = meta__ign; void *p = p__alloc; - __bpf_obj_drop_impl(p, meta ? meta->record : NULL); + __bpf_obj_drop_impl(p, meta ? meta->record : NULL, false); } __bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign) @@ -1979,7 +1985,7 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, */ if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { /* Only called from BPF prog, no need to migrate_disable */ - __bpf_obj_drop_impl((void *)n - off, rec); + __bpf_obj_drop_impl((void *)n - off, rec, false); return -EINVAL; } @@ -2078,7 +2084,7 @@ static int __bpf_rbtree_add(struct bpf_rb_root *root, */ if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { /* Only called from BPF prog, no need to migrate_disable */ - __bpf_obj_drop_impl((void *)n - off, rec); + __bpf_obj_drop_impl((void *)n - off, rec, false); return -EINVAL; } diff --git a/kernel/bpf/internal.h b/kernel/bpf/internal.h index e233ea83eb0a..4c3cfdd6e6a2 100644 --- a/kernel/bpf/internal.h +++ b/kernel/bpf/internal.h @@ -6,6 +6,6 @@ struct btf_record; -void __bpf_obj_drop_impl(void *p, const struct btf_record *rec); +void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); #endif /* __BPF_INTERNAL_H_ */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7de4b9f97c8f..8dfc5d39c91d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -662,8 +662,8 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) field->kptr.btf_id); migrate_disable(); __bpf_obj_drop_impl(xchgd_field, pointee_struct_meta ? - pointee_struct_meta->record : - NULL); + pointee_struct_meta->record : NULL, + fields[i].type == BPF_KPTR_PERCPU); migrate_enable(); } else { field->kptr.dtor(xchgd_field); -- 2.29.2