From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50B6EC00A98 for ; Fri, 20 Oct 2023 13:50:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A450C8D00D0; Fri, 20 Oct 2023 09:50:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A6A68D0003; Fri, 20 Oct 2023 09:50:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F98F8D00D0; Fri, 20 Oct 2023 09:50:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 714748D0003 for ; Fri, 20 Oct 2023 09:50:22 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 452331CBF4E for ; Fri, 20 Oct 2023 13:50:22 +0000 (UTC) X-FDA: 81365974284.29.47BD257 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) by imf24.hostedemail.com (Postfix) with ESMTP id 924E6180007 for ; Fri, 20 Oct 2023 13:50:18 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; spf=none (imf24.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697809820; a=rsa-sha256; cv=none; b=N2Z6Y6BbUGWQQqnRfneoAY+Ua2hRHPY3zB8S+PY+Lj2S6IM7hcnAtU2qmTyF+BLoStSTWq FIiG825N+zkgjRzvgSKJ4+swX3eVkwo+bpoNU6144A8BEiVp6IrdNo8ZohQDybXLo4qid9 qtYKSYVVPNxO+s4Ndcpf0tn7MnGZDnI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; spf=none (imf24.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697809820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n5XqlYEhcPnWYFKo0I+TP7AYESMtLfneJDVzaJMx6w4=; b=UutGP34DJZjwzOD/D3Hi4m60OFFsAg/xmcfL2ffvCxVZrOxNs2ie2AP6yJc7Qx+Bd1T8YF WMWem55SrNeGew4Ss4wp9ZvpgtV1f2zJUoeWiWdfOtxqvNSkZxfmwKwwhAHBcxGJBWbB6u nYmLk/jTSFsM+T7YRv3C88BF4XwGPQ0= Received: from mail02.huawei.com (unknown [172.30.67.153]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4SBlpS0kCTz4f3n6F for ; Fri, 20 Oct 2023 21:30:56 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP4 (Coremail) with SMTP id gCh0CgBn+dgIgTJlmYjjDQ--.7231S11; Fri, 20 Oct 2023 21:30:59 +0800 (CST) From: Hou Tao To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: Martin KaFai Lau , Alexei Starovoitov , Andrii Nakryiko , Song Liu , Hao Luo , Yonghong Song , Daniel Borkmann , KP Singh , Stanislav Fomichev , Jiri Olsa , John Fastabend , houtao1@huawei.com, Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton Subject: [PATCH bpf-next v3 7/7] selftests/bpf: Add more test cases for bpf memory allocator Date: Fri, 20 Oct 2023 21:32:02 +0800 Message-Id: <20231020133202.4043247-8-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20231020133202.4043247-1-houtao@huaweicloud.com> References: <20231020133202.4043247-1-houtao@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:gCh0CgBn+dgIgTJlmYjjDQ--.7231S11 X-Coremail-Antispam: 1UD129KBjvJXoW3Wr4rXw15Cw4kJw1fJFyrJFb_yoWfArW3pF y0yrySyr1kWr1Ig34Yvw4UCa4rZF4fWw15GrZ5WFyUury3X34UArn5CFyUJF95GFZ2gr45 Aas0gF97KF4xA3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBSb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4UJVWxJr1lIxAI cVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r4j6F4UMIIF0xvEx4A2js IEc7CjxVAFwI0_Gr1j6F4UJbIYCTnIWIevJa73UjIFyTuYvjxUFgAwUUUUU X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 924E6180007 X-Stat-Signature: ezp3zu7x845br1m7dqtrr6whpkaa1jho X-HE-Tag: 1697809818-102569 X-HE-Meta: U2FsdGVkX1/V1L4y4mn9UVNvlrGf1ZPi8L/oLjn2YUkMsiXTM5YX9u1xnYcaIaTNNuL699kpBDpZLbHCFMSnTHTxlVZNn8AvdR/nHdkmZDL7o8VxSwsVYZq36CmUJDNRLatZsI4FrKQIvz/Lm5/aRSfpnwN0+B5CB6UzGPbnuinFPkWdoMcBbWfR+RlLVMPBJ94IE3fCDf2sXAnKKPiQ1K19A2cZ1o4BeJmZyBm9t/mtHq0qeuUQeRqUa8S4ZW0oawunAecDcBrlrl+HnqZJV1kJL4Uct5lInj4vEjuSGWuKK2e9bXA6IZvg3duwNc3IYJZSfA1Tua86NPALQbeOXnhj30jv8xTgwYz9gFts+vAp3d8jeVrhMuOx1THKKBrIJZ2XC1H/OQzPl7aJ1g7kPLJNZ2nMWm1VtKMJEbqRAe+HYb+Cks2XfCM2yPNLaxVgJ5XIU3I3LfU9aBsm05fwdLbCHS/bWnPLajwJApDg/IZ0ApBiCtwI//tqYTwwImL5ORDEPsfGpEdZVisFhxTNx1Og2J6+AG5sZMKKqWBW/9IbqMD6Q5hBowtov0wnlkL+3lY3OygBssNk/up5gWmS29v+3eNAa8XGFx3uMxVh7AgK01FLQn0TII+iHIWUIwQbBafl9PtnYbEhWEzVdIfPkjaNI97kuHqKCzZxpJ6LoSfTI4ThpJxMJEfN+o/eul6RwASrw+QGol1aNmrc5LI8+V2OrfqNkWHsaMXKs5FjWOEuNuNdH5luxmqWcccAT18qK/OZ27MdG24EmGGrYYiVP6dnPwv6tV65+eP9VmvHQzBc1so49PQZwVr3fb8/IUfNJ3hpqlG0TVskw4GQvw6R1U0gtf/wU9I2H0nuzbkOA/zW+9k1x+9rDuA7odu9YSWtFWWLFtz0hc+n7qwOLXiuX2fSkWCsib1ja/oZOaByG44YMM0KOHIa2R0PgwqL+GQ93Z4JGY9lSG+VCXuEv8U MJNaiTes N7rFpOJwgCmiyTPhijyLaKp6Z+ESsriaN6he1mrBwmIo0ybQsegr4e4aiROrLr1Mu1sroaOg1aOsowY8hSBLoPrQ+oivs+GywAyxmtKFKsYKtkB2hI4+Lk3deD61WWN70o4h+iY533mz5nnQOLcnn5PBjE+lsBQp1giJRW+OS7Nr2x3RdNgHKYTSO6bHy34dP8fFkXRhZtM+AxoEMsE4puBaziROinZVUPXZwW1DUxkfVZHCY5IV7ISAF/2IEbKqfdEe5lG8WxxHBy8U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hou Tao Add the following 3 test cases for bpf memory allocator: 1) Do allocation in bpf program and free through map free 2) Do batch per-cpu allocation and per-cpu free in bpf program 3) Do per-cpu allocation in bpf program and free through map free For per-cpu allocation, because per-cpu allocation can not refill timely sometimes, so test 2) and test 3) consider it is OK for bpf_percpu_obj_new_impl() to return NULL. Signed-off-by: Hou Tao --- .../selftests/bpf/prog_tests/test_bpf_ma.c | 20 +- .../testing/selftests/bpf/progs/test_bpf_ma.c | 180 +++++++++++++++++- 2 files changed, 193 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/test_bpf_ma.c b/tools/testing/selftests/bpf/prog_tests/test_bpf_ma.c index 0cca4e8ae38e0..d3491a84b3b98 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_bpf_ma.c +++ b/tools/testing/selftests/bpf/prog_tests/test_bpf_ma.c @@ -9,9 +9,10 @@ #include "test_bpf_ma.skel.h" -void test_test_bpf_ma(void) +static void do_bpf_ma_test(const char *name) { struct test_bpf_ma *skel; + struct bpf_program *prog; struct btf *btf; int i, err; @@ -34,6 +35,11 @@ void test_test_bpf_ma(void) skel->rodata->data_btf_ids[i] = id; } + prog = bpf_object__find_program_by_name(skel->obj, name); + if (!ASSERT_OK_PTR(prog, "invalid prog name")) + goto out; + bpf_program__set_autoload(prog, true); + err = test_bpf_ma__load(skel); if (!ASSERT_OK(err, "load")) goto out; @@ -48,3 +54,15 @@ void test_test_bpf_ma(void) out: test_bpf_ma__destroy(skel); } + +void test_test_bpf_ma(void) +{ + if (test__start_subtest("batch_alloc_free")) + do_bpf_ma_test("test_batch_alloc_free"); + if (test__start_subtest("free_through_map_free")) + do_bpf_ma_test("test_free_through_map_free"); + if (test__start_subtest("batch_percpu_alloc_free")) + do_bpf_ma_test("test_batch_percpu_alloc_free"); + if (test__start_subtest("percpu_free_through_map_free")) + do_bpf_ma_test("test_percpu_free_through_map_free"); +} diff --git a/tools/testing/selftests/bpf/progs/test_bpf_ma.c b/tools/testing/selftests/bpf/progs/test_bpf_ma.c index ecde41ae0fc87..b685a4aba6bd9 100644 --- a/tools/testing/selftests/bpf/progs/test_bpf_ma.c +++ b/tools/testing/selftests/bpf/progs/test_bpf_ma.c @@ -37,10 +37,20 @@ int pid = 0; __type(key, int); \ __type(value, struct map_value_##_size); \ __uint(max_entries, 128); \ - } array_##_size SEC(".maps"); + } array_##_size SEC(".maps") -static __always_inline void batch_alloc_free(struct bpf_map *map, unsigned int batch, - unsigned int idx) +#define DEFINE_ARRAY_WITH_PERCPU_KPTR(_size) \ + struct map_value_percpu_##_size { \ + struct bin_data_##_size __percpu_kptr * data; \ + }; \ + struct { \ + __uint(type, BPF_MAP_TYPE_ARRAY); \ + __type(key, int); \ + __type(value, struct map_value_percpu_##_size); \ + __uint(max_entries, 128); \ + } array_percpu_##_size SEC(".maps") + +static __always_inline void batch_alloc(struct bpf_map *map, unsigned int batch, unsigned int idx) { struct generic_map_value *value; unsigned int i, key; @@ -65,6 +75,14 @@ static __always_inline void batch_alloc_free(struct bpf_map *map, unsigned int b return; } } +} + +static __always_inline void batch_free(struct bpf_map *map, unsigned int batch, unsigned int idx) +{ + struct generic_map_value *value; + unsigned int i, key; + void *old; + for (i = 0; i < batch; i++) { key = i; value = bpf_map_lookup_elem(map, &key); @@ -81,8 +99,72 @@ static __always_inline void batch_alloc_free(struct bpf_map *map, unsigned int b } } +static __always_inline void batch_percpu_alloc(struct bpf_map *map, unsigned int batch, + unsigned int idx) +{ + struct generic_map_value *value; + unsigned int i, key; + void *old, *new; + + for (i = 0; i < batch; i++) { + key = i; + value = bpf_map_lookup_elem(map, &key); + if (!value) { + err = 1; + return; + } + /* per-cpu allocator may not be able to refill in time */ + new = bpf_percpu_obj_new_impl(data_btf_ids[idx], NULL); + if (!new) + continue; + + old = bpf_kptr_xchg(&value->data, new); + if (old) { + bpf_percpu_obj_drop(old); + err = 2; + return; + } + } +} + +static __always_inline void batch_percpu_free(struct bpf_map *map, unsigned int batch, + unsigned int idx) +{ + struct generic_map_value *value; + unsigned int i, key; + void *old; + + for (i = 0; i < batch; i++) { + key = i; + value = bpf_map_lookup_elem(map, &key); + if (!value) { + err = 3; + return; + } + old = bpf_kptr_xchg(&value->data, NULL); + if (!old) + continue; + bpf_percpu_obj_drop(old); + } +} + +#define CALL_BATCH_ALLOC(size, batch, idx) \ + batch_alloc((struct bpf_map *)(&array_##size), batch, idx) + #define CALL_BATCH_ALLOC_FREE(size, batch, idx) \ - batch_alloc_free((struct bpf_map *)(&array_##size), batch, idx) + do { \ + batch_alloc((struct bpf_map *)(&array_##size), batch, idx); \ + batch_free((struct bpf_map *)(&array_##size), batch, idx); \ + } while (0) + +#define CALL_BATCH_PERCPU_ALLOC(size, batch, idx) \ + batch_percpu_alloc((struct bpf_map *)(&array_percpu_##size), batch, idx) + +#define CALL_BATCH_PERCPU_ALLOC_FREE(size, batch, idx) \ + do { \ + batch_percpu_alloc((struct bpf_map *)(&array_percpu_##size), batch, idx); \ + batch_percpu_free((struct bpf_map *)(&array_percpu_##size), batch, idx); \ + } while (0) DEFINE_ARRAY_WITH_KPTR(8); DEFINE_ARRAY_WITH_KPTR(16); @@ -97,8 +179,21 @@ DEFINE_ARRAY_WITH_KPTR(1024); DEFINE_ARRAY_WITH_KPTR(2048); DEFINE_ARRAY_WITH_KPTR(4096); -SEC("fentry/" SYS_PREFIX "sys_nanosleep") -int test_bpf_mem_alloc_free(void *ctx) +/* per-cpu kptr doesn't support bin_data_8 which is a zero-sized array */ +DEFINE_ARRAY_WITH_PERCPU_KPTR(16); +DEFINE_ARRAY_WITH_PERCPU_KPTR(32); +DEFINE_ARRAY_WITH_PERCPU_KPTR(64); +DEFINE_ARRAY_WITH_PERCPU_KPTR(96); +DEFINE_ARRAY_WITH_PERCPU_KPTR(128); +DEFINE_ARRAY_WITH_PERCPU_KPTR(192); +DEFINE_ARRAY_WITH_PERCPU_KPTR(256); +DEFINE_ARRAY_WITH_PERCPU_KPTR(512); +DEFINE_ARRAY_WITH_PERCPU_KPTR(1024); +DEFINE_ARRAY_WITH_PERCPU_KPTR(2048); +DEFINE_ARRAY_WITH_PERCPU_KPTR(4096); + +SEC("?fentry/" SYS_PREFIX "sys_nanosleep") +int test_batch_alloc_free(void *ctx) { if ((u32)bpf_get_current_pid_tgid() != pid) return 0; @@ -121,3 +216,76 @@ int test_bpf_mem_alloc_free(void *ctx) return 0; } + +SEC("?fentry/" SYS_PREFIX "sys_nanosleep") +int test_free_through_map_free(void *ctx) +{ + if ((u32)bpf_get_current_pid_tgid() != pid) + return 0; + + /* Alloc 128 8-bytes objects in batch to trigger refilling, + * then free these objects through map free. + */ + CALL_BATCH_ALLOC(8, 128, 0); + CALL_BATCH_ALLOC(16, 128, 1); + CALL_BATCH_ALLOC(32, 128, 2); + CALL_BATCH_ALLOC(64, 128, 3); + CALL_BATCH_ALLOC(96, 128, 4); + CALL_BATCH_ALLOC(128, 128, 5); + CALL_BATCH_ALLOC(192, 128, 6); + CALL_BATCH_ALLOC(256, 128, 7); + CALL_BATCH_ALLOC(512, 64, 8); + CALL_BATCH_ALLOC(1024, 32, 9); + CALL_BATCH_ALLOC(2048, 16, 10); + CALL_BATCH_ALLOC(4096, 8, 11); + + return 0; +} + +SEC("?fentry/" SYS_PREFIX "sys_nanosleep") +int test_batch_percpu_alloc_free(void *ctx) +{ + if ((u32)bpf_get_current_pid_tgid() != pid) + return 0; + + /* Alloc 128 16-bytes per-cpu objects in batch to trigger refilling, + * then free 128 16-bytes per-cpu objects in batch to trigger freeing. + */ + CALL_BATCH_PERCPU_ALLOC_FREE(16, 128, 1); + CALL_BATCH_PERCPU_ALLOC_FREE(32, 128, 2); + CALL_BATCH_PERCPU_ALLOC_FREE(64, 128, 3); + CALL_BATCH_PERCPU_ALLOC_FREE(96, 128, 4); + CALL_BATCH_PERCPU_ALLOC_FREE(128, 128, 5); + CALL_BATCH_PERCPU_ALLOC_FREE(192, 128, 6); + CALL_BATCH_PERCPU_ALLOC_FREE(256, 128, 7); + CALL_BATCH_PERCPU_ALLOC_FREE(512, 64, 8); + CALL_BATCH_PERCPU_ALLOC_FREE(1024, 32, 9); + CALL_BATCH_PERCPU_ALLOC_FREE(2048, 16, 10); + CALL_BATCH_PERCPU_ALLOC_FREE(4096, 8, 11); + + return 0; +} + +SEC("?fentry/" SYS_PREFIX "sys_nanosleep") +int test_percpu_free_through_map_free(void *ctx) +{ + if ((u32)bpf_get_current_pid_tgid() != pid) + return 0; + + /* Alloc 128 16-bytes per-cpu objects in batch to trigger refilling, + * then free these object through map free. + */ + CALL_BATCH_PERCPU_ALLOC(16, 128, 1); + CALL_BATCH_PERCPU_ALLOC(32, 128, 2); + CALL_BATCH_PERCPU_ALLOC(64, 128, 3); + CALL_BATCH_PERCPU_ALLOC(96, 128, 4); + CALL_BATCH_PERCPU_ALLOC(128, 128, 5); + CALL_BATCH_PERCPU_ALLOC(192, 128, 6); + CALL_BATCH_PERCPU_ALLOC(256, 128, 7); + CALL_BATCH_PERCPU_ALLOC(512, 64, 8); + CALL_BATCH_PERCPU_ALLOC(1024, 32, 9); + CALL_BATCH_PERCPU_ALLOC(2048, 16, 10); + CALL_BATCH_PERCPU_ALLOC(4096, 8, 11); + + return 0; +} -- 2.29.2