From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2C79FEFB60 for ; Fri, 27 Feb 2026 15:37:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA5696B008C; Fri, 27 Feb 2026 10:37:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E0BCD6B0099; Fri, 27 Feb 2026 10:37:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE3DD6B009D; Fri, 27 Feb 2026 10:37:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ACD0F6B0093 for ; Fri, 27 Feb 2026 10:37:39 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6360213B947 for ; Fri, 27 Feb 2026 15:37:39 +0000 (UTC) X-FDA: 84490641438.14.EEF51D7 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) by imf29.hostedemail.com (Postfix) with ESMTP id AFC4C12000E for ; Fri, 27 Feb 2026 15:37:37 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=mZkLpNSx; spf=pass (imf29.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 158.69.130.18 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1772206657; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TkHwF7BlYff6jdk+S19J6dVNCPWWYptmECBBQYyXzl8=; b=fb0W7pkE3DHqguLjySzK1+n4JfK6qEyrenXk8AcLuwREBuw7pxE6N83/0H6UBRe4a1ekkC Z8bfme97p6+WsxRHW5Z/FjCCeSfIBGK1A6HRy1JyArkNFR9mweVDB1b7ihdxsHd+VFOq4M AWSyank92UVgdMqlF2cnK4NOyDqJL5I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1772206657; a=rsa-sha256; cv=none; b=6TX0ROyX3yEwqYKak7DFnpB4ql0K46RXMKfg5ld4OWWc7U5wO9zBIRNW8Enmvi0wYEhDQ2 9eEjE2tvClaf/+OPYYmjMB5+y3UFjKWyk+wcV+PVOmgmmK8+Sg51oK/P5EF65dgRMbu+5X ocAfbJsC8kT0vOHc+4Jg+dxVqqgM5Hs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=mZkLpNSx; spf=pass (imf29.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 158.69.130.18 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1772206657; bh=TkHwF7BlYff6jdk+S19J6dVNCPWWYptmECBBQYyXzl8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mZkLpNSx9822xzcf/BesAB0krI1At1wDqeTKB5mw71wTCgO3xnm6e+/Jqa5jnNQs4 +2DWJb7I5DcZkG+dfUkJlGSQfH7YZau0+Ego9UObawBmp4T7Sx0K02vC9nhMHhntaK bk8QYEsqZomV1hXDT3wLnK2yz02ysyHIVMGDJm+/iaxkXD3cEkjYI3bh9ccJZ5Ijnm GBi3G1BLlgf9OtoOmY45YcB/r2YX2D89ZjQe04PbYn3WPy/aWVZwlIitrHPDXhbQtH 3d1/z+Lr5NCob1bwlS29la9h6oTpyvAl/WmNQ+rfMuOnOOvlJshYccLBJhi5Waa/Tx q1PbyEekLSS5g== Received: from thinkos.internal.efficios.com (mtl.efficios.com [216.120.195.104]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4fMssF0B9MzKCt; Fri, 27 Feb 2026 10:37:37 -0500 (EST) From: Mathieu Desnoyers To: Andrew Morton Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , "Paul E. McKenney" , Steven Rostedt , Masami Hiramatsu , Dennis Zhou , Tejun Heo , Christoph Lameter , Martin Liu , David Rientjes , christian.koenig@amd.com, Shakeel Butt , SeongJae Park , Michal Hocko , Johannes Weiner , Sweet Tea Dorminy , Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Vlastimil Babka , Christian Brauner , Wei Yang , David Hildenbrand , Miaohe Lin , Al Viro , linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, Yu Zhao , Roman Gushchin , Mateusz Guzik , Matthew Wilcox , Baolin Wang , Aboorva Devarajan Subject: [PATCH v18 2/3] lib: Test hierarchical per-cpu counters Date: Fri, 27 Feb 2026 10:37:29 -0500 Message-Id: <20260227153730.1556542-3-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260227153730.1556542-1-mathieu.desnoyers@efficios.com> References: <20260227153730.1556542-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: AFC4C12000E X-Stat-Signature: po9gqdnardizf8r9mszxdbrbrdju4b38 X-Rspam-User: X-HE-Tag: 1772206657-235586 X-HE-Meta: U2FsdGVkX1+beEyl6UxnOJS531MZSJtcX94WaC7QBVHRWxNNXnwGW8dH5NHoN9x9T4/wBNHpMCD4KVn6eP4Z+FZnJN9/I+HQZEOaaYykbB1LIFi4hx7Ioz4E/rtrzyDjnfgjyTLPfGKq+wUdf0ixIUYFuxfPAQ7GFX9kpBHoWNl+sHrJyaDD4tDVzTfGwST41EtO2v0EEUTWA249kMGICGh9cF4fcxwwBtgeFL5xrychMVf+Qfq4MwrNeT7+y6onm8K17vYUu2H6DnQEiOs3TKVuB8bPgkBXOuQ0fE4Rd+h/VlE2G4Q8RwVTZ8Dz3todSKwVhFDRr1Fd7dH6TdisAxkCvFx2I5lSGYLiBTpWC+rFMaPrGOlXMCcQdHvkIBJfNKVzLlOdl5rC2lAxXxpz0HHNFfYwpSjgxcGZE1K8eimkOI7VxfaccuQcrzZPNmHENVk5O5Zbg4CXWK198B2psUJqFPWlDzTCd51JKXNgSWeWuDuX5+03TOBdHsQb9G03OIurOeqI8arS27Ey0jjO/bhZgRcZUfqM5fi6mbHvLMmlrFd35gMznjvpRmBC4+2m5Bb4Nruxe2KoNqIIiYsC/D/l4yXh8aw0SDdgmBaeZPwUaRVqlhvCKfzVNo9rqtwMZ26VkeLaa5oxH3t/N+bqOy/UVlTjZwDg4YwmI75Eb3r3MwrA8ygmg2i+0/H2hZwwbVGgxJLlPDFIudSKqXZAWbNcr16XnxsWohiaf7zpEg1uevdkcOl5kEmHs2XcZx2pyCRHVgQrpgwZP/ip6Ki3B/JJleGU2YenQsSV8c5a2AWqoXmBO9BDE/ccjcXyA5bNmyzZvuKgGnUedGar0MD+fJmWVrqFiLq0xGLJNU0a30GjjPu5x7ffg8IDE13d3BWTw6UTO7YUooWMgBuC/WnUJ68/wsoObx+a0GO7Hnn5IveNrSsGmkCtuualggHfuf+GfI6DrW/7EbMzEDJjA7O WV3yoINX bFF4P62CDB0dHApRlPUmeT6lB/P+QfWlkod+WU1T+45/vXElNFRfLrPnpIaumwLJTM/4EvoK6gJzmBT8z2MAWt4wIUUldL+fKfF8tA2QNHNtBWZiQia0qKbdT94fXM+rSYBiFhFJV/OAG/Sr/r/LcXsLvejaHfd3K16eSIE9Ry529ZCpcEHf/jy2B5JoGYTw1ex8q82W+IlPH+nOZkXB1rwDvGipJJH4bQbOdpNmHnWgdGRDZ5LrpN8IHDt/Hyeab71gZ8nPJJnDdXbZq8GShDyLJ55MN2GH1jWjKJ7+BR6K+ad5HgsdYlWbTsHB8Vo87Lb7TnvWx3KHc8IZW4/6W/kSzKuRX+5L6gA6fqW/QaPrt/9qruG/y/C6wbtMXjGfQpGguTf4J3VQddZ3lJvyoswuw3wd/7HR/oJ1kwfwNPZlTqhaTRFlCLkqG970FyTkDqS4Oty8pAG9Dq02+n6SVtpuCkFHyjbSXqv4AaXkWf4lQLSFNdLdmA29HSFmMtoRCQ+k8Ho+yVYq1NXsjRLBQJzyt1/uCAhjcilApvPSQ6kWFxc9CHSqqp3sNrFH+P9E/yX9TpUn56WxWm85icbU3rRCkDGHZlLD/dOrq8DTAglaFRQHuwt/xUXlJIOAleWRAjXRmMyulnEYNy364LjoSWTbcOeQIXWrj1Ge9ceOhdsXYsG/C1LptPzcLGz77cWToEi5KzgS/aaodIAuQ3mLSS6tVRYPR2SVlo19NlIgoxLTbursA+hWRGyCgFHAgqCW70G/ClXCRNvEAJTuYAn3m3vCVJVozw+97VZl/ICNCrZlKBvG/EbC55m06W/yW7dbaPc3yM4ImWF+hI1h0+rPW/g1KOwFMqvoU+W1oOIXm8uQqm3h5n9UYd8Ohg9Vi9zW+/PIamTQOJuxqHOOdY2RpkulzKk3IFpXJgySO2w9I2jM3h+EJnSnfBqUNN6N5IaVAhcut2AJ5z+YzU+p2Q1QfmcwANTJC KcTx1/tB t3v3sMPOMLFjOyBxat/V+Y7pYgVbN4Ae8b902kvC0vUSLnQ9jOOtVa5aAcg2O9VeJp/0tEIq6HW6KW+S3I5KNg== Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce Kunit tests for hierarchical per-cpu counters. Keep track of two sets of hierarchical counters, each meant to have the same precise sum at any time, but distributed differently across the topology. Keep track of an atomic counter along with each hierarchical counter, for sum validation. Those tests cover: - Single-threaded (no concurrency) updates. - Concurrent updates of counters from various CPUs. Perform the following validations: - Compare the precise sum of counters with the sum tracked by an atomic counter. - Compare the precise sum of two sets of hierarchical counters. - Approximated comparison of hierarchical counter with atomic counter. - Approximated comparison of two sets of hierarchical counters. - Validate the bounds of approximation ranges. Run with the following .kunit/.kunitconfig: CONFIG_KUNIT=y CONFIG_SMP=y CONFIG_PREEMPT=y CONFIG_NR_CPUS=32 CONFIG_HOTPLUG_CPU=y CONFIG_PERCPU_COUNTER_TREE_TEST=y With the following execution (to use SMP): ./tools/testing/kunit/kunit.py run --arch=x86_64 --qemu_args="-smp 12" Signed-off-by: Mathieu Desnoyers Cc: Andrew Morton Cc: "Paul E. McKenney" Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Martin Liu Cc: David Rientjes Cc: christian.koenig@amd.com Cc: Shakeel Butt Cc: SeongJae Park Cc: Michal Hocko Cc: Johannes Weiner Cc: Sweet Tea Dorminy Cc: Lorenzo Stoakes Cc: "Liam R . Howlett" Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Christian Brauner Cc: Wei Yang Cc: David Hildenbrand Cc: Miaohe Lin Cc: Al Viro Cc: linux-mm@kvack.org Cc: linux-trace-kernel@vger.kernel.org Cc: Yu Zhao Cc: Roman Gushchin Cc: Mateusz Guzik Cc: Matthew Wilcox Cc: Baolin Wang Cc: Aboorva Devarajan --- Changes since v17: - Use percpu_counter_tree_{init,destroy}_many, - Add coverage for percpu_counter_tree_{init,destroy}, - Add coverage for percpu_counter_tree_set. - Fix wait queue head reinit bug. Use DECLARE_WAIT_QUEUE_HEAD instead. --- lib/Kconfig | 12 + lib/tests/Makefile | 2 + lib/tests/percpu_counter_tree_kunit.c | 399 ++++++++++++++++++++++++++ 3 files changed, 413 insertions(+) create mode 100644 lib/tests/percpu_counter_tree_kunit.c diff --git a/lib/Kconfig b/lib/Kconfig index 0f2fb9610647..0b8241e5b548 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -52,6 +52,18 @@ config PACKING_KUNIT_TEST When in doubt, say N. +config PERCPU_COUNTER_TREE_TEST + tristate "Hierarchical Per-CPU counter test" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + This builds Kunit tests for the hierarchical per-cpu counters. + + For more information on KUnit and unit tests in general, + please refer to the KUnit documentation in Documentation/dev-tools/kunit/. + + When in doubt, say N. + config BITREVERSE tristate diff --git a/lib/tests/Makefile b/lib/tests/Makefile index 05f74edbc62b..d282aa23d273 100644 --- a/lib/tests/Makefile +++ b/lib/tests/Makefile @@ -56,4 +56,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o +obj-$(CONFIG_PERCPU_COUNTER_TREE_TEST) += percpu_counter_tree_kunit.o + obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/ diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c new file mode 100644 index 000000000000..a79176655c4b --- /dev/null +++ b/lib/tests/percpu_counter_tree_kunit.c @@ -0,0 +1,399 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers + +#include +#include +#include +#include +#include + +struct multi_thread_test_data { + long increment; + int nr_inc; + int counter_index; +}; + +#define NR_COUNTERS 2 + +/* Hierarchical per-CPU counter instances. */ +static struct percpu_counter_tree counter[NR_COUNTERS]; +static struct percpu_counter_tree_level_item *items; + +/* Global atomic counters for validation. */ +static atomic_long_t global_counter[NR_COUNTERS]; + +static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq); +static atomic_t kernel_threads_to_run; + +static void complete_work(void) +{ + if (atomic_dec_and_test(&kernel_threads_to_run)) + wake_up(&kernel_threads_wq); +} + +static void hpcc_print_info(struct kunit *test) +{ + kunit_info(test, "Running test with %d CPUs\n", num_online_cpus()); +} + +static void add_to_counter(int counter_index, unsigned int nr_inc, long increment) +{ + unsigned int i; + + for (i = 0; i < nr_inc; i++) { + percpu_counter_tree_add(&counter[counter_index], increment); + atomic_long_add(increment, &global_counter[counter_index]); + } +} + +static void check_counters(struct kunit *test) +{ + int counter_index; + + /* Compare each counter with its global counter. */ + for (counter_index = 0; counter_index < NR_COUNTERS; counter_index++) { + long v = atomic_long_read(&global_counter[counter_index]); + long approx_sum = percpu_counter_tree_approximate_sum(&counter[counter_index]); + unsigned long under_accuracy = 0, over_accuracy = 0; + long precise_min, precise_max; + + /* Precise comparison. */ + KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[counter_index]), v); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(&counter[counter_index], v)); + + /* Approximate comparison. */ + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&counter[counter_index], v)); + + /* Accuracy limits checks. */ + percpu_counter_tree_approximate_accuracy_range(&counter[counter_index], &under_accuracy, &over_accuracy); + + KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - under_accuracy)), 0); + KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + over_accuracy)), 0); + KUNIT_EXPECT_GT(test, (long)(approx_sum - (v - under_accuracy - 1)), 0); + KUNIT_EXPECT_LT(test, (long)(approx_sum - (v + over_accuracy + 1)), 0); + + /* Precise min/max range check. */ + percpu_counter_tree_approximate_min_max_range(approx_sum, under_accuracy, over_accuracy, &precise_min, &precise_max); + + KUNIT_EXPECT_GE(test, v - precise_min, 0); + KUNIT_EXPECT_LE(test, v - precise_max, 0); + KUNIT_EXPECT_GT(test, v - (precise_min - 1), 0); + KUNIT_EXPECT_LT(test, v - (precise_max + 1), 0); + } + /* Compare each counter with the second counter. */ + KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]), percpu_counter_tree_precise_sum(&counter[1])); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1])); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1])); +} + +static int multi_thread_worker_fn(void *data) +{ + struct multi_thread_test_data *td = data; + + add_to_counter(td->counter_index, td->nr_inc, td->increment); + complete_work(); + kfree(td); + return 0; +} + +static void test_run_on_specific_cpu(struct kunit *test, int target_cpu, int counter_index, unsigned int nr_inc, long increment) +{ + struct task_struct *task; + struct multi_thread_test_data *td = kzalloc(sizeof(struct multi_thread_test_data), GFP_KERNEL); + + KUNIT_EXPECT_PTR_NE(test, td, NULL); + td->increment = increment; + td->nr_inc = nr_inc; + td->counter_index = counter_index; + atomic_inc(&kernel_threads_to_run); + task = kthread_run_on_cpu(multi_thread_worker_fn, td, target_cpu, "kunit_multi_thread_worker"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, task); +} + +static void init_kthreads(void) +{ + atomic_set(&kernel_threads_to_run, 1); +} + +static void fini_kthreads(void) +{ + /* Release our own reference. */ + complete_work(); + /* Wait for all others threads to run. */ + wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0)); +} + +static void test_sync_kthreads(void) +{ + fini_kthreads(); + init_kthreads(); +} + +static void init_counters(struct kunit *test, unsigned long batch_size) +{ + int i, ret; + + items = kzalloc(percpu_counter_tree_items_size() * NR_COUNTERS, GFP_KERNEL); + KUNIT_EXPECT_PTR_NE(test, items, NULL); + ret = percpu_counter_tree_init_many(counter, items, NR_COUNTERS, batch_size, GFP_KERNEL); + KUNIT_EXPECT_EQ(test, ret, 0); + + for (i = 0; i < NR_COUNTERS; i++) + atomic_long_set(&global_counter[i], 0); +} + +static void fini_counters(void) +{ + percpu_counter_tree_destroy_many(counter, NR_COUNTERS); + kfree(items); +} + +enum up_test_inc_type { + INC_ONE, + INC_MINUS_ONE, + INC_RANDOM, +}; + +/* + * Single-threaded tests. Those use many threads to run on various CPUs, + * but synchronize for completion of each thread before running the + * next, effectively making sure there are no concurrent updates. + */ +static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1, enum up_test_inc_type type) +{ + unsigned long batch_size_order = 5; + int cpu0 = _cpu0; + int cpu1 = _cpu1; + int i; + + init_counters(test, 1UL << batch_size_order); + init_kthreads(); + for (i = 0; i < 10000; i++) { + long increment; + + switch (type) { + case INC_ONE: + increment = 1; + break; + case INC_MINUS_ONE: + increment = -1; + break; + case INC_RANDOM: + increment = (long) get_random_long() % 50000; + break; + } + if (_cpu0 < 0) + cpu0 = cpumask_any_distribute(cpu_online_mask); + if (_cpu1 < 0) + cpu1 = cpumask_any_distribute(cpu_online_mask); + test_run_on_specific_cpu(test, cpu0, 0, 1, increment); + test_sync_kthreads(); + test_run_on_specific_cpu(test, cpu1, 1, 1, increment); + test_sync_kthreads(); + check_counters(test); + } + fini_kthreads(); + fini_counters(); +} + +static void hpcc_test_single_thread_first(struct kunit *test) +{ + int cpu = cpumask_first(cpu_online_mask); + + do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE); + do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE); + do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM); +} + +static void hpcc_test_single_thread_first_random(struct kunit *test) +{ + int cpu = cpumask_first(cpu_online_mask); + + do_hpcc_test_single_thread(test, cpu, -1, INC_ONE); + do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE); + do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM); +} + +static void hpcc_test_single_thread_random(struct kunit *test) +{ + do_hpcc_test_single_thread(test, -1, -1, INC_ONE); + do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE); + do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM); +} + +/* Multi-threaded SMP tests. */ + +static void do_hpcc_multi_thread_increment_each_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment) +{ + int cpu; + + init_counters(test, batch_size); + init_kthreads(); + for_each_online_cpu(cpu) { + test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment); + test_run_on_specific_cpu(test, cpu, 1, nr_inc, increment); + } + fini_kthreads(); + check_counters(test); + fini_counters(); +} + +static void do_hpcc_multi_thread_increment_even_cpus(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment) +{ + int cpu; + + init_counters(test, batch_size); + init_kthreads(); + for_each_online_cpu(cpu) { + test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment); + test_run_on_specific_cpu(test, cpu & ~1, 1, nr_inc, increment); /* even cpus. */ + } + fini_kthreads(); + check_counters(test); + fini_counters(); +} + +static void do_hpcc_multi_thread_increment_single_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment) +{ + int cpu; + + init_counters(test, batch_size); + init_kthreads(); + for_each_online_cpu(cpu) { + test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment); + test_run_on_specific_cpu(test, cpumask_first(cpu_online_mask), 1, nr_inc, increment); + } + fini_kthreads(); + check_counters(test); + fini_counters(); +} + +static void do_hpcc_multi_thread_increment_random_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment) +{ + int cpu; + + init_counters(test, batch_size); + init_kthreads(); + for_each_online_cpu(cpu) { + test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment); + test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment); + } + fini_kthreads(); + check_counters(test); + fini_counters(); +} + +static void hpcc_test_multi_thread_batch_increment(struct kunit *test) +{ + unsigned long batch_size_order; + + for (batch_size_order = 2; batch_size_order < 10; batch_size_order++) { + unsigned int nr_inc; + + for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) { + long increment; + + for (increment = 1; increment < 100000; increment *= 10) { + do_hpcc_multi_thread_increment_each_cpu(test, 1UL << batch_size_order, nr_inc, increment); + do_hpcc_multi_thread_increment_even_cpus(test, 1UL << batch_size_order, nr_inc, increment); + do_hpcc_multi_thread_increment_single_cpu(test, 1UL << batch_size_order, nr_inc, increment); + do_hpcc_multi_thread_increment_random_cpu(test, 1UL << batch_size_order, nr_inc, increment); + } + } + } +} + +static void hpcc_test_multi_thread_random_walk(struct kunit *test) +{ + unsigned long batch_size_order = 5; + int loop; + + for (loop = 0; loop < 100; loop++) { + int i; + + init_counters(test, 1UL << batch_size_order); + init_kthreads(); + for (i = 0; i < 1000; i++) { + long increment = (long) get_random_long() % 512; + unsigned int nr_inc = ((unsigned long) get_random_long()) % 1024; + + test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 0, nr_inc, increment); + test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment); + } + fini_kthreads(); + check_counters(test); + fini_counters(); + } +} + +static void hpcc_test_init_one(struct kunit *test) +{ + struct percpu_counter_tree pct; + struct percpu_counter_tree_level_item *counter_items; + int ret; + + counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL); + KUNIT_EXPECT_PTR_NE(test, counter_items, NULL); + ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL); + KUNIT_EXPECT_EQ(test, ret, 0); + + percpu_counter_tree_destroy(&pct); + kfree(counter_items); +} + +static void hpcc_test_set(struct kunit *test) +{ + static long values[] = { + 5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0, + -5, -100, -127, -128, -255, -256, -4095, -4096, -500000, + }; + struct percpu_counter_tree pct; + struct percpu_counter_tree_level_item *counter_items; + int i, ret; + + counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL); + KUNIT_EXPECT_PTR_NE(test, counter_items, NULL); + ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL); + KUNIT_EXPECT_EQ(test, ret, 0); + + for (i = 0; i < ARRAY_SIZE(values); i++) { + long v = values[i]; + + percpu_counter_tree_set(&pct, v); + KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), v); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, v)); + + percpu_counter_tree_add(&pct, v); + KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 2 * v); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 2 * v)); + + percpu_counter_tree_add(&pct, -2 * v); + KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 0); + KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 0)); + } + + percpu_counter_tree_destroy(&pct); + kfree(counter_items); +} + +static struct kunit_case hpcc_test_cases[] = { + KUNIT_CASE(hpcc_print_info), + KUNIT_CASE(hpcc_test_single_thread_first), + KUNIT_CASE(hpcc_test_single_thread_first_random), + KUNIT_CASE(hpcc_test_single_thread_random), + KUNIT_CASE(hpcc_test_multi_thread_batch_increment), + KUNIT_CASE(hpcc_test_multi_thread_random_walk), + KUNIT_CASE(hpcc_test_init_one), + KUNIT_CASE(hpcc_test_set), + {} +}; + +static struct kunit_suite hpcc_test_suite = { + .name = "percpu_counter_tree", + .test_cases = hpcc_test_cases, +}; + +kunit_test_suite(hpcc_test_suite); + +MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters"); +MODULE_LICENSE("Dual MIT/GPL"); -- 2.39.5