From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <owner-linux-mm@kvack.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17])
	by smtp.lore.kernel.org (Postfix) with ESMTP id DF965C4332F
	for <linux-mm@archiver.kernel.org>; Tue, 12 Dec 2023 19:22:25 +0000 (UTC)
Received: by kanga.kvack.org (Postfix)
	id 729686B0342; Tue, 12 Dec 2023 14:22:25 -0500 (EST)
Received: by kanga.kvack.org (Postfix, from userid 40)
	id 6D9526B0343; Tue, 12 Dec 2023 14:22:25 -0500 (EST)
X-Delivered-To: int-list-linux-mm@kvack.org
Received: by kanga.kvack.org (Postfix, from userid 63042)
	id 553556B0344; Tue, 12 Dec 2023 14:22:25 -0500 (EST)
X-Delivered-To: linux-mm@kvack.org
Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12])
	by kanga.kvack.org (Postfix) with ESMTP id 410046B0342
	for <linux-mm@kvack.org>; Tue, 12 Dec 2023 14:22:25 -0500 (EST)
Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1])
	by unirelay04.hostedemail.com (Postfix) with ESMTP id 1A1D31A0A69
	for <linux-mm@kvack.org>; Tue, 12 Dec 2023 19:22:25 +0000 (UTC)
X-FDA: 81559137450.07.283570D
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
	by imf28.hostedemail.com (Postfix) with ESMTP id 654B8C000B
	for <linux-mm@kvack.org>; Tue, 12 Dec 2023 19:22:23 +0000 (UTC)
Authentication-Results: imf28.hostedemail.com;
	dkim=pass header.d=kernel.org header.s=k20201202 header.b=lSUwxVgD;
	dmarc=pass (policy=none) header.from=kernel.org;
	spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com;
	s=arc-20220608; t=1702408943;
	h=from:from:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references:dkim-signature;
	bh=PE3Ud83rQ8nRHqXBgbyIm2JkshmsAUhc2ptltMD0cvg=;
	b=ZQgeZ9XgvXYy8sZ9DR/cBfs/V1TtY+yJlEULy166QkozMXQJmULcn5KZVZ0hIP9gv0BJE+
	+xayT/f7QBrruwsUs11pzWANf1Aj+sK9pdGbuITLTFdx3lETty+TvEteMsffH8PJWF0LLS
	VThiTGuj8RE036j4+ZS+Ovf+lsxH+Go=
ARC-Authentication-Results: i=1;
	imf28.hostedemail.com;
	dkim=pass header.d=kernel.org header.s=k20201202 header.b=lSUwxVgD;
	dmarc=pass (policy=none) header.from=kernel.org;
	spf=pass (imf28.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org
ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702408943; a=rsa-sha256;
	cv=none;
	b=AEmOX5k7Eh8IOlYocX8RQQ0jQjWoq1nadkjKYBTOWmihiCust85f7K8JQHkyYcsYDRpzYc
	pDuRMWYnB0x2JmcW76u6Mez0XwizjP34ZWvMY5CHxRz1Ej2LYNPR55hijPPiKqRw7sIIOa
	cl7LMnxe43ivK7YN1TH5BGFqcKxNyp8=
Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])
	by dfw.source.kernel.org (Postfix) with ESMTP id 71B696123B;
	Tue, 12 Dec 2023 19:22:22 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id DDE4CC433C8;
	Tue, 12 Dec 2023 19:22:20 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
	s=k20201202; t=1702408941;
	bh=i5eA2fXhaq3Do9ywFpJzwHOFQOS7qYLx5BFeshX3BZI=;
	h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
	b=lSUwxVgDk2j4cnZPH/Zy7xE7vjsRTnpogG2abp5kyPRCtT/TZgdYhJerl+EIt8XO+
	 AZCxf+fPMQpNamCpOTnjqC4DOxO2jnXg7wVICwPQJUTf6RfCL14WhPZuHupkqX7l0O
	 Hq+nJ2IAFAZtDifyGfoR91Vlrlzdh0w8cg/NWZlNbz/mzlhu3X4Kwv8yuSz6GB6uHz
	 PVUgx+D3XmLxKMS4vyobPs59mP5iwt/lGzRutQe8tew0eS6GZcND9rnTgF9gVWr8sg
	 0Nerc0Xi7synedvS/VCWgFV37xO/7MVPkRGY//8a+vJkxCv/bYg7Qcn9afsFwQw62R
	 IjULLbahcHfEg==
From: SeongJae Park <sj@kernel.org>
To: SeongJae Park <sj@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Shuah Khan <shuah@kernel.org>,
	damon@lists.linux.dev,
	linux-mm@kvack.org,
	linux-kselftest@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 4/5] selftests/damon: add a test for update_schemes_tried_regions sysfs command
Date: Tue, 12 Dec 2023 19:22:18 +0000
Message-Id: <20231212192218.54095-1-sj@kernel.org>
X-Mailer: git-send-email 2.34.1
In-Reply-To: <20231212191206.52917-5-sj@kernel.org>
References: 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: 654B8C000B
X-Rspam-User: 
X-Rspamd-Server: rspam02
X-Stat-Signature: i3ufb83rso68p5qor4sjwees1siafimg
X-HE-Tag: 1702408943-318595
X-HE-Meta: U2FsdGVkX18rYwd0Dp3JijBoy469CfAtkgRIQnxtuaSKmNv8AppI7FRJchUcpA8WWv5PZ4DTM6M/UCUQDw+ijmWuBUnBVa41oR0w7kYnwtuF6mXOoxjE+FkijepeSKq96udf5CDhY9xBY4nzgBwisXZkofBSE34/sTUMwMSKhKMxhYlj+SwRxjMIliq9KFlP+UA6k7X24RrKMUxDNDHUVdH7dirYqJkyWZUIniQTwXFVf/cdtUagR9HQXcy/TFHnyVoK12nuj8XHImyEpozImbMSt3CZOpjFTlfr8iIG4WKNsG1nmO2lDBJrXCCp9311bSXLV5kfaF67Vmttflk842MyBlX+yZVayAYp2kr0Mag9+vNJjB4c+4jrNDVunMO/pYLwkI0MWsy/2k3znBOjHQ2AhHcbVxDKVnL1KPLEna1YtJnTqQ0dHOA94UqyS4WkSnPWIXgyuxhkdDkkmG3ipI5uPEg0KlbtcuF5ewIe5/wIki+qP67e+l21SVX3gBq5n/xSEgxONvV/HPtHncO/NGhqqvNXhZ3Nz62q0DOEwxiFC5OP6EsM17Z+PLLcsfpGLQtQiedb05iAdLHSWWew4bWLxucwUsVNCWEAF6tcXq32zC0A0/nCkRuB9XQFOVdZ5eODrU+n/Kzq0QBio61PUNYWQBfI5EZ8gn3O6cZLpt3fsRHn9qk5x5EvuvsHHLP/IL0Mduffys87Zu+3hPjZTdQgONHWCr3DEsmyh14RqPjSveqbDAbJeNG+aSpthtMIyt5TBpcPivY5bG16CSu3ljBhfhVnxA4KZc9uMmVjgYc7MTQrwH1VMlef8yWd6Lfsw/I88Co1Iv632iCQ4tZ1H08Gmlk06kvhl0D9pZCffTb/SBLWZjWo6sk0S8ztQlgvVceJooGF/q9K6a5HGr2yInCQ8o4KZQVHv4afDm7fPiOpx4Y/LRT8NmQYxlY6F6GxrdtJDu4vKvnJ0xAV/bm
 ojGpekqn
 ts5d5uxDT1zVde2es2GQ9tg/gi+S8EXm0eEsGpeY/IkiBRW8b5lVY99XkalNAc7xkq81vnFt+Ks6uinM1BJDamNdMB4r0aK8J9AiRFGtWUkIOjRFVyQwrjNd3k5faM1c4Q//jz41fJhMs0SSXpoEPgMJfVKyoh29JM9P3VWzlYb/RIsFDAK11dy4da+UJL2PkwwGw76i204ha9Xsp8UdFjkudJQH1GsbqTl4kKn7pD3k2K2X0Uqae8WkZZUrN7QIwpBagGm+TM5w+ZJkjwsAvcvfmba/asjjPizjKWACQbTuIZl2OHnsGX+Fh6QYUZhRuOO7c
X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4
Sender: owner-linux-mm@kvack.org
Precedence: bulk
X-Loop: owner-majordomo@kvack.org
List-ID: <linux-mm.kvack.org>
List-Subscribe: <mailto:majordomo@kvack.org>
List-Unsubscribe: <mailto:majordomo@kvack.org>

This and fifth patch of this patchset may not cleanly applicable, since those
are made on top of my out-of-tree experimental changes.  I will rebase these
and send v2.  Sorry for the noise.


Thanks,
SJ

On 2023-12-12T19:12:05+00:00 SeongJae Park <sj@kernel.org> wrote:

> Add a selftest for verifying the accuracy of DAMON's access monitoring
> functionality.  The test starts a program of artificial access pattern,
> monitor the access pattern using DAMON, and check if DAMON finds
> expected amount of hot data region (working set size) with only
> acceptable error rate.
> 
> Note that the acceptable error rate is set with only naive assumptions
> and small number of tests.  Hence failures of the test may not always
> mean DAMON is broken.  Rather than that, those could be a signal to
> better understand the real accuracy level of DAMON in wider
> environments.  Based on further finding, we could optimize DAMON or
> adjust the expectation of the test.
> 
> Signed-off-by: SeongJae Park <sj@kernel.org>
> ---
>  tools/testing/selftests/damon/Makefile        |  2 +
>  tools/testing/selftests/damon/access_memory.c | 41 ++++++++++++++
>  ...te_schemes_tried_regions_wss_estimation.py | 55 +++++++++++++++++++
>  3 files changed, 98 insertions(+)
>  create mode 100644 tools/testing/selftests/damon/access_memory.c
>  create mode 100755 tools/testing/selftests/damon/sysfs_update_schemes_tried_regions_wss_estimation.py
> 
> diff --git a/tools/testing/selftests/damon/Makefile b/tools/testing/selftests/damon/Makefile
> index d2105d41ea25..1363987709c6 100644
> --- a/tools/testing/selftests/damon/Makefile
> +++ b/tools/testing/selftests/damon/Makefile
> @@ -4,6 +4,7 @@
>  TEST_GEN_FILES += huge_count_read_write
>  TEST_GEN_FILES += dbgfs_target_ids_read_before_terminate_race
>  TEST_GEN_FILES += dbgfs_target_ids_pid_leak
> +TEST_GEN_FILES += access_memory
>  
>  TEST_FILES = _chk_dependency.sh _debugfs_common.sh
>  TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh
> @@ -11,6 +12,7 @@ TEST_PROGS += debugfs_empty_targets.sh debugfs_huge_count_read_write.sh
>  TEST_PROGS += debugfs_duplicate_context_creation.sh
>  TEST_PROGS += debugfs_rm_non_contexts.sh
>  TEST_PROGS += sysfs.sh sysfs_update_removed_scheme_dir.sh
> +TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py
>  TEST_PROGS += reclaim.sh lru_sort.sh
>  TEST_PROGS += dbgfs_target_ids_read_before_terminate_race.sh
>  TEST_PROGS += dbgfs_target_ids_pid_leak.sh
> diff --git a/tools/testing/selftests/damon/access_memory.c b/tools/testing/selftests/damon/access_memory.c
> new file mode 100644
> index 000000000000..585a2fa54329
> --- /dev/null
> +++ b/tools/testing/selftests/damon/access_memory.c
> @@ -0,0 +1,41 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Artificial memory access program for testing DAMON.
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <time.h>
> +
> +int main(int argc, char *argv[])
> +{
> +	char **regions;
> +	clock_t start_clock;
> +	int nr_regions;
> +	int sz_region;
> +	int access_time_ms;
> +	int i;
> +
> +	if (argc != 4) {
> +		printf("Usage: %s <number> <size (bytes)> <time (ms)>\n",
> +				argv[0]);
> +		return -1;
> +	}
> +
> +	nr_regions = atoi(argv[1]);
> +	sz_region = atoi(argv[2]);
> +	access_time_ms = atoi(argv[3]);
> +
> +	regions = malloc(sizeof(*regions) * nr_regions);
> +	for (i = 0; i < nr_regions; i++)
> +		regions[i] = malloc(sz_region);
> +
> +	for (i = 0; i < nr_regions; i++) {
> +		start_clock = clock();
> +		while ((clock() - start_clock) * 1000 / CLOCKS_PER_SEC <
> +				access_time_ms)
> +			memset(regions[i], i, 1024 * 1024 * 10);
> +	}
> +	return 0;
> +}
> diff --git a/tools/testing/selftests/damon/sysfs_update_schemes_tried_regions_wss_estimation.py b/tools/testing/selftests/damon/sysfs_update_schemes_tried_regions_wss_estimation.py
> new file mode 100755
> index 000000000000..cdbf19b442c9
> --- /dev/null
> +++ b/tools/testing/selftests/damon/sysfs_update_schemes_tried_regions_wss_estimation.py
> @@ -0,0 +1,55 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import subprocess
> +import time
> +
> +import _damon_sysfs
> +
> +def main():
> +    # access two 10 MiB memory regions, 2 second per each
> +    sz_region = 10 * 1024 * 1024
> +    proc = subprocess.Popen(['./access_memory', '2', '%d' % sz_region, '2000'])
> +    kdamonds = _damon_sysfs.Kdamonds([_damon_sysfs.Kdamond(
> +            contexts=[_damon_sysfs.DamonCtx(
> +                ops='vaddr',
> +                targets=[_damon_sysfs.DamonTarget(pid=proc.pid)],
> +                schemes=[_damon_sysfs.Damos(
> +                    access_pattern=_damon_sysfs.DamosAccessPattern(
> +                        # >= 25% access rate, >= 200ms age
> +                        nr_accesses=[5, 20], age=[2, 2**64 - 1]))] # schemes
> +                )] # contexts
> +            )]) # kdamonds
> +
> +    err = kdamonds.start()
> +    if err != None:
> +        print('kdmaond start failed: %s' % err)
> +        exit(1)
> +
> +    wss_collected = []
> +    while proc.poll() == None:
> +        time.sleep(0.1)
> +        err = kdamonds.kdamonds[0].update_schemes_tried_bytes()
> +        if err != None:
> +            print('tried bytes update failed: %s' % err)
> +            exit(1)
> +
> +        wss_collected.append(
> +                kdamonds.kdamonds[0].contexts[0].schemes[0].tried_bytes)
> +
> +    wss_collected.sort()
> +    acceptable_error_rate = 0.2
> +    for percentile in [50, 75]:
> +        sample = wss_collected[int(len(wss_collected) * percentile / 100)]
> +        error_rate = abs(sample - sz_region) / sz_region
> +        print('%d-th percentile (%d) error %f' %
> +                (percentile, sample, error_rate))
> +        if error_rate > acceptable_error_rate:
> +            print('the error rate is not acceptable (> %f)' %
> +                    acceptable_error_rate)
> +            print('samples are as below')
> +            print('\n'.join(['%d' % wss for wss in wss_collected]))
> +            exit(1)
> +
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.34.1