From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABD17C64ED6 for ; Tue, 28 Feb 2023 09:37:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C2516B0071; Tue, 28 Feb 2023 04:37:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34A8B6B0072; Tue, 28 Feb 2023 04:37:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C62A6B0073; Tue, 28 Feb 2023 04:37:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0536B6B0071 for ; Tue, 28 Feb 2023 04:37:58 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D97F9ABD5C for ; Tue, 28 Feb 2023 09:37:57 +0000 (UTC) X-FDA: 80516198994.26.5CD48F6 Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by imf26.hostedemail.com (Postfix) with ESMTP id D3CC014001D for ; Tue, 28 Feb 2023 09:37:55 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JxTwuIbc; spf=pass (imf26.hostedemail.com: domain of dvyukov@google.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=dvyukov@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677577076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pLx88XxeINaxlh9c88no1ufEASvBrkn7OcCaxFAtYeg=; b=lWar35l/g/y4DRrKYwvZo2L5oLVa36KWcPBgXZRPq0F6qzzVlAESUhyp6kzG39u5SdRURe CqcmTV/G8Nmbyjp3Ksm6VaqKdMF1WI4FdVGLkUV2JvOjPwCtRAYw6qHVkveuss6q69zpQz dPzxHgIwPW9gFp/vfWpe/EzXryz4jY4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JxTwuIbc; spf=pass (imf26.hostedemail.com: domain of dvyukov@google.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=dvyukov@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677577076; a=rsa-sha256; cv=none; b=4eyrfExxrbx/tHy6eOQ2WM09YK8M1PngaVmERajRej+dTCqrXiRW6qU1X8LS6x2Qnh6acj 4XIsC6farHdgHeKUxo8EgdN03YDSsiyjvG5O9XjlOkOuM9/nklY+rMeKj/PlcgcMWFg+Hf TlGH1t1YuwGUPSw9764Z/djTI1s66Ec= Received: by mail-lj1-f169.google.com with SMTP id b10so9554207ljr.0 for ; Tue, 28 Feb 2023 01:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1677577074; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pLx88XxeINaxlh9c88no1ufEASvBrkn7OcCaxFAtYeg=; b=JxTwuIbcVbdZ0XB/Ke6thXX+EuLbCMglRHbuTsDcHuG1WcxKzsWuKh9IzH+XQrjVfT eA24c9QoJvrTMcNgSstG5r6ULEbW/tS1vJ6HgO//wh8IWWUoWqSgsluInD0IqX9pHy9M t6RapCbSZ8CEfTGAVRtBewJSQITH2IKxarMs7I3yPvOnQvrJgrdOPfhzSvF6h6/LHYpJ MVhN6erdapv9BGhw1tzBMGn5ESETGuU7f29FhzFtExEbCsCwIcsa8p98zUg4WYLA39g5 ZGOoVDtOQZOfFtk0xFbaeSiUT4GFozwbyeH/6z2GYiRVxeEHzLej2H27CLZOrJtAbcOC kzig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677577074; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pLx88XxeINaxlh9c88no1ufEASvBrkn7OcCaxFAtYeg=; b=58KydKpoONE2H+5MT/Pms4cbq+IZ1vrJhFG8D5Lg25pN2JoPsOpdvLE1eJF7jtDXXl yJf5YPI4Ak03KGHXHUijdemLLP2pF4YzrKo1amelEcYyIwauHmoPYFs4/AjJGuibuksy CoUUfEG9uj4j+gPhsf9M3id+ozAkrTf5qk9Ff0DXx5h01gGk2jiuL0NIpqNkSNvcTcPg uolWlUdDkpGWvPEG+YYIhxq0TJNr1Qsg5G8IZWL7WEamvaoIPDdaWsls4N5ShEPmfhDP 7dTcXOgrHuobYz9JS0x0FXiylSeVBxB85iSJG4Qyk+Jt7+/23oFpgnHxhETYzBxDFL0Q mZqA== X-Gm-Message-State: AO0yUKVmLzsLfj9u6l15MYe7ykyu//3Tk5ZqW5ffcCrGbPIgYYoJaydE hSJUX8INEqoW57MplhgwGWPpa4LL9HDHn5dSjx+/Ag== X-Google-Smtp-Source: AK7set812PqxuX3MxJLxRLKJAHUohcBUnCC1CLWhxpTualMPtkhO/vvhti24y2V4Lpve70nRZvqahc7LvQktEGV+1VA= X-Received: by 2002:a05:651c:1242:b0:295:b83b:ab11 with SMTP id h2-20020a05651c124200b00295b83bab11mr576748ljh.4.1677577073761; Tue, 28 Feb 2023 01:37:53 -0800 (PST) MIME-Version: 1.0 References: <0b5efd70e31bba7912cf9a6c951f0e76a8df27df.1677517724.git.andreyknvl@google.com> In-Reply-To: <0b5efd70e31bba7912cf9a6c951f0e76a8df27df.1677517724.git.andreyknvl@google.com> From: Dmitry Vyukov Date: Tue, 28 Feb 2023 10:37:40 +0100 Message-ID: Subject: Re: [PATCH] kcov: improve documentation To: andrey.konovalov@linux.dev Cc: Andrey Konovalov , Marco Elver , Alexander Potapenko , kasan-dev@googlegroups.com, Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D3CC014001D X-Rspam-User: X-Stat-Signature: wymonshibuwt8c3ec3gt8etncozghfjt X-HE-Tag: 1677577075-259667 X-HE-Meta: U2FsdGVkX19OzIgyCTa6xuYEST2JIxg5a6mTZQaDGwWbWld0rXnmG3h24pYRJioFJtZj4ORnnPdxLnBB4qoxqO7pgZ/gP3hXKuXfSdmS7Cw4tEI7j6mI4qeWS4fe/mMK28fIbOW4HI9jIQYvO3+32RJIMqRR5Fcu4HC8Wo5RBhcg8sBI+kjeAh7s+G6PoPM1oLvHz5yqAnuCF/Dqseb6FG3XBE7DcwIR/+leMG6VXOnMyqKmwHFTvok9ZUVj55FMS5GWoX40faq83APnlzoW72uhJTmI2EtjqtZEej95FNaJYB/znXt2WgUnDhYF32Y/+CLgzjWomHWoRUDJ5kBy7Ush+1KDxq3CNNzyAnqZ6wFVNlOtqPqxNacknFZVxwZLJ4AePiP/CDfvW6CnH5xdZNtUiQO5Ld7p7hrndutDVJjpuoIA6eXorMUT4Chol0Zak9+NXbghSHc5dvRmxEFadJPqpZ+VEaXSTA8FHlCoEtNL4QrWWrX8lpafD3dP7q/RH9T3tTEgN9I7on7Se1mW1B/mzUp551MxSqhH1CLvLs+CJSSWxBcdLDv0Y7R9AS+B7HBhHjHgS5Kn40y5aOxcBCq5nbZ6CJu+mqYAslnPJ3FPxYvsI7vwxokQhP7ECbFTaagTAc87N/nob9nBDPP5S5eHz42hFtYlWj7tZQD9U5xqa5d3BqHDmdkSZh9TxUX+kFwRtBMNFyFuPKX8RwHXlxk0+NAHmeDwX1HHxwCDVcxoHk6DKCkVHjJ+LMrcR5UtrwRXmBGtSCu5okgKaP9j5cQHwDMDGLps/Vrmuo5+rJz1dooVHEfQQohwud0ys10168lMNGjZ0nbU0a8MNTJEXiH56uHKLzHUhRVdcnauYwqX7aorFVa9Z155FXwPzjn1Pv1oYVRPLa3IHiPQv3r/6YePSewI4nQNnvOh5rFXwrEpcKPaCeJ55yJA/8KTq3bwewc5CF8n/q35zPNH1pQ UyysjrSW iWYqYEfM8Dw1Nfrce6EHv5ddydJxfKzrSue9WYFMBsf2+1+g7KWzmbw99Y62cMOuHusr6tc6aLkCrreGpTG+oRjsA0yd+dShcouNvcZ3FlK1pmA78gxfCyOOe/jkRkvpIDZFG1LsZJM+4+i/BaWXJZSnNQh7K1lSINHO9ESViKbA10EeukaM+ZgdOuCPXpDGq49YJ+xq9TWUlTLJyBDyPssIgjUGgdmmjDVow/BNespETdEpX0MWCYai9ZQgShkmBaVQ7fUuVgzVU30I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 27 Feb 2023 at 18:17, wrote: > > From: Andrey Konovalov > > Improve KCOV documentation: > > - Use KCOV instead of kcov, as the former is more widely-used. > > - Mention Clang in compiler requirements. > > - Use ``annotations`` for inline code. > > - Rework remote coverage collection documentation for better clarity. > > - Various smaller changes. > > Signed-off-by: Andrey Konovalov > --- > Documentation/dev-tools/kcov.rst | 169 +++++++++++++++++++------------ > 1 file changed, 102 insertions(+), 67 deletions(-) > > diff --git a/Documentation/dev-tools/kcov.rst b/Documentation/dev-tools/kcov.rst > index d83c9ab49427..a113a03a475f 100644 > --- a/Documentation/dev-tools/kcov.rst > +++ b/Documentation/dev-tools/kcov.rst > @@ -1,42 +1,50 @@ > -kcov: code coverage for fuzzing > +KCOV: code coverage for fuzzing > =============================== > > -kcov exposes kernel code coverage information in a form suitable for coverage- > -guided fuzzing (randomized testing). Coverage data of a running kernel is > -exported via the "kcov" debugfs file. Coverage collection is enabled on a task > -basis, and thus it can capture precise coverage of a single system call. > +KCOV collects and exposes kernel code coverage information in a form suitable > +for coverage-guided fuzzing. Coverage data of a running kernel is exported via > +the ``kcov`` debugfs file. Coverage collection is enabled on a task basis, and > +thus KCOV can capture precise coverage of a single system call. > > -Note that kcov does not aim to collect as much coverage as possible. It aims > -to collect more or less stable coverage that is function of syscall inputs. > -To achieve this goal it does not collect coverage in soft/hard interrupts > -and instrumentation of some inherently non-deterministic parts of kernel is > -disabled (e.g. scheduler, locking). > +Note that KCOV does not aim to collect as much coverage as possible. It aims > +to collect more or less stable coverage that is a function of syscall inputs. > +To achieve this goal, it does not collect coverage in soft/hard interrupts > +(unless remove coverage collection is enabled, see below) and from some > +inherently non-deterministic parts of the kernel (e.g. scheduler, locking). > > -kcov is also able to collect comparison operands from the instrumented code > -(this feature currently requires that the kernel is compiled with clang). > +Besides collecting code coverage, KCOV can also collect comparison operands. > +See the "Comparison operands collection" section for details. > + > +Besides collecting coverage data from syscall handlers, KCOV can also collect > +coverage for annotated parts of the kernel executing in background kernel > +tasks or soft interrupts. See the "Remote coverage collection" section for > +details. > > Prerequisites > ------------- > > -Configure the kernel with:: > +KCOV relies on compiler instrumentation and requires GCC 6.1.0 or later > +or any Clang version supported by the kernel. > > - CONFIG_KCOV=y > +Collecting comparison operands is only supported with Clang. Are you sure? I see -fsanitize-coverage=trace-cmp in gcc sources and man page. Otherwise looks good to me. > -CONFIG_KCOV requires gcc 6.1.0 or later. > +To enable KCOV, configure the kernel with:: > > -If the comparison operands need to be collected, set:: > + CONFIG_KCOV=y > + > +To enable comparison operands collection, set:: > > CONFIG_KCOV_ENABLE_COMPARISONS=y > > -Profiling data will only become accessible once debugfs has been mounted:: > +Coverage data only becomes accessible once debugfs has been mounted:: > > mount -t debugfs none /sys/kernel/debug > > Coverage collection > ------------------- > > -The following program demonstrates coverage collection from within a test > -program using kcov: > +The following program demonstrates how to use KCOV to collect coverage for a > +single syscall from within a test program: > > .. code-block:: c > > @@ -84,7 +92,7 @@ program using kcov: > perror("ioctl"), exit(1); > /* Reset coverage from the tail of the ioctl() call. */ > __atomic_store_n(&cover[0], 0, __ATOMIC_RELAXED); > - /* That's the target syscal call. */ > + /* Call the target syscall call. */ > read(-1, NULL, 0); > /* Read number of PCs collected. */ > n = __atomic_load_n(&cover[0], __ATOMIC_RELAXED); > @@ -103,7 +111,7 @@ program using kcov: > return 0; > } > > -After piping through addr2line output of the program looks as follows:: > +After piping through ``addr2line`` the output of the program looks as follows:: > > SyS_read > fs/read_write.c:562 > @@ -121,12 +129,13 @@ After piping through addr2line output of the program looks as follows:: > fs/read_write.c:562 > > If a program needs to collect coverage from several threads (independently), > -it needs to open /sys/kernel/debug/kcov in each thread separately. > +it needs to open ``/sys/kernel/debug/kcov`` in each thread separately. > > The interface is fine-grained to allow efficient forking of test processes. > -That is, a parent process opens /sys/kernel/debug/kcov, enables trace mode, > -mmaps coverage buffer and then forks child processes in a loop. Child processes > -only need to enable coverage (disable happens automatically on thread end). > +That is, a parent process opens ``/sys/kernel/debug/kcov``, enables trace mode, > +mmaps coverage buffer, and then forks child processes in a loop. The child > +processes only need to enable coverage (it gets disabled automatically when > +a thread exits). > > Comparison operands collection > ------------------------------ > @@ -205,52 +214,78 @@ Comparison operands collection is similar to coverage collection: > return 0; > } > > -Note that the kcov modes (coverage collection or comparison operands) are > -mutually exclusive. > +Note that the KCOV modes (collection of code coverage or comparison operands) > +are mutually exclusive. > > Remote coverage collection > -------------------------- > > -With KCOV_ENABLE coverage is collected only for syscalls that are issued > -from the current process. With KCOV_REMOTE_ENABLE it's possible to collect > -coverage for arbitrary parts of the kernel code, provided that those parts > -are annotated with kcov_remote_start()/kcov_remote_stop(). > - > -This allows to collect coverage from two types of kernel background > -threads: the global ones, that are spawned during kernel boot in a limited > -number of instances (e.g. one USB hub_event() worker thread is spawned per > -USB HCD); and the local ones, that are spawned when a user interacts with > -some kernel interface (e.g. vhost workers); as well as from soft > -interrupts. > - > -To enable collecting coverage from a global background thread or from a > -softirq, a unique global handle must be assigned and passed to the > -corresponding kcov_remote_start() call. Then a userspace process can pass > -a list of such handles to the KCOV_REMOTE_ENABLE ioctl in the handles > -array field of the kcov_remote_arg struct. This will attach the used kcov > -device to the code sections, that are referenced by those handles. > - > -Since there might be many local background threads spawned from different > -userspace processes, we can't use a single global handle per annotation. > -Instead, the userspace process passes a non-zero handle through the > -common_handle field of the kcov_remote_arg struct. This common handle gets > -saved to the kcov_handle field in the current task_struct and needs to be > -passed to the newly spawned threads via custom annotations. Those threads > -should in turn be annotated with kcov_remote_start()/kcov_remote_stop(). > - > -Internally kcov stores handles as u64 integers. The top byte of a handle > -is used to denote the id of a subsystem that this handle belongs to, and > -the lower 4 bytes are used to denote the id of a thread instance within > -that subsystem. A reserved value 0 is used as a subsystem id for common > -handles as they don't belong to a particular subsystem. The bytes 4-7 are > -currently reserved and must be zero. In the future the number of bytes > -used for the subsystem or handle ids might be increased. > - > -When a particular userspace process collects coverage via a common > -handle, kcov will collect coverage for each code section that is annotated > -to use the common handle obtained as kcov_handle from the current > -task_struct. However non common handles allow to collect coverage > -selectively from different subsystems. > +Besides collecting coverage data from handlers of syscalls issued from a > +userspace process, KCOV can also collect coverage for parts of the kernel > +executing in other contexts - so-called "remote" coverage. > + > +Using KCOV to collect remote coverage requires: > + > +1. Modifying kernel code to annotate the code section from where coverage > + should be collected with ``kcov_remote_start`` and ``kcov_remote_stop``. > + > +2. Using `KCOV_REMOTE_ENABLE`` instead of ``KCOV_ENABLE`` in the userspace > + process that collects coverage. > + > +Both ``kcov_remote_start`` and ``kcov_remote_stop`` annotations and the > +``KCOV_REMOTE_ENABLE`` ioctl accept handles that identify particular coverage > +collection sections. The way a handle is used depends on the context where the > +matching code section executes. > + > +KCOV supports collecting remote coverage from the following contexts: > + > +1. Global kernel background tasks. These are the tasks that are spawned during > + kernel boot in a limited number of instances (e.g. one USB ``hub_event`` > + worker is spawned per one USB HCD). > + > +2. Local kernel background tasks. These are spawned when a userspace process > + interacts with some kernel interface and are usually killed when the process > + exits (e.g. vhost workers). > + > +3. Soft interrupts. > + > +For #1 and #3, a unique global handle must be chosen and passed to the > +corresponding ``kcov_remote_start`` call. Then a userspace process must pass > +this handle to ``KCOV_REMOTE_ENABLE`` in the ``handles`` array field of the > +``kcov_remote_arg`` struct. This will attach the used KCOV device to the code > +section referenced by this handle. Multiple global handles identifying > +different code sections can be passed at once. > + > +For #2, the userspace process instead must pass a non-zero handle through the > +``common_handle`` field of the ``kcov_remote_arg`` struct. This common handle > +gets saved to the ``kcov_handle`` field in the current ``task_struct`` and > +needs to be passed to the newly spawned local tasks via custom kernel code > +modifications. Those tasks should in turn use the passed handle in their > +``kcov_remote_start`` and ``kcov_remote_stop`` annotations. > + > +KCOV follows a predefined format for both global and common handles. Each > +handle is a ``u64`` integer. Currently, only the one top and the lower 4 bytes > +are used. Bytes 4-7 are reserved and must be zero. > + > +For global handles, the top byte of the handle denotes the id of a subsystem > +this handle belongs to. For example, KCOV uses ``1`` as the USB subsystem id. > +The lower 4 bytes of a global handle denote the id of a task instance within > +that subsystem. For example, each ``hub_event`` worker uses the USB bus number > +as the task instance id. > + > +For common handles, a reserved value ``0`` is used as a subsystem id, as such > +handles don't belong to a particular subsystem. The lower 4 bytes of a common > +handle identify a collective instance of all local tasks spawned by the > +userspace process that passed a common handle to ``KCOV_REMOTE_ENABLE``. > + > +In practice, any value can be used for common handle instance id if coverage > +is only collected from a single userspace process on the system. However, if > +common handles are used by multiple processes, unique instance ids must be > +used for each process. One option is to use the process id as the common > +handle instance id. > + > +The following program demonstrates using KCOV to collect coverage from both > +local tasks spawned by the process and the global task that handles USB bus #1: > > .. code-block:: c > > -- > 2.25.1 >