From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D2B3C35274 for ; Fri, 15 Dec 2023 13:58:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76F478D012E; Fri, 15 Dec 2023 08:58:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F5208D0121; Fri, 15 Dec 2023 08:58:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E3848D012E; Fri, 15 Dec 2023 08:58:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 416708D0121 for ; Fri, 15 Dec 2023 08:58:52 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 170DE1C1772 for ; Fri, 15 Dec 2023 13:58:52 +0000 (UTC) X-FDA: 81569208504.22.74E41E9 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 115F0120021 for ; Fri, 15 Dec 2023 13:58:49 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702648730; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SrfLqHE68QYTPUPck9t/kihtncnywic4c4Cm0ZJ3440=; b=pLu02bbz05HxddD4/2ZyRCS2TbFrc4uYmFTa4m+j9L2OkNaN0hePi8QZiSMTkrYpg4OH7u qq2nIV6dz9BgBsgg+oLOdqvAuUBv01YnCyyFq2ttdnEEQ2xWirHn7JP5h3DrQvDnCOzT0x cHzzjCBOSYwrpyW6a62zwextxN2wZjg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702648730; a=rsa-sha256; cv=none; b=3dolkRg1gMnfh13bM5xI7UHfVHCwn5njFsvmx5KemzmwJDWHEYhVUVfa9QeMGVR+kKyjex jq4PeGQVm6y5wKbXVm7JR1wUHjdPIV80LgiOwBTt3cUmDAl6s3NXhGrNvcyflzX+v9AZpd U7W669JfM3XcPxdq3syO927dItHHpdM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 352BDC15; Fri, 15 Dec 2023 05:59:34 -0800 (PST) Received: from [10.57.76.37] (unknown [10.57.76.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8DC223F5A1; Fri, 15 Dec 2023 05:58:47 -0800 (PST) Message-ID: <71228821-cbd3-4a3c-9ed5-18f6d5ebcfc0@arm.com> Date: Fri, 15 Dec 2023 13:58:45 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format To: Mark Brown Cc: Andrew Morton , Shuah Khan , Peter Xu , Aishwarya TCV , linux-mm@kvack.org, linux-kselftest@vger.kernel.org References: <20231214162434.3580009-1-ryan.roberts@arm.com> Content-Language: en-GB From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: ukn6tcq8b9pnezqks91rcqdr9d1xa7h5 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 115F0120021 X-Rspam-User: X-HE-Tag: 1702648729-289472 X-HE-Meta: U2FsdGVkX18unq716aMDufTRwrEu5na8hq0LB+4lPVxaC/s7mH0g7WgvfR8FQOsqQGggKXS0XNw/8ujYadnmTzBPSEp0TXzbL85s/ZpFWwAsOwcvJykrBUFLZECH/dfRhAhi1U48aUaj92PBeO3XdO2XatCCB8vTAcdbgjYJ6I89EeAmEd56slocZMJjGqORZHLnA133lloBws0SZgcVJFmwFk93kL4PcE7AY3jjyf3GwHou9pim8JK2nkYShYh//pcKdt1XqWbJ6iKfQq8ER7xBIRYIyJCpdWe8WoWL34QpmAk8VpL2RVofmUpct8iYlu2aAuO0c6QwLE8BBw7h/6nmN1a4nWt5ma+FzOpONjRHvk2Lo7axIsf7G7lS6+Cgt0yd3LBjxjOIRRvZA0rR9RypyinM0NHUmoD/2ckxWvy3xlVGz274l7kIuBnHhMn7w/GysliKuT33MTmYraNQTyzMi8/TEECXQVrBgjjUJV0as4Nww6bjKvaSmax/jowKcrdnTk/Pp0u04BXSrjUNbvDGsAy7ansOivEIdlj+UTJSoqOvDUyvCcH1GYI/2e/qW04CpUqDAUbvcnRBZj3RXa2JCMu7wyTEVgG14MoI0iXNzI+huVHD9N0FwN1JcEyGe5vN7ngLsVzx5VX61eRQ1RcVdofRG7r6zJfQOo44oahmrpj3K9QPcwb5If9Og7N5k+mbBimHLmCqSRklLyzGuZamVFekUBTwpLaQB4k1rjAx6ZgT40183kuWRVvF4Ee4Mv5XWvmrQaqJQQqxkYtoCnTmCwzwdsZicRW2ZCD9/jpAYLrEqTKypJqpYCQ5TqYxFiXriAUnxNOPKznPtO1ckrNtYmD8A15PnR4t9P3ORykTWUSJy+l4tM+wTQYZJ+1Po+znwFJSASXYVolZg3XzmIywO5QycxNolaHbBZAr7Z4KADtJdpPcm7D8CJbSC4qXqQ6SJorZuWblbthT8nW qit/v9oV Zb9vNuHnu661MNyOSBOcc63zIF9ugt6fSKr7izFxpt89OSudAz/jeZP8eiLPF9s73OZ82SKYbhcKXtK0tW/C9s/N0AgU4na29T+zo8V1/Dzm61YZnrGNFCH3orMOPDGYhBe6XIXX5oWmO55KLqSnN9a7QA/7ggbXzFSjwXtaPfnyFAaB0b3XXq4h6ELfOtWhKUM32/Ag8j7wJq6EdzjHsaTtBUXI3sw4buFs6jiQHpSml4mh3Urwkx0CjAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 15/12/2023 13:54, Mark Brown wrote: > On Thu, Dec 14, 2023 at 04:24:34PM +0000, Ryan Roberts wrote: >> When running tests on a CI system (e.g. LAVA) it is useful to output >> test results in TAP format so that the CI can parse the fine-grained >> results to show regressions. Many of the mm selftest binaries already >> output using the TAP format. And the kselftests runner >> (run_kselftest.sh) also uses the format. CI systems such as LAVA can >> already handle nested TAP reports. However, with the mm selftests we >> have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> >> individual test binaries) and the middle level did not previously >> support TAP, which breaks the parser. > > Reviewed-by: Mark Brown Thanks! > >> Let's fix that by teaching run_vmtests.sh to output using the TAP >> format. Ideally this would be opt-in via a command line argument to >> avoid the possibility of breaking anyone's existing scripts that might >> scrape the output. However, it is not possible to pass arguments to >> tests invoked via run_kselftest.sh. So I've implemented an opt-out >> option (-n), which will revert to the existing output format. > > What I did for ftrace which had a similar situation was make a wrapper > script which invokes the test runner, make the test runner a > TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure > automatically and make the wrapper a normal TEST_PROGS. Neither option > is especially lovely. Yeah that's a good idea... I'll wait and see if anyone shouts that this has broken something. If nothing is broken, I think it is better to just make TAP the default rather than adding yet another wrapper.