From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5476C35274 for ; Mon, 18 Dec 2023 11:32:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 614E18D0009; Mon, 18 Dec 2023 06:32:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C2A68D0001; Mon, 18 Dec 2023 06:32:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 465518D0009; Mon, 18 Dec 2023 06:32:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 328BE8D0001 for ; Mon, 18 Dec 2023 06:32:56 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D0B47A111D for ; Mon, 18 Dec 2023 11:32:55 +0000 (UTC) X-FDA: 81579727110.08.15D136F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id CE5DC120009 for ; Mon, 18 Dec 2023 11:32:53 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702899174; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OxscQ+ygqaEV587y3juakSV/n9Wuz++rt6TkmzcTrJc=; b=rhqlURsPW8gvThh+QQEHrX+XFlHNkNBVXSxcX45FiMRRW2sqbyQXbv5zyd2H3ky2qrJ1cW ZomGM0RW39m/nr5LQYwoAzsImcC9f/nUZdEc+xbSNbhOkU7I20qlgK+9WtTmsQ1VSR7n07 tVdR1COPo6Xxs598VN1g/IbXgeKh5nE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702899174; a=rsa-sha256; cv=none; b=nviV+iiq6XFXZxGFtJIzrQTG3VVyWCp8DWoKMVIpxmGOtAc6fhWYrIozxDj9lix6Gb5YIe 3eBlDe7Yyk8LQzGPSR09t06RPMeYl1U2BfQwG6VCG2HxbQpKyurcH6vsjzh4ycrClnBHfj Qb026ninBx9B3R1hz6CaIva5THRZpos= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FF771FB; Mon, 18 Dec 2023 03:33:37 -0800 (PST) Received: from [10.57.75.230] (unknown [10.57.75.230]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AA4693F738; Mon, 18 Dec 2023 03:32:51 -0800 (PST) Message-ID: Date: Mon, 18 Dec 2023 11:32:49 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format Content-Language: en-GB To: John Hubbard , Mark Brown Cc: Andrew Morton , Shuah Khan , Peter Xu , Aishwarya TCV , linux-mm@kvack.org, linux-kselftest@vger.kernel.org References: <20231214162434.3580009-1-ryan.roberts@arm.com> <71228821-cbd3-4a3c-9ed5-18f6d5ebcfc0@arm.com> <07193932-941d-46f6-b152-d6c5fe09b26b@sirena.org.uk> <76abe3b9-3f66-4336-b09d-d5c137ff6582@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: CE5DC120009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: jaxdnkukckckyzubg6ytc4sftifi1a1r X-HE-Tag: 1702899173-604034 X-HE-Meta: U2FsdGVkX1+nSfvtM+cf8go5kZfLbZsJrOY0QzpjnvXbnjqSQ1tVdFgOjWkVjz7GGVhraS08AX6yWhW3aun2rfmQNPVuCeH9PFkWVKNgzsTMGj7MX/W/TOep2aYnpgSDaT2W9S+PS8fbpiqNW8+QqJlJ7rIE9d1rKQfv10VKPqha2/N+dxYHS/iqrjjVwt5SOqA6CnGQ/j/5wBCzRpWS15YvuXyN9EUWiYH1vwz8OiFnSDQnxIybAfDie60iP7xo/7igcPCx0U9iKKJ38rpVV6qjklX3tglSDR/3yXkwFYsoq+kGHkelSULKMpJYc0/mG/zpMvQgXVVg6N/D8fSdBWOXXobEf98WFBJi1d7sSQU7gkY2PYSoDsDlJZ7m+hKSxn3m8vY7FSIMWPl4Na5wsc63iAas45TOjnspOFBHFbGFk/0Mg9A0E4qFFIhkZcfoXRSc+hoqJDAlKV+JjxDKNmFH15gP9XtU1w2oIXwPNm/3vkVFSLKCYPvApvJaLmVzz0aJXYWHMOsb1SInmD89ICz39gxtfewkGJoGTt7m8Rsa2pLxwBa9s1nI2/pu3kdOJvf8Nfwcswq6rpwdnUSz7XecFNfHaxOLG6wXy7R3FA0+kbgxerOUBnSLKzUbSzJ12QZQdvULtC7ShYvTQ4sRgbnH7zo7BlEh3Uli36HTJ4qWn8kB2Zsa2d8Hvo0KFy/g65DcQv79DIO0snWBP5AwUm8qu8PdQRtEEYANN8d6O2yiv1BRxeNTPl0tm7JzzroIY7pmRr68SSsPv2ItDZjrtcvU5de0Pnlg7E4pYNgLKesm9p5JZP7wiLPW6CTCLwkR9d60xxPAz/WPfYg/YHgmSrKF/BAHEN2D9Hr02pwmIr5D9cVqM6uzFp07cO9gphqZl0+9PQUGc1E/xJhkJZSNCowf3J6IJmz7snIhgOeUGCPQeLx8rengktgyg6dto+vH5OvGCPvDmcRNJuN5MVG Kw0rrGBl 15JSZBZdp94BhO8qQ6v7leJJYg0t/VCj+glk1UiHScNSolXiqwZ6khUTioq5BMJszrvQIvIPrnaJ5XYLadMtegpUnO/UAlvTXftlOnoszMI+yoytnndwq1H3UWQfqH67SA6/M7Tzs9Z0y0/p8ysapD+gEF6gJkQYea+BCrym3RPO0w88A5CNxhsbtyBRIwifIvTNVtZlGxOT1A1LnIil3pzav+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 16/12/2023 02:40, John Hubbard wrote: > On 12/15/23 18:25, John Hubbard wrote: >> On 12/15/23 06:28, Ryan Roberts wrote: >> ... >>> I've kept all the existing "pretty" output and results summary as is, it just >>> gets a hash in front of it when TAP is enabled. >>> >>> so this: >>> >>> ----------------------- >>> running ./hugepage-mmap >>> ----------------------- >>> Returned address is 0xffff89e00000 >>> First hex is 0 >>> First hex is 3020100 >>> [PASS] >>> SUMMARY: PASS=1 SKIP=0 FAIL=0 >>> >>> becomes this: >>> >>> TAP version 13 >>> # ----------------------- >>> # running ./hugepage-mmap >>> # ----------------------- >>> # Returned address is 0xffff89e00000 >>> # First hex is 0 >>> # First hex is 3020100 >>> # [PASS] >>> ok 1 hugepage-mmap >>> # SUMMARY: PASS=1 SKIP=0 FAIL=0 >>> 1..1 >>> >>> If you think the latter is ofensive, then I can do the wrapping as you suggest. >> >> I applied this and ran the tests, all while carefully reminding myself >> to "think like a human". :) And from that perspective, to me, the output >> is effectively the same: the leading '#' characters do not really change >> anything, from a readability point of view. >> >> So IMHO you're on perfectly solid ground, if you just switch over >> directly to this format. Great thanks for taking a look! >> >> Tested-by: John Hubbard >> > > I should also point out that some of the subtests already attempt a TAP > output. So now we end up with TAP-within-TAP output for those programs. It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :) > > For example: >     # ----------------------- >     # running ./madv_populate >     # ----------------------- >     # TAP version 13 >     # 1..21 >     # # [RUN] test_prot_read >     # ok 1 MADV_POPULATE_READ with PROT_READ >     # ok 2 MADV_POPULATE_WRITE with PROT_READ >     # # [RUN] test_prot_write >     # ok 3 MADV_POPULATE_READ with PROT_WRITE >     ...etc... > > Note the double level of leading '#' characters. > > Again, this is still readable enough for humans. But it should probably > be removed in subsequent patches to the subtests. I personally don't agree with this. It would be difficult to flatten to a single TAP instance because the top level doesn't have a clue how many test cases the child is running. Trying to do this will make things more fragile and less modular. LAVA can certainly deal with nested test cases and correctly parses everything to test case names that contain the test name at each level of nesting. The thing I was trying to solve with this patch was that previously the top level (run_kselftest.sh) and the bottom level (individual mm test binaries) were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was confusing the LAVA parser. > > > thanks,