From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE3C2C41535 for ; Tue, 19 Dec 2023 08:33:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35C976B0089; Tue, 19 Dec 2023 03:33:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 30CAF6B008C; Tue, 19 Dec 2023 03:33:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D4396B0092; Tue, 19 Dec 2023 03:33:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0DBBE6B0089 for ; Tue, 19 Dec 2023 03:33:09 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D27A9A1DC7 for ; Tue, 19 Dec 2023 08:33:08 +0000 (UTC) X-FDA: 81582902856.28.47D0071 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 15F4A100009 for ; Tue, 19 Dec 2023 08:33:06 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702974787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P5nKSDzl5zTecRFuqW7BaBCnXe1rpMbzNDFD5t+jvyg=; b=v6gdOHgWOOyTdM9W9eAtbfnefx5xWhgguWJAJc2uNrQUPoPy7EJl3i5TpMjQkT1aS7z9/C SGGyDaNCfas4PXZV04G/X/IIOrJseZGxe2Uat4c/lSeIEV1d/gNPKPubRITUePjj0Kbh0s UAf2ZnS142hMqcYPzyqG8WxeWNS7FJA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702974787; a=rsa-sha256; cv=none; b=T0t1GpJIlfsJBoGUfFK3PSekL6glkXVlscz0qnt8BpfWHpIgin5/tUey2+QbMqFeKm5QNp giZpReiBtIQwxHUPY0JjwXXTKtLACaDhS3FHATcCjTsLOEcitYYEIYiFxId0kZh7kBdywZ d/18n0A/nh5NNZ18kB8D6vy1T1lWWE8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6F5841FB; Tue, 19 Dec 2023 00:33:50 -0800 (PST) Received: from [10.57.75.230] (unknown [10.57.75.230]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2027E3F738; Tue, 19 Dec 2023 00:33:05 -0800 (PST) Message-ID: Date: Tue, 19 Dec 2023 08:33:03 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format Content-Language: en-GB To: John Hubbard , Mark Brown Cc: Andrew Morton , Shuah Khan , Peter Xu , Aishwarya TCV , linux-mm@kvack.org, linux-kselftest@vger.kernel.org References: <20231214162434.3580009-1-ryan.roberts@arm.com> <71228821-cbd3-4a3c-9ed5-18f6d5ebcfc0@arm.com> <07193932-941d-46f6-b152-d6c5fe09b26b@sirena.org.uk> <76abe3b9-3f66-4336-b09d-d5c137ff6582@arm.com> <7663374f-ed7e-4d86-a07f-e71c6bcef841@nvidia.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: 8k9f9ch41m86rrpqxw54ufpapqok5dy4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 15F4A100009 X-HE-Tag: 1702974786-516246 X-HE-Meta: U2FsdGVkX1+i720+NsoW8Ss9Cmls5aTxptJgVEr795HA4LVsFBoM6xLbgma/w3tq0YrSMr7DixdVBEr9fwEbukaae7KM4ezCA72YguNOQvadF7SoAwKxGjLt1EJwjyeAhb0ChRIvAnLCT/sJa3MvfrmE9jZ+HVUl7c8Tb9dk2qj2CSGfXLFR8bSuG7K2jS+xdgG3j9m0Q2mQ1R9tBS0JB/nTbN2C+U7ad6VKKTl7at71tdcc2nDxp8jIbkuMCdeh1MzjaOq+cyzNY9JTmarrf1NmNiUfcCqWDNQ4MYPZpGWSaaC5rt1xg7yJKLH6aKox1QmanS5oD/y85Elvdu0Nw+wmIFsU2dWbqTp9BSXTNE85iRu1A9fdFToHpc7FojeY58dO4Vpt/86JgNn387t+Pi9tNQw4sEmmq654j0fVi90LCuzNhNlJzgU2+ysl5tSpaZuL7r5I8FZYyA2XOWu55UJzsw5hlYM94T3sbsgupBM8KbTyReNU4aUjf4CCi7bsBL7dXtnGDpLRJPT7ZhBIjNX19fcV2upS+3oFJJ0h9eQaJiZMs4Zmfh9ZlMBgKhm/Mc7G48nGP6QNVo815drFmPUH87hoHOS+EdcrALGOpOPfo0EDYsTXH5Nfw6BTMqAG28fT614Rx/yE6WB6J17NKg/PuuNCHhLDWb0WIv8Y7NRp3Mk/vpAT5coBNEyhYuLqe7unB4hqji3TigOZ+sSBixe5C1DZYnPIIvO49M0YAMv8uCo4hgX1FVF4YDL5/qgQJbN7m27l6sxpPJlKb+Xc4a9FkMa+/HOp9A+rw+Qej44dLYaLmtHZjnUo2dV/NhSRb+nD3LBSDKE7n8qE7MWFEUG4kMZckEvMSxwpKdu6/0hvffrP1R6zd1S5gfJbp8JQ9GSfiOtqEX5g98JrDKAErzOyp20SZWCMDmnLHxZ22D0wxaJGlzWj703j4TR703WivL2I7VPsymLwMMG6R3O pSFcLI9g cGAq/sxDQyZd4Hmkk52jcpX2zir9moS7gRKAwJ7+kD6ZM50hTfo4vdX35VJmwMY44Oa0pHysgOWm8CBGjzS/EvDJQ0EyJjwvqXJQe0IMmV98taJ4zKEMm0KWyx1Eh0P6WO9g3ZtF3lkx3R8LG9w2PmU31he1K+Kk2vDEtF0syAcIvqfTmQzXXAdNa8jbE7VBTOXuP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 19/12/2023 00:55, John Hubbard wrote: > On 12/18/23 16:51, John Hubbard wrote: >> On 12/18/23 03:32, Ryan Roberts wrote: >> ... >>>> I should also point out that some of the subtests already attempt a TAP >>>> output. So now we end up with TAP-within-TAP output for those programs. >>> >>> It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :) >>> >>>> >>>> For example: >>>>      # ----------------------- >>>>      # running ./madv_populate >>>>      # ----------------------- >>>>      # TAP version 13 >>>>      # 1..21 >>>>      # # [RUN] test_prot_read >>>>      # ok 1 MADV_POPULATE_READ with PROT_READ >>>>      # ok 2 MADV_POPULATE_WRITE with PROT_READ >>>>      # # [RUN] test_prot_write >>>>      # ok 3 MADV_POPULATE_READ with PROT_WRITE >>>>      ...etc... >>>> >>>> Note the double level of leading '#' characters. >>>> >>>> Again, this is still readable enough for humans. But it should probably >>>> be removed in subsequent patches to the subtests. >>> >>> I personally don't agree with this. It would be difficult to flatten to a single >>> TAP instance because the top level doesn't have a clue how many test cases the >> >> That's not quite what I had in mind... >> >>> child is running. Trying to do this will make things more fragile and less >>> modular. LAVA can certainly deal with nested test cases and correctly parses >>> everything to test case names that contain the test name at each level of >>> nesting. The thing I was trying to solve with this patch was that previously the >>> top level (run_kselftest.sh) and the bottom level (individual mm test binaries) >>> were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was >>> confusing the LAVA parser. >>> >> >> I was thinking more along these lines: >> >> a) For the individual programs (binaries), there is actually neither need nor >> desire to create TAP output at that level, because frameworks like LAVA only >> care about running a lot of tests and parsing the output. >> >> b) Therefore, just stop specifying TAP output at the leaf level, and let >> run_vmtests.sh and run_kselftest.sh do it. >> >> Looking at madv_populate.c, I see that it scatters calls to ksft_*() around. >> And I was thinking that this is all just redundant, isn't it? >> > > Although I suppose that the counter argument is that the subtests in > madv_populate.c really *do* want to be specifically printed in TAP > format. > > arggh, I guess this is just not worth fooling around with after all. Yes; I wouldn't want to lose the fine granularity we have currently. For example cow.c has ~900 test cases now that I've multiplied everything up for mTHP. 16 of those are known to fail (hugetlb issue) and 1 is skipped. I wouldn't want to reduce that down to a single cow test case that always fails; that's not helpful to understand if I've regressed something. But sounds like we are both on the same page now. >   > > thanks,