From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F462C3ABD8 for ; Fri, 16 May 2025 12:30:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AB376B0153; Fri, 16 May 2025 08:30:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80A7A6B0154; Fri, 16 May 2025 08:30:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FCAF6B0155; Fri, 16 May 2025 08:30:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1DE656B0153 for ; Fri, 16 May 2025 08:30:01 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 85D6912032F for ; Fri, 16 May 2025 12:30:02 +0000 (UTC) X-FDA: 83448703044.02.6C24AC9 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf16.hostedemail.com (Postfix) with ESMTP id C72E3180008 for ; Fri, 16 May 2025 12:30:00 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="TQS/7nvA"; spf=pass (imf16.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747398600; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w4KxbqLUYNlaeliDLSzW7eh9S4Q0txRrXMK3MdkuDUM=; b=KNALLu2YlZhulfHpy0mwjR/g9104GvHVBsP64BXYY2hux59UHtXONWLxQ1vwn4QOjuzOjL UOLIC2TnfeZRghm4AfMrntSfoQzQEF8buYoW0OvombQUBe2qdmD/NuFeHAKS2LrZUDP9DK qACX7mizypamgvlkhdlI3rb9Y7rUd+Y= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="TQS/7nvA"; spf=pass (imf16.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747398600; a=rsa-sha256; cv=none; b=xPs9h3c4n1KB29T2Fkmk4OvYNQgyo5C7ggu6RFFPjboZPltkmOiaARZ10NwGPlaCyWSRgw Jr+EfJ+KbPC1hv6K2GCB7+GOnL3HqDHa+1q5FHzA3vVil4pzf0kv0B9iNLZma5/vSXyA6D m27NGa6hAPsJzl4fWrOL8PsNE248SjM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 1EB44A4E77D; Fri, 16 May 2025 12:30:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77DF2C4CEE4; Fri, 16 May 2025 12:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747398599; bh=v60gkmBBs9thffCH+KRcGOvOSb27plbXUWEj4N9vc9Q=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=TQS/7nvAYZH5uI9JzECirMI3mQVI54ZX42fKXb7ca0ktFWDVPErTwR2nCXq1dGXs5 oPkLLcVa7crcsKSYuY7nuLx9KrNbQS8eNCqxUIZwkvd9l2Qfv9tnxDQ4BnUpSZmjJ3 WdLzM/EISz3OXP6zRfhUjUlnLPpimkmoEaJaboNEAARasn2H4P8JgqZ9WXbNBqmWrC PyuOgkjq8WMFO10qUDYicyK4yVtQbKKQxgoyK6HsexlHqewLb2uWkmEYPaJNOAXL/4 nIhoRBJtoTHh6axJ1K6gGGibFJ4xb4W4S7vNqvBMXP3rQ9SFLsdkZlpxxL4yg11cYr XpN3fFkTf6+QQ== Date: Fri, 16 May 2025 14:29:57 +0200 From: Mark Brown To: David Hildenbrand Cc: Andrew Morton , Shuah Khan , linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] selftests/mm: Fix test result reporting in gup_longterm Message-ID: References: <20250515-selftests-mm-gup-longterm-dups-v1-1-05f8f731cf63@kernel.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="+lLg9Zve3xwFMbDn" Content-Disposition: inline In-Reply-To: X-Cookie: Well begun is half done. X-Rspam-User: X-Rspamd-Queue-Id: C72E3180008 X-Rspamd-Server: rspam09 X-Stat-Signature: ht8q1m39dchmxw95majqkuefdf86cdzt X-HE-Tag: 1747398600-329962 X-HE-Meta: U2FsdGVkX1/8gOsVMilp695y/u+6lQ+nc45zW0GpkdxCBtRmjZQX/QPc6kbA97DwLihkRG11NBheAprVxg+IJ4x5wMzPeZcztB+8HYDlrm2kwH4naDr78DEDg+HqrtiriOEh/pV/23obYKeeghqC+FbSqbrMU06ma2il+U1a2HTODkpVYGODqdHqCDKZVdMFFqWeXOpma5JjBJlwCZD/zqjPsg0/zixHQN0m8fgHUpwDxBBpdI1VHt63bkcVrewTGH8ZG/YEt/8Y8D/ZXEa4iARL1n9oq4Vtk1eGe93HSvyqBHl8PMvVRz9d9hoq6ZtF/06KkDpIm3KRcLxdCDvxNtT/cR7TYfvXHMAwG1byNuWKCnGoLRG7bKV02c48jHm03+J9nV6YkJXgaEYCI4OER1NIpyrHoCcnrV9EzsZwwmJk1r/M3PJGgSDlOH4ZADvmFl8MTEhtKnUsUpKlcWBeCzHm+M3ZkTfmC+5tL91Tpw9VRoRLIiqf3SzSnO+bI+9lQWNzKWGX05oSGs2GOkNz5ikN5xu5ebCtQNtHYodqgh7QQicbLoLKxL8yL+n+Z76xqIhVdstrGG4rOBJWOG05KoWi5OOILnoF785hNhgs3+hb6PVAzXGrXMIq08ObB1mSBksql3UKYMwNp4Io3bdcjUo31hK1JzrWZNtg/ac5I+IV9+XcyOL3bMCQMAtWPFgQ9a3n2tSqm4Ig4IQU/uKC/u0bIi8i2I/zH+iLz1Iaj8dsgv++ATynWrcvet20WQZpqr2oHpNqfZxZOEvC+JUHc9K9G21ykt2HGudoHMkHW9jfTU8JTX5XAtBS1LwNWfraOHo3WQMrX6/ClAJVclO6CCR0NZeB8R4wwQTLFcAKnkNGekMSrdunb8IjcYZJX5PoXeJTAYmCXuo7gjh07efORcJLtOohPT7CrImj0MO4hLEoV+MVaHhyjGK8gdouwYJRFvc0fYvfYflkDON9vms MAJqXLVt xgGLo8UOLlb7v5VGea4UDLszt5bhBRv2BhwvtdKnfEV/oAmzOBAgE+aCXS33cCwUTZgEA8xL1Ygs4mT9NbndqAzA81vK2iwDBWSKZ1Np8a/mcyYsdTUbRAEJpUjH44JVBGSvO9blpyMuxPCYSKai1QR4TDeYeqj6/32huuoakN7tO1os1rJUwECSLWomFAOHUuCcz7hrIM8sVh9ChgFvIQyLncDFne5Ig8NnFHfbdSmDepKOh1SjkQWJ7S6V5pknY82kmICTIUAzVVeEhmwnAEnGwqoEYjeTIAWYdJRvVJkixV8juoUhxFgu/Xka634iP1IIgh/8avbvDAavntmpayDjBLPZPTTOvC7QN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --+lLg9Zve3xwFMbDn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, May 16, 2025 at 10:02:16AM +0200, David Hildenbrand wrote: > On 15.05.25 10:57, Mark Brown wrote: > > The kselftest framework uses the string logged when a test result is > > reported as the unique identifier for a test, using it to track test > > results between runs. The gup_longterm test completely fails to follow > > this pattern, it runs a single test function repeatedly with various > > parameters but each result report is a string logging an error message > > which is fixed between runs. > As the person who wrote that test (that you apparently didn't CC for some I just CCed whoever get_maintainers told me to CC for the patch. > reason), what exactly is the problem with that? > We run tests. If all pass, we're happy, if one fails, we investigate. None of the tooling is able to either distinguish between the multiple tests that are being run in gup_longterm, nor compare the results of multiple runs effectively. If all the tests run they report themselves as being duplicates of the same test name, if one of them starts failing the effect is that one of the duplicates disappears and we get an entirely new test that's never passed reported. If multiple tests fail it's even worse. This means that UIs displaying test results end up reporting things unclearly (Was there a regression or was a new test that never worked added? What was the test?). Since it's difficult to track the tests between runs tooling that does reporting of things like "This last worked in X" in the UI doesn't work properly, and tool driven bisection of issues similarly struggles since it can't tell what's going on with any of the tests between runs. Basically, the output is garbled and vastly less useful for people running this as a matter of routine or as part of a broader kselftest run. For example with my own automation I probably won't notice that a previously working test failed unless every single test fails, and newly added tests that never worked are a much lower priority to the point where I may never look at them depending on where they are. If a selftest is reporting multiple tests it should report them with names that are stable and unique. --+lLg9Zve3xwFMbDn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAmgnL74ACgkQJNaLcl1U h9CSJQf/fXhm7O5lg8Qrz14Y3i/j8Av9aIR/LUBLxRkJ/v19xwDexB6y0bQQVmCF j6QFkQ9Ifxz08T3R00Ravg5WUQRjeSmhDlksSli170TW3QPRJUWmXuLuG2RBo8s9 TmunvK5zQyOIqYCoApXIygewwr1ITKIa1sg5sYnIpDp71Lk2jr77pZG89tQ3AJcu Yd4HyhtuyA8CIfYO56eZFWyDI5xUQNrBNw3HS3YqQVw1SkU8LOqjjTcxADfovi6J ddcFWTUCNhJ0fUzyY1QlUCXkocczidPmBLhYSeBXWU/oPnELAFm4OR6rxRLOOQq5 GQ7BFmSkcer8MMNiQRHM7MGpIpUKZw== =/Hzi -----END PGP SIGNATURE----- --+lLg9Zve3xwFMbDn--