From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0832C61D9D for ; Wed, 22 Nov 2023 20:51:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231429AbjKVUvw (ORCPT ); Wed, 22 Nov 2023 15:51:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230377AbjKVUvv (ORCPT ); Wed, 22 Nov 2023 15:51:51 -0500 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 792111B6 for ; Wed, 22 Nov 2023 12:51:47 -0800 (PST) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1ce656b9780so1421745ad.2 for ; Wed, 22 Nov 2023 12:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1700686307; x=1701291107; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=QDYyRLoiycUaY49QoLf26F930m6gNVp1RU3PlgGtl7M=; b=eERjgB3sh2hm3upFresTqzAHyiUxEGuZZ62OrpJkl095cpFA1GYlAE53MC6PsvAsV8 6Ut0z2EsXM57RdECbRVT28bCZDG4UXCKcpVHQUhB/hYiRSouQ65gxmHp1nnJolRJLtwy MO8cBR42HRqBYTMseGGfpfyjUVnUZk+Ci90GLmOJaKbM7zpzCJ5IGXv1VlGRLTgGmttl sKSwefTFaYH55NOgeDF96nVKzXwWMfRBUZn2oO7Zd/c9714f1/pdRSbfKYeVyvOwBcos dgNBKY7DeJ+yn0GwXNTq6mXZQVzlmcAdqQldobbq9nem2zHtvkRo2OKJkvpS/4zZjDaX prLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700686307; x=1701291107; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=QDYyRLoiycUaY49QoLf26F930m6gNVp1RU3PlgGtl7M=; b=egQmEhSCJn5l4u0jfSSuRN8IylMdl9vwggIIpyUWAnyiseHeq/+gJbHidNOp03ItTu LeaeS/C+oCgzMex7PhtqtScHPGDKeMaS1bz1neHcx6U/0WDt8M5UJOCF4AauTqF0Lsvq luxhII+gilt2aLPjRsKd7WynIKcEYBtbwJu6EgJRtnmci7nUKrZj7APurDgHVAJcggwe prVdSsfPwkX7/UQ8P7mhhnQVagEIZeTalLJ2xpAIvKY9HVpP9W7LkqOUbthMXEz64acJ c35tAgl3b2nXX4UGtiCF8feDGJRhgp7F4ngmMnLX71KX1zBKwB2YcEN2VmbAkyQtzaJ9 vuGg== X-Gm-Message-State: AOJu0YyRYMxDw1HC2LkYTud0WtTSH3fx/b9Y0rFf3kcR3Mp/oo5xQMcP es+fp7hYW9aASLD2YA8S3pOsOg== X-Google-Smtp-Source: AGHT+IE6oL5OBU3IwVHKwsZfJXYmwuWE0TSV4peQofShC5i+2wEn3Ehzw2AsBZVt3VWIgqiavzxf1g== X-Received: by 2002:a17:902:8649:b0:1cc:e823:c8cc with SMTP id y9-20020a170902864900b001cce823c8ccmr3253781plt.41.1700686306783; Wed, 22 Nov 2023 12:51:46 -0800 (PST) Received: from dread.disaster.area (pa49-180-125-5.pa.nsw.optusnet.com.au. [49.180.125.5]) by smtp.gmail.com with ESMTPSA id z3-20020a170902ee0300b001cc2ebd2c2csm100557plb.256.2023.11.22.12.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 12:51:46 -0800 (PST) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1r5uCF-00G9zk-1o; Thu, 23 Nov 2023 07:51:43 +1100 Date: Thu, 23 Nov 2023 07:51:43 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Nikolai Kondrashov , Theodore Ts'o , Chandan Babu R , workflows@vger.kernel.org, Joe Perches , Andy Whitcroft , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Chandan Babu R Subject: Re: [PATCH 2/3] MAINTAINERS: Require kvm-xfstests smoke for ext4 Message-ID: References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> <20231115175146.9848-3-Nikolai.Kondrashov@redhat.com> <20231115185808.GD36211@frogsfrogsfrogs> <87v8a096cr.fsf@debian-BULLSEYE-live-builder-AMD64> <20231119225437.GA292450@mit.edu> <20231122161746.GM36211@frogsfrogsfrogs> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231122161746.GM36211@frogsfrogsfrogs> Precedence: bulk List-ID: X-Mailing-List: workflows@vger.kernel.org On Wed, Nov 22, 2023 at 08:17:46AM -0800, Darrick J. Wong wrote: > On Wed, Nov 22, 2023 at 04:44:58PM +0200, Nikolai Kondrashov wrote: > > On 11/20/23 00:54, Theodore Ts'o wrote: > > > So as for *me*, I'm going to point people at: > > > > > > https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md > > > > ... > > > > > (And note that I keep the xfstests-bld repo's on kernel.org and > > > github.com both uptodate, and I prefer using the using the github.com > > > URL because it's easier for the new developer to read and understand > > > it.) > > > > I already queued a switch to the kernel.org URL, which Darrick has suggested. > > I'll drop it now, but you guys would have to figure it out between yourselves, > > which one you want :D > > > > Personally, I agree that the one on GitHub is more reader-friendly, FWIW. > > For xfstests-bld links, I'm ok with whichever domain Ted wants. > > > > And similarly, just because the V: line might say, "kvm-xfstests > > > smoke", someone could certainly use kdevops if they wanted to. So > > > perhaps we need to be a bit clearer about what we expect the V: line > > > to mean? > > > > I tried to handle some of that with the "subsets", so that you can run a wider > > test suite and still pass the Tested-with: check. I think this has to be > > balanced between allowing all the possible ways to run the tests and a > > reasonable way to certify the commit was tested automatically. > > > > E.g. name the test "xfstests", and list all the ways it can be executed, thus > > communicating that it should still say "Tested-with: xfstests" regardless of > > the way. And if there is a smaller required subset, name it just "xfstests > > smoke" and list all the ways it can be run, including the simplest > > "kvm-xfstests smoke", but accept just "Tested-with: xfstests-smoke". > > > > I'm likely getting things wrong, but I hope you get what I'm trying to say. > > Not entirely -- for drive-by contributions and obvious bugfixes, a quick > "V: xfstests-bld: kvm-xfstests smoke" / "V: fstests: ./check -g smoke" > run is probably sufficient. For trivial drive-by contributions and obvious bug fixes, I think this is an unnecessary burden on these potential contributors. If it's trivial, then there's little burden on the reviewer/maintainer to actually test it, whilst there is significant burden on the one-off contributor to set up an entirely new, unfamiliar testing environment just to fix something trivial. If you want every patch tested, the follow the lead of the btrfs developers: set up a CI mechanism on github and ask people to submit changes there first and then provide a link to the successful test completion ticket with the patch submission. > (Insofar as n00bs running ./check isn't sufficient, but that's something > that fstests needs to solve...) > > For nontrivial code tidying, the author really ought to run the whole > test suite. It's still an open question as to whether xfs tidying > should run the full fuzz suite too, since that increases the runtime > from overnightish to a week. Yes, the auto group tests should be run before non-trivial patch sets are submitted. That is the entire premise of the the auto group existing - it's the set of regression tests we expect to pass for any change. However, the full on disk format fuzz tests should not be required to be run. Asking people to run tests that take a week for general cleanups and code quality improvements is just crazy talk - the cost-benefit equation is all out of whack here, especially if the changes have no interaction with the on-disk format at all. IMO, extensive fuzz testing is something that should be done post-integration - requiring individual developers to run tests that take at least a week to run before they can submit a patchset for review/inclusion is an excessive burden, and we don't need every developer to run such tests every time they do something even slightly non-trivial. It is the job of the release manager to co-ordinate with the testing resources to run extensive, long running tests after code has been integrated into the tree. Forcing individual developers to run this sort of testing just isn't an efficient use of resources.... > For /new features/, the developer(s) ought to come up with a testing > plan and run that by the community. Eventually those will merge into > fstests or ktest or wherever. That's how it already works, isn't it? -Dave. -- Dave Chinner david@fromorbit.com