From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B0EFC61D90 for ; Tue, 21 Nov 2023 18:02:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231163AbjKUSC1 (ORCPT ); Tue, 21 Nov 2023 13:02:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229472AbjKUSC0 (ORCPT ); Tue, 21 Nov 2023 13:02:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F23B712A for ; Tue, 21 Nov 2023 10:02:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700589742; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yf6+OqDU33FdIyPnh42M7nerK/HZctSltnWYJNnLM3k=; b=PBgEpDRjTbbfmsChPTQISEDL5XuDYjeMWMacXRTkJFXedrbi5ufaXgw2Gz7pbcgvXEVSLv jSZbGaePA9hZWkOsbtsRAybsFA0Y2VpmYd6eXIVJnL5meMH5eesF4t6siFzlz8d3AOZHPe 0XP6+ul7+qP/TfVjonaY1uujIESXADU= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-135-pmWcyGAeM1muIxwr7KDUnA-1; Tue, 21 Nov 2023 13:02:20 -0500 X-MC-Unique: pmWcyGAeM1muIxwr7KDUnA-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-67804869762so55443496d6.0 for ; Tue, 21 Nov 2023 10:02:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700589740; x=1701194540; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yf6+OqDU33FdIyPnh42M7nerK/HZctSltnWYJNnLM3k=; b=sSDIGF3x8agV4FCnoWcTcUOjSvpnQuIp/uUSXtr1vVLn212gMiOBfumedO7AFvOTzG /N16a5Tj3Hy5YCKm0jCn/Qe71kWRyxxAb+W/cctP4a1r1Ju6OZUvZbmGG/gf1U5NhJ3x Yd1OQMvVzB/4RzjeWuxNgIGQbJfojoou/oovk4BzAOLWftYUrBHgspBfoCKxie/H03FD 8icS0L5C/ITTCzBYaGt6II1yWcCBl7gkYnlu3ScTyc8mfNr5wJGVHRG5iGsU56nLKEqw tm6ts0Qr+dvJGu+aoDn1fQuQVzAP76cJfppau+3znFwqwGiUuDGuPBlOGvCk73XCQ0cD xvSQ== X-Gm-Message-State: AOJu0Yz6o3YZDWJ/nNlofaVtkfJk8ut8/5TJN5+bkV8T1RRVAdgaf6/D Cn/MiYidujkEBUoqCmyqA845MvzAoZJojrP+EDL7EWWh0z/QwfSVVs/wKgWh+WbNRiDq9ixoyFP ljSrLvXi4Xruu/eMewNSN X-Received: by 2002:a05:6214:1945:b0:677:fb3c:2189 with SMTP id q5-20020a056214194500b00677fb3c2189mr18408410qvk.39.1700589740193; Tue, 21 Nov 2023 10:02:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IFoEG1UlzpT2qdb7kbat2sfVcRwhd5K3h5HOqjrN1kpEGZpnFJp5FxbddZ4WY+0Qnz3ytPIAw== X-Received: by 2002:a05:6214:1945:b0:677:fb3c:2189 with SMTP id q5-20020a056214194500b00677fb3c2189mr18408374qvk.39.1700589739925; Tue, 21 Nov 2023 10:02:19 -0800 (PST) Received: from [192.168.0.118] (88-113-27-52.elisa-laajakaista.fi. [88.113.27.52]) by smtp.gmail.com with ESMTPSA id pw9-20020a05620a63c900b0076db1caab16sm3809476qkn.22.2023.11.21.10.02.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Nov 2023 10:02:19 -0800 (PST) Message-ID: <6e50731e-1391-4a8e-9f32-adaa4f8a5119@redhat.com> Date: Tue, 21 Nov 2023 20:02:15 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests Content-Language: en-US To: =?UTF-8?Q?Ricardo_Ca=C3=B1uelo?= Cc: workflows@vger.kernel.org, joe@perches.com, apw@canonical.com, tytso@mit.edu, davidgow@google.com, rostedt@goodmis.org, broonie@kernel.org, skhan@linuxfoundation.org, djwong@kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, vkabatov@redhat.com, cki-project@redhat.com, kernelci@lists.linux.dev References: <87sf50imba.fsf@collabora.com> From: Nikolai Kondrashov In-Reply-To: <87sf50imba.fsf@collabora.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: workflows@vger.kernel.org Hi Ricardo, On 11/20/23 15:30, Ricardo Cañuelo wrote: > On mié, nov 15 2023 at 19:43:49, Nikolai Kondrashov wrote: >> Introduce a new tag, 'Tested-with:', documented in the >> Documentation/process/submitting-patches.rst file. The tag is expected >> to reference the documented test suites, similarly to the 'V:' field, >> and to certify that the submitter executed the test suite on the change, >> and that it passed. > > I think the 'V:' field in MAINTAINERS is a good addition to document > what developers are supposed to test for every subsystem, but in the > case of the per-commit "Tested-with:" tag, I think the real value of it > would be in using it for accountability and traceability purposes > instead, that is, to link to the actual results of the (automatic) tests > that were used to validate a commit. > > This would provide two important features: > > 1. Rather than trusting that the tester did things right and that the > test environment used was appropriate, we'd now have proof that the > test results are as expected and a way to reproduce the steps. > > 2. A history of test results for future reference. When a regression is > introduced, now we'd have more information about how things worked > back when the test was still passing. > > This is not trivial because tests vary a lot and we'd first need to > define which artifacts to link to, and because whatever is linked (test > commands, output log, results summary) would need to be stored > forever. But since we're doing that already for basically all kernel > mailing lists, I wonder if something like "public-inbox for test > results" could be possible some day. I agree that it would be good to have a record of the actual test results uploaded somewhere. For the start, I think it's fine to just have them uploaded to places like Dropbox or Google Drive, or whatever can be accessed with an unauthenticated URL. Otherwise I'm seriously considering opening up submissions to KCIDB for the general (authenticated) public (with pre-moderation and whitelisting). That would require a bunch of work, though. We already have basic artifact mirroring system, but it relies on the submitter hosting the files somewhere in the first place. So we'd have to add some file upload service to that. And then we'd have to think really hard on how to keep the access public, and at the same time not to go bankrupt from somebody scraping our archive in S3 storage. Any help would be super-welcome! I think we can make space for the results URL after the test name in the Tested-with: tag. We can probably make up some syntax for the V: field that would say if the URL is required or not, but it could just be always accepted. It will be the maintainer's call to require it or not. I think it should be up to the test to define what their output should be, and it would be very hard (and not that useful) to unify them in a single output format (speaking from Red Hat's CKI experience executing many different suites for the kernel). The test name in the Tested-with: tag would help identify the format, if necessary. Nick