From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 778C8C001B0 for ; Wed, 19 Jul 2023 20:46:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CD1428008F; Wed, 19 Jul 2023 16:46:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67D4228004C; Wed, 19 Jul 2023 16:46:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51E4028008F; Wed, 19 Jul 2023 16:46:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3FE4828004C for ; Wed, 19 Jul 2023 16:46:06 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F2CB916045C for ; Wed, 19 Jul 2023 20:46:05 +0000 (UTC) X-FDA: 81029543490.24.94A31C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id D9390140007 for ; Wed, 19 Jul 2023 20:46:02 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fe+aX742; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689799562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b12/aDc/W25MU+G31DwKlpA3J3XW9Rd+FvX9Yt4/KPo=; b=BvuagKPbq2LHQb97XsGrbBP2dW7vWSismt4tg5RL+R7ehPkxmtyHKltq8qyVoTgs5DWvQK dzIumovqAsvIwo/okFm6+v9cU6UgDER2HBNScXB9DVNBCohUTYH6pVgU+pRq4cn1M3+kNi oGEHuIIUjHKqKrDNBdK/PXfd+h0+jlw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689799562; a=rsa-sha256; cv=none; b=6bwrb9WaZuoNk4gFylXPrzuJYfC8ympSaR8bBunXPVZDnx+7IZhCw45s7z2vAjEcpW+2Pm eNODqHnG5N1ozcexZG6fX2WI6UkXZq46O9znNLjjDC1bKiL1auXQeRMsWCuQg4DP3rU9If q3UvC+vYSg8PEirOx8YQjaU8yiWtroA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fe+aX742; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689799561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=b12/aDc/W25MU+G31DwKlpA3J3XW9Rd+FvX9Yt4/KPo=; b=fe+aX742ZA9Y7lG91hhP1eA7XgDct+a3LSE34nGkrrpCgMQEFBCtUe0/OlTRpnFHDPVpTe kwRiKf0MHlL/R5jd1g/5iSaDeYrXIfrqeRSYJUrAixFuyCgI2fOem0hUu4NPCJ+q/VMnJh 12hkxvC/PYSAF7pcpdRCN8XIoM7/L48= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-zYrE-4NIMbC3L2-h9VR4vQ-1; Wed, 19 Jul 2023 16:45:58 -0400 X-MC-Unique: zYrE-4NIMbC3L2-h9VR4vQ-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-4053d3613f9so367231cf.0 for ; Wed, 19 Jul 2023 13:45:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689799558; x=1690404358; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=b12/aDc/W25MU+G31DwKlpA3J3XW9Rd+FvX9Yt4/KPo=; b=fV6HiEGIuH9gaelh7lXgzgs6X9Xi1KCZiNH6aUGJBNiaVng0cBTdXkT9Wpyz62WgQJ Czhvn3b4r5EGtRXQ2iSJP29N6e97XzCLgOaPPGHH6/0j0lnuCI/3RMVAAT9u+34KlOMG Mcc+byWlpPcUekEff3G1RWH6BIR0gamBYhZh2ODwvo8adTUQyTiqHzQgZrXgMED2Altr 4vV96BijbnzIP3kzPIom2f3bqWKLmgfPnZXPCsmOVZQ7LfHgVkfJ2tC/2WDOOI2mWog1 +azOlc4hd437zdrRzXo/RgSgzYSLe3zqz6mYq2qyLDwHMtSt7FaUrenAoGo0opw/+IYb deXw== X-Gm-Message-State: ABy/qLZh3Wn1Az7sB4HHVci5ASZgSVkTuY5+K9p6Atag6L3Mq7w79y7a h/ba4GM7PtfE2sQNh1q298uCcoQO55gvmIe11aUGIu8TI8YxTAyPGVP5uMkIgSJF9sGXeqdok9u xRPV6SBrxh7M= X-Received: by 2002:ac8:5a8d:0:b0:400:8036:6f05 with SMTP id c13-20020ac85a8d000000b0040080366f05mr1056334qtc.2.1689799558105; Wed, 19 Jul 2023 13:45:58 -0700 (PDT) X-Google-Smtp-Source: APBJJlGSHzDb9t3k9dqegflbS3XBeYFDxCtTpS4UPlvDTuKAeHWVA2dhbsFeIaJTUjfOppuCRIL0qg== X-Received: by 2002:ac8:5a8d:0:b0:400:8036:6f05 with SMTP id c13-20020ac85a8d000000b0040080366f05mr1056324qtc.2.1689799557835; Wed, 19 Jul 2023 13:45:57 -0700 (PDT) Received: from x1n (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id ci23-20020a05622a261700b004037cd9bd40sm1592793qtb.15.2023.07.19.13.45.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 13:45:57 -0700 (PDT) Date: Wed, 19 Jul 2023 16:45:45 -0400 From: Peter Xu To: David Hildenbrand Cc: Ryan Roberts , Andrew Morton , Shuah Khan , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Mark Brown , John Hubbard , Florent Revest , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Subject: Re: [PATCH v2 8/8] selftests/mm: Run all tests from run_vmtests.sh Message-ID: References: <20230717103152.202078-1-ryan.roberts@arm.com> <20230717103152.202078-9-ryan.roberts@arm.com> <5a12536a-2e33-7a68-6cfd-fd991ddf875a@redhat.com> MIME-Version: 1.0 In-Reply-To: <5a12536a-2e33-7a68-6cfd-fd991ddf875a@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: D9390140007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: rb5g1usex4ygyho9rx3jzhbpur8bo9rr X-HE-Tag: 1689799562-70574 X-HE-Meta: U2FsdGVkX18ViX5HLp7YrsVdK0J+1atPTbDnN2AVetGDr4b1EUawFjYHrSWpqJAveC301nbBwio6jp1t1qNqaunwbF2nxa1pvGuh6duzSwZRGEUYTPw3vLx+aMcvMbxtvAG0pHpkMtwotWD0rdPHqUx7jn53zY2uYpZU98PKS1ACBCnv/rCQHnXgYkCiRff5nxgnGEaXtXeUKpcIzMHjbZfoIEPMRYtsriUyTccf1cC4gEuxpYctKqYOf5sdAR2UO1OwlLEC6K90+sM9tNbTNDKrwQQCCMkj7mj0IIdsuBv77w31F/rQy8CFnUpIJY3Jw31s8wCP8a5zHft5S4REv7Vmq+wltbOBV0ojf+gu2u5P6SCpTlKPLak434aq+ECPK23at+k4WVw3/aAgxLNthsWGcqoweJSUrSC9xh0CuIiG2i7afasCYeO12LUNgi3QLUoinzaLcXm7vUBqin6yiHjG8XratWEqiQ+AxylWBradIpZYivQYpACNtYayZ8kcBAHa7vt17U6DCm6dLF+65RoI1HvCvngCgkqF8Qiw4OYoRgDbU+6T7goYQXx1DxBsAWPbT2gFN6CIW4zo4zV65gNpn0ShRJ2S3AL/0uLDRDe6j4LigMEUvD4c0oRbTiF3QE0B1esmeQ96wO1wivMQrIrOX1edaFA0MW9efZhjc2yiimypH/F3rj0bI4U+NOc8pIyKKM99N5qwrTh3gTsXt8duJuu2hCWyg2MM7/ddbPmIm63z6pjyjtPXPHulGBZ2bYTJVZDV/MmgPIEf3XsgyxZnhQ+aLfmd3G8iP29fkkGUr2W+T73SpATPZe4KXMRkSENnyPK3roGDlivN5wDzDR6KgzEdUmKc2Y9s+2HqJ4BJEj4KKgI5vi5Qa4b2H9loNTOeQZf8km6OVPm2H+grCV9hDsGhvzrIjoPMd4nFL4nZw05XP9iQzmmZsaCITuatjtshK4WIX6qalucRivD X8FowRi0 g8CQwtOE1qJpmsQQqrrog2ws3GVx0Aj+ceo/H0zRKnfXUgwl/uCqLbG0KZq9r8cAJp08SUsNAInUz8ZwMHglO1bAlMrkdk22Lkwo4bd268fjqQkzRoMBSUUaJ4nsuXf8yDevOAAcVvTwwEfNctN2SqmUuapYsDEzxrfqWE+rhbgQKwKMPJjieioPLMGSjW8W07sGyXnVv7EZ8LElJ9V1kb67bm1jMdNf+vaYcFo4siyRcBvIQZjQMPvQoZIyKASU8/dwqZCDJ/xJqmtDXY+zYaCrsVNDZ94iTm6F8BB7UcigrkPges7HIGnX+Q37kVoT+9EnTS9HIyHwE9rrDg/Y5KIVopq5nYkDYTcGr70TuTEx7LwF3oSbnjcvdhp6vZY0diWCGTsO4oqSlXGN2cOOG/exCpoZn8/DIyouOAr6qLuIP88qDJAcqt3eiSa6Bik9ZJfBbosW8mJvWqz72DrtDxOj1INVyY/Tqk420iSbypmab6Bw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.007887, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jul 17, 2023 at 07:27:13PM +0200, David Hildenbrand wrote: > On 17.07.23 12:31, Ryan Roberts wrote: > > It is very unclear to me how one is supposed to run all the mm selftests > > consistently and get clear results. > > > > Most of the test programs are launched by both run_vmtests.sh and > > run_kselftest.sh: > > > > hugepage-mmap > > hugepage-shm > > map_hugetlb > > hugepage-mremap > > hugepage-vmemmap > > hugetlb-madvise > > map_fixed_noreplace > > gup_test > > gup_longterm > > uffd-unit-tests > > uffd-stress > > compaction_test > > on-fault-limit > > map_populate > > mlock-random-test > > mlock2-tests > > mrelease_test > > mremap_test > > thuge-gen > > virtual_address_range > > va_high_addr_switch > > mremap_dontunmap > > hmm-tests > > madv_populate > > memfd_secret > > ksm_tests > > ksm_functional_tests > > soft-dirty > > cow > > > > However, of this set, when launched by run_vmtests.sh, some of the > > programs are invoked multiple times with different arguments. When > > invoked by run_kselftest.sh, they are invoked without arguments (and as > > a consequence, some fail immediately). > > > > Some test programs are only launched by run_vmtests.sh: > > > > test_vmalloc.sh > > > > And some test programs and only launched by run_kselftest.sh: > > > > khugepaged > > migration > > mkdirty > > transhuge-stress > > split_huge_page_test > > mdwe_test > > write_to_hugetlbfs > > > > Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this > > case all the test programs invoked by both scripts are run twice! > > > > Needless to say, this is a bit of a mess. In the absence of fully > > understanding the history here, it looks to me like the best solution is > > to launch ALL test programs from run_vmtests.sh, and ONLY invoke > > run_vmtests.sh from run_kselftest.sh. This way, we get full control over > > the parameters, each program is only invoked the intended number of > > times, and regardless of which script is used, the same tests get run in > > the same way. > > > > The only drawback is that if using run_kselftest.sh, it's top-level tap > > result reporting reports only a single test and it fails if any of the > > contained tests fail. I don't see this as a big deal though since we > > still see all the nested reporting from multiple layers. The other issue > > with this is that all of run_vmtests.sh must execute within a single > > kselftest timeout period, so let's increase that to something more > > suitable. > > > > In the Makefile, TEST_GEN_PROGS will compile and install the tests and > > will add them to the list of tests that run_kselftest.sh will run. > > TEST_GEN_FILES will compile and install the tests but will not add them > > to the test list. So let's move all the programs from TEST_GEN_PROGS to > > TEST_GEN_FILES so that they are built but not executed by > > run_kselftest.sh. Note that run_vmtests.sh is added to TEST_PROGS, which > > means it ends up in the test list. (the lack of "_GEN" means it won't be > > compiled, but simply copied). > > > > Signed-off-by: Ryan Roberts > > --- > > Acked-by: David Hildenbrand Thanks for letting me know, David. Sorry for the late response, still catching up things. I used to justify that from mm/ itself that everything should be PROG, but I see that from higher level where TEST_GEN_FILE|PROG is really used this makes sense. As long as vm_utils.o will be properly linked I'll be happy enough.. Acked-by: Peter Xu Thanks, -- Peter Xu