From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8346EB64DA for ; Thu, 20 Jul 2023 08:14:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AFDA2800D3; Thu, 20 Jul 2023 04:14:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15FFC28004C; Thu, 20 Jul 2023 04:14:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04F732800D3; Thu, 20 Jul 2023 04:14:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E947028004C for ; Thu, 20 Jul 2023 04:14:24 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B5ED41A01EB for ; Thu, 20 Jul 2023 08:14:24 +0000 (UTC) X-FDA: 81031278048.13.8263FB2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 6F3AE4001E for ; Thu, 20 Jul 2023 08:14:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689840862; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bhD0LmqdX6k4L1ayub13WbtK3NYiTAi81Lf8VthdNkE=; b=AgpxHIyem7KjN+ttp3USifJ8DNbkbyXm7+ZnoBzEUy2q1lrj0GOXydMcmBs06Ea3q7To/Y MfbaWEFYiVYNY0SgwD8+pMkonMlU4eHZFDLLFsTyutMDd7oPtrOsEmO0RNzsC9e6Lj8/EX eoPJ+S5EPMdyxp1PNWFjpfEo5oNfSp4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689840862; a=rsa-sha256; cv=none; b=BW5Xh7o/hO/vOuqO+ewZh5nz7pXLw+/jXNihsY+McoqrwRG8tcpdje6vrHriJ2FriMdo2I OlPisLDO/EZrAzr7KBWRF8tnMx7YGw4VAdR03YUF4Qo61r21RP6eKe/pCMaQMKvysvRZM2 SmeyhSIo8gLJveDNy/VZ/W8QpQDPqo8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 767472F4; Thu, 20 Jul 2023 01:15:03 -0700 (PDT) Received: from [10.1.38.38] (C02Z41KALVDN.cambridge.arm.com [10.1.38.38]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A74923F67D; Thu, 20 Jul 2023 01:14:18 -0700 (PDT) Message-ID: <0e512516-6453-f7cb-67ee-f32c17eceea2@arm.com> Date: Thu, 20 Jul 2023 09:14:17 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v2 8/8] selftests/mm: Run all tests from run_vmtests.sh To: Peter Xu , David Hildenbrand Cc: Andrew Morton , Shuah Khan , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Mark Brown , John Hubbard , Florent Revest , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org References: <20230717103152.202078-1-ryan.roberts@arm.com> <20230717103152.202078-9-ryan.roberts@arm.com> <5a12536a-2e33-7a68-6cfd-fd991ddf875a@redhat.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 6F3AE4001E X-Rspam-User: X-Stat-Signature: q9foko8bjhzsgtjj9nuimhrimd3f75gq X-Rspamd-Server: rspam01 X-HE-Tag: 1689840861-959614 X-HE-Meta: U2FsdGVkX1/2XL1LixeTK3QoTAg7Kqt2OkIcfl0f0yt5wEmXTVtMdi3qKF/arWlqodE51aK5HNF2FZJcngyXHmE/hv9zagJO2y2IrZgo7r/UQJB/e+MGikz2xpqi5/Hf2aYWb3h/DVg+8a8ZH6PitqHzygkwLOP6ZVI40m4+6fn+KSusArnxFV99pg4Sad4a0SrBx8abriz7gQUaWCMa8CS8TzhYru7DR4SQfrLKTNEo0v7mMWe8uhGhzr126190Lz4F6OrMZ4u+/hyYfwKhBTsFg8zTTy/Ss+bieA+Di13rEhd7N0ps6Oi6RhqTDNTXOxWS642VHQo42M3PLOj7scy3I6S/ayalGsVMy6OBu0ge+DHcHh5Mx0biHobJC733N0A5fSMV0k6PO/1HioLCDv+693hReCpxnzv6DUIqmjeZLQokhmuaMmxpaAyPhWR5MsneIy9eGm3DhOcbaoPuhTEyTZrmk+dTU/nNQY9NbsxFbsgYorAS3BJEY/hGwF37azEzOqLldoZ+aN40vYcLG15/uLfj+nq7CpXmZxM0ubUjmtghQtI2gD9V3tkL7Uow6jFIU+3ra/icPvLKM/fTuXKiBrsAKXYdvFxI/wJUENW2vxXYRKUs14ueJgfpSHe34AK56MSqArHMkGLvKyhR/liDWA7SCkEWhlhp4Na+gWpGFfEaw8lcRDPUxp87aIGv4e26bpEN7BHxeifiuFJKieCgzD4YyzQBiS3p8w8Z2wAG4TcN7pnL7Kq1h2RvF9SOVuSBFENY4cpj/2fRpiCnI2tPuzzdp5OrpD20o5NJUYSIUDs8FMTtdg7diCfr0FtJlrudyND1HLIbrK7GzNzc6oBlqjicFSmGscyD95gnb+M/afmTj2xmw+sjatsuCB+oYBveBuoEJ4DzruG2Rs5SXTT+LjkLOPntl2gml66Gpb7TUJbJRSERDU9OKV7zcc9Hgt8xZ2hlDHd1AALXCZD heDyTfpI 9QEMS0opqQ4vSEhB9tx2R6I9MakeSgCQliMYSkU/dHvWJXSvjI0yqpEudNoU2yxieHJ8pn31Zvm82LA48sLLUQNKIhvSZB304RvYmqOKgX4zUho1z7tHjh6wk/oM+ZRofzr/buGLbWmrmOXXtaXlBqU/BGTirRSxgxrYwAJ4DtKymMTJ6nm58OcjaLRBJasFITSdxUmtCx2NlIWczgZb5n0+s8r4a7ra2zJQwv5wBaL0ZnOI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 19/07/2023 21:45, Peter Xu wrote: > On Mon, Jul 17, 2023 at 07:27:13PM +0200, David Hildenbrand wrote: >> On 17.07.23 12:31, Ryan Roberts wrote: >>> It is very unclear to me how one is supposed to run all the mm selftests >>> consistently and get clear results. >>> >>> Most of the test programs are launched by both run_vmtests.sh and >>> run_kselftest.sh: >>> >>> hugepage-mmap >>> hugepage-shm >>> map_hugetlb >>> hugepage-mremap >>> hugepage-vmemmap >>> hugetlb-madvise >>> map_fixed_noreplace >>> gup_test >>> gup_longterm >>> uffd-unit-tests >>> uffd-stress >>> compaction_test >>> on-fault-limit >>> map_populate >>> mlock-random-test >>> mlock2-tests >>> mrelease_test >>> mremap_test >>> thuge-gen >>> virtual_address_range >>> va_high_addr_switch >>> mremap_dontunmap >>> hmm-tests >>> madv_populate >>> memfd_secret >>> ksm_tests >>> ksm_functional_tests >>> soft-dirty >>> cow >>> >>> However, of this set, when launched by run_vmtests.sh, some of the >>> programs are invoked multiple times with different arguments. When >>> invoked by run_kselftest.sh, they are invoked without arguments (and as >>> a consequence, some fail immediately). >>> >>> Some test programs are only launched by run_vmtests.sh: >>> >>> test_vmalloc.sh >>> >>> And some test programs and only launched by run_kselftest.sh: >>> >>> khugepaged >>> migration >>> mkdirty >>> transhuge-stress >>> split_huge_page_test >>> mdwe_test >>> write_to_hugetlbfs >>> >>> Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this >>> case all the test programs invoked by both scripts are run twice! >>> >>> Needless to say, this is a bit of a mess. In the absence of fully >>> understanding the history here, it looks to me like the best solution is >>> to launch ALL test programs from run_vmtests.sh, and ONLY invoke >>> run_vmtests.sh from run_kselftest.sh. This way, we get full control over >>> the parameters, each program is only invoked the intended number of >>> times, and regardless of which script is used, the same tests get run in >>> the same way. >>> >>> The only drawback is that if using run_kselftest.sh, it's top-level tap >>> result reporting reports only a single test and it fails if any of the >>> contained tests fail. I don't see this as a big deal though since we >>> still see all the nested reporting from multiple layers. The other issue >>> with this is that all of run_vmtests.sh must execute within a single >>> kselftest timeout period, so let's increase that to something more >>> suitable. >>> >>> In the Makefile, TEST_GEN_PROGS will compile and install the tests and >>> will add them to the list of tests that run_kselftest.sh will run. >>> TEST_GEN_FILES will compile and install the tests but will not add them >>> to the test list. So let's move all the programs from TEST_GEN_PROGS to >>> TEST_GEN_FILES so that they are built but not executed by >>> run_kselftest.sh. Note that run_vmtests.sh is added to TEST_PROGS, which >>> means it ends up in the test list. (the lack of "_GEN" means it won't be >>> compiled, but simply copied). >>> >>> Signed-off-by: Ryan Roberts >>> --- >> >> Acked-by: David Hildenbrand > > Thanks for letting me know, David. Sorry for the late response, still > catching up things. > > I used to justify that from mm/ itself that everything should be PROG, but > I see that from higher level where TEST_GEN_FILE|PROG is really used this > makes sense. As long as vm_utils.o will be properly linked I'll be happy > enough.. Yep that's the case; I've set it up so that vm_utils.o is linked in for both TEST_GEN_FILE and TEST_GEN_PROG binaries. > > Acked-by: Peter Xu Thanks! > > Thanks, >