From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA7D6C001B0 for ; Mon, 17 Jul 2023 17:27:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92EFE6B0074; Mon, 17 Jul 2023 13:27:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DFA46B0075; Mon, 17 Jul 2023 13:27:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 780426B0078; Mon, 17 Jul 2023 13:27:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 685AC6B0074 for ; Mon, 17 Jul 2023 13:27:22 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4130D1C8D17 for ; Mon, 17 Jul 2023 17:27:22 +0000 (UTC) X-FDA: 81021785124.09.1834437 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id D43BB4000D for ; Mon, 17 Jul 2023 17:27:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=D++w7+50; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689614839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w5WtHncCWF3+ygwdoPe907VvBow+yKOAvExoyl7spoY=; b=5K39QOelE4iLmatXr+sx2JQ7cxZwxHe7fUXAaeQf/kfMHjecQJCpcPtKGABDBv2Zzaa8lc Wtbtarot28n9ede3xfzregG2w02edF9U9a8V4aog+EtcdVYtNaHCBstrWsmMyAr85Gvl4T O63dGn11YXKNkAsSCv/BPenrMMtJ1qE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=D++w7+50; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689614839; a=rsa-sha256; cv=none; b=vC53FfhA+DBgHnPGGd9lCzTeS5Mq+TcGCfjL3PkNz3A4c0Fr8gpLYvuzvwfN/2bXfRPX2G 3eJiyh3W13M7GqVmUKvQr1+9+pECzxe32A/haexUI5xUAgyUmtOk/k2/zDDFhWKV6T4d+9 echnTHbW1Ac/FhCB5PPs8MpD8BP75sM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689614838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w5WtHncCWF3+ygwdoPe907VvBow+yKOAvExoyl7spoY=; b=D++w7+50s3+TzPcD8DRF3B8mgAg8tdzEHy3L8b4eSy0f4ewAA2p35iBPFLoS1OpGiavWYK SPeMoUstpY7wTc7mYoF+ncHJCSUjTKKmR79su4kxx4TRXxEuaus4vjtXiFlNTjOoIKC9jt yrrOGruE4UzaZfIINLtBGM9aqWe+q3o= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-499-O00Rfy5aPaOhzdIOWTxa9Q-1; Mon, 17 Jul 2023 13:27:17 -0400 X-MC-Unique: O00Rfy5aPaOhzdIOWTxa9Q-1 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-4fb76659d44so3758486e87.3 for ; Mon, 17 Jul 2023 10:27:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689614835; x=1692206835; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=w5WtHncCWF3+ygwdoPe907VvBow+yKOAvExoyl7spoY=; b=mI+0LLlvDbc49jY47ADykNrQ268pQtPiFVte0NmoBzDNaZEmdkWf8iuyDiSw+EaRGz IerctUzFdDiLy2Y3eYzbNA92yccuMxvk+QxyB0W0c1jjPMJ+v4/oHWvumGh+mBb0UwKE zBttOs4WlGCbZU8SKbzZKrsqwiXn8uS8x1cSb3JrVfYqFi97/i61UuZN5deBiB2ySssa PVMsD5i1ZxObgJ/YPuMfGiepEXOQTZRfFFXIiU7pxtK+xdj7JEWce/NVmcV5x/DvhJ3z r44Db9bcTWzHA6ndxquXrvojZgTSIR1VDIcPvmi1GD0T0p5R5fYG2xDB78X+OMp3F96h FUMQ== X-Gm-Message-State: ABy/qLYmQgbheQmGEfreAxgGsnpTDaBydelSGxwek+wYybt3NGGUYTGe aKFGgMQzmHhKObUXJ+f/kzZf8sQZN0/ZL/Mgng8/zskhI3iB55us1x5cexE7s8f9xKhQhD1fjAQ R2Cd4KvROaEM= X-Received: by 2002:a05:6512:acc:b0:4fb:8492:83fb with SMTP id n12-20020a0565120acc00b004fb849283fbmr9203911lfu.68.1689614835691; Mon, 17 Jul 2023 10:27:15 -0700 (PDT) X-Google-Smtp-Source: APBJJlENfHmen2cNCbd2icPUMMHdFvRR1Wfpqdk+4Ck+wP3t30aPgs1aZcy9uL2FIxVFjUn1iDZz8Q== X-Received: by 2002:a05:6512:acc:b0:4fb:8492:83fb with SMTP id n12-20020a0565120acc00b004fb849283fbmr9203886lfu.68.1689614835294; Mon, 17 Jul 2023 10:27:15 -0700 (PDT) Received: from [192.168.3.108] (p5b0c62d6.dip0.t-ipconnect.de. [91.12.98.214]) by smtp.gmail.com with ESMTPSA id r1-20020adfdc81000000b0031412b685d2sm36662wrj.32.2023.07.17.10.27.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 17 Jul 2023 10:27:14 -0700 (PDT) Message-ID: <5a12536a-2e33-7a68-6cfd-fd991ddf875a@redhat.com> Date: Mon, 17 Jul 2023 19:27:13 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v2 8/8] selftests/mm: Run all tests from run_vmtests.sh To: Ryan Roberts , Andrew Morton , Shuah Khan , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Mark Brown , John Hubbard , Florent Revest , Peter Xu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org References: <20230717103152.202078-1-ryan.roberts@arm.com> <20230717103152.202078-9-ryan.roberts@arm.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <20230717103152.202078-9-ryan.roberts@arm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D43BB4000D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8uymhg5p8k66aydpe8w9kbyj1stpt7t5 X-HE-Tag: 1689614839-167815 X-HE-Meta: U2FsdGVkX190GQ8NqOVWB6PmrA4x7OhGbrvXeic1vf5yZkCEJ0YQpQ5a+Js7XOj+iWdctJQOzAC003dNfmcqTO34IQrZioZIK96K8Aky8F0ZaeWr01svSQ2Nkj8Mlak7+963DpTC0c4jSghdJSIy8hvetXsVsOu12+9S2Bp93hIqnfkAX5Prs39IgbmQyT3nraqMgVsnbJwn3iqdNtv0R/GnDit9BBqJ6I24V/uT42hUeacogWDFNjXRGczc548kruA/l7/sAdIejg2ElRXvOWBVMasVf0htEaeOnTQsSSyJzEgOIdjNE5mvPkl2usjwpve6vt6ZK8U98Lp+OsUoQGXXZlYba0NeFkYk+dSVAlxYhqDS+eSA3odDkJfIQBz7vSev2s7ITFYpR+bugt/IS4UPBwY/Py5CTcIbUoFled/6T6VoRxbhubbaSkrAhPp/o22bXGG1wHk++uiyFHcqeCkehK5yLM9xoz65qdUMGWUcvl9+SzODnFLw1irSypOzB9vxOf39b0NOzk2qJf9c3o0Hco1K/eGI95u0wkwNGaHhPF11N7yjfdnYy/KyAn+AiD3ARQvQOemAjJ2SaaJ3AVYQ9NWEzrlCbcP4VXyp5KDrVG+9OP7oUVJqI8KLtp26F6Yhmy/ZrkczydG51zbGYKw4X5BT3WLNBiO2RpUwE0nn/7Hm4ZZH57RHNZtdv4KfvOIPfWGc9fhyt/yLhUCg2lNnVS593b+Lz3xm6pSqn7Sabw2SH5vr4cDtIpLD2KrtW8hfFe/lC9+J6OmgJDwM+6fUkUl+Fs//ZSq6NTkhP1UvnWsfKYhLOa+UNkbzvaf/aY6oZ6Zdd4M/iKVaW+Bqfz9e6Mi8PbfMM11mxKRAusxPU41zmYwMSQX9XsndwVi2/qU2X7UcS0qhBtmsnyXdVP6bdSujkBs7EBCR4kYeWvJ3t3tqLibO6LtT+0Tmr5fvGSjZcmbFFTuCaLeG2qF A6Fzs+zQ ZNVLjsz0z/z5TFaVAbp1wZu1/AQ8WTEFA9fvX8GYJORM9qrDb5mKFQ8Q9r9iv3vYykjeZjg54/5VPUXQhMXyWNrJDgUUy15pgxqvuL38+gHU4OXFEDpOQi1UlS7HLm414Ryb8e0u5XJs0gTvnbN0I0XeMxaAxcg5f6H5kNaL5/PW+uvf/IiaDCSQykqS08ZX7QJcj9VQNyjbD10o7onY8mcqItHUiW8xXjBBtFjZFt8Ep59tmlr+W2Aj0OxeWgpP//EfQT/2ojjnRCQqzBRwFt/oxWoiZP4YR2IrqRibEno1E6fI454yp1EYpXgRXWnXHSxwn5wMQGLQkh9J9aQcH60qqlvF/Jg+bHMMzcMRvUGYx2HjtNLztjgfZSodFXl75WdcEYWHVdeadFTOZlTFnDHRxCyj0+0JvgTosX29P1lG1TdmGKmE2P0Tr686IC2SgQB6MzQAV6MQXVsa8mVf7i/ouwcRPpCf+v4YC0vE0nF4VapuVnNtroWBt+TT7+P8LiOhHM/5zP3aKB2nPysV5Ib7BUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 17.07.23 12:31, Ryan Roberts wrote: > It is very unclear to me how one is supposed to run all the mm selftests > consistently and get clear results. > > Most of the test programs are launched by both run_vmtests.sh and > run_kselftest.sh: > > hugepage-mmap > hugepage-shm > map_hugetlb > hugepage-mremap > hugepage-vmemmap > hugetlb-madvise > map_fixed_noreplace > gup_test > gup_longterm > uffd-unit-tests > uffd-stress > compaction_test > on-fault-limit > map_populate > mlock-random-test > mlock2-tests > mrelease_test > mremap_test > thuge-gen > virtual_address_range > va_high_addr_switch > mremap_dontunmap > hmm-tests > madv_populate > memfd_secret > ksm_tests > ksm_functional_tests > soft-dirty > cow > > However, of this set, when launched by run_vmtests.sh, some of the > programs are invoked multiple times with different arguments. When > invoked by run_kselftest.sh, they are invoked without arguments (and as > a consequence, some fail immediately). > > Some test programs are only launched by run_vmtests.sh: > > test_vmalloc.sh > > And some test programs and only launched by run_kselftest.sh: > > khugepaged > migration > mkdirty > transhuge-stress > split_huge_page_test > mdwe_test > write_to_hugetlbfs > > Furthermore, run_vmtests.sh is invoked by run_kselftest.sh, so in this > case all the test programs invoked by both scripts are run twice! > > Needless to say, this is a bit of a mess. In the absence of fully > understanding the history here, it looks to me like the best solution is > to launch ALL test programs from run_vmtests.sh, and ONLY invoke > run_vmtests.sh from run_kselftest.sh. This way, we get full control over > the parameters, each program is only invoked the intended number of > times, and regardless of which script is used, the same tests get run in > the same way. > > The only drawback is that if using run_kselftest.sh, it's top-level tap > result reporting reports only a single test and it fails if any of the > contained tests fail. I don't see this as a big deal though since we > still see all the nested reporting from multiple layers. The other issue > with this is that all of run_vmtests.sh must execute within a single > kselftest timeout period, so let's increase that to something more > suitable. > > In the Makefile, TEST_GEN_PROGS will compile and install the tests and > will add them to the list of tests that run_kselftest.sh will run. > TEST_GEN_FILES will compile and install the tests but will not add them > to the test list. So let's move all the programs from TEST_GEN_PROGS to > TEST_GEN_FILES so that they are built but not executed by > run_kselftest.sh. Note that run_vmtests.sh is added to TEST_PROGS, which > means it ends up in the test list. (the lack of "_GEN" means it won't be > compiled, but simply copied). > > Signed-off-by: Ryan Roberts > --- Acked-by: David Hildenbrand -- Cheers, David / dhildenb