From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9427EB64DC for ; Sun, 18 Jun 2023 19:19:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EB498D0002; Sun, 18 Jun 2023 15:19:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 09B528D0001; Sun, 18 Jun 2023 15:19:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECBE88D0002; Sun, 18 Jun 2023 15:19:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D9A788D0001 for ; Sun, 18 Jun 2023 15:19:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AE167A0298 for ; Sun, 18 Jun 2023 19:19:54 +0000 (UTC) X-FDA: 80916833508.09.EF08AB1 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf21.hostedemail.com (Postfix) with ESMTP id 00CDF1C0006 for ; Sun, 18 Jun 2023 19:19:51 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=eeTeUO73; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687115992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JbR6FYiZlzxET1aZbxa1LnPa25BlJwTh4qhaLJ4bvzg=; b=jI2vtSOOLxfCS4HLV8tVsFxvAdbh2wPb0+JVsbTM4Kw05lpCHc3DGls8058yAU9+F508Wo TrSqlKnmQTfjpS/8JfWjl2HizAY80u/RqItguiMdp2vS/1rV660gxq7fSEklGCPMR69G+O e9Gq6p838uEf0PCtZKGxDTZogll275Q= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=eeTeUO73; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687115992; a=rsa-sha256; cv=none; b=iRTFasd/FIMa0Vuozho1T03yMaB8Koie5yGvdLM6+c1QWcU9CjcvfIX0FdkiDUc4BBPaFt NWXB6Oi6eSUpT6uHJjKQBIGwEvBVPvOiUX+JuqUdToIsGx7Z4R0u7bC8K3cdaEP/ZjNSrT ZH77I1M1S9wBm5GHwCwPRjMGUVqnYko= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-3fddc620055so202311cf.1 for ; Sun, 18 Jun 2023 12:19:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687115991; x=1689707991; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=JbR6FYiZlzxET1aZbxa1LnPa25BlJwTh4qhaLJ4bvzg=; b=eeTeUO735vurVqYffUSvtO4InMdD7ecrFouGxTvZhYEKJAFj+iNmhaN9tu2ONEh2zq 89mW0ayvjTPwrfBsoKUi9fd5fr9ze0ICHF7BxV4woHO5JqvPZRPKLY2xrrn26G+NJH0x DUxE98voVWezGuxg9OUVAFu8rupSLoeVL/QnMQxAmb/k40sScPQjWMg9w2/P2DgTkMb8 SHdvwblgqJJHWx+7iPLuQxgNy100g4QEuXp4/vYMvNUabSEKRH66I5WIJ93nZtC1CnKk 7RYEIUbSVvCf3kGwuVSB8mzNNo/aPYY38t591pX2WAN2fS9n4tJiTPSz8cqRXqfOVBu7 hOsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687115991; x=1689707991; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JbR6FYiZlzxET1aZbxa1LnPa25BlJwTh4qhaLJ4bvzg=; b=ESNgJbN8hMyT4SVV7leokGV7dCpcpp1Wsc9PCVd0SRQaD8h0KTUN1C90vxehoCwtni aljTnD+NSY8XwjPJeAdEM6F5uMuvXABYaOu8MoAOpjN6QmbTga7wwobveDsBIBbgLcfJ lCouFDbBiw80J+VhYYNWPnqE6j9d9twN6/N0YKX5//2+MLaLMIWeKRWhptLhVZaJnBsB VOK+FT2wNPbd+LzIs155WobHGvPXZNqXkdUgl31lxQMil4oV3HmKOxBrPYKKxSuKMJpn vm6qljosxDCmdklrGN1LbiapQ1v0v/mQzSJV3jmAplv1n5VKjVEn3cLz6svdaa8xJduf ELiA== X-Gm-Message-State: AC+VfDx1CLdEnhH2R3oLfvfG55buyi+E78znUHjxl3kQCo72TN+Ouzrk GlErm1QdVGYnw9beE13jdy5yDpGGVVTnPltCFP/asw== X-Google-Smtp-Source: ACHHUZ4amVIobuPycpFLC2fHeYKXTl1nn9tY9t3tZcZ8KQLrci6OwIYFpzYKhnwEJ6EGI3MQyCTT/zihLmvQ2yroRTk= X-Received: by 2002:a05:622a:589:b0:3ed:210b:e698 with SMTP id c9-20020a05622a058900b003ed210be698mr754708qtb.7.1687115990926; Sun, 18 Jun 2023 12:19:50 -0700 (PDT) MIME-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> <20230609005943.43041-1-yuzhao@google.com> In-Reply-To: <20230609005943.43041-1-yuzhao@google.com> From: Yu Zhao Date: Sun, 18 Jun 2023 13:19:14 -0600 Message-ID: Subject: Re: kvm/x86: multichase benchmark To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 00CDF1C0006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zuyjudbk79n8fkihop6beswxircx5iic X-HE-Tag: 1687115991-786776 X-HE-Meta: U2FsdGVkX1/Mx4+Q/euqgM2bdlcZa+E8z/CkjLxT4ZkCMyV9St2wsu9qDjn+50y/O2tDlB8mK3d6nryaRQtrdUG+ik2UOYXE9VdU3GwDSBPTjXH3k/FxZeCXMC3n9UfGfuDIC4o7FnE5B7KAe71Jiqjqcs0pxloJOCjl13AnxvPg7jyPnDrjLRLev3G0rqD2LKlm7uPorEHI07+yE8/lH4fV15LnxsfXolQofUe0yswFOCkwZTiGrXpupEReT/Y8pfoOhUO2e8cwmSl795K/d7S+SqOmVi/RzNgNrrdGB1t0CR7ZEuPGbKahpqnhx0eOesCaqqUPUHMSyc4vDMUqP3AJAs5Fvh2WcmlKHWgt7fDQtmOB1unbdJtTWKrdmExgkhroI4x7u6QTGq4OrJu+PioanB5+scKT0y/XQ4papFgq7m57Ta6YdYIflPkdWDBgcO54hKQqdspOmAuEk4TNwWy7C3WzP8YqRsGojXrokbyCmy7LrcFbx6vgKqyGNk28bONO/VDerQTb0K0VLHmgNlntuRJ8wEkVDAF+Nh4gpGgIASYP80tgfOIk1xTjQpj+rN8qCsx4HH76bMV+YriH7Ue1CWFk7uy2y4h4RyNO08la6zurlDpVRSpnwp51D4O18t99Oovp7A//t/nImRAxPoTlvX081pMYEzIfV8qTlmBI635iera71B+Hnn0pfZPx7AEkHLlyDojIVsmhetsmrDDL3fzcQSpIwhiuUunZV071Er5dc6cOa7KH1jqcXZx4FHAJ9UTKV8+2hvB+sMOIol8Y0JUi8v6ELuOmlkuPVGTBww44i4GjF62O3FvcIWPKtjTa0Niyd0n+m3F444nkpKiZtNXJFLjb80QrrfcGzq696JB10SEo70nx8C5WYDrmQBRMYYCi9K4a+hzUKkSKa5KQ7BlUEWd6wjXLp9dnFIELThCp50uTmgiy6Q5WqbnG6CSGtep61pFbuoEWNFt 7azDWsg3 jTUnl7P5wSZVK1pcu7epCWzgdfnJKB9CRtkMkehBsz5KMtK4eVLH1YoJoxRXL86A2GCBwXibEYfRZO4jFx+p1tHZhLujoM0NHHPWCzy/3bj4TO1P/qh0hg/9JYyw59q7fuZljIWfRrOGRSJUNTMeWld02ymF7cG1NKl8p2I2IC3zkN3fWENyoZF0VNE+5VxfkZyG/Tqs689HgS+Ly0iDA9x4b0d8FDqF2KF8Ql9Z+ovQ6EDjYAAB+7ZTWNQKD0UPriFTuHTERgdW+nga2jTjUyEWiavom2p3ppa23E4GcpKnlWAwv0u9tSbufySyE0/8t3r2fjO7/IpaEeEeuz4whusXemmrybOPhuvLY0r5eNhhebjc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 8, 2023 at 6:59=E2=80=AFPM Yu Zhao wrote: > > TLDR > =3D=3D=3D=3D > Multichase in 64 microVMs achieved 6% more total samples (in ~4 hours) af= ter this patchset [1]. > > Hardware > =3D=3D=3D=3D=3D=3D=3D=3D > HOST $ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Address sizes: 43 bits physical, 48 bits virtual > Byte Order: Little Endian > CPU(s): 128 > On-line CPU(s) list: 0-127 > Vendor ID: AuthenticAMD > Model name: AMD Ryzen Threadripper PRO 3995WX 64-Cores > CPU family: 23 > Model: 49 > Thread(s) per core: 2 > Core(s) per socket: 64 > Socket(s): 1 > Stepping: 0 > Frequency boost: disabled > CPU max MHz: 4308.3979 > CPU min MHz: 2200.0000 > BogoMIPS: 5390.20 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr= pge mca cmov pat pse36 clflush mmx fxsr sse sse2 > ... > Virtualization features: > Virtualization: AMD-V > Caches (sum of all): > L1d: 2 MiB (64 instances) > L1i: 2 MiB (64 instances) > L2: 32 MiB (64 instances) > L3: 256 MiB (16 instances) > NUMA: > NUMA node(s): 1 > NUMA node0 CPU(s): 0-127 > Vulnerabilities: > Itlb multihit: Not affected > L1tf: Not affected > Mds: Not affected > Meltdown: Not affected > Mmio stale data: Not affected > Retbleed: Mitigation; untrained return thunk; SMT enabled = with STIBP protection > Spec store bypass: Mitigation; Speculative Store Bypass disabled vi= a prctl > Spectre v1: Mitigation; usercopy/swapgs barriers and __user = pointer sanitization > Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP = always-on, RSB filling, PBRSB-eIBRS Not affected > Srbds: Not affected > Tsx async abort: Not affected > > HOST $ numactl -H > available: 1 nodes (0) > node 0 cpus: 0-127 > node 0 size: 257542 MB > node 0 free: 224855 MB > node distances: > node 0 > 0: 10 > > HOST $ cat /sys/class/nvme/nvme0/model > INTEL SSDPF21Q800GB > > HOST $ cat /sys/class/nvme/nvme0/numa_node > 0 > > Software > =3D=3D=3D=3D=3D=3D=3D=3D > HOST $ cat /etc/lsb-release > DISTRIB_ID=3DUbuntu > DISTRIB_RELEASE=3D22.04 > DISTRIB_CODENAME=3Djammy > DISTRIB_DESCRIPTION=3D"Ubuntu 22.04.1 LTS" > > HOST $ uname -a > Linux x86 6.4.0-rc5+ #1 SMP PREEMPT_DYNAMIC Wed Jun 7 22:17:47 UTC 2023 = x86_64 x86_64 x86_64 GNU/Linux > > HOST $ cat /proc/swaps > Filename Type Size Used Priority > /dev/nvme0n1p2 partition 466838356 0 -2 > > HOST $ cat /sys/kernel/mm/lru_gen/enabled > 0x000f > > HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled > always madvise [never] > > HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag > always defer defer+madvise madvise [never] > > Procedure > =3D=3D=3D=3D=3D=3D=3D=3D=3D > HOST $ git clone https://github.com/google/multichase > > HOST $ > HOST $ > > HOST $ cp multichase/multichase ./initrd/bin/ > HOST $ sed -i \ > "/^maybe_break top$/i multichase -t 2 -m 4g -n 28800; poweroff" \ I was reminded that I missed one parameter above, i.e., "/^maybe_break top$/i multichase -N -t 2 -m 4g -n 28800; poweroff" \ ^^ > ./initrd/init > > HOST $ > > HOST $ cat run_microvms.sh > memcgs=3D64 > > run() { > path=3D/sys/fs/cgroup/memcg$1 > > mkdir $path > echo $BASHPID >$path/cgroup.procs And one line here: echo 4000m >$path/memory.min # or the largest size that doesn't cause OOM k= ills > qemu-system-x86_64 -M microvm,accel=3Dkvm -cpu host -smp 2 -m 6g \ > -nographic -kernel /boot/vmlinuz -initrd ./initrd.img \ > -append "console=3DttyS0 loglevel=3D0" > } > > for ((memcg =3D 0; memcg < $memcgs; memcg++)); do > run $memcg & > done > > wait > > Results > =3D=3D=3D=3D=3D=3D=3D > Before [1] After Change > ---------------------------------------------- > Total samples 6824 7237 +6% > > Notes > =3D=3D=3D=3D=3D > [1] "mm: rmap: Don't flush TLB after checking PTE young for page > reference" was included so that the comparison is apples to > Apples. > https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/