From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2036DC7EE29 for ; Fri, 9 Jun 2023 00:59:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAD0A8E000D; Thu, 8 Jun 2023 20:59:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A363F8E0001; Thu, 8 Jun 2023 20:59:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88A9E8E000D; Thu, 8 Jun 2023 20:59:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7307F8E0001 for ; Thu, 8 Jun 2023 20:59:48 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3EE6E402FB for ; Fri, 9 Jun 2023 00:59:48 +0000 (UTC) X-FDA: 80881402056.21.32D1A97 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 70C1118000B for ; Fri, 9 Jun 2023 00:59:46 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=2gQvv8El; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3gXmCZAYKCKggchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3gXmCZAYKCKggchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272386; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ddXgsJixIEMqGA5wc+jEsC/2PCjKU+k92CAoSHzENIQ=; b=psstrFYTADkjYXzVij+E6l8dsxRtrmxK2WSkdx7cZ7mdC8Tuj7PHv5k1VPv92Q7TJ2qMpK apCSZn/hKzhE2YBXAxWv3VYNqvASgbgfM115NSICQHHUylpjzyTkdQJpHDDPVOYko7/zqq qOZR8C3+PEFIY4FAAlXyDHS1RauPReU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=2gQvv8El; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3gXmCZAYKCKggchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3gXmCZAYKCKggchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272386; a=rsa-sha256; cv=none; b=EnsEe+SJd3vigHju6kicP1LQuGfBbt7s4K32n73MmkLsSx4MdlFRY91Ixfo3DSY8UTgFdV rw2PfDh7I+S8Wo9c3NLEVxarr3wglI6POs33vkFTDk7pIGZV1OQWNCEzQ17aMLIDQWDHS9 VPCsBypLQeXUnuRdQCWWfEI1leCRdQE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-568fa455b8fso16074317b3.3 for ; Thu, 08 Jun 2023 17:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272385; x=1688864385; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ddXgsJixIEMqGA5wc+jEsC/2PCjKU+k92CAoSHzENIQ=; b=2gQvv8Elto4XccdRwaHPDqIj9nLGBVTBNmb86njoPvqhsd0tjtN6HCelUzSemN7wyY 71LauLyB1lnldzxzQTkfe6nQZecdjOxXtr/4StosXGI1CrlQDTJck4wzTXo8dKjjpnV4 D20udOm2zFIzHEQ51hLEe5UarHwCpaC6lXinX2YwMS98bkU6dZgz2pBUAz+mzsFIorM8 xdmbpgNfTyxvCAzBTy+SieKUu3Kw2LKLwSnetOswWodGDZgbdEkzkIsg0JKEhFrOZU8p T4WBKVoS8Yje74KiF7RZvIKnuSmaC0kPLDwvUqdU+Fm/Ozb38xa28j2ttI0hrtP0j54l z55w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272385; x=1688864385; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ddXgsJixIEMqGA5wc+jEsC/2PCjKU+k92CAoSHzENIQ=; b=HWIPbKxQdjSMluyyO99UwI6oCUIczsSZe33CyISa1ed6eTzBUSBFXoe6tjyn5BgSzc j1w/jMsRRj9C+HnDvUd3qv1/KaQVXb3CPARPeGPKfbqLB6NPCLRMpnkZfFSSgBjCkRKm INi99T40jm/KF0BnNA/ZTyAAyDmaawcuXVanttXFp7h6xrc/f62D0ZTq5lC2IBrKf2uP UdYv4/MVC4pWrLJeYb5n0mw33DLW9Y7l9khWfaIiWbysh+g04pgNpeogZdv0cn7zj9vk 8JA/3qMevQ3uxQpGHIqYLE7q4WOMSonH+cPVGFtpl8lRR7+LDuyLHN9vfa+WSMSyRKoL UDbA== X-Gm-Message-State: AC+VfDxbYjKn/BHhBnDctwfjG5UslES7nAuzEkdcrq4tNwf/+3TYoOmz 6lukWIopQ5xsW64zviYeq+fL1VtsILI= X-Google-Smtp-Source: ACHHUZ48Ctq25/jOWo2nOrUYiTkPu+9cUUGT9KZFEWhgs36Jat11A7iC8UGGnM5T4c2tHVmOSHQ7ebLSwlw= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a05:690c:702:b0:565:ebd4:304d with SMTP id bs2-20020a05690c070200b00565ebd4304dmr731769ywb.4.1686272385574; Thu, 08 Jun 2023 17:59:45 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:43 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005943.43041-1-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/x86: multichase benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 70C1118000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: d48gkuwfumtef44femsc3t986xucidz4 X-HE-Tag: 1686272386-251854 X-HE-Meta: U2FsdGVkX18/os+7N1Uy7Xehb5Ix9sLhYvKPH/H2UGaWv1pqy5/Z9WBRT+C/W6bs60uMIppBsrUATzitmD7AYv5dc4rZSn431I5zitc6uAny0bhQPJOtgJgp6IHeNj1xWZ9x9YceiZ6ccvuLgOv0NEc9i8xE0FjZGIJT3OiuYCpsxhjjW5Oy7zfiMe14U3FU9ldcPN3lUGXkuI+Yd6oA0nXRnTovc7YCk99Z//AgqEXwaSD6MFYn+9z/ZWaCoAr70RYNQNbk57vWPA/vqtxPruq7FXwX7SFiXbpyNVsIAfaRnVm3+AfLimt/k788mXyJZiY+r73Smi/mQ/GYiGdZUArn6qHesVngc+HxlIaO3DSl3np159IIsKR7t6P5IqpfXGask1nYCMPK0jNMBNbVDzUveUUWZ8vfjr6ujKgJtNW2ys0UxbDT2NiXVmYc68mquIXm/dkVP7/V5uelm/tbFI6DT4zb2ypOdn6+SJVajCK/uY5WU9qh9O4WmSZC7zFA8QWtl078mNXFAwY3PVQb+1hnzam0lC3db/1OWyvguqdwZ27aoVgnd8QG0IJKsopcOZOGwFD5vDrFy2Ra/c8JuiWT7Lpyx7ZDq1D/04kpJbJ4CP4+r06VslltN6z69+Vhex5fOFOvikiTKQ512hGX4FsXfbygj1XsT2sAR/GuJtXvXO49JJMx8Wj8kxgD0XW15J41EvjzVZRrrgbiUUX6BEJQExRh/iE9WmxXNWc8xRfiDVZlWRwY/33r/LGAqwHEqDsIUkq9+LuKYYhxeK3+DxvICqCm+3C3ohDiBQ0ep0EBJrT+lGhQ+CFgYhTgW6EIihITNcvBCI6/PaAXEQlUcheI3QWQfHOv1x9Lon7m8bz8QWFQiLlbzxJAOjNdyVFY2ZyTj2lQ0kuaEtGhxY2v8DNIu7biRJ0krZwM5mevzjJIUYvWSzgQ/ALvizaDm2B1d1MbpniOqNqzBcgqbs0 qt5dfnM+ 1BLaAnhLtELJ5lz/h3HQkAfZLC7PYv4j+b1S6qR17H/xCPu8LyOHmDJnYWDFJucgEQej8mTR3G7tDYLIgWH9Ijt60olsvbppDdVLYVduyuoG4NFHoEaE1RGN/I9WBSgwJNXpk0rTDNauz7MdJTVg8LfsLwZYH8gwKPxepSKaVdlXEFcCdfPgkd3of0aRyDs0WiADGyFmKethqtbNjBYU4C1O8b9w6R4CMNDxSVri+IUZsF0l36SPzxy8ea1NKNXTw5wKHPU25HjByTcxamxUIOcsz0QddBhMDYsnc+GLq4H89oaSNEpLi9XwrPfqgjqdPsW8lWpqGuwj/id5DIQyFvvtttLXfxddoZov313KKmK4+3Qs6fNArAhuB3FLbYuQMtRwFak0qC7sqRNS4cvphARY/CYHatzmYqLNbkCbG2G1/nmHle3ZyubJ13sD5qKHZcUU1yUzjJWVSlWKYXjxeV/lJmhl2YQPMpsCIgomMhEDrAFq7+ke4DCbsSLDC4BuUppOHHPd7dTGVuXWHr7i99Jm5Y2TuvTinnDhfysIZd1lC3jfSsi6BX+BbCjPHZF3LeOvYeUwNT8GKBwQeRs/DbV4Yw779dO4ZneiTGwW8Qq1RB7sYS/OSjtli73JaidpePcU5FIaehx+TE2MfajuqwSRtv5YrN0rpwaJp1Rfz4twbMUx6C8GT2OFheg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TLDR ==== Multichase in 64 microVMs achieved 6% more total samples (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: AuthenticAMD Model name: AMD Ryzen Threadripper PRO 3995WX 64-Cores CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 1 Stepping: 0 Frequency boost: disabled CPU max MHz: 4308.3979 CPU min MHz: 2200.0000 BogoMIPS: 5390.20 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ... Virtualization features: Virtualization: AMD-V Caches (sum of all): L1d: 2 MiB (64 instances) L1i: 2 MiB (64 instances) L2: 32 MiB (64 instances) L3: 256 MiB (16 instances) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-127 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 1 nodes (0) node 0 cpus: 0-127 node 0 size: 257542 MB node 0 free: 224855 MB node distances: node 0 0: 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" HOST $ uname -a Linux x86 6.4.0-rc5+ #1 SMP PREEMPT_DYNAMIC Wed Jun 7 22:17:47 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838356 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x000f HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] Procedure ========= HOST $ git clone https://github.com/google/multichase HOST $ HOST $ HOST $ cp multichase/multichase ./initrd/bin/ HOST $ sed -i \ "/^maybe_break top$/i multichase -t 2 -m 4g -n 28800; poweroff" \ ./initrd/init HOST $ HOST $ cat run_microvms.sh memcgs=64 run() { path=/sys/fs/cgroup/memcg$1 mkdir $path echo $BASHPID >$path/cgroup.procs qemu-system-x86_64 -M microvm,accel=kvm -cpu host -smp 2 -m 6g \ -nographic -kernel /boot/vmlinuz -initrd ./initrd.img \ -append "console=ttyS0 loglevel=0" } for ((memcg = 0; memcg < $memcgs; memcg++)); do run $memcg & done wait Results ======= Before [1] After Change ---------------------------------------------- Total samples 6824 7237 +6% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/