From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7896EC7EE23 for ; Fri, 9 Jun 2023 00:59:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1EA38E000B; Thu, 8 Jun 2023 20:59:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA7398E0001; Thu, 8 Jun 2023 20:59:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B22F08E000B; Thu, 8 Jun 2023 20:59:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9B0E28E0001 for ; Thu, 8 Jun 2023 20:59:42 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6754EAF007 for ; Fri, 9 Jun 2023 00:59:42 +0000 (UTC) X-FDA: 80881401804.01.8469890 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 943D7180011 for ; Fri, 9 Jun 2023 00:59:40 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BDzc1aMR; spf=pass (imf16.hostedemail.com: domain of 3e3mCZAYKCKIaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3e3mCZAYKCKIaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0D7F9pDbG9FXqs6+iQJrvOEyVDkJYxW5kzmn8NxTXSM=; b=GqxJBVZfVzqvGeN0k06qBu+4IzMnNb5sdv028o/SbeqkzEFANAwpztna2wMgAXFtexj/Kn LVVP7sDPz/JgnLVXn/PxwdhqMscFJ2AcemENyRKyLvqYxuT2rdUMxpFgytDbw36Ll4re61 1gokV8mTveD5J/KiI8bPXdTTMUC82/c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272380; a=rsa-sha256; cv=none; b=Zfpt7T4wRf9ZTuo1XyUxcFGDgMsBaz85TKDsB5z5X6uqG+Kop8fYcGxPcXNw8VzrsplZiZ OoL+d8P51m1tWoLh8d5uXIrADvDjc1/FMnTu0A0KzpHKEaC7lk+Awyb+Is+HmhQqfKF8kn QA1ZD14+35NbSmHEqWVdczONZRyWNe8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BDzc1aMR; spf=pass (imf16.hostedemail.com: domain of 3e3mCZAYKCKIaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3e3mCZAYKCKIaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5618857518dso16260087b3.2 for ; Thu, 08 Jun 2023 17:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272379; x=1688864379; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0D7F9pDbG9FXqs6+iQJrvOEyVDkJYxW5kzmn8NxTXSM=; b=BDzc1aMRHWXMiLk4E+pANQvv85EBP8Fbmqip6g4XnCghOWEy+T3hHMA98y7gu3MrzJ m9ZUmOHYqjH9lz2cLAdIslB62deMkVm5FAytrF1l0ADpuk18V34OtWN8S0hJyOMsYZKG DZ3FeQwAgU/HrpcxXBWW2H8tt/UuIX4ypLlLVKfqSh3rXART4r4Tw/QKjnEtg7JS8CHS CWcIW0q6hoRoggptEjKDZYq373BjG6IuT0/lhqlLaPYC66B04C52gE09wIc5hlW88owA Gr5feq82Y0tlKosId/vUE/V228jTsXcOxUsj2CaGmnAfl6wsi8uq3/SNbii3CwrF5QN8 PobQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272379; x=1688864379; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0D7F9pDbG9FXqs6+iQJrvOEyVDkJYxW5kzmn8NxTXSM=; b=ig1I0vnEx71I3zIKoTuLqimg6eIGzTuU23hwQ0kaTvCMngoI+191uuQ3BuVTK2ZiJl BcBAzTgQPjb01QkCZdLPEslTvnDh7eW6uK8LQOIbXXjusi0vi0Jkm6Psd1Cy3wTZNWxU HXVrl7kiqHn4N4uemu4u8+SwYtgTH8XOl97j6+qZhHn/1MSBTSyH4AOwDfkNCgvnOZjU VJ6XCm+YZB/MSY/In+Wt8Q5uKINWLWIuWgfj12yov4BDYhZlWZvVFfkTMf5ukgt1Elhe 3sYuQ10ufx4F+0o/gr/R4GoqJRvwqBI7PMLFOohYGSOtDcecCDnCD4BL8sw9fURMEE8L OBnQ== X-Gm-Message-State: AC+VfDweJNlVQfD63uPOLllt7zpDBeQDG/GYB7U4VJ7RW4UtHF1kingy LGFeYZqIqmzQUKZ0XtSKTNXCVDgm21w= X-Google-Smtp-Source: ACHHUZ4+oKNZvW2k4pUHaG+wjpqVYAXO5ZmY9/JryBcaWXlbxmXfyr05oUuxiW1jF2Vui38QPi/ByPbILxk= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a81:ae60:0:b0:565:e712:422c with SMTP id g32-20020a81ae60000000b00565e712422cmr809562ywk.1.1686272379675; Thu, 08 Jun 2023 17:59:39 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:35 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005935.42390-1-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/arm64: Spark benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 943D7180011 X-Rspam-User: X-Stat-Signature: tnb4nendjha6j91bn87cp3g74xcme9mt X-Rspamd-Server: rspam03 X-HE-Tag: 1686272380-192025 X-HE-Meta: U2FsdGVkX1+fa0yL7aq/idWTQLz1X2cKOjsLkRmVaeQCPzdkthZiEzXd7187NRBUA7SzJgfPJDQ84eXa1a6CJ75UrmErTIOLcpW8Klatoo3IMtnr0A53PcbEBpMl6ZZfcE4b0i2k/TraoKI6nwrch/xRYtLnbU6JzRHvMnEkHRAFfrN/xOzppQjgnRFJxVa3ntKJfnI39uLxgKhlzYVTTxgDbRTTOKPX/ARqS9LQPHHknXfUbKIpSbaNQX7CQnAyt4tK/LfYdI5sa8pQusMQTk38CetBx+awXLsqaW4MA8lEGBVwR8jkUZ+nDLB9Ryx0qI9cyfd+uspQv9zPqESgM8oJ/yqvLg0i++7fP1kB8WyB+65gcuR26e9LHYfElsV2HsveDU85S7tAVvICpWw8LJQevDO7Dtyip31Eud56grzIA2dqBMZ7t5ul4fqDg6Nhk2jRBqv1P7oQ8nu7do0uwEcjllRo+zM/5LXLk1Ptm5IsMkcH29FpeMtmaGJAyyHWu9nxQuI3cw8bW4jBAfdIgSCaHLsIR5wsmhUewwdbjBZZ+8/KlGLU7tR8aZlYx5/9xqyX+1gtCzfu/W0LFbbAgeprCqgMQ//FOVrTPvslJb1lltjLuRNF1ikY0wHbXA+qGu0LzpPimu+lAfgZzM/hVpSbIvtZZybG0aSBQAPPANtNNHcVIVyawZOgtu0DHlqX+xaq12ySeK7yL4wwwFmVhRdG4box7YEgWA+3jcKZPb/1Pe/KxW+niraNT6ZJ1K2+LBfvBQ4utscN9EpCQnpSst6xGS3pOh8vT+ZovF45zsLrpf2zCNv4OAQhAuOEu7aYW2vg2euYCT/NZfWHbUCFJJV4fz3HAhbdhc0RNWgry4TsijBHDk6a6ZTwfP+AjIyptjhLzSUBhv2/vo6Kkc1fdowkiz5qbWzVY5YrNBs4pmjyoWuTwL5Y1tQng5tAMWVU4Zz5p0y52n7Ppukbri/ KibwxJ31 fAWJ0w5qkyaBcjCXaeBbdVpQePgq132whBH2L4wBrMoXvtmn/0iCEkNgBR+f7JqK4IbHxgv4ilC+1Rz2wq4sNDW7uB4w7HvJKTKwQa2kkV4I/Gs3OGI//PiU3I4vRcVvWsUKgBdisJNVbQwhE1vrix98Jzyz4ylUtBbpzuUmiEaXOuqmcilbGjHNYr/9EkXARHOOEVA197ZE0/Vf175779P33uYpLXDqNb2cEw3bpJ1pzPMFw3SRp04KFe3jseTviOGAbaxZIIduBrcYtZFs3uRurSeqm6hKN0W7nCoMnH7VVR5vJWxSiZXU2AW+HTwCBXGUNmPkdoGooJW3xu0ZY93qkdo7PK/TB8zm8lCjtnb3mZPURsTjYoK1WmBLfXHAutqlRibqj1v0gaIbFW711THR7NqQ/GsQPxyuQbZLNu+c+SI4U26/tdf3vItB84aERwW1aZZ+IHjWsVFoewhtEf3UfU8rCJ+iiY8Tyik0AEiBC8o0uei3XKPuSIKKNWkh8ZmqMWoxaNl4vZ1NnkD2DZQaw4Vs/UV4VvIf9taLoT5orDKvE2n4ja0ETAtH9hEMzC594GY4rBAf2aJ0WrJbmB857Pc9Lm5GE9i623+zPvt/tQNXhVF2lp370wVlWoJT5rqRFmwEQT4rPUKZpLMQRcDflipiRb2ogM3rVayrdJ2JQwuUsT6D2qNGMiu7laR/ExQqH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TLDR ==== Apache Spark spent 12% less time sorting four billion random integers twenty times (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: ARM Model name: Neoverse-N1 Model: 1 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 2 Stepping: r3p1 Frequency boost: disabled CPU max MHz: 2800.0000 CPU min MHz: 1000.0000 BogoMIPS: 50.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs Caches (sum of all): L1d: 8 MiB (128 instances) L1i: 8 MiB (128 instances) L2: 128 MiB (128 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-63 NUMA node1 CPU(s): 64-127 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, BHB Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-63 node 0 size: 257730 MB node 0 free: 1447 MB node 1 cpus: 64-127 node 1 size: 256877 MB node 1 free: 256093 MB node distances: node 0 1 0: 10 20 1: 20 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" HOST $ uname -a Linux arm 6.4.0-rc4 #1 SMP Sat Jun 3 05:30:06 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838356 116922112 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x000b HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-aarch64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS" GUEST $ java --version openjdk 17.0.7 2023-04-18 OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu122.04.2) OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu122.04.2, mixed mode, sharing) GUEST $ spark-shell --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.4.0 /_/ Using Scala version 2.12.17, OpenJDK 64-Bit Server VM, 17.0.7 Branch HEAD Compiled by user xinrong.meng on 2023-04-07T02:18:01Z Revision 87a5442f7ed96b11051d8a9333476d080054e5a0 Url https://github.com/apache/spark Type --help for more information. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-aarch64 \ -M virt,accel=kvm -cpu host -smp 64 -m 300g -nographic -nic user \ -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ cat gen.scala import java.io._ import scala.collection.mutable.ArrayBuffer object GenData { def main(args: Array[String]): Unit = { val file = new File("/dev/shm/dataset.txt") val writer = new BufferedWriter(new FileWriter(file)) val buf = ArrayBuffer(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L) for(_ <- 0 until 400000000) { for (i <- 0 until 10) { buf.update(i, scala.util.Random.nextLong()) } writer.write(s"${buf.mkString(",")}\n") } writer.close() } } GenData.main(Array()) GUEST $ cat sort.scala import java.time.temporal.ChronoUnit import org.apache.spark.sql.SparkSession object SparkSort { def main(args: Array[String]): Unit = { val spark = SparkSession.builder().getOrCreate() val file = sc.textFile("/dev/shm/dataset.txt", 64) val results = file.flatMap(_.split(",")).map(x => (x, 1)).sortByKey().takeOrdered(10) results.foreach(println) spark.stop() } } SparkSort.main(Array()) GUEST $ cat run_spark.sh export SPARK_LOCAL_DIRS=/dev/shm/ spark-shell