From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E364EB64D7 for ; Sun, 18 Jun 2023 20:11:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0AC18D0002; Sun, 18 Jun 2023 16:11:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBB2A8D0001; Sun, 18 Jun 2023 16:11:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B82E88D0002; Sun, 18 Jun 2023 16:11:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A9A1F8D0001 for ; Sun, 18 Jun 2023 16:11:50 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7289F1C7BD0 for ; Sun, 18 Jun 2023 20:11:50 +0000 (UTC) X-FDA: 80916964380.23.3747194 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf03.hostedemail.com (Postfix) with ESMTP id AE81A20011 for ; Sun, 18 Jun 2023 20:11:48 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BI4IVSXI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687119108; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mpfISiVhGBFvYu3AQzytJ9eyp0qWbNPzT0F/TqGcEF0=; b=Ch1JVGEe3r6o+IZscSV/MJH2/trrdlGCeU/BdU/TJON52P1HxepIUCEwbpAf7G2d+jRSQd xXBNx0fgkZ8GONMqPGHS3tng5cw9hb7XSPL/0bZkFpawD78YqoJqq/xo9SZQYI4k/gbBIY 9Y9LdYfvr9gx3ZSNPhdNPpKBbxtgb2Y= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BI4IVSXI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687119108; a=rsa-sha256; cv=none; b=8ECaeWmNRfMEyd6cjHMPq0+GNAiA7cAhZkOPxGuiX5K4Eula632XU6Q2LpJ504dEDdUUJK G1JSxM7ZAwoGGI+l/TJryBGD3NAOL/WG/Yp4tpZZEXK3niDEzDLfio4D7Ozme9mOvYV1DN yEWQrXzJTO/gNkdJDagrjQqmpl9wGmI= Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-3fddc620055so206331cf.1 for ; Sun, 18 Jun 2023 13:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687119108; x=1689711108; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=mpfISiVhGBFvYu3AQzytJ9eyp0qWbNPzT0F/TqGcEF0=; b=BI4IVSXI0OmI7HSHsHlrapOhxlYQr88irtqXtejFQDdwi0ENBjBoT+XLb8GlVX5ye2 2RKZIVa3SgmrzIbhLz9gPTVxEaHp2aLuGzJABn2V8qcyj1r5n7W74CILO0agM4yRBdqp 1RaptWkZvNHDOFx+VjmcCLiDP8C+KTiX8SbGmrEKzvHmrSeItS1P0Xmdw5h0oqtFiDy4 AOhW0Kb490uZt+xyTNM54SZyWZrBQ/OOSALJEaw5URHLWQ5/4/9LTWbU9KUB5nzr8SkC EiNHGrA/DNhBI3N/ps22/XwwMptyPwh1WFUc37QSRtqZWWbsx8o8/qqFNZmnqhSquaII BNYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687119108; x=1689711108; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mpfISiVhGBFvYu3AQzytJ9eyp0qWbNPzT0F/TqGcEF0=; b=ECK+prMplIxHkl2roc4UUjhJeysfcERrwfvnnF9ca9NOPTTkK6NANA2gnrAaAWwZU0 GWLW4uzXn/aQB2za/NoLHs8ScYQnZ0jasaF+CkiZ+cy0cr9BO7F+WOewcjCxpjig7jqt HPMh/pM7Aps/PfhmBh6TfON7+rIXUywDVwWt6siCyEo0SJFhX2mjGgLnYfkSTuWUUiXc 7A2lq8UmBhbb0Ldzymxwsx+9X2YpNqHg9v+1IIv2nPLLKRj6D2mwgHLvFCDAlLF9g3kY 7GqZPMjvyglyXNqIk5GniE1GiN0qTWFRS+mNNxEdecA6nmN04IJbr5pEdIA1TFGJ973e KPSw== X-Gm-Message-State: AC+VfDxZ6c9EbVxu+5gMYFMcP1bBDTQ6u+jev8e71XwyU+qczzUFIxuI Edf78mBWOfMGrWU5fn8wTfLQs1q+xx/FMPuH8WmN7g== X-Google-Smtp-Source: ACHHUZ7+FzP5AfTCxoLANN0O1kovXzprxLRO2NDRfP+79+jkGgGwKBzMX+UybO5p4uMqphT40cGSU06Ta5mNNZ2MZbY= X-Received: by 2002:a05:622a:85:b0:3f8:5b2:aeed with SMTP id o5-20020a05622a008500b003f805b2aeedmr773688qtw.21.1687119107794; Sun, 18 Jun 2023 13:11:47 -0700 (PDT) MIME-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> <20230609005935.42390-1-yuzhao@google.com> <873530okh0.wl-maz@kernel.org> In-Reply-To: <873530okh0.wl-maz@kernel.org> From: Yu Zhao Date: Sun, 18 Jun 2023 14:11:11 -0600 Message-ID: Subject: Re: kvm/arm64: Spark benchmark To: Marc Zyngier Cc: Andrew Morton , Paolo Bonzini , Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AE81A20011 X-Stat-Signature: 7xjmfgifxo5tjanbtrcsp9qhb5o3qa31 X-HE-Tag: 1687119108-558040 X-HE-Meta: U2FsdGVkX18zzQveNKXt8XjSvuUtC3wm0mqpSnxgk2j+JLYXL5kXm779hybbps3GWkzakdL4XzgeFYVkTggB6/oyQvW0HAK8bPUNP/hA5j6Uzkz9XZrt3dl0oW/HVVmec2ImwSlglKTjuzGo4DtbrPHroXW8Xvbmj9HLR1vNdvtVyC0blGiwTcR9skVvZrhSf3ICN62nI8/flkS6mcLW+Nvb0afgnJ+VtIKhqxP2K95SphxmBOy5MojgcNDtmXPf6VqABudCjkKuMyUuDmmq8fmp220997nKgb6jCXrFZsJ5SedvhYqGv5HOmc6tWad22tMsiJtX57nHnpm7Uy7NDjd7lzFhapQ3DG1H9F/JUezoZ93oC5tLDv04J2e+roedqQTiOTKzazlg9TH1shqjcnAYL7BozjK4Tz3P3IBmcXsX0ZisVTBlkJy0Qv+6iB/RpbEvhZZNNJLs+laGsAri0tslhNGeZMYeVqcHTCm8+FvJadf7cy1M+Z1WU4cpI5yMhAQ3fZf2WTO7sciGpJCjF+obPdIW93RpSpidkLZFBEbgZz7Lqzxu8kcMiIXICzKME23Tuk0sed69RIxKW2QXNWwFmmctfpMzZPXsc+Vb37vrLZQ6359IErxiLH2SypNvhekqopxFduCQaTBbYBPBVgZ3Ge4lbbg71DsCU1jOQHsKJ6B8Kg5zTHKDcJlIjzlXK5nuYofVmWcWLkqTd416ye28LO8G61MNjNuxe1yqIT+xqcLRa2vcZNWKrgfiA+ZRqWI6HBAy3U4DaKxGZPvSvEDl8AX2yQrvvsj6Y8+uIwmL51toB7M1RkeXiuMMmQM7mIZEohHFH76A8tJmyDla10KTsOEX6WlDwDMjNqb+NL+Dqsu3G3a+0dHfOO5Py1UV1Fo9vHcqeZBvPa6dt8c98SjrEkJZQvy+eoPPeJ2Rc8uKwZDtB7IkF2aJeQgjZPU8gK5CHBzs9GugALaXv5A FTCBB6ej Lut0RKwBhq85RtCcMajdDGidBT/+WNUj2CT3+YispG8q3GhLVBVWe6mgcvyeYJDMF7pnljZnlKpZT4kwPIAxNQXPs0khv/WlVxBbxadtWGWnlBVs3BNV8RT6fKOt85ibWClXMzRyztzbmDhEUyEG1neN4JQT20GO7yJBj0tKEWs+hEAPAvXi1pEeU4zwn5gppYVX5c9NauRzwAOQhl1tUcZ0DFMllT/nt1VjuCdUxMW1QygdrcE8Uzvzcj6q2lTPQ1r/wYadOB92SYR7MEZa7gxf2YsVTNQWHGieAjg6B+n8C7MIAhta1BsceWQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jun 9, 2023 at 7:04=E2=80=AFAM Marc Zyngier wrote: > > On Fri, 09 Jun 2023 01:59:35 +0100, > Yu Zhao wrote: > > > > TLDR > > =3D=3D=3D=3D > > Apache Spark spent 12% less time sorting four billion random integers t= wenty times (in ~4 hours) after this patchset [1]. > > Why are the 3 architectures you have considered being evaluated with 3 > different benchmarks? I was hoping people having special interests in different archs might try to reproduce the benchmarks that I didn't report (but did cover) and see what happens. > I am not suspecting you to have cherry-picked > the best results I'm generally very conservative when reporting *synthetic* results. For example, the same memcached benchmark used on powerpc yielded >50% improvement on aarch64, because the default Ubuntu Kconfig uses 64KB base page size for powerpc but 4KB for aarch64. (Before the series, the reclaim (swap) path takes kvm->mmu_lock for *write* on O(nr of all pages to consider); after the series, it becomes O(actual nr of pages to swap), which is <10% given how the benchmark was set up.) Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency ------------------------------------------------------------------------ Before 639511.40 0.09940 0.04700 0.27100 22.52700 After 974184.60 0.06471 0.04700 0.15900 3.75900 > but I'd really like to see a variety of benchmarks > that exercise this stuff differently. I'd be happy to try other synthetic workloads that people think that are relatively representative. Also, I've backported the series and started an A/B experiment involving ~1 million devices (real-world workloads). We should have the preliminary results by the time I post the next version.