From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42305C83003 for ; Fri, 9 Jun 2023 00:59:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6D4E8E000C; Thu, 8 Jun 2023 20:59:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF5A68E0001; Thu, 8 Jun 2023 20:59:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6FC38E000C; Thu, 8 Jun 2023 20:59:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A1ADD8E0001 for ; Thu, 8 Jun 2023 20:59:45 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6CBE71A039A for ; Fri, 9 Jun 2023 00:59:45 +0000 (UTC) X-FDA: 80881401930.09.08B0A5E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf02.hostedemail.com (Postfix) with ESMTP id B258180004 for ; Fri, 9 Jun 2023 00:59:43 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FMN3dhDu; spf=pass (imf02.hostedemail.com: domain of 3fnmCZAYKCKUdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3fnmCZAYKCKUdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=V1wVMmQBD63nDzmLiM9DM9qFnh0ZEBCCD/JU8jyUV47PQmanRAHQfS9csCA9BPJfh3D3PH s8nsL0wZv56132koAB/DpDBAH5Y+/j2HyhWdlxvKm2iV4m3f1nBpzTRd8ZDVLNAc13k9wy SLbdueNg3NGRPSfmDOEJoLdK7r33YM0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272383; a=rsa-sha256; cv=none; b=uuDSwTDWVSyNApA09OwhmPyy4OsnCQBRnEOYDyGsvx3E3P9FOgAthe5OInHQQh6IEn22zZ s/DizQrd/wAZoQERsfIfXu/omkiFxSt8tTn7TRhcOPjof1XGHrLx2s/Jlb/3LgIShP7kho x/Ppd81k8vdG+nJG7zS0puKwNfVgRvA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FMN3dhDu; spf=pass (imf02.hostedemail.com: domain of 3fnmCZAYKCKUdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3fnmCZAYKCKUdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-bac6a453dd5so1731347276.2 for ; Thu, 08 Jun 2023 17:59:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=FMN3dhDuy89W3YL34iGrOcxlPjg1bzMkLLXo1TdyO66RGSvbHlps7gdvHr44ss3LGk MR0VFD1DRCFTNC9tyqK5ReMEOiPx2JNiCbTUTyUv9FMGeK9nCNoAU6WnXWYbBC+Dw27V niwud/7XN7pCGjqFT4B3VS+EBw+uLus4pO0EQ6k6SszjrU1zFcD9yQh/qzHLrmZknfvX XuK4cXbAzi1gsEYKdbrLfMfylLB6JTl2vYXAwWR55yMmpFjT1Lojk1XODC7sstmSmHo4 uXWebCC18fjzj/uapeQiZlzXqVxfdSytOTcShko5dqCAeC6bbLhPv2RLA/d/FJo+pzzy govw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=ip7sNdrNlvHAUebbkMP5ChJsvv36GW8Ff+pgrd2G8yYQgiMTaetmpEmhpY6FBWVLw9 ZD5bXPZdWF8uUT2XqNv7SDq2HpJ+kCBKgaFkERcF9jnNb8tNeMnx4mj2GPhwrzc0XN8c vNejQxiR+3Hzd+Vv+A4kjaihdbBTPXOciHcVBEVh4yL+rBR4wDRatwUvoJ/y9jEZrJMB owziZcWATis6tYwZnY1CBTlmOAdZQWlDsdlzOsvmbw9qLx8hqpdtEmlARjqDTHP1sfwo YYJWCANukC0V8oQ4F9DdJS9M4jaHs4p7Z9wk9t4/tJQ7Q3IFzPk8MFIgiCQ1nDBBkxV/ IcMg== X-Gm-Message-State: AC+VfDwrIIMJX8zXEPXYEbbvSSoeFpFG02sc3WLfJKrLdr+goWDa9PcE SNckc8+VArks6SmLv29Rbt4aqNm1cEg= X-Google-Smtp-Source: ACHHUZ6QOfrw0l3bdpvvlmXUNnamszuXXqEgiHjhlXZ1bthx0t+lBEyPW/rr7RnEuOk5SiS3J1yRldpoYHg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a5b:784:0:b0:bad:99d:f087 with SMTP id b4-20020a5b0784000000b00bad099df087mr697256ybq.6.1686272382836; Thu, 08 Jun 2023 17:59:42 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:39 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005940.42722-1-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/powerpc: memcached benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: B258180004 X-Rspam-User: X-Stat-Signature: uem64sg6snkrdi5abjhm1uids41zhg5u X-Rspamd-Server: rspam03 X-HE-Tag: 1686272383-224685 X-HE-Meta: U2FsdGVkX180gA3HEnIoshZcQabAVQftGotedAzCo3lbXRQ++zEJ+f2nckp5PUtq9/nLJdggqwi/64DdOdXc7AlQ0u37Dv6BwFUACoFR+7SkJjdE9Lupo02n8suuVXzGVvWlhiDSEASCGFZjqr9l6Eb5FSwTUVX5mDEnF1L3ofD1kmdp23IGAwGmx5IjFvdkBfiaZ4NW7IlcFKJ6geLe9a7y0uxxqkWGSDHoCClAMb4OsahQ/kJ9QdS+BTa8Rx2lMODNMlWfCGeNkJl2JIaalKR53Uib0u6ownhZp4hyPQ8zZbC5eA+x2/JYs3AnL0pdp7WSI1dT0TACXoONxfxj/lb8hcGK9MPQkIcQjS5Z1OSgiD7L99vi4igw3PEy4f+MrdBJTyVLhHDTG/Zqf0XTkiLiQghSRvO35Czdsstrl3cUiB2W0516qNjEuyp2cA2Duf4ChhSqNZylgko878/UA0chh5AVirae2H1gppc1bij9cFkRiOqpEU/eQ/bkugDByiTES+nmyvuiIPqiBF2L0AnW00HVro1zR7B/PRS5eErO4GgeqWzI1hcyZxT8cbLq3ROZScbmsJ59AKZ5ul9E5vbKyNTzmhkunBMPwzHVCXL7lIKjaGs3YpVF0U0Ywrtly+r8TfBhLjxW/LJlRZft+KbYnKhmpjmaZbFn/0v6sj2XoXk2zCJrR677uzKdAwecVJuKYQCni+bKF2VJHixbkjeuv3G1Lpbp4+gVZk3liZlo1cWPEvR3XPAXFfwfKkpFzL2kOshd2/+81MlBPq2XZ3zk2ZEyRTRTBzo0AO4jv3EwgA7ZTTNfDNIzogvtjJjsO5Xk1XIwnnBikXzZJ8ZiBRq3ldIMKk9l9Sxd1VrcqsExICXMGiIQFjH4qWQSjlFSkuRuGwi/uDJDUoYTLWC8MetnZo90CwBHg7LUik9sPWfBRMXiEcTWx+eejoNuYsI0kLUV6ZC6YkN+yKsnNL3 3zDTgKOF tRhW1GssDNd9ATeK0owETio/KnVZWcVtq+XRZEVwKxVCw5G1Z/tlCdGKNvdUsib4DW07GSGU/oZoM25GbILqXwj3CmN+CmOlHbNFNUGg1GyvhjT99UJ4+31SMF6Nh9pOz28PzaTVmbUP/2bCHy8CwUjb9ip7sfB/s1wZvenKzrVObPRf93Hya0YuXiD+WTIu4SJNOKBHhH12LcZoV9Kyi9WGF8zRKkG1XDu08pAY8hhD9tsOdrigt1II24GRShhKIPFsqFwRHsYOFJvUoKio+OWDDyvW9KA68jy1GCeXLkxESr1IRz01kA+FHwBfJUe9UMSOXiaRzabn++yRziInmPbQ2Bf0d81xeXjudajqf5boNbgfTdAcd4jc1uT0ILjgPzRJNrzqDu8Z85m8ZsT9nLdt4mVFY6IHLWBUIWIsSdm3c84ZZNhrPDDMIvj2CmNOypFWfAomYNg/nr8R1taUuLl7G91WMI+6c4ypAg8WGUdHxX81CT/a4VqAJztdWDo/PwAVMSPRJyziwP6V3tNv+80Or8K2alXk7aS4IRfP6nxTA2ziW4fKNR1fD1b9xCp9GRQibUStr1mCgW9Czvk5jyC65iT5EsSu+YAx2zzYg6APFQyGS45y5FL0JfsOKFx64jHWwCSiTXOfDL1olgtxlaS4UZhDq2AS8pc/diczLAUfX1OcU4kqZEjNvAsqSO1peuulZM8t4fmcK0vACPIkqLmFt71BSLOGl2CWRVOMsfECgf3N8iQoC9yxXpGIIyIrglqWm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TLDR ==== Memcached achieved 10% more operations per second (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 184 On-line CPU(s) list: 0-183 Model name: POWER9 (raw), altivec supported Model: 2.2 (pvr 004e 1202) Thread(s) per core: 4 Core(s) per socket: 23 Socket(s): 2 CPU max MHz: 3000.0000 CPU min MHz: 2300.0000 Caches (sum of all): L1d: 1.4 MiB (46 instances) L1i: 1.4 MiB (46 instances) L2: 12 MiB (24 instances) L3: 240 MiB (24 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-91 NUMA node1 CPU(s): 92-183 Vulnerabilities: Itlb multihit: Not affected L1tf: Mitigation; RFI Flush, L1D private per thread Mds: Not affected Meltdown: Mitigation; RFI Flush, L1D private per thread Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio) Spectre v1: Mitigation; __user pointer sanitization, ori31 speculation barrier enabled Spectre v2: Mitigation; Indirect branch serialisation (kernel only), Indirect branch cache disabled, Software link stack flush Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-91 node 0 size: 261659 MB node 0 free: 259152 MB node 1 cpus: 92-183 node 1 size: 261713 MB node 1 free: 261076 MB node distances: node 0 1 0: 10 40 1: 40 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" HOST $ uname -a Linux ppc 6.3.0 #1 SMP Sun Jun 4 18:26:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838272 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x0009 HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-ppc64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" GUEST $ cat /etc/memcached.conf ... -t 92 -m 262144 -B binary -s /var/run/memcached/memcached.sock -a 0766 GUEST $ memtier_benchmark -v memtier_benchmark 1.4.0 Copyright (C) 2011-2022 Redis Ltd. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-ppc64 \ -M pseries,accel=kvm,kvm-type=HV -cpu host -smp 92 -m 270g -nographic -nic user \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 1:0 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=P:P \ -n allkeys -d 2000 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 0:1 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=R:R \ -n allkeys --randomize --distinct-client-seed Results ======= Before [1] After Change ------------------------------------------------- Ops/sec 721586.10 800210.12 +10% Avg. Latency 0.12546 0.11260 -10% p50 Latency 0.08700 0.08700 N/C p99 Latency 0.28700 0.24700 -13% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/