From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 597FCC3DA4A for ; Wed, 14 Aug 2024 22:00:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF1386B0099; Wed, 14 Aug 2024 18:00:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA1BC6B009B; Wed, 14 Aug 2024 18:00:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA83D6B00AD; Wed, 14 Aug 2024 18:00:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9B2C76B0099 for ; Wed, 14 Aug 2024 18:00:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5083B1C4DC8 for ; Wed, 14 Aug 2024 22:00:47 +0000 (UTC) X-FDA: 82452221334.06.6E141FE Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 723F71A0023 for ; Wed, 14 Aug 2024 22:00:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xXP1ec8J; spf=pass (imf19.hostedemail.com: domain of 3Cym9ZgYKCI4Aws51uy66y3w.u64305CF-442Dsu2.69y@flex--seanjc.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3Cym9ZgYKCI4Aws51uy66y3w.u64305CF-442Dsu2.69y@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723672774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=H5D3RXK29Yyt3M5CRguL/n/uC+sGzyd4zhpQCsEWaFIuRzvlpfiFhYIDvSCXduy/SfsyRH 8ykLPV7ogQL+Aveh76h3hzi5DBHvrloYQE44kHu7o3mgB8ASgqHP02ZJJd1Fos6q4RYvl8 oUbrms5bfv2AXQSISrGvAb/86pZ0dvs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723672774; a=rsa-sha256; cv=none; b=y9sy8E3E3OFhzoIBbx6QCGs/EKbEwihVgY84oEhbdqN1P3677SPHssdHFtkAvL7kiw9qAl iFwMokOo+kTza3myqkfS/LSJOiShiVT0n4wukl97O7fk+Gzio80utv++4M/QubMXKpt6p2 UIKiwSck1iX5AnEXsB5+tSfKTyN3WzA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xXP1ec8J; spf=pass (imf19.hostedemail.com: domain of 3Cym9ZgYKCI4Aws51uy66y3w.u64305CF-442Dsu2.69y@flex--seanjc.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3Cym9ZgYKCI4Aws51uy66y3w.u64305CF-442Dsu2.69y@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-201f0280971so1185515ad.2 for ; Wed, 14 Aug 2024 15:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723672844; x=1724277644; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=xXP1ec8JUQtT8PhJA5fcBT+aHAn6r9qcVJ+BJAYmjeza9Q92NGV6C9oCWp+POvy0Yt o3MRdFFVLfkxn+tV8BVkgicyqvgvOqfT4E/M/hCVz8b6njVU60liy0aoeVc3lw2PSclV bJv4XOwR37Ghx+FCsz7InuhOrgdaOI5oPSj0m08nzRbRbP/Bn8uVrbvxZy7BPCKNviQb J5yTF3H28l5WB68cSS+Q8nic3tqqchaK0v5QlohNGdaTH+csCH1SfvYrbwldpiFOkQ0K uDkie5ZHhMTyP2FY4lrstklrLXdBPTkjvWbZ9b6CTyGK1x/erNoGdOL441/nspvwi42L ofYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723672844; x=1724277644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=VVe1yJUpkR8ILxLf06XXNMEtV/r/LFSC/AMYlbQPdhgfdrOyPFvAx8gc3SZy2ZWHHa PulEL/Mz3kGNiAn9xZdtgataQiPfUO/6xgUrbhH64B3G+FkZurFypcCJ9PIwQdLGhe8H ED+OGJ8T0JA6NVBLNAkSR2ZB9jYmwN1CmqIiny32Y1eISbRdGF0bNfAIKt6AEXKItKQB NDE0k2Oglyi2X0hPUVijzayh9BXt/iNS2o0KuUR40FuLfIqm5U0VA9rl0IO1ged1oPu2 M3IoW4MgrARq1sftMAvG5uW81GZ1VbFfxH4i+PvwO4S7qM91ffZ/kwrkKdbDfcvQ9QNH xvPA== X-Forwarded-Encrypted: i=1; AJvYcCXJOBS3I/kQfMU/mTisfoLpNyBR7OSDcDptMV/gBLEGMLZjg705Rno7+ZMMwhyMxDXSBKF/bsXNGQpvWy1ieZo9vuU= X-Gm-Message-State: AOJu0YzP0rhjLpm5ya5/Y8DQKI6a2S5xNKvl+Da6USXcZ5DmXRkAc96l ua2UyNywbcMBlv+nWhn6K7dJTlxlvsJD/hzuvGar9KPNdHHxIb6FKVg++lcTN83iizETvvH0uyu tfQ== X-Google-Smtp-Source: AGHT+IELSmNcIfcWqD1y6wfROgLi5On6yYa9ztLE8WB0+uhpire+X0WHbKhlLbGeYLBF6mQ5SB34xpmbxzM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:da8b:b0:1fd:6529:7443 with SMTP id d9443c01a7336-201d64b4f7bmr3318245ad.11.1723672843639; Wed, 14 Aug 2024 15:00:43 -0700 (PDT) Date: Wed, 14 Aug 2024 15:00:42 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240809160909.1023470-1-peterx@redhat.com> <20240814123715.GB2032816@nvidia.com> <20240814144307.GP2032816@nvidia.com> Message-ID: Subject: Re: [PATCH 00/19] mm: Support huge pfnmaps From: Sean Christopherson To: Jason Gunthorpe Cc: Peter Xu , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Oliver Upton , Marc Zyngier Content-Type: text/plain; charset="us-ascii" X-Rspamd-Queue-Id: 723F71A0023 X-Stat-Signature: wycwjc487mshmn83uqxdozq1iytaqkzy X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723672845-884999 X-HE-Meta: U2FsdGVkX1/EtnsGE43rdMCQGJEV6c4c1b93jyR1GxRTGotmaT3rsxQVVnbuMETpKQIRlZ9yP7q4PYEyRNcNSpBFkaw+LbN1sspIhT0Afc/wWh4DDXysMQsQ/zTx7K968eykLTMnC6Y2YrCYm/j0NNEuhi4Kur9Rr7FIcQETsWmkwUN2TZ7zZsxbwA+lr2tNPTo6KKmek/lD2a9bDCf+R1ChaJb+nJ2DbX7U4gIT0yhfBRwRhephRVtpy2JNLVrKtmQKTJRmWOqm9gbGX8pk1UmbuCzmZ8OYpaAy59f6gYygLEipfAnJcPzN0OQXTbcvIEU3AXG8TVklHNOBFuahN9M5S5ptAWUDF8GOuxES4mGTiC0mOH/vwb4my6a7RUN1KcocFuEGynY4Fdds3ZToccjv+fsruX6++Teh4tr/35oOaTzKQq1zMdnXgDBqH5vIEV3MW+nzFrKQ9dr+3dJ7D9fRk72gmfOi0k+FJEFYoxrThtBlMS1O2I8gDvHI3LbGwEKwuwOgP2PdDyI+TJn7T7R4mt2ROqibB/i0QQb9rXFdBPOzjx2gfYayLSh1fAFhlpp9yNnx1ZUod5MRAwVoVo3v+qvCHHtj7wbONPq9Gc5ojF5I8xSGAeTag7vx9HvrhIuSx6q8drzYPOuF9Fs1CWkAhju9qQcyFopkNfKa9GFjXYeHv66BP97LdOfaGJ7gqw/aTGLyyoDlQX9N1B2cUWdV8s9EkOl9i2m3xCGdMdMjwBjf7K7a2oUAv+gGO+Q1UIib6J4JICCRvDLdi0wGtR4R+3EHJUiO3qYQZDg1h6RtRD/VQ62wq4tLtLJFIgn7sQ7u/0UIwGBycenIqXErvpyNlBBrP6QPvJjUARo5BXP1QIvq5JEKz/faEloVLkfFVJ985z9lZkGMPziUXtoIfr/YBdT4TeTE464FhsV7/6vFzCDxkoFKAOhi35OvMfw+EXafvkF62Kw7Evy3AC7 haQ0k3FZ YTqdVZQcr1R0/D6UHdyG/fmqVZhfdHjirVuaBcUEt4hs9Xp85ayaa2zdqJCecbw3w6FoyqviZOfFshaIV79DF85GuyH6wsOfuFqkLeRzrl0QvIcPPVKPOcqQq69evb6r5T/jroCsJE3AVJA/vlixAtpIjyJrPbpbS/qIej18dj/+3k/MjPCxRyo2ytgxCuKWE54X/YM8RQBTQLG+xh9GQ/tMitR5p+MQcQT0c3xk/PLNgrKv3b/YNxOypWBod4I68fKlO/dkR+BC/Ng2JqHoGwemUno1JGm8fWU0ppQSyWv03qL3w317PSOAARtT4Dxq56WcurXeuYy8Hj2UCiiAbE+us0EffRdmmpe469y4S7rnQ5rc6GSGGHfppARoR7V1JUI1FjIpp8f+ehS0y5qkb2meJhK+eeJ+VOeW0ESlB37UJLdl/nH6pebYRqvNtb9MgevYQhCLw7xIxHoJ6WJkHCqnS+L8+kEPGeu6gGwJL0jklrYP2zk0BqYLEgPpEt6nZCdcTz++CBeWY9Wog606ykz3RIokYkNFwnu9U2wvKXqFY8mAMLlmlm5u8Yw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000021, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Aug 14, 2024, Sean Christopherson wrote: > TL;DR: it's probably worth looking at mmu_stress_test (was: max_guest_memory_test) > on arm64, specifically the mprotect() testcase[1], as performance is significantly > worse compared to x86, and there might be bugs lurking the mmu_notifier flows. > > When running mmu_stress_test the mprotect() phase that makes guest memory read-only > takes more than three times as long on arm64 versus x86. The time to initially > popuplate memory (run1) is also notably higher on arm64, as is the time to > mprotect() back to RW protections. > > The test doesn't go super far out of its way to control the environment, but it > should be a fairly reasonable apples-to-apples comparison. > > Ouch. I take that back, it's not apples-to-apples, because the test does more > work for x86. On x86, during mprotect(PROT_READ), the userspace side skips the > faulting instruction on -EFAULT and so vCPUs keep writing for the entire duration. > Other architectures stop running the vCPU after the first write -EFAULT and wait > for the mproptect() to complete. If I comment out the x86-only logic and have > vCPUs stop on the first -EFAULT, the mprotect() goes way down. > > /me fiddles with arm64 > > And if I have arm64 vCPUs keep faulting, the time goes up, as exptected. > > With 128GiB of guest memory (aliased to a single 2GiB chunk of physical memory), > and 48 vCPUs (on systems with 64+ CPUs), stopping on the first fault: > > x86: > run1 = 6.873408794s, reset = 0.000165898s, run2 = 0.035537803s, ro = 6.149083106s, rw = 7.713627355s > > arm64: > run1 = 13.960144969s, reset = 0.000178596s, run2 = 0.018020005s, ro = 50.924434051s, rw = 14.712983786 > > and skipping on -EFAULT and thus writing throughout mprotect(): > > x86: > run1 = 6.923218747s, reset = 0.000167050s, run2 = 0.034676225s, ro = 14.599445790s, rw = 7.763152792s > > arm64: > run1 = 13.543469513s, reset = 0.000018763s, run2 = 0.020533896s, ro = 81.063504438s, rw = 14.967504024s Oliver pointed out off-list that the hardware I was using doesn't have forced write-back, and so the overhead on arm64 is likely due to cache maintenance.