From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EB59C433EF for ; Fri, 3 Jun 2022 16:42:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78A06B0071; Fri, 3 Jun 2022 12:42:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B27038D0001; Fri, 3 Jun 2022 12:42:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E8CA6B0073; Fri, 3 Jun 2022 12:42:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 86FB06B0071 for ; Fri, 3 Jun 2022 12:42:19 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 571DD10DA for ; Fri, 3 Jun 2022 16:42:19 +0000 (UTC) X-FDA: 79537492398.22.668B371 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf24.hostedemail.com (Postfix) with ESMTP id C0A6D180015 for ; Fri, 3 Jun 2022 16:42:02 +0000 (UTC) Received: by mail-qk1-f172.google.com with SMTP id o68so6377971qkf.13 for ; Fri, 03 Jun 2022 09:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=pZiq+DaP0uIJbqmhpHEQocGuha+dXUbZZZqTwIB2IaE=; b=rrQxT3vbT7GMAWiBE8NKh82hTKWmXoBOIgjSoB8gWHlxJCtpicQfJi6YHcKEu0Skmg abttTcfRqmTOcz4L3ohQC6WWhlA31fuHxoEfeuJpbUAg7kX6/vtplGcJ9j4IrzJVxD9O vdLoYfObodGls9T269h7hJyDm997n4snH1eyk5q8lGm0DuwzVfKoDQFEQSeKxajjX2Yz 5TiL9zM/J27+HT1KqOA1cB9necDYn5jb3A6o3ARG7LxeA2ES2j5uBkLAZdLyHaiO4lvt ER/oF7LY8Hxyq/pFtt8IclZxYPm5bLZuRep0Ojk6gIJnuU/og9qHiw7H4aRy54/0OtvI pIJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=pZiq+DaP0uIJbqmhpHEQocGuha+dXUbZZZqTwIB2IaE=; b=sX7mFz0mjwmziIRCK4Y34tSAAYnCd1sXYB9WjeEw6X6dlFI8qznq3EpkntxIF65ZKi 4RW0DBop03sDpxHHVgwQkk48BZFtg5ooNibKFZTukyGLRUKVs9q5NkR+jqsyo7Cu0/os Eb3qQxVlWc/D4ofiMffAkAInQsYC0X23P81rrbdn5u6cu6GZvpO+sd4BAIvDyiN8Ide5 VllPZqN0ROCm6GzJQgDlGfdZ/IgTrOjGwBz5nVH/Zq7adNqYFUHuEhZYjt/witALEQ3b J0QCJGbunp4+BXEETdFGwwXhbdWmJ4BAsDvyyYtPvCyTs6Pt9KYOVCzbVVyQNQutNgsX L/yA== X-Gm-Message-State: AOAM5306SpsR8B8YM9sqlHyF6wgGEBrjYL0MsKb9NReOYh7IWJIOjgBD wrUZ5BL0QpDB4FAuapVK1vsO6g== X-Google-Smtp-Source: ABdhPJwD1IT8PgvyiFpaFJ0oQgGHCzkg9tknn5j7X5fVKzbiMKtPXk1vgrTU2p+vhNcLbOMRztlAVw== X-Received: by 2002:a05:620a:2845:b0:6a3:646f:9ba8 with SMTP id h5-20020a05620a284500b006a3646f9ba8mr7509475qkp.56.1654274537770; Fri, 03 Jun 2022 09:42:17 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:1d66]) by smtp.gmail.com with ESMTPSA id w184-20020a3794c1000000b006a098381abcsm5550650qkd.114.2022.06.03.09.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jun 2022 09:42:17 -0700 (PDT) Date: Fri, 3 Jun 2022 12:42:15 -0400 From: Johannes Weiner To: Yosry Ahmed Cc: Sean Christopherson , Shakeel Butt , Marc Zyngier , Tejun Heo , Zefan Li , James Morse , Alexandru Elisei , Suzuki K Poulose , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Andrew Morton , Michal Hocko , Roman Gushchin , Oliver Upton , Cgroups , Linux Kernel Mailing List , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, Linux-MM Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=rrQxT3vb; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.172 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C0A6D180015 X-Stat-Signature: aqs7q5wps1kbiyneuipcr4cap4k8yk8n X-HE-Tag: 1654274522-470376 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, May 27, 2022 at 11:33:27AM -0700, Yosry Ahmed wrote: > On Wed, May 25, 2022 at 5:39 PM Sean Christopherson wrote: > > > > On Wed, May 25, 2022, Johannes Weiner wrote: > > > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > > > I don't have enough context to say whether we should piggyback KVM MMU > > > > pages to the existing NR_PAGETABLE item, but from a high level it > > > > seems like it would be more helpful if they are a separate stat. > > > > Anyway, I am willing to go with whatever Sean thinks is best. > > > > > > Somebody should work this out and put it into a changelog. It's > > > permanent ABI. > > > > After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. > > > > It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will > > scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something > > bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to > > allocate a large number of page tables even though the guest isn't accessing that > > much memory. > > > > But, someone would need to either understand how KVM works to make that connection, > > or know (or be told) to go look at KVM's stats if they're running VMs to better > > decipher the stats. > > > > And even in the little bit of time I played with this, I found having > > nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very > > informative. E.g. when backing a VM with THP versus HugeTLB, > > nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an > > order of a magnitude higher with THP. I'm guessing the THP behavior is due to > > something triggering DoubleMap, but now I want to find out why that's happening. > > > > So while I'm pretty sure a clever user could glean the same info by cross-referencing > > NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the > > very least prove to be helpful for understanding tradeoffs between VM backing types, > > and likely even steer folks towards potential optimizations. > > > > Baseline: > > # grep page_table /proc/vmstat > > nr_page_table_pages 2830 > > nr_secondary_page_table_pages 0 > > > > THP: > > # grep page_table /proc/vmstat > > nr_page_table_pages 7584 > > nr_secondary_page_table_pages 140 > > > > HugeTLB: > > # grep page_table /proc/vmstat > > nr_page_table_pages 3153 > > nr_secondary_page_table_pages 153 > > > > Interesting findings! Thanks for taking the time to look into this, Sean! > I will refresh this patchset and summarize the discussion in the > commit message, and also fix some nits on the KVM side. Does this > sound good to everyone? Yes, thanks for summarizing this. Sounds good to me!