From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 203A5D42B9C for ; Tue, 12 Nov 2024 15:45:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 881B56B00AA; Tue, 12 Nov 2024 10:45:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 830E56B00AF; Tue, 12 Nov 2024 10:45:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AD3B6B00B1; Tue, 12 Nov 2024 10:45:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 419886B00AA for ; Tue, 12 Nov 2024 10:45:32 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AEBED1A0313 for ; Tue, 12 Nov 2024 15:45:31 +0000 (UTC) X-FDA: 82777866402.26.8E93BBF Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf16.hostedemail.com (Postfix) with ESMTP id 232E6180010 for ; Tue, 12 Nov 2024 15:44:47 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zh5owlQ6; spf=pass (imf16.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731426156; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LC0oAB3hsBtzyot9DwM7yE0bq5av8B40/ay1VJm5kIY=; b=lUVWGLdAE6JjL1Ma8GqxVnX5e8bszHrhaq+Kz8oVr9EEnVih4EKDYliTXMf162uDeZ1rv2 y3f9RhKxU0AyG4QkoB1yubmq9O7k+VjERXMUvCVtDMtelfhgjJreka864elkIJqDBVAOfN bnZhi5nBbtNysbfcAijDJh28S9ANcaQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zh5owlQ6; spf=pass (imf16.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731426156; a=rsa-sha256; cv=none; b=lVHw4Wc+HH9lJa8TEJCw+LcabfjcOA8ZtsAPYF/RvBBqhHFB675cWS2SskyntQfpDjqPXD QRc4jBYRJZG3PM5WfZkZ8eyfxj23zgi5TLzrkxI91f8SejTQFwJh4q+QjCpEz3QF20WAfO tTzpVtLfaqMIOFi//eq4VhXX2Smuc7Q= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-460969c49f2so313611cf.0 for ; Tue, 12 Nov 2024 07:45:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731426329; x=1732031129; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=LC0oAB3hsBtzyot9DwM7yE0bq5av8B40/ay1VJm5kIY=; b=zh5owlQ6tdV6tceEFEEqw5ZbLT2OJV7YrmKAl14CsFRuDnKJsZ9jDcoFooOwOgrek9 VLgkcCAqGEDs7qMgKxI5w9Eti2QTF38Z5UAPyluQRmRVj5Ki4qpza0aC6nMwMZhpgoiQ QvXkKSUI/51UcOcUeVquGeGItPPm7ma4HqJRrTMUGjDfNKhOhH0lPtzacdRo/iTOyUhc 6K/7F0qs7C9NcocrF46UnCwunO86M0ckHsaqUwxtf2XF52/bJuihQQ2vud/bqc+SbXgR mVGO97ZsuhYdWx1LISQZjinvJaX6TlsvBU5djuTINoFl+kPt6eZ1H1W7cJtxy/hFMR3h WQkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731426329; x=1732031129; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LC0oAB3hsBtzyot9DwM7yE0bq5av8B40/ay1VJm5kIY=; b=anENqKhTaCJMAcDs+9g92HtG0sks9CNYC502eskxKQDxN+ZQ9a0jQHen1HB+WBbDBB NC5YDZLOuTyRWyewbv3sZeSTSwZ5nXJVKwVegvE4YGkWzmasNVbxl4OkPCKgXaV7/ne7 gm+MKlbzrHE3V5ITBTIpddoDcszBBoYGN+VxelIt8DMSW1o1hj2WIXuWcd7mGBorRUjg WKQEI6tL5RpwSF2an4WgQYpbHh8aG7apEim+8wON8+e7RoxVBg85e7pc7JJXERsmc90g t84LC5mjUg/Q5hTXgYwERu+g9zq2WiN9nt2nkGTW0+snCzr7hx89E49Pv+mUIOQ3cOdt JsNA== X-Forwarded-Encrypted: i=1; AJvYcCXTAcMCiJUPnr7zgYKRFDh9IUbgj9ChETxBegHsldJdy/dp7k4fA04IBXd3yw3K2S6tW/vtipFHzA==@kvack.org X-Gm-Message-State: AOJu0Yw6n0H/pLxTmqlYBk6s5zjqS9n06BB42h3S8gORugFX6om2PVPW egF26V00UuzOatUGALtYQSeCCGW02GKNeX7vADV7FmIdFPY083JJylZakfAZ+b/m7R0T7GvBNk8 bn7C8hxXvWNcnGTK7cUBXZMNuY/IkbisCfSMs X-Gm-Gg: ASbGncsBZGO17qiB5BGBrPjLZbu9XWFyNgkh5q9fzfWNUFCKprVmlHXTMpMvunFXAXH CW7hFTFE0a+sqMIsv2m4YyKhSEdtY3w4CSrkWnvKR2G7J3RDQ8WTe21Yovc+8bw== X-Google-Smtp-Source: AGHT+IHITYhV1PsQizhz8Y+qiNc1Fv3ooQCiIBNrErc3MxHIp7FJycJEqDOHsW5Ie2/I+HtgciMvWSgYDyi1lwxy5Cc= X-Received: by 2002:a05:622a:4b0f:b0:460:b4e3:49e with SMTP id d75a77b69052e-4634288b366mr2509591cf.9.1731426328226; Tue, 12 Nov 2024 07:45:28 -0800 (PST) MIME-Version: 1.0 References: <20241111205506.3404479-1-surenb@google.com> <20241111205506.3404479-5-surenb@google.com> In-Reply-To: From: Suren Baghdasaryan Date: Tue, 12 Nov 2024 07:45:17 -0800 Message-ID: Subject: Re: [PATCH 4/4] mm: move lesser used vma_area_struct members into the last cacheline To: Vlastimil Babka Cc: akpm@linux-foundation.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 232E6180010 X-Stat-Signature: mu3s1k49f1xzppu7pr4p3cpij8em4bw9 X-Rspam-User: X-HE-Tag: 1731426287-30940 X-HE-Meta: U2FsdGVkX1+rELjH5xJBO5zTEFMNCAWkkUNdZbf3VYMUQP7D7Ke8B5pu19tSdzwPDxvqsIcHxTzmKaJon/5XSnRfsKRVHg/dooGevVDUmWR/T7LeojnyQTepqN1t3XvMJPn0/QUuicbd/N83G4xSPViQKRinXZQ99jeDZt2/ZfOjiM0bxd3iRHUkfMCrfZg1a0Bvd0eZi/qlREbHvi6TNpml7UryJagWw33CgGvnWoNHiK+9nEJ5wriT5eaazN3Z3R+HBwQQgotlLoyGwJ3tdqjtRVKtNX4rTb74KPU5JejglHp/EsHoqYwXO2Hn4DqMoTidOHgEZl4XXJIwa8Buotzhxx3WMlSHxKAWb1HhMmzcaSrWpyVwpNhja/HQJvkf1PsJ2clw6gDwLFCFGjjXarVjkZ5wG2OBa2hDT6QlzSKLCDzB//xW/moFrBdAfDt2a3GxZ5IF3M4cyGEXis6PyjSiS4+YNn093VSxmsWQApgFqo5hMWsosVfi1jXgPD9do+lmnD0awBNxhENbFNNi51CzNAbqQaDo3mcQS1gSsDGoeVFM8IIIWaj2tIuG3WfRObnpAF5rwPytKc3txnteBiElbCWNnGlRLp3QfZ1n4lL4SJpO5vhmb47b8llQaugbfiqfgI+oV/3Uvw7+JXoybnwLcytAmuOtlb0YjAq4ySWk2C6twWqWBnV2IjydtsaqCh80pmZfz8wBK0kchqlTuYDjoNqUNDZpL5RIi/hRAlIdtwNAELMErbh2wuLsNtH6sRL0ywSR0cBGe/5xnjDw0deBxMZcvkFn9dFCeiiq3LQucaG1ieminKDfbuLJI8QHx2UHw22QoyhthH0wTfxM+BXfFuWZNX56eV9XA1f+E4crJNieQfm7MmYQiq/PzXQvvYlZY3HhlBFhGJ4kZZfaYp5Z03Exzg8Dddckeky2FNJ6I7lLIXK8P7HxFsh2w++pULI2PhpFbM0C6XZZUrB v8cRFdkD 6vUFemnul7FrfrU1fF4asuthzeN/MMIzKMiexjjKPfrYj/rEG/ovtmfLsRJO3S7FkKR70E7LArl3a9GLZ99O30rRDD98nqSefPNBJbteG1m5P+HiEWfX94Yir4wYGBMTO1ANe/WV2cFVAmlq27qq2UoxtcXAAWXoD+VdHw/PpzUUemwFENfMRDh1T9N6W5G7Ez3V9/36L85DowPUug+8aHMGQgdrkPzCsN2LZjUJpO6KzdfLhm1fsQe3+fi8Oo4fyCF3n5GZT61jbmVeX9Tm0ZGn6TUxEEA92V4s0WwaRUHF1e6U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Nov 12, 2024 at 2:07=E2=80=AFAM Vlastimil Babka wr= ote: > > On 11/11/24 21:55, Suren Baghdasaryan wrote: > > Move several vma_area_struct members which are rarely or never used > > during page fault handling into the last cacheline to better pack > > vm_area_struct. As a result vm_area_struct will fit into 3 cachelines > > as opposed to 4 cachelines before this change. New vm_area_struct layou= t: > > > > struct vm_area_struct { > > union { > > struct { > > long unsigned int vm_start; /* 0 8 *= / > > long unsigned int vm_end; /* 8 8 *= / > > }; /* 0 16 *= / > > struct callback_head vm_rcu ; /* 0 16 *= / > > } __attribute__((__aligned__(8))); /* 0 16 *= / > > struct mm_struct * vm_mm; /* 16 8 *= / > > pgprot_t vm_page_prot; /* 24 8 *= / > > union { > > const vm_flags_t vm_flags; /* 32 8 *= / > > vm_flags_t __vm_flags; /* 32 8 *= / > > }; /* 32 8 *= / > > bool detached; /* 40 1 *= / > > > > /* XXX 3 bytes hole, try to pack */ > > > > unsigned int vm_lock_seq; /* 44 4 *= / > > struct list_head anon_vma_chain; /* 48 16 *= / > > /* --- cacheline 1 boundary (64 bytes) --- */ > > struct anon_vma * anon_vma; /* 64 8 *= / > > const struct vm_operations_struct * vm_ops; /* 72 8 *= / > > long unsigned int vm_pgoff; /* 80 8 *= / > > struct file * vm_file; /* 88 8 *= / > > void * vm_private_data; /* 96 8 *= / > > atomic_long_t swap_readahead_info; /* 104 8 *= / > > struct mempolicy * vm_policy; /* 112 8 *= / > > > > /* XXX 8 bytes hole, try to pack */ > > > > /* --- cacheline 2 boundary (128 bytes) --- */ > > struct vma_lock vm_lock (__aligned__(64)); /* 128 4 *= / > > > > /* XXX 4 bytes hole, try to pack */ > > > > struct { > > struct rb_node rb (__aligned__(8)); /* 136 24 *= / > > long unsigned int rb_subtree_last; /* 160 8 *= / > > } __attribute__((__aligned__(8))) shared; /* 136 32 *= / > > struct vm_userfaultfd_ctx vm_userfaultfd_ctx; /* 168 0 *= / > > I don't see anon_name in the output, I thought it was added for Android? = :) Yes, this output is generated with defconfig. That's why you see some holes in this structure. On my x86 machine I have non-zero vm_userfaultfd_ctx and numab_state, on Android I have vm_userfaultfd_ctx and anon_name. > > > > > /* size: 192, cachelines: 3, members: 17 */ > > /* sum members: 153, holes: 3, sum holes: 15 */ > > /* padding: 24 */ > > Instead you seem to have padding so an attempt to use SLAB_TYPESAFE_BY_RC= U > should use that and not add more up to 256 pages. Yes, thanks for the tip about SLAB_TYPESAFE_BY_RCU freelist. In actual configurations where I saw SLAB_TYPESAFE_BY_RCU causing this structure to grow I had less padding at the end. > Perhaps this pahole output wasn't generated with a fully representative c= onfig? You are right. I'll replace it with the actual output from my x86 setup (Android probably has a smaller interested audience). > > > /* forced alignments: 3, forced holes: 2, sum forced holes: 12 */ > > } __attribute__((__aligned__(64))); > > > > > > Memory consumption per 1000 VMAs becomes 48 pages: > > > > slabinfo after vm_area_struct changes: > > ... : ... > > vm_area_struct ... 192 42 2 : ... > > > > > > Signed-off-by: Suren Baghdasaryan > > --- > > include/linux/mm_types.h | 37 ++++++++++++++++++------------------- > > 1 file changed, 18 insertions(+), 19 deletions(-) > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 789bccc05520..c3755b680911 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -733,16 +733,6 @@ struct vm_area_struct { > > unsigned int vm_lock_seq; > > #endif > > > > - /* > > - * For areas with an address space and backing store, > > - * linkage into the address_space->i_mmap interval tree. > > - * > > - */ > > - struct { > > - struct rb_node rb; > > - unsigned long rb_subtree_last; > > - } shared; > > - > > /* > > * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_v= ma > > * list, after a COW of one of the file pages. A MAP_SHARED vma > > @@ -762,14 +752,6 @@ struct vm_area_struct { > > struct file * vm_file; /* File we map to (can be NULL). = */ > > void * vm_private_data; /* was vm_pte (shared mem) */ > > > > -#ifdef CONFIG_ANON_VMA_NAME > > - /* > > - * For private and shared anonymous mappings, a pointer to a null > > - * terminated string containing the name given to the vma, or NUL= L if > > - * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. > > - */ > > - struct anon_vma_name *anon_name; > > -#endif > > #ifdef CONFIG_SWAP > > atomic_long_t swap_readahead_info; > > #endif > > @@ -782,11 +764,28 @@ struct vm_area_struct { > > #ifdef CONFIG_NUMA_BALANCING > > struct vma_numab_state *numab_state; /* NUMA Balancing state *= / > > #endif > > - struct vm_userfaultfd_ctx vm_userfaultfd_ctx; > > #ifdef CONFIG_PER_VMA_LOCK > > /* Unstable RCU readers are allowed to read this. */ > > struct vma_lock vm_lock ____cacheline_aligned_in_smp; > > #endif > > + /* > > + * For areas with an address space and backing store, > > + * linkage into the address_space->i_mmap interval tree. > > + * > > + */ > > + struct { > > + struct rb_node rb; > > + unsigned long rb_subtree_last; > > + } shared; > > +#ifdef CONFIG_ANON_VMA_NAME > > + /* > > + * For private and shared anonymous mappings, a pointer to a null > > + * terminated string containing the name given to the vma, or NUL= L if > > + * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. > > + */ > > + struct anon_vma_name *anon_name; > > +#endif > > + struct vm_userfaultfd_ctx vm_userfaultfd_ctx; > > } __randomize_layout; > > > > #ifdef CONFIG_NUMA >