From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC983D41D4E for ; Tue, 12 Nov 2024 03:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 588F06B00B6; Mon, 11 Nov 2024 22:23:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EC126B00B7; Mon, 11 Nov 2024 22:23:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EF186B00BA; Mon, 11 Nov 2024 22:23:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0A2006B00B6 for ; Mon, 11 Nov 2024 22:23:44 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 36A8E1A1C49 for ; Tue, 12 Nov 2024 03:23:43 +0000 (UTC) X-FDA: 82775997570.11.3C285B7 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf02.hostedemail.com (Postfix) with ESMTP id 83CF980006 for ; Tue, 12 Nov 2024 03:22:23 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d06RbChf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of surenb@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731381646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XZ4JdoZQje+kaOr0XJAZHdsqYNTycxGtA/GqVUYAWS0=; b=dyfB+LJINrFqJSMYpPblwuKzsdRYjYPKfx63f7XdvsRM/z/ihq2C7fxos4alfuklG9CxBW ury2KeVNHI2PIjDuHAUy1QsJBBJyKsV7c6ivHIEYorXNvv7FmmKdUwkH6j7MhGQOZO9SkH mu6tg2U1Vdb5UESnX3wCAUlFH+Af/Ys= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d06RbChf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of surenb@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731381646; a=rsa-sha256; cv=none; b=JiVob93xFubvREQZVxVjzOuBSmf5GNKsG5B/rjCq2qKSZR8R9/MxDybivDs8bPaZj6vhjz VCGE9xaH5jbtcgAEvuyJwCBNNzt0hxR4RRpT4MPZZjRZUOk6EbySBFVEPvBGsMa/kgUpMS EkEqD5CwP6zN5lFVO/TIpgRPRX841/I= Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-460b295b9eeso84931cf.1 for ; Mon, 11 Nov 2024 19:23:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731381820; x=1731986620; darn=kvack.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XZ4JdoZQje+kaOr0XJAZHdsqYNTycxGtA/GqVUYAWS0=; b=d06RbChfWS3N6Vgsr5/pkx5bYJL4OW71c7ip3c+bfcuWnRSS0Wafdq1D0elqycMz/c f9YO/Dj+P5J4KOd1eQFnO3/vwkqJ0iftwkE7lp59dm+nE/lxcAUsuXDUjb050brCub2g PSlSf/swugyHULdUPc+wB/yrlGuWfKsKGgOKr2uhHcVe5wKwatfIw/NHL/chEVOR0p7G KPtCOeVYcPaVByKxXlL6Vrp0eHtBl1zRCRB5TzHxmYdg8vn0h6Vj0QKe5JeVwHHNFEgM CcjXf6PJnIPCgYHUyzjsFBtb3zy6t56X+GXPMzr6dK9BWOAHq2nFsRQCX3ziVqWlWjQe iHYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731381820; x=1731986620; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XZ4JdoZQje+kaOr0XJAZHdsqYNTycxGtA/GqVUYAWS0=; b=Dfq7mldoxnSonudPcOsiwoPGw6MC+V9Iu3y09870dqFsx2dWlZiegYd5qBKTrlDvS/ R3HsZfP+1eFxUNuXig2NIMUpAUazOpHYiNZAQXZiUfYwkZBr9mperXtdPX1Y47dYd+LQ WfQttBj1OGXvDPXa7KNzMIJwRcub6OucZSeje9Jj5n5VDSM8G2bXf7KPuGmoCDbcyMg/ U7cPvTfOy6TeBVvVeD1uk0r3EsBCnpLegEd2f7oY/xLc4QVQX9reMO6Tgynyg1F78lLe esWd8OkiyFVqOaGMlZdpZrgnkCT7k9q3EKkib9sxl1/eok8TpCt4f4DyolL0xYCaZ+RC YI3g== X-Forwarded-Encrypted: i=1; AJvYcCV2nY5Yf38zQaZZvvjiNNJrNyJzr7rluBD9Sc4wcNIQoD5VAr33bSVs9oNFgPq3vmAjMuZ+5lTgDQ==@kvack.org X-Gm-Message-State: AOJu0Yw7uiftQVPrRZfORvhbzxMQJnRDmLpFNpyWwJoETILsaWnDj7bT 95Wd4fYkH9XAlLleYh1yqa6BeGiVOMnqMQaEGF9p5WzWFvuxqu5capNVxCkJJV50J7TWhzCBbUA zOCCpAvh3+QkX2A/k6oVEx4/iNOJumsXXw0Tr X-Gm-Gg: ASbGncvvjcsHK+OB4AnLbDscfoQiMJBRCIwp2krhtDrQaVOXyqKVyXd6XHHtedKGSTk Jopq26zBJphXnM5riv5n7wEoB2Z3l8VHXkQIyONbr+jTjzHnv+gay71W/1iy1JA== X-Google-Smtp-Source: AGHT+IH2X7R9EaxE8yEh/tFib5yP3+5ARuM042kvBNJm/BiVdasUXL+ESmr0khwLuuJ9NriFWoyJUV1Agr69/n8aGG0= X-Received: by 2002:a05:622a:190a:b0:462:c96a:bb30 with SMTP id d75a77b69052e-4633ef60502mr2057411cf.2.1731381820135; Mon, 11 Nov 2024 19:23:40 -0800 (PST) MIME-Version: 1.0 References: <20241111205506.3404479-1-surenb@google.com> In-Reply-To: From: Suren Baghdasaryan Date: Mon, 11 Nov 2024 19:23:29 -0800 Message-ID: Subject: Re: [PATCH 0/4] move per-vma lock into vm_area_struct To: "Liam R. Howlett" , Suren Baghdasaryan , akpm@linux-foundation.org, willy@infradead.org, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 83CF980006 X-Stat-Signature: zbaoq7xewhz8crk1t7dta517kpewzi8n X-Rspam-User: X-HE-Tag: 1731381743-981948 X-HE-Meta: U2FsdGVkX1/F/92zcgi9vSDBm5/KBSbZ/IInj2UU0PnQnOMxB6glWOQlGVuvtVl+QWLar/rSfXEW17IIWX8d2dgUuIKMuAW1LeiIV1o0qitG43CuCVojqhLWxqByPLW2lMJLik3MHqzt3oqHL9kTQPkpE78W7PvCOcYc9u5yHfQXqCIVKMfV8+ZcRMsEnbCw7g2dBkWtGbuuGri1+7CgnaSuCGa+q1nzorYM1DVpvDixbpzCG0QT4G55EZ1bIpKC2Q4L8HyMh/Ka/KdzPPT9TIlHIZQwPoJ4Qhopkp93NITCNe62iKKl3pxBC+y2CrVNJ8RnfViTRD8kK71mpcc0AxmTgB4HapfkMyPBRE6CRXSoqlHpZzGdwHc4E2LF67yzt+Mi+6jclkzHy6RpH+fONzP+zpth4Z4eFLrok/i6DW39lNVYYkmmg2otpuqkRuoMzE4aPfW6evHxj276x/O9MjJ9lNM5VEyhW9MxNQ5uqhzqiLBbWqRXCqdI2Rjbo0pb958NTf3eGDHE4rt+UcoRfJgqbf4n4Z43TmN38yDQltih9pnOUlCA9Ae1kVC77FRpVpUlRdkPETwBnFjedEXiAuMSgDuXNqMmC1HCru2OXfJPV7m2rMP5T/ykhkQxe8IGQ3gxdPCR1CcQo3OnfOrrDkjLfPyA6N8PFwqvjxeBaGbMdthufE1vNxW10EE1zMm4zsQhWa32uT428GyqA+j3y8TYcKi3Qx5JmeJ2L+Qwp5FfloQfivVXbODDWuctPayM4BhbwtaRQIyrLQjuZsVLjS5TQKrInWSla+OLCYbeocvmUVvkqM+2CN+qnASSNMGJ0b+vP/vEbSPr+4X39B8NRPwapaiQhTfP16qUh9AbG9HYBc3WRgWytEWETaqGfYpj1LtzBENcpHiJmdCzhbM+2/m5n94idRNe3srYeqKhzIIQhjDGOp3PbSlb1ahgy16Qu5mBDFKNe7oljZRqJH8 Jy/Qtbz4 TN9YbV3RqUcXUsAqlP5z7zVVC3Pi6Smox4MUOxsEZ3uC8VCMADjeUx9fijGEUCCIIbX52QtmCCFHnIkog3+/dSvIh68dFO65as61GU495xmOs15kwIJnVnlgTO95jgbF3lieuPoTmySMG8ajd/+cVTSlJ8dJxT6FSbmV7GsG49E9YCn7wrqcTZxRsAtuuhy4LFUVmLkivVe6zVpjn17WnFyfd/SdCIvEImJASBsEkulH5qm2+noKkviedGCIwlfVT9iImf3FSv6RIEKLRyYr96tMV0ofOH7vs0K0RF6zGapGU3ECgqR4wiYluy+5XFlLNj9I8BGcu74s0UoH25ZzxdygGYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Nov 11, 2024 at 6:35=E2=80=AFPM Liam R. Howlett wrote: > > * Suren Baghdasaryan [241111 15:55]: > > Back when per-vma locks were introduces, vm_lock was moved out of > > vm_area_struct in [1] because of the performance regression caused by > > false cacheline sharing. Recent investigation [2] revealed that the > > regressions is limited to a rather old Broadwell microarchitecture and > > even there it can be mitigated by disabling adjacent cacheline > > prefetching, see [3]. > > This patchset moves vm_lock back into vm_area_struct, aligning it at th= e > > cacheline boundary and changing the cache to be cache-aligned as well. > > This causes VMA memory consumption to grow from 160 (vm_area_struct) + = 40 > > (vm_lock) bytes to 256 bytes: > > > > slabinfo before: > > ... : ... > > vma_lock ... 40 102 1 : ... > > vm_area_struct ... 160 51 2 : ... > > > > slabinfo after moving vm_lock: > > ... : ... > > vm_area_struct ... 256 32 2 : ... > > > > Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 page= s, > > which is 5.5MB per 100000 VMAs. > > To minimize memory overhead, vm_lock implementation is changed from > > using rw_semaphore (40 bytes) to an atomic (8 bytes) and several > > vm_area_struct members are moved into the last cacheline, resulting > > in a less fragmented structure: > > > > struct vm_area_struct { > > union { > > struct { > > long unsigned int vm_start; /* 0 8 *= / > > long unsigned int vm_end; /* 8 8 *= / > > }; /* 0 16 *= / > > struct callback_head vm_rcu ; /* 0 16 *= / > > } __attribute__((__aligned__(8))); /* 0 16 *= / > > struct mm_struct * vm_mm; /* 16 8 *= / > > pgprot_t vm_page_prot; /* 24 8 *= / > > union { > > const vm_flags_t vm_flags; /* 32 8 *= / > > vm_flags_t __vm_flags; /* 32 8 *= / > > }; /* 32 8 *= / > > bool detached; /* 40 1 *= / > > > > /* XXX 3 bytes hole, try to pack */ > > > > unsigned int vm_lock_seq; /* 44 4 *= / > > struct list_head anon_vma_chain; /* 48 16 *= / 40 + 16 > > /* --- cacheline 1 boundary (64 bytes) --- */ > > struct anon_vma * anon_vma; /* 64 8 *= / 56 + 8 > > const struct vm_operations_struct * vm_ops; /* 72 8 *= / > > long unsigned int vm_pgoff; /* 80 8 *= / > > struct file * vm_file; /* 88 8 *= / > > void * vm_private_data; /* 96 8 *= / > > atomic_long_t swap_readahead_info; /* 104 8 *= / > > struct mempolicy * vm_policy; /* 112 8 *= / > > > > /* XXX 8 bytes hole, try to pack */ > > > > /* --- cacheline 2 boundary (128 bytes) --- */ > > struct vma_lock vm_lock (__aligned__(64)); /* 128 4 *= / > > > > /* XXX 4 bytes hole, try to pack */ > > > > struct { > > struct rb_node rb (__aligned__(8)); /* 136 24 *= / > > long unsigned int rb_subtree_last; /* 160 8 *= / > > } __attribute__((__aligned__(8))) shared; /* 136 32 *= / > > struct vm_userfaultfd_ctx vm_userfaultfd_ctx; /* 168 0 *= / > > This is 8 bytes on my compile, I guess you have userfaultfd disabled? Yeah, I show here results with the defconfig. After I move things around at the end we will have space for to keep everything under 3 cachelines. > > Considering this will end up being 256 anyways these changes may not > matter, but we can pack this better. > 1. move vm_lock to after anon_vma (ends up at 64B in the end) > 2. move vm_lock_seq to after vm_lock Nope, for performance reasons it's important to keep vm_lock_seq in the first cacheline. It's used very extensively when read-locking the VMA. > 4. move struct to the new 112 offset (which is 8B aligned at 112) > 3. move detached to the end of the structure detached also should stay in the first cacheline, otherwise we will get performance regression. I spent a week experimenting with what we can and can't move :) > > This should limit the holes and maintain the alignments. > > The down side is detached is now in the last used cache line and away > from what would probably be used with it, but maybe it doesn't matter > considering prefetch. It very much does matter :) > > But maybe you have other reasons for the placement? > > > > > /* size: 192, cachelines: 3, members: 17 */ > > /* sum members: 153, holes: 3, sum holes: 15 */ > > /* padding: 24 */ > > /* forced alignments: 3, forced holes: 2, sum forced holes: 12 */ > > } __attribute__((__aligned__(64))); > > > > Memory consumption per 1000 VMAs becomes 48 pages, saving 2 pages compa= red > > to the 50 pages in the baseline: > > > > slabinfo after vm_area_struct changes: > > ... : ... > > vm_area_struct ... 192 42 2 : ... > > > > Performance measurements using pft test on x86 do not show considerable > > difference, on Pixel 6 running Android it results in 3-5% improvement i= n > > faults per second. > > > > [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google= .com/ > > [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ > > [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT= _kbfP_pR+-2g@mail.gmail.com/ > > > > Suren Baghdasaryan (4): > > mm: introduce vma_start_read_locked{_nested} helpers > > mm: move per-vma lock into vm_area_struct > > mm: replace rw_semaphore with atomic_t in vma_lock > > mm: move lesser used vma_area_struct members into the last cacheline > > > > include/linux/mm.h | 163 +++++++++++++++++++++++++++++++++++--- > > include/linux/mm_types.h | 59 +++++++++----- > > include/linux/mmap_lock.h | 3 + > > kernel/fork.c | 50 ++---------- > > mm/init-mm.c | 2 + > > mm/userfaultfd.c | 14 ++-- > > 6 files changed, 205 insertions(+), 86 deletions(-) > > > > > > base-commit: 931086f2a88086319afb57cd3925607e8cda0a9f > > -- > > 2.47.0.277.g8800431eea-goog > >