From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4D6AD597C7 for ; Wed, 13 Nov 2024 02:57:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D6BB6B00BB; Tue, 12 Nov 2024 21:57:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 987156B00BC; Tue, 12 Nov 2024 21:57:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FFD86B00BD; Tue, 12 Nov 2024 21:57:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 615DE6B00BB for ; Tue, 12 Nov 2024 21:57:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DA0B6C0725 for ; Wed, 13 Nov 2024 02:57:43 +0000 (UTC) X-FDA: 82779560724.18.C8B7161 Received: from mail-il1-f174.google.com (mail-il1-f174.google.com [209.85.166.174]) by imf16.hostedemail.com (Postfix) with ESMTP id D6DC618000A for ; Wed, 13 Nov 2024 02:56:59 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ASUYvY5n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of surenb@google.com designates 209.85.166.174 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731466532; a=rsa-sha256; cv=none; b=yQ3S8JiW9uv8lT5OexYgzOtcDjFgxf21wBV94J8+0ummmvazDEBw9NJiWBK2KRiMceY2Qz MaUJjdFW66nPxwI3x3nTT+M2uAlNxZOw6mfRRXsT6NNHQrkMv9ZZbh1lfJoSpZWqf0KqBi M5hG8CO7z8z412T0pjrEFopo3jg0V+s= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ASUYvY5n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of surenb@google.com designates 209.85.166.174 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731466532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BiPGZH28TjM74y6PlEbeCLpg1G4vxPdKyGtVU4e+ltk=; b=G11LFLmxk/f8FZf8liitmG8TK0rIpa/glVdmo+1/EMJwXmAsbooy9EK41hspJAHpBW2aKz D6yX5YwN5n/HjauS8VRtMfKDAlQbrHMKgvCvKmb6qGUMDUj4nhVPJNiy5PhJc8F+hrUJCJ nmi8dGVfiVinMu5PSQFP9/qclunvBBM= Received: by mail-il1-f174.google.com with SMTP id e9e14a558f8ab-3a4e54d3cefso62465ab.0 for ; Tue, 12 Nov 2024 18:57:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731466661; x=1732071461; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=BiPGZH28TjM74y6PlEbeCLpg1G4vxPdKyGtVU4e+ltk=; b=ASUYvY5nfvrURgTrsthI7NXdQlSEQsKuKcjhFq8xw/Vy14CAl6Z9q+WvFiicpE/syL rv98KizzhBeFsGLWz4cKCwwayjnnRuNTiTNFImwfB0ICCTe7udnlWsJKaUIPRuANQcnz kRuhf4jQK6KUzeMuZiAtuFeN8JXiBo7gqD/+QqOIEUSIABPZswEVfbj9hv1RN5uHM4LX 5+R6ERg4T+NiMErGlJ54PsCvqLwVUR3E/hV0cH5DVScYteDhB4tMST2zVTFJIawCu7N6 idiRdoiDZDFEcKDm0FykcFkc5AHCp1SKrQ/YGdvhkfKlY8WekluzhK4oH35JMld/h7QG NGrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731466661; x=1732071461; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BiPGZH28TjM74y6PlEbeCLpg1G4vxPdKyGtVU4e+ltk=; b=eXUf7xraPpgvdYHHtMm/rqVLGRDZB6XSg/iChHUZNEeSkVq2cKpRciZ2XDak7R5XIs V62TJeiTYdRMSEERM+ypqlVXXFV8tYgosIh9KinruUs2/Msjmr7OcyQIM3wWSvA1KLYr vCyNRdNw+WHNnzcyHgAg/39Dj6PWQKbDERhJ9XCUOgk+vy6ee8SWlJB5rniesUiD3owu +DFqOIsYVD99wiIAc7C3p46E2wK9h0WvfTc7wGHG5KgPDyCyNulfJQKLb6TyrYua1vFT Ix9L0x24+MKP921Ktnai8f97Qy0//7cBtfVL4XOn/pXNUtMH4ZRLOrUAgkPLlH1eWpwQ M3Kw== X-Forwarded-Encrypted: i=1; AJvYcCUVRbktSPjJPHA42w7YwA2dprs7/EK9xdXojQAkI1zNMMNdTTvhzVf+cH+YmT+mIk9/Gvb04qOVdw==@kvack.org X-Gm-Message-State: AOJu0YyVdFIAkOwDEgH9kdE3mboYGrFLMAE4gMxAkhEQk3A5XTPEzkKH M/Bp9BsstMfgE1n0Z4+EO2Tj8/gqTzXiYYsjPiZ/9z1lChQtjply8qx0VXskJtgWbsh09IYI8of Z0lqCFKXCxi0ej2EQ/kyYG1sogN0zPFtI4afk X-Gm-Gg: ASbGnculd3KE/gd64nuR31+4VsHYrlGu9dG1MoqGkMNQvydRFRwaVc1Lq++LxByeOCA GKiBS/kTRerZ0kNrxg39rS7eh4/XzPMQ= X-Google-Smtp-Source: AGHT+IEECaI8sMTzzH3P3OtOAOiKpt4aBBrFl39wfr2y6vpKiA8N//g2AVMBj6w97P3qp6YBF6Zwm7sETS31Ttr2DS8= X-Received: by 2002:a05:6e02:19cc:b0:3a0:8cb6:3ad1 with SMTP id e9e14a558f8ab-3a715c010eamr1580395ab.20.1731466660621; Tue, 12 Nov 2024 18:57:40 -0800 (PST) MIME-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> <20241112194635.444146-5-surenb@google.com> In-Reply-To: <20241112194635.444146-5-surenb@google.com> From: Suren Baghdasaryan Date: Tue, 12 Nov 2024 18:57:29 -0800 Message-ID: Subject: Re: [PATCH v2 4/5] mm: make vma cache SLAB_TYPESAFE_BY_RCU To: Suren Baghdasaryan Cc: akpm@linux-foundation.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: D6DC618000A X-Stat-Signature: o16q3zh1b1k4no1npgxs9yoiiq1nibfp X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1731466619-177903 X-HE-Meta: U2FsdGVkX19E9wswmsqpLXITnc06ozpkRPHMdUIEvNbyb9JITpUWYOx0Yq0dvee4NNv8AS6/QOPGFktXrkaBLeIvRSNQXvFO2fWaj/wlNOl7bIAKghmXA6d2dsIrBo0xMVXqAPt7tKa1BXEqw/NPXH7NDGjQXUFOqUbds2431OOu4V9m4brZu5k55f+ZoW8SkOt/znyD9aMvVdTm9/1qOLcPRGCFujdiDf2PEMYDR/ID22LxsBlsVR4EqNAx9627vfOmHz5hur/m5dt73OD4nVZTrkqYJcUx1s3t29xIea8siTa4CHkZmOhcYKg20z2K0lXQsUEILaTyrOh9/dy3Y088gwHlJWXq/ZHkIg1L6QK1up90IXJ6wKss07LKRf661MFLDyD7tzcWwIXJ2SuK6X15wKFg5JkLL2ZqYv+J9jg6WrViYcjM3bfFc5hFpavKJtiQqT1jnPws/hVJ2i0kV8Z2F6hqIB71rsqCbK1u7N1vDLP00/pB6z0qMcmk1/uqjZKo94bmUxRrV/cc8gaav55DjawRNhrVQDmOBoJS6rLIASplyOG+uCtYh6feJ1RgTvNiKpUbRb+zh9eDHQBSjVa92pibWwxdlX3SYBjvU5S6TQxwkbZj6yuk4Vbn6DlIUgKPslK/4FUNVGZdF7AE0RYtQn/B/3EI6RVW6e7Eu7DGgw9YgpuzgAnmi3LqDJAHnexn5i+ROO0aViq2neOdqFguOv165SBuP9PkUdfEIKYdMshFwFaf3OmCBmSWxCunYcbqUi/zcCvQFyBLgjsd9fup10nEAhsBxBFxOjofD8ZrcgSP1tMwm5URzHQYBFNeMm3tMD27ts07TBkM9bv+CTW7rLYrpkQUmfY95Fs2psSnL2qwbu866G4yDLdKYLz6RFsa0S4csZeEj+HTxKzyecXfQ0MszWhHYXad4WZVsQQWsXFUbr7HpQHmTwQtHn8qzVbDfWji5wNFbL/QpRX DrGeeTvz iTJA2Os4+0zMS1DNgh1KN+KXoGAyztqwoVjjXqov78hkDizZz63ocMCRrCmLyg7p1Mo4AfndTxAT4Kb5WEdPUqKp0KEslX1pWKail3wEGWjpD1+hr2qfcoPI/dRiA0uYb1LQVvDS3zsZCZLyBrUcl9h3VaMzbcbutODxUBvkwX6AcFBW6UyWJHwTIq1GXTotabpMG34fqDfR/h1ACOI8WRAzUgY8NnzlG5eGNUUfkGvL1TFwKcUo0zs1aKuQqSi2zPqn4Q/WtZGwr3YwIZWmx7n02zcUCKsWH9JD7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Nov 12, 2024 at 11:46=E2=80=AFAM 'Suren Baghdasaryan' via kernel-te= am wrote: > > To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that > object reuse before RCU grace period is over will be detected inside > lock_vma_under_rcu(). > lock_vma_under_rcu() enters RCU read section, finds the vma at the > given address, locks the vma and checks if it got detached or remapped > to cover a different address range. These last checks are there > to ensure that the vma was not modified after we found it but before > locking it. Vma reuse introduces a possibility that in between those > events of finding and locking the vma, it can get detached, reused, > added into a tree and be marked as attached. Current checks will help > detecting cases when: > - vma was reused but not yet added into the tree (detached check) > - vma was reused at a different address range (address check) > If vma is covering a new address range which still includes the address > we were looking for, it's not a problem unless the reused vma was added > into a different address space. Therefore checking that vma->vm_mm is > still the same is the the only missing check to detect vma reuse. Thinking about this some more, I don't think this works. I'm relying on vma_start_read() to stabilize the vma, however the lock I'm taking is part of the vma which can be reused from under us. So, the lock I'm taking might be reinitialized after I take the lock... I need to figure out a way to stabilize the vma in some other manner before taking this lock. > Add this missing check into lock_vma_under_rcu() and change vma cache > to include SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct > reuse and will minimize the number of call_rcu() calls. > Adding vm_freeptr into vm_area_struct avoids bloating that structure. > lock_vma_under_rcu() checks of the detached flag guarantees that vma > is valid and attached to a tree, therefore unioning vm_freeptr with > vm_start/vm_end is not an issue even though lock_vma_under_rcu() is > using them. > As part of this change freeptr_t declaration is moved into mm_types.h > to avoid circular dependencies between mm_types.h and slab.h. > > Signed-off-by: Suren Baghdasaryan > --- > include/linux/mm_types.h | 10 +++++++--- > include/linux/slab.h | 6 ------ > kernel/fork.c | 29 +++++++++++++---------------- > mm/memory.c | 2 +- > 4 files changed, 21 insertions(+), 26 deletions(-) > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index 5c4bfdcfac72..37580cc7bec0 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -32,6 +32,12 @@ > struct address_space; > struct mem_cgroup; > > +/* > + * freeptr_t represents a SLUB freelist pointer, which might be encoded > + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. > + */ > +typedef struct { unsigned long v; } freeptr_t; > + > /* > * Each physical page in the system has a struct page associated with > * it to keep track of whatever it is we are using the page for at the > @@ -673,9 +679,7 @@ struct vm_area_struct { > unsigned long vm_start; > unsigned long vm_end; > }; > -#ifdef CONFIG_PER_VMA_LOCK > - struct rcu_head vm_rcu; /* Used for deferred freeing. */ > -#endif > + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY= _RCU */ > }; > > /* > diff --git a/include/linux/slab.h b/include/linux/slab.h > index b35e2db7eb0e..cb45db2402ac 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -212,12 +212,6 @@ enum _slab_flag_bits { > #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED > #endif > > -/* > - * freeptr_t represents a SLUB freelist pointer, which might be encoded > - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. > - */ > -typedef struct { unsigned long v; } freeptr_t; > - > /* > * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. > * > diff --git a/kernel/fork.c b/kernel/fork.c > index 7823797e31d2..946c3f9a9342 100644 > --- a/kernel/fork.c > +++ b/kernel/fork.c > @@ -478,25 +478,15 @@ void __vm_area_free(struct vm_area_struct *vma) > kmem_cache_free(vm_area_cachep, vma); > } > > -#ifdef CONFIG_PER_VMA_LOCK > -static void vm_area_free_rcu_cb(struct rcu_head *head) > +void vm_area_free(struct vm_area_struct *vma) > { > - struct vm_area_struct *vma =3D container_of(head, struct vm_area_= struct, > - vm_rcu); > - > +#ifdef CONFIG_PER_VMA_LOCK > + /* The vma should be detached while being destroyed. */ > + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); > /* The vma should not be locked while being destroyed. */ > VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); > - __vm_area_free(vma); > -} > #endif > - > -void vm_area_free(struct vm_area_struct *vma) > -{ > -#ifdef CONFIG_PER_VMA_LOCK > - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); > -#else > __vm_area_free(vma); > -#endif > } > > static void account_kernel_stack(struct task_struct *tsk, int account) > @@ -3115,6 +3105,11 @@ void __init mm_cache_init(void) > > void __init proc_caches_init(void) > { > + struct kmem_cache_args args =3D { > + .use_freeptr_offset =3D true, > + .freeptr_offset =3D offsetof(struct vm_area_struct, vm_fr= eeptr), > + }; > + > sighand_cachep =3D kmem_cache_create("sighand_cache", > sizeof(struct sighand_struct), 0, > SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RC= U| > @@ -3131,9 +3126,11 @@ void __init proc_caches_init(void) > sizeof(struct fs_struct), 0, > SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, > NULL); > - vm_area_cachep =3D KMEM_CACHE(vm_area_struct, > - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| > + vm_area_cachep =3D kmem_cache_create("vm_area_struct", > + sizeof(struct vm_area_struct), &args, > + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RC= U| > SLAB_ACCOUNT); > + > mmap_init(); > nsproxy_cache_init(); > } > diff --git a/mm/memory.c b/mm/memory.c > index d0197a0c0996..9c414c81f14a 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm= _struct *mm, > goto inval; > > /* Check if the VMA got isolated after we found it */ > - if (is_vma_detached(vma)) { > + if (is_vma_detached(vma) || vma->vm_mm !=3D mm) { > vma_end_read(vma); > count_vm_vma_lock_event(VMA_LOCK_MISS); > /* The area was replaced with another one */ > -- > 2.47.0.277.g8800431eea-goog > > To unsubscribe from this group and stop receiving emails from it, send an= email to kernel-team+unsubscribe@android.com. >