From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 337FFE7718E for ; Thu, 26 Dec 2024 17:07:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E6806B008C; Thu, 26 Dec 2024 12:07:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 748176B0092; Thu, 26 Dec 2024 12:07:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60D946B0093; Thu, 26 Dec 2024 12:07:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3B6A36B008C for ; Thu, 26 Dec 2024 12:07:25 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E7E088112B for ; Thu, 26 Dec 2024 17:07:24 +0000 (UTC) X-FDA: 82937739990.22.DE44566 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf17.hostedemail.com (Postfix) with ESMTP id 2B3E240011 for ; Thu, 26 Dec 2024 17:06:50 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=uzIWEtaq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3SY1tZwYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3SY1tZwYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735232824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uC/uS/GmF3UbaohKrad0ksO3lGJZa9/e0a1asiVVa0A=; b=CwX/xT6+j43vEwwzURawakDllVW/qwSFxSHFzJYSTRt4juvoKQibb8uSwtpUyISz6PW19d oDUekBfE7ihmWHGk5hj2l/B5vqPHiiSfpHm3j5yN6mhbFPEvB2LRP3kykYDhyyA4bjGYgA jE0BA0YKABkRucL13yoQRZfN6dLsjYE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735232824; a=rsa-sha256; cv=none; b=i1FqOj9k4mMQh+Rms2SEPJJh+Eqp+NAE+Ejz71WMHgUs5r/ttrA7GZFR9JobhScwCTp4sm zQ9RkGKd/zwNtn4kVxYWKp3dL4FGiKt3qHCZB434UmM6jG5AUBxw/w6idolr0X8wedvZbk Sequ0d8satNnqdMtdd/ZHDcsdQkeCTA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=uzIWEtaq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3SY1tZwYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3SY1tZwYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2eedd15c29eso8009505a91.3 for ; Thu, 26 Dec 2024 09:07:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735232842; x=1735837642; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uC/uS/GmF3UbaohKrad0ksO3lGJZa9/e0a1asiVVa0A=; b=uzIWEtaqH+Xglai4E6j6IAq6UM6yd60zeK6LaIHWQ8aFYFrmWi40zzAIkKzGXXpMxW 70D7GpS7ZI8FEJxXZf6/7ALSJy3Nt2SOq4pFQ5m2zYlTUxcSF0ilfJRbmtaDUgHcZBZW SR3HaTSAJy3gsHQH8G31Y/nsRoR5UDHrJkh8eoO6WtEazm//NDq2rSgqT7GNHxVf1HrC c8R3tLSgd5yDkMNiA+s6YlyKSs1CGo8rmWrZKsSSCf71saHKSJjbcgZkCaSLex7Hqk6v l/Dt76YEUuQYllf7KzsoY/WXnQPAxBhtwjF5oPsNUHbsKMtdISAyjH201kL4fPhe7vJs AInw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735232842; x=1735837642; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uC/uS/GmF3UbaohKrad0ksO3lGJZa9/e0a1asiVVa0A=; b=tKJju2oGISTnXuSF/pO/iPhkIZXJ5FmD06DrbtfsfkPkU1AmPj0s7TcfcQPC9zWNer 7IWct87+KZbcho32ZBc68yCDvrLBmUV2WSWCgYSKVdEtg6/V7sm0AI/iwEWqSSxwuzAX SNxivPIxD89lGxNPZVGRhwuFQ9CeO3MgMoLP+EazNyWQHpCM4Cvp64VakVBsUcw3ZNmF lDSjeki+ntfDwtUFnQbQqYtuTcYbF/DX6eK4Xkrqy2XD7xbu7bU234KZs9wZUpD1Xpiu 4/CCgECESic8i7V99wmSdlTLHOT6TkurUpRbL0sSgY3nySBWt0H0dAkQ+83MPIZ2gKzN tW8Q== X-Forwarded-Encrypted: i=1; AJvYcCWxWqnyzNXd6xVdM8g2ybIQOtwf1j9AIoGpLCH9V0U1IHiU8y3pzD4IMQ3L3hD6lZy54TtPbSZvJw==@kvack.org X-Gm-Message-State: AOJu0Ywgfvd2l8+imiRIH77RxxgEah5gu/NrbV0X1yGkM81i5VelkhJ4 zb2wrAX0r4hNrnY5BMZ7QGYWFt6oI9TJOKhBjSgJK6CQzcGMr0zuY90myXZaCupKQY2chtt9/7y LDA== X-Google-Smtp-Source: AGHT+IHCdGYEXz7DM0Oi4jA5FcxiPuZb/VK8PE48RzKkuHUHkyLtSTzBzCK0XU3Squr5UgqXBJEcCRqnrjo= X-Received: from pjyr4.prod.google.com ([2002:a17:90a:e184:b0:2ea:3a1b:f493]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:fc4f:b0:2ea:5054:6c44 with SMTP id 98e67ed59e1d1-2f452eeb641mr29738883a91.31.1735232841920; Thu, 26 Dec 2024 09:07:21 -0800 (PST) Date: Thu, 26 Dec 2024 09:06:56 -0800 In-Reply-To: <20241226170710.1159679-1-surenb@google.com> Mime-Version: 1.0 References: <20241226170710.1159679-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241226170710.1159679-5-surenb@google.com> Subject: [PATCH v7 04/17] mm: modify vma_iter_store{_gfp} to indicate if it's storing a new vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: qre35mtkououyotkab3869rtfww1xrek X-Rspamd-Queue-Id: 2B3E240011 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1735232810-284135 X-HE-Meta: U2FsdGVkX18RDBrgGrooBb+K0ZME55DOuCV+Wl0RlrUuiTBaeUtuccFODF4PJjCdWIjd2LmFeV9d1clM97HUoprUz6SANm7RqAlk5m0+/YA7hOpJXorOva1ySOcfwCijZRieDYXGJt4kccSi3WVHoS/YKJSpXnMwSmCQ0I0okDsMlXZn8aX4v5EFnzZh1mX90mTVcLb4u6of37yvCY++lJiZ78VitH8Xt8HA5AQbK+3wA2lZn12mMoms8c0u7E3M99MNwh9gYzA1ubG/Qxh8hvYrPs9Gv5QOwIToJLZiNTy6KS2rqZQEwuUo0ioXc4jymTa9FaqJmasVHH8+rk25CyYGf2G3cV7PyFdnuhJFfyWf2h/ArKIFSfkyqYWHsY/VUIZN/K6XUGwNVFzWePbyE0VjIQuhtd50GVgMxADsxhqqDUNQB4Qg79EtxbPeNVuNJtCCnFSDS/p7cITBLnzh6b6jAvaVJ17kwlFK3taKKKbAVK1Engm+NV12OgiXUUKIjuK+P5Es9Tc93d+NsPbz3V4PTj+ohd3h923nnifNETEwGmPTENJ7ooNnCtc7jXgUkJ1UfUfkZJGmi1ATVGU85Z3LGoQ7SmDz3UisoAvKVTK4dbx+GwIQtOdasne5IGX34HjOrZApdFp8pPaHq3/+MJzB5u7QyIxG+tyIJYPJYWpRyKLdfeuRlqynRpZpnJTmjt3K/E8EoSJMey4xXh11KGOgoRqhzWjkDiACiuCgkLHcJgqGbZkKO8w8n44q/ba1SAsmRnlBY1ypCoa+lTPLjyhcFT9/FObGGDUa5qHB82reILactIak4g4nDE3gxTH7JGVNKYfmadiD6Yt0k6OA7VeQz+FN7oy0lMZlQw7GZcf5ap0gR5KH1yD5SztSykxwb3/hBg5DSqQZSBfghCiLI1TCtVHFAfPqeCg6YEJorBSIfTcXuecmafnc+8FgEx4gmG7JdsK5fTbRxM6J8um gbpvtLWS GEWEGnK8L29ueNXnLei1arpSgeWqvJxKZ7Vkf3enV+jgTYHi2A72XVuY6WPQeGFFRRmfKtjudOrZRaSkgUA/5pYFoyCcZBuo9oQZlQ4EOrJDBbvsSJ1gQuSHQYW19+ed7jvbu6R4Mf2cIb3+MPJJjl6METSXAcAV16Zlmm0sF/PnUHCOSZRcD0o8P6bHK/UcA4bftdCjZ0qZWp46F1HmdyFIj1tKd7OhUWGuH8ygVryTs4T8bAFBK9/gQVQYVO+/hy4dhCiU+PW5ZbEF/28rpr9NoVytdBTRs1m06ySd2plBCYK6BjI8BmyQG0HJxPZe8rFk5KCIZGkcrMUnvQZ9Wz8w9L7C3TR9JvKmdjP+Gazp3qkSdPgWF6120dG866Ot0iGxN3qPj5FkJ1PYj5MqjQfvCDH55UcDjeBybVCOsbn0roflKv/jwVWUFU7NKOsrvMg87BSa4dkWtBbZ4ygI0zTHpQ6LwDTsswNWHceHigL9XPjdAVlRpTqAARmHNj0leBR7PE2AkPEnamhSgcIWUGNp4UG9pMJ2TAJZ5FGnU4laQrEYe5m1bxaikJPEk1eamXILcxR1O4v+yIho= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_iter_store() functions can be used both when adding a new vma and when updating an existing one. However for existing ones we do not need to mark them attached as they are already marked that way. Add a parameter to distinguish the usage and skip vma_mark_attached() when not needed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 12 ++++++++++++ mm/nommu.c | 4 ++-- mm/vma.c | 16 ++++++++-------- mm/vma.h | 13 +++++++++---- 4 files changed, 31 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 081178b0eec4..c50edfedd99d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,6 +821,16 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } +static inline void vma_assert_attached(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(vma->detached, vma); +} + +static inline void vma_assert_detached(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(!vma->detached, vma); +} + static inline void vma_mark_attached(struct vm_area_struct *vma) { vma->detached = false; @@ -866,6 +876,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } +static inline void vma_assert_attached(struct vm_area_struct *vma) {} +static inline void vma_assert_detached(struct vm_area_struct *vma) {} static inline void vma_mark_attached(struct vm_area_struct *vma) {} static inline void vma_mark_detached(struct vm_area_struct *vma) {} diff --git a/mm/nommu.c b/mm/nommu.c index 9cb6e99215e2..72c8c505836c 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1191,7 +1191,7 @@ unsigned long do_mmap(struct file *file, setup_vma_to_mm(vma, current->mm); current->mm->map_count++; /* add the VMA to the tree */ - vma_iter_store(&vmi, vma); + vma_iter_store(&vmi, vma, true); /* we flush the region from the icache only when the first executable * mapping of it is made */ @@ -1356,7 +1356,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); - vma_iter_store(vmi, new); + vma_iter_store(vmi, new, true); mm->map_count++; return 0; diff --git a/mm/vma.c b/mm/vma.c index 476146c25283..ce113dd8c471 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -306,7 +306,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, * us to insert it before dropping the locks * (it may either follow vma or precede it). */ - vma_iter_store(vmi, vp->insert); + vma_iter_store(vmi, vp->insert, true); mm->map_count++; } @@ -660,14 +660,14 @@ static int commit_merge(struct vma_merge_struct *vmg, vma_set_range(vmg->vma, vmg->start, vmg->end, vmg->pgoff); if (expanded) - vma_iter_store(vmg->vmi, vmg->vma); + vma_iter_store(vmg->vmi, vmg->vma, false); if (adj_start) { adjust->vm_start += adj_start; adjust->vm_pgoff += PHYS_PFN(adj_start); if (adj_start < 0) { WARN_ON(expanded); - vma_iter_store(vmg->vmi, adjust); + vma_iter_store(vmg->vmi, adjust, false); } } @@ -1689,7 +1689,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) return -ENOMEM; vma_start_write(vma); - vma_iter_store(&vmi, vma); + vma_iter_store(&vmi, vma, true); vma_link_file(vma); mm->map_count++; validate_mm(mm); @@ -2368,7 +2368,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap) /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); - vma_iter_store(vmi, vma); + vma_iter_store(vmi, vma, true); map->mm->map_count++; vma_link_file(vma); @@ -2542,7 +2542,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vm_flags_init(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); vma_start_write(vma); - if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL)) + if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL, true)) goto mas_store_fail; mm->map_count++; @@ -2785,7 +2785,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) anon_vma_interval_tree_pre_update_vma(vma); vma->vm_end = address; /* Overwrite old entry in mtree. */ - vma_iter_store(&vmi, vma); + vma_iter_store(&vmi, vma, false); anon_vma_interval_tree_post_update_vma(vma); perf_event_mmap(vma); @@ -2865,7 +2865,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) vma->vm_start = address; vma->vm_pgoff -= grow; /* Overwrite old entry in mtree. */ - vma_iter_store(&vmi, vma); + vma_iter_store(&vmi, vma, false); anon_vma_interval_tree_post_update_vma(vma); perf_event_mmap(vma); diff --git a/mm/vma.h b/mm/vma.h index 24636a2b0acf..18c9e49b1eae 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -145,7 +145,7 @@ __must_check int vma_shrink(struct vma_iterator *vmi, unsigned long start, unsigned long end, pgoff_t pgoff); static inline int vma_iter_store_gfp(struct vma_iterator *vmi, - struct vm_area_struct *vma, gfp_t gfp) + struct vm_area_struct *vma, gfp_t gfp, bool new_vma) { if (vmi->mas.status != ma_start && @@ -157,7 +157,10 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; - vma_mark_attached(vma); + if (new_vma) + vma_mark_attached(vma); + vma_assert_attached(vma); + return 0; } @@ -366,7 +369,7 @@ static inline struct vm_area_struct *vma_iter_load(struct vma_iterator *vmi) /* Store a VMA with preallocated memory */ static inline void vma_iter_store(struct vma_iterator *vmi, - struct vm_area_struct *vma) + struct vm_area_struct *vma, bool new_vma) { #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) @@ -390,7 +393,9 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); - vma_mark_attached(vma); + if (new_vma) + vma_mark_attached(vma); + vma_assert_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) -- 2.47.1.613.gc27f4b7a9f-goog