From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF246E77188 for ; Thu, 26 Dec 2024 17:07:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 859146B008A; Thu, 26 Dec 2024 12:07:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 808C76B008C; Thu, 26 Dec 2024 12:07:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ADCC6B0092; Thu, 26 Dec 2024 12:07:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 472766B008A for ; Thu, 26 Dec 2024 12:07:23 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DDDD31A042D for ; Thu, 26 Dec 2024 17:07:22 +0000 (UTC) X-FDA: 82937739192.19.B13433D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf19.hostedemail.com (Postfix) with ESMTP id 7F84A1A0019 for ; Thu, 26 Dec 2024 17:06:37 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BUpaHBq1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3R41tZwYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3R41tZwYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735232822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cAz4Nb0K6HeFkqWg5wvl8l9ILYUUia2eSNvAxmroFu8=; b=c4KQn255lWCxCLk7JJZbbehFFv6sYbJdu4FHWW8htx+vAbI5Hd7Ix0v3bd/aKjIVHGd9z6 67EBMVUxTdZpc73zb+OdxBKvGgw/KogGRcb3Tv1ovf5sh/usyO3cXgrp+l5qf3+2GvheZu 5sd4By8YYUc+DJL4TeJrFnX1MSa/H74= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735232822; a=rsa-sha256; cv=none; b=bceCp+v0oyt3YSm7m87/335/UVDK8R5caH1vp7ke3ZIrO8Q8RTndwIZv3lcyX6i5f9SIeq SD2owGeLZNma3GU2Bv1nAlFATE3nr1xfhE/golw0eF9cPAyQkpvdU+3Zv5k6scpNNjFxvh XnxhsUhcJXKIHpDQDnkXqjOpF6vFN2Y= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BUpaHBq1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3R41tZwYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3R41tZwYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef8c7ef51dso7828340a91.1 for ; Thu, 26 Dec 2024 09:07:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735232840; x=1735837640; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cAz4Nb0K6HeFkqWg5wvl8l9ILYUUia2eSNvAxmroFu8=; b=BUpaHBq1IEN4YjofmDf9qCyWNFwrzYbyk79KrMuZuF+Ub5QMVxPWkGTL8lcD3cIpQD X6LdR73g0MnimGZ01/cTP6m9lvoy4iIAYoE3uxGRW5E2/cvpY07YyJ/+gVocRkmbBzVd /h1vYBBEi8uplCgJCExw9o5e1SIzKxW9bV3DFCFkV+Y6NudoVqdu0+7KlbTPOdn7MUiO 8sMGNg9NAL3gIOQQ0gdKG6lju3l4qiit98ZNVEskb5Y8GnnLW9IIbAopHLUk3I473tTQ FhpLmqQYyugr96V+LPQZeiDmu3aUHI2gbjnWMEsj0ezqDQ7mCBMI/9/t5M/PgZJXFchF YVqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735232840; x=1735837640; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cAz4Nb0K6HeFkqWg5wvl8l9ILYUUia2eSNvAxmroFu8=; b=MA7J7sBz9tUM2/cMwPdAzVUh0wyY4VA7yu0vAJ+zeYMS/J4+TDZNx+GrLr5qok0gG4 +6IoVG5nK8+neQDd+p1J6d94dTZNDPVKyQl5b6jLyMXJxjsE3KuHh7EWOxsM39eKxDnJ V1V4JHtKj0fqa6gOr7CfCe208di6cIuprJ35PjAxGNJOwjGRXL9XA1Yyj21+1Tzluit2 prXjolwMMPaTCd9ZTWDIYXP+ejLXLxbu90qLdWmppGE/EYR8IQBmSEiXKr3JJBHDu5Za lcERNRnqb+nA7SdXEgC3cjF68wsqrkmH/aXJoo1H1LA4+JFcASCTDdjxGUZyBvpHHEjy Wpuw== X-Forwarded-Encrypted: i=1; AJvYcCVvtN3njbxUdwsyKOH/qoG51SitgN6kTaMmOzd8Rw3EjGmXMgvaVNI1CquXKjL91bTXLJ5JVdreTg==@kvack.org X-Gm-Message-State: AOJu0YyuspzV3GinAf9YjCgffDDIrdmkl7258dlKNTH5XqtxUG1k1wlM NkPyU9fA5p8JxY2aPqu45rf3rjyxAGHBbhGJomrky8s/BjYS6vP4fB1BDwvSuwjME564WZJrLup sQQ== X-Google-Smtp-Source: AGHT+IHG75OaLxopGAtkxZlDDT2v+C2m79NALqlAe4xW7nBDhmw9thNrZbKpynlUfVT/naik0hkFrK8zsZY= X-Received: from pjbtb12.prod.google.com ([2002:a17:90b:53cc:b0:2ef:7483:e770]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5251:b0:2f2:a90e:74ef with SMTP id 98e67ed59e1d1-2f44353f0b2mr41720800a91.1.1735232839810; Thu, 26 Dec 2024 09:07:19 -0800 (PST) Date: Thu, 26 Dec 2024 09:06:55 -0800 In-Reply-To: <20241226170710.1159679-1-surenb@google.com> Mime-Version: 1.0 References: <20241226170710.1159679-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241226170710.1159679-4-surenb@google.com> Subject: [PATCH v7 03/17] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: omapu9kkdw8or63jon9w8tjcn8qdeepp X-Rspamd-Queue-Id: 7F84A1A0019 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1735232797-164809 X-HE-Meta: U2FsdGVkX18pFunXIIeJEMMaDr2Y09xPATjvftY0BMR44pBD0VDUdqRMNdIdZfISkJob0VNQnH0bV3JPLBx2A0Af3x/D3IVwWl0J1X7owIbHJnotvnSYza6amHXwyZm6gZPOULdivS9KRCoXz0lZig/sKQZkM/fG/E/yqgwWltWq4U0ygd3kppuVO/Bu9WYFkwsp1igNB1QxTkD0PUYqTUw4AolJ2vRpdkcobaFaqs8OZhkE+fYlDQD6U8CjNN28GqwwsQUqCMmBGM+LhqNRsuumT0Ow1xrKl5btSi4EtJ31VyjJ5OjtTtKfhyuwCUku0p3E9LnV3jN3CdYXytMEn1u0jATHCRY5t56DHXx1JkwnhARNhu/tyGmTtgfuOiOLNf32/Tz+sNYWOEfRG6qrI7D1AUHtOzMagcOxqjKD/Ta0Vab6L5wnJpdf/fEIfZet61icZo2nxK4+ZgKF+DlTbSzCUS954/olTRG3eIOjDKKOJ8W060mZJP7U1AR5YYz5y4/8bdxefJgbYmsFyRGh61qBPCTToGFh5W8SDhilu4F6lVl+x7eiqbrlykDvraV/mCjJM3npqS3wJ9SIESW2mCqD8YNFU+g7B95V5sZ0PzqY96Pz+xaa+3LoZ9jr0ziyG6h+fMbKOLLQXMlopLeYdS55qDvt8G5iw7v+VdrMemVAtuaH24e8pRdf34TtQKuFHSoxuCQ/CbawYLXcJ5ExFxUfs0xz/bnZhx1y0KoRsiyAK4JTMeM+wshw3BsMZARsoSqHQnn1CwhE5KAHrsgWVwuaJYavP0C/mRrfohJzt44ByghNlnnyhKdCpvluiwU3KsANal2a3a69SXI03L26aadgQg9NfFPwo2CWhgwDRN8iY16jdKMThKl08kTGFVudYr642YrGDInRMXV3I1/13gIlZ1SBJkJrAxODOjdIeiKmR+5FZnw/JnAcxrLnlBBR0/846AHDZ2uNSmOJUhR QJ6/1XNp vgFB7pc40vTsEGl20VAArlG1NnfTD/IsrDUA2K5aqWBh8Rutsppf+PMGrPH+jPrEkB7nvySRQST0caAO1Z+R/fN70H1pSGlILpm+UIE5ONrQngC2//qsa84yh2kGw/ARCl9PSjgq8wr2OrgGyQaEMJMgpWkdbCKcQgzpYK2OjB3GVDmJt8wiDY+i8rBL3DL+0+gRDOXKF2RyAgDtZDd4LzbRhYEf5ItIa6rqziNBTTgcuMB6VM07OM8O3VL/AF+T/wmqDlnqP9a/VeOaYhrUz3gQNPUWsz6CP/U+r05/T1z1YUSN10jbDIIJ2zZoc/d00jtH4Gz4zyMQLdaWvokTSZz96mI3SKhOjU19F27RahNAUo3vdI1uF42e+eGPBVRxH9N5NFegogIokZQDK5yZB/S/RkV/80q2bDgs0yyp200G1oLIul5fWRdG7m7/JssdvXyfge2PgLRWY2PkcsO9StYh8aqQA19TQni2vVNfFIL8+7Kf71bEXxu43zM2Lu5Q8XIAsxFzTUXc1LvON4fuY9GasHEF8x2LkkCFbTqlv3+jpdr6QuKbuPnxgU5KCagI+NhJl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f92ba8f5fe..081178b0eec4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 40a8e615499f..f2f9e7b427ad 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 2a20e3810534..d0dee2282325 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6349,7 +6349,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 0caaeea899a9..476146c25283 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 61ed044b6145..24636a2b0acf 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -157,6 +157,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -389,6 +390,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index d19ce6fcab83..2a624f9304da 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; } -- 2.47.1.613.gc27f4b7a9f-goog