From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B82C3C83030 for ; Fri, 4 Jul 2025 06:07:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 114856B8009; Fri, 4 Jul 2025 02:07:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 09F5D6B00FF; Fri, 4 Jul 2025 02:07:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED0476B8009; Fri, 4 Jul 2025 02:07:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D7C9A6B00FE for ; Fri, 4 Jul 2025 02:07:50 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 74ACE161C02 for ; Fri, 4 Jul 2025 06:07:50 +0000 (UTC) X-FDA: 83625551100.30.23485A6 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf07.hostedemail.com (Postfix) with ESMTP id A89BE4000B for ; Fri, 4 Jul 2025 06:07:48 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="LkTM/dgp"; spf=pass (imf07.hostedemail.com: domain of 3s29naAYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3s29naAYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751609268; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2vVO5jxx/ZJrgheBL7RPOjE/kXWGqgLicQNUNDfyGkA=; b=lP1OLcIW5/KbI2pb110GmceXT+wnsKpttlr7GJA3Kwa7i3KmU7oQlFycIlU7vZd4m/f/Am QcQqetldXqGcwGuDSeFEfJiOrxxF3lkH+9qqfO+gW4XW9GvyawWjaQtwTzbqkqL6aZRq4c MPOv0sHE+y7dyXIBxTWPDaP/h9RHlIY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751609268; a=rsa-sha256; cv=none; b=Ij9r8CmD7ISC94+7qwIx+mtqyp9df1vjv1b85DzxXVdCOftlp4WpIhMz6HCwIj2Ap/fwpe odaT2Ukkz3R5aMSakgo+LCz0wvMNyw2W3LlEdvr+dk5znJjlTmuSCHyu9VU/d2145KW/sc dbABY4Z+Gs+Ts81grj3XUeS7soy3pcc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="LkTM/dgp"; spf=pass (imf07.hostedemail.com: domain of 3s29naAYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3s29naAYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-74ad608d60aso549423b3a.1 for ; Thu, 03 Jul 2025 23:07:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1751609267; x=1752214067; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2vVO5jxx/ZJrgheBL7RPOjE/kXWGqgLicQNUNDfyGkA=; b=LkTM/dgpFDPizqEgkD1fjqNNCJVNPGUDfQPiVUIQHDzQjF51G8gauCu3clJFH1tUl8 a6xZaq8ELPtBWRDCkl6GIVkw2QQkeDIkUra3n0gsSNmVWoJ94B7qKHcP5KDfYG6LM0cU lajFwqTBrhP1m/a5uhfbYNMM3Ua7EJk12eB+B7SkWsnQOWGQN1Q7IJYBrnpiY9e7+fUi 4/pOx9z+Un8BWzEejA4RLNWSoY2KbIHJRDb9gLfR2+YpcSDYewcrPMDHdkftV4EqrcOY hsIN1kN1+/sfG+HnOM+HIE3sI4EwK4y62er1vlJWfpQHk5fhder1fTJeYvo4HW/I/rt7 H1xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751609267; x=1752214067; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2vVO5jxx/ZJrgheBL7RPOjE/kXWGqgLicQNUNDfyGkA=; b=aigDVItOU4K/eBwhyEnNohuWcQJNEKesmvPK5ehtcert7Mu9MdOaNhBiRfOE3TRcLu ZHqNNB6gA9NU8s1FDiaVMpjLtFkffROMOGMdCKIZZaGSXFH2fp8TFucbTRojzqB6ICyb ppgEs+BeF6i8HE0AFiDTDC9JBTRi4fBicYwkV38vnZfp57izqwsaveYDLExc5q2NxCRD ivrh+kskK6Q4YWVovSv8CyeHbdIxmvro/k6owKfa58lwgP/oH7MYO1GXGvMSAx21kBlq hd6mpxzQ+2fTyEm5l2YNcA2wUPqGFEi4oobqHZWLUtLpulmnNycknfaNmbg14K4noxKj Qh1Q== X-Forwarded-Encrypted: i=1; AJvYcCVKrFQ8VNCagI2GelRnyWttSRwxiYqZymU6zugBND22PDNhHUNwn2RWW1DhR9LapKn2BYkdp2zwtw==@kvack.org X-Gm-Message-State: AOJu0YyZSWchKSHN9VuP2YyI+9BJKVa10gGenprYhsefq4JY/XuGjG5y PFFchQbU+ApLozCcc58xIOdVZgPcBUF1UL0LqrX21weud+qtl/0HMN0i/hDSnpdtxKp/ry7ysUV dp0EKCw== X-Google-Smtp-Source: AGHT+IGK2PS7F/Js/FFS6q/dLVUjbaZBo7PGzhlpQvuYSYZ/AxLr1RaHAdn4y2306pdsG4prKULLG7rLEWE= X-Received: from pftb16.prod.google.com ([2002:a05:6a00:2d0:b0:746:3185:144e]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3c8b:b0:748:3485:b99d with SMTP id d2e1a72fcca58-74ce65c53bamr2551001b3a.18.1751609267282; Thu, 03 Jul 2025 23:07:47 -0700 (PDT) Date: Thu, 3 Jul 2025 23:07:26 -0700 In-Reply-To: <20250704060727.724817-1-surenb@google.com> Mime-Version: 1.0 References: <20250704060727.724817-1-surenb@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250704060727.724817-9-surenb@google.com> Subject: [PATCH v6 8/8] fs/proc/task_mmu: execute PROCMAP_QUERY ioctl under per-vma locks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, vbabka@suse.cz, peterx@redhat.com, jannh@google.com, hannes@cmpxchg.org, mhocko@kernel.org, paulmck@kernel.org, shuah@kernel.org, adobriyan@gmail.com, brauner@kernel.org, josef@toxicpanda.com, yebin10@huawei.com, linux@weissschuh.net, willy@infradead.org, osalvador@suse.de, andrii@kernel.org, ryan.roberts@arm.com, christophe.leroy@csgroup.eu, tjmercier@google.com, kaleshsingh@google.com, aha310510@gmail.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, surenb@google.com Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A89BE4000B X-Stat-Signature: armtm3itzuxtx5idoe1ksqjxcdhknxzf X-HE-Tag: 1751609268-469693 X-HE-Meta: U2FsdGVkX1/8aZsCxTExZKBc7gidqF76d3cD3I048Qed7jpDQ9cCmdJvaQZVLwVLjALOXxzdHU6Z3f2Gejxz/C2w/UQ+zEJvFKEZ1wwh5ePgUFQ4QORAnVE/8TPy8RVDUrqD646bXZ+xwKxn/mi4XeKFYF38p8Unhkx7Gc66/P0MsovmIQuMDb82bmdG2/qVMK9ssAheFVYR+CgiL/tVIAHZuxjCHIMEI0liBwCXe80E0LYpyBVSnlN+o0mp/Wp4ZfphqLqm398NY3YQZm3erM2SZNMjgXyw0fp/FCFgJ/Qv8h5Vk0G7bNkmfX/WgIWAevsVGvUzVXjg+i0iPO8IiGnCOws0QJrQDLe/00vRMveduZDGJ85P/h7VcfP6twV1etd7gVDTRUrpWsY6UkZnxCOP3gDaYEHIS+utlfsBbWzfvuPIebc/u8WridOBVHCpsuzQyM55cc5fpVyrR9VDh4bghHolKyIeRJJQQWmTqxSPHziSDwb/F1KFXv6AiaVKKu2lw0HSl/E8TRxqNeyl6/aUM4t/iIrbUcwDvBgKiYYUCChVmWSm7XF4r0tzJ4K9ssMdgOexvjKM8QzTMwKNyCnrjENPLutYdnwGdPGI+ukOXVrLDTQGL9VHKMHHmoBXxlgokdHljwVMbTs63j5KRHHztjd82qFffLOOdR+vX2dqtWuP7//O47TvSgc1MnxAZzp+NFm4MptNANzZboq0Z++YEM2VGBNd5QTMPmCvv3573qIhmtxSoVCwjLrS80iesd8JCzGzIFT0W9pPghfwi+gpsGlradakfhEdEi/3cOfFfk/b2f1u41VV11nJKRBVdNI4bmib09HJOVRe43Zzt7ZZucj4lYKkLBKqn/ZyIQ5brptL62pr/hP0eo8CJxhIx0Rh95vbBqJueMyoXJbL1ZG3f7w3bM96qxHYe7XryzSkFSFIboibROGG/iyMLsr4wg1hj/PtqumVjZFSDLw 1WleYoQs EvgCj25iE1ydc72FEmANAE2l1vk+Taa9Yblzq10Zbe2lo0+NOsoRzYCOqyRGkV3przJSlNtvwpxhPv4ZbJFx3E8BCW8g6v/JYJJL1+bQrOqupkjg3o/dNFbH7MR7L5JkGukiIktyDZ0KGrBGwfGsAdm5GztKR+pN/1OQjFZniR9fVpY1hwJOf12MH4pfM2hfPxu0ts3jrgGNEkp5OG15xQCIpke+qCDS8zCuMC2xWShwUNoudzva8Hx0RZz5QXIsT+8oNAD8Mo1+eJ4C0pcY8XbIWA18m18371Ui+s9D6puvHPw6e/z00fEc6LhXt3tWgyCboJDRz4jFdQxPYHaSNULRHdxnHZ739wQ/vDoRNrwBrTxKB4ClmiSPi8IME4l43qmZB4FmEWvB+d7tUiOzCYicsTif2+uvkXBZG3gFHsJyoQmICfCHFSFj7vDW/dDjt1s7ECuPg5gMtXiC2a68LzFXJEYTt+fpp740hpjf6kd67gr12gGDYB9EImdXrjg3jN5gWs532uKlQb/BRIU2sogy50ubzoUjmOd3Cfn4XqaDSNETn8dNMfUQDspFJaER9mAwXV9LPokevIuoXs1myUYZJtS1y8DjbJ8vZMbDM9QyviAzAD3jy4wg66Pj59a2uc2IsXqnBNfDPTiy9F9NOYB8HfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Utilize per-vma locks to stabilize vma after lookup without taking mmap_lock during PROCMAP_QUERY ioctl execution. While we might take mmap_lock for reading during contention, we do that momentarily only to lock the vma. This change is designed to reduce mmap_lock contention and prevent PROCMAP_QUERY ioctl calls from blocking address space updates. Signed-off-by: Suren Baghdasaryan Acked-by: Andrii Nakryiko --- fs/proc/task_mmu.c | 60 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index ff3fe488ce51..0496d5969a51 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -487,28 +487,64 @@ static int pid_maps_open(struct inode *inode, struct file *file) PROCMAP_QUERY_VMA_FLAGS \ ) -static int query_vma_setup(struct mm_struct *mm) +#ifdef CONFIG_PER_VMA_LOCK + +static int query_vma_setup(struct proc_maps_private *priv) { - return mmap_read_lock_killable(mm); + priv->locked_vma = NULL; + priv->mmap_locked = false; + + return 0; } -static void query_vma_teardown(struct mm_struct *mm, struct vm_area_struct *vma) +static void query_vma_teardown(struct proc_maps_private *priv) { - mmap_read_unlock(mm); + unlock_vma(priv); +} + +static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_private *priv, + unsigned long addr) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma_iter_init(&priv->iter, priv->mm, addr); + vma = get_next_vma(priv, addr); + rcu_read_unlock(); + + return vma; } -static struct vm_area_struct *query_vma_find_by_addr(struct mm_struct *mm, unsigned long addr) +#else /* CONFIG_PER_VMA_LOCK */ + +static int query_vma_setup(struct proc_maps_private *priv) { - return find_vma(mm, addr); + return mmap_read_lock_killable(priv->mm); } -static struct vm_area_struct *query_matching_vma(struct mm_struct *mm, +static void query_vma_teardown(struct proc_maps_private *priv) +{ + mmap_read_unlock(priv->mm); +} + +static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_private *priv, + unsigned long addr) +{ + return find_vma(priv->mm, addr); +} + +#endif /* CONFIG_PER_VMA_LOCK */ + +static struct vm_area_struct *query_matching_vma(struct proc_maps_private *priv, unsigned long addr, u32 flags) { struct vm_area_struct *vma; next_vma: - vma = query_vma_find_by_addr(mm, addr); + vma = query_vma_find_by_addr(priv, addr); + if (IS_ERR(vma)) + return vma; + if (!vma) goto no_vma; @@ -584,13 +620,13 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) if (!mm || !mmget_not_zero(mm)) return -ESRCH; - err = query_vma_setup(mm); + err = query_vma_setup(priv); if (err) { mmput(mm); return err; } - vma = query_matching_vma(mm, karg.query_addr, karg.query_flags); + vma = query_matching_vma(priv, karg.query_addr, karg.query_flags); if (IS_ERR(vma)) { err = PTR_ERR(vma); vma = NULL; @@ -675,7 +711,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) } /* unlock vma or mmap_lock, and put mm_struct before copying data to user */ - query_vma_teardown(mm, vma); + query_vma_teardown(priv); mmput(mm); if (karg.vma_name_size && copy_to_user(u64_to_user_ptr(karg.vma_name_addr), @@ -695,7 +731,7 @@ static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) return 0; out: - query_vma_teardown(mm, vma); + query_vma_teardown(priv); mmput(mm); kfree(name_buf); return err; -- 2.50.0.727.gbf7dc18ff4-goog