From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 943F7C5B543 for ; Wed, 4 Jun 2025 23:11:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2F016B00A1; Wed, 4 Jun 2025 19:11:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDFDE6B00A2; Wed, 4 Jun 2025 19:11:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCDB76B00A3; Wed, 4 Jun 2025 19:11:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8838D6B00A1 for ; Wed, 4 Jun 2025 19:11:58 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 26706C0BA2 for ; Wed, 4 Jun 2025 23:11:58 +0000 (UTC) X-FDA: 83519267916.12.3A1C30E Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 6DC85160010 for ; Wed, 4 Jun 2025 23:11:56 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mFNBXYO3; spf=pass (imf08.hostedemail.com: domain of 3u9JAaAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3u9JAaAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749078716; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=3npSTljTroS8UV+to+K7B0gejhUv63EMca56GSqeKhA=; b=ojJzpp2Z/9tS+pc66toP9hpta4GmNs/T7uE2ekbTFKIfNQVMCK0gPterenIuaeNnMHX+HW R9lj18Woemx7PMWnNz5H4WzKihOYXtWIlFcB499UxtUHATZg8LA4fdv1FTFSnqJVEFOW2d l+j3Uh92/2PhZ9Qz2QqO/cnBol8xZwM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749078716; a=rsa-sha256; cv=none; b=a1B24O5Eu28D8nADuQH92aW7pImxXaaNvJJfFWd/lMapoyPvaePEi5Pt83DlXcHUPLv7/E mKEnAoo0mxS3ZM6McoJ5XG6bgfp1r6oRGt3uPd3mdWarxpaq3WIQkOXyySFyOdvy4fPBC+ zjsbVVl9dzoWQiMkAPSDcJZip+skizY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mFNBXYO3; spf=pass (imf08.hostedemail.com: domain of 3u9JAaAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3u9JAaAYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-742a969a4d0so324167b3a.2 for ; Wed, 04 Jun 2025 16:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1749078715; x=1749683515; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=3npSTljTroS8UV+to+K7B0gejhUv63EMca56GSqeKhA=; b=mFNBXYO3RPTPSDTFNjT351KkHpkmiy1gzBbCKvHP5F9ND4Gx0YhaZvLLgvGBEs0U1I b2h9/FOPiUO3CoGE3mPAnuUk4/zACFjudb4Hpi+HWkGPpaeaEaejq+pnVtPLf8RXKjtR R5rgDdiPEQkTdDnHcZ+C1+zBxaaTQ4qlcEy7lpBWaEcchraKWfttni1mnlnep96bFcUa enIfFzrVO3zlYLwaDFDesRL+yoXfdzT3A/kZyka2E+6nVBxuJQ8BmFL1duKRS2iGMFRY 6jOsnePg2CFGms23nOD+VQXCfVl3ZR1vdAcTKqj3tpkBVBnfTDHqe64axpK5CyB2K4FD oj6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749078715; x=1749683515; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3npSTljTroS8UV+to+K7B0gejhUv63EMca56GSqeKhA=; b=lU65zJnEiajEWAsdMiybv0kftkDuzT4dM0m0Vf1YiaIyg1QKUDbZIe0ipQ/nK7FhlO IxP/XGY3ayFmIulh4/Xf66TzGdfJbq4Ptnm1EMO03F4ZJRs6NkGxGx8BppZ0Mh3SD9Iz ERR9H1/aitXitLU3h5j4QgMvOhyq2Y+NyfwXA58GUuvY0S/PDDllCdkZqX2EIWLRikVh Ednsmp3g0O7mJDz+XMAmcvg6HRFGtT1+3H9L1B5Q1Mxff1vY6aJ0djOSMwWTQKR7djna q+5JPrL6+k2keDCTnZKNeTc+JvWExEM1wDHQdT6UBogSr1gt0uuQ76NzShd2CYWy9OOy pa5A== X-Forwarded-Encrypted: i=1; AJvYcCUkAhBzImNNmAvcTtRrXTTpCVsjtyMAuJYkOr7SXN6dGzCl5lBGsXuYmx7qWKmxu4efhztxlOdpXQ==@kvack.org X-Gm-Message-State: AOJu0YxyzEh3UqVCbd8NXSYco9GT2PUFkrIbedZzLmGXr7tUfvAPN9xJ mudUXIVsHkEXxph22CZiTv6PJ4h+mc89+WZ8W58D/ffWIWqp1uLHQteR3Jhv+/Suj5c2ekgSa35 3HGDsIA== X-Google-Smtp-Source: AGHT+IE5vyjpR0b4JnX96tHDLjXZe/ay0gti7aWB7+1hKtbXecOEAJBb165hSe99jrOqgpdnI0DBhGLXyQk= X-Received: from pgac22.prod.google.com ([2002:a05:6a02:2956:b0:b2c:3dd5:8139]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6b02:b0:216:1ea0:a516 with SMTP id adf61e73a8af0-21d22d4ce34mr7310828637.41.1749078715129; Wed, 04 Jun 2025 16:11:55 -0700 (PDT) Date: Wed, 4 Jun 2025 16:11:44 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.49.0.1266.g31b7d2e469-goog Message-ID: <20250604231151.799834-1-surenb@google.com> Subject: [PATCH v4 0/7] use per-vma locks for /proc/pid/maps reads and PROCMAP_QUERY From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, david@redhat.com, vbabka@suse.cz, peterx@redhat.com, jannh@google.com, hannes@cmpxchg.org, mhocko@kernel.org, paulmck@kernel.org, shuah@kernel.org, adobriyan@gmail.com, brauner@kernel.org, josef@toxicpanda.com, yebin10@huawei.com, linux@weissschuh.net, willy@infradead.org, osalvador@suse.de, andrii@kernel.org, ryan.roberts@arm.com, christophe.leroy@csgroup.eu, tjmercier@google.com, kaleshsingh@google.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, surenb@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6DC85160010 X-Stat-Signature: 7xsm7n5hmuqwnr3yqrximexxj3chezgz X-Rspam-User: X-HE-Tag: 1749078716-538170 X-HE-Meta: U2FsdGVkX1+6+mux7tqXvpr+Q8DA/XcJ14/sOtDdFJJEkn31I9Csqvyx67Q3b4jX/ujZNHg6hgXUoaEyLCVJskfqhgnfyWJ7nv4kX/7yEEoCW5mfU1CtS68SK+jAEzsM0jaG5A+7yBsbWsWLQ6u1+LsR/jT4BOV4LNT708rroT0NKeIjDeoLpL2aCCC9LANpVgWIfiADKBVr4O+JS6zhqjeC0XZ9qTwUZP2ZtF9HcGbWj9iPiwxObaLU+laFN7r45QeXhQ7y3Rg3O7R3gwxRzwemtIMojF2hB6kVLgpvhK8hI6YX9OWuQvwLnRNt930rnRIcIvlr4IoaGGeU21NuroLSWmKxIEWyAowJajYEamkKI6ogQv98HiGcX+aRgaR9lRwIXI87JBDCH0WkPKef0sLEFZIrfMk+xL2AjFPZGdOib/Pn4OCwIWqkAmhkNMx/pVeTC7SgTrq2zc3x9P7CyIiOksQHbg+wQIOVUe8kSVz+/PaiO2n/OwEVsdHvaeDj2xcEqVXqzpkzDKq6a+bhl43299dUkZ9NU9Rii+II8KcwvpHryjZaQu2wolCG+rAUy+j3os5lyRMW98luVt+TkB/Lv1NgE9eoYgGjnyhKYI0Jel3Ip6JvQ83qj1RUPkeZkJmSq1SAF7P4S/VCSYMmu8TRQco2tyIjwvynZbRqY6mZKOEkF0X1nR/Tc9wNQvdeYsYYvlkBL7Z06rnzhb2IsTrb9qYlUwd+mZQ7DY7bmIIVvRbcpoga8XcDeHK3xhQcQXWtb30qwbUwk4m1zNwVW1isUsa2qwmV0yNg/lWrOfNPxXGVbMat7EfBIKIesZ/cTWP3PSXE2BcOaRIGRBE4w9hDXRHnZX9y0XhVo8xsadXR5LT53cbzjT9jJQBvu8J/QXRf20k/JjzeuZtbDsDxi+NuVxudfLPLL+QKovp4ZLKwp63EN04NedgkRb976QwYvasaf5YqShgOEJ9Am9w EpQNt+3k pEkcw0M7uAvctFbEmBUOA71xzAU90d/XkFfXQNWBOkNjIM8vFoQtSkqURc63VQPHv9TE19mhdNESu4oh3w4kMugGSBQrs0RQ5Ay+Wj/LFHrtG5NCDSF8ZE5KVj2Oy3f4jTcS3BGiFgc50mo7xZVZISh6HaSKwWD8fL3WQqpIpIsUaYUi6P8lME9KPDzvzVMP6mwxEeI11JBMIGbzcz3zqk1V4InKUqbPNPh4SMg2E2+FQvQ08XwlDHNWt0Q57Aq8w/+JTtLUzVcmzC3bMZq08y50KlrnkFo02kiQ/c9oCaI1pabKTzfaRpmWVNT/V4b5TjWAimQpap5kr4AgVJoI/FpOgvmNDUL08ai8hGhbI6n1DBfWtcoiG1ZlYL2mXIv0qW7nUhHIHeppe+Rvl3fLJO7CJxMxfOUGxPD3G9yOY1bhE6pXwoW4HolvF5y3BnOVvdrLN8/URC+gzUXYyOp/uoJCnFRD+7Je2XJJKzbSXPoFX90EkPWPobOkaSLtaokTnoEGwB8FCh5NN7WFV9pHm63S6yy8WRVnZpMhfeof3ZY9VTKVBT61vjMbdX6d2GBcj7kBwMF9m29J5BHXLTz1vSmjK39ipA6v265jCPlCJhhKO51vCtNmKVEe1hGyCbSzNMkhE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Reading /proc/pid/maps requires read-locking mmap_lock which prevents any other task from concurrently modifying the address space. This guarantees coherent reporting of virtual address ranges, however it can block important updates from happening. Oftentimes /proc/pid/maps readers are low priority monitoring tasks and them blocking high priority tasks results in priority inversion. Locking the entire address space is required to present fully coherent picture of the address space, however even current implementation does not strictly guarantee that by outputting vmas in page-size chunks and dropping mmap_lock in between each chunk. Address space modifications are possible while mmap_lock is dropped and userspace reading the content is expected to deal with possible concurrent address space modifications. Considering these relaxed rules, holding mmap_lock is not strictly needed as long as we can guarantee that a concurrently modified vma is reported either in its original form or after it was modified. This patchset switches from holding mmap_lock while reading /proc/pid/maps to taking per-vma locks as we walk the vma tree. This reduces the contention with tasks modifying the address space because they would have to contend for the same vma as opposed to the entire address space. Same is done for PROCMAP_QUERY ioctl which locks only the vma that fell into the requested range instead of the entire address space. Previous version of this patchset [1] tried to perform /proc/pid/maps reading under RCU, however its implementation is quite complex and the results are worse than the new version because it still relied on mmap_lock speculation which retries if any part of the address space gets modified. New implementaion is both simpler and results in less contention. Note that similar approach would not work for /proc/pid/smaps reading as it also walks the page table and that's not RCU-safe. Paul McKenney's designed a test [2] to measure mmap/munmap latencies while concurrently reading /proc/pid/maps. The test has a pair of processes scanning /proc/PID/maps, and another process unmapping and remapping 4K pages from a 128MB range of anonymous memory. At the end of each 10 second run, the latency of each mmap() or munmap() operation is measured, and for each run the maximum and mean latency is printed. The map/unmap process is started first, its PID is passed to the scanners, and then the map/unmap process waits until both scanners are running before starting its timed test. The scanners keep scanning until the specified /proc/PID/maps file disappears. This test registered close to 10x improvement in update latencies: Before the change: ./run-proc-vs-map.sh --nsamples 100 --rawdata -- --busyduration 2 0.011 0.008 0.455 0.011 0.008 0.472 0.011 0.008 0.535 0.011 0.009 0.545 ... 0.011 0.014 2.875 0.011 0.014 2.913 0.011 0.014 3.007 0.011 0.015 3.018 After the change: ./run-proc-vs-map.sh --nsamples 100 --rawdata -- --busyduration 2 0.006 0.005 0.036 0.006 0.005 0.039 0.006 0.005 0.039 0.006 0.005 0.039 ... 0.006 0.006 0.403 0.006 0.006 0.474 0.006 0.006 0.479 0.006 0.006 0.498 The patchset also adds a number of tests to check for /proc/pid/maps data coherency. They are designed to detect any unexpected data tearing while performing some common address space modifications (vma split, resize and remap). Even before these changes, reading /proc/pid/maps might have inconsistent data because the file is read page-by-page with mmap_lock being dropped between the pages. An example of user-visible inconsistency can be that the same vma is printed twice: once before it was modified and then after the modifications. For example if vma was extended, it might be found and reported twice. What is not expected is to see a gap where there should have been a vma both before and after modification. This patchset increases the chances of such tearing, therefore it's event more important now to test for unexpected inconsistencies. [1] https://lore.kernel.org/all/20250418174959.1431962-1-surenb@google.com/ [2] https://github.com/paulmckrcu/proc-mmap_sem-test Suren Baghdasaryan (7): selftests/proc: add /proc/pid/maps tearing from vma split test selftests/proc: extend /proc/pid/maps tearing test to include vma resizing selftests/proc: extend /proc/pid/maps tearing test to include vma remapping selftests/proc: test PROCMAP_QUERY ioctl while vma is concurrently modified selftests/proc: add verbose more for tests to facilitate debugging mm/maps: read proc/pid/maps under per-vma lock mm/maps: execute PROCMAP_QUERY ioctl under per-vma locks fs/proc/internal.h | 6 + fs/proc/task_mmu.c | 233 +++++- tools/testing/selftests/proc/proc-pid-vm.c | 793 ++++++++++++++++++++- 3 files changed, 1011 insertions(+), 21 deletions(-) base-commit: 2d0c297637e7d59771c1533847c666cdddc19884 -- 2.49.0.1266.g31b7d2e469-goog