From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF4C6C3ABA9 for ; Thu, 1 May 2025 03:27:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68CD76B009C; Wed, 30 Apr 2025 23:27:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 615ED6B009D; Wed, 30 Apr 2025 23:27:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B5AC6B009E; Wed, 30 Apr 2025 23:27:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 284A76B009C for ; Wed, 30 Apr 2025 23:27:44 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C8EB38181C for ; Thu, 1 May 2025 03:27:44 +0000 (UTC) X-FDA: 83392904448.25.C391A78 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf18.hostedemail.com (Postfix) with ESMTP id E9B821C0006 for ; Thu, 1 May 2025 03:27:42 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WHStRRIW; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746070063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wX6llm03C0LIofZKVhngpskpWSkctaSgtaEvqaJgRKA=; b=LSgRneg8Vg5YFrszuIH7tKZ7vFFFxjGLTyQsKRPbsd7rENW8uPV7qBCj68DD4kiG2+G4E/ eFm8VhJlEB7/WGEkwqYU20060QkK+XfnRypUR6/8fEijMF7J+ga2H6AU3WrBY1knqEduHr SXdSW/PXdKmrU27N8kFrFRcyD5MaRgE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WHStRRIW; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746070063; a=rsa-sha256; cv=none; b=obdspieC92pwNjY9BvzhxfvD3sqcJ5ZiW7flhd0OdZFCVobrxWV/bdwnlKC98Kmd9oB2x/ 1ZH+sYs2neouEO4+q0OVwEc/Yq436L26Or8ADO1r6Tqd9LxiCPKTqFyhdLZftjsb8fpZDL SAL+Dhzr41mbrpbWdxhV/RGKaNR6dQs= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-73c17c770a7so792570b3a.2 for ; Wed, 30 Apr 2025 20:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746070062; x=1746674862; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wX6llm03C0LIofZKVhngpskpWSkctaSgtaEvqaJgRKA=; b=WHStRRIWBbtO3wL9IPJo0bnWytWv9dqsZ1C9BLUei/vxYndiqJiXq12jCvCPX10YYv +8y4uza7pzmbRfG+zaXzbjy/mSXpo2NxLZor3LfbYLFwEIo6i49QgIQhSH/JXrPLhQzQ jTTmCafggstIghGhwMpB5MGH+N4I+r5HvNLAOTFFz6VfA7xET9AP3w9CNdsxuw6bHytJ Rb3GpKn7OoOMO3y5O5uy/U9QHhJfBftXmQ7IMVGN9YJ5lkqqkCh6xNec4iACcXnlGZMF gt0SatiWYSSOkAgK3uGiNJFuIoTnSn956cCSnNws7xtubg2QxjJJXh+LE+BA1MYI7HYx 6wqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746070062; x=1746674862; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wX6llm03C0LIofZKVhngpskpWSkctaSgtaEvqaJgRKA=; b=ilF5BDXmdXsQQoaAD4JxNkDR3luyXksdWTYHrrEfgnEwvILDQxIyA3UGBXzsytCM2b W7gP5yXSBc1bAr7CrLlSi/dPXRRje7qlAD9XwdBvkMiiOQFgoueDet8SHjPtJ5CVDmln 9uLgzBqAdoEJX47aEu1uCqq9dVu4cZ4Ga6cTufsLK8vZq3mxIvV/XhuZb6HnOTgtFznf ue/sJTBxDGkxGZpSOSh40HCQsdC2eK3u44BzTlFil+bmfpNokQ68Jv0Wy1aJD9vg7zjS A7ARADO6ZJ+SrClf1LEuI9oZnI/53LCz74vOcSUhpm6EaPB8i+MJTqT3W0HczHZYbikk NO4A== X-Forwarded-Encrypted: i=1; AJvYcCXaSzfsJTNCX4GyW8f/cPyiOznPO2UWzQ85fmMDkNIk0T8dX8Sf0LoCBd0/SmllucsyLpCqczoLEw==@kvack.org X-Gm-Message-State: AOJu0YxvYnckQ3eXtaIKTosVwkUUFm/AKgVlbIAS6hiL1+IM70/+yFyr Jetk06+mMs71KLkIJUU7OyFkKVDbRLPXWynCYNAFzys8aqjlETrj X-Gm-Gg: ASbGncswMfycbCXZ/pAFqxyl6jGB9FBrwUDUseXzO9L8HyjxDcwdFvpGoC55FVchpb0 znwcvexZqyqLwp1dZfs0KIETO0J0bSj3D4VjV5my0nE/YwxAMXsSnFFc3ZRZTHREXi8qk3zNYTE waVrvbvUYLFu8O8KO6cN6HkO95NQ1F0taJw+RxzH5DbqkSbWUrSS4vhrLePF9jzoxP5zD/LkfQw i6kId6dvvQhf3eoztsSSMzMT88WN0s2f+SoVG1472vqIIU6bmvO94Fmb5muxqjn40kvD/8jo6h8 CiBMKAiG7IVkvxVMueBHTHQ2SSK5qDsPbZG3mM5m/Nbv+pUPnBLdluVYki3i13qp5lhb X-Google-Smtp-Source: AGHT+IGnPb76TyUdO9Ok/eAIZ2iA6j3BcFpbdz59/hEoRzxhRnG5xDhSPIVqOw3bH3ISf3RLwvXNzw== X-Received: by 2002:a05:6a20:d705:b0:1f5:8262:2c0b with SMTP id adf61e73a8af0-20bd6656f44mr1758597637.2.1746070061886; Wed, 30 Apr 2025 20:27:41 -0700 (PDT) Received: from localhost.localdomain ([2620:10d:c090:400::5:13f8]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-740398f9aa0sm2616253b3a.10.2025.04.30.20.27.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 30 Apr 2025 20:27:41 -0700 (PDT) From: Alexei Starovoitov To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: vbabka@suse.cz, harry.yoo@oracle.com, shakeel.butt@linux.dev, mhocko@suse.com, bigeasy@linutronix.de, andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, rostedt@goodmis.org, hannes@cmpxchg.org, willy@infradead.org Subject: [PATCH 4/6] locking/local_lock: Introduce local_lock_irqsave_check() Date: Wed, 30 Apr 2025 20:27:16 -0700 Message-Id: <20250501032718.65476-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250501032718.65476-1-alexei.starovoitov@gmail.com> References: <20250501032718.65476-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: yygajhfqr3i6j3ok4zj51fwpkgmy3qyg X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E9B821C0006 X-Rspam-User: X-HE-Tag: 1746070062-504185 X-HE-Meta: U2FsdGVkX1+gBFB2wguzkedeBWdtGOJXunPRulvB5Pe8uVb+k8ztsOCLOUmulS2bxVaFnYjZIP954I/vv5aJtan1KtaIVFh6auZ7iYKBpDlEHuKgdPkiWXjWxNYm1nDz21O447Wbx5jRXY76RgH++ISu1uy+8h2CH8faRdcA6bTk6R/Zs3SiQShQlB1oA9IUzRvR77D1y2MgXRZB9gOMxPTm2SEf2HjVRLr5+i/88APHytHKRjbk6AovmK3Dp1N+ExuNuaekqa0Yz6tyv49xetQe0xjWP7lWA7Qnu1HwNUylI+/FM1GisgJoHmOolIZqlHTTbcLy2jZxjCkS/703Jitccc/8BMPNZCN3TEZBWIvaa46XEwOOBpI0PKV0lE2lSOP3u9fOBpkoK9r5V4DM0nDXLqZl/YGbBfX3b9thhrMoqVlwWHGhCYE2evfy/9TXhVtBXoFMT1DGvf59R26tDM/cX4A1FqMc8fesQ21z40a9DUbyuICB6NqVLJXAtrMC1auLQbpKmaYbVigX9eSqSuSOnkaQTwPGSQ+hWUONGCE2gzwoHsGldfp48eIxDml74VPNNwS9wCkCs30uYCF642I1noqYAGbmJ0eqtot+6nSFDYEfcNRRCFo974tyHKTDHPVC05w1VJ9iwklvghNW/b1Z9X8hbZy25UUQaMu3FhTOt/rl2D1oKM0wT7dNi78+HkbiLrKg14tc3HDE+T90mUwP1NsmJufmMv7lXtHzzFiRxxiXKLZUP+kgmDwAc4shESCf65IWCxJczn3Du8zyqfH9+sIPgjgBDcSnPP1BmcYrccejiVQGCY59wR2Rt24zAP5plFlt/rDNmMASqM14OobTTmpkRte4WFUBMhozuGdQ6p4p3ZEws/9OW3y/TPSKQ7y4lWaOaCoH9m3l6qyBYEUG1W30dbPo4HAKdVJ/cxQskpjnNtoXN2DFd495sq+pGomcfE2hiu7t0iug5vB W+zzuBr8 KjG99qfAD77/Q9R5/He1arKOExXtg838/qL1NrwQetYmH++j/4QtOqkNoXY1f2iBnKqIP5KUrHn6GDokoaMNy09Qsc3PEvVWwd33xJMo5beW9gYat1n2OxCbt2ZdZAIANBNOz/1V87j8w3/RVoVA/VKQaSCCUdYZunU7tv8uNkmLoRRjqV1n1RZzxEUNBR8RJfBVurplYTE8ylKEWi7zcJGXttWGpUFYXFrJWYkbx6ULFX4WITgGJ0ldk7TgCDOAEaBlMoWzhCHdhzo7Q7RbM1oz+6HcoJBphXzNkexKP79TsttresaGLqNL/enVBcEV5v0iWhaOnW8xVEHso2NP9FHMcxaz9jLA04NAjqMZAvFz1knyLAf7914vj53T9dAGAzTFCALt7swc30GJ3gg4IPkAczJQHSVNcyWyJfNn+Yk84UDUPTpfiAxaSQhFlXr+03HT4HwYQD+BbBYJo/NdPBbrMfVso+txZ6LqGp+LOS8WnK85Oeadj0IV4zg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce local_lock_irqsave_check() to check that local_lock is not taken recursively. In !PREEMPT_RT local_lock_irqsave() disables IRQ, but re-entrance is possible either from NMI or strategically placed kprobe. The code should call local_lock_is_locked() before proceeding to acquire a local_lock. Such local_lock_is_locked() might be called earlier in the call graph and there could be a lot of code between local_lock_is_locked() and local_lock_irqsave_check(). Without CONFIG_DEBUG_LOCK_ALLOC the local_lock_irqsave_check() is equivalent to local_lock_irqsave(). Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 13 +++++++++++++ include/linux/local_lock_internal.h | 19 +++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 092ce89b162a..0d6efb0fdd15 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -81,6 +81,19 @@ #define local_trylock_irqsave(lock, flags) \ __local_trylock_irqsave(lock, flags) +/** + * local_lock_irqsave_check - Acquire a per CPU local lock, save and disable + * interrupts + * @lock: The lock variable + * @flags: Storage for interrupt flags + * + * This function checks that local_lock is not taken recursively. + * In !PREEMPT_RT re-entrance is possible either from NMI or kprobe. + * In PREEMPT_RT it checks that current task is not holding it. + */ +#define local_lock_irqsave_check(lock, flags) \ + __local_lock_irqsave_check(lock, flags) + DEFINE_GUARD(local_lock, local_lock_t __percpu*, local_lock(_T), local_unlock(_T)) diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 263723a45ecd..7c4cc002bc68 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -168,6 +168,15 @@ do { \ /* preemption or migration must be disabled before calling __local_lock_is_locked */ #define __local_lock_is_locked(lock) READ_ONCE(this_cpu_ptr(lock)->acquired) +#define __local_lock_irqsave_check(lock, flags) \ + do { \ + if (IS_ENABLED(CONFIG_DEBUG_LOCK_ALLOC) && \ + (!__local_lock_is_locked(lock) || in_nmi())) \ + WARN_ON_ONCE(!__local_trylock_irqsave(lock, flags)); \ + else \ + __local_lock_irqsave(lock, flags); \ + } while (0) + #define __local_lock_release(lock) \ do { \ local_trylock_t *tl; \ @@ -293,4 +302,14 @@ do { \ #define __local_lock_is_locked(__lock) \ (rt_mutex_owner(&this_cpu_ptr(__lock)->lock) == current) +#define __local_lock_irqsave_check(lock, flags) \ + do { \ + typecheck(unsigned long, flags); \ + flags = 0; \ + migrate_disable(); \ + if (IS_ENABLED(CONFIG_DEBUG_LOCK_ALLOC)) \ + WARN_ON_ONCE(__local_lock_is_locked(lock)); \ + spin_lock(this_cpu_ptr((lock))); \ + } while (0) + #endif /* CONFIG_PREEMPT_RT */ -- 2.47.1