From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B6A9C02183 for ; Fri, 17 Jan 2025 16:13:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F12E36B0082; Fri, 17 Jan 2025 11:13:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EC33D6B0085; Fri, 17 Jan 2025 11:13:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8A896B008C; Fri, 17 Jan 2025 11:13:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BB2596B0082 for ; Fri, 17 Jan 2025 11:13:40 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 746D68053C for ; Fri, 17 Jan 2025 16:13:40 +0000 (UTC) X-FDA: 83017439400.21.051F386 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 442B540014 for ; Fri, 17 Jan 2025 16:13:38 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Hz3dFp4I; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737130419; a=rsa-sha256; cv=none; b=3+qfxQyVfj/sugyXIlBPK1yry9e9xv4md9g0yL/HHZt2mBddqrg5LwxhVj/sOY3E+2F9+4 AX7SyEdVDSQK+l9AEp/lUCgyZ86DSzPp5mnNtAf/3Piz/oFx5w8IpSrFpM5q085KEDhiPj v51q8ivoRR0tGZJT9UN81A/lJLZsTio= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Hz3dFp4I; dmarc=none; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737130418; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PZ+0eV5Tl2QBZ5nlEFCXeRqgvcPKGVoHTW6KrpY0SOo=; b=wr7Oac3K+gdCX2+baQpXrQTZFHd+4qEUOFHtyx8O/pzTo+ePfvun9bCkYlHKWBHekND85+ +XKCG1xubSSbsgr5hMQNb7FBvNJ6Z+4qhiyHAYO8v9nMDM3Oq8oM+EV9VzNhk2jG9ChAFQ kq4dLmG4JyGDR8NhtmfQFcn0+VSFNWk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=PZ+0eV5Tl2QBZ5nlEFCXeRqgvcPKGVoHTW6KrpY0SOo=; b=Hz3dFp4II8RFwyg0NPGIqTbz+p eZ240rr/LgdcMP68NMxP6as+JlvN2aSedYfCbd8ThhJyvQVtD1O9XOlhmSlAl7wGvRkN4jfII3f7y HHGVuXHodRGI5UWJ30yf/HsPuwTj3XMy5outOICLQfGXdtVfp+n8luDnBHA9/UC276KhaUI/dodq5 n4MqE2s2ywqejDtOtxOaS0hH0k7v71Q3DcS94622gLh2GoYK91wdq1478WbM9S/WMe//5wjSN7Tbl GOEumt3/fyXbyCMBVAx1lmpOiSVuY5CEJwq3kp+zWmu58YxdqFnUdl72RHj2UjaeHQ+7T3Et0GPDH sQqiyfMw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tYoyC-0000000Fc3u-1csH; Fri, 17 Jan 2025 16:13:16 +0000 Date: Fri, 17 Jan 2025 16:13:16 +0000 From: Matthew Wilcox To: Peter Zijlstra Cc: Suren Baghdasaryan , will@kernel.org, boqun.feng@gmail.com, mark.rutland@arm.com, Mateusz Guzik , akpm@linux-foundation.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com Subject: Re: [PATCH] refcount: Strengthen inc_not_zero() Message-ID: References: <20250111042604.3230628-1-surenb@google.com> <20250111042604.3230628-12-surenb@google.com> <20250115104841.GX5388@noisy.programming.kicks-ass.net> <20250115111334.GE8385@noisy.programming.kicks-ass.net> <20250115160011.GG8385@noisy.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250115160011.GG8385@noisy.programming.kicks-ass.net> X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 442B540014 X-Stat-Signature: axy5no16jbmc7okohtjht4mrctdg43ah X-Rspam-User: X-HE-Tag: 1737130418-538324 X-HE-Meta: U2FsdGVkX1/AP62DRA0Z+ty6fM5eBomNTLtBSdrzmrk33pXJLZ+AFQJr2HSACG9Xk6tTmacGXuc2wTwIM1KzHGkXxUDirw7dPMbmHe5dprZM3pE/dck7c2yE6X7j9gNVJUsWk7yK1+3xey4ZJn2Cj7/vDVSTgleVfjxK/owsT+HHuUQMTToKgaHP6zD4ktApqtuMsw8HSGEAGIGZlIZ62kkZhwo/yi5swDcNhPdjwBSOah5Wbb3xth4pMMRVaq9JpU5G/4iQwcIydDkZ0tFc5evDubBH6sBJhml39FLW3Tpp34iT4bnYF1YLAWGkeHA0hryvJqISQJgtkyN/F8uzcYVQgD9PdPd9jf9ajCXG+dr2wL/F/tr+/uezT+gZuRtqApCwyhzqL3ZRQt7ubBjvTYXqIJdCDEnr+XYYNGAUmAUGOPTl/A8EP5YfQ1KOICK1QTgwPy9KtCMu2SNYX3CllB2lHiQwqSW9hEoCU5uVSvV4E1+fF9c802fzZWkZq5QH0J5GCqLTZyvCa/fX6uCKMAkf/lsftCLmcqHyCxrov2ykNDqy/YIovDHzH1HSrGNho8RE5GT30bNHgiFXOiTP141FLKMDQQE9EWCAkKnWNvwQHonN70PMbw7+YahQBkoK66OqSiMo2TgOGWOBEXD4ok62uaytRIxKqI8fn3l104zhqdsiEmodGPdNDTIBLJ9L7hWs8OLQImKyHFIKW8thp6uBJUhgBo9Hv6U3u1wBuhfsbdODVNAtpkeLU2oCU3IBo8oj0TfSIHckLJE0sf/pvWtCw5hFxOlvXn5L2TeH4m9V/EkBMDNA6vxXew2AXxF8eyPShGsPGWL2JYcsgAEJpYoSMQ5G7YBpz5H6DN5x1UOBPbQRnrVyOcA7zLms0rK6FrtHvdOCrttK1EIgpf7F+pYNvO1EqTVte7tr1bQoJWTETeoccRgSatTwxgsbKjPA1c+xuDsWJaFFkNZvVzI 0TNun7+S 0K2gvYXWOaVEBauxGVewbD/Ae06K3hr+knTokRfuC1DGzM7uEZ9QHso86iIREoW4IHcNA0JiMDOszyEhOqXf4V3lu0AZ/xaruYkIr5ZhX34eb4B1kuB95FaIocNic1EwZ/QYkgB9r3DfifbwXuufgxNxrva0T8tE262MIgQGUhEo+SttMSb1oaAzdXcgu74FcWos3vWPhvPZSlT/NqR9p5fXjX0eyW5Ss9zeRXW19kn5nmR6Nl91wgwq8d5jMgFWID/GJWzdAdssS7sRSJgX7ThQVLMj5c1GIBR/fYd6wMYGbTlVNpDa2vIoqp9jA8zIybET4u/yBBKTPhsHYM0XO3w+/sD+v+Wx7kbFb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 15, 2025 at 05:00:11PM +0100, Peter Zijlstra wrote: > Subject: refcount: Strengthen inc_not_zero() > > For speculative lookups where a successful inc_not_zero() pins the > object, but where we still need to double check if the object acquired > is indeed the one we set out to aquire, needs this validation to happen > *after* the increment. > > Notably SLAB_TYPESAFE_BY_RCU is one such an example. While you're looking at inc_not_zero(), have you already thought about doing something like this? ie failing rather than saturating since all users of this already have to check for failure. It looks like two comparisons per call rather than three. diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 35f039ecb272..3ef7d316e870 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -142,16 +142,13 @@ bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) int old = refcount_read(r); do { - if (!old) + if (old <= 0 || old + i < 0) break; } while (!atomic_try_cmpxchg_relaxed(&r->refs, &old, old + i)); if (oldp) *oldp = old; - if (unlikely(old < 0 || old + i < 0)) - refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF); - return old; } $ ./scripts/bloat-o-meter before.o after.o add/remove: 0/0 grow/shrink: 0/4 up/down: 0/-91 (-91) Function old new delta io_wq_for_each_worker.isra 162 158 -4 io_worker_handle_work 1403 1387 -16 io_wq_activate_free_worker 187 158 -29 io_queue_worker_create 367 325 -42 Total: Before=10068, After=9977, chg -0.90% (that's io_uring/io-wq.o as an example user of refcount_inc_not_zero())