From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A6AC433EF for ; Tue, 5 Oct 2021 05:41:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BEFC760F5D for ; Tue, 5 Oct 2021 05:41:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org BEFC760F5D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 64D766B0075; Tue, 5 Oct 2021 01:41:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D6576B0078; Tue, 5 Oct 2021 01:41:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 450806B007B; Tue, 5 Oct 2021 01:41:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 37B586B0075 for ; Tue, 5 Oct 2021 01:41:14 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EF85A1818CC08 for ; Tue, 5 Oct 2021 05:41:13 +0000 (UTC) X-FDA: 78661285626.27.259F9CC Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf19.hostedemail.com (Postfix) with ESMTP id A5996B001CD1 for ; Tue, 5 Oct 2021 05:41:13 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id g13-20020a17090a3c8d00b00196286963b9so1092067pjc.3 for ; Mon, 04 Oct 2021 22:41:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FIPuS4OLzQCF5AgK/VmYDt3q+TnxbO8XesSIDRxgCzY=; b=IB/e9E/C1kr8o0K5p8DzE62VkGgywadLxe9xGEIVZ9iGC/sIxNgo2/87mURnDogKP7 m8HBiNbsUvDrIoE43mIFRcXZSGZ9tctHdgAwm3etPMRH6gJs8iyQcs6qegXbIu+l44O9 7HoVSFbODN5nUtrtzlzDy3nKiDJDLm6qbTXFZR5BCvQCuG4DCx7b/G23X5rN+kYfgp1u s1xj1BKNkrR5a9CcA+ETt9hwhj1A2+wJaxnJ70fiYtihkSPJbEZTFOkXgw5FF7UpTfjo tfncNoW/B7FETJce8ZEtvy7GWga+cqQ76Og9QapjpdZBbFOdSIGTs7khxf/6tSqZAV3n oU5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FIPuS4OLzQCF5AgK/VmYDt3q+TnxbO8XesSIDRxgCzY=; b=0oMqqow4pp9h46TwIDlBmIkfh2gbx3xL5jLUiqocENPPViTWiNmd8rhnj0U2xLG1yj D+cCxp+3/MQTetzk2Z7txx9K3Ya4xIHxGBfmJnGlGCMIfC2FZ2f3aljYjjSgdae2DElP EiwPK0As92CNBlUYdpSCqsIKvwblGjz/8IBTUz/Xm1rBMhl0aSfmY3sLOS9XZd85rtam WWX6TLZiMEk3SMiFvGjhtu351Fz7gQdS5nd5ndFfL/Shp0f551uBhNqYrEHAg4ejFnIc L3vN7uZPebtls+FE92RTXN8fTOpk7d1UmIP1v3OZYT9lHmrb5uVwxBk8+nClQWExUmpJ e9Ug== X-Gm-Message-State: AOAM532J2gYzth2u3CLpoVPcHtn1quX663anqQwNGLnMsl8t7rEL1VM5 QY+GVXVuUwlnxWZcAAu92R4= X-Google-Smtp-Source: ABdhPJz8WyWYoFwsoQMeCU3ssjLVOLt9VS4oUnytD2/OaWtFurLWyTtrhH5Pr8dODfpZCySuXm3Ejw== X-Received: by 2002:a17:902:c78a:b0:13e:cac2:4b41 with SMTP id w10-20020a170902c78a00b0013ecac24b41mr3494570pla.17.1633412472703; Mon, 04 Oct 2021 22:41:12 -0700 (PDT) Received: from localhost (searspoint.nvidia.com. [216.228.112.21]) by smtp.gmail.com with ESMTPSA id j12sm16719053pfj.206.2021.10.04.22.41.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Oct 2021 22:41:12 -0700 (PDT) From: Yury Norov To: Stephen Rothwell Cc: Yury Norov , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mmc@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, "James E.J. Bottomley" , Alexander Lobakin , Alexander Shishkin , Alexey Klimov , Andrea Merello , Andy Shevchenko , Arnaldo Carvalho de Melo , Arnd Bergmann , Ben Gardon , Benjamin Herrenschmidt , Brian Cain , Catalin Marinas , Christoph Lameter , Daniel Bristot de Oliveira , David Hildenbrand , Dennis Zhou , Geert Uytterhoeven , Heiko Carstens , Ian Rogers , Ingo Molnar , Jaegeuk Kim , Jakub Kicinski , Jiri Olsa , Joe Perches , Jonas Bonn , Leo Yan , Mark Rutland , Namhyung Kim , Palmer Dabbelt , Paolo Bonzini , Peter Xu , Peter Zijlstra , Petr Mladek , Rasmus Villemoes , Rich Felker , Samuel Mendoza-Jonas , Sean Christopherson , Sergey Senozhatsky , Shuah Khan , Stefan Kristiansson , Steven Rostedt , Tejun Heo , Thomas Bogendoerfer , Ulf Hansson , Will Deacon , Wolfram Sang , Yoshinori Sato Subject: [PATCH 05/16] lib: add find_first_and_bit() Date: Mon, 4 Oct 2021 22:40:48 -0700 Message-Id: <20211005054059.475634-6-yury.norov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211005054059.475634-1-yury.norov@gmail.com> References: <20211005054059.475634-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A5996B001CD1 X-Stat-Signature: 8zydqrtdy6ngwkdbnxr9oo4j5ph8nfzd Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="IB/e9E/C"; spf=pass (imf19.hostedemail.com: domain of yury.norov@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=yury.norov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1633412473-573156 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently find_first_and_bit() is an alias to find_next_and_bit(). Howeve= r, it is widely used in cpumask, so it worth to optimize it. This patch adds its own implementation for find_first_and_bit(). On x86_64 find_bit_benchmark says: Before (#define find_first_and_bit(...) find_next_and_bit(..., 0): Start testing find_bit() with random-filled bitmap [ 140.291468] find_first_and_bit: 46890919 ns, 32671 iteratio= ns Start testing find_bit() with sparse bitmap [ 140.295028] find_first_and_bit: 7103 ns, 1 iteratio= ns After: Start testing find_bit() with random-filled bitmap [ 162.574907] find_first_and_bit: 25045813 ns, 32846 iteratio= ns Start testing find_bit() with sparse bitmap [ 162.578458] find_first_and_bit: 4900 ns, 1 iteratio= ns (Thanks to Alexey Klimov for thorough testing.) Signed-off-by: Yury Norov Tested-by: Wolfram Sang Tested-by: Alexey Klimov --- include/linux/find.h | 27 +++++++++++++++++++++++++++ lib/find_bit.c | 21 +++++++++++++++++++++ lib/find_bit_benchmark.c | 21 +++++++++++++++++++++ 3 files changed, 69 insertions(+) diff --git a/include/linux/find.h b/include/linux/find.h index ea57f7f38c49..6048f8c97418 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -12,6 +12,8 @@ extern unsigned long _find_next_bit(const unsigned long= *addr1, const unsigned long *addr2, unsigned long nbits, unsigned long start, unsigned long invert, unsigned long le); extern unsigned long _find_first_bit(const unsigned long *addr, unsigned= long size); +extern unsigned long _find_first_and_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long size); extern unsigned long _find_first_zero_bit(const unsigned long *addr, uns= igned long size); extern unsigned long _find_last_bit(const unsigned long *addr, unsigned = long size); =20 @@ -123,6 +125,31 @@ unsigned long find_first_bit(const unsigned long *ad= dr, unsigned long size) } #endif =20 +#ifndef find_first_and_bit +/** + * find_first_and_bit - find the first set bit in both memory regions + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @size: The bitmap size in bits + * + * Returns the bit number for the next set bit + * If no bits are set, returns @size. + */ +static inline +unsigned long find_first_and_bit(const unsigned long *addr1, + const unsigned long *addr2, + unsigned long size) +{ + if (small_const_nbits(size)) { + unsigned long val =3D *addr1 & *addr2 & GENMASK(size - 1, 0); + + return val ? __ffs(val) : size; + } + + return _find_first_and_bit(addr1, addr2, size); +} +#endif + #ifndef find_first_zero_bit /** * find_first_zero_bit - find the first cleared bit in a memory region diff --git a/lib/find_bit.c b/lib/find_bit.c index 0f8e2e369b1d..1b8e4b2a9cba 100644 --- a/lib/find_bit.c +++ b/lib/find_bit.c @@ -89,6 +89,27 @@ unsigned long _find_first_bit(const unsigned long *add= r, unsigned long size) EXPORT_SYMBOL(_find_first_bit); #endif =20 +#ifndef find_first_and_bit +/* + * Find the first set bit in two memory regions. + */ +unsigned long _find_first_and_bit(const unsigned long *addr1, + const unsigned long *addr2, + unsigned long size) +{ + unsigned long idx, val; + + for (idx =3D 0; idx * BITS_PER_LONG < size; idx++) { + val =3D addr1[idx] & addr2[idx]; + if (val) + return min(idx * BITS_PER_LONG + __ffs(val), size); + } + + return size; +} +EXPORT_SYMBOL(_find_first_and_bit); +#endif + #ifndef find_first_zero_bit /* * Find the first cleared bit in a memory region. diff --git a/lib/find_bit_benchmark.c b/lib/find_bit_benchmark.c index 5637c5711db9..db904b57d4b8 100644 --- a/lib/find_bit_benchmark.c +++ b/lib/find_bit_benchmark.c @@ -49,6 +49,25 @@ static int __init test_find_first_bit(void *bitmap, un= signed long len) return 0; } =20 +static int __init test_find_first_and_bit(void *bitmap, const void *bitm= ap2, unsigned long len) +{ + static DECLARE_BITMAP(cp, BITMAP_LEN) __initdata; + unsigned long i, cnt; + ktime_t time; + + bitmap_copy(cp, bitmap, BITMAP_LEN); + + time =3D ktime_get(); + for (cnt =3D i =3D 0; i < len; cnt++) { + i =3D find_first_and_bit(cp, bitmap2, len); + __clear_bit(i, cp); + } + time =3D ktime_get() - time; + pr_err("find_first_and_bit: %18llu ns, %6ld iterations\n", time, cnt); + + return 0; +} + static int __init test_find_next_bit(const void *bitmap, unsigned long l= en) { unsigned long i, cnt; @@ -129,6 +148,7 @@ static int __init find_bit_test(void) * traverse only part of bitmap to avoid soft lockup. */ test_find_first_bit(bitmap, BITMAP_LEN / 10); + test_find_first_and_bit(bitmap, bitmap2, BITMAP_LEN / 2); test_find_next_and_bit(bitmap, bitmap2, BITMAP_LEN); =20 pr_err("\nStart testing find_bit() with sparse bitmap\n"); @@ -145,6 +165,7 @@ static int __init find_bit_test(void) test_find_next_zero_bit(bitmap, BITMAP_LEN); test_find_last_bit(bitmap, BITMAP_LEN); test_find_first_bit(bitmap, BITMAP_LEN); + test_find_first_and_bit(bitmap, bitmap2, BITMAP_LEN); test_find_next_and_bit(bitmap, bitmap2, BITMAP_LEN); =20 /* --=20 2.30.2