From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAEF2C77B7E for ; Wed, 24 May 2023 03:13:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DA4F280009; Tue, 23 May 2023 23:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66DFE280001; Tue, 23 May 2023 23:13:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44134280009; Tue, 23 May 2023 23:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 32089280001 for ; Tue, 23 May 2023 23:13:33 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0E8BBA069A for ; Wed, 24 May 2023 03:13:33 +0000 (UTC) X-FDA: 80823678306.19.E87F69D Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf14.hostedemail.com (Postfix) with ESMTP id 20E8C100006 for ; Wed, 24 May 2023 03:13:30 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JB3kp4BJ; spf=pass (imf14.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684898011; a=rsa-sha256; cv=none; b=YbWu/eQAdo0oHEHwfYzNw+2eePA+aRqBCck+A+cfZ8QVXeOKVnbRjZk+XNQdAJfN4pTFxW SQbRxG2EoWRUI7c22DLxO8AXfEwPEjJEp9cL00SZwqM6gVYZ7pogvoGz8MBBAqHHi2gYPZ kem0zo1Xk38souLTviPlk8XuVFc3d9w= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JB3kp4BJ; spf=pass (imf14.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684898011; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Sfx1V5T4mPIwVqP+RrJR35TgWmrm3iAxbhRMi/Gcd/w=; b=MttQHWxHKO3f1I8NbCXuiVediCHMbBYL96+kf1O2oJ9Vc6XBwWyJRYTBxVrUDr+HNbKlfK /ZMwkwk8Yiqyuvp05ohXgvRJqwNmIF9b0YUt5aT/2PTSEtDzFke/Ztvg2BQs0p5lxTIQbq uhb2utdc0fLjsPiSHtUKntwVThhh9o4= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1ae4c5e1388so4345425ad.1 for ; Tue, 23 May 2023 20:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1684898010; x=1687490010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sfx1V5T4mPIwVqP+RrJR35TgWmrm3iAxbhRMi/Gcd/w=; b=JB3kp4BJlOTs+7cy33BIr9gKo327JK5pA7ulU3SsjOpt3OCFJdJD7nK6aH5tKH88dJ rFNBih93HAFSgVUwvO50dFWu8MWGNidvUluFSfNOqUZpHn6gautW+XcY602ivtqiQmOd DHaviW5bqiNIiZr18jVVyRGEvqNQCnp2SPyV9uat+WK4D/AHQ4dwM2H04pC/D3F41xcv gUBrRudDRolPnEB+3e8vALOAFWlf6PovASzbMUtuHxm32FFhN4tq1uzAtRRQKIOo2+Fc En7/cwMNU5GHxKFqTz8+PyPf7Md0NP/OSjUlmMq5pNdNop7bHIIHzPJ4YWnVZKtXiwBL rHpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684898010; x=1687490010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sfx1V5T4mPIwVqP+RrJR35TgWmrm3iAxbhRMi/Gcd/w=; b=YqNYe5Z4fdN13fca/vd1EY4e7xON/40F4t0G9G5mRtldiO1j0uKdBAY9hpdJlV177V 7XXfcWERccStgTqwtKk4EnHE5JHbl9HXMd4i0F/BQVW0/8F9X8P7pG5RuwxLdh12C1xN 1QqJP+2xQOungknCrmuIUig/jKrZ0Pye5MltvAAcLzlj65mDFx2FpWSPsstMXwkrLk/x 4Mpb/8vZ8a4Fx3BtLUAU86jwoUhoJL96qRZ2dmEUUC9ZbVySPsEy9jR4fW4PVqzNC2ZI FGuwwAn1UlATNG0+kc0tGpEvE6I/GLaMN9wlEtCEiiPugV9ueshkpsCeWyZTo4VT95+M ajXQ== X-Gm-Message-State: AC+VfDz+RXvnr0rkoGc+wcmuYT5OSqT5HYyD4nWQNPJD1ySV9ilAoVi2 W/bGjcxa083F/WA5Vef/GXbcfw== X-Google-Smtp-Source: ACHHUZ6KdkbBdG/NhhrmPGJii84sxlUGH4WE/c+JtFzlhs/OksNqs2HD2kzrCPZt4nc2+TgevXB2ww== X-Received: by 2002:a17:903:120b:b0:1a6:d15f:3ce1 with SMTP id l11-20020a170903120b00b001a6d15f3ce1mr19866697plh.34.1684898010071; Tue, 23 May 2023 20:13:30 -0700 (PDT) Received: from GL4FX4PXWL.bytedance.net ([139.177.225.249]) by smtp.gmail.com with ESMTPSA id i4-20020a17090332c400b001ab061e352bsm7480769plr.195.2023.05.23.20.13.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 23 May 2023 20:13:29 -0700 (PDT) From: Peng Zhang To: Liam.Howlett@oracle.com Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, Peng Zhang Subject: [PATCH v4 06/10] maple_tree: Add mas_wr_new_end() to calculate new_end accurately Date: Wed, 24 May 2023 11:12:43 +0800 Message-Id: <20230524031247.65949-7-zhangpeng.00@bytedance.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20230524031247.65949-1-zhangpeng.00@bytedance.com> References: <20230524031247.65949-1-zhangpeng.00@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 20E8C100006 X-Stat-Signature: y9eg71kxqhykgoojzhjmj4yfg7sm9bor X-Rspam-User: X-HE-Tag: 1684898010-43683 X-HE-Meta: U2FsdGVkX1+UdJ6tsJ5fkJ6Jm/2fbLuIo47L3bh5vHEtLEvEyQF8tFF3rLOWOwyDzIMYRYc7RPCK5U0UEL17LyRSNmAr7HdiPgEXTJqKZpgdx3p+M2mdLrghLxAAlj9eCGSdRzw+tfa6HYWJfvqmczt4Ur7FAGmZdZ0l96/ZUOtfAb+i+u2r3335VCfFV2TMWdpHMDrmbwozI+TOJEWwPjOTI9UtA7ZFo9joJFC98BlvmSvJcIZ+nrYW6wu9OhukNXAXE8h/t2PB8eMCv+52Hot2s3ALg5kQC42YkWn0DScpj3mo3vVG53hqVxDZXhOTbOYhTywiCtotaONV1lKIcGIySRrcC1ihp9lLPrL3Gk55MqZb4xcETRNJOAjbGt+iMT6hKrxLB3IIoEmaIlddn9GcGN4ZG1pDBRZJC7UPIt+xTp08gbPVcVcPfGvI21WUyDcsjLlJs3Ry1Jt0JX1zQZ58uh6YnQpBnYKpTVR9C/IzVSKCHO+Fc1tx/byuW2gu+CHdwDNGn0hZQ5ULmeI4ci5Drega9YXezMdJnP9E3hVHuC7uFGoi2PU21H6QOtcAAVI0aDIucb+SpAjRYTVwlJzY1L2JQjSg3ihgU0t/dGFoRC9kR+TVyzuradxtLez2kjccwksallRwjG5u24QFij0yKJijIKfF1i7sLrd51PMng3nlMZRLcGGal6IF/r3oQ6i0C1apXSpWw/Ev6XTxqyIV7yi8Z5SR2Cu4j5k7pLxD18F9aqunfRHps44AxBbPPTEcHypMQ2udmf7vGiM5ftawYi90DtRwNW3wiZLnPt5ACpyuFB4df8Mzh78igXPdbQcDYV8a46GMdagdpEdlwd54ft8jWis+xzI3UhY7XYrHCvO8ncAMyxuYDbz/8BtIMki/Tcp4xP0NROOuEM+uVGR7QEnsWy8XgyyRSFHHTX4OwwqqbjI+Zqo56skJ7liUCCk29HF3rTeiVVphnHy ocV2QZd4 ueHXWuAbOTWu6RFvr3C2keSiAlimv6TQ/+7xgPwUOTOTyHkPaLLRfMbTT81gY1LN2tp/n01VJuPSApE6sbsHncbm6KugqJ1QJ7xndiiFPTIfKQrykmXSPb+JhpssWXW0h1MTnxGhecDq++In6VoQyARF5pd6pFeJeM73JhDhv6r6fPSvtHRdOnhS+/6ap8KyqzCz5J48k4H5ZpUdDy6nWUtiyW4NueoFsIsFAbyKuQizuMx/Y8SmE7ZD4kRc7IqHsisJKRKj0jgHkHdQfYQmPiTEPlkyhxVYEUSc8F6pDvzJNxyOwQM6y0P4lA//G0k+Klw4saZLExnkNuNrEr96FlUNqGCXxSYLKftMcfbNQ+b5Fqzf1+7Bk6lE+9UVbKpEpkcq2jiWBsldBeNUbKDgjoXVZ2Wy5Im6lQMUor66S3yzHCxa3rsVA0OU2/iPJzwQ5RgK4gEWywxh2csHgO1pf5Yxkh/D5lBVxKwuwERfGdYeHKHI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous new_end calculation is inaccurate, because it assumes that two new pivots must be added (this is inaccurate), and sometimes it will miss the fast path and enter the slow path. Add mas_wr_new_end() to accurately calculate new_end to make the conditions for entering the fast path more accurate. Signed-off-by: Peng Zhang Reviewed-by: Liam R. Howlett --- lib/maple_tree.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e6e42e1ba44c..07c5c7afd415 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -4296,6 +4296,21 @@ static inline void mas_wr_extend_null(struct ma_wr_state *wr_mas) } } +static inline unsigned char mas_wr_new_end(struct ma_wr_state *wr_mas) +{ + struct ma_state *mas = wr_mas->mas; + unsigned char new_end = wr_mas->node_end + 2; + + new_end -= wr_mas->offset_end - mas->offset; + if (wr_mas->r_min == mas->index) + new_end--; + + if (wr_mas->end_piv == mas->last) + new_end--; + + return new_end; +} + static inline bool mas_wr_append(struct ma_wr_state *wr_mas) { unsigned char end = wr_mas->node_end; @@ -4351,9 +4366,8 @@ static void mas_wr_bnode(struct ma_wr_state *wr_mas) static inline void mas_wr_modify(struct ma_wr_state *wr_mas) { - unsigned char node_slots; - unsigned char node_size; struct ma_state *mas = wr_mas->mas; + unsigned char new_end; /* Direct replacement */ if (wr_mas->r_min == mas->index && wr_mas->r_max == mas->last) { @@ -4363,17 +4377,15 @@ static inline void mas_wr_modify(struct ma_wr_state *wr_mas) return; } - /* Attempt to append */ - node_slots = mt_slots[wr_mas->type]; - node_size = wr_mas->node_end - wr_mas->offset_end + mas->offset + 2; - if (mas->max == ULONG_MAX) - node_size++; - - /* slot and node store will not fit, go to the slow path */ - if (unlikely(node_size >= node_slots)) + /* + * new_end exceeds the size of the maple node and cannot enter the fast + * path. + */ + new_end = mas_wr_new_end(wr_mas); + if (new_end >= mt_slots[wr_mas->type]) goto slow_path; - if (wr_mas->entry && (wr_mas->node_end < node_slots - 1) && + if (wr_mas->entry && (wr_mas->node_end < mt_slots[wr_mas->type] - 1) && (mas->offset == wr_mas->node_end) && mas_wr_append(wr_mas)) { if (!wr_mas->content || !wr_mas->entry) mas_update_gap(mas); -- 2.20.1