From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 449F1C001DF for ; Fri, 20 Oct 2023 13:31:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D92678D00D9; Fri, 20 Oct 2023 09:31:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D42878D00BC; Fri, 20 Oct 2023 09:31:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C32308D00D9; Fri, 20 Oct 2023 09:31:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AEDCE8D00BC for ; Fri, 20 Oct 2023 09:31:16 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 751E1C0F1E for ; Fri, 20 Oct 2023 13:31:16 +0000 (UTC) X-FDA: 81365926152.01.0A59AAD Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) by imf24.hostedemail.com (Postfix) with ESMTP id 81A4E18002D for ; Fri, 20 Oct 2023 13:31:12 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=none; spf=none (imf24.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697808674; a=rsa-sha256; cv=none; b=NhyYVl3y3nAV3w+KWm+8bnQYNcV5xctIaeSbPz894N7+al3SHF5R5nkaGRMsnL1xVgOot5 wx/wGn1/K8VUxHuVj2J60zGx8zz8nZxoN0SjdH/3sYPlTS80z9UHHUwXz0ofB045dMTKEb vhvZ3C7Na7gbr/6jWjuXSDeug0iJ07o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=none; spf=none (imf24.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.51) smtp.mailfrom=houtao@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697808674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BkXSHMJLApoxDTUP/2k9YP0rtOA+bkaY0N+xDuGX3sI=; b=5a2piCniBXSxGKGPspUryJh29k7Sk/yODbMNwKizlfuI1wvHHu6ZxFzLlZ8XHTJyE2YGyB gBi/JL+KcLb4DGSv5PDGNLaIyRSpGBKzfOg2lHeLFdxqDzaDXKVPrVS39J0LXKjkYd5qE/ GUCjXtqlZR0w/dGz9iEzJ8DIVJBVAdg= Received: from mail02.huawei.com (unknown [172.30.67.153]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4SBlpQ0FnYz4f3mHJ for ; Fri, 20 Oct 2023 21:30:54 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP4 (Coremail) with SMTP id gCh0CgBn+dgIgTJlmYjjDQ--.7231S8; Fri, 20 Oct 2023 21:30:56 +0800 (CST) From: Hou Tao To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: Martin KaFai Lau , Alexei Starovoitov , Andrii Nakryiko , Song Liu , Hao Luo , Yonghong Song , Daniel Borkmann , KP Singh , Stanislav Fomichev , Jiri Olsa , John Fastabend , houtao1@huawei.com, Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton Subject: [PATCH bpf-next v3 4/7] bpf: Use pcpu_alloc_size() in bpf_mem_free{_rcu}() Date: Fri, 20 Oct 2023 21:31:59 +0800 Message-Id: <20231020133202.4043247-5-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20231020133202.4043247-1-houtao@huaweicloud.com> References: <20231020133202.4043247-1-houtao@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:gCh0CgBn+dgIgTJlmYjjDQ--.7231S8 X-Coremail-Antispam: 1UD129KBjvJXoW7urWrCw1UJry8uFykArWrZrb_yoW5JrWDpF W7Kr18AF4kXF45W3W2gr1xAa45Jw1Ig3WxKay7Zry5uFWfWr1DGr4kGry7XFn0krWUGaya yrykKr4furWUA3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY 6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Gr0_Cr1lIxAIcVC2z280aV CY1x0267AKxVW8Jr0_Cr1UYxBIdaVFxhVjvjDU0xZFpf9x07UZo7tUUUUU= X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 81A4E18002D X-Stat-Signature: cek8pied3xnttqc61kbh38uha1jyk6fy X-Rspam-User: X-HE-Tag: 1697808672-151456 X-HE-Meta: U2FsdGVkX1/ro5T3IwsJzFJMMjqbI3bKAMThSSClgLk/UqwfIlOv9A0NZeoufHbYn7Is3WQ2BFimlH0Z6CdddRZFd6tgc1De28TVir5OYJlHPSqizPScz//uE6coriCtTRKyoWOYK5HAcmemV4PaYMgbdXMCJ8ekDkO69VrERxDhNi10PwB4Gq7j0gR+G9KQ/sK2U3wexdKikIN9yTY4ODyTqfScPXr48ABOi+Km7HJRIcjMvFNoV2wFeiVhwnmmoHuwKPX56e9IvGkFXhqpFJEZhhJiDdCZzzcYVPGm8GNmCYZ7epIV6r+BPx9yvcbmGLlUrNJONbvScaBbX9pGiuhYE036190SumYchC/23II4HdTX3ZeGUqDOQbg3GCFalIAZ0dNJ/1pdG5488oorD1Gg71ZpabJNemJdehHw5PECD0MzFbmg1mpvxYkzMQMxjMykT4WwPJKV8wLmD6v0vPsCCb7yxxSixnEiXRpKVOvphZhmGiWkpTA0VsASg11xcf7DsUWXz+ivwfgVh2kjzf4uBu8IEwP0QGkXnhJcs3M06lGIvNWaw6jsAFrB20Zt/ChGZuhe6G/1MoAjcy/t4eUvQUjXEyLkQrlb72l7jCiMIq2eacSAjf1ycdLIuuERgbzTnRjvCgg2yCVrX13guctYO0efRC0ru+WNkowr82nWZwvuI+aGVgc6Z1ngmxyrS+1VzIpRJJ01HM5zDNcLfNisHF5Owojhua4bfW+DpAb0TQrrzRYdGIACVBqt+NiKVKoZUDx97V0UbRZoeaC5Mac14k/s7wJI68CnWKubZrR5gJtw7ZZeWCoJrcuaYJgnR6U384/vn+xiL99UGc7lM4lO/hjsY3qiwBQvC11hZxdku8ykFMK6/vp6zTJOzIsjR1+Dk+fUrz8rlcWHsM+VUI0ScMWnzJOOjJwnxGQPCfSljbgE9th2OwmW8Z/dM4tGkJ5iMuhHyepuAYKIBr8 84cXKolp fg5SRxsBWLdThqLTQrlYFru1zIDk9v9zTRKMsLxCoxWH3ODh0Tqrv1X31WLYocp0G2miW7f9iSw+249nudP179JGypD/f/gSbAzz4Fl/aTWjo+5vyazpHSbWK32jcPYK0D0KXefo9PfTpBOhfdjb6O42OCmtWBBX30tggfaeFmsWmEadSI3w9YY47yTmRLZ8qcce8bxOH/KYYyR0NPXnBXrhKtHv6erz6av3h3Ln9ajSataV6yoeCvndY9qZxifZbIqPRHfqLSPnBVUVAijJS/d25lNDmuvszTdmaFFKF/3RA+QhLE4RK4V7lrQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hou Tao For bpf_global_percpu_ma, the pointer passed to bpf_mem_free_rcu() is allocated by kmalloc() and its size is fixed (16-bytes on x86-64). So no matter which cache allocates the dynamic per-cpu area, on x86-64 cache[2] will always be used to free the per-cpu area. Fix the unbalance by checking whether the bpf memory allocator is per-cpu or not and use pcpu_alloc_size() instead of ksize() to find the correct cache for per-cpu free. Signed-off-by: Hou Tao --- include/linux/bpf_mem_alloc.h | 1 + kernel/bpf/memalloc.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf_mem_alloc.h b/include/linux/bpf_mem_alloc.h index d644bbb298af4..bb1223b213087 100644 --- a/include/linux/bpf_mem_alloc.h +++ b/include/linux/bpf_mem_alloc.h @@ -11,6 +11,7 @@ struct bpf_mem_caches; struct bpf_mem_alloc { struct bpf_mem_caches __percpu *caches; struct bpf_mem_cache __percpu *cache; + bool percpu; struct work_struct work; }; diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c index 776bdf5ffd80b..5308e386380af 100644 --- a/kernel/bpf/memalloc.c +++ b/kernel/bpf/memalloc.c @@ -525,6 +525,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) /* room for llist_node and per-cpu pointer */ if (percpu) percpu_size = LLIST_NODE_SZ + sizeof(void *); + ma->percpu = percpu; if (size) { pc = __alloc_percpu_gfp(sizeof(*pc), 8, GFP_KERNEL); @@ -874,6 +875,17 @@ void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size) return !ret ? NULL : ret + LLIST_NODE_SZ; } +static notrace int bpf_mem_free_idx(void *ptr, bool percpu) +{ + size_t size; + + if (percpu) + size = pcpu_alloc_size(*((void **)ptr)); + else + size = ksize(ptr - LLIST_NODE_SZ); + return bpf_mem_cache_idx(size); +} + void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr) { int idx; @@ -881,7 +893,7 @@ void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr) if (!ptr) return; - idx = bpf_mem_cache_idx(ksize(ptr - LLIST_NODE_SZ)); + idx = bpf_mem_free_idx(ptr, ma->percpu); if (idx < 0) return; @@ -895,7 +907,7 @@ void notrace bpf_mem_free_rcu(struct bpf_mem_alloc *ma, void *ptr) if (!ptr) return; - idx = bpf_mem_cache_idx(ksize(ptr - LLIST_NODE_SZ)); + idx = bpf_mem_free_idx(ptr, ma->percpu); if (idx < 0) return; -- 2.29.2