From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59AA8CDB47E for ; Wed, 18 Oct 2023 11:48:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF2048E0006; Wed, 18 Oct 2023 07:48:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA27D8D0016; Wed, 18 Oct 2023 07:48:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B90B78E0006; Wed, 18 Oct 2023 07:48:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AAA2D8D0016 for ; Wed, 18 Oct 2023 07:48:13 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6F687120213 for ; Wed, 18 Oct 2023 11:48:13 +0000 (UTC) X-FDA: 81358408866.30.A94F607 Received: from dggsgout12.his.huawei.com (unknown [45.249.212.56]) by imf30.hostedemail.com (Postfix) with ESMTP id 7EFB480004 for ; Wed, 18 Oct 2023 11:48:09 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; dmarc=none; spf=none (imf30.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.56) smtp.mailfrom=houtao@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697629691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7DRL6PXnOwUjzRpMja5o3pNOzDEF7qhxt7epvC+LHhQ=; b=F22UHHL20MAS4llYreKXE3pQjiZB8IVgmTQADPmBvW6HQyTEwG17R9ftF0UkGX9R399hBl 6jWYWfMnywy7W9uLWxO1oVOZ0GUCPq2fVqJmITKYLXH2eFivhBuhxZG7nnSOmIwEes7YB3 Ol1Hj4smVuFAyAUpKsoLNjbMnW5vMR8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; dmarc=none; spf=none (imf30.hostedemail.com: domain of houtao@huaweicloud.com has no SPF policy when checking 45.249.212.56) smtp.mailfrom=houtao@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697629691; a=rsa-sha256; cv=none; b=mOKAsD+oujMxfrPJmWFGuNcOasA/PSg0IDCQBp12fNIyQBFi4qCu1oFwNBgyiY0E/X7Vor /gj/VimtSHnR5r4YPCIh410llTpH0PFYDTFVWp81zzQx/Lq8x73A6/Z12cUA1JHMNDVhdw Gv4YZSDrLprcoy5SN7WgBEKXHjJRR4w= Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4S9TGh6mvTz4f3mJR for ; Wed, 18 Oct 2023 19:32:28 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP4 (Coremail) with SMTP id gCh0CgDnfd1Mwi9l9jYmDQ--.41845S8; Wed, 18 Oct 2023 19:32:34 +0800 (CST) From: Hou Tao To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: Martin KaFai Lau , Alexei Starovoitov , Andrii Nakryiko , Song Liu , Hao Luo , Yonghong Song , Daniel Borkmann , KP Singh , Stanislav Fomichev , Jiri Olsa , John Fastabend , houtao1@huawei.com, Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton Subject: [PATCH bpf-next v2 4/7] bpf: Use pcpu_alloc_size() in bpf_mem_free{_rcu}() Date: Wed, 18 Oct 2023 19:33:40 +0800 Message-Id: <20231018113343.2446300-5-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20231018113343.2446300-1-houtao@huaweicloud.com> References: <20231018113343.2446300-1-houtao@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:gCh0CgDnfd1Mwi9l9jYmDQ--.41845S8 X-Coremail-Antispam: 1UD129KBjvJXoW7urWrCw1UJry8uFykArWrZrb_yoW5JrW7pF W7Kr10yr4kXF4rG3W2gr1xAa45Jw1Ig3WxKa47ury5uFWfWr1DGr4kGry7XFn09rWUGaya yrykKr4fCrWUA3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY 6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aV CY1x0267AKxVW8Jr0_Cr1UYxBIdaVFxhVjvjDU0xZFpf9x07UZo7tUUUUU= X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7EFB480004 X-Stat-Signature: b6dhoi5xt88cozfqzujrnnyz9oqf75tq X-HE-Tag: 1697629689-844057 X-HE-Meta: U2FsdGVkX1+PGHdA1rNhhlYMEvZn/oLpaKS6as50hm6LXmz90QFfMnD5o8LHhtR2YrZcltcJKdcPewx3jBS0ED9yZf3gu/njXR/WMJBChjrLTA0r373W5Sa8P9OiUuVzRGt8d6Fzi8iEcizjlg/dhvCFSmdOvWGz6pljpKjMUOEtDMAcWpG9HPEFGl/FPrBBRyqwYmYILy/tOd0zf9XWeSSuBuDhpzSdJLevB4Fq3I/ZB2Qt8igAC6AG+99LEFsQzQSPiLb+0cvCd4vg0m66fqMDlt9PjrgebZfaUITG/jBMK79a9lS3UI8qBk0W4tNeatv7ocSwFTOy4ALoTWmPLvbZTcb/eeIzD+Td7bhVt6xi76PrSSWI5F8GYXZDPgo5wy4gWEGlU4UoGBurjyKJKggRJz9n6zpgAVFMSH/8WX1q67eIKP8eyErv0BIrJ7RaZQUftXC/M+KIUNM4/iGiiR/6b6JuB+m8Gh3/fubOA+vIwUweHzck4faygosYswb5nbl+XA1NpAP+XQXeT+jep2ZpNvmRefLFl+FWob2NaIqFujvs6d/E9IpDbD9PZMOt5U/qT+JPiTvd+6Rq++l6NSygWkxkXUzBpcknm6MYrozizq2oGRKBsjp4AXOrne4PxwTFnlqMGiW85YCFwL3rospOTEIZsFPDzXEhBVJVNpV27cgjLx6+J/ZSIl3I/iJjzYwYWq4EZQvrWxLuINhQtK+Vi6cKeA+M+hB5VWeCnNbmQ4Sbk3bR8AI04199XB0bXOsTVRarEO/r5uI50KpIkQKzGlYFIzHcCl2793XqUqDsPZNPKJKfBxfn28ZTzgEcfUEwYs05Iq6r3eL19dtgI1r0mLHED32WYTL3UmmNrtKWnu5fMHURgrOOY8gTCukRqhK4hMqYz+FFoOxzkH5fC1k/xnmHRtzwa0PvDBFIHMElxeFVsU6dyYdBr+rkWrUpz4QOWoq7KGyRihoSpAk q/Yq1uvT Y9zpAfWOdmIuXazm/xQvjqHBOK5MM1ZiEv4mUZna2H+xSOnLRHgS3fxN/Mv+EYhGAnPbrS0COjrm0d9VBaJ70I6k4jZ12LATX+wbztZDNzsHHcNR2hOv87IpK2LRHQ/HI4Zfoo5A73X7iHneLKtj56PSalH1ejWCKrObfkF+ZouoBQFfpMRmtML3loPHhMRtIMdzNmrisKe2XiYPFF2LFJCmgw7L4ek89zAUxo1dk4oxabhmlKZzNxDOTWcVjwm5N6sRZxILLoXCmkFJA7ldvqv958kkOYwr53STzLZyivfBHLqtBL1yizhKdTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hou Tao For bpf_global_percpu_ma, the pointer passed to bpf_mem_free_rcu() is allocated by kmalloc() and its size is fixed (16-bytes on x86-64). So no matter which cache allocates the dynamic per-cpu area, on x86-64 cache[2] will always be used to free the per-cpu area. Fix the unbalance by checking whether the bpf memory allocator is per-cpu or not and use pcpu_alloc_size() instead of ksize() to find the correct cache for per-cpu free. Signed-off-by: Hou Tao --- include/linux/bpf_mem_alloc.h | 1 + kernel/bpf/memalloc.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf_mem_alloc.h b/include/linux/bpf_mem_alloc.h index d644bbb298af..bb1223b21308 100644 --- a/include/linux/bpf_mem_alloc.h +++ b/include/linux/bpf_mem_alloc.h @@ -11,6 +11,7 @@ struct bpf_mem_caches; struct bpf_mem_alloc { struct bpf_mem_caches __percpu *caches; struct bpf_mem_cache __percpu *cache; + bool percpu; struct work_struct work; }; diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c index e52ef1e106ae..43bd4ce3947b 100644 --- a/kernel/bpf/memalloc.c +++ b/kernel/bpf/memalloc.c @@ -531,6 +531,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) /* room for llist_node and per-cpu pointer */ if (percpu) percpu_size = LLIST_NODE_SZ + sizeof(void *); + ma->percpu = percpu; if (size) { pc = __alloc_percpu_gfp(sizeof(*pc), 8, GFP_KERNEL); @@ -880,6 +881,17 @@ void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size) return !ret ? NULL : ret + LLIST_NODE_SZ; } +static notrace int bpf_mem_free_idx(void *ptr, bool percpu) +{ + size_t size; + + if (percpu) + size = pcpu_alloc_size(*((void **)ptr)); + else + size = ksize(ptr - LLIST_NODE_SZ); + return bpf_mem_cache_idx(size); +} + void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr) { int idx; @@ -887,7 +899,7 @@ void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr) if (!ptr) return; - idx = bpf_mem_cache_idx(ksize(ptr - LLIST_NODE_SZ)); + idx = bpf_mem_free_idx(ptr, ma->percpu); if (idx < 0) return; @@ -901,7 +913,7 @@ void notrace bpf_mem_free_rcu(struct bpf_mem_alloc *ma, void *ptr) if (!ptr) return; - idx = bpf_mem_cache_idx(ksize(ptr - LLIST_NODE_SZ)); + idx = bpf_mem_free_idx(ptr, ma->percpu); if (idx < 0) return; -- 2.29.2