From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A46C4345F for ; Thu, 11 Apr 2024 16:40:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 254716B00B7; Thu, 11 Apr 2024 12:40:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 203E26B00B8; Thu, 11 Apr 2024 12:40:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CC546B00B9; Thu, 11 Apr 2024 12:40:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E34376B00B7 for ; Thu, 11 Apr 2024 12:40:34 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A030DA1809 for ; Thu, 11 Apr 2024 16:40:34 +0000 (UTC) X-FDA: 81997814388.03.304C54F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id B7DA0180019 for ; Thu, 11 Apr 2024 16:40:31 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=ZRDGRD1C; spf=pass (imf06.hostedemail.com: domain of jianfeng.w.wang@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=jianfeng.w.wang@oracle.com; dmarc=pass (policy=quarantine) header.from=oracle.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712853631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=zc8zh/JRJr57ikJjyKewTyO/joTKoeW778VhZry+YzE=; b=5mMzb/pJqmzWfwhzCXk+tdoJ6ZGRvPSOYVeJbCg2UBdzSf7vGHNhFIqd5EC6waRXtPsANa TZlFvdjkM0fZY0K056U2ZqZJyN3hOTy+2kBGk4Rmg+VTDVof+MemOThoMXWS1TsQqKby9o ktpV2JBme8NZ4Iw27TRpJjeJqizCBYo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712853631; a=rsa-sha256; cv=none; b=h8rPTsLW4cw32v0tKKiBqe3fewEnThScrGfh8kCJLeksjk5/AlOxCyxTKWH7EHeOKoICfo PfZwT6IiC0DmDldOsM9NQii3/tO50GJZ4W5F/NwGd3+5l0RhywHVLSM3sYi3F4PltIQ6ZY KLfDN+lULTrjl0rgIPMianlHXpr23nY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=ZRDGRD1C; spf=pass (imf06.hostedemail.com: domain of jianfeng.w.wang@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=jianfeng.w.wang@oracle.com; dmarc=pass (policy=quarantine) header.from=oracle.com Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 43BGYf1e014847; Thu, 11 Apr 2024 16:40:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=corp-2023-11-20; bh=zc8zh/JRJr57ikJjyKewTyO/joTKoeW778VhZry+YzE=; b=ZRDGRD1CdK6G/GXzV3ozOOs+o84j4iR+AsKHEYm68h7+yQLgstOqjIhHJ1ooH8jewcXA A+tCorR+xZ6KZpNk8ExJNouqY5v2uTtta623VUDqmlaidkxs26zHrPIzaaP45QvA1uPe +wt0u7WYA6RPe00qu7QwPTzj5y8yGL4IIsizUxdOu9er2SYqRtiQfgExmpDCqJQadxge 05tZtzy+gpwkLOBt8KvuLHb0b0Z4k+6DqHxebb2P3bzFJa8T9WKcpqpZ4RcivmY9ux6U th/Erj+xsdSM2rlrCQwkv+zkpKBAUro020GwWrh9FNwkqX5aIecmPLLmd/dK+lkuuOe1 Ug== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3xed4jrudt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Apr 2024 16:40:25 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 43BFoP7h032343; Thu, 11 Apr 2024 16:40:24 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3xavuadhnd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Apr 2024 16:40:24 +0000 Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 43BGduJQ008535; Thu, 11 Apr 2024 16:40:24 GMT Received: from jfwang-mac.us.oracle.com (dhcp-10-159-230-44.vpn.oracle.com [10.159.230.44]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3xavuadhma-1; Thu, 11 Apr 2024 16:40:24 +0000 From: Jianfeng Wang To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, junxiao.bi@oracle.com Subject: [PATCH] slub: limit number of slabs to scan in count_partial() Date: Thu, 11 Apr 2024 09:40:23 -0700 Message-ID: <20240411164023.99368-1-jianfeng.w.wang@oracle.com> X-Mailer: git-send-email 2.42.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-11_09,2024-04-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 adultscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000 definitions=main-2404110122 X-Proofpoint-GUID: -SwOfn8GKUt0X4zWUsz5phuu6G-mtB0D X-Proofpoint-ORIG-GUID: -SwOfn8GKUt0X4zWUsz5phuu6G-mtB0D X-Stat-Signature: 81tj5yu9iy5nr4hufpeini4b84atj7ft X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B7DA0180019 X-Rspam-User: X-HE-Tag: 1712853631-408133 X-HE-Meta: U2FsdGVkX19UAcCEiELAiTjqwdoDZh/BMj9ftxDrwJvkn2tcMbZXdmv/a9gaxSfw7wfroHsGgWkVwvl9OTvnnIE9UWeeKx+6YC/9Pks2VY0kTj37WUrJDqSQjbqrJ0+adgAE6x9CbA98NOM+UxtSgS/NGOBGV7wR4Dl/dnlUZLkZd/Gm1lBNLOMVmpSXV4R37M2OiAv8GzIDdGbVzTCt1n0ZfSlQD+L/km/zyMyql7USvUGZiO+65nKTZgOvzS5oekOMEmd0dXV6z/FHe+BAkcjapZ20L2uDEnKLubJKAyhQr4rLkDKaXHFwuZow6cm31WSn2Tti2up/OLTkiUFsk6JmfoubpYuDIpmpQ6tn3bw5DM5U/cC8GM1dutamE0mfyhk3WuKfiyecJZ/IWgLQoXhMRbcBO6cTjfoEaVfTg4m1UkQUgFkEYr3bNE0dR+bqpCw/vajC0JlhK5FF6Hdti62TM8g6ENUKbeVkW81/YC5BITmgEJ8DKWTqhVct3hK7FWWJHHiERLshalfQmYTtTXibq+ibaRLmlU+i3FtkWPt0xj1cRWQ7wi+yta0jPrzQa740U+LO/5FrNyeYPY+7fhof9g+MXcrf0l4FlZSeV6qqCjSL5zlAocl2m4rFF0gCMdXKREMQDUldrDzH7s9VpyEuz9Uq3lZjmIQFr0RifHAvXm1RXAKkf15adMuk+Y/fE0Qxk1hJXL8gGyVIl5FsNnc7Mk+f9w/ZmQNlK8UAto+rxrLwqKy0yr4ydiBiO3jFD0LIirVi96MPODGyAQX4n7blyKswEYHb0Ob4jbMkxbhtexpb9hVd23WWJU5rGRlmWd70WUWQU5/vUe/rBa9xHVFwdKBwxer++9nS4gsW2cjucfNlBi+rg1W74p0WrItXFhsgx+UdK9jLldCJ43sPBaJSye4KlX9y5LE/my0H6g0ZO/UJlD9UwlSfT2L+cCi8S/zQR6OHG7aZje1LGHl E7MVDHAR UwpR3sLoQGaHcw5rFTsgpGSSePXCuHp/k6G0KXLUhu5YmERe834np9c8J00uPsDZdZydCw2BzJUKAItlrU+NQhkX8abTOTAI+Y9wEQnflj0gRNLIQSZXK5FBwsOq1X3ztUuBSKtaJQiXcw6owTsQTqYOHhGKLPrnm752+ignPU0/9GazoSKCzAi1LY0ZHN214QeWJf3E2YRT+pr4ACuxup/ArnUoSSpl+DcWV0jU4rV9ti3RJk5Qrtkw7MaWcUamhYix9f5S1sFIOB52tqhUkETBjZXFmejmggkjfE26vQkHwMkg08e30QJ/TAlugzSh4B3MuzKHeF3up1K2K7QWLLO5lAj2i9ZXPKbXSclZ8O0A9RPf8KiBaes/eZtre0BuAGeZnkJa/LINf3fipC/Vv9DNsIw+qOGb3cWAeMTKEy12NyCItU/GQ6iMsmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When reading "/proc/slabinfo", the kernel needs to report the number of free objects for each kmem_cache. The current implementation relies on count_partial() that counts the number of free objects by scanning each kmem_cache_node's list of partial slabs and summing free objects from every partial slab in the list. This process must hold per kmem_cache_node spinlock and disable IRQ and may take a long time. Consequently, it can block slab allocation requests on other CPU cores and cause timeouts for network devices etc., when the partial slab list is long. In production, even NMI watchdog can be triggered due to this matter: e.g., for "buffer_head", the number of partial slabs was observed to be ~1M in one kmem_cache_node. This problem was also confirmed by several others [1-3] in the past. Iterating a partial list to get the exact count of objects can cause soft lockups for a long list with or without the lock (e.g., if preemption is disabled), and is not very useful too: the object count can change right after the lock is released. The approach of maintaining free-object counters requires atomic operations on the fast path [3]. So, the fix is to limit the number of slabs to scan in count_partial(), and output an approximated result if the list is too long. Default to 10000 which should be enough for most sane cases. [1] https://lore.kernel.org/linux-mm/ alpine.DEB.2.21.2003031602460.1537@www.lameter.com/T/ [2] https://lore.kernel.org/lkml/ alpine.DEB.2.22.394.2008071258020.55871@www.lameter.com/T/ [3] https://lore.kernel.org/lkml/ 1e01092b-140d-2bab-aeba-321a74a194ee@linux.com/T/ Signed-off-by: Jianfeng Wang --- mm/slub.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 1bb2a93cf7b6..5ed998ec7d6d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3213,16 +3213,25 @@ static inline bool free_debug_processing(struct kmem_cache *s, #endif /* CONFIG_SLUB_DEBUG */ #if defined(CONFIG_SLUB_DEBUG) || defined(SLAB_SUPPORTS_SYSFS) +#define MAX_PARTIAL_TO_SCAN 10000 + static unsigned long count_partial(struct kmem_cache_node *n, int (*get_count)(struct slab *)) { unsigned long flags; unsigned long x = 0; + unsigned long scanned = 0; struct slab *slab; spin_lock_irqsave(&n->list_lock, flags); - list_for_each_entry(slab, &n->partial, slab_list) + list_for_each_entry(slab, &n->partial, slab_list) { x += get_count(slab); + if (++scanned > MAX_PARTIAL_TO_SCAN) { + /* Approximate total count of objects */ + x = mult_frac(x, n->nr_partial, scanned); + break; + } + } spin_unlock_irqrestore(&n->list_lock, flags); return x; } -- 2.42.1