From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF627C3DA59 for ; Mon, 22 Jul 2024 21:39:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C5F76B007B; Mon, 22 Jul 2024 17:39:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 475776B0083; Mon, 22 Jul 2024 17:39:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33CFB6B0085; Mon, 22 Jul 2024 17:39:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0D0716B007B for ; Mon, 22 Jul 2024 17:39:03 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 871DAA1A88 for ; Mon, 22 Jul 2024 21:39:02 +0000 (UTC) X-FDA: 82368704124.09.F2A6C75 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf10.hostedemail.com (Postfix) with ESMTP id D42B9C0007 for ; Mon, 22 Jul 2024 21:39:00 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BaVFeAdD; spf=pass (imf10.hostedemail.com: domain of "SRS0=xUIm=OW=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=xUIm=OW=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721684295; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=FYGU2sR4bsgU45HK71Y7T/ptrrY7HH1sV8MLXm1VsKE=; b=a6Enu+nOUa5KHB4KzPfQUwEgc7dYDGSpp4/JkSD+VreSMHIQpnJFsevHnAiuIwxj/mQWSn 40wBQBjSDjxc4RbJDvUDOSVNHHptjPcaMVmWwdGqTcAmpWacEnpp0T/SeQiMmz1dEUWnGh 4YFbiCtGFnqd+eO6XEc/WRMssvAsp7E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721684295; a=rsa-sha256; cv=none; b=23kl9cx+Pfeb7A6xDlfo+3GFDn6Mu7OYwIS4ZDPVA4eJUsuHJcSBVJhLEMz6SYprkrx+bc zFBAFOsq7/RoNFT2/ADuWlzpc4xk/vCsflMwkSWgAzCfO4Oi+5/10uGzm/76gu2pQlXKNW 7UWzftN00arGR6dVqlfza+ZE43QXJqo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BaVFeAdD; spf=pass (imf10.hostedemail.com: domain of "SRS0=xUIm=OW=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=xUIm=OW=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 8EB7460347; Mon, 22 Jul 2024 21:38:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DC02C116B1; Mon, 22 Jul 2024 21:38:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721684339; bh=Rmt1EHpAenm5mlggL4qEvhD0ErTUNnAcYszDJw3mMDk=; h=Date:From:To:Cc:Subject:Reply-To:From; b=BaVFeAdDuAKKbaXJBRnQpl9wtpPjyZn57+ZDbkF+9m3dHzAVAfHUhDz5ky4ABOylC XJcKDRlaxC22UvZn84L9LyLJP6swWN+2KGnG947s5NYI1HaXaBX3SHy9D6yj2yZHqB EAK3HK7wzPAv4P0cnV/9f98la4/OBFlQ+rxwiqH/3ZGHUWmnKTAIUx+21qaG/KAOhk eKHMI3r6aC7FDgfOkUFCn75WJkP3dAyMz7Nku5R3KBOCLbFck3Y88JzikxPlHKx+sx VipDl6ugr4clRku69fJvHtonB0EkPWNVISht1rWtiDxZomtE3qUz1E2SrsA1oH5IZJ 8crSUeKvBFPjA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C5283CE0F8C; Mon, 22 Jul 2024 14:38:53 -0700 (PDT) Date: Mon, 22 Jul 2024 14:38:53 -0700 From: "Paul E. McKenney" To: linux-mm@kvack.org Cc: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, hannes@cmpxchg.org, willy@infradead.org, rcu@vger.kernel.org Subject: Possible slab-allocator issue Message-ID: Reply-To: paulmck@kernel.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D42B9C0007 X-Stat-Signature: ggioe5uea1g7jby377az5m51n34mc3hd X-HE-Tag: 1721684340-237844 X-HE-Meta: U2FsdGVkX198qyt3N6dWLfL5zMkNGRRpwJvOjf4kmSA9ksCt0Lb3CGJ8zGE2OjYKW1PNSJFsKd/YwGZAXoAIH3ujcj7saegGa5KoN94gmQbRjxMimKaY27QJSwZaQT7zhVaOqzrV+aHTut1/NESu3Ay+O/Wn+cOtDuscwgPk/vDW9n2rz+h7zZ+Byz/UY9gTjkJW8T7t9Vs1g6RwyEqb40alMqNIsi08xwelCd7NyxDCctKJz5M+op60EYy7YS8P2FM49xmxQMgGZzEURd9Uf3NnWuWvKz0JyFXxPbbcFQgarmFxema1bUDg1X0+GYia/gZGzJsaGV6pFJB5n436zDCe4p8CiROhoesNyTWonWKqCZNByXfMyMOzXVkoFbh2ReqvvC3k6zr9SyZf9FFzQrsV673sf5JANZpKWqv59PcqkburR4LoqNgOff4WeHGaLQn+tsnyhRiRza9xktgDyBcF+QNK8zytaZVg3og2G2DgsLC6M+IiZrwtSfQjK3OLhcta5XwGFTpqGRYn0/umqfWL4okNr6wjpnmdoEspusXWP4oTBIDHzA8JViF0CeXC4JWQ1IQi362z1tvQZUzb3NLgi2yG/xMaUts3TAc1PK4F/tfH4145xOzyfFUMpkRGVCaQDwkeGDC6njhSGFbscRxEXWyAPeid7bgsiJjR7qJEyavxti4ZLKr43t135y9DBVzsgaBgSenTZzLlIzJjHBFeoCl4Y6/Iyi6CaYhKL7GRDOuKPyscbsu3mVdAWu6XLijKZy6NuecL+maTrl/8daski2Bk9vzNR7p/bQMxHOs19qOAhx6ByWKYVoQAXnuAlUPi+CXTNwgb78Bixz/3txymji+ksErl6sUB+e0OF86CwynOQQQ/3jLWLYjFovcbUzyuzrCcFGAnunTWhOFqNxAW8EPIjfpWJeyV0nytoPukFJjjWENPpHudPJID0J1KGPymEjvBOtN1AlwnQfi w4+dVpiM jGxW0RRTJWXD3yr8/RDvBDt/fSGYy+NP8J/gF3vdc9XH62tN2JiwnCze2slzOt2q77Yx2c7xJUAEItsS5tTJJDty50enmjPsF463VhjyhWUYu8/V6lX7+S0zTXMm1mtvVJM7XUwzDiY99dg+pIrZZ8qTqrKx0LdwmWD4mWnBSgfuFHc/uQ4epCUQoo3M8Arl5BytuAGlcG/X6/I+n2YHpkn51Jruokm/A65L8LMQWyiYAqesYNiArwdPb0aqCiMj76iYdB2KNTHFE2uounW/6766vbV87eZ/ZYLgtfuWY4EuuK4uJs89TzC0nhYHukA83MRaX08AVfN1nrhwy/i0p6F+Rm3t4rPINLlMi5te/zGST8h+W8YiLRgu4V2Ec6WoOzhZeN3Wb18pXyafQ2SCom8qyn13DCDaOKcIkq3VITKSrFsHQAqyFgyGDsEBT2zTJLHPR8a7RRWLjV2Tp4psQwpZ1qaauokULEeu+eURxSC45ftnEDoih5cwr83H8lcYKM/mpZiFZjrVagKIaculdiiqWvqrIDo8cM1X4bFQyGC0KlU64ogp6HyqXlPRyav3+WlGqLfMrdscvVnAOTcdSMpuIptmq7leSLnOsH2Nk7NoK2L88wqaBayzlkVJH2b3CM7iVZRt/z1cwZORdutVSlRb9hvsE/ektoRZso7utkqTm+JJU0u4gyx7c9quEuqTjnilAdY97Wcfg/5dzc7S5k3jiD8C9/w6Da3aq24QuL0BAoCmZmMh4n2BKu66fzM225LnRtDCpZl3BcSr54ET0mJLPdQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello! I have been chasing a hang [1] in rcuscale, which is a kernel module that does performance testing for RCU updaters. I have a good long-term workaround for this problem, namely an extremely stupid custom allocator, so I don't need a fix, but I am nevertheless posting this just in case this affects others. Or in case I am doing something stupid that I should avoid doing in the future. I first encountered this on RCU's "dev" branch, but the problem also exists in current mainline [2]. One high-probability reproducer for this hang, tested on both AMD and Intel x86, is as follows: tools/testing/selftests/rcutorture/bin/kvm.sh --torture rcuscale --allcpus --duration 5 --configs TREE --bootargs "rcuscale.scale_type=tasks rcuscale.minruntime=5 rcuscale.nreaders=0 rcuscale.nwriters rcuscale.gp_async=1 rcuscale.gp_async_max=100000" --trust-make This script builds a kernel, requires that virtualization be enabled, and uses the qemu-system-x86_64 command. I have not run this on a system with fewer than 16 CPUs. When it fails, this takes a bit less than ten minutes to fail, which results in a non-zero exit code from kvm.sh. Rare successes normally consume kernel-build time plus less than a minute of run time. The failure mode is CPU 0 (and always CPU 0) failing to get out of this (admittedly ugly, since reworked) "retry" loop in the rcu_scale_writer() function in kernel/rcu/rcuscale.c: 1 retry: 2 if (!rhp) 3 rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); 4 if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { 5 atomic_inc(this_cpu_ptr(&n_async_inflight)); 6 cur_ops->async(rhp, rcu_scale_async_cb); 7 rhp = NULL; 8 } else if (!kthread_should_stop()) { 9 cur_ops->gp_barrier(); 10 goto retry; 11 } else { 12 kfree(rhp); /* Because we are stopping. */ 13 } The loop begins with the "retry" label on line 1, finds "rhp" equal to NULL on line 2, gets a NULL from kmalloc(..., GFP_KERNEL) on line 3, proceeds to the "else if" clause on lines 8-11, invokes rcu_barrier_tasks() on line 9 (because that is what the rcuscale.scale_type=tasks kernel boot parameter tells it to do), then line 10 jumps back to the retry label. This repeats thousands of times, consuming some minutes. There is no drama in the console log, as in no mention of OOM killer, allocation failures, or much of anything other than Tasks RCU switching to per-CPU callback queueing due to the heavy load and the expected rcuscale output. Note that all 16 CPUs are looping allocating and invoking either call_rcu_tasks() when allocation succeeds or rcu_barrier_tasks() when allocation fails. So CPU 0 could simply be quite unlucky or have other work so that when the rcu_barrier_tasks() waits for memory to be released, the other CPUs allocate it all before CPU 0's rcu_scale_writer() task is awakened. But again, there is no sign of memory-system drama on the console log. I am good, as I implemented a stupid per-CPU-array-based allocator, which avoids this issue, admittedly quite crudely. Johannes suggested turning on tracing for the kmem:kmalloc and kmem:mm_page_alloc trace events, which I did using this command: tools/testing/selftests/rcutorture/bin/kvm.sh --torture rcuscale --allcpus --duration 5 --configs TREE --kconfig "CONFIG_DEBUG_KERNEL=y CONFIG_RCU_TRACE=y CONFIG_FTRACE=y" --bootargs "rcuscale.scale_type=tasks rcuscale.minruntime=5 rcuscale.nreaders=0 rcuscale.nwriters rcuscale.gp_async=1 rcuscale.gp_async_max=100000 trace_event=kmem:kmalloc,kmem:mm_page_alloc trace_buf_size=3k" --trust-make I collected the .config file [3] and console output [4]. The console output features more than 1500 kmalloc trace events and not quite 10 mm_page_alloc trace events. Please let me know if more information would be helpful. Thanx, Paul [1] https://docs.google.com/document/d/1M_SJ_hVUe-jtkH6BjHzzRFOnAiu1sqThlFnx8Jfp-fg/edit?usp=sharing [2] bbb3556c014d ("Merge tag 'keys-next-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd"), for example. [3] https://drive.google.com/file/d/1lZmIvatLyVYzdRK_rv2xNNWDBiGwyQvk/view?usp=sharing [4] https://drive.google.com/file/d/1jh4hJG877j0Rp65153wtmzKm6jErXZ9n/view?usp=sharing