From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90199D11100 for ; Sun, 3 Nov 2024 15:03:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ED3D6B007B; Sun, 3 Nov 2024 10:03:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 99C3E6B0083; Sun, 3 Nov 2024 10:03:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B2506B0085; Sun, 3 Nov 2024 10:03:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6AAAB6B007B for ; Sun, 3 Nov 2024 10:03:45 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E6C771C76F3 for ; Sun, 3 Nov 2024 15:03:44 +0000 (UTC) X-FDA: 82745102580.04.237515E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id 9540516002B for ; Sun, 3 Nov 2024 15:03:22 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Qy8gOKmM; spf=pass (imf08.hostedemail.com: domain of "SRS0=b2af=R6=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=b2af=R6=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730646140; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IadLORJd8TUnpt493OKxzjK06ithZB89GPZehVhaZiY=; b=of9QxlqfuiIupaY6WOlKnrbFs/PK7MCqvfXtZuA5iB/1fLxsnVU2MLBHA1CvAvxDtqVSfQ Fusx4uNlpG54/qrnwSVggh6Fnnhp5Ok/9iCx+rDEYGvty1Yjk5NhdhtkHH4P4SG0sUdY/5 +Oznive0GsVmN3251+AhqUtbIHT9AFk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730646140; a=rsa-sha256; cv=none; b=bk77GNX89CtYAKoyjygwenTQjydVsMYskutoh0KIaFjsqZGqaAAvduK1BKF3LYvtZjXuQ0 S2lbquWm5RiPh6tHoaBHE5o0r8n4jJchvpaC9R5lWXh3GrNVaZmh/N8vxvxKNHHN/dq0es +6GFUoRyOTGFO7zcwIoo0jfO2xePpJc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Qy8gOKmM; spf=pass (imf08.hostedemail.com: domain of "SRS0=b2af=R6=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=b2af=R6=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org"; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 4F9705C4A0F; Sun, 3 Nov 2024 15:02:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB12EC4CECD; Sun, 3 Nov 2024 15:03:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730646221; bh=563EfkE5D0QNLDN9RLp0kZKWZbAwx5LWLn3gA89ZDCk=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=Qy8gOKmMxEPwCYBx5xJgYEuqXaFBWZoeFhihwMlG0nJLISo/ynHbP/XHe1kkbVijj L+VjC3xqq0QIbPRWAtj3TBf/jMJvnE6Xu73jUwiJKd8cbpNEy12h9edJJ1xGV4CE5R YWhGr2inXn9zIKScw18blnRbSwnW6HoI4roiDeae+gto1nTIqxUGYQ+VKGg6YEo/KT EkfYmQgj+N7go4j1tOOZv9yBXc7/73d2gG/FlY56CIbbViyL1ljdffwaKgijVxB2uf yDCdYmJiVonyR3wdDFG1l6hTnRst0q3+NP3MqUolk7tNLKQ1usq4vPkJgVdeWtLWGa craUhhQ0Il6cA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6454ECE0F53; Sun, 3 Nov 2024 07:03:41 -0800 (PST) Date: Sun, 3 Nov 2024 07:03:41 -0800 From: "Paul E. McKenney" To: Boqun Feng Cc: Sebastian Andrzej Siewior , Vlastimil Babka , Marco Elver , linux-next@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, sfr@canb.auug.org.au, longman@redhat.com, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, Thomas Gleixner , Peter Zijlstra Subject: Re: [PATCH] scftorture: Use workqueue to free scf_check Message-ID: <88694240-1eea-4f4c-bb7b-80de25f252e7@paulmck-laptop> Reply-To: paulmck@kernel.org References: <20241101195438.1658633-1-boqun.feng@gmail.com> <37c2ad76-37d1-44da-9532-65d67e849bba@paulmck-laptop> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam10 X-Stat-Signature: zfymraeeok5kxrc4h6ikskyriz3nnnut X-Rspamd-Queue-Id: 9540516002B X-Rspam-User: X-HE-Tag: 1730646202-563885 X-HE-Meta: U2FsdGVkX1/zrzaqdLsGKDdkdYU/YEA5iiytJ+kzLPcEUebSmrJuQz2rYohmkH66FCm4Lu7EEGs8bpS6CfPNHR3nxqa+ZPjI/QwNJQYBXuVWOTGITQDnDY/25ut5HHZAtcqO/3W5ZZ8bG74T7KFwhOXmMf5T04/gr4Eiobe7Knvcll+ckxOx+Kw05R+f/eAo5y76nzhfLNveUy5fDMN7CkVVSGeX/orKbytZLxI5t/h4J59tIr5VAfhiyrpqQ9SGm2wDxZZASGzZwKW4Bu+GCoIm2jIk8x/85hzZnLosh8Lc0u+f0Rzmiq5hQav/yeFi1yCv1ZzfQMTkBWBHGQVyliL0GU3EWwpy2e6iHZBoqi3ltp0Mg4ZM9BAa3zQizN8K2HDCDv4cWC4NK9tp1Ljmuwt4NGf/dna0oW0Y77v3ap2Ofjo4PwO/z0rKWowjKj2oOGAibzpEyZFbgzScPBnkSsXUF/bWuuOPiy6BZnpwe7qLxuqEDSMhm1xcfHHj3wREOXFL36R4fs54i0tNDO6ZEppUQvBvbgeoOP18S4VH/4AAB4YaQfn2pE81NH77J8Fi61TtUc4F1R4TBgM7b1a7Tyc+BAVGv8K/v9P4n1yt5UO1VXGBSsTmvUsHfOpq1cTtPTjJf2ago1z1UniY2Dq+OrGsmRkLOS03Zk29Ayg6v8/scml6cZrUAyD4AJTlPN4+vt8ZbGAOXa31Au19djNUD7MK3j1yUK6CCOWRn1yVv7p5w73Oau0VOjNg7kwFO+AqC9G2xFws8+cc7hMKbiLprUUYc2c+TZoIho8Uw5FaK/JOwpTTjKgaJBuDjei3vCNZGlMimHxs/7wApqHxNupbhgAi41T2KfaALBc9nBOW8QyUDAftEJNHLLUCwsTrYg4nrZCdYTfu90HKK8hyKslASz5nReefdwJtvxrbaryfSkPPoRWU4bEzywPZkKeYD+jahPhRA5JxdLN/B21LtWx CpBF4rRg mvDv98D/dGn1hltWwG4H2Dap5Ql8LbzsWb0yzuFGo7HhZHdvmK0h5iQ2W4HzO9Q35JBukQMxU68zunXyDnASduwLT3GFkzUdtM1oJEw8FSOOhJOIXzhNjGpUKjB/SXHWWsnQnMRuwkHknNd5m3FmCw2ZjUBUetoqMzTPFIvK1DRi/Ym9Z1hUMqwdr1SNJcv/uenBQjflVl8TNlWUECqx9p0GR5b9ZFuDg6/8iT+X8FqSvrDIjZ5gbG366tZ/OtEfbWUPpuWL9Xv+KWPtHtdzwURASVyNLmRrhA3dCrlJcqc377yfz8Duf56MiEWTWUXZSdux62YUcZQ1NYTqF+pi5AHJXdgJV0qrVcO0ZeMHBnlqWwKVd2C05WiJOvbH13VrXlaueqAGbztFeZ13cFxbK91xBJEu1ARdtRCYh8yeRAVqRsmc/DgDL8V9Zr3NQ/SdUomGxYL3qrf/FTnt0EiNfE841vsk/OuVWr9yuWsF+Jj6lGBLN90Hslz18xqSFaPR51sm9Xejrc4S7/0KkuQdAfF1asymxPZ01+HmBniPCdlERyU0cVXIWQrC7Z6IVCtOCk51OgOB2Da/1bNG+13YMFbQD6wAmvV7PQrpPFXsPhBmYbMRmUdhax5Ek9aEg6+B24dE36GKvYcQuzEJ8Gh2v5JltKUCVkyWx5SDbU3glxwfWU3STjGc8k8Ya5VVsRT94vG1QNOWJus2xVWliDDLdkbrRrduixYOL+0w50HpGACfS3rVl/W6e8BRfwcE/6+oAsMsrcBL6YzOIo8c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Nov 02, 2024 at 08:35:36PM -0700, Boqun Feng wrote: > On Fri, Nov 01, 2024 at 04:35:28PM -0700, Paul E. McKenney wrote: > > On Fri, Nov 01, 2024 at 12:54:38PM -0700, Boqun Feng wrote: > > > Paul reported an invalid wait context issue in scftorture catched by > > > lockdep, and the cause of the issue is because scf_handler() may call > > > kfree() to free the struct scf_check: > > > > > > static void scf_handler(void *scfc_in) > > > { > > > [...] > > > } else { > > > kfree(scfcp); > > > } > > > } > > > > > > (call chain anlysis from Marco Elver) > > > > > > This is problematic because smp_call_function() uses non-threaded > > > interrupt and kfree() may acquire a local_lock which is a sleepable lock > > > on RT. > > > > > > The general rule is: do not alloc or free memory in non-threaded > > > interrupt conntexts. > > > > > > A quick fix is to use workqueue to defer the kfree(). However, this is > > > OK only because scftorture is test code. In general the users of > > > interrupts should avoid giving interrupt handlers the ownership of > > > objects, that is, users should handle the lifetime of objects outside > > > and interrupt handlers should only hold references to objects. > > > > > > Reported-by: "Paul E. McKenney" > > > Link: https://lore.kernel.org/lkml/41619255-cdc2-4573-a360-7794fc3614f7@paulmck-laptop/ > > > Signed-off-by: Boqun Feng > > > > Thank you! > > > > I was worried that putting each kfree() into a separate workqueue handler > > would result in freeing not keeping up with allocation for asynchronous > > testing (for example, scftorture.weight_single=1), but it seems to be > > doing fine in early testing. > > I shared the same worry, so it's why I added the comments before > queue_work() saying it's only OK because it's test code, it's certainly > not something recommended for general use. > > But glad it turns out OK so far for scftorture ;-) That said, I have only tried a couple of memory sizes at 64 CPUs, the default (512M), which OOMs both with and without this fix and 7G, which is selected by torture.sh, which avoids OOMing either way. It would be interesting to vary the memory provided between those limits and see if there is any difference in behavior. It avoids OOM at the default 512M at 16 CPUs. Ah, and I did not check throughput, which might have changed. A quick test on my laptop says that it dropped by almost a factor of two, from not quite 1M invocations/s to a bit more than 500K invocations/s. So something more efficient does seem in order. ;-) tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --configs PREEMPT --duration 30 --bootargs "scftorture.weight_single=1" --trust-make Thanx, Paul > Regards, > Boqun > > > So I have queued this in my -rcu tree for review and further testing. > > > > Thanx, Paul > > > > > --- > > > kernel/scftorture.c | 14 +++++++++++++- > > > 1 file changed, 13 insertions(+), 1 deletion(-) > > > > > > diff --git a/kernel/scftorture.c b/kernel/scftorture.c > > > index 44e83a646264..ab6dcc7c0116 100644 > > > --- a/kernel/scftorture.c > > > +++ b/kernel/scftorture.c > > > @@ -127,6 +127,7 @@ static unsigned long scf_sel_totweight; > > > > > > // Communicate between caller and handler. > > > struct scf_check { > > > + struct work_struct work; > > > bool scfc_in; > > > bool scfc_out; > > > int scfc_cpu; // -1 for not _single(). > > > @@ -252,6 +253,13 @@ static struct scf_selector *scf_sel_rand(struct torture_random_state *trsp) > > > return &scf_sel_array[0]; > > > } > > > > > > +static void kfree_scf_check_work(struct work_struct *w) > > > +{ > > > + struct scf_check *scfcp = container_of(w, struct scf_check, work); > > > + > > > + kfree(scfcp); > > > +} > > > + > > > // Update statistics and occasionally burn up mass quantities of CPU time, > > > // if told to do so via scftorture.longwait. Otherwise, occasionally burn > > > // a little bit. > > > @@ -296,7 +304,10 @@ static void scf_handler(void *scfc_in) > > > if (scfcp->scfc_rpc) > > > complete(&scfcp->scfc_completion); > > > } else { > > > - kfree(scfcp); > > > + // Cannot call kfree() directly, pass it to workqueue. It's OK > > > + // only because this is test code, avoid this in real world > > > + // usage. > > > + queue_work(system_wq, &scfcp->work); > > > } > > > } > > > > > > @@ -335,6 +346,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra > > > scfcp->scfc_wait = scfsp->scfs_wait; > > > scfcp->scfc_out = false; > > > scfcp->scfc_rpc = false; > > > + INIT_WORK(&scfcp->work, kfree_scf_check_work); > > > } > > > } > > > switch (scfsp->scfs_prim) { > > > -- > > > 2.45.2 > > >