From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 885B7C433EF for ; Sun, 19 Jun 2022 12:14:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B27456B0071; Sun, 19 Jun 2022 08:14:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD6E76B0073; Sun, 19 Jun 2022 08:14:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C77A6B0074; Sun, 19 Jun 2022 08:14:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8CAE06B0071 for ; Sun, 19 Jun 2022 08:14:10 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 594C5775 for ; Sun, 19 Jun 2022 12:14:10 +0000 (UTC) X-FDA: 79594877460.09.1C0F363 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) by imf13.hostedemail.com (Postfix) with ESMTP id B8EAD200A2 for ; Sun, 19 Jun 2022 12:14:09 +0000 (UTC) Received: from [2a02:8108:963f:de38:eca4:7d19:f9a2:22c5]; authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) id 1o2toa-0007gd-H3; Sun, 19 Jun 2022 14:14:04 +0200 Message-ID: <67d39f78-8eed-f49a-b3b0-18f77f9821cd@leemhuis.info> Date: Sun, 19 Jun 2022 14:14:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: [patch v5] mm: lru_cache_disable: replace work queue synchronization with synchronize_rcu Content-Language: en-US To: Michael Larabel , Andrew Morton , Marcelo Tosatti Cc: Borislav Petkov , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Matthew Wilcox , Mel Gorman , Nicolas Saenz Julienne , Juri Lelli , Thomas Gleixner , Sebastian Andrzej Siewior , "Paul E. McKenney" , Stefan Wahren , "regressions@lists.linux.dev" References: <20220304163554.8872fe5d5a9d634f7a2884f5@linux-foundation.org> <20220528141809.732788db257253396462c79e@linux-foundation.org> <134065a4-0446-bbbc-fca8-59f32798cf08@MichaelLarabel.com> <0ce05ce7-1a00-82df-f37a-bf7f9e216504@MichaelLarabel.com> From: Thorsten Leemhuis In-Reply-To: <0ce05ce7-1a00-82df-f37a-bf7f9e216504@MichaelLarabel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-bounce-key: webpack.hosteurope.de;regressions@leemhuis.info;1655640849;1c531aa5; X-HE-SMSGID: 1o2toa-0007gd-H3 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655640849; a=rsa-sha256; cv=none; b=viQUR8VbtG38tHBfPNgs+fLmo3nixGeYc0iT6DezCHft75cbP+gx0xuEBdUizSe7ew/I7a PdLq6lPh4W866fDt2+pOAjmjE5O0xe5AtnaoINwv2VC7xCKrferIk5Pwt/EIUafFmmnMOQ SISKuSL1/14aFe7avnT5X+YFnav+Cr8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf13.hostedemail.com: domain of regressions@leemhuis.info designates 80.237.130.52 as permitted sender) smtp.mailfrom=regressions@leemhuis.info ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655640849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qJiC3sMrO3QNKp3fQi3UwYnb5iQTKUvG3pgF+skzCts=; b=sKh79Pwp6ZUZiM2uA6/BVgqqsy9mvzbQrR02eE+4c7BABEwdILnT/CsELn+9igGIZnSdkK gNR4cZ8gzh0n9G6DzOAPxdA7Gul4rryXSf6WS7CbYYZJ+foXdykr1RldGVVRs7ubO237m7 XkD+DqDmlEbSrN6lqsTGVkszJ37doBw= Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf13.hostedemail.com: domain of regressions@leemhuis.info designates 80.237.130.52 as permitted sender) smtp.mailfrom=regressions@leemhuis.info X-Stat-Signature: w78en5b64t98ferbm66qzm8kjfq6otd4 X-Rspamd-Queue-Id: B8EAD200A2 X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1655640849-93450 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, this is your Linux kernel regression tracker. On 29.05.22 02:48, Michael Larabel wrote: > On 5/28/22 17:54, Michael Larabel wrote: >> On 5/28/22 16:18, Andrew Morton wrote: >>> On Thu, 28 Apr 2022 15:00:11 -0300 Marcelo Tosatti >>> wrote: >>>> On Thu, Mar 31, 2022 at 03:52:45PM +0200, Borislav Petkov wrote: >>>>> On Thu, Mar 10, 2022 at 10:22:12AM -0300, Marcelo Tosatti wrote: >>>>> Someone pointed me at this: >>>>> https://www.phoronix.com/scan.php?page=news_item&px=Linux-518-Stress-NUMA-Goes-Boom >>>>> >>>>> which says this one causes a performance regression with stress-ng's >>>>> NUMA test... >>>> >>>> This is probably do_migrate_pages that is taking too long due to >>>> synchronize_rcu(). >>>> >>>> Switching to synchronize_rcu_expedited() should probably fix it... >>>> Can you give it a try, please? >>> I guess not. >>> >>> Is anyone else able to demonstrate a stress-ng performance regression >>> due to ff042f4a9b0508?  And if so, are they able to try Marcelo's >>> one-liner? >> >> Apologies I don't believe I got the email previously (or if it ended >> up in spam or otherwise overlooked) so just noticed this thread now... >> >> I have the system around and will work on verifying it can reproduce >> still and can then test the patch, should be able to get it tomorrow. >> >> Thanks and sorry about the delay. > > Had a chance to look at it today still. I was able to reproduce the > regression still on that 5950X system going from v5.17 to v5.18 (using > newer stress-ng benchmark and other system changes since the prior > tests). Confirmed it also still showed slower as of today's Git. > > I can confirm with Marcelo's patch below that the stress-ng NUMA > performance is back to the v5.17 level of performance (actually, faster) > and certainly not like what I was seeing on v5.18 or Git to this point. > > So all seems to be good with that one-liner for the stress-ng NUMA test > case. All the system details and results for those interested is > documented @ https://openbenchmarking.org/result/2205284-PTS-NUMAREGR17 > but basically amounts to: > >     Stress-NG 0.14 >     Test: NUMA >     Bogo Ops/s > Higher Is Better >     v5.17: 412.88 >     v5.18: 49.33 >     20220528 Git: 49.66 >     20220528 Git + sched-rcu-exped patch: 468.81 > > Apologies again about the delay / not seeing the email thread earlier. >lru_cache_disable: replace work queue synchronization with synchronize_rcu > Thanks, > > Michael > > Tested-by: Michael Larabel Andrew, is there a reason why this patch afaics isn't mainlined yet and lingering in linux-next for so long? Michael confirmed that this patch fixes a regression three weeks ago and a few days later Stefan confirmed that his problem was solved as well: https://lore.kernel.org/regressions/79bb603e-37cb-d1dd-1e12-7ce28d7cfdae@i2se.com/ Reminder: unless there are good reasons it shouldn't take this long to for reason explained in https://docs.kernel.org/process/handling-regressions.html Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight. >>>> diff --git a/mm/swap.c b/mm/swap.c >>>> index bceff0cb559c..04a8bbf9817a 100644 >>>> --- a/mm/swap.c >>>> +++ b/mm/swap.c >>>> @@ -879,7 +879,7 @@ void lru_cache_disable(void) >>>>        * lru_disable_count = 0 will have exited the critical >>>>        * section when synchronize_rcu() returns. >>>>        */ >>>> -    synchronize_rcu(); >>>> +    synchronize_rcu_expedited(); >>>>   #ifdef CONFIG_SMP >>>>       __lru_add_drain_all(true); >>>>   #else >>>> >>>>