From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF784C433F5 for ; Mon, 23 May 2022 06:19:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 301D06B0005; Mon, 23 May 2022 02:19:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B20C6B0006; Mon, 23 May 2022 02:19:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A06A6B0007; Mon, 23 May 2022 02:19:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0B5A46B0005 for ; Mon, 23 May 2022 02:19:54 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id D429F80C85 for ; Mon, 23 May 2022 06:19:53 +0000 (UTC) X-FDA: 79496007066.23.4A8E15A Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.133]) by imf05.hostedemail.com (Postfix) with ESMTP id 06AF610002D for ; Mon, 23 May 2022 06:19:25 +0000 (UTC) Received: from [192.168.1.107] ([37.4.249.139]) by mrelayeu.kundenserver.de (mreue012 [212.227.15.167]) with ESMTPSA (Nemesis) id 1MlbPO-1nSArI0Q0w-00iluk; Mon, 23 May 2022 08:19:46 +0200 Message-ID: Date: Mon, 23 May 2022 08:19:41 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: vchiq: Performance regression since 5.18-rc1 Content-Language: en-US To: paulmck@kernel.org, Phil Elwell Cc: Marcelo Tosatti , Andrew Morton , Nicolas Saenz Julienne , Borislav Petkov , Minchan Kim , Mel Gorman , Juri Lelli , Thomas Gleixner , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linux ARM , regressions@lists.linux.dev, riel@surriel.com, viro@zeniv.linux.org.uk References: <77d6d498-7dd9-03eb-60f2-d7e682bb1b20@i2se.com> <20220521234616.GO1790663@paulmck-ThinkPad-P17-Gen-1> <20220523044818.GS1790663@paulmck-ThinkPad-P17-Gen-1> From: Stefan Wahren In-Reply-To: <20220523044818.GS1790663@paulmck-ThinkPad-P17-Gen-1> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K1:TSDqooD2qpB0H55oseeE5i/UKDv81lLRHUSkMpF+f6pSH2efkxU EU1yOKtb2vv2XBeZV3Z7rj2N+rYHqjAVfDIQZINjoMrt2oFSMCeDKZzYpZMzO4+eh1XlMCL EbgZilH5dplMtxW9j4NPfAfbUvgy9HZiKPFs3sP5lpcGKSJ266604KhH7wxBK4GbrFDJNpd zOc/AR8e3w/D9hnZ80PLw== X-UI-Out-Filterresults: notjunk:1;V03:K0:iSxJ5LbKmG0=:oLSd99D0npyJB5y7h8PO0P vI/k3/ewqFXa7tarqLA+EjDKGxVUUtmVqjUQUYuV8j2dpW4eHS8HAoporAFr3lcOKKQrDShWg ieZUvfPM0POY6flXugUFuXb0NrESaR5UdnKELk74HNrLgRVfMrdmq4CQa33a//3jQ4pUi2Npu O39M+pWcJrZajr3y4YRPjRS2DVIVk1kvKLZq3CMeKfpW/CRcEJSVcM88kr0wa6xPFMc1c6e5B vevcxxCTKGrPdf+CjbxPQXT2sZwHvyKVakYs/t+Ym0cahYcn6uJM+0v3ZZqZPLVeOVSfIf7l0 8G2AeFjO00I+aILsQxuQshdi7Rp73gJTKAWea4cejPzbxP3XGqLeAvPGFObRGFa4vKmfBVAEm ukxtuIh1YXFEo7XXLM7M+3H0UyLyIa/EtJrGlUm/wTA4ngVN6Eyrd7cywuoABWqMft5qEtuix 0BKebGd4d6pf9vGt5Mj1Prz9croLGpYAdBifycj9vuHuMnAz9Cv3rTCCvGoksat9ybCS8HvD3 53kY/gtXxueELkw4CO2+v1+Wofe1UtnG7Tz48+5e8TNjbCD3ciqthr5y0zYnMGBkm3fzb8/Vn rY4zCguzf4TGRKdy6bwnJV9/yYreYkzMerYe72BBiR5znQVqtXH0CcZuRc5H4cgdsdKF1wUHR cTdYVcx5mc6SbYLBvIqK9vlNkcnIwZut4FWL+7GWTLwxWR00FYYPVSsQhk71dQMyMlNxj6wo9 AWySY8M0ZXDFWZJbmmrQpFU0TlWyx/OPqtyxb9mRwS+Bi6gyY4SGSdtzOJg= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 06AF610002D X-Stat-Signature: d5zoke6tkqfrozktheudscr8yhn6ixuo Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf05.hostedemail.com: domain of stefan.wahren@i2se.com designates 212.227.126.133 as permitted sender) smtp.mailfrom=stefan.wahren@i2se.com X-Rspam-User: X-HE-Tag: 1653286765-835451 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Paul, Am 23.05.22 um 06:48 schrieb Paul E. McKenney: > On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote: >> Hi Paul, >> >> Am 22.05.22 um 01:46 schrieb Paul E. McKenney: >>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote: >>>> Hi, >>>> >>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my >>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance >>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: >>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu >>>> >>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1]. >>>> >>>> Before commit: >>>> >>>> real    0m1,500s >>>> user    0m0,068s >>>> sys    0m0,846s >>>> >>>> After commit: >>>> >>>> real    7m11,449s >>>> user    0m2,049s >>>> sys    0m0,023s >>>> >>>> Best regards >>>> >>>> [1] - https://github.com/raspberrypi/userland >>> Please feel free to try the patch shown below. Or the pair of patches >>> from Rik here: >>> >>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ >>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ >> I tried your patch and Rik's patches but in both cases vchiq_test runs 7 >> minutes instead of ~ 1 second. > That is surprising. Do you boot with rcupdate.rcu_normal=1? No, not explicit. > That would > nullify my patch, but I would expect that Rik's patch would still provide > increased performance even in that case. I will retest with a fresh SD card image. > > Could you please characterize where the slowdown is occurring? Unfortunately i don't have a deep insight into driver and vchiq_test tool. Just a user view. Do you think an strace would be a good starting point? @Phil Any advices to analyse this issue? > > Thanx, Paul > >> Best regards >> >>> There is work ongoing to produce something better, but ongoing slowly. >>> Especially my part of that work. >>> >>> Thanx, Paul >>> >>> ------------------------------------------------------------------------ >>> >>> From paulmck@kernel.org Mon Feb 14 11:05:49 2022 >>> Date: Mon, 14 Feb 2022 11:05:49 -0800 >>> From: "Paul E. McKenney" >>> To: clm@fb.com >>> Cc: riel@surriel.com, viro@zeniv.linux.org.uk, linux-kernel@vger.kernel.org, >>> linux-fsdevel@vger.kernel.org, kernel-team@fb.com >>> Subject: [PATCH RFC fs/namespace] Make kern_unmount() use >>> synchronize_rcu_expedited() >>> Message-ID: <20220214190549.GA2815154@paulmck-ThinkPad-P17-Gen-1> >>> Reply-To: paulmck@kernel.org >>> MIME-Version: 1.0 >>> Content-Type: text/plain; charset=us-ascii >>> Content-Disposition: inline >>> Status: RO >>> Content-Length: 1036 >>> Lines: 32 >>> >>> Experimental. Not for inclusion. Yet, anyway. >>> >>> Freeing large numbers of namespaces in quick succession can result in >>> a bottleneck on the synchronize_rcu() invoked from kern_unmount(). >>> This patch applies the synchronize_rcu_expedited() hammer to allow >>> further testing and fault isolation. >>> >>> Hey, at least there was no need to change the comment! ;-) >>> >>> Cc: Alexander Viro >>> Cc: >>> Cc: >>> Not-yet-signed-off-by: Paul E. McKenney >>> >>> --- >>> >>> namespace.c | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/fs/namespace.c b/fs/namespace.c >>> index 40b994a29e90d..79c50ad0ade5b 100644 >>> --- a/fs/namespace.c >>> +++ b/fs/namespace.c >>> @@ -4389,7 +4389,7 @@ void kern_unmount(struct vfsmount *mnt) >>> /* release long term mount so mount point can be released */ >>> if (!IS_ERR_OR_NULL(mnt)) { >>> real_mount(mnt)->mnt_ns = NULL; >>> - synchronize_rcu(); /* yecchhh... */ >>> + synchronize_rcu_expedited(); /* yecchhh... */ >>> mntput(mnt); >>> } >>> } >>>