From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B68EC83F01 for ; Wed, 30 Aug 2023 06:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B41D48E0044; Wed, 30 Aug 2023 02:26:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF34E8E0009; Wed, 30 Aug 2023 02:26:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B94A8E0044; Wed, 30 Aug 2023 02:26:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8BB258E0009 for ; Wed, 30 Aug 2023 02:26:07 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5CA841A019B for ; Wed, 30 Aug 2023 06:26:07 +0000 (UTC) X-FDA: 81179785974.09.162A6B9 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id 82DC0A0013 for ; Wed, 30 Aug 2023 06:26:05 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=a6CVy3vA; dmarc=pass (policy=none) header.from=linuxfoundation.org; spf=pass (imf25.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693376765; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C4/YoH83bvXx+UErcEY8rjjCGPACrB3xnlVduS7inaA=; b=uoqvjIkWybg4wvO8n9o0DC/PJ3AwhAC1wkYNWVXrk1kPx4gmxY/PQ/p2h+VWxia1S5ghSl CBo6GHKnSadGVQochYJZViVjKiQTXSq3tB7BuMO7MK/eTsiPA2qhyNhXyWYL+98Of5a3HM c1ze3v5BJaquwPDZvoLmH27kHCGHAdo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=a6CVy3vA; dmarc=pass (policy=none) header.from=linuxfoundation.org; spf=pass (imf25.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693376765; a=rsa-sha256; cv=none; b=Oajt2yI4PmVh60ODwLyUPxAuW51iNhRIrjiwctlgfmmh6cOzfOr3ezseOewBGldDgBaHtL J5Rr9KMoc0lnLG5N+ziyK1AAMDsphHRuYgSHXerhHkr3lrBuIrQeqo7G7/kij6IiaRNta9 QAQfT/FJWH08QrBmlcDaFpSy0ysWHG4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 66F6062A11; Wed, 30 Aug 2023 06:26:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BABDC433C8; Wed, 30 Aug 2023 06:26:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1693376764; bh=5CTRaQi87lWjlnEJg7LzQ49T5UnzTi0vTd1SjwA50cw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=a6CVy3vAwAqiqNjRcl44SAEIcqzREnVPXyBJHY2g9sZmbunT1GBDyATpX6TB1ZKeL iftxn9XFvTMVnKLs9ChzY61M2kWqd8B0a0+KXc9+xgggCOtyAwbngSAuTAYFxOCgqn k4ZF1DMAD8Ie6vlwkTONkSA8kswHZDt2kw89j/Oo= Date: Wed, 30 Aug 2023 08:26:01 +0200 From: Greg KH To: Maximilian Heyne Cc: stable@vger.kernel.org, Linus Torvalds , Michael Larabel , Matthieu Baerts , Dave Chinner , Matthew Wilcox , Chris Mason , Jan Kara , Amir Goldstein , Andrew Morton , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Luis Chamberlain , Kees Cook , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH] mm: allow a controlled amount of unfairness in the page lock Message-ID: <2023083048-eraser-imprison-5cbd@gregkh> References: <20230823061642.76949-1-mheyne@amazon.de> <2023082731-crunching-second-ad89@gregkh> <20230828101420.GA54787@dev-dsk-mheyne-1b-c1362c4d.eu-west-1.amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230828101420.GA54787@dev-dsk-mheyne-1b-c1362c4d.eu-west-1.amazon.com> X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 82DC0A0013 X-Stat-Signature: ze857b9buydszubqrh3ydqa4w1dymwft X-Rspam-User: X-HE-Tag: 1693376765-93302 X-HE-Meta: U2FsdGVkX18h1O1FgVY+OssNefHecJMIWCWyhn/4MADWyEvW/n7O3F0JVPorXgqrUGW8CGy58fDX/v4Cj46ASWUIDYTwrni/naL8qSJr200T3O6IJvPWStF/Gj1VfECgvbtgb7LvyDP/VFlSjARumuLvKYX/YWRSHWPoFlRJ3xIW4CHLFhtbU1A27dziDtg8NMM2xvgBQ5J3Gu1Tz3GPfwYwaqDN422/BzOWUgqnFs2WPKLahoCX3a50SdUZUoiDu7BrGrb9r0Uqo/KCqDldm9EEJW63zAnTq4kYBR3zuim+EaD6+4WDNtC+9ouIr78bzxdgHS3l5k8+aecJ7OaCA1kge0G5r3wAhiHhEOqXCbMEyD/VH/5bt4pL3HK5zGibgoBpN+ym2avZQ5ZkAEN7OpjXMgk6W8l9jgMujvkrx8OUswkO/j6g+JO5uIt4Kir7/VHbI68tOFeE0ztUdpWxMYD57rSgzC+ECG5oY3WZif4gRoA91vw1vj7NO03Tninz4Xai5C44x9woJZV6ljLelnY2nw1F69MNDrp4BNR6DYxXxCXl/EDjmMT+jujgJxywltvUNlyKRsxEHhVwx5QF14VQXylc39hoHbyEfd7VvDDlE7BP+AxA4NAH8fz4dl8RgueVzCppJbbwvv8cCxF07m49lvz2eiDKTVjBfLtG5p9aAPoiwQISCWi8VxPVAFX2nBsn7BAKDJVMVAg5NDo7m4r1pwJbW6EjWozLl8lT6uj+nd7e25e+qCLdDsWhorl0kW8jpi1EQVzm050ftO/0vy4JXTlEkdMqoWJVGPgOqYFhG9sVAfThI/4nEsA1clZIQKjijMchfai83j3A/ZJzHNiJZMMS38k2Ncu/B5FFUTNEjqn3Mw1I6r+ez3tfMX1CMy3g5hd0e0JP+80dCj1BjkuEz448y+B0+kfCPlpaQrJV2QxlJuCmKgDSiiI+IqlPnEaF4Kfos4lbdZ5FTRx S531ZPK/ iEXqdeVTo+ey0lfTB4oHlxI8uhw3xEjdC6ZXBCbsqlRHTpruDFGRdylEtX5Uq4cu4BmJZr8xbChfKTpwTuMYCc+/+jNPj1S9ViL1YoHyLRZS2PVrP6woSNdWmdZxfNytpVjVC06lP7ye+Z69cRQEUhcSzOp+P5kWd8x7vDfzvApwaW6bc6tQkfAXovPlFqoaM/e4RUaCad2kPbHMgGFkMId5yNqNf0SGzGghOhAePRGB/b2Eps09JyP1s9xUqNkdHUeU/GjRHO6cOrFJ9yfvBXCOS6p3VOMCJrKpz88vDEw+0a03nRg8QuGrHUNxqSR7rHZ+xHvPqoIeHoLwVJ/q/RL3jGkfwbAUYNSAGosATNVmdaKJHv72YnQ8JF9NESm9tgzXupU9sGOHgwdjqBvVBg9uxIRPmTXvcCddoeHqS9FHrv9mNdQunVG+3l7G1hxzjJOIhNZz/lDUBasDLd5IjY7EsDI9ekTRmGoNA/EV/0jKaUd6pQKIkwyo6ajo7RWSnOnttnrqz0Y1SkthEHKrX9fcf44MgryNMdPAL5m9XUzHjiWP/y8xQn1wEwzGyqA5GeXo6NMzsvXgSzgAkD/ZO4NVaVbdhfjAOIj4Ns4hHVpw9KEB98ytBDjQr91/U32yCctE3wmZYGdU7n5BWzH7ev/NCxhm9U2alZc8SALV6H7DqePx+ZegMCKdJY9WGlBH30ehHUu2ILnnwASU/IvOoGv+MUkpzSe/e5ZN/XUAQK74ZCiEjUCWZbJ8vCTTKuhNqAovKd25JS4WTC2QG+qS+3TWhFBDYw4xnYx8RcaT6dr3UAqKe29zB88Ka/eZLJbQUxVPHvj6HWbVl1CgHlDi6nTzkKiGY0GcoZAfinLCYBE20VgcjAL8OyjHF4FEn7tEl7libMtwNdnmzfn63X1otnuDbQpr2Hz0Q7/Jap/ufAwcDvAMQ4BwkA79TFHRwf7H/oHpq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Aug 28, 2023 at 10:14:20AM +0000, Maximilian Heyne wrote: > On Sun, Aug 27, 2023 at 10:54:03AM +0200, Greg KH wrote: > > On Wed, Aug 23, 2023 at 06:16:42AM +0000, Maximilian Heyne wrote: > > > From: Linus Torvalds > > > > > > [ upstream commit 5ef64cc8987a9211d3f3667331ba3411a94ddc79 ] > > > > > > Commit 2a9127fcf229 ("mm: rewrite wait_on_page_bit_common() logic") made > > > the page locking entirely fair, in that if a waiter came in while the > > > lock was held, the lock would be transferred to the lockers strictly in > > > order. > > > > > > That was intended to finally get rid of the long-reported watchdog > > > failures that involved the page lock under extreme load, where a process > > > could end up waiting essentially forever, as other page lockers stole > > > the lock from under it. > > > > > > It also improved some benchmarks, but it ended up causing huge > > > performance regressions on others, simply because fair lock behavior > > > doesn't end up giving out the lock as aggressively, causing better > > > worst-case latency, but potentially much worse average latencies and > > > throughput. > > > > > > Instead of reverting that change entirely, this introduces a controlled > > > amount of unfairness, with a sysctl knob to tune it if somebody needs > > > to. But the default value should hopefully be good for any normal load, > > > allowing a few rounds of lock stealing, but enforcing the strict > > > ordering before the lock has been stolen too many times. > > > > > > There is also a hint from Matthieu Baerts that the fair page coloring > > > may end up exposing an ABBA deadlock that is hidden by the usual > > > optimistic lock stealing, and while the unfairness doesn't fix the > > > fundamental issue (and I'm still looking at that), it avoids it in > > > practice. > > > > > > The amount of unfairness can be modified by writing a new value to the > > > 'sysctl_page_lock_unfairness' variable (default value of 5, exposed > > > through /proc/sys/vm/page_lock_unfairness), but that is hopefully > > > something we'd use mainly for debugging rather than being necessary for > > > any deep system tuning. > > > > > > This whole issue has exposed just how critical the page lock can be, and > > > how contended it gets under certain locks. And the main contention > > > doesn't really seem to be anything related to IO (which was the origin > > > of this lock), but for things like just verifying that the page file > > > mapping is stable while faulting in the page into a page table. > > > > > > Link: https://lore.kernel.org/linux-fsdevel/ed8442fd-6f54-dd84-cd4a-941e8b7ee603@MichaelLarabel.com/ > > > Link: https://www.phoronix.com/scan.php?page=article&item=linux-50-59&num=1 > > > Link: https://lore.kernel.org/linux-fsdevel/c560a38d-8313-51fb-b1ec-e904bd8836bc@tessares.net/ > > > Reported-and-tested-by: Michael Larabel > > > Tested-by: Matthieu Baerts > > > Cc: Dave Chinner > > > Cc: Matthew Wilcox > > > Cc: Chris Mason > > > Cc: Jan Kara > > > Cc: Amir Goldstein > > > Signed-off-by: Linus Torvalds > > > CC: # 5.4 > > > [ mheyne: fixed contextual conflict in mm/filemap.c due to missing > > > commit c7510ab2cf5c ("mm: abstract out wake_page_match() from > > > wake_page_function()"). Added WQ_FLAG_CUSTOM due to missing commit > > > 7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem") ] > > > Signed-off-by: Maximilian Heyne > > > --- > > > include/linux/mm.h | 2 + > > > include/linux/wait.h | 2 + > > > kernel/sysctl.c | 8 +++ > > > mm/filemap.c | 160 ++++++++++++++++++++++++++++++++++--------- > > > 4 files changed, 141 insertions(+), 31 deletions(-) > > > > This was also backported here: > > https://lore.kernel.org/r/20230821222547.483583-1-saeed.mirzamohammadi@oracle.com > > before yours. > > > > I took that one, can you verify that it is identical to yours and works > > properly as well? > > Yes it's identical and fixes the performance regression seen. Therefore, > > Tested-by: Maximilian Heyne > > for the other patch. Thanks, I've added this to the patch now. greg k-h