From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 760E7C433FE for ; Sun, 9 Jan 2022 20:48:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB59D6B0072; Sun, 9 Jan 2022 15:48:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B64A56B0073; Sun, 9 Jan 2022 15:48:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A06096B0074; Sun, 9 Jan 2022 15:48:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0057.hostedemail.com [216.40.44.57]) by kanga.kvack.org (Postfix) with ESMTP id 8ECF66B0072 for ; Sun, 9 Jan 2022 15:48:31 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 53A6E824C421 for ; Sun, 9 Jan 2022 20:48:31 +0000 (UTC) X-FDA: 79011936822.12.486BCCB Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf03.hostedemail.com (Postfix) with ESMTP id 64E1620012 for ; Sun, 9 Jan 2022 20:48:30 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 12140B80E3C; Sun, 9 Jan 2022 20:48:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3623DC36AE3; Sun, 9 Jan 2022 20:48:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1641761307; bh=LsHeiNVZfdCTH6ss1wKhfos+xXMSUASjz79/xSlYqL4=; h=In-Reply-To:References:Date:From:To:Cc:Subject:From; b=krDGV2YzWlKoFmCkH2ukFo1N3fRA8GMf8aU/kIWrzKO0Kgvv83DJXfI9K1431nI5Z 3I8Ykynazlre9CGfJcc50NnXSkO/THagrvnd48LF4pgp98t5zLuGyCqq5+KdFLQAHd lal+KB3b4rfIE3v8vzgEHeo4wn5pOig0lPf6FCkXobT0u+Tm++vbHx4wcNUbXbBHjK w1Ol6ViIbdH7OElwLtOQN/QYOJTMCjzL1hKMaTmwl9OusfBaI7i64sZvnVHovvrAJq GgNcOzOW5NgQ+gIqLRtefj919/3LEyppsD5fDlZTodIwCl4jpev2bL8Yb75g3qE1qH OAXy0xLdUaTkQ== Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailauth.nyi.internal (Postfix) with ESMTP id 0734227C0054; Sun, 9 Jan 2022 15:48:25 -0500 (EST) Received: from imap48 ([10.202.2.98]) by compute6.internal (MEProxy); Sun, 09 Jan 2022 15:48:26 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrudegkedgudegfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedftehn ugihucfnuhhtohhmihhrshhkihdfuceolhhuthhosehkvghrnhgvlhdrohhrgheqnecugg ftrfgrthhtvghrnheptdfhheettddvtedvtedugfeuuefhtddugedvleevleefvdetleff gfefvdekgeefnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrh homheprghnugihodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdduudeiudek heeifedvqddvieefudeiiedtkedqlhhuthhopeepkhgvrhhnvghlrdhorhhgsehlihhnuh igrdhluhhtohdruhhs X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 7ABA521E006E; Sun, 9 Jan 2022 15:48:25 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-4526-gbc24f4957e-fm-20220105.001-gbc24f495 Mime-Version: 1.0 Message-Id: In-Reply-To: <0B23F7F5-96F1-4E2C-8C17-0ECC21CA14C6@gmail.com> References: <7c9c388c388df8e88bb5d14828053ac0cb11cf69.1641659630.git.luto@kernel.org> <739A3109-04DD-4BA5-A02B-52EE30E820AE@gmail.com> <355c148c-06a8-4e15-a77b-0ea2e22bf708@www.fastmail.com> <0B23F7F5-96F1-4E2C-8C17-0ECC21CA14C6@gmail.com> Date: Sun, 09 Jan 2022 13:48:05 -0700 From: "Andy Lutomirski" To: "Nadav Amit" Cc: "Linus Torvalds" , "Andrew Morton" , Linux-MM , "Nicholas Piggin" , "Anton Blanchard" , "Benjamin Herrenschmidt" , "Paul Mackerras" , "Randy Dunlap" , linux-arch , "the arch/x86 maintainers" , "Rik van Riel" , "Dave Hansen" , "Peter Zijlstra (Intel)" , "Mathieu Desnoyers" Subject: Re: [PATCH 16/23] sched: Use lightweight hazard pointers to grab lazy mms Content-Type: text/plain Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=krDGV2Yz; spf=pass (imf03.hostedemail.com: domain of luto@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=luto@kernel.org; dmarc=pass (policy=none) header.from=kernel.org X-Rspamd-Queue-Id: 64E1620012 X-Stat-Signature: xgbnoeurej4aqhdr5p3irgrc57osiuju X-Rspamd-Server: rspam04 X-HE-Tag: 1641761310-98926 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Jan 9, 2022, at 1:34 PM, Nadav Amit wrote: >> On Jan 9, 2022, at 11:52 AM, Andy Lutomirski wrote: >> >> On Sun, Jan 9, 2022, at 11:10 AM, Linus Torvalds wrote: >>> On Sun, Jan 9, 2022 at 12:49 AM Nadav Amit wrote: >>>> >>>> I do not know whether it is a pure win, but there is a tradeoff. >>> >>> Hmm. I guess only some serious testing would tell. >>> >>> On x86, I'd be a bit worried about removing lazy TLB simply because >>> even with ASID support there (called PCIDs by Intel for NIH reasons), >>> the actual ASID space on x86 was at least originally very very >>> limited. >>> >>> Architecturally, x86 may expose 12 bits of ASID space, but iirc at >>> least the first few implementations actually only internally had one >>> or two bits, and hashed the 12 bits down to that internal very limited >>> hardware TLB ID space. >>> >>> We only use a handful of ASIDs per CPU on x86 partly for this reason >>> (but also since there's no remote hardware TLB shootdown, there's no >>> reason to have a bigger global ASID space, so ASIDs aren't _that_ >>> common). >>> >>> And I don't know how many non-PCID x86 systems (perhaps virtualized?) >>> there might be out there. >>> >>> But it would be very interesting to test some "disable lazy tlb" >>> patch. The main problem workloads tend to be IO, and I'm not sure how >>> many of the automated performance tests would catch issues. I guess >>> some threaded pipe ping-pong test (with each thread pinned to >>> different cores) would show it. >> >> My original PCID series actually did remove lazy TLB on x86. I don't remember why, but people objected. The issue isn't the limited PCID space -- IIRC it's just that MOV CR3 is slooooow. If we get rid of lazy TLB on x86, then we are writing CR3 twice on even a very short idle. That adds maybe 1k cycles, which isn't great. > > Just for the record: I just ran a short test when CPUs are on max freq > on my skylake. MOV-CR3 without flush is 250-300 cycles. One can argue > that you mostly only care for one of the switches for the idle thread > (once you wake up). And waking up by itself has its overheads. > > But you are the master of micro optimizations, and as Rik said, I > mostly think of TLB shootdowns and might miss the big picture. Just > trying to make your life easier by less coding and my life simpler > in understanding your super-smart code. ;-) As Rik pointed out, the mm_cpumask manipulation is also expensive if we get rid of lazy. Let me ponder how to do this nicely.