From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3287BD6B6B3 for ; Wed, 30 Oct 2024 17:27:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B84436B009F; Wed, 30 Oct 2024 13:27:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B33816B00A1; Wed, 30 Oct 2024 13:27:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D45B6B00A6; Wed, 30 Oct 2024 13:27:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 803BD6B009F for ; Wed, 30 Oct 2024 13:27:51 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 295B3C099C for ; Wed, 30 Oct 2024 17:27:51 +0000 (UTC) X-FDA: 82730950890.28.41483DF Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) by imf17.hostedemail.com (Postfix) with ESMTP id BE9DD4000F for ; Wed, 30 Oct 2024 17:27:28 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=fastmail.fm header.s=fm3 header.b=cFhPKo7a; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="a z7yaEr"; dmarc=pass (policy=none) header.from=fastmail.fm; spf=pass (imf17.hostedemail.com: domain of bernd.schubert@fastmail.fm designates 202.12.124.151 as permitted sender) smtp.mailfrom=bernd.schubert@fastmail.fm ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730309138; a=rsa-sha256; cv=none; b=G7WL3zqRSMR5DBb9wxnAJZgHUNMQgj6Nfd9/UHpOEJTQRdaywn/AT57tIrMnHWtyCWIOyM GUmi/F7WAR8zAcRrKtUvQMYTLOgbPMUQ2NaosqxOYtxLmlQwSNbDE3ePSqYQFK4BNtEVXI Xwrdm5VK5DXl50spRSf77fqiOQ4LYMM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=fastmail.fm header.s=fm3 header.b=cFhPKo7a; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="a z7yaEr"; dmarc=pass (policy=none) header.from=fastmail.fm; spf=pass (imf17.hostedemail.com: domain of bernd.schubert@fastmail.fm designates 202.12.124.151 as permitted sender) smtp.mailfrom=bernd.schubert@fastmail.fm ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730309138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HEHVsGBzrirAoADNA8r+pSWIy2xvTKgTmjRILTAQNCE=; b=HHzyLd6Vxu5/1gbishgD3KgnIHeykI24PuN+yngoi0QRP8Xc6U8qu1Ydh3Zdz4HXZtyzDY 80R8Wv8m/ABlLUQmgzHS2fHYyyRZu3cLgO6gAWhlRQn6QzgHJJWiQ01sW+jsSjzxZ5CR2S YgrpZ6Ig8pSfkrd9+JkDWBQ51FztDyc= Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfout.stl.internal (Postfix) with ESMTP id BE5FC1140130; Wed, 30 Oct 2024 13:27:47 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Wed, 30 Oct 2024 13:27:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.fm; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1730309267; x=1730395667; bh=HEHVsGBzrirAoADNA8r+pSWIy2xvTKgTmjRILTAQNCE=; b= cFhPKo7aDcTxAqsu4fw8apdPyHLlMo0fkPzqOeN1CMtBbhPHGpt61p4Zb++cM6gO +LBmsG24i5z+ngcOMT0OB6+583KtG+13H5yFcjN/axH//Pt3KIv7eU8NRuWs0fT/ bnZId7CJFWVjFy/IBqOd+ibRqugBWKzFWOIUQ3XrP2hn0QxTRPGrxTCX9fPFjQRw mjFaEvAXYG43aGGKXbgJLwIYJ/UC2t6SExdh/SFWdvkWhQyvNMp7L82yPcTq0mIb bxJGv9W1PeohSBKMCzmI37mhtarIdP3BSs3rg/xUhMtCfMdVwg6y8ciHs5l2CfZS 0LdWU9d+5wVdL28r21P2lA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1730309267; x= 1730395667; bh=HEHVsGBzrirAoADNA8r+pSWIy2xvTKgTmjRILTAQNCE=; b=a z7yaErcupzsBl5CSthoBbxmbQM+VPq7XLa55yp4X6aKHedmVsttO2z6WOnO+P3tB Eh2kKciIBsTKSTgpYLMg8/EdRk1N5tB3IsPP9oICKyO/btUbwddRoPGHLSnxIJTu 4bxLzwKlzjpTL0m0tGI4RFXY6Xyrki/pVj4RjqWdMu7pAX0/4nKYUqWWOygSoI+c JfGDYVvv6bXXx+wl40c6eZp+j5dMiWDKoRXMs0Cva6ouPhLpA4cRii+CZe2mtp/w P5RYSmqBFXkwF+IWC089X2kHJwvB5xcCb1KSCpZTpb8KJHsn7REDalLMPUGm8VW0 lSy4rwhWzFEr3OCQrzVJg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdekfedguddttdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefkffggfgfuvfevfhfhjggtgfesthekredttddv jeenucfhrhhomhepuegvrhhnugcuufgthhhusggvrhhtuceosggvrhhnugdrshgthhhusg gvrhhtsehfrghsthhmrghilhdrfhhmqeenucggtffrrghtthgvrhhnpeduleefvdduvedu veelgeelffffkedukeegveelgfekleeuvdehkeehheehkefhfeenucevlhhushhtvghruf hiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsvghrnhgurdhstghhuhgsvghr thesfhgrshhtmhgrihhlrdhfmhdpnhgspghrtghpthhtohepledpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepjhhorghnnhgvlhhkohhonhhgsehgmhgrihhlrdgtohhmpdhr tghpthhtohepjhgvfhhflhgvgihusehlihhnuhigrdgrlhhisggrsggrrdgtohhmpdhrtg hpthhtohepmhhikhhlohhssehsiigvrhgvughirdhhuhdprhgtphhtthhopehshhgrkhgv vghlrdgsuhhttheslhhinhhugidruggvvhdprhgtphhtthhopehlihhnuhigqdhfshguvg hvvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepjhhoshgvfhesthho gihitghprghnuggrrdgtohhmpdhrtghpthhtohephhgrnhhnvghssegtmhhpgigthhhgrd horhhgpdhrtghpthhtoheplhhinhhugidqmhhmsehkvhgrtghkrdhorhhgpdhrtghpthht ohepkhgvrhhnvghlqdhtvggrmhesmhgvthgrrdgtohhm X-ME-Proxy: Feedback-ID: id8a24192:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 30 Oct 2024 13:27:44 -0400 (EDT) Message-ID: Date: Wed, 30 Oct 2024 18:27:42 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 2/2] fuse: remove tmp folio for writebacks and internal rb tree To: Joanne Koong Cc: Jingbo Xu , Miklos Szeredi , Shakeel Butt , linux-fsdevel@vger.kernel.org, josef@toxicpanda.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@meta.com References: <20241014182228.1941246-1-joannelkoong@gmail.com> <3e4ff496-f2ed-42ef-9f1a-405f32aa1c8c@linux.alibaba.com> <0c3e6a4c-b04e-4af7-ae85-a69180d25744@fastmail.fm> <023c4bab-0eb6-45c5-9a42-d8fda0abec02@fastmail.fm> From: Bernd Schubert Content-Language: en-US, de-DE, fr In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: BE9DD4000F X-Stat-Signature: ojfz4548ecepxgdzdye5oszfz5ifq4hb X-Rspam-User: X-HE-Tag: 1730309248-805059 X-HE-Meta: U2FsdGVkX1+u8R/zoqnsyQwNF06Ya5b4sPxWGRyDfa5mnY0TbKFpRMaKK7opWwmYRjzVxi9ENZvkUxLo5H9Ls8coS3XDfan0H65IESPZgrPSS/wN3KfVGpWI5gGy66T8Xjh3EXnRBk8X/YE0lPxgpaNe4ERSYZYlIhL9WEXk40/Kf2fuKBDsRZIUL0YJNSnXnm2pWnikPd5aTnTlX1Efib+H3htYs/Fky5/Ljl6/JPTDBTpeHpKmTjz+r7IZ0DH5JtxqA4BoDEmoV1YbpybHcnlStIWaSTKV8d3H0DrLvAgarAAr6Dehd4Vyhxt1+CblfjZ9pHHMIBxYE1FSFUYwNBMQStzqLDTmvnRAAwb/w/CyUnUOeBEiNtdHsb2gP1+72nbff+bM62HWicB2l4SGUaKSh4a+Eg0k0DFS+GWuzrWGrQwtfJqcxkRjKgJ+HJTfPiJLtwYhLE0kvhBa4kJ7srPzAk08o0xgkqGClP2u9H7nyjUROq4FzYp/PxABZT2d8P1zTyizDXmXM1SAPzPjTNpuWgw2TuwLxf/WolhDHYh+TyVatMX/ZFIQUREtifyc0k28ORBD42HNG2+CvErkVVmi0fDY4J3h+1zva6rf3DWJHx6qC1IA+DATOqY5szu9OOcZcA7R5QDiFyApEh2MvsKJlDaazrrpHU24xJTARYDRNvQISaUlV30TvrkDez/tlNvZ9REFPh4JI6diCkKaEPOHVseSV04ZkEERn4J31czjUYJZwd1t86fcBZ7O3f8sOczbFVXlEOhXr4A49gpBtgBkkduEDjIfsVNyfRT09tKDrWRrolHeJcXeK/oiTYi4tHZ718YpPULgFRtDC0GN5vdb4kB2xwVXy4B/X9C8PoOhdvzp9Uc4kjjAS5yEjn4AimBLx0iZnbJDK+yuL8uGRC3rX4yFdjHnHRgKj7OsqsemFSeCJzVJ4AFnazqw2ZUGgfraci6/HyJb4PDnEN+ gtzFo4xB gFXx9PP1XpXRypctGuSDDeAUTEFwKX68oc3dCBOkb1YneI5DP6Z9sjfNKGC6qw/XBGtgnm5T2KEKDWo2dOcATNBKsRDTHRLHGtIS/kryjVgCvaUZbWQMGrBNvV2pLr03Nx6wTt2pj8Uq3Z9x58j75gXIM1mpcVyJdRxxcLTkaZ2Q3ubTRpzM5Icz1FHcLVg1DnSmSy6rsaJpdKzhfjwyU9UcXcIrhZ3JlOntL7MpDCAawfQ9eCf5EBJJX0NEU9VB2pn1mUS9jhZqyWiytCGoCMQizNgA4vD9n3TEh1UceW7ghhWyyZgSeE+N/QdjGgNlBY+gA8IRQf+7m8hJ/5rCmhvuYZgkWh4aJu3LBxkQPa0HHkL/jzQl+uh1TJNZXrwH3vFzjpTKJL29Vhy58ITHwxiwCpr8OoYhf30dV5l4KcMYQ1P39fwYc9ZDYDQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 10/30/24 18:02, Joanne Koong wrote: > On Wed, Oct 30, 2024 at 9:21 AM Bernd Schubert > wrote: >> >> On 10/30/24 17:04, Joanne Koong wrote: >>> On Wed, Oct 30, 2024 at 2:32 AM Bernd Schubert >>> wrote: >>>> >>>> On 10/28/24 22:58, Joanne Koong wrote: >>>>> On Fri, Oct 25, 2024 at 3:40 PM Joanne Koong wrote: >>>>>> >>>>>>> Same here, I need to look some more into the compaction / page >>>>>>> migration paths. I'm planning to do this early next week and will >>>>>>> report back with what I find. >>>>>>> >>>>>> >>>>>> These are my notes so far: >>>>>> >>>>>> * We hit the folio_wait_writeback() path when callers call >>>>>> migrate_pages() with mode MIGRATE_SYNC >>>>>> ... -> migrate_pages() -> migrate_pages_sync() -> >>>>>> migrate_pages_batch() -> migrate_folio_unmap() -> >>>>>> folio_wait_writeback() >>>>>> >>>>>> * These are the places where we call migrate_pages(): >>>>>> 1) demote_folio_list() >>>>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode >>>>>> >>>>>> 2) __damon_pa_migrate_folio_list() >>>>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode >>>>>> >>>>>> 3) migrate_misplaced_folio() >>>>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode >>>>>> >>>>>> 4) do_move_pages_to_node() >>>>>> Can ignore this. This calls migrate_pages() in MIGRATE_SYNC mode but >>>>>> this path is only invoked by the move_pages() syscall. It's fine to >>>>>> wait on writeback for the move_pages() syscall since the user would >>>>>> have to deliberately invoke this on the fuse server for this to apply >>>>>> to the server's fuse folios >>>>>> >>>>>> 5) migrate_to_node() >>>>>> Can ignore this for the same reason as in 4. This path is only invoked >>>>>> by the migrate_pages() syscall. >>>>>> >>>>>> 6) do_mbind() >>>>>> Can ignore this for the same reason as 4 and 5. This path is only >>>>>> invoked by the mbind() syscall. >>>>>> >>>>>> 7) soft_offline_in_use_page() >>>>>> Can skip soft offlining fuse folios (eg folios with the >>>>>> AS_NO_WRITEBACK_WAIT mapping flag set). >>>>>> The path for this is soft_offline_page() -> soft_offline_in_use_page() >>>>>> -> migrate_pages(). soft_offline_page() only invokes this for in-use >>>>>> pages in a well-defined state (see ret value of get_hwpoison_page()). >>>>>> My understanding of soft offlining pages is that it's a mitigation >>>>>> strategy for handling pages that are experiencing errors but are not >>>>>> yet completely unusable, and its main purpose is to prevent future >>>>>> issues. It seems fine to skip this for fuse folios. >>>>>> >>>>>> 8) do_migrate_range() >>>>>> 9) compact_zone() >>>>>> 10) migrate_longterm_unpinnable_folios() >>>>>> 11) __alloc_contig_migrate_range() >>>>>> >>>>>> 8 to 11 needs more investigation / thinking about. I don't see a good >>>>>> way around these tbh. I think we have to operate under the assumption >>>>>> that the fuse server running is malicious or benevolently but >>>>>> incorrectly written and could possibly never complete writeback. So we >>>>>> definitely can't wait on these but it also doesn't seem like we can >>>>>> skip waiting on these, especially for the case where the server uses >>>>>> spliced pages, nor does it seem like we can just fail these with >>>>>> -EBUSY or something. >>>> >>>> I see some code paths with -EAGAIN in migration. Could you explain why >>>> we can't just fail migration for fuse write-back pages? >>>> >> >> Hi Joanne, >> >> thanks a lot for your quick reply (especially as my reviews come in very >> late). >> > > Thanks for your comments/reviews, Bernd! I always appreciate them. > >>> >>> My understanding (and please correct me here Shakeel if I'm wrong) is >>> that this could block system optimizations, especially since if an >>> unprivileged malicious fuse server never replies to the writeback >>> request, then this completely stalls progress. In the best case >>> scenario, -EAGAIN could be used because the server might just be slow >>> in serving the writeback, but I think we need to also account for >>> servers that never complete the writeback. For >>> __alloc_contig_migrate_range() for example, my understanding is that >>> this is used to migrate pages so that there are more physically >>> contiguous ranges of memory freed up. If fuse writeback blocks that, >>> then that hurts system health overall. >> >> Hmm, I wonder what is worse - tmp page copies or missing compaction. >> Especially if we expect a low range of in-writeback pages/folios. >> One could argue that an evil user might spawn many fuse server >> processes to work around the default low fuse write-back limits, but >> does that make any difference with tmp pages? And these cannot be >> compacted either? > > My understanding (and Shakeel please jump in here if this isn't right) > is that tmp pages can be migrated/compacted. I think it's only pages > marked as under writeback that are considered to be non-movable. > >> >> And with timeouts that would be so far totally uncritical, I >> think. >> >> >> You also mentioned >> >>> especially for the case where the server uses spliced pages >> >> could you provide more details for that? >> 7> > For the page migration / compaction paths, I don't think we can do the > workaround we could do for sync where we skip waiting on writeback for > fuse folios and continue on with the operation, because the migration > / compaction paths operate on the pages. For the splice case, we > assign the page to the pipebuffer (fuse_ref_page()), so if the > migration/compaction happens on the page before the server has read > this page from the pipebuffer, it'll be incorrect data or maybe crash > the kernel. > >> >> >>> >>>>>> >>>>> >>>>> I'm still not seeing a good way around this. >>>>> >>>>> What about this then? We add a new fuse sysctl called something like >>>>> "/proc/sys/fs/fuse/writeback_optimization_timeout" where if the sys >>>>> admin sets this, then it opts into optimizing writeback to be as fast >>>>> as possible (eg skipping the page copies) and if the server doesn't >>>>> fulfill the writeback by the set timeout value, then the connection is >>>>> aborted. >>>>> >>>>> Alternatively, we could also repurpose >>>>> /proc/sys/fs/fuse/max_request_timeout from the request timeout >>>>> patchset [1] but I like the additional flexibility and explicitness >>>>> having the "writeback_optimization_timeout" sysctl gives. >>>>> >>>>> Any thoughts on this? >>>> >>>> >>>> I'm a bit worried that we might lock up the system until time out is >>>> reached - not ideal. Especially as timeouts are in minutes now. But >>>> even a slightly stuttering video system not be great. I think we >>>> should give users/admin the choice then, if they prefer slow page >>>> copies or fast, but possibly shortly unresponsive system. >>>> >>> I was thinking the /proc/sys/fs/fuse/writeback_optimization_timeout >>> would be in seconds, where the sys admin would probably set something >>> more reasonable like 5 seconds or so. >>> If this syctl value is set, then servers who want writebacks to be >>> fast can opt into it at mount time (and by doing so agree that they >>> will service writeback requests by the timeout or their connection >>> will be aborted). >> >> >> I think your current patch set has it in minutes? (Should be easy >> enough to change that.) Though I'm more worried about the impact >> of _frequent_ timeout scanning through the different fuse lists >> on performance, than about missing compaction for folios that are >> currently in write-back. Hmm, if tmp pages can be compacted, isn't that a problem for splice? I.e. I don't understand what the difference between tmp page and write-back page for migration. >> > > Ah, for this the " /proc/sys/fs/fuse/writeback_optimization_timeout" > would be a separate thing from the > "/proc/sys/fs/fuse/max_request_timeout". The > "/proc/sys/fs/fuse/writeback_optimization_timeout" would only apply > for writeback requests. I was thinking implementation-wise, for > writebacks we could just have a timer associated with each request > (instead of having to grab locks with the fuse lists), since they > won't be super common. Ah, thank you! I had missed that this is another variable. Issue with too short timeouts would probably be network hick-up that would immediately kill fuse-server. I.e. if it just the missing page compaction/migration, maybe larger time outs would be acceptable. Thanks, Bernd