From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF506D6B6B3 for ; Wed, 30 Oct 2024 17:02:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 381738D0003; Wed, 30 Oct 2024 13:02:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 331428D0001; Wed, 30 Oct 2024 13:02:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AB758D0003; Wed, 30 Oct 2024 13:02:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E8D1E8D0001 for ; Wed, 30 Oct 2024 13:02:48 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8C635ACCFC for ; Wed, 30 Oct 2024 17:02:48 +0000 (UTC) X-FDA: 82730886882.19.32B93AE Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf09.hostedemail.com (Postfix) with ESMTP id B960B140052 for ; Wed, 30 Oct 2024 17:02:27 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fFBmrbsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of joannelkoong@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=joannelkoong@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730307710; a=rsa-sha256; cv=none; b=BvTQ90e9Sl3gUnTcpGve92YKk3iDl+wGc/on3QLupjNoyIHGo7F1yS7YphGCDlcpM1HVhI YDSJyHLzkg7IUwy93WQpp/SYQU7+kVjmXhu/011P6iyoxSnXvqUwYBruhQGortUzmUpJnS +n1x56iaifLfVynso0mwWg2giNVRwec= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fFBmrbsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of joannelkoong@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=joannelkoong@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730307710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ErSVIe1pDdqmUfCBsYvxa835SchST7bDaaEk/Wl0F+w=; b=hjoAsa9JX9TnCf0yfYVhaauDlESUFeVo1KPQ8mD4FcjNQPSlcuPaS7rsaAVoBr8vM7FKg2 JGo7P6808/vC45McBIHmC4BE7j2nb9lfHfUO8qTO8alfl1okLEyQEhr8W/f+jNogLc74VY xcgwom6jKdfcrNH2Awk8QKCIAe47xwk= Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4609d75e2f8so10423591cf.1 for ; Wed, 30 Oct 2024 10:02:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730307765; x=1730912565; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ErSVIe1pDdqmUfCBsYvxa835SchST7bDaaEk/Wl0F+w=; b=fFBmrbsNEo9NLKjxBiSCY+msdb3ybh1K8k3LjGO9j7wqQPO+4ScXZ4rMGeIYtCIuZX d3A8o9ykVaVZ7wKLQHtfgIOSy1w1/gW8rW9kkCyebiFfJSYoCXjzjUgxjkZH6wfp++7Z 0tTwUEgjB8txfYFdazmeGuPHZG+Z6dwpPd7fZBPRGc7ib/sZ0GvXB03ft8ch1vUd3de9 le5d+8/+TNafbir2gs6srRR45UjmmfDZR2ZslJrH7Gx1eLlXWR7Y7+GZoTm2qK/QmHeY wHH7P22IWDpGyPFxHkwnKyQGPTgrqyxs50T2c8NnvJQLf6ZrnJ7fbbhRvEffDmO8kGK4 j+Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730307765; x=1730912565; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ErSVIe1pDdqmUfCBsYvxa835SchST7bDaaEk/Wl0F+w=; b=Gt4UTSecV01hOfDG733eGZtoQmdZRIkdxRO7AQh4bZhEVohybt85f5rRsrPh6MlPCF i4Ag84HQ6geCzG8D7KfCy6jrDjBda95MnOU0ITRHeLu8sA0eGDsjbqX4HtWyQjt9Gxnz 1qNAqaOlFEJd1CmPMKYpxfYqAcgVUNaP3tFxb/Z3jxMFI9kU7hV9cZyQ3jdpp7F1wQ8U kkd+zRdKG1sEg1bJivkt8gCzgTCz9MXdbqeQEKbumruABEfcxUw4zH8yUlmShRU8DggX 3gv4vH63iiDPFX84lumr0faQK0O2dxZKLbfm3AVIGRKu5okF4WerljmXoKrEzrVKuPSC LTdA== X-Forwarded-Encrypted: i=1; AJvYcCWGoJU7/16QMnBBgQ5fBlgxx8Nprxx3+GXKU6Iy0y8HxJBkvZlmn/at+FhnIK2RjPEESvlBkmN42g==@kvack.org X-Gm-Message-State: AOJu0YySowwOwklg/r1nxJarwqkOhBUmnk7ExiJTMw4tW7Tv5oP8hXlm yM7kCD46Oz/TpLIbAfVwBzZbNOVye4QRMozK0t9X3oxY1xLJV94b8VeAaSheGLbX5ksc54umTTo Tl2wart4RSVnj7xCdb8OvnhG4Bbw= X-Google-Smtp-Source: AGHT+IGIwX1OJKitl+q7bsCqXDDs+7Kp/1uhtNHeVnWpk/nljaLb8jeBtsIxpM29kjs7zWuH6QSG3Vxx9MYh/XFhn88= X-Received: by 2002:a05:622a:391:b0:456:919a:11e4 with SMTP id d75a77b69052e-46168459349mr113594461cf.20.1730307764493; Wed, 30 Oct 2024 10:02:44 -0700 (PDT) MIME-Version: 1.0 References: <20241014182228.1941246-1-joannelkoong@gmail.com> <3e4ff496-f2ed-42ef-9f1a-405f32aa1c8c@linux.alibaba.com> <0c3e6a4c-b04e-4af7-ae85-a69180d25744@fastmail.fm> <023c4bab-0eb6-45c5-9a42-d8fda0abec02@fastmail.fm> In-Reply-To: <023c4bab-0eb6-45c5-9a42-d8fda0abec02@fastmail.fm> From: Joanne Koong Date: Wed, 30 Oct 2024 10:02:33 -0700 Message-ID: Subject: Re: [PATCH v2 2/2] fuse: remove tmp folio for writebacks and internal rb tree To: Bernd Schubert Cc: Jingbo Xu , Miklos Szeredi , Shakeel Butt , linux-fsdevel@vger.kernel.org, josef@toxicpanda.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@meta.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: B960B140052 X-Rspamd-Server: rspam01 X-Stat-Signature: 7uza5839unirm6uxud5yiwtws7b58wto X-HE-Tag: 1730307747-808286 X-HE-Meta: U2FsdGVkX1/62gC+82cpIKOCNNz5PEukgdu4ezQfciCkvSVJeLzubykeR+f2jVJq50brnULaxV9Le7skxx7tqu5bJU2AsqVDPYBICmiUfe2bh3zqnoWdE4Ntw6Zd4wcD9VpXYQx54xA/UdwDMjiSa0UIglQpAmbJDkLXXCExBOVEsykWlYVg/dwqmtBahg/V/zhbXlrywVudXVfsAAoAoKCcYjvYB2yq4KC0CspBHxoFponA18u9PpDlMPKCntSztlDV1uab6o59yh/yS8ii6ZXpzsB61q9IpF5Vhu3GiixMInGhFZsPHtq+Uf6GGI2rJ353rovewCfK6xR0FrzjJgtqSfabI+KcJ9JisiQe1aQrqSrAxYHMH5wdcp4KEIRaxCiherKV1Ru63W28EFTx1Ok9tS5uXvO17TxizFABn4KUOaj+Ep8eHWSGbtKyQpmYfJbKkjTkHu0O0yhd6Cy9JYDI/cFit+ORa99iB5Ye9QrSqTUsFa9sSFB6CEilb7fN38PkILXl5wGmD8PfVZlZf+fzOCjD6/gg0qSuReG6+lsPY4qX436BWE/k15UoKrOGzg/rTUVr18ct6hg+DGPv4sj6Fqa0WNp1mA3YWN6bmfI23zEuIP/vm3vfyyQ4kX0FlSw2PeETh34+xh+LYWh5r6tyj92bq9OHbSTUlczZzJrwuQxOQkU8jZysNT1I8wAkttYDr//WtRQC7ytwN+T1yp1ijXyJadrxo8Q6LtBiMyoznCN/MIDwUUoa0DcFkB6BTl1j1sErPd3QiKChIKHEO1Fwf2CdwLBB8UoRFB3mDxvyVcyGmG0WYwtG5FRhIXM1Ogb/4nwbqwRgp0wEW7rnboxrA63k/DxvsZBA+agQge9Sa+GP4UI6a9wWExoLWJPHT6eXw7tDu/SMVaxgJxle4XDug0Y04UjfcCFC9UK+qs5gfQT8TWmOdbNymEJmake2HfHobAOYsrrlz7FgiWo j9XuaJoh CVQfRzJIJPyr1tFmM1yKBUDih/mmeEHfyqsLqBSpyOpoIkEdxnP/FYPaQprRqrxNrQKpDCindOR9J/MW+JBfbNn26LIEoNt+n5JkeErmMHcGou0T943CPQKTa1xSC4C2b4KOpEov35j8t13cDTuRlnzKI/vrX42NvFPl4sq4E228+areQFvWB55o0KlXBge8/pRnI/1AaTLDuypbH/G6IepLndTRmgBM0x3lI7iuVhkaD9EAuR+e0NwbE4Vat1l0ADI4QTo4Da4vaCijmdRyPU0uVeE6vadbetMj9bWVBCvqZeRXKhLjajbqQdYLnU/XGNL71+BV8ab0VWaxX1vuZ+zJbiQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 30, 2024 at 9:21=E2=80=AFAM Bernd Schubert wrote: > > On 10/30/24 17:04, Joanne Koong wrote: > > On Wed, Oct 30, 2024 at 2:32=E2=80=AFAM Bernd Schubert > > wrote: > >> > >> On 10/28/24 22:58, Joanne Koong wrote: > >>> On Fri, Oct 25, 2024 at 3:40=E2=80=AFPM Joanne Koong wrote: > >>>> > >>>>> Same here, I need to look some more into the compaction / page > >>>>> migration paths. I'm planning to do this early next week and will > >>>>> report back with what I find. > >>>>> > >>>> > >>>> These are my notes so far: > >>>> > >>>> * We hit the folio_wait_writeback() path when callers call > >>>> migrate_pages() with mode MIGRATE_SYNC > >>>> ... -> migrate_pages() -> migrate_pages_sync() -> > >>>> migrate_pages_batch() -> migrate_folio_unmap() -> > >>>> folio_wait_writeback() > >>>> > >>>> * These are the places where we call migrate_pages(): > >>>> 1) demote_folio_list() > >>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode > >>>> > >>>> 2) __damon_pa_migrate_folio_list() > >>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode > >>>> > >>>> 3) migrate_misplaced_folio() > >>>> Can ignore this. It calls migrate_pages() in MIGRATE_ASYNC mode > >>>> > >>>> 4) do_move_pages_to_node() > >>>> Can ignore this. This calls migrate_pages() in MIGRATE_SYNC mode but > >>>> this path is only invoked by the move_pages() syscall. It's fine to > >>>> wait on writeback for the move_pages() syscall since the user would > >>>> have to deliberately invoke this on the fuse server for this to appl= y > >>>> to the server's fuse folios > >>>> > >>>> 5) migrate_to_node() > >>>> Can ignore this for the same reason as in 4. This path is only invok= ed > >>>> by the migrate_pages() syscall. > >>>> > >>>> 6) do_mbind() > >>>> Can ignore this for the same reason as 4 and 5. This path is only > >>>> invoked by the mbind() syscall. > >>>> > >>>> 7) soft_offline_in_use_page() > >>>> Can skip soft offlining fuse folios (eg folios with the > >>>> AS_NO_WRITEBACK_WAIT mapping flag set). > >>>> The path for this is soft_offline_page() -> soft_offline_in_use_page= () > >>>> -> migrate_pages(). soft_offline_page() only invokes this for in-use > >>>> pages in a well-defined state (see ret value of get_hwpoison_page())= . > >>>> My understanding of soft offlining pages is that it's a mitigation > >>>> strategy for handling pages that are experiencing errors but are not > >>>> yet completely unusable, and its main purpose is to prevent future > >>>> issues. It seems fine to skip this for fuse folios. > >>>> > >>>> 8) do_migrate_range() > >>>> 9) compact_zone() > >>>> 10) migrate_longterm_unpinnable_folios() > >>>> 11) __alloc_contig_migrate_range() > >>>> > >>>> 8 to 11 needs more investigation / thinking about. I don't see a goo= d > >>>> way around these tbh. I think we have to operate under the assumptio= n > >>>> that the fuse server running is malicious or benevolently but > >>>> incorrectly written and could possibly never complete writeback. So = we > >>>> definitely can't wait on these but it also doesn't seem like we can > >>>> skip waiting on these, especially for the case where the server uses > >>>> spliced pages, nor does it seem like we can just fail these with > >>>> -EBUSY or something. > >> > >> I see some code paths with -EAGAIN in migration. Could you explain why > >> we can't just fail migration for fuse write-back pages? > >> > > Hi Joanne, > > thanks a lot for your quick reply (especially as my reviews come in very > late). > Thanks for your comments/reviews, Bernd! I always appreciate them. > > > > My understanding (and please correct me here Shakeel if I'm wrong) is > > that this could block system optimizations, especially since if an > > unprivileged malicious fuse server never replies to the writeback > > request, then this completely stalls progress. In the best case > > scenario, -EAGAIN could be used because the server might just be slow > > in serving the writeback, but I think we need to also account for > > servers that never complete the writeback. For > > __alloc_contig_migrate_range() for example, my understanding is that > > this is used to migrate pages so that there are more physically > > contiguous ranges of memory freed up. If fuse writeback blocks that, > > then that hurts system health overall. > > Hmm, I wonder what is worse - tmp page copies or missing compaction. > Especially if we expect a low range of in-writeback pages/folios. > One could argue that an evil user might spawn many fuse server > processes to work around the default low fuse write-back limits, but > does that make any difference with tmp pages? And these cannot be > compacted either? My understanding (and Shakeel please jump in here if this isn't right) is that tmp pages can be migrated/compacted. I think it's only pages marked as under writeback that are considered to be non-movable. > > And with timeouts that would be so far totally uncritical, I > think. > > > You also mentioned > > > especially for the case where the server uses spliced pages > > could you provide more details for that? > For the page migration / compaction paths, I don't think we can do the workaround we could do for sync where we skip waiting on writeback for fuse folios and continue on with the operation, because the migration / compaction paths operate on the pages. For the splice case, we assign the page to the pipebuffer (fuse_ref_page()), so if the migration/compaction happens on the page before the server has read this page from the pipebuffer, it'll be incorrect data or maybe crash the kernel. > > > > > >>>> > >>> > >>> I'm still not seeing a good way around this. > >>> > >>> What about this then? We add a new fuse sysctl called something like > >>> "/proc/sys/fs/fuse/writeback_optimization_timeout" where if the sys > >>> admin sets this, then it opts into optimizing writeback to be as fast > >>> as possible (eg skipping the page copies) and if the server doesn't > >>> fulfill the writeback by the set timeout value, then the connection i= s > >>> aborted. > >>> > >>> Alternatively, we could also repurpose > >>> /proc/sys/fs/fuse/max_request_timeout from the request timeout > >>> patchset [1] but I like the additional flexibility and explicitness > >>> having the "writeback_optimization_timeout" sysctl gives. > >>> > >>> Any thoughts on this? > >> > >> > >> I'm a bit worried that we might lock up the system until time out is > >> reached - not ideal. Especially as timeouts are in minutes now. But > >> even a slightly stuttering video system not be great. I think we > >> should give users/admin the choice then, if they prefer slow page > >> copies or fast, but possibly shortly unresponsive system. > >> > > I was thinking the /proc/sys/fs/fuse/writeback_optimization_timeout > > would be in seconds, where the sys admin would probably set something > > more reasonable like 5 seconds or so. > > If this syctl value is set, then servers who want writebacks to be > > fast can opt into it at mount time (and by doing so agree that they > > will service writeback requests by the timeout or their connection > > will be aborted). > > > I think your current patch set has it in minutes? (Should be easy > enough to change that.) Though I'm more worried about the impact > of _frequent_ timeout scanning through the different fuse lists > on performance, than about missing compaction for folios that are > currently in write-back. > Ah, for this the " /proc/sys/fs/fuse/writeback_optimization_timeout" would be a separate thing from the "/proc/sys/fs/fuse/max_request_timeout". The "/proc/sys/fs/fuse/writeback_optimization_timeout" would only apply for writeback requests. I was thinking implementation-wise, for writebacks we could just have a timer associated with each request (instead of having to grab locks with the fuse lists), since they won't be super common. Thanks, Joanne > > Thanks, > Bernd