From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4C1FC25B78 for ; Tue, 4 Jun 2024 09:32:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40F936B00B7; Tue, 4 Jun 2024 05:32:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C0928D0001; Tue, 4 Jun 2024 05:32:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238CA6B00B8; Tue, 4 Jun 2024 05:32:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 035406B0092 for ; Tue, 4 Jun 2024 05:32:39 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A976F1A0D4B for ; Tue, 4 Jun 2024 09:32:39 +0000 (UTC) X-FDA: 82192691238.14.ECB3E4B Received: from fout7-smtp.messagingengine.com (fout7-smtp.messagingengine.com [103.168.172.150]) by imf16.hostedemail.com (Postfix) with ESMTP id 74CBF180027 for ; Tue, 4 Jun 2024 09:32:37 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=fastmail.fm header.s=fm1 header.b=SOeEetZh; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="b ths4Di"; dmarc=pass (policy=none) header.from=fastmail.fm; spf=pass (imf16.hostedemail.com: domain of bernd.schubert@fastmail.fm designates 103.168.172.150 as permitted sender) smtp.mailfrom=bernd.schubert@fastmail.fm ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717493557; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z/p/FhNJ0mP2yG98DTjrbvV/ibnqnwF1h5OakqDfidw=; b=v7IjfI+YgK0VuEUIh8QFNNmsimtk/BibCekwpN1DJnUA6rbZDK0uEecrbX7WqG4i3T2yXW Dct5jebOALgJVQSlyTVtkYl0YXkogID3k1FpTOH587iPLbD0XDYlcnDGdAi89o5DtdbSKM uNgmlJfct0xrtiY9VwzcTvPwV2NxK8g= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=fastmail.fm header.s=fm1 header.b=SOeEetZh; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="b ths4Di"; dmarc=pass (policy=none) header.from=fastmail.fm; spf=pass (imf16.hostedemail.com: domain of bernd.schubert@fastmail.fm designates 103.168.172.150 as permitted sender) smtp.mailfrom=bernd.schubert@fastmail.fm ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717493557; a=rsa-sha256; cv=none; b=hwQwtMJ9VHnl7VE4Pn9hlYPA7uiZjGcsWmnESt2dnkCGBQohXJn4ASv8eE9oqvcOl1svNH uY7cY0q5lVfVxfu3g4hwKGC7EVpEaBygTeAJzW/5FftFaSaFmNYgDm9A6+BR2kRKwrAbrj VEKEYj715eg9RYV4/QCmVTkZ0Y95kRI= Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.nyi.internal (Postfix) with ESMTP id A00D11380118; Tue, 4 Jun 2024 05:32:36 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Tue, 04 Jun 2024 05:32:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.fm; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1717493556; x=1717579956; bh=Z/p/FhNJ0mP2yG98DTjrbvV/ibnqnwF1h5OakqDfidw=; b= SOeEetZh0Ow/keZ1Ik1aO0oSKhs+/s4Kw/VEx4YcD/SmVXSWLDOY76xpjfP9WJlI RQDosdFOlCnE6boiapTohMGYH+X6vwr3CGDMraPUZlmuBw7NHwK1MvtxilAk4/ia OzggNKn46Ah+LsOQjzMdKxuEm4VuMsYkxjodEAtEp7J4BYSG+uAWT8CAdn0ysnCZ Ic8yXAthhRuOu7y+c99QUiBvQ5pbl7rkZSKprDmW9mgqx9MkLzt+4KCjV0f0HJJi pypGuHNJ+Aapql6o6sfrax0/UhrvywLdRI3VM1LaVhHxFIBqSjlU6smyWAxt+rZB 1NfPVftFM5lz0nvOGfStww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1717493556; x= 1717579956; bh=Z/p/FhNJ0mP2yG98DTjrbvV/ibnqnwF1h5OakqDfidw=; b=b ths4DilJb7/VHsQJgh3N5g0pO5MmvqdZ4Y+PbEWZf7Lg7H2stEYHv8NAjztb5PIS nAwfw6HvDoxFQ21bkzqSntl+Z7RyhW+J8k70zgsUpRa3NNm/gncZhzL13N+czJID Jf2pba+9AkXuCUZzfAm7nvMbyEdWAJ9PwkoCw/oFxkomH0tdwajfNl7GvfXa4Zfs vMzIGDIqWHPY1islMfyBzKuc7mlLIyjNLWdsCzfCEkPzEXHIi2FYnIchPEu3D14f Jw/0hYACvkWjC0Ep/STKlnpsn/b8XMPDZoYzNPaV+Ecig2sWisrP8ZKTts1vC/zQ 65FVrkeapHJFswq6Wi36Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdelgedgudefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeeuvghr nhguucfutghhuhgsvghrthcuoegsvghrnhgurdhstghhuhgsvghrthesfhgrshhtmhgrih hlrdhfmheqnecuggftrfgrthhtvghrnhepvefhgfdvledtudfgtdfggeelfedvheefieev jeeifeevieetgefggffgueelgfejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsggvrhhnugdrshgthhhusggvrhhtsehfrghsthhmrghilhdr fhhm X-ME-Proxy: Feedback-ID: id8a24192:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 4 Jun 2024 05:32:34 -0400 (EDT) Message-ID: <2f834b5c-d591-43c5-86ba-18509d77a865@fastmail.fm> Date: Tue, 4 Jun 2024 11:32:32 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [HELP] FUSE writeback performance bottleneck To: Jingbo Xu , Miklos Szeredi Cc: "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" , lege.wang@jaguarmicro.com, "Matthew Wilcox (Oracle)" , "linux-mm@kvack.org" References: <495d2400-1d96-4924-99d3-8b2952e05fc3@linux.alibaba.com> <67771830-977f-4fca-9d0b-0126abf120a5@fastmail.fm> From: Bernd Schubert Content-Language: en-US, de-DE, fr In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 74CBF180027 X-Stat-Signature: 64138xzipn7iib33pyeroqqiw6be6z6f X-HE-Tag: 1717493557-617209 X-HE-Meta: U2FsdGVkX1+AM0A4IdkzN5/yJ7Apr24D625JOcJgOWXnaun8ZJlSBudrI8oRIZ9YvF/cyWKSmcxmKqg7szzwkCL51pzAh5hgRbyuNsLg+iRMkFXV28AcqmkWk9aTST7GN2v106hy35M9nmGYNIF2CgP+7EKjNbm+n4LUY+jX/2zDbWSneCesBamhN4F0lO80xL/mMVgWI4YeTOkrZ7z2+aMxuvdFYvYvDMvjiaKgsbMYnDkh3bOVvP0HAExwFJfmaEijvM19LZQYj4pkziyA1gNjgAUwH8XHPDoqz9zOL9XQcHchlL4X11epLd001HG9hPgVMFASGN5gM9nc59uCI1UCJobaxuFFMh/WLDFxHLihjLmhXf2Ww3eGPZDtojDSWmu4fggVt0jjiREYQX4mvtliWTLBPLjKmgU1qwEwvsIRCLyPu2/gRXzTYit4V2K8Kt4OD4H4Lx7A+jdZSBjIBkxtJCZWnd7edkuDw0iDsrj+GovfeV4uERtk/uGIbqqb1EFDKkmfAP3MmCiAhl+nasbulviqaV/uS398C8t19plcXY1Q1X1X68dzQlG8gZpVqQaJtWLxYjLVyTNqNfwnFuLAFraXI3J/7WQBS9Wyz4QMxIdsf/TMJ6//m4S7SgC+xzvGywHZQYTAbr/JaR5NJyiPs75OEQK+0Sfw3i0L05eYeoRIS5eBzLmUtA6v5FchwGaNEL1Wt+G1b0hD6ACcVltCFNEbYBptSIaN5BuPzjRRFoB2J7I4wuQ9dniD4Nb6yX4mANImfxsUll1jX21LN787BHaaBCaVDezIefYjMHVMUQyuLHn7eY+LNVgH5W7qOTywxNET1tNFFZQCMYDgZ8V+q0MjdGUJ7lO+wh66EIHSimc0vWEU4/aY+QxOE53iVjREhUVYfPcc5Rv4HivG8IHvAZrGCUVxtbKShDdU3Y/jwjIegLkTXSYpkLbxusuL/EEZ/rFtNPcTKsAK//3 J8atsRlw ectioiFn61sjWaYVz84yh0M7MNAPeo68MNgW6ChKnxDHxE76lpq91WY8i2+ku7Jg3WGbxDkzkFvqwuIX1HAqjRXGdqXxF7f3IeRfAlmfte++0O/BXjugiPCHsLaUYiz5h6m3NAcnRlOpFSOX4Yb+HZoTfEoMT9oMznfrepV+rmf4Wv1PJe1hdpLx67ooWB/z0btNIqnnuMUACvBkuvfCcRjtb4Zv1HpYplNHpKqZB1isqs3CyAik7XfsNNM0a4NZGrMlrS4k9SbXVFrjIbbnkMNb8TXVtBwWxlQ+0Pfi7KI0Ef8+g54mh3z2Qrau+8kTfQUcmZgxFGnt/r1ro9yZcPCsHwb/1JO69hy4vT30myEqI6mI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/4/24 09:36, Jingbo Xu wrote: > > > On 6/4/24 3:27 PM, Miklos Szeredi wrote: >> On Tue, 4 Jun 2024 at 03:57, Jingbo Xu wrote: >> >>> IIUC, there are two sources that may cause deadlock: >>> 1) the fuse server needs memory allocation when processing FUSE_WRITE >>> requests, which in turn triggers direct memory reclaim, and FUSE >>> writeback then - deadlock here >> >> Yep, see the folio_wait_writeback() call deep in the guts of direct >> reclaim, which sleeps until the PG_writeback flag is cleared. If that >> happens to be triggered by the writeback in question, then that's a >> deadlock. >> >>> 2) a process that trigfgers direct memory reclaim or calls sync(2) may >>> hang there forever, if the fuse server is buggyly or malicious and thus >>> hang there when processing FUSE_WRITE requests >> >> Ah, yes, sync(2) is also an interesting case. We don't want unpriv >> fuse servers to be able to block sync(2), which means that sync(2) >> won't actually guarantee a synchronization of fuse's dirty pages. I >> don't think there's even a theoretical solution to that, but >> apparently nobody cares... > > Okay if the temp page design is unavoidable, then I don't know if there > is any approach (in FUSE or VFS layer) helps page copy offloading. At > least we don't want the writeback performance to be limited by the > single writeback kworker. This is also the initial attempt of this thread. > Offloading it to another thread is just a workaround, though maybe a temporary solution. Back to the background for the copy, so it copies pages to avoid blocking on memory reclaim. With that allocation it in fact increases memory pressure even more. Isn't the right solution to mark those pages as not reclaimable and to avoid blocking on it? Which is what the tmp pages do, just not in beautiful way. Thanks, Bernd