From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F10ACC27C52 for ; Wed, 5 Jun 2024 05:50:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58DC16B0083; Wed, 5 Jun 2024 01:50:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F0DA6B0089; Wed, 5 Jun 2024 01:50:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3698D6B008C; Wed, 5 Jun 2024 01:50:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0F7D86B0083 for ; Wed, 5 Jun 2024 01:50:05 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 844CEA0E83 for ; Wed, 5 Jun 2024 05:50:04 +0000 (UTC) X-FDA: 82195759128.03.667AD28 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf06.hostedemail.com (Postfix) with ESMTP id C0900180009 for ; Wed, 5 Jun 2024 05:50:01 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UwcQq8Hh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of amir73il@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=amir73il@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717566601; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qr2WBNrHmUTP/4akueajoP+o69Bup6GqxHj8Jk3mdxE=; b=ii8l9xKavtccxdoMPIUM3uZ5OTGIg4DWrKB6mTQdcL6f4zc6zpicxAtdjDtFTQWpD/ymIV tN4bl1eYXUWXRTaX7MTeUxCkCAHgqduUwfg0FkJQvE/pHTntOm/j5hz7eUARuPip4GtFLm 8ajng3BXriVfy/OKG+uI+UGEAAq9ih8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UwcQq8Hh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of amir73il@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=amir73il@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717566601; a=rsa-sha256; cv=none; b=h0TrPVirUU4BH0F7aUjCt7vgneeD7xDhKlxeKr1h67GO+fPDMz98bV8NFfaxj+V9vcz42V aieuqXOgP9y8ENY42fnlac02wgEQhEeRk0+I/xgr9IFYMEhmlT2dxGdJ4kJ5bbZxxgYfqc YWFrCYkTlzsWgUObbQsURgBhse/AGJc= Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-795186ae3efso80435985a.1 for ; Tue, 04 Jun 2024 22:50:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717566601; x=1718171401; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=qr2WBNrHmUTP/4akueajoP+o69Bup6GqxHj8Jk3mdxE=; b=UwcQq8Hh+cQEmZdheBKvaz0sYlADCaObYSO4I4htJ5m6C1TAd5nYFVFNS+gQyEwHjI FPi8SV+MFESxXiKSgIUGj+IIkFBIA/ZE8qjzMOuGxPz0d2N9RKXdoXPOGNw4y7cqeb5b DiFJh4kZoaB5ek3NY+5LB+8/U4QggDZbUV3Y3FmD+LaNi1ax0C56fb/s0Wgkig59V87K 7bTrl0Q2yX6OuB2RLiQInbxZEBA1OujVd/e/LmSNvyYC6x3caoFxsbsG3xQO3QcALbKJ Wg9dYNmo3WZThpzPImnF7n590ruPCeBQ4RUFT7Il/6lMgTzAntDFgdvTMm5kpyUdgj/n Dt8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717566601; x=1718171401; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qr2WBNrHmUTP/4akueajoP+o69Bup6GqxHj8Jk3mdxE=; b=O+aBrYDWaQv+RbTuXlzTbVfvSLmRBHujAyhlDNH3KHonMIUAwhoBD37UWuBIbWINBY opdiQIkPh72YqoTpxaTxSaXj/xTXuo5iLq3Uh81SAqN99lQ+i/4eMa/yk+69e0rAN7YF YiQYdJrTLjI4pp6TKqOCgL+zejUbskoRJ4LEpyUEr3RIIzpUTeFoI2W398v3gTwrKgMW Sb/4vP+8N4atoym5aMQyM1KvuMH2YR2YAnJMTec+ytc1sLJh/mtGzcBlZv3M30KKrPko mWEt9YP5MNV/Zr2wEYRh+4LEWt4xcYA2czA9IH/Wcq0xs0mLBdoVePVZsAyhKwV6ZJok tIiA== X-Forwarded-Encrypted: i=1; AJvYcCX8QDqb5jxd9qcSvtDQvBoxKZ59ioySKzlLO3CTV0OlqxbKsdqkk3INb95a+51VzmHVD0EQ5xDr0MH8QDLwY0cvFtM= X-Gm-Message-State: AOJu0Yx0yE/Dyep+T5IFm18KlKeS5hfAYBtCBmYi1wSvLESIy3q1J0Ht YeYOwmKXpSeSifyLSZM5pV2iHl30kwi2QshyRkoeQCRBeV3uBUcFI0mpY/lsWViHQr7/E6RLIr4 5BqrTw/orzyvUdQWFBSiebtpTKwk= X-Google-Smtp-Source: AGHT+IGcQ1cPWxGbVNyd3pwcoakHuFvvgQuZ9N/5xFSdQxMvy0LV2gfCKQA3/ySDjUj4T5GP625izNpPx+rQJwWZXqY= X-Received: by 2002:a05:620a:8d6:b0:795:219d:c68e with SMTP id af79cd13be357-79523d35d2dmr148351685a.14.1717566600473; Tue, 04 Jun 2024 22:50:00 -0700 (PDT) MIME-Version: 1.0 References: <67771830-977f-4fca-9d0b-0126abf120a5@fastmail.fm> <2f834b5c-d591-43c5-86ba-18509d77a865@fastmail.fm> <21741978-a604-4054-8af9-793085925c82@fastmail.fm> <20240604165319.GG3413@localhost.localdomain> <6853a389-031b-4bd6-a300-dea878979d8c@fastmail.fm> <20240604221654.GA17503@localhost.localdomain> In-Reply-To: <20240604221654.GA17503@localhost.localdomain> From: Amir Goldstein Date: Wed, 5 Jun 2024 08:49:48 +0300 Message-ID: Subject: Re: [HELP] FUSE writeback performance bottleneck To: Josef Bacik Cc: Bernd Schubert , Miklos Szeredi , Jingbo Xu , "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" , lege.wang@jaguarmicro.com, "Matthew Wilcox (Oracle)" , "linux-mm@kvack.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: C0900180009 X-Stat-Signature: ks8rmte7fxshfa7wybnutd4d78xaoo8q X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717566601-4662 X-HE-Meta: U2FsdGVkX18dNy3Ao0bI2VNvlr+nBw16jAQvnxM5pzOc0khb2ZBU/i/WHgsoUIQFh788uDnHotphHdGp9e7woM8I8nK3ZCgSD3w4mLN0+p5u1KTlu2UA5iBkkP81sus+Kz4FcPaFK4ayZW2NwMd4YO8imR82ibXUji+kwrdPDj8aPDAdMc8FowCmTXURkZzvmr05IezKbKKE2KsPbMHNZQ4TsDTtm4MMMaz5OfgZ5DaqGsRwRakhO+cZ/JiaYvZSuWHZFfBb36+cRADBH7V2tKAvoTmc75p049/FBDpFvBosmEHEZUnKGo4lDlACRYOFlp2nYAjZikP32fI1QnCawcs6xmsJDAo4WHtXgfEgG3Qo2TMdZa5loygpbE+EgXmlgQCKW04RNwMlmqOk/Ju9TX8VQiexnyDPDX0/AyhhHr+CMMEugePWNDcVwxp4dKo0u/eJhWDpGTF3R82JmTS9vwkEwBtZtbO3dsjqGngC7tQLwcEpJMIWXXfQ3XLrJPJ4Wwuk/PVf3Rm37mLCpheBP9F5I7skguk24glB3JIuVyjSHbtKn05fE0ZKWjstzUbF5cRu2jerjqrbvNoxQV77KUDJEe1djlLul3bd+CB2weEUM8i1sjxFNWdP+zakrjr4RZpcxgVrxhGn+5JdkZtokqgAWdqS5fdXro4fwjrfBG4qmbJWzxv3+SaY5Y6kvLDlC/lDmAD1tW/jHzYSa5VMWRQrhnp+Yv/ErCAurWa8Ck61RjL+88Z1S/LvvFSv75Ah6j1AaGCOwbeGAbW/fPp7iohtHjOkz1g+qljgSfd4Wbhg1MXVng2Z4/SGb5BmEFFkpIfRpjE0Xd1PvHovgMOFg2OnVbswPesHOVWyGm1ExaZey4o/Mt8Qlo775NIkJ6xRdbAJ1EPMiPtmAl8fOFh0dKrbhqMUKy7C0JvdS8JtqXhSbTrSGMlXreYoYS0GxsZhaa9hx1yNmMR/tXYM10Y CJoZWxGK s38Yr5MDt1RRA4M8cwkrKSf+abks4PNkDRXZoTalsRmxrF6YUP0knw8yI5wqAlLkg7S52IPK1MK7Z4ZD31HBC1hArGA41mc7yv8SRXvCBtg/qfYz1Ritkl0ARGfkqT/Z6/f/99z9gW+I1cG0czKufaS3jb7LxRzsQOdSm3oXZKXZJWHm0fatsBfw6htVnqaWA1v3EpgsWP50ZArzxhJPXjQccyUyqVTE1u7koYljVfqn04UX9L9HA7pDLMRv+5qXLwttjnRwDnoLbSa4sN09jDdggZIax3hgbkVH2kX62+HNlnv5u3OhPTw9GyvoLmUKchtP1ZfjKoWnfwK8aJuUY3K3hF+tyFCgT/YSwiAuSNZyZjKR1HGrML0ArVD+rR7VZE9oquPhhhyKL8aT2DPcPXrlFdrEKs9L/EtQ0RQ0t06VGBEX+FJ0i5cJ2qIvS7UWFBCMCMRAvq03pBps= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jun 5, 2024 at 1:17=E2=80=AFAM Josef Bacik w= rote: > > On Tue, Jun 04, 2024 at 11:39:17PM +0200, Bernd Schubert wrote: > > > > > > On 6/4/24 18:53, Josef Bacik wrote: > > > On Tue, Jun 04, 2024 at 04:13:25PM +0200, Bernd Schubert wrote: > > >> > > >> > > >> On 6/4/24 12:02, Miklos Szeredi wrote: > > >>> On Tue, 4 Jun 2024 at 11:32, Bernd Schubert wrote: > > >>> > > >>>> Back to the background for the copy, so it copies pages to avoid > > >>>> blocking on memory reclaim. With that allocation it in fact increa= ses > > >>>> memory pressure even more. Isn't the right solution to mark those = pages > > >>>> as not reclaimable and to avoid blocking on it? Which is what the = tmp > > >>>> pages do, just not in beautiful way. > > >>> > > >>> Copying to the tmp page is the same as marking the pages as > > >>> non-reclaimable and non-syncable. > > >>> > > >>> Conceptually it would be nice to only copy when there's something > > >>> actually waiting for writeback on the page. > > >>> > > >>> Note: normally the WRITE request would be copied to userspace along > > >>> with the contents of the pages very soon after starting writeback. > > >>> After this the contents of the page no longer matter, and we can ju= st > > >>> clear writeback without doing the copy. > > >>> > > >>> But if the request gets stuck in the input queue before being copie= d > > >>> to userspace, then deadlock can still happen if the server blocks o= n > > >>> direct reclaim and won't continue with processing the queue. And > > >>> sync(2) will also block in that case.> > > >>> So we'd somehow need to handle stuck WRITE requests. I don't see = an > > >>> easy way to do this "on demand", when something actually starts > > >>> waiting on PG_writeback. Alternatively the page copy could be done > > >>> after a timeout, which is ugly, but much easier to implement. > > >> > > >> I think the timeout method would only work if we have already alloca= ted > > >> the pages, under memory pressure page allocation might not work well= . > > >> But then this still seems to be a workaround, because we don't take = any > > >> less memory with these copied pages. > > >> I'm going to look into mm/ if there isn't a better solution. > > > > > > I've thought a bit about this, and I still don't have a good solution= , so I'm > > > going to throw out my random thoughts and see if it helps us get to a= good spot. > > > > > > 1. Generally we are moving away from GFP_NOFS/GFP_NOIO to instead use > > > memalloc_*_save/memalloc_*_restore, so instead the process is mark= ed being in > > > these contexts. We could do something similar for FUSE, tho this = gets hairy > > > with things that async off request handling to other threads (whic= h is all of > > > the FUSE file systems we have internally). We'd need to have some= way to > > > apply this to an entire process group, but this could be a workabl= e solution. > > > > > > > I'm not sure how either of of both (GFP_ and memalloc_) would work for > > userspace allocations. > > Wouldn't we basically need to have a feature to disable memory > > allocations for fuse userspace tasks? Hmm, maybe through mem_cgroup. > > Although even then, the file system might depend on other kernel > > resources (backend file system or block device or even network) that > > might do allocations on their own without the knowledge of the fuse ser= ver. > > > > Basically that only in the case that we're handling a request from memory > pressure we would invoke this, and then any allocation would automaticall= y have > gfp_nofs protection because it's flagged at the task level. > > Again there's a lot of problems with this, like how do we set it for the = task, > how does it work for threads etc. > > > > 2. Per-request timeouts. This is something we're planning on tacklin= g for other > > > reasons, but it could fit nicely here to say "if this fuse fs has = a > > > per-request timeout, skip the copy". That way we at least know we= 're upper > > > bound on how long we would be "deadlocked". I don't love this app= roach > > > because it's still a deadlock until the timeout elapsed, but it's = an idea. > > > > Hmm, how do we know "this fuse fs has a per-request timeout"? I don't > > think we could trust initialization flags set by userspace. > > > > It would be controlled by the kernel. So at init time the fuse file syst= em says > "my command timeout is 30 minutes." Then the kernel enforces this by hav= ing a > per-request timeout, and once that 30 minutes elapses we cancel the reque= st and > EIO it. User space doesn't do anything beyond telling the kernel what it= 's > timeout is, so this would be safe. > Maybe that would be better to configure by mounter, similar to nfs -otimeo and maybe consider opt-in to returning ETIMEDOUT in this case. At least nfsd will pass that error to nfs client and nfs client will retry. Different applications (or network protocols) handle timeouts differently, so the timeout and error seems like a decision for the admin/mounter not for the fuse server, although there may be a fuse fs that would want to set the default timeout, as if to request the kernel to be its watchdog (i.e. do not expect me to take more than 30 min to handle any request). Thanks, Amir.