From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAA6AC432BE for ; Thu, 5 Aug 2021 18:50:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5C37A60F58 for ; Thu, 5 Aug 2021 18:50:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5C37A60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CB3836B006C; Thu, 5 Aug 2021 14:50:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C3C256B0071; Thu, 5 Aug 2021 14:50:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADCD66B0072; Thu, 5 Aug 2021 14:50:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 8E94A6B006C for ; Thu, 5 Aug 2021 14:50:35 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2F4D1182F4F79 for ; Thu, 5 Aug 2021 18:50:35 +0000 (UTC) X-FDA: 78441918030.36.0B1337A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id B396A9002585 for ; Thu, 5 Aug 2021 18:50:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1628189434; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IiYjOVol2JvYBzSm9nmbOsFi1gSlVR+sNt5v+Iro+eM=; b=ZHbbAU61oS1H+2oKLhh8moglBhhHg9ftrLE4x06oC+5N59lgrVeSpWNOFolj5VDfdjOBjg JHdkv0sLAG+RU8NV0HI3je+fFAMDQabsFcLI6ODWV8Z0zBC50uzg2lkHxV5LkcU81uU/9x ldeg5zLEeBhlU26u5Fl6xXD+FK36/ZQ= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-374-V4EJdhZ6Mxi7WqY5xJnNjA-1; Thu, 05 Aug 2021 14:50:33 -0400 X-MC-Unique: V4EJdhZ6Mxi7WqY5xJnNjA-1 Received: by mail-qk1-f197.google.com with SMTP id b9-20020a05620a1269b02903b8bd5c7d95so4786828qkl.12 for ; Thu, 05 Aug 2021 11:50:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=IiYjOVol2JvYBzSm9nmbOsFi1gSlVR+sNt5v+Iro+eM=; b=JQKAcaBbL80LXODzRKCoUrdEsiMzOfyFa0KrJhXWUGoPphYLckrJ0/Q5N8KrGwkYgq 21FrwJ95zXO5y0HOl/FPKhwILyT98rK5Ae8uvw3+yvdU1hK/X6kcgQBU4zhYvNEWFA4d Mn1ypM0lJbXAbGtPRgI7ERLPGZNT0GPtSURtXkwvBne6VxRH5Ut6K+It8sGw3+LTBKD9 SDDBithcmMObFQxsanC8UWkHuhJrReQ/JYXgLEilA/En2BOg6Qk7MYg9JbJRHl5kjmDn 1ZXi8NSh40yZeBSz1Ad2cwGejuAy63tOMDnnO8Y72roANAnuzP983i+9CmPxHZnypEXJ +utQ== X-Gm-Message-State: AOAM532rkhvE8T8AJsKRz+LQcUeCRmtXdGI/9Qst1eVHi4k6LN85uDD7 DvVkalwA/jKgwC3+JJrOaZuMGjk6pXfJfDqp7lQngu/9obTMYbk2SGR3DI+7tJNs0R4hmVvPrNr bk13g72UvZoQ= X-Received: by 2002:a05:620a:b44:: with SMTP id x4mr4723784qkg.11.1628189432782; Thu, 05 Aug 2021 11:50:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwCNAmptp/YxuNLEFQ2GKThqZsDaw/EkAl6/uzN/fIMPdEnuhdMjnc4Ro9mfYivPJI45LPC9Q== X-Received: by 2002:a05:620a:b44:: with SMTP id x4mr4723755qkg.11.1628189432545; Thu, 05 Aug 2021 11:50:32 -0700 (PDT) Received: from [192.168.1.3] (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154]) by smtp.gmail.com with ESMTPSA id c190sm3509913qkg.46.2021.08.05.11.50.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 11:50:32 -0700 (PDT) Message-ID: <90a2a17aeae0447793496426d21794a3b0f7c197.camel@redhat.com> Subject: Re: Canvassing for network filesystem write size vs page size From: Jeff Layton To: David Howells , Anna Schumaker , Trond Myklebust , Steve French , Dominique Martinet , Mike Marshall , Miklos Szeredi Cc: "Matthew Wilcox (Oracle)" , Shyam Prasad N , Linus Torvalds , linux-cachefs@redhat.com, linux-afs@lists.infradead.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs-developer@lists.sourceforge.net, devel@lists.orangefs.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Date: Thu, 05 Aug 2021 14:50:30 -0400 In-Reply-To: <1219713.1628181333@warthog.procyon.org.uk> References: <1017390.1628158757@warthog.procyon.org.uk> <1170464.1628168823@warthog.procyon.org.uk> <1186271.1628174281@warthog.procyon.org.uk> <1219713.1628181333@warthog.procyon.org.uk> User-Agent: Evolution 3.40.3 (3.40.3-1.fc34) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="ISO-8859-15" Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B396A9002585 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZHbbAU61; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf29.hostedemail.com: domain of jlayton@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=jlayton@redhat.com X-Stat-Signature: bbqdn48kcx3niwg1qttj39o7jkads6xw X-HE-Tag: 1628189434-588044 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 2021-08-05 at 17:35 +0100, David Howells wrote: > With Willy's upcoming folio changes, from a filesystem point of view, we're > going to be looking at folios instead of pages, where: > > - a folio is a contiguous collection of pages; > > - each page in the folio might be standard PAGE_SIZE page (4K or 64K, say) or > a huge pages (say 2M each); > > - a folio has one dirty flag and one writeback flag that applies to all > constituent pages; > > - a complete folio currently is limited to PMD_SIZE or order 8, but could > theoretically go up to about 2GiB before various integer fields have to be > modified (not to mention the memory allocator). > > Willy is arguing that network filesystems should, except in certain very > special situations (eg. O_SYNC), only write whole folios (limited to EOF). > > Some network filesystems, however, currently keep track of which byte ranges > are modified within a dirty page (AFS does; NFS seems to also) and only write > out the modified data. > > Also, there are limits to the maximum RPC payload sizes, so writing back large > pages may necessitate multiple writes, possibly to multiple servers. > > What I'm trying to do is collate each network filesystem's properties (I'm > including FUSE in that). > > So we have the following filesystems: > > Plan9 > - Doesn't track bytes > - Only writes single pages > > AFS > - Max RPC payload theoretically ~5.5 TiB (OpenAFS), ~16EiB (Auristor/kAFS) > - kAFS (Linux kernel) > - Tracks bytes, only writes back what changed > - Writes from up to 65535 contiguous pages. > - OpenAFS/Auristor (UNIX/Linux) > - Deal with cache-sized blocks (configurable, but something from 8K to 2M), > reads and writes in these blocks > - OpenAFS/Auristor (Windows) > - Track bytes, write back only what changed > > Ceph > - File divided into objects (typically 2MiB in size), which may be scattered > over multiple servers. The default is 4M in modern cephfs clusters, but the rest is correct. > - Max RPC size is therefore object size. > - Doesn't track bytes. > > CIFS/SMB > - Writes back just changed bytes immediately under some circumstances cifs.ko can also just do writes to specific byte ranges synchronously when it doesn't have the ability to use the cache (i.e. no oplock or lease). CephFS also does this when it doesn't have the necessary capabilities (aka caps) to use the pagecache. If we want to add infrastructure for netfs writeback, then it would be nice to consider similar infrastructure to handle those cases as well. > - Doesn't track bytes and writes back whole pages otherwise. > - SMB3 has a max RPC size of 16MiB, with a default of 4MiB > > FUSE > - Doesn't track bytes. > - Max 'RPC' size of 256 pages (I think). > > NFS > - Tracks modified bytes within a page. > - Max RPC size of 1MiB. > - Files may be constructed of objects scattered over different servers. > > OrangeFS > - Doesn't track bytes. > - Multipage writes possible. > > If you could help me fill in the gaps, that would be great. -- Jeff Layton