From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BF34C021B1 for ; Thu, 20 Feb 2025 20:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71B842802C6; Thu, 20 Feb 2025 15:46:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A43B2802A7; Thu, 20 Feb 2025 15:46:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51E832802C6; Thu, 20 Feb 2025 15:46:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2CEC12802A7 for ; Thu, 20 Feb 2025 15:46:48 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E5554142EC6 for ; Thu, 20 Feb 2025 20:46:47 +0000 (UTC) X-FDA: 83141506854.27.76E69DF Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf09.hostedemail.com (Postfix) with ESMTP id F1C51140004 for ; Thu, 20 Feb 2025 20:46:45 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=davidfrank-ch.20230601.gappssmtp.com header.s=20230601 header.b=jVmktg8P; spf=none (imf09.hostedemail.com: domain of david@davidfrank.ch has no SPF policy when checking 209.85.219.170) smtp.mailfrom=david@davidfrank.ch; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740084406; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ki4BqDr2jfoSCfBXflnZeD/HvCxdHbHAAIJAh4w+1oM=; b=32mpRmd/WVrs5VtBf6z4CQCvv55paHXRPDeYpEAwJurEJmv0wfLqR6DVsrCHR2sNQZ7vUL T+EuM4mXZ4Fv3WVN5up3JcL6cmRsqEURDiMVLt8pR/OJ5Xn8qGoe78ANLCMAm4OZGECxSs u5JSSXxEDAay+xOgvM8cf04/kzfxzAQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=davidfrank-ch.20230601.gappssmtp.com header.s=20230601 header.b=jVmktg8P; spf=none (imf09.hostedemail.com: domain of david@davidfrank.ch has no SPF policy when checking 209.85.219.170) smtp.mailfrom=david@davidfrank.ch; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740084406; a=rsa-sha256; cv=none; b=NT66DrHWScGRap8QehxEdO9LEvwCv4P2QQJeoOkMhfLdq5Yu53Pa45vpaVhxjpBZ9b6rWy CE93n6XPhgO48Y2JS8LhHRbZFVQIRAlVzoEGU7hEwC2D7tOzLYwikxs5Aw39+wSOp0OGhp 5qEh1gDiorcJuwuwznGVQoceXBU1Vs8= Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-e479e529ebcso1173531276.3 for ; Thu, 20 Feb 2025 12:46:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidfrank-ch.20230601.gappssmtp.com; s=20230601; t=1740084405; x=1740689205; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ki4BqDr2jfoSCfBXflnZeD/HvCxdHbHAAIJAh4w+1oM=; b=jVmktg8P3rVeuiAdk+lji1Q0uFnUoz/z2mQSxlNKznpP8W2/yBNMngmw62YpzUi9HW 8WjVUuw88gwzFC8ovsv/k9BUA1pd79g803CMp4t4IBpN81Yxfzhyh8fFMez//V/vOhrR ww1aIg+T0wwCO/Suwt3PTj8XiBqp437QteyYk4eWSYZlfGxlZdqUH1GgphJtyjW21pY2 sbGGXfFYA22euWNwKJvOidiP6FZ4HmNEZxvPqALWzAuNTNWLccHKfGSHoE2aiob8urI+ VTqH9MWHGeGFXroKLRskrpIvKcBztnAU7pllMXjDQSLxZ/po91+WxMvRBXNFyyvr5TNf ek2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740084405; x=1740689205; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ki4BqDr2jfoSCfBXflnZeD/HvCxdHbHAAIJAh4w+1oM=; b=n6fhIiiG3VSpm4cAXlwuZuemMGNTTAHnCuev1v7ca8uAQZxQLpoiMy3JqiPVQ556aY njAbuH0qTI4x+9xPgUbgP84zFLT3L2T4MYJNr+qD4egaZEkMOE6h26FKw4F5SfXZQla0 QicmS8iptWxS7AOaNj1ZHKIz+YEBHiONt2wNC7phwxo8TxKGCq4PmcawgFunm5ke8VM6 vbbyzIhOKhLStTPK2BNYjvvGMXvNZ/0nwfGp2ADUkDHMb4glRnshaieysiMShYoilqwo sdYmWwU512uC2k//39yXuf13ZO8Ziih4zygVREwfKNsXm3QFTomfJBisXdYFCtr/G11i NlTQ== X-Gm-Message-State: AOJu0Yy8FRpK/WhkH3n6oFHDLQGx1prmfROKI9f60XFL4OYYcSVUWT0k Er1YTsb/5XCCsmR8jA7/jVwhqWuktv2H3+cXm6Yab6mrOeoB9llK/WOZRB2WE610k+r+7uPuPak yWhoPE7J7AUeIhkjnGE9Y1ySRZlxq34wTPmQ+VA== X-Gm-Gg: ASbGnct4QWnNkfrv+FXl88DL1nYhtHBZXDhLJVZSJ1jDTXdHkGeht/uuNd+y3GlP7UY PW/Zp0Do/zBwOuUtNMJd6eYvFesL2VTcQrkOGxcCYU8WHZibj1OHbVGa9iZGpi/mJkYioQITJJK yUMkfLA8GgkuPu3lnTedo96LTbVg== X-Google-Smtp-Source: AGHT+IECScJWDOKRG/qBJ8UnxAsKCIAGMu+3ZEUFJrWLypTdYZOvw75h5GMNXvgZAYJhA00gHT0/bZsGQCVHkXM8isA= X-Received: by 2002:a05:6902:1ac3:b0:e5d:cdc6:7ad2 with SMTP id 3f1490d57ef6-e5e8b067de6mr99742276.46.1740084405031; Thu, 20 Feb 2025 12:46:45 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: David Frank Date: Thu, 20 Feb 2025 21:46:08 +0100 X-Gm-Features: AWEUYZlkfgrMYlKjcUpCPWBRs8CxAmJiDWQaTfFwzPjscJTOpGasU7i9OgJWXII Message-ID: Subject: Re: Efficient mapping of sparse file holes to zero-pages To: Matthew Wilcox Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: F1C51140004 X-Stat-Signature: f5pnozga3zq81erd1ctd7pyjy8xmxk3h X-HE-Tag: 1740084405-235955 X-HE-Meta: U2FsdGVkX1+v3+g3JVlwTgjRVb6rYSTeijMrK/cjar27UZV6d2Q/sRMKUnU4g310ASmcqJQnMyDb8KkXjs2g1jP/jb4xlvIPbCfaja9KQuKEuHkzV0VqCTwlqkCENtm/T3Zv5veKymvQ2OUEBlPeHyPpSJtQd6iw0uOuYL+1r0uMITbS9EzeHTtAy/UeeJQmsjivRR+BKbjnC5VzKvNT7ot4qUVBpfxjLjc6WVboxLIiIjwPHyM/t2p68SON5WwZCOMz+GsiynKm+gm9EAE35wGdigig7eFp5c5vkWABvE0LRHx/tHqGgFuvRQG+av8ZHE47/i0D+u3yZqM3pencytZ+CmvtaORjDpeZU8DLgcoiJlU3mZSO4poSKgdRNSSkPTy3CxD8PZajy10UPVNXZbpVzUtGsgWQ47K9z0B7N7ladehtHJlg080dxl2KKU3kHA9Xq3dJTEobWzzG2kybx8wIGJJKu4rh7o6vioidob9Uwq4Zr6dLuiBDWRCxWRc+yA5DYrVLw4d14XU/cxl2Qdc3AdDrgvEMEVFObMwH+Pb8vSAj//RuGyYp8WpncQTs3I7ugt2XuQ7EfIRs3jnSqZUJlZSwVKjTKxUEPs9nUFFQkpPbPQDE7GDk8BB/gGGxIK5be1z/SlgP/rkoSGHrTNvPq5Bkz20rDnJ8TK28QWesM94yVfoOP5Zz3AF2ABEsMUw4xgFkp2PIqpNYjla3V9U4rFYg9cEKaS4OU8LdhPyuYgRdyQhj9OCKPQRJHyxvp6li3OzW/urR3y86DKFs8bwm6KA4lTsEHMbMAHuXw04TKvQObRAOYVqclTrA4GJNt8E1K02RzQpO4gl0dMPWiXTbST0njKDBrlauDf/4zajAn0awrNQFN4Jkf7jiWgvLhVVxZ9VGhGNXHsC3tkDQPcmLBiosyCbBGgTkek4fXuRiT4FHPviMxRoZjf546X82U7eLrucWTjPHzECAzOY j9DGCzN4 9FiiNPNJymUBLFa5zj3mEXeBDLdwJx2686h/zH7UsamUqolLnk8GFuhEzwt9o/LPSVz+BfM4l4VXUlLEH1QUR5zzf2tNXfLXYvl6FlmAVzrdboSANVQw0+XyNRIsfPGR1jNVyQj9q8y9GnL8usMZtjily5kDcZXF2H0nO6HcAj6yNTgxtKujuiiTo66jX0Nzr2DfKWQXjeFOB0NNkhFAvWzQF522Elqs28DoJhnYiQlFsOdp61C8HEvj4Xm/4hbpX2PWif4Daq2R7RN6K76QVzGPGj5k8/vEzpstQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.011236, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Thank you, Matthew, for your reply. What do you think about the complexity of this task? I'd be interested in taking a look but I don't have kernel development experience so I would need guidance. On Thu, 20 Feb 2025 at 14:47, Matthew Wilcox wrote: > > On Thu, Feb 20, 2025 at 01:48:18PM +0100, David Frank wrote: > > I'd like to efficiently mmap a large sparse file (ext4), 95% of which > > is holes. I was unsatisfied with the performance and after profiling, > > I found that most of the time is spent in filemap_add_folio and > > filemap_alloc_folio - much more than in my algorithm: > > > > - 97.87% filemap_fault > > - 97.57% do_sync_mmap_readahead > > - page_cache_ra_order > > - 97.28% page_cache_ra_unbounded > > - 40.80% filemap_add_folio > > + 21.93% __filemap_add_folio > > + 8.88% folio_add_lru > > + 7.56% workingset_refault > > + 28.73% filemap_alloc_folio > > + 22.34% read_pages > > + 3.29% xa_load > > Yes, this is expected. > > The fundamental problem is that we don't have the sparseness information > at the right point. So the read request (or pagefault) comes in, the > VFS allocates a page, puts it in the pagecache, then asks the filesystem > to fill it. The filesystem knows, so could theoretically tell the VFS > "Oh, this is a hole", but by this point the "damage" is done -- the page > has been allocated and added to the page cache. > > Of course, this is a soluble problem. The VFS could ask the filesystem > for its sparseness information (as you do in userspace), but unlike your > particular usecase, the kernel must handle attackers who are trying to > make it do the wrong thing as well as ill-timed writes. So the VFS has > to ensure it does not use stale data from the filesystem. > > This is a problem I'm somewhat interested in solving, but I'm a bit > busy with folios right now. And once that project is done, improving > the page cache for reflinked files is next on my list, so I'm not likely > to get to this problem for a few years. >