From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 293F4C47422 for ; Mon, 29 Jan 2024 05:15:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BE0B6B0072; Mon, 29 Jan 2024 00:15:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76E316B0075; Mon, 29 Jan 2024 00:15:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 635A06B0078; Mon, 29 Jan 2024 00:15:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 50CCD6B0072 for ; Mon, 29 Jan 2024 00:15:27 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 21E061C0D08 for ; Mon, 29 Jan 2024 05:15:27 +0000 (UTC) X-FDA: 81731185494.30.138C0FF Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf16.hostedemail.com (Postfix) with ESMTP id 2EB21180006 for ; Mon, 29 Jan 2024 05:15:24 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=uWGgssV9; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf16.hostedemail.com: domain of david@fromorbit.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706505325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MzCJhbr94fRn8OFzgvOn2yQbiBfC+VkQTRw5GLu7Xvk=; b=HlTsggp8YDuaZVn/uEk+9+ufi4bJsWg0QlRZ7fkFGjn31mtqSo8Wo03Q7HUg0h7Ck7JB++ 7L1TWruohh1YqJTBFChL0Phun/mxXUiNpn1vcNrG2AfZKGu9GTIN3U8HlZRT0kn6Ea52Cy +fFH4GELYLlEV2Drl1YiadMLrbZEkSw= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=uWGgssV9; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf16.hostedemail.com: domain of david@fromorbit.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706505325; a=rsa-sha256; cv=none; b=T9tJXvq/PnFTF0BmranP4IPsS0TfmQNOM7XUUZIWSyRejcJFXV5CpzqTmg1GOmBr+hGCv5 gu95lheacPsy1gZFqZLQI81s+sBZqC2pkNRGWAyH+YooynhGAxgdWOBZravxt7Ka7ZkYjx iKhgzkWWsZEQWp6JpfxcRpsIBOPwI+Q= Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6ddc1fad6ddso2184940b3a.0 for ; Sun, 28 Jan 2024 21:15:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1706505324; x=1707110124; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=MzCJhbr94fRn8OFzgvOn2yQbiBfC+VkQTRw5GLu7Xvk=; b=uWGgssV9E3DCJhefM0CcLI8+EfhGs00fwCuOq9SCng3eLvWW4tsOcfP/gck3aDjrBG bIP/Cb1HTsgRyk2hMAvn3BaGO4Wx+dWZcoMU/e4yEb7EhwirtYHYkczZOmOvrgkbskSK HFYe5Z9gFNmUJOijFE1GaAug+x0waLTyA10S9rzVEjp3gsYEtGMv+22aAD/EZ1PYss3T J3wS7w99jc+0Z7a0zvgpRwqxJ7/SGpyR4BQSs1TLyzpylGnXci/aVKluf0M35+NOdgHu v6Wlmish0G10FO5Uqd0C5fU7zfqTRBlF75+g01wIlzoflPV6kG/ts1i9VS7tLao6v0Q/ 5bAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706505324; x=1707110124; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MzCJhbr94fRn8OFzgvOn2yQbiBfC+VkQTRw5GLu7Xvk=; b=vtYpwO9ZVMOvye4CwwVaxk+ZSQeMQmTVnIR0Eq79sx6xZVt/IOkTJBf2yFRebVyV5l RMpXvBqzjr0eXNFyn3M9yekHtSYlwjB9NjDkYzgQW3N2VMnisLglTmlRMmUJr0D90PWT vjUmma2PBWVRrE7QHaLo62HkpIw6hRKf3/ZO2PLMK2C2AvmrFXhs3yTkFkl3Nc7JVUg3 nrM1b0lcBGQE0uR1Q4WkbZ/wH8OWRIwNJdp5+FnVsj9NC6+xkAV8xQyMPX6ZhyU1tEcV M/3i6Y1yWG/bX8Z2jnyXMpZmPXRupfuQV4oxMrfjcqFIHvKzCwFaOQmJ+CyCdGDgU0HC H8lw== X-Gm-Message-State: AOJu0YxyBs483x1sElczofcnJwUumKywCtvxPXkSYVopaunPitKZ5m/J KERBZzSdRa/fR2E0RjyRR7lu9L7fdIstzSeWT4u2lIfAu+JeI7uWReWBQwaeWgk= X-Google-Smtp-Source: AGHT+IGANgJ4Z2U1cic1qfkdu+LrVftCh5DoXKhkXGOBg6TiBffjvJTExqp//91HJcFTxKSobu8QBQ== X-Received: by 2002:a05:6a00:d74:b0:6dd:8a25:e167 with SMTP id n52-20020a056a000d7400b006dd8a25e167mr3050277pfv.34.1706505323985; Sun, 28 Jan 2024 21:15:23 -0800 (PST) Received: from dread.disaster.area (pa49-181-38-249.pa.nsw.optusnet.com.au. [49.181.38.249]) by smtp.gmail.com with ESMTPSA id h10-20020a056a00218a00b006dbd1b13d29sm4911312pfi.208.2024.01.28.21.15.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 21:15:23 -0800 (PST) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1rUJzI-00GjMK-2X; Mon, 29 Jan 2024 16:15:16 +1100 Date: Mon, 29 Jan 2024 16:15:16 +1100 From: Dave Chinner To: Ming Lei Cc: Mike Snitzer , Matthew Wilcox , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Don Dutile , Raghavendra K T , Alexander Viro , Christian Brauner , linux-block@vger.kernel.org Subject: Re: [RFC PATCH] mm/readahead: readahead aggressively if read drops in willneed range Message-ID: References: <20240128142522.1524741-1-ming.lei@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2EB21180006 X-Stat-Signature: j4d6ixroszgwoyzzfari4cne8zfx3ncr X-HE-Tag: 1706505324-715531 X-HE-Meta: U2FsdGVkX18SuhRlMHdL41WYZeLT8EwFdfAtfNeg9ZsnSeC6sZKn1rsok5MXnYDEKFyJCbj+KpnBamVgce/SRd50gEXhk56N/PqfBe0L7Mr3WzIrmWEyM4hvWEE9uXPUGxCTDI0JLpUyaND5V8sY6RGsEM4xGIuaaf+bIIn4u01uisF7tgG4WLYEBzbeG7JI2B7R3NnicahJaxJw9j+B7DfSZB9EH3H5HE3KoD9S7myEMiwfIKOAQLGUU8lngdL9BIDdp63WsjIuaTJpyZIngKRUExonWZ9yA+e2LkeZkFuw1hs4Xf3DhI7kFL4FD7OkNh6LAtqbYfh09FRAjYV4gsn55RQXTjQSMM2uCf0WBZDPlw3w0AQ46ay/JPui6nVLIBVv1VT9ZNkAqnepDUokS2ouLxu3Bb9owk/x6S3HEDTXr+kPgzKM8pshQJhgk4g8impEN6W20K8m1K5vb6kBVKWf88VVnHp3yaWO9b6vqHYWO5CN3mBve0D1e3Al41ELcBqwdFIvm22tYLxeAMh3Gt+PruH6/5AXUzMBr6DJTdmeruV64QS40GdpFofJawUw1N9qPZRKPqyS59xLy0Gf4ucwYceF89IpTul3jde0+6AyqJNxL0tzNAPPt0iFawYc0SdT+Tgab1G2qTSedXU1iznWHroKLOyDNybu+I9uciQZkrz8JM/k5P01Z/5EfU/EePj/U+F2DxZdlvxyJ3spli9SCxmVoGZgkZQ8I6dsYl87AF2sSjEeso+oyqyQAMzHr4i0Hjyx53hpxc+neJ/WPL+Uw4XWMUzQxYjMC+NH/29988NuH9xU19dmQ0GFvKshMGWY9fZIk+SXq6oPYYFZwQ37Uf+NYtsoYKTRkflTrYyMmxOtPN/2ExjOGPEs4GXIj2T8lIsQHswMctyxG3PfTd+R60NNoXPtbMAGAnZtfdq0TARJ2SZigXvdr7n9kjNqHa5KX5EO34n0HLE/EkN e3vJSLGO ladWps4PUKU6nbJqXnuLQVCKsP9X5tXUmDAcjdNW41vygBLI7V6IukVk8hfIl4YLWak4WM4s/4TgSNfxv2HwkLx2CEER5cOIcEL+Gr2nqR+4lPYHz8vPQFznbT1Po7LySwLR5rGseTx8DQLIWrQMwzrZThICxSXtLNm8Zz1qONIQwGgULfy7qN0DGYB2Kk65yGZdhA1Egg2WBYbpeWX/wFt6TnMEK+8dGzpZNkVEj0qBtUClHXNh4Pep2YtijfjKgexAN9RHeoy4umnMC4Erf+3aqB+MvW4BJxEr+w/zYIErqmKCmF/que9RiHNYzzoxmX5NuXwWZN/T9hB+1BKPyhzQ+y2zoFCrvBUMbkdbzEQfH82pXiWsdQnRa0rao0AYDSAT8wFe03BAjApQ65MlDYKK/zwe/OST8m0QjP75PMG+YS0vJcZdEk9ZTMPp27IpA2hFp+1pMV46/XR6IiQKYKPHkWaIDzqdyKpSKwhB0irWFElQ7zTzFXC7kUnlNuqBYgefj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jan 29, 2024 at 11:57:45AM +0800, Ming Lei wrote: > On Mon, Jan 29, 2024 at 12:47:41PM +1100, Dave Chinner wrote: > > On Sun, Jan 28, 2024 at 07:39:49PM -0500, Mike Snitzer wrote: > > > On Sun, Jan 28, 2024 at 7:22 PM Matthew Wilcox wrote: > > > > > > > > On Sun, Jan 28, 2024 at 06:12:29PM -0500, Mike Snitzer wrote: > > > > > On Sun, Jan 28 2024 at 5:02P -0500, > > > > > Matthew Wilcox wrote: > > > > Understood. But ... the application is asking for as much readahead as > > > > possible, and the sysadmin has said "Don't readahead more than 64kB at > > > > a time". So why will we not get a bug report in 1-15 years time saying > > > > "I put a limit on readahead and the kernel is ignoring it"? I think > > > > typically we allow the sysadmin to override application requests, > > > > don't we? > > > > > > The application isn't knowingly asking for readahead. It is asking to > > > mmap the file (and reporter wants it done as quickly as possible.. > > > like occurred before). > > > > ... which we do within the constraints of the given configuration. > > > > > This fix is comparable to Jens' commit 9491ae4aade6 ("mm: don't cap > > > request size based on read-ahead setting") -- same logic, just applied > > > to callchain that ends up using madvise(MADV_WILLNEED). > > > > Not really. There is a difference between performing a synchronous > > read IO here that we must complete, compared to optimistic > > asynchronous read-ahead which we can fail or toss away without the > > user ever seeing the data the IO returned. > > Yeah, the big readahead in this patch happens when user starts to read > over mmaped buffer instead of madvise(). Yes, that's how it is intended to work :/ > > We want required IO to be done in as few, larger IOs as possible, > > and not be limited by constraints placed on background optimistic > > IOs. > > > > madvise(WILLNEED) is optimistic IO - there is no requirement that it > > complete the data reads successfully. If the data is actually > > required, we'll guarantee completion when the user accesses it, not > > when madvise() is called. IOWs, madvise is async readahead, and so > > really should be constrained by readahead bounds and not user IO > > bounds. > > > > We could change this behaviour for madvise of large ranges that we > > force into the page cache by ignoring device readahead bounds, but > > I'm not sure we want to do this in general. > > > > Perhaps fadvise/madvise(willneed) can fiddle the file f_ra.ra_pages > > value in this situation to override the device limit for large > > ranges (for some definition of large - say 10x bdi->ra_pages) and > > restore it once the readahead operation is done. This would make it > > behave less like readahead and more like a user read from an IO > > perspective... > > ->ra_pages is just one hint, which is 128KB at default, and either > device or userspace can override it. > > fadvise/madvise(willneed) already readahead bytes from bdi->io_pages which > is the max device sector size(often 10X of ->ra_pages), please see > force_page_cache_ra(). Yes, but if we also change vma->file->f_ra->ra_pages during the WILLNEED operation (as we do for FADV_SEQUENTIAL) then we get a larger readahead window for the demand-paged access portion of the WILLNEED access... > > Follows the current report: > > 1) usersapce call madvise(willneed, 1G) > > 2) only the 1st part(size is from bdi->io_pages, suppose it is 2MB) is > readahead in madvise(willneed, 1G) since commit 6d2be915e589 > > 3) the other parts(2M ~ 1G) is readahead by unit of bdi->ra_pages which is > set as 64KB by userspace when userspace reads the mmaped buffer, then > the whole application becomes slower. It gets limited by file->f_ra->ra_pages being initialised to bdi->ra_pages and then never changed as the advice for access methods to the file are changed. But the problem here is *not the readahead code*. The problem is that the user has configured the device readahead window to be far smaller than is optimal for the storage. Hence readahead is slow. The fix for that is to either increase the device readahead windows, or to change the specific readahead window for the file that has sequential access patterns. Indeed, we already have that - FADV_SEQUENTIAL will set file->f_ra.ra_pages to 2 * bdi->ra_pages so that readahead uses larger IOs for that access. That's what should happen here - MADV_WILLNEED does not imply a specific access pattern so the application should be running MADV_SEQUENTIAL (triggers aggressive readahead) then MADV_WILLNEED to start the readahead, and then the rest of the on-demand readahead will get the higher readahead limits. > This patch changes 3) to use bdi->io_pages as readahead unit. I think it really should be changing MADV/FADV_SEQUENTIAL to set file->f_ra.ra_pages to bdi->io_pages, not bdi->ra_pages * 2, and the mem.load() implementation in the application converted to use MADV_SEQUENTIAL to properly indicate it's access pattern to the readahead algorithm. Cheers, Dave. -- Dave Chinner david@fromorbit.com