From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5F5BC7EE23 for ; Wed, 1 Mar 2023 06:01:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B4256B0071; Wed, 1 Mar 2023 01:01:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 464336B0072; Wed, 1 Mar 2023 01:01:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32C8F6B0073; Wed, 1 Mar 2023 01:01:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 23B9C6B0071 for ; Wed, 1 Mar 2023 01:01:04 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D61E51C6B2C for ; Wed, 1 Mar 2023 06:01:03 +0000 (UTC) X-FDA: 80519281206.11.754D561 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf24.hostedemail.com (Postfix) with ESMTP id 784B9180003 for ; Wed, 1 Mar 2023 06:01:00 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of hsiangkao@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=hsiangkao@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677650461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tDzZArxQz/7Dh9kBvNqDDdoi10UPAl33iVXjXNC1SO8=; b=GCagTuh3nVuoSHEq9yZlmCIFAQwJUiJW+gDej2oo0+qetT+1PFa7vP4L/zQI3LWhxXZ7jv 44o32ZHeKAEAdzOecmDZJzMEG02LOtzysQO4S8dViTWI0ujSBtPiHpSNqv7wmRBG75ziRQ d0aCZcnPX5MWuGjIyxmlSj59qZCy5zE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of hsiangkao@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=hsiangkao@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677650461; a=rsa-sha256; cv=none; b=UwOCyLFYixGjg7Q11k41qXk01jWTZVnjlMs/SjgoSzIHmofIe06Tj3EAa7hV3u1ZqNuesh QMFmQIbpquGx5bl35G1/YBbQ6PPbuqY9jnjEDMxxl6WUr59h8ZIvPerVHM4AdWdKrzdqGj vAcQ/LjJ8B8qW1kvt4nyw8p6gChG9ks= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VcoLMzW_1677650456; Received: from 30.97.48.239(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VcoLMzW_1677650456) by smtp.aliyun-inc.com; Wed, 01 Mar 2023 14:00:56 +0800 Message-ID: Date: Wed, 1 Mar 2023 14:00:55 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [LSF/MM/BPF TOPIC] Cloud storage optimizations From: Gao Xiang To: Matthew Wilcox Cc: Theodore Ts'o , lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org References: <49b6d3de-e5c7-73fc-fa43-5c068426619b@linux.alibaba.com> <01ff76e3-87fd-0105-c363-44eecff12b57@linux.alibaba.com> In-Reply-To: <01ff76e3-87fd-0105-c363-44eecff12b57@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: hfmrzm59ni668awysntaf6hb974ogokz X-Rspamd-Queue-Id: 784B9180003 X-HE-Tag: 1677650460-294436 X-HE-Meta: U2FsdGVkX19VMkVGVmCb5SkjlKll33MYpQ6ZIfyUIq8j6c97kfB3aKy9sHVuMPfyYXl6eMjcV2fpqTZIV90Y4Fx5M/HlDIBwfvXRhUpkIwP9pwiHj78EtbMC3UGlhHWwV7yuN6kBo7q7rlQsVHdwZu6RTIPmRLv8OojV8qiK7YFRFelU57gQropi85x1tQfhFLYjq4wwDcrwwfCMBMG/4lHWHjIApht0h+H4Q8ui4bKsxJuB53f3GhADeLLpvBwrfFwi0ZW1wtoIgt4e53edyTvbONgl6NffQmeqVbieC1ZKcPvyQOB9Q0PVyuEYxrTYOezCD0hFL18DTBX8FxFW9uzDAAUi1pU4cM+xcZzzlhxXwqRrmKRDorBSyAABleKuwq4ULgymFuQIHgzKOq4jv71Y4EVKErEsd5FGEVRNAz9SsiQDYvUGmFtG+IodRkd9YAdL/vL1cPvZvhnW71AtXBkDofC6+4xUa0okHa34G9Yxory/js3ujoATsTDTqGUUCu3QWjT4iYQfHSsEg7lNT8pMus8t60jwqQihPIPMzMJK/0zrNviBzcp+FX/KEo6xw2lBF6m3q2yu/TR1m68uOdqAMwWm0jNpzXR7zUCl5qxbnbAhTIhoOc4nZ0Aly8Ek3wfnM6d7HLhEb2or6U2eKtI6oZgwjKRCbfOyDn25HIvWfPL+Fq7812ij8wVad4zr/f2DeTwbu0Ndko7YrWg24JS9gTcIeHVre+7LGCs3arEG5P6+WuzbzR85REPK2e0MRqDLash/NVDqqTs7T0WmV/2AoTCfSNT0Z0tLqc7hDQKRDB4Z9ei0uEuVgIUAf+fGvNNsn3nXrc0ChstaDzS2djJyvTwLrNR7vCKUH+wNjJQ7jau+bZnmIpKD+lYtQox3Z34y4vjUdtY21GGp7WXBGS2829Z+NVbBaX5jkx7e0Hvb91ZjY9IaV4Am2JlyZgUEERAcn5eck+fsHN00d2X dfTmGeeh NIQDDucjnzxsx0NdaHWtnJZ+SB9em/3lGPUdqZ5V/BhjLRkjsvnX8sDJiFBTjmqc8ctNUoAlqDXnfzOUFufm2UpW9O6a/QaUbKqpAtclSPr159YgYtmHIzocvIBXz4aW7YuM8XqzPawinqXNPDdKoTu4WQRf/5iVWKNOP24DMdG+zRbmmZsjjtLIpvoufoFmvgYJBbHs3NV7BuKc35Jd6tu8i1h9b1W61CtT/E59YH24UcFd58SgvtbU08rCjgG04i6U/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2023/3/1 13:51, Gao Xiang wrote: > > > On 2023/3/1 13:42, Matthew Wilcox wrote: >> On Wed, Mar 01, 2023 at 01:09:34PM +0800, Gao Xiang wrote: >>> On 2023/3/1 13:01, Matthew Wilcox wrote: >>>> On Wed, Mar 01, 2023 at 12:49:10PM +0800, Gao Xiang wrote: >>>>>> The only problem is that the readahead code doesn't tell the filesystem >>>>>> whether the request is sync or async.  This should be a simple matter >>>>>> of adding a new 'bool async' to the readahead_control and then setting >>>>>> REQ_RAHEAD based on that, rather than on whether the request came in >>>>>> through readahead() or read_folio() (eg see mpage_readahead()). >>>>> >>>>> Great!  In addition to that, just (somewhat) off topic, if we have a >>>>> "bool async" now, I think it will immediately have some users (such as >>>>> EROFS), since we'd like to do post-processing (such as decompression) >>>>> immediately in the same context with sync readahead (due to missing >>>>> pages) and leave it to another kworker for async readahead (I think >>>>> it's almost same for decryption and verification). >>>>> >>>>> So "bool async" is quite useful on my side if it could be possible >>>>> passed to fs side.  I'd like to raise my hands to have it. >>>> >>>> That's a really interesting use-case; thanks for bringing it up. >>>> >>>> Ideally, we'd have the waiting task do the >>>> decompression/decryption/verification for proper accounting of CPU. >>>> Unfortunately, if the folio isn't uptodate, the task doesn't even hold >>>> a reference to the folio while it waits, so there's no way to wake the >>>> task and let it know that it has work to do.  At least not at the moment >>>> ... let me think about that a bit (and if you see a way to do it, feel >>>> free to propose it) >>> >>> Honestly, I'd like to take the folio lock until all post-processing is >>> done and make it uptodate and unlock so that only we need is to pass >>> locked-folios requests to kworkers for async way or sync handling in >>> the original context. >>> >>> If we unlocked these folios in advance without uptodate, which means >>> we have to lock it again (which could have more lock contention) and >>> need to have a way to trace I/Oed but not post-processed stuff in >>> addition to no I/Oed stuff. >> >> Right, look at how it's handled right now ... >> >> sys_read() ends up in filemap_get_pages() which (assuming no folio in >> cache) calls page_cache_sync_readahead().  That creates locked, !uptodate >> folios and asks the filesystem to fill them.  Unless that completes >> incredibly quickly, filemap_get_pages() ends up in filemap_update_page() >> which calls folio_put_wait_locked(). >> >> If the filesystem BIO completion routine could identify if there was >> a task waiting and select one of them, it could wake up the waiter and >> pass it a description of what work it needed to do (with the folio still >> locked), rather than do the postprocessing itself and unlock the folio > > Currently EROFS sync decompression is waiting in .readahead() with locked > page cache folios, one "completion" together than BIO descriptor > (bi_private) in the original context, so that the filesystem BIO completion > just needs to complete the completion and wakeup the original context > (due to missing pages, so the original context will need the page data > immediately as well) to go on .readhead() and unlock folios. > > Does this way have some flew? Or I'm missing something? In this way, EROFS sync decompression is just all handled with a completion in .readahead() and mark uptodate & unlock folios before out of .readahead(), so filemap_update_page() will (almost) always succeed in folio_test_uptodate() before filemap_update_page(): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/erofs/zdata.c?h=v6.2#n1167 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/erofs/zdata.c?h=v6.2#n1167 So I think core-MM just needs to export "bool async" for fses... Thanks, Gao Xiang > > Thanks, > Gao Xiang > >> >> But that all seems _very_ hard to do with 100% reliability.  Note the >> comment in folio_wait_bit_common() which points out that the waiters >> bit may be set even when there are no waiters.  The wake_up code >> doesn't seem to support this kind of thing (all waiters are >> non-exclusive, but only wake up one of them).