From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E378EC433F5 for ; Thu, 28 Oct 2021 15:03:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 604A260C51 for ; Thu, 28 Oct 2021 15:03:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 604A260C51 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E31BD6B006C; Thu, 28 Oct 2021 11:03:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE1D36B0071; Thu, 28 Oct 2021 11:03:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAA67940007; Thu, 28 Oct 2021 11:03:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0218.hostedemail.com [216.40.44.218]) by kanga.kvack.org (Postfix) with ESMTP id A3E7B6B006C for ; Thu, 28 Oct 2021 11:03:08 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 220F51843F4E5 for ; Thu, 28 Oct 2021 15:03:08 +0000 (UTC) X-FDA: 78746164056.14.F2A9B79 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) by imf08.hostedemail.com (Postfix) with ESMTP id 03E54300024A for ; Thu, 28 Oct 2021 15:02:59 +0000 (UTC) Received: by mail-il1-f172.google.com with SMTP id y17so7182556ilb.9 for ; Thu, 28 Oct 2021 08:03:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=aTWh4Okm1+p/YQAesadMe8LIMp+KQN7lDMfYNMC59vk=; b=I49wb6nTPYEiCcPmqQj2n46yc6VDg5xJgv+Od4DeshlfH84fQ3GMV4aaWszeHc5rdm uJDqP1NyNCGpOdtxQFuEAF5aghmyjwERCNz9Y+UFgSNvFDl1ne81cjxI3abKWSImzohh ggt95mbS0NdwPCb4NHlpHTYQ7eZI2UHW9xgzltEdBdtbbhBW+TPAV/6qfA50qIznebpP VlnKLs9+jwZ6e5K5b5AgL7Qy+a6G0e4e0jlUPKZxvg2TpmdKJf8juQBpIE2gokpYyaHY N4Mi4dRdfg5JLmO6F3QXg152B7uyzrKHjfLyPfszJ5xTNp3QTCiXEpcBhpBNUqxQ7LGi njrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=aTWh4Okm1+p/YQAesadMe8LIMp+KQN7lDMfYNMC59vk=; b=YJ/fRe46y/3PUoP9bgXiMzUbwzlD8LWJlECPdBzoSg7wcUqePK6tvQIruNDw1vDw6e U5gJ+txv5p/8LbcF8DA4r4pxddIFWs+M/hlR3LndeWhNdp9leLL0/89s5rRQIqqnZGi9 bZzrtlmljF84DKPEj7dlSHkw9cK7SlOzbL2VGgnm/psHE91lc4Sm0WcldoJh66h/fuft +Hss1QFvIkKKsdEIpSFBl16VAAc9aldsslCKYFht/99UIPVvB+KDHH4Is/JLfhT1+xZv NtZnPdaIdx2MPtThiHDL1G37Ftwj6CucvKvvoHjNi9/RVPl72vUA3iY4IhpKmmqDMYxU TZXg== X-Gm-Message-State: AOAM533LsHRF+dygHrYj9wlRNtBJTdFl9RRgZt4ji99spqyH9fQdWTdp NuO94wdceOOyrCGJhKdqCuLAZN/Kk0422oV6FpA= X-Google-Smtp-Source: ABdhPJwm7cZOSkD4PDVjAKfDvpzMKJHc86x0QxxDtkB6D7ZEBvFKVv7YgkT+HMOK7q7M6BWeZYGIgOl3bBCrMNO7ds8= X-Received: by 2002:a05:6e02:2169:: with SMTP id s9mr3630226ilv.27.1635433386956; Thu, 28 Oct 2021 08:03:06 -0700 (PDT) MIME-Version: 1.0 References: <20211020173729.GF16460@quack2.suse.cz> <20211021080304.GB5784@quack2.suse.cz> <20211022093120.GG1026@quack2.suse.cz> <20211025155713.GD12157@quack2.suse.cz> In-Reply-To: <20211025155713.GD12157@quack2.suse.cz> From: Zhengyuan Liu Date: Thu, 28 Oct 2021 23:02:55 +0800 Message-ID: Subject: Re: Problem with direct IO To: Jan Kara Cc: viro@zeniv.linux.org.uk, Andrew Morton , tytso@mit.edu, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org, =?UTF-8?B?5YiY5LqR?= , Zhengyuan Liu Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 03E54300024A Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=I49wb6nT; spf=pass (imf08.hostedemail.com: domain of liuzhengyuang521@gmail.com designates 209.85.166.172 as permitted sender) smtp.mailfrom=liuzhengyuang521@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: nprwod6w11xeqqt568gch7hjngjqtdxi X-Rspamd-Server: rspam06 X-HE-Tag: 1635433379-164365 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 25, 2021 at 11:57 PM Jan Kara wrote: > > On Sat 23-10-21 10:06:24, Zhengyuan Liu wrote: > > On Fri, Oct 22, 2021 at 5:31 PM Jan Kara wrote: > > > > > > > Can you post output of "dumpe2fs -h " for the filesystem where the > > > > > > > problem happens? Thanks! > > > > > > > > > > > > Sure, the output is: > > > > > > > > > > > > # dumpe2fs -h /dev/sda3 > > > > > > dumpe2fs 1.45.3 (14-Jul-2019) > > > > > > Filesystem volume name: > > > > > > Last mounted on: /data > > > > > > Filesystem UUID: 09a51146-b325-48bb-be63-c9df539a90a1 > > > > > > Filesystem magic number: 0xEF53 > > > > > > Filesystem revision #: 1 (dynamic) > > > > > > Filesystem features: has_journal ext_attr resize_inode dir_index > > > > > > filetype needs_recovery sparse_super large_file > > > > > > > > > > Thanks for the data. OK, a filesystem without extents. Does your test by > > > > > any chance try to do direct IO to a hole in a file? Because that is not > > > > > (and never was) supported without extents. Also the fact that you don't see > > > > > the problem with ext4 (which means extents support) would be pointing in > > > > > that direction. > > > > > > > > I am not sure if it trys to do direct IO to a hole or not, is there any > > > > way to check? If you have a simple test to reproduce please let me know, > > > > we are glad to try. > > > > > > Can you enable following tracing? > > > > Sure, but let's confirm before doing that, it seems Ext4 doesn't > > support iomap in > > V4.19 which could also reproduce the problem, so if it is necessary to > > do the following > > tracing? or should we modify the tracing if under V4.19? > > Well, iomap is just a different generic framework for doing direct IO. The > fact that you can observe the problem both with iomap and the old direct IO > framework is one of the reasons why I think the problem is actually that > the file has holes (unallocated space in the middle). > > > > echo 1 >/sys/kernel/debug/tracing/events/ext4/ext4_ind_map_blocks_exit/enable > > > echo iomap_dio_rw >/sys/kernel/debug/tracing/set_ftrace_filter > > > echo "function_graph" >/sys/kernel/debug/tracing/current_tracer > > If you want to trace a kernel were ext4 direct IO path is not yet > converted to iomap framework you need to replace tracing of iomap_dio_rw > with: > > echo __blockdev_direct_IO >/sys/kernel/debug/tracing/set_ftrace_filter Sorry for the late reply, because it is hard to reproduce when enabling ftrace. I'll post here right after getting useful data. > > > And then gather output from /sys/kernel/debug/tracing/trace_pipe. Once the > > > problem reproduces, you can gather the problematic file name from dmesg, find > > > inode number from "stat " and provide that all to me? Thanks! >