From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9EA4C433FE for ; Mon, 25 Oct 2021 15:57:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5F64A60C49 for ; Mon, 25 Oct 2021 15:57:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5F64A60C49 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B8FC4940008; Mon, 25 Oct 2021 11:57:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B16B2940007; Mon, 25 Oct 2021 11:57:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B6D3940008; Mon, 25 Oct 2021 11:57:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0004.hostedemail.com [216.40.44.4]) by kanga.kvack.org (Postfix) with ESMTP id 8651B940007 for ; Mon, 25 Oct 2021 11:57:17 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4144C2C5A6 for ; Mon, 25 Oct 2021 15:57:17 +0000 (UTC) X-FDA: 78735414114.39.0BC26A5 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf05.hostedemail.com (Postfix) with ESMTP id 8618C5085CC1 for ; Mon, 25 Oct 2021 15:57:07 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 85FB2218EE; Mon, 25 Oct 2021 15:57:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1635177435; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Sjc87XfL1WuZPJro7Vq4FUjnrPJgaAca3TEnfkMGFQU=; b=y+29YJzTI+Rj/0CN58+POSX0QZMcdtrzVssxi1+pXRJEvRbzvgZEtiHiQ3jaquq723xt8C k/H5xZnWppnhLMNizOAdON59mxMq2zGrNCHH54Q+u4D3DpmByuMgc+SuhohDLyr5v7U1Js qejVkRfxMaK2fOMj+whbhnLafQUNqmU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1635177435; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Sjc87XfL1WuZPJro7Vq4FUjnrPJgaAca3TEnfkMGFQU=; b=wOcxGJuaXT3sNaVtUIfhYyh03F6r5p1HMOOR1lzQKSSCONe4CHh16zsLVdXfJyqM5oq5sz LeYRw1LmZ1nqLLAA== Received: from quack2.suse.cz (unknown [10.100.224.230]) by relay2.suse.de (Postfix) with ESMTP id 6DBA7A3B81; Mon, 25 Oct 2021 15:57:13 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 45D681E0A2B; Mon, 25 Oct 2021 17:57:13 +0200 (CEST) Date: Mon, 25 Oct 2021 17:57:13 +0200 From: Jan Kara To: Zhengyuan Liu Cc: Jan Kara , viro@zeniv.linux.org.uk, Andrew Morton , tytso@mit.edu, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org, =?utf-8?B?5YiY5LqR?= , Zhengyuan Liu Subject: Re: Problem with direct IO Message-ID: <20211025155713.GD12157@quack2.suse.cz> References: <20211020173729.GF16460@quack2.suse.cz> <20211021080304.GB5784@quack2.suse.cz> <20211022093120.GG1026@quack2.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: 8618C5085CC1 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=y+29YJzT; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=wOcxGJua; spf=pass (imf05.hostedemail.com: domain of jack@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Stat-Signature: 36s33akzg5oxh9j98mbzq5pt7qo3eset X-Rspamd-Server: rspam06 X-HE-Tag: 1635177427-891954 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat 23-10-21 10:06:24, Zhengyuan Liu wrote: > On Fri, Oct 22, 2021 at 5:31 PM Jan Kara wrote: > > > > > > Can you post output of "dumpe2fs -h " for the filesystem where the > > > > > > problem happens? Thanks! > > > > > > > > > > Sure, the output is: > > > > > > > > > > # dumpe2fs -h /dev/sda3 > > > > > dumpe2fs 1.45.3 (14-Jul-2019) > > > > > Filesystem volume name: > > > > > Last mounted on: /data > > > > > Filesystem UUID: 09a51146-b325-48bb-be63-c9df539a90a1 > > > > > Filesystem magic number: 0xEF53 > > > > > Filesystem revision #: 1 (dynamic) > > > > > Filesystem features: has_journal ext_attr resize_inode dir_index > > > > > filetype needs_recovery sparse_super large_file > > > > > > > > Thanks for the data. OK, a filesystem without extents. Does your test by > > > > any chance try to do direct IO to a hole in a file? Because that is not > > > > (and never was) supported without extents. Also the fact that you don't see > > > > the problem with ext4 (which means extents support) would be pointing in > > > > that direction. > > > > > > I am not sure if it trys to do direct IO to a hole or not, is there any > > > way to check? If you have a simple test to reproduce please let me know, > > > we are glad to try. > > > > Can you enable following tracing? > > Sure, but let's confirm before doing that, it seems Ext4 doesn't > support iomap in > V4.19 which could also reproduce the problem, so if it is necessary to > do the following > tracing? or should we modify the tracing if under V4.19? Well, iomap is just a different generic framework for doing direct IO. The fact that you can observe the problem both with iomap and the old direct IO framework is one of the reasons why I think the problem is actually that the file has holes (unallocated space in the middle). > > echo 1 >/sys/kernel/debug/tracing/events/ext4/ext4_ind_map_blocks_exit/enable > > echo iomap_dio_rw >/sys/kernel/debug/tracing/set_ftrace_filter > > echo "function_graph" >/sys/kernel/debug/tracing/current_tracer If you want to trace a kernel were ext4 direct IO path is not yet converted to iomap framework you need to replace tracing of iomap_dio_rw with: echo __blockdev_direct_IO >/sys/kernel/debug/tracing/set_ftrace_filter > > And then gather output from /sys/kernel/debug/tracing/trace_pipe. Once the > > problem reproduces, you can gather the problematic file name from dmesg, find > > inode number from "stat " and provide that all to me? Thanks! Honza -- Jan Kara SUSE Labs, CR