From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F1A6C433FE for ; Wed, 20 Oct 2021 17:37:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8C748610EA for ; Wed, 20 Oct 2021 17:37:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8C748610EA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0381E6B0071; Wed, 20 Oct 2021 13:37:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F298F6B0080; Wed, 20 Oct 2021 13:37:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF1946B0081; Wed, 20 Oct 2021 13:37:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id CFCB16B0071 for ; Wed, 20 Oct 2021 13:37:34 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8B1191818CBF6 for ; Wed, 20 Oct 2021 17:37:34 +0000 (UTC) X-FDA: 78717522828.07.4A1B733 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf30.hostedemail.com (Postfix) with ESMTP id C796DE0016A1 for ; Wed, 20 Oct 2021 17:37:28 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id A0FBE1F770; Wed, 20 Oct 2021 17:37:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1634751452; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=fwUSPhdBez3rOtdXQZGr/c7TTcI5ppZjDsY4VW5mwVw=; b=aUB1mJkA23iGtnk8d4hV70kSXH0UErgLyfLnEk22MEms6lrAjuGhe1dKpHmiGSxPVqESMS OmXRp0EWWTWOPIBPu7LSmeo9Ox3rQ0uiaqXKu6Xjy4RN2qHagZ9wRoKFdkEHP1Kp+IPI2E BWjrzpYWxxtX4Dmi0k+gU/Lgp2xKF/k= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1634751452; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=fwUSPhdBez3rOtdXQZGr/c7TTcI5ppZjDsY4VW5mwVw=; b=BKdycMabWA4i1T6wCjXXcMDBiKO49lic7/TZvFZHu312eVRxElPJrI1z/BtMRMGkjxu5cC t+WUbA2LXiWtWtCg== Received: from quack2.suse.cz (unknown [10.100.224.230]) by relay2.suse.de (Postfix) with ESMTP id A5345A3B84; Wed, 20 Oct 2021 17:37:31 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 3804D1F2C7D; Wed, 20 Oct 2021 19:37:29 +0200 (CEST) Date: Wed, 20 Oct 2021 19:37:29 +0200 From: Jan Kara To: Zhengyuan Liu Cc: viro@zeniv.linux.org.uk, akpm@linux-foundation.org, tytso@mit.edu, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, mysql@lists.mysql.com, linux-ext4@vger.kernel.org, =?utf-8?B?5YiY5LqR?= , Zhengyuan Liu Subject: Re: Problem with direct IO Message-ID: <20211020173729.GF16460@quack2.suse.cz> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C796DE0016A1 X-Stat-Signature: t71g8ujpg6ne4767xuxc8eocd8csmn7e Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=aUB1mJkA; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=BKdycMab; spf=pass (imf30.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-HE-Tag: 1634751448-84197 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 13-10-21 09:46:46, Zhengyuan Liu wrote: > Hi, all > > we are encounting following Mysql crash problem while importing tables : > > 2021-09-26T11:22:17.825250Z 0 [ERROR] [MY-013622] [InnoDB] [FATAL] > fsync() returned EIO, aborting. > 2021-09-26T11:22:17.825315Z 0 [ERROR] [MY-013183] [InnoDB] > Assertion failure: ut0ut.cc:555 thread 281472996733168 > > At the same time , we found dmesg had following message: > > [ 4328.838972] Page cache invalidation failure on direct I/O. > Possible data corruption due to collision with buffered I/O! > [ 4328.850234] File: /data/mysql/data/sysbench/sbtest53.ibd PID: > 625 Comm: kworker/42:1 > > Firstly, we doubled Mysql has operating the file with direct IO and > buffered IO interlaced, but after some checking we found it did only > do direct IO using aio. The problem is exactly from direct-io > interface (__generic_file_write_iter) itself. > > ssize_t __generic_file_write_iter() > { > ... > if (iocb->ki_flags & IOCB_DIRECT) { > loff_t pos, endbyte; > > written = generic_file_direct_write(iocb, from); > /* > * If the write stopped short of completing, fall back to > * buffered writes. Some filesystems do this for writes to > * holes, for example. For DAX files, a buffered write will > * not succeed (even if it did, DAX does not handle dirty > * page-cache pages correctly). > */ > if (written < 0 || !iov_iter_count(from) || IS_DAX(inode)) > goto out; > > status = generic_perform_write(file, from, pos = iocb->ki_pos); > ... > } > > From above code snippet we can see that direct io could fall back to > buffered IO under certain conditions, so even Mysql only did direct IO > it could interleave with buffered IO when fall back occurred. I have > no idea why FS(ext3) failed the direct IO currently, but it is strange > __generic_file_write_iter make direct IO fall back to buffered IO, it > seems breaking the semantics of direct IO. > > The reproduced environment is: > Platform: Kunpeng 920 (arm64) > Kernel: V5.15-rc > PAGESIZE: 64K > Mysql: V8.0 > Innodb_page_size: default(16K) Thanks for report. I agree this should not happen. How hard is this to reproduce? Any idea whether the fallback to buffered IO happens because iomap_dio_rw() returns -ENOTBLK or because it returns short write? Can you post output of "dumpe2fs -h " for the filesystem where the problem happens? Thanks! Honza -- Jan Kara SUSE Labs, CR