From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4636AC4332F for ; Mon, 18 Oct 2021 18:43:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C891F6127B for ; Mon, 18 Oct 2021 18:43:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C891F6127B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 591E7900003; Mon, 18 Oct 2021 14:43:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 540A6900002; Mon, 18 Oct 2021 14:43:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 430C3900003; Mon, 18 Oct 2021 14:43:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id 346A5900002 for ; Mon, 18 Oct 2021 14:43:54 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C71F032346 for ; Mon, 18 Oct 2021 18:43:53 +0000 (UTC) X-FDA: 78710432346.05.57ABDCE Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP id 268B1B0000A2 for ; Mon, 18 Oct 2021 18:43:51 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 32EF36103C; Mon, 18 Oct 2021 18:43:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1634582632; bh=H3l6oRMr0L9AgsGKguVWXHWB0Kjms53ahKLBQ0RTSWQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=Q/1WxzcG7z261AoQfugx2edEXS+zsK0tf/QatW/mnNvREcNMWDsfaqZkL1JB640lJ 97DSLSno9TypB5MIqczA3GCTn/OIg9CnKessQzxe8pHKFmdUBFwA/Qa/QEb/rfb6rK rVgSevWZbw23i4E0pyO7syOaI4swCRYbn/EYpAU8= Date: Mon, 18 Oct 2021 11:43:49 -0700 From: Andrew Morton To: Zhengyuan Liu Cc: viro@zeniv.linux.org.uk, tytso@mit.edu, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, mysql@lists.mysql.com, linux-ext4@vger.kernel.org, =?UTF-8?Q?=E5=88=98=E4=BA=91?= , Zhengyuan Liu Subject: Re: Problem with direct IO Message-Id: <20211018114349.b80a27af9bfa7f16162b0ec4@linux-foundation.org> In-Reply-To: References: X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.31; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 268B1B0000A2 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b="Q/1WxzcG"; spf=pass (imf19.hostedemail.com: domain of akpm@linux-foundation.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none X-Stat-Signature: 1tog3ii9p5f35dsq18rjdx8bur6a8qom X-Rspamd-Server: rspam05 X-HE-Tag: 1634582631-716342 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 18 Oct 2021 09:09:06 +0800 Zhengyuan Liu wrote: > Ping. > > I think this problem is serious and someone may also encounter it in > the future. > > > On Wed, Oct 13, 2021 at 9:46 AM Zhengyuan Liu > wrote: > > > > Hi, all > > > > we are encounting following Mysql crash problem while importing tables : > > > > 2021-09-26T11:22:17.825250Z 0 [ERROR] [MY-013622] [InnoDB] [FATAL] > > fsync() returned EIO, aborting. > > 2021-09-26T11:22:17.825315Z 0 [ERROR] [MY-013183] [InnoDB] > > Assertion failure: ut0ut.cc:555 thread 281472996733168 > > > > At the same time , we found dmesg had following message: > > > > [ 4328.838972] Page cache invalidation failure on direct I/O. > > Possible data corruption due to collision with buffered I/O! > > [ 4328.850234] File: /data/mysql/data/sysbench/sbtest53.ibd PID: > > 625 Comm: kworker/42:1 > > > > Firstly, we doubled Mysql has operating the file with direct IO and > > buffered IO interlaced, but after some checking we found it did only > > do direct IO using aio. The problem is exactly from direct-io > > interface (__generic_file_write_iter) itself. > > > > ssize_t __generic_file_write_iter() > > { > > ... > > if (iocb->ki_flags & IOCB_DIRECT) { > > loff_t pos, endbyte; > > > > written = generic_file_direct_write(iocb, from); > > /* > > * If the write stopped short of completing, fall back to > > * buffered writes. Some filesystems do this for writes to > > * holes, for example. For DAX files, a buffered write will > > * not succeed (even if it did, DAX does not handle dirty > > * page-cache pages correctly). > > */ > > if (written < 0 || !iov_iter_count(from) || IS_DAX(inode)) > > goto out; > > > > status = generic_perform_write(file, from, pos = iocb->ki_pos); > > ... > > } > > > > From above code snippet we can see that direct io could fall back to > > buffered IO under certain conditions, so even Mysql only did direct IO > > it could interleave with buffered IO when fall back occurred. I have > > no idea why FS(ext3) failed the direct IO currently, but it is strange > > __generic_file_write_iter make direct IO fall back to buffered IO, it > > seems breaking the semantics of direct IO. That makes sense. > > The reproduced environment is: > > Platform: Kunpeng 920 (arm64) > > Kernel: V5.15-rc > > PAGESIZE: 64K > > Mysql: V8.0 > > Innodb_page_size: default(16K) This is all fairly mature code, I think. Do you know if earlier kernels were OK, and if so which versions? Thanks.