From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9AAEC433FE for ; Mon, 18 Oct 2021 01:09:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5947B610A6 for ; Mon, 18 Oct 2021 01:09:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5947B610A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 80FDB6B006C; Sun, 17 Oct 2021 21:09:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BE976B0071; Sun, 17 Oct 2021 21:09:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AD9F900002; Sun, 17 Oct 2021 21:09:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id 5C9FF6B006C for ; Sun, 17 Oct 2021 21:09:19 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 149018249980 for ; Mon, 18 Oct 2021 01:09:19 +0000 (UTC) X-FDA: 78707774838.25.B3710FF Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) by imf06.hostedemail.com (Postfix) with ESMTP id 19885801A89B for ; Mon, 18 Oct 2021 01:09:17 +0000 (UTC) Received: by mail-il1-f180.google.com with SMTP id y17so13167558ilb.9 for ; Sun, 17 Oct 2021 18:09:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WfeOjNSr14+hr9YL9DUQsshVwCDugysEM+0F+xx1LNs=; b=LCWXoLiFeq/6CvXKyYDIpZX75/Z6HSNEQHQN/yKd8EwdLAyfQpzjX21S4UryzIOWgz 463zMzy/5GqP2+tE4lOB1gD+tSGH2qoq8YlF84mKq46+CRxF2RpSBpqt6z1eTCchUV9e B6fwq8W2z1PBKaFIvHtGCK5L1OQTd7QxksM+soX9Iso8Eg1SrU1oeS7Z+8QOlCSdzuLp YNcE1FFJ6hWp1KTJNUC4aTRpW3z160myslZknXu/ZWm7vIxZRfP21FU1VQygI0ynL3cP OZq8cYCK2L5CCiE1MnmI5Tmf4zdtxRpdgF3j7IIy5JVHhu2A87tFa2DC6n9DnUjUyPXt NG1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WfeOjNSr14+hr9YL9DUQsshVwCDugysEM+0F+xx1LNs=; b=TYfBCoBdjZjB2Kq/fM/lbN03e4Uh0Sd5k24Sa9b7P97gEZWVC+qtJrdw2IYiqwcZ4G GG3PKwaRq73Biexzmxjq3aE6fCTRKTRHCq+RRIlUURT1MOZ1CmABpxSqihK1jeThppy7 MnaIRT2VwnJeLbH2T0eJYqwI6I9XjNNKZqAIpgED/z84P9K+Om8JGM3kbZqWB0JiiBiL V7VkzS6nOzRK/V0GzAVjMpEOik3ChnXxiON2flQVunT4RnCuR8AvWfLAl/4UK2wi0MWK NBQ1i125PCEVMsGvNRtWHKo3/JUq6x4UaK/eagBlSrlKAraalPo8OnJ2htNS3CVNOH7R gjvg== X-Gm-Message-State: AOAM530tA7R+c299amV2glIIeAqHJbwFgt/ZI0oZjOou9NfWJPyqjBTA nhFpcXpQjTdZgiUQPtMm03sAYEiXguozC6fflxM= X-Google-Smtp-Source: ABdhPJyZSh3itSF+VXrY4OH3OZp6AdtHKiAionYTGWjW2Fk28B/StczyWg2y7en5AEMUGL5dW5S27vz4XCBz1ck8+ws= X-Received: by 2002:a05:6e02:19c9:: with SMTP id r9mr3302469ill.98.1634519358012; Sun, 17 Oct 2021 18:09:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Zhengyuan Liu Date: Mon, 18 Oct 2021 09:09:06 +0800 Message-ID: Subject: Re: Problem with direct IO To: viro@zeniv.linux.org.uk, akpm@linux-foundation.org, tytso@mit.edu Cc: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, mysql@lists.mysql.com, linux-ext4@vger.kernel.org, =?UTF-8?B?5YiY5LqR?= , Zhengyuan Liu Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 19885801A89B X-Stat-Signature: xdqi6djb7hbhnuxtq1wcxdmqws3a1r5g Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LCWXoLiF; spf=pass (imf06.hostedemail.com: domain of liuzhengyuang521@gmail.com designates 209.85.166.180 as permitted sender) smtp.mailfrom=liuzhengyuang521@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1634519357-615620 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Ping. I think this problem is serious and someone may also encounter it in the future. On Wed, Oct 13, 2021 at 9:46 AM Zhengyuan Liu wrote: > > Hi, all > > we are encounting following Mysql crash problem while importing tables : > > 2021-09-26T11:22:17.825250Z 0 [ERROR] [MY-013622] [InnoDB] [FATAL] > fsync() returned EIO, aborting. > 2021-09-26T11:22:17.825315Z 0 [ERROR] [MY-013183] [InnoDB] > Assertion failure: ut0ut.cc:555 thread 281472996733168 > > At the same time , we found dmesg had following message: > > [ 4328.838972] Page cache invalidation failure on direct I/O. > Possible data corruption due to collision with buffered I/O! > [ 4328.850234] File: /data/mysql/data/sysbench/sbtest53.ibd PID: > 625 Comm: kworker/42:1 > > Firstly, we doubled Mysql has operating the file with direct IO and > buffered IO interlaced, but after some checking we found it did only > do direct IO using aio. The problem is exactly from direct-io > interface (__generic_file_write_iter) itself. > > ssize_t __generic_file_write_iter() > { > ... > if (iocb->ki_flags & IOCB_DIRECT) { > loff_t pos, endbyte; > > written = generic_file_direct_write(iocb, from); > /* > * If the write stopped short of completing, fall back to > * buffered writes. Some filesystems do this for writes to > * holes, for example. For DAX files, a buffered write will > * not succeed (even if it did, DAX does not handle dirty > * page-cache pages correctly). > */ > if (written < 0 || !iov_iter_count(from) || IS_DAX(inode)) > goto out; > > status = generic_perform_write(file, from, pos = iocb->ki_pos); > ... > } > > From above code snippet we can see that direct io could fall back to > buffered IO under certain conditions, so even Mysql only did direct IO > it could interleave with buffered IO when fall back occurred. I have > no idea why FS(ext3) failed the direct IO currently, but it is strange > __generic_file_write_iter make direct IO fall back to buffered IO, it > seems breaking the semantics of direct IO. > > The reproduced environment is: > Platform: Kunpeng 920 (arm64) > Kernel: V5.15-rc > PAGESIZE: 64K > Mysql: V8.0 > Innodb_page_size: default(16K) > > Thanks,