From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5436C433EF for ; Wed, 13 Oct 2021 01:48:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2AEC461039 for ; Wed, 13 Oct 2021 01:48:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2AEC461039 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7F2A16B006C; Tue, 12 Oct 2021 21:48:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A2166B0071; Tue, 12 Oct 2021 21:48:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 66930900002; Tue, 12 Oct 2021 21:48:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id 555436B006C for ; Tue, 12 Oct 2021 21:48:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0E6D18249980 for ; Wed, 13 Oct 2021 01:48:29 +0000 (UTC) X-FDA: 78689729538.07.AF31468 Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) by imf02.hostedemail.com (Postfix) with ESMTP id BD0C17001A0A for ; Wed, 13 Oct 2021 01:48:28 +0000 (UTC) Received: by mail-io1-f44.google.com with SMTP id p68so1001462iof.6 for ; Tue, 12 Oct 2021 18:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to:cc; bh=FBxwHwk0X+rOYnO9GmJcwxmQ+H/dHZFzpYpSSxOsQlE=; b=EiJehrHGbGuNa0m2kRByW7VYY1r3nUsGqr1LCl+pPIvBQa3nfUI/1ACJtoTF6VIWol jUqCuKAXbqIGvHtoT+tJEf0KbXINj+KBddNSg72CDpX25d/DgI/Y5SWjjBoJVvSDmuwA fazDfQ+DpKtgbs/XXbfQ+Vs1niQ2eq4515nbRbtMbBV2JXGBohWDAz3LOx2jZw6+yB+R jIQ7o6BPcUKJSwTy2TQWMa3PjRUuGK0lVLCrOKJ7tT497TAk6+KJd2Q6rPKaK351zAmO /lguqa9foooRmchrBThpd28ZEnLsW/PHL/ES3gNB9oKti+GtyNTc+YhdHR17xvaiz5SV B1BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=FBxwHwk0X+rOYnO9GmJcwxmQ+H/dHZFzpYpSSxOsQlE=; b=xp+EoMrlNEozW0yobhJgprAP/qY1wFDjewSUPSt9q4cC7k9GczhqjTVXxHASu53Sb9 89/2Jh//fWi5PbW8E5N/mzZMufoEbQmwUqrgyceRsNLZQXirRtkERhY1GanrNKHPvk5/ jOe1U1eqL3Lxh7eikPwoYVzNhcveQW5hLNgkBcltHzFgvwOihCM/HnWyOomwh1ius9bE NI38N/8yQZJxBVuP2M3gAEmNOA9yjCvLCLow8TpeixuXoBfCI9XxIWwPooAgFdRUFK0T Q2GxjOADFCtQui0ByCTKZfiIjDmpxY2TygIBVeqQEynbCoL19TogdWYpFkgg6G89peTA W/4Q== X-Gm-Message-State: AOAM531Uj+beO4lFxgMGOVhsPdcnbzyE2FYnU0OrpYEhFGarN04LFsbU jK+LlIi2xMlhJ9lnLlIQq0SKgs2nOUalgEW4OSY= X-Google-Smtp-Source: ABdhPJzAx0KZf8Scqwt+AFfO79rxNTLWd7BGbokaPTfJl1+YaRp7NguEHUVCYCjezYdVmGKsSR9y29Zqwwlxp9KlqRU= X-Received: by 2002:a05:6602:2c88:: with SMTP id i8mr18638993iow.48.1634089708166; Tue, 12 Oct 2021 18:48:28 -0700 (PDT) MIME-Version: 1.0 From: Zhengyuan Liu Date: Wed, 13 Oct 2021 09:46:46 +0800 Message-ID: Subject: Problem with direct IO To: viro@zeniv.linux.org.uk, akpm@linux-foundation.org, tytso@mit.edu Cc: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, mysql@lists.mysql.com, linux-ext4@vger.kernel.org, =?UTF-8?B?5YiY5LqR?= , Zhengyuan Liu Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BD0C17001A0A X-Stat-Signature: 5zysbuhhgmyaywd1gdwd599e8n5rxffn Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EiJehrHG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of liuzhengyuang521@gmail.com designates 209.85.166.44 as permitted sender) smtp.mailfrom=liuzhengyuang521@gmail.com X-HE-Tag: 1634089708-232968 X-Bogosity: Ham, tests=bogofilter, spamicity=0.108137, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, all we are encounting following Mysql crash problem while importing tables : 2021-09-26T11:22:17.825250Z 0 [ERROR] [MY-013622] [InnoDB] [FATAL] fsync() returned EIO, aborting. 2021-09-26T11:22:17.825315Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: ut0ut.cc:555 thread 281472996733168 At the same time , we found dmesg had following message: [ 4328.838972] Page cache invalidation failure on direct I/O. Possible data corruption due to collision with buffered I/O! [ 4328.850234] File: /data/mysql/data/sysbench/sbtest53.ibd PID: 625 Comm: kworker/42:1 Firstly, we doubled Mysql has operating the file with direct IO and buffered IO interlaced, but after some checking we found it did only do direct IO using aio. The problem is exactly from direct-io interface (__generic_file_write_iter) itself. ssize_t __generic_file_write_iter() { ... if (iocb->ki_flags & IOCB_DIRECT) { loff_t pos, endbyte; written = generic_file_direct_write(iocb, from); /* * If the write stopped short of completing, fall back to * buffered writes. Some filesystems do this for writes to * holes, for example. For DAX files, a buffered write will * not succeed (even if it did, DAX does not handle dirty * page-cache pages correctly). */ if (written < 0 || !iov_iter_count(from) || IS_DAX(inode)) goto out; status = generic_perform_write(file, from, pos = iocb->ki_pos); ... } >From above code snippet we can see that direct io could fall back to buffered IO under certain conditions, so even Mysql only did direct IO it could interleave with buffered IO when fall back occurred. I have no idea why FS(ext3) failed the direct IO currently, but it is strange __generic_file_write_iter make direct IO fall back to buffered IO, it seems breaking the semantics of direct IO. The reproduced environment is: Platform: Kunpeng 920 (arm64) Kernel: V5.15-rc PAGESIZE: 64K Mysql: V8.0 Innodb_page_size: default(16K) Thanks,