From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A81CC10DCE for ; Mon, 4 Dec 2023 14:41:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA6A76B020B; Mon, 4 Dec 2023 09:41:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B56956B020F; Mon, 4 Dec 2023 09:41:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1E666B02BE; Mon, 4 Dec 2023 09:41:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 956886B020B for ; Mon, 4 Dec 2023 09:41:11 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 73868C025C for ; Mon, 4 Dec 2023 14:41:11 +0000 (UTC) X-FDA: 81529398342.11.4F1E698 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf02.hostedemail.com (Postfix) with ESMTP id 13FA08000C for ; Mon, 4 Dec 2023 14:41:08 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=wI8PwYjE; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="WDJyYUy/"; spf=pass (imf02.hostedemail.com: domain of jack@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701700869; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oZ5GBwYG9yYyA/BLwztwhsbyqCCEQU8wZSQHMYMGfGY=; b=LeLopYxvAmqgJgDBFG26TtqwVVt3NSkKjICj2R/tL6shpPjUXS9z+OaM2UieerDDyRfrg0 NRRr9gpMdNrok9We/XsOqQmUaEKZPxC/0YX+1XkSGZQZesJ6ik0rQCoG78Z5h+Nj1OFeu0 ovmsmJ2JioFDDp3yD2LKXOs/pkWm+sA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701700869; a=rsa-sha256; cv=none; b=nc89ISKFiZ7lsJEP2HuQsshrctajV4NXGkQv3AfjIbEguAmoN3s1aqxXt21h4fR3vdgOPe z+gor9C8gboo29+Z7MrR8VrL1DBbkBux3PzqQ2YSLdBdIhSQmK+MvDjuin/xgDM/QHhVfx +9aQeRNAh0yVFgj1Iz+68kavxLT6lXw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=wI8PwYjE; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="WDJyYUy/"; spf=pass (imf02.hostedemail.com: domain of jack@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 6870A1FCE8; Mon, 4 Dec 2023 14:41:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1701700867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=oZ5GBwYG9yYyA/BLwztwhsbyqCCEQU8wZSQHMYMGfGY=; b=wI8PwYjECnoMx/1ik3UIhSJavowayEM+Pt/v9tfVpsUA+C4764jEXunRHprIqNyw/U1Sbk KjgDmS4iz2mCv68mYCe4BgGO+rxagYTrhx4H0BbFblac0C4iqIQ4gddyq86tZiC+RNwCh/ b0eaE0rM9MSkL8xJHsQOUbQ5/TwQ7jw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1701700867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=oZ5GBwYG9yYyA/BLwztwhsbyqCCEQU8wZSQHMYMGfGY=; b=WDJyYUy/wKTW5vIqC5AvqhpoCXROn9S9aKSSGTZ9qQGvx1ZPFMk61NomQeOKeZKbFPsVeR bYjWQyquj9MyxnCA== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 5A433139E2; Mon, 4 Dec 2023 14:41:07 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id wngFFgPlbWUwGAAAn2gu4w (envelope-from ); Mon, 04 Dec 2023 14:41:07 +0000 Received: by quack3.suse.cz (Postfix, from userid 1000) id EDD24A07DB; Mon, 4 Dec 2023 15:41:06 +0100 (CET) Date: Mon, 4 Dec 2023 15:41:06 +0100 From: Jan Kara To: Baokun Li Cc: Jan Kara , linux-mm@kvack.org, linux-ext4@vger.kernel.org, tytso@mit.edu, adilger.kernel@dilger.ca, willy@infradead.org, akpm@linux-foundation.org, ritesh.list@gmail.com, linux-kernel@vger.kernel.org, yi.zhang@huawei.com, yangerkun@huawei.com, yukuai3@huawei.com Subject: Re: [PATCH -RFC 0/2] mm/ext4: avoid data corruption when extending DIO write race with buffered read Message-ID: <20231204144106.fk4yxc422gppifsz@quack3> References: <20231202091432.8349-1-libaokun1@huawei.com> <20231204121120.mpxntey47rluhcfi@quack3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 13FA08000C X-Rspam-User: X-Stat-Signature: 6kicckgerp8f78bxruon5jh6kbp15a8f X-Rspamd-Server: rspam03 X-HE-Tag: 1701700868-943682 X-HE-Meta: U2FsdGVkX1+in8HogPPzfS4LEs80YnLLVnvbNuuI3/NZfDKhtQlhM2nVtaZNvagvyW211u51amJAuUUml9vZ1YB8iW/MpWw625yR/ISb0Vo5XiZF09LwfUaVVKy234xTdcXYA3tGj0CKmc/REU8p7U69hdYTaMB8A/mziz8KKnD0PXUVyQwbGvFFMX0FM+SDn9fxMwpKtYqviUmopMP5Q+74JsOjf323xCBxyxpTS+E+m9eMLIz1z6KRU6RDhXjh/XmsHGjR3YrVg78n+YAqNDXFVJBRnlTlKnxQFH0oMgNvBK0oBNsxcdeTSSXZP1+TWCIVev1iHXq241cLiMqkdbUgWT/IIBjnD7+78SFZKw6eD2OcyKxPxmTQmaED8Cawl2LYXdSaA6gJ0mTuQsHf1xje1KsqEBge35qu8b4DnJO9e07rGARxEqn5efjKnU0sEHgVEVctTLb+TAQ1/Sj08gEJgowrd5WDU0XtCR5srvy6J4xZVFEq8X93LQRGE7n2h1OyEUWbcX/ePyDoCQxCmwOC6IxEdoP0BeaE9uco89OCoAfGbEJjXtkFykdISoZmbE/XjBZdfwnnkvBQBZE/k05EgTIVgc3GJOakaiSeuxwe+wvqBjq6WkZ5Z/Jf/go+qOWoD3nXjgJNGr8mDVuiAd0FnI4adLAojSyBHqo5uTBcsrwR24vHCPOtNjG2o2R16Z05nLsD71QujrFD1fbEA1DuOsYHKoBDUZjGLmuMe6phBzSLQ4pElo6hhlh+hZXnJl8W+6/Y80ykLdZqnLXPqv1T3DYNMRwa+3pBUEDrnJLdz9173dKwiXZmllbKu9wt1Ed58EkKb1ivsLTljDVBkizNJeOp/Rb3e3Ua8Bwu3r9gCxxTtN7Tm7XU+MVq/77itDecDlraBbmxnjDJcY/VzuAyhEsay0kswipLmFHP3eHWTqzHt3cQqAFHRJqtwKAJ7EvJYcS+RxESZxQaRSY dk9SX/B/ 0a/eVc6xrZRfzjrAj42zd/aXwMgw+Yhl3kefpYJpLOmAa35jh5YLqP6LekJE+quU7tB2ZiZbkDqHX7Gsmcr2QVya9F5oelkV4yFBH2DxrEweFJeqARHcZ5TfLvn5H1DJfhihqqT5bD4pVy2OStYgnllbNMWm+ot252g2JlY8itI2yyjJbHPckjKLnlDEN44oL9Hp2Uyma4D8mK3V505uPr9odmzcs4wxairhNbc4Wiz2l4JbZJdnA6dKwApzO4NufSHW0kJkenXpxcRfImM2YMxSQj/sVXnd/W6rVKUrrupYWmDW9MDo4SgXtp/DBPP7zrxz9X3SHWOabsPqzrfBn1j8a1c94UOnqU4zo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon 04-12-23 21:50:18, Baokun Li wrote: > On 2023/12/4 20:11, Jan Kara wrote: > > On Sat 02-12-23 17:14:30, Baokun Li wrote: > > > Recently, while running some pressure tests on MYSQL, noticed that > > > occasionally a "corrupted data in log event" error would be reported. > > > After analyzing the error, I found that extending DIO write and buffered > > > read were competing, resulting in some zero-filled page end being read. > > > Since ext4 buffered read doesn't hold an inode lock, and there is no > > > field in the page to indicate the valid data size, it seems to me that > > > it is impossible to solve this problem perfectly without changing these > > > two things. > > Yes, combining buffered reads with direct IO writes is a recipe for > > problems and pretty much in the "don't do it" territory. So honestly I'd > > consider this a MYSQL bug. Were you able to identify why does MYSQL use > > buffered read in this case? It is just something specific to the test > > you're doing? > The problem is with a one-master-twoslave MYSQL database with three > physical machines, and using sysbench pressure testing on each of the > three machines, the problem occurs about once every two to three hours. > > The problem is with the relay log file, and when the problem occurs, the > middle dozens of bytes of the file are read as all zeros, while the data on > disk is not. This is a journal-like file where a write process gets the data > from > the master node and writes it locally, and another replay process reads the > file and performs the replay operation accordingly (some SQL statements). > The problem is that when replaying, it finds that the data read is > corrupted, > not valid SQL data, while the data on disk is normal. > > It's not confirmed that buffered reads vs direct IO writes is actually > causing this issue, but this is the only scenario that we can reproduce > with our local simplified scripts. Also, after merging in patch 1, the > MYSQL pressure test scenario has now been tested for 5 days and has not > been reproduced. > > I'll double-check the problem scenario, although buffered reads with > buffered writes doesn't seem to have this problem. Yeah, from what you write it seems that the replay code is using buffered reads on the journal file. I guess you could confirm that with a bit of kernel tracing but the symptoms look pretty convincing. Did you try talking to MYSQL guys about why they are doing this? > > > In this series, the first patch reads the inode size twice, and takes the > > > smaller of the two values as the copyout limit to avoid copying data that > > > was not actually read (0-padding) into the user buffer and causing data > > > corruption. This greatly reduces the probability of problems under 4k > > > page. However, the problem is still easily triggered under 64k page. > > > > > > The second patch waits for the existing dio write to complete and > > > invalidate the stale page cache before performing a new buffered read > > > in ext4, avoiding data corruption by copying the stale page cache to > > > the user buffer. This makes it much less likely that the problem will > > > be triggered in a 64k page. > > > > > > Do we have a plan to add a lock to the ext4 buffered read or a field in > > > the page that indicates the size of the valid data in the page? Or does > > > anyone have a better idea? > > No, there are no plans to address this AFAIK. Because such locking will > > slow down all the well behaved applications to fix a corner case for > > application doing unsupported things. Sure we must not crash the kernel, > > corrupt the filesystem or leak sensitive (e.g. uninitialized) data if app > > combines buffered and direct IO but returning zeros instead of valid data > > is in my opinion fully within the range of acceptable behavior for such > > case. > > > I also feel that a scenario like buffered reads + DIO writes is strange. > But theoretically when read doesn't return an error, the data read > shouldn't be wrong. And I tested that xfs guarantees data consistency in > this scenario, which is why I thought it might be buggy. Yes, XFS has inherited stronger consistency guarantees from IRIX times than Linux filesystems traditionally had. We generally don't even guarantee buffered read vs buffered write atomicity (i.e., buffered read can see a torn buffered write). Honza -- Jan Kara SUSE Labs, CR