From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D621EC636D3 for ; Wed, 8 Feb 2023 19:37:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61A306B007D; Wed, 8 Feb 2023 14:37:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C9AB6B007E; Wed, 8 Feb 2023 14:37:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4921A6B0080; Wed, 8 Feb 2023 14:37:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A41F6B007D for ; Wed, 8 Feb 2023 14:37:38 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C7AC41C5FDB for ; Wed, 8 Feb 2023 19:37:37 +0000 (UTC) X-FDA: 80445134154.09.8287BEA Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by imf02.hostedemail.com (Postfix) with ESMTP id E587A8000D for ; Wed, 8 Feb 2023 19:37:35 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=fail ("headers rsa verify failed") header.d=mit.edu header.s=outgoing header.b=fWZexWOd; spf=pass (imf02.hostedemail.com: domain of tytso@mit.edu designates 18.9.28.11 as permitted sender) smtp.mailfrom=tytso@mit.edu; dmarc=pass (policy=none) header.from=mit.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675885056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WS8yi8V+z1lLpmoH+0uIKeV1wOV5nUwuLs7CM/oWWsE=; b=6MfIFeQ6pFimpLJYa3GUBSjyJIVvXcAsoIelG/uQS6Exs8Moe7es/8jvxa7/TmzX3ZabKk 9d7gf4azbPmMBOB8FJf/PKH1anUakzy3ZBfAFkCf6jXBJ6SPzTtFc1AdlILVSmmK6qClkn BvARXucq72VsnQ0s1sQzboeRpYyuGhU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=fail ("headers rsa verify failed") header.d=mit.edu header.s=outgoing header.b=fWZexWOd; spf=pass (imf02.hostedemail.com: domain of tytso@mit.edu designates 18.9.28.11 as permitted sender) smtp.mailfrom=tytso@mit.edu; dmarc=pass (policy=none) header.from=mit.edu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675885056; a=rsa-sha256; cv=none; b=KcMUINlAErHSPkHH2RNk5PobYKnL36eFyaWMg10feLeecinmXT+kwrR+xvkLTwhvzWyF9R 8XV9FNcwOHBTwxO7xb+tx8mcM/uTsyrxs2RCaiXMWj/RQPxgHWJiKn/+I4h6ZiykVqr5+1 0vlj53nZjF7JFEVpM833O85YOXJuQko= Received: from cwcc.thunk.org (pool-173-48-120-46.bstnma.fios.verizon.net [173.48.120.46]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 318JbUVR012588 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 8 Feb 2023 14:37:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1675885052; bh=WS8yi8V+z1lLpmoH+0uIKeV1wOV5nUwuLs7CM/oWWsE=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=fWZexWOdUQPoK2iQ5J7DxyBlN1+HNKAgl/kjEZ6I64BzOGEO8heMMJHy9k+IVPxeS VRS2ZTx6lbpRHO272sSRLoNUoA78BxZ23iUK1GHZPzJ+7A62Xhd0+z2c9dK/KNGIyW wxnQJlRFs8jZ7axYmWilPLcI8Is54nrN+U+2OfPte5vD7GuyuRGDeQucE7Jb8SK4+z Zs1HF5K0VCG9C6Z+jfcGsYJhSkXmtTkqrKadfcsflc2zFPpDrgK73vh41bBcirG/VN iHmka7egx9sOAjDmK5YiOgi79HDBp8fNyWbilNlhm/dKfKbRuFWBO3jIcZ684GtSPD xj5fg03OZiv9w== Received: by cwcc.thunk.org (Postfix, from userid 15806) id 125F715C35A2; Wed, 8 Feb 2023 14:37:30 -0500 (EST) Date: Wed, 8 Feb 2023 14:37:30 -0500 From: "Theodore Ts'o" To: Jan Kara Cc: "Bhatnagar, Rishabh" , akpm@linux-foundation.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org, abuehaze@amazon.com Subject: Re: EXT4 IOPS degradation between 4.14 and 5.10 Message-ID: References: <053b60a6-133e-5d59-0732-464d5160772a@amazon.com> <20230126093231.ujn6yaxhexwzizp5@quack3> <20230127121721.lerrb36nhj7gdiwm@quack3> <6a0fcca5-b869-ffb7-426b-b49a6782c1c0@amazon.com> <20230208140247.rt62xdtriopfdb4o@quack3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230208140247.rt62xdtriopfdb4o@quack3> X-Stat-Signature: ggnama67a4fx9e5pfqzrc83xhoa93gc3 X-Rspam-User: X-Rspamd-Queue-Id: E587A8000D X-Rspamd-Server: rspam06 X-HE-Tag: 1675885055-693346 X-HE-Meta: U2FsdGVkX1+TQCUWDlWoX7NZIX1hDRylQTAGyL+lpqHfU4/+Mxqt1u/mubL03fBagjlegw+VmuwkjfxkZIlQ3OgXdfZmcW94Er7LAoGMup5/4s0kubYFvi1jbiZy0VC73KvXR/LDziRZYIs8L8zi7F2OoWg0rz5vQv0z0wuASEB+YZRw9DBLAH4TnDBORvRtObeFyLUL/F2lBoZBGL/kDZykbDS3CJdFTedPEdBeIDTjQWt8bvc+Jdh+91w+OzpKuxlsK7GZrx7ZsvZZwqqLHBPDTWhRaK0v0OQVJKV5cjfXC8UrB9uPI7nXS7D9xwkjdd7fjhxHu3zcx8iXV9qICABUiBAxWBXppo6Y+9HNDDeaPoQX1PPNOIBN86jL3Mwgiq/PK5+TLIp3wR28lwWmLrwgajFuvcAOktflMNz7Mn029W6N3hvpnIOGIEVT+vGF2+QstoeKDc2UsFIKbgvPv2PaiUJb2p4ZbqFtgH5JddH2aj5Cmz4olQpayYZthouqAHdNbqzF60ES/gyG4fGRgZ4luRMW0sqGmN/9GI+oNThOpIGnonRCcdobpDxTFWc4owN8QUTCa0QbVY0SmGCsqBTjLQpQyQSTN7t1OqX/r/G7WQXwDOJIA3MqS4kHLnpHKUGBjAcnJYhezPDQmqOrS1ODJ5h5CDpG7e+JDO8hqTLmQbjOlnMJfFWp6DD+ZI46KlABp2lQHsGwbbIgopzev3xMs1w0k4eW7H25bZvE77ss9q1ypi1Kp6fASBxqivCdWcSK8XpQ1uoU8FL2M85MYyroJOjEzWHSYId9qNHN99amjcDPjvbdk/QP+Vp9MEvdoT4qdSM4ssy31n9Fatm4yHwhdfaJRqKibi5ZZ3CfMzUCuzGF2EWgsLZ8Eq3aa54x/BVU2WiRg1di7+e5GBH7BRmehrtjTmSDuZvEmmUAUekm/JRcv0hdcYLs5c3W5OlrYykc5qyZx04SftQNmmP v2PhmvKZ JMuUu8R0rc2VNvNjVKKw8sqHnewRiaxwlUTbddFCh59EvjKfKDs54XAp05g5cnQpdmrZvVdRPNfjfIG6ljyCd99bXWHzCXJAwjDURll5h2/yVxhfXXA1ygfJAcBtlAr+tR1BYyXJW/KvNW7gNSFEkhT7wo/tcA9BUfJH6cOtUjHRWueVXf8IxmFhzt2lcDbdS1Pm2KNzOp2UhY8Uy+p4MyGiAiMXMzeCbwfLqmCphkjJl76c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.002950, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 08, 2023 at 03:02:47PM +0100, Jan Kara wrote: > > I believe its the MySQL database though not so sure. > > Well, in that case I think your MySQL DB is somewhat misconfigured. At > least as far as we have been consulting MySQL / MariaDB developers > regarding benchmarking, they suggested we should configure the database to > use direct IO and increase DB internal buffers instead of relying on > buffered IO and pagecache behavior. And if your fio job is representative > of the IO load the DB really creates, I'd agree that that would be a saner > and likely more performant configuration ;) Could it possibly be Postgres? I happen to know that Amazon RDS and Google Cloud SQL support both MySQL and Postgres, and there are some optimizations that some of us in the Cloud space have been pursuing which are much easier because MySQL uses Direct I/O, but unfortunately Postgres uses buffered I/O and doens't support DIO. :-( - Ted