From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E9E3C433EF for ; Sun, 8 May 2022 14:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D65FA6B0072; Sun, 8 May 2022 10:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D162F6B0073; Sun, 8 May 2022 10:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B69096B0074; Sun, 8 May 2022 10:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A0FBA6B0072 for ; Sun, 8 May 2022 10:47:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7A0AD2F329 for ; Sun, 8 May 2022 14:47:02 +0000 (UTC) X-FDA: 79442853084.12.56A4815 Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by imf01.hostedemail.com (Postfix) with ESMTP id 2BAA140071 for ; Sun, 8 May 2022 14:46:50 +0000 (UTC) Received: from pps.filterd (m0250809.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 248EkhCp009155; Sun, 8 May 2022 07:46:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : cc : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=PPS06212021; bh=czRpmkdLFj2dU8soGIByve4jBKOCypuLbdF86yim2eg=; b=VCPNod0eaXGSgWMVbXd8SeS/AwtSOXBzXuklPMb4NfL0oMFQb+FtwMt/M32LycBGSjxW KYa9oFesdmmyUSju3eDdDf4N0BWakvbrv1b9uFbKPXm/FSutnFyOYlOJJ2oWBUNZu5fM tcXvWcYoywbD+w+e4NpSP7CjOnfNo45krZ8ir/Ch6WCoKRPwgadEUsbeWGPl066JstOp dbdbZbClYhI2r8qfpnQMGZJ+FEig8jPUEMSv0+/BfTs9wPG8uyLSk0RvI2D1rW+CF4Fa yZQOHiTFbRAYZPGHyIUJaigPwfQgbe0iOP8+iD/bpgmwRSlRWp2ncdGLjinVxuKvzhiT Kg== Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2173.outbound.protection.outlook.com [104.47.57.173]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3fwr9k0jmy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 08 May 2022 07:46:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EsZRZtcCCNxrC4HBtMwvuF1DslG5Hftaddh4/eC4kANzCHICMrna8zILBmgjpq6QbxCumXO3NvVu7f3gwjUKwr+HGSqlKEqSD7tyyPvx/IwXuB4CEVnbh/j8qMOT0nuQL5e8W4za9oRKiViUPhwjeX0RXRkZwKzqiL4EHIK5IxvBaTctGH/Q/VCLF2gAN7P95sBUd8R/9WfaGOytVirgLVcNh5fElPUmfqrfkpKjS4iBpRN8zCU0zEFSAz5lu0EW35nDVHF302ooYpweI2OdNOjIzWhVS2VIE6i1Aa/Ol7Y41CgD+efW5xscHneps91PVDCw3MPNAt9wmHpskiclhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=czRpmkdLFj2dU8soGIByve4jBKOCypuLbdF86yim2eg=; b=EFRLLwTEHXbYPL/0NQCgmrGANN4NNdJGhbtwR5BE3NUFxfmAvt8gOfQo3UlSeDSucsKop4I2OODUB2ZfMrultgAJXbvzBEC7jceddlmOPoZQxrc6qDYM7pO5FfQCLB3jp/N7VQ8UCmBxGlCnA8qE4wRFqMh2RAzEeWBl89V/pRnjhhTHxq7K9mMKLXbE/Ol2vLg1iBic2ZUihqLVtVNwzZVt69HUILGJvUQpHOWBYb3qKUIKFMiOWdiyyUc5LLBq7B/uzs5OFRmF9EVwnHU1P5IJ9vFfc6AV8NL6rvNKByhC1KqrEMNysh9lttb+1+KXP+yHmZPVnnBoBMc6F8i0Xg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from PH0PR11MB5192.namprd11.prod.outlook.com (2603:10b6:510:3b::9) by BY5PR11MB4291.namprd11.prod.outlook.com (2603:10b6:a03:1c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Sun, 8 May 2022 14:46:41 +0000 Received: from PH0PR11MB5192.namprd11.prod.outlook.com ([fe80::1d6f:9e73:f47:3ba8]) by PH0PR11MB5192.namprd11.prod.outlook.com ([fe80::1d6f:9e73:f47:3ba8%7]) with mapi id 15.20.5227.023; Sun, 8 May 2022 14:46:40 +0000 From: "Song, Xiongwei" To: "phillip@squashfs.org.uk" , "willy@infradead.org" CC: "linux-mm@kvack.org" , "squashfs-devel@lists.sourceforge.net" Subject: squashfs performance regression and readahea Thread-Topic: squashfs performance regression and readahea Thread-Index: Adhi6Hqd2L+DZdiJQaOdnMUF8OAN3A== Date: Sun, 8 May 2022 14:46:40 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bf8ce95a-aa1b-4f41-f920-08da31018ffe x-ms-traffictypediagnostic: BY5PR11MB4291:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: EWTbQ4EqIDMbv99mzx8cUpGWCGHsiLTJQCpJJkjYPL8qq4izazxahoLuXgDKw/5P86/stufcrdOC10qTthR3U2VG401fYGW/sDsWDVyvJFB2F1f5tGH9lmpjPDfryQepIk02xsw5zwi9emvSYdplj9G2mKW4fDHHZjnycGaHQ8nfVnYjlt+fxF4oG5W5yJInXAUnV7lHg5Yfwtd8/0BgN5GFacl4jsS63zK9Pj6OJpVvxsT8m5vrFQWuE32ybFp4urvzkgpy+QNO+fjQlQ/Iy8QoVLPrBM4sQG5OCW2NNoRkWxPd7SujbQdOofPEbQAmietLu8mETLM4GG8FZpIiwMDHmTzGVZ8Gfkc98j3Q1eCSaz9QgbyoZdm17YpCrEmoHQC7Gi3xXlyPZVPS8fhVPrBFCwoh1NVedK/1qZmT2lk4cPebo513/qQz3aZPiUsSdl1KLCOdbNEAWiRCbw0ZR9Okf1kN3bsHxdSdi3/JzU9oBC2yUEt+BxDHmkJE4FrF85aVBTTqXJSwJfk7hBPbfK8SBecHLpHUf3LVsoGi4GAjXFxtK58gZG1ufipj9BSFDkut4y2aRP/7KkVFLwQ6gbJK8AU8F9j/XsPl6dmP6Qs3UdP7xhZroqUZh91WG0lcwZ2C/XpUeGlm8GoX76JQy/9h2LlPGS6IdMV+RZDj93zhie9fv0S+BhepP8ySF6XmYLUxK/2znS0e/vHzyFjcjdz4VaYIUD68WofFqR0eSuI0gjiV7Kj7wozcpAhhjLOV97N/H7Ahqx6xDdBib64zCwx0c9ZO/Z73wAoZHLIgi5o= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR11MB5192.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(52536014)(508600001)(966005)(64756008)(8936002)(66946007)(76116006)(66476007)(66446008)(66556008)(5660300002)(9686003)(2906002)(38100700002)(38070700005)(86362001)(4326008)(110136005)(8676002)(7696005)(54906003)(55016003)(71200400001)(26005)(33656002)(122000001)(83380400001)(186003)(6506007)(316002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?G90sm4dW01QwXtfS2GjKCccmAQ/SzPrF/U0PGunwMlVGISMeNCDpW4/A48Sc?= =?us-ascii?Q?RsW+euhUCiwfs/xBsCgGa5nqDSY2piP1CbtKphM7Jgi5OlWtwVkfx/wgZ+gJ?= =?us-ascii?Q?hrulD6VmIg0iPr9R7OQRM+4LCWXdNT/gBR0eT/0VhxTwjd9N3Y6XRFQwdgKJ?= =?us-ascii?Q?Lg1fx776N8xBynklrMByFS+3HR/IfA3o5y+PfyJvcixSLKkbMlG6k5/8RKUM?= =?us-ascii?Q?BY3hIg4c7DCemGk5yRHZSqlahADClxR6b96ceDNVX7Hv/3csJt4I6CgQt6gl?= =?us-ascii?Q?hOCAB5Uc2n3pxdLphBdTL9E0q5z6QH8iAUx22cRbeV4S3WIE1eF9Mb/Zvor1?= =?us-ascii?Q?t/Ud4/93wiMeVadNYyrhWvIgGKFbSWZNy3ficonJSYWiSHgSIF3phJ/NK9Sj?= =?us-ascii?Q?Zkbw7fDGorExJ0K9ICPpL72t33Zoi40C0PEl//yv4zUw37lw0hug4Iwa68qT?= =?us-ascii?Q?nMUXoltZW6+iUHmhyDBvDYQYQ3k1xrogXPdMZMWjTZAwMnXwfO+kLvtqYM08?= =?us-ascii?Q?zJ/42kzFWo66JJNsD0y9D3vyFvvPjWrGrBAOOQFwHG/YKoENZwiPLRUul9WX?= =?us-ascii?Q?x6kqbwY3ZO3A5Eb/+SV1cRAnT2LmQVUBUQIRbD67wSLLOGb3Qbw4hzEwu+H9?= =?us-ascii?Q?XWREt+Hckb5/MOvi1nFBWvxSrd1qY+QiSiURPFRoqHU8p7EdiLaa91DuOkQt?= =?us-ascii?Q?TlfL2/Zhiqo0M15JJW146cdD/gwPygmDqexCHMXoXeFEsYI3zLCKwIC6+B21?= =?us-ascii?Q?sCkBnMN5vhPVdgy4MmukAi4NaBV8BNOiskns+U/6Xr5HaE6Lawxv+FpHQJap?= =?us-ascii?Q?L4nN6VoE2SAGZv71cjsKJqQNyJwyJ5dNbb91ozmoh8hHo2N16Rb9SFOoRXTQ?= =?us-ascii?Q?hN45iyPsu5VDj0dtGOLkNiLkq0Bc0OiKyoXr3yLdI8huGLBUrEA11QlcGs6t?= =?us-ascii?Q?hDLfEPx9XNeqI2SZ/RDcU/ihzqYeOhLSwIYY3pZxY75o1JmJRoxSuiFUrci2?= =?us-ascii?Q?Cu1M6krdSiL4f+2olrvrSb+TGIcI1cvG8wY+KAFKC1X/ks33Q50s5TSf47ew?= =?us-ascii?Q?AM1zzNNdXI04sAG5kDTPGe32pANPCP+v5UbSmTtSQ7JD6aQwGfbjKRttk1hN?= =?us-ascii?Q?0tei7xe/PwaO4jZwgFwVBLRNNXDaCpD8n8M3KPQPjA/ha54LNXbupWiR4aqG?= =?us-ascii?Q?exgcqBZY5K9npxSYZQDVOEBQYWPdj8fBJjr0/KuNtmsQkKZEi/p0JDjPfCxh?= =?us-ascii?Q?/NqbTZL8KraQvut5oN+oQKJjtiz0odaHxw+zCDyqVD6Jqwi9Y/axboOAV5CI?= =?us-ascii?Q?KPwicasqVSsmeMVt7w1b0JF7IN29pTfn0+7p1UP8TeuGuslTCbzGQ9rnKv/O?= =?us-ascii?Q?Rcqb23IZma7O1NeGcs5VzfbB3B2xrXFQYl8PcJvu2nAXUUDXwYbDDIGxtieJ?= =?us-ascii?Q?H4BToYoOU80gMoma6xrM5uBDoBVXmXzZ+f/MekLLUW2fVW6azrRPY/NrCe0c?= =?us-ascii?Q?6Ttg5YvAErsh90IjAWKliDVtU6wGH4dhsGuCnl88qASeXQMeL2lde8BgTMYv?= =?us-ascii?Q?gtEySwjwD88tihvPN+bQtS5Rgn2lUY3Tioz9kXSt6HBt/WZ3ZfKnp4xhPXSH?= =?us-ascii?Q?akIU6pTehjMhU6mSZL26FCrKGtdmOrAIQvK2mt511NVGzulbA1qJIKJwf4KX?= =?us-ascii?Q?rZcIqCpgELqYGWdOSeNgULx4umkgJs3NkJpbgD0DUeFABUDPE3t32a7Juxn/?= =?us-ascii?Q?bXxF2Sno3n39F2UAtpvJeMVcLEyYDR4=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5192.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf8ce95a-aa1b-4f41-f920-08da31018ffe X-MS-Exchange-CrossTenant-originalarrivaltime: 08 May 2022 14:46:40.6457 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: IuYtV7PKNuH90Wb7kBSOWHswFAkFHJTizGqg0H7lBX+o4dzyhUsszK9fsTClp+rBpKbBfNzT8+BslXFTiAuJhKpkTtKvp3gYhypQ3clkmv0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4291 X-Proofpoint-GUID: mr9l80J1OKJq0iLeJCqILjcSYnNWqkmW X-Proofpoint-ORIG-GUID: mr9l80J1OKJq0iLeJCqILjcSYnNWqkmW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-08_04,2022-05-06_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 impostorscore=0 spamscore=0 mlxlogscore=389 phishscore=0 malwarescore=0 suspectscore=0 clxscore=1011 mlxscore=0 lowpriorityscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205080098 X-Stat-Signature: cyqqe3rr5eet9taeofe4qjnyoujte8rj X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2BAA140071 X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=windriver.com header.s=PPS06212021 header.b=VCPNod0e; dmarc=pass (policy=none) header.from=windriver.com; spf=pass (imf01.hostedemail.com: domain of "prvs=2127e39853=xiongwei.song@windriver.com" designates 205.220.166.238 as permitted sender) smtp.mailfrom="prvs=2127e39853=xiongwei.song@windriver.com" X-HE-Tag: 1652021210-717337 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, I am facing a performance regression on squashfs. There are many squashfs= =20 partitions on our board. I am doing the operations below on 90 squashfs partitions: "for cnt in $(seq 0 9); do echo 3 > /proc/sys/vm/drop_caches; echo "Loop ${= cnt}:"; time -v find /squashfs/part[0-9][0-9] | xargs -P 24 -i cat {} > /de= v/null 2>/dev/null; echo ""; done" On linux 4.18, I got the elapsed time statistics below with command above(r= un=20 find/xargs/cat commands 10 times): 1:22.80 (1m + 22.80s) 0:59.76 1:01.43 1:02.48 1:03.03 1:02.92 1:03.19 1:03.22 1:03.26 1:03.14 On linux 5.10, huge performance regression: 5:48.69 (5m + 48.69s) 5:52.99 6:06.30 6:01.43 5:50.08 6:26.59 6:09.98 6:04.72 6:05.21 6:21.49 By "git bisect", I found this regression is related to readahead. After re= verting=20 c1f6925e1091 ("mm: put readahead pages in cache earlier") and 8151b4c8bee4 ("mm: add readahead address space operation") on linux 5.10, the performance is improved: 1:37.16 (1m + 37.16s) 1:04.18 1:05.28 1:06.07 1:06.31 1:06.58 1:06.80 1:06.79 1:06.95 1:06.61 Also, I found disabling readahead is helpful with 9eec1d897139 ("squashfs: = provide=20 backing_dev_info in order to disable read-ahead"): 1:06.18 (1m + 6.18s) 1:05.65 1:06.34 1:06.88 1:06.52 1:06.78 1:06.61 1:06.99 1:06.60 1:06.79 I have also tired with the upstream linux 5.18, see the results below: 1:12.82 (1m + 12.82s) 1:07.68 1:08.94 1:09.65 1:09.87 1:10.32 1:10.47 1:10.34 1:10.24 1:10.34 =20 As we can see that even if the readahead disabled, there is still extra 2 ~= 3s overhead than linux 4.18. BTW, the reverted two commits above are from " Change read= ahead API " series, see the following link: https://lore.kernel.org/all/20200414150233.24495-11-willy@infradead.org/T/#= m22d6de881c24057b776758ae8e7f5d54e2db8026 . I would appreciate your comments and inputs. Regards, Xiognwei