From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8E3FC00144 for ; Mon, 1 Aug 2022 04:53:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FF4D8E0002; Mon, 1 Aug 2022 00:53:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AED28E0001; Mon, 1 Aug 2022 00:53:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 475EB8E0002; Mon, 1 Aug 2022 00:53:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 37BAF8E0001 for ; Mon, 1 Aug 2022 00:53:30 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0AE671C609F for ; Mon, 1 Aug 2022 04:53:30 +0000 (UTC) X-FDA: 79749805380.29.FE0AAB7 Received: from p3plwbeout16-05.prod.phx3.secureserver.net (p3plsmtp16-05-2.prod.phx3.secureserver.net [173.201.193.62]) by imf09.hostedemail.com (Postfix) with ESMTP id 44D08140108 for ; Mon, 1 Aug 2022 04:53:29 +0000 (UTC) Received: from mailex.mailcore.me ([94.136.40.143]) by :WBEOUT: with ESMTP id INQioAh4oKlDUINQioorWJ; Sun, 31 Jul 2022 21:53:25 -0700 X-CMAE-Analysis: v=2.4 cv=BZodbph2 c=1 sm=1 tr=0 ts=62e75c46 a=EhJYbXVJKsomWlz4CTV+qA==:117 a=84ok6UeoqCVsigPHarzEiQ==:17 a=ggZhUymU-5wA:10 a=IkcTkHD0fZMA:10 a=biHskzXt2R4A:10 a=pGLkceISAAAA:8 a=M5qS2GdLtPm0FGHqoQ0A:9 a=QEXdDO2ut3YA:10 X-SECURESERVER-ACCT: phillip@squashfs.org.uk X-SID: INQioAh4oKlDU Received: from 82-69-79-175.dsl.in-addr.zen.co.uk ([82.69.79.175] helo=[192.168.178.33]) by smtp02.mailcore.me with esmtpa (Exim 4.94.2) (envelope-from ) id 1oINQh-0000wA-9b; Mon, 01 Aug 2022 05:53:23 +0100 Message-ID: <28fd8f61-8005-96f8-7bfb-4a2519be5f18@squashfs.org.uk> Date: Mon, 1 Aug 2022 05:53:18 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [PATCH v5 3/3] squashfs: implement readahead To: Xiongwei Song Cc: Hsin-Yi Wang , Matthew Wilcox , Xiongwei Song , Marek Szyprowski , Andrew Morton , Zheng Liang , Zhang Yi , Hou Tao , Miao Xie , "linux-mm @ kvack . org" , "squashfs-devel @ lists . sourceforge . net" , Linux Kernel Mailing List , xiaohong.qi@windriver.com References: <20220606150305.1883410-1-hsinyi@chromium.org> <20220606150305.1883410-4-hsinyi@chromium.org> From: Phillip Lougher In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailcore-Auth: 439999529 X-Mailcore-Domain: 1394945 X-123-reg-Authenticated: phillip@squashfs.org.uk X-Originating-IP: 82.69.79.175 X-CMAE-Envelope: MS4xfI4RVxeWUBVO3GBbJENv8CO239lwCGDsYmDwTgv3mfMRzygo5solL/JWhRIEea+NTmtjE7D9EW8oSScRsA3kD7750L5HUgoHW3RPQ9DdTavVpqtyZVpF tAvclnrGG9b1n/R83/gjvP7KHWx3PQKEHcvelVThsqMPB/G/fa+RysVK1gZ7QPjyLW08flDX2OAckWzLTBqV7mvNbrapScdvgh8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659329609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a/or55facFeDmDIb1TYDzZYpLrGtYt8bgXsx5GuUn6Y=; b=O0kc6eVRmU1tsAFB855SLnA5AKI5jxhJx+C43TsSQfL9NWp/FmZ0cy8aqThwoVFnqNq1h9 W9HP1hhsSfIf4x9GuP6322Tq964vyaeMP68hmTOGmi5MjACAyQmgyT7QRfvynEYr/W7ltc Z+mboxSTCBHoxs84aHDIDJ4nmZZzgrg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=none; spf=none (imf09.hostedemail.com: domain of phillip@squashfs.org.uk has no SPF policy when checking 173.201.193.62) smtp.mailfrom=phillip@squashfs.org.uk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659329609; a=rsa-sha256; cv=none; b=y7Z1/UtOXgr7+bkgYWMS7JaonDuxHD25wwLayzkI7lYJP2hU3fNwyFzUYBTWUJiq0W2e4z wC5pgpHic7AArpvYTJmFh51EB4Z+gLLQDJae6SkvrbaUqA47Pz6jsrjLqefeUlvizI3Oqh VUUm3meuTndUldEeuKc9nuvNu7ldKXg= X-Rspamd-Server: rspam02 X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=none; spf=none (imf09.hostedemail.com: domain of phillip@squashfs.org.uk has no SPF policy when checking 173.201.193.62) smtp.mailfrom=phillip@squashfs.org.uk X-Stat-Signature: yeygikxa57ux8yt8iz1xq91tmrq9m4ux X-Rspamd-Queue-Id: 44D08140108 X-HE-Tag: 1659329609-307344 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 29/07/2022 06:22, Xiongwei Song wrote: > Hi Phillip, > > Gentle ping. > > Regards, > Xiongwei > > On Fri, Jul 15, 2022 at 9:45 AM Xiongwei Song wrote: >> >> Please see the test results below, which are from my colleague Xiaohong Qi: >> >> I test file size from 256KB to 5120KB with thread number >> 1,2,4,8,16,24,32(run ten times and get it’s average value). The read >> performance is shown below. The difference of read performance between >> 4.18 kernel and 5.10(with squashfs_readahead() patch v7) seems is >> caused by the files whose size is litter than 256KB. >> >> T1 T2 T4 T8 >> T16 T24 T32 >> All File Size >> 4.18 136.8642 100.479 96.5523 96.1569 96.204 >> 96.0587 96.0519 >> 5.10-v7 138.474 103.1351 99.9192 99.7091 99.7894 >> 100.2034 100.4447 >> Delta 1.6098 2.6561 3.3669 3.5522 >> 3.5854 4.1447 4.3928 To clarify what was mentioned later in the email - these results were obtained using SQUASHFS_DECOMP_MULTI_PERCPU, on a 12 core system? If so these results are unexpected. There is very little extra parallelism shown when increasing the threads. There is about a 36% increase in performance moving from 1 thread to 2 threads, which is about what I expected, but from there on there is almost no parellelism improvement, even though you should have 12 available Squashfs decompressors. This is the results I get on a rather old 4-core X86_64 system using virtualisation, off SSD with a Squashfs filesystem created from a set of Linux kernel repositories and distro root filesystems. So a lot of small files and some larger files. ************************ 1 Thread real 8m4.435s user 4m1.401s sys 2m57.680s 2 Threads real 5m16.647s user 3m16.984s sys 2m35.655s 4 Threads real 3m46.047s user 2m58.669s sys 2m20.193s 8 Threads real 3m0.239s user 2m41.253s sys 2m27.935s 16 Threads real 2m38.329s user 2m34.478s sys 2m26.303s *************************** This is the behaviour I would expect to see, a steadily decreasing overall clock time, as more threads in parallel mean more Squashfs decompressors are used. Due to user-space overheads and context switching, you will generally expect to see a decreasing clock time even after the number of threads is more than the number of cores available. The rule of thumb is always to use at least double the number of real cores. As such your results are confusing, because they max out after only 2 parallel threads. This may indicate there is something wrong somewhere in your system, where I/O is bottlenecking early, or it cannot accomodate multiple parallel reads and it is locking reads out. These results remind me of the old days using rotating media, where there was an expensive disk head SEEK to data blocks. Trying to read multiple files simultaneously was often self-defeating because the extra SEEK time swallowed up any parallelism improvements, leading to negligible, flat and decreasing performance improvement as more threads were added. Of course I doubt seek time is involved here, but, a lot of things can emulate seek time, such as a constant unexpected cost. As this effect is observed with the "original" Squashfs, this is going to be external to Squashfs, and unrelated to the readhead patches. >> >> Fsize < 256KB >> 4.18 21.7949 14.6959 11.639 10.5154 10.14 >> 10.1092 10.1425 >> 5.10-v7 23.8629 16.2483 13.1475 12.3697 12.1985 >> 12.8799 13.3292 >> Delta 2.068 1.5524 1.5085 1.8543 >> 2.0585 2.7707 3.1867 >> This appears to show the readhead patch is performing much worse with files less than 256KB, than larger files. Which would indicate a problem with the readahead patch. But, this may be a symptom of whatever is causing your general lack of parallelism. i.e. external to Squashfs. When read sizes are small, any extra fixed costs loom large in the result because they are a significant proportion of the overall cost. When read sizes are large, any extra fixed costs are a small proportion of the overall cost and show up marginally or not at all in the results. In otherwords, there is already a suspicion there are some unexpected fixed costs to doing I/O, which results in poor parallel performance. These fixed costs if they are worse on the later kernel, will show up here where read sizes are small, and may not show up elsewhere. I have instrumented and profiled the readahead patches on a large number of workloads, with various degrees of parallelism and I have not experienced any unexpected regressions in performance as reported here on small files. This is not to say there isn't an undiscovered issue with the readahead patch, but, I have to say the evidence more points to an issue with your system rather than the readahead patch. What I would do here is first investigate why you apear to have poor parallel I/O scaling. Phillip