From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F3B3C001E0 for ; Thu, 3 Aug 2023 08:21:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF683280218; Thu, 3 Aug 2023 04:21:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA61B2801EB; Thu, 3 Aug 2023 04:21:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C94C3280218; Thu, 3 Aug 2023 04:21:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B99F92801EB for ; Thu, 3 Aug 2023 04:21:20 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7B576803B1 for ; Thu, 3 Aug 2023 08:21:20 +0000 (UTC) X-FDA: 81082098720.18.05F4945 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 7FB891C0026 for ; Thu, 3 Aug 2023 08:21:18 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691050878; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FpnwQ5u5IkE8QK1p7DUGeVCfR4DV9YycNqhwOWy94hU=; b=xqWo+41o39kZH5MDsl6Mc5XKbxoAOODQLqVEgAQFkAGG9c22D6exZ4QqpVIbYN6PdvVw1p 43UzJ1CHg+uDIAPkXId8bWDQvDztdZfTvwthBGki8bQ2ar+xEeFL+DHEvXqGYg+GQmNyB6 VOt6f8XD8gNXvW5bzFIURMwasncJB/8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691050878; a=rsa-sha256; cv=none; b=TUb/m3vpaNj1MqZFkKbulNbJZ+X88CP3CXZD35PtSXyqF7i9ckVhUaeZOjIabmeG1MYmF1 JrR8cgsF5uKVkKLV2mRjtXbLSxroSRmlJ3RC481xajP35j2uopeIOcrvMEVrrdh+7E5vRU WdMdK021uByAn3K6rqZtZ258gJGhIuI= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E15D113E; Thu, 3 Aug 2023 01:22:00 -0700 (PDT) Received: from [10.57.77.90] (unknown [10.57.77.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 12E9D3F5A1; Thu, 3 Aug 2023 01:21:14 -0700 (PDT) Message-ID: <49142e18-fd4e-6487-113a-3112b1c17dbe@arm.com> Date: Thu, 3 Aug 2023 09:21:13 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance To: Yin Fengwei , Yu Zhao Cc: Andrew Morton , Matthew Wilcox , David Hildenbrand , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230726095146.2826796-1-ryan.roberts@arm.com> <20230726095146.2826796-3-ryan.roberts@arm.com> <8c0710e0-a75a-b315-dae1-dd93092e4bd6@arm.com> <4ae53b2a-e069-f579-428d-ac6f744cd19a@intel.com> From: Ryan Roberts In-Reply-To: <4ae53b2a-e069-f579-428d-ac6f744cd19a@intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7FB891C0026 X-Stat-Signature: t5h3crf5i51r94jo4zghnk5obrck1wn4 X-Rspam-User: X-HE-Tag: 1691050878-81433 X-HE-Meta: U2FsdGVkX18HaRDqshTaj2Xw/cVn7gj3etSUjyUNbvgh6fH0x911fW9f5BVjDmAk6RwFJwLbLhtx/Hg/6prfCzLxH1acN2cUVFHVH8uYguJwmc2MAiSHH+Q/Ubh/5R5cY9DuDRNEdVis6Ba8WSIq0g7onko3wsrmDxc9VrifS3y7lNvgYbAcRAJmUAC0g4OEkZxfv56BKFxbVskHfzfq/QBsxd7geQeidQaXn8kiSPTSIKjmMjj3mwn9vFrJTOImjvsznKDiY1fXvBfMKNTkGR0Lo+A7/YBaeQVMtL4CijVRGHDqvkEkG72tf6HoD7ZPwS4Zzth3g7FbwffrVpKtWsUh7OImbcJJWbwXARZpuUpQayelW8MfNEBr13akgszJ3+IipdNLS0+JNEpEOty1oGc1CcMOnG6UhyssfP76Z2F9m+USr4fkR/GCW1v6SwYLJ7e9EEzaKcI6kG2KV6zv5iIXF2s0q6T9MxWkhSCXsYVnVNpKJTICpDvx3kNgijWIjx3QPXuutY2m0ROqxDoLs9LUN3fzFghaPAQy22rdcVgPb7QYYcTFwF5rtNJAU7xbpEFdOPREfsG4frZMt9/CGjZuv6y0rnKBZv+Pi+KGFo6eyUaotHRM7BzqutmkEqgFR/tiJzQ7GSr6y6+rdkcdbCCdJlUIYAeeIa09hhEZoQdfMSp2jdw9E9nG/FlqFCbRGbvHFIqa6NDuc6FTj6NQUTKBLoEclONnvZ9kdAFRnTK4xRjm2kGU0nte8w6J/rrJPuxsgWFRjebifBLi++bruVSAGtv2MiufhJAtXPrAptFTQBn0Y/oxR6h+JmK6cpFrBcLEKsK+om5aTkbk1WPX45QDi5f4XGRjsprh6FO2F+Jp5247/akZVof8vQ6ezIvmUxDPv/ThnLernXGYHRSWAeojXoSMYfLxtPB7mMxOXo6VisLA9gMuP4RVDEd7n2NKiJ1KKIM/+/RotsqNzcC mJ5ODRkz 6Q+Lm9BMU+qZiTHhXyqFt6WlRMduogk3XiA5SGavZzFmQ51nnkskAbmbx0HEC4az96hh4wZy7ceLEkq7y3MleAjbPK/niSe92kl0ov6IY7Oe68a5djVhk/ApuSj6h6bxxYOoZ1m1waP9ZDBSMwR1HuhtYh4vjSpDnC0xrS1abTiEYkyftXkuWg7b3FPJJZNifE+Y+yt104aPAwrkTpGl034kZyFhGqJDulMBg6hf+b9b71ao= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 03/08/2023 09:05, Yin Fengwei wrote: ... >> I've captured run time and peak memory usage, and taken the mean. The stdev for >> the peak memory usage is big-ish, but I'm confident this still captures the >> central tendancy well: >> >> | MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | >> |:-------------------|------------:|------------:|------------:|:------------| >> | 4k | 0.0% | 0.0% | 0.0% | 0.0% | >> | 16k | -3.6% | -26.5% | -0.5% | -0.1% | >> | 32k | -4.8% | -37.4% | -0.6% | -0.1% | >> | 64k | -5.7% | -42.0% | -0.6% | -1.1% | >> | 128k | -5.6% | -42.1% | -0.7% | 1.4% | >> | 256k | -4.9% | -41.9% | -0.4% | 1.9% | > > Here is my test result: > > real user sys > hink-4k: 0% 0% 0% > hink-16K: -3% 0.1% -18.3% > hink-32K: -4% 0.2% -27.2% > hink-64K: -4% 0.5% -31.0% > hink-128K: -4% 0.9% -33.7% > hink-256K: -5% 1% -34.6% > > > I used command: > /usr/bin/time -f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all > to build kernel and collect the real time/user time/kernel time. > /sys/kernel/mm/transparent_hugepage/enabled is "madvise". > Let me know if you have any question about the test. Thanks for doing this! I have a couple of questions: - how many times did you run each test? - how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid. - what does "hink" mean?? > > I also find one strange behavior with this version. It's related with why > I need to set the /sys/kernel/mm/transparent_hugepage/enabled to "madvise". > If it's "never", the large folio is disabled either. > If it's "always", the THP will be active before large folio. So the system is > in the mixed mode. it's not suitable for this test. We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that MADV_NOHUGEPAGE must continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting. My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios. We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought. Anyway, as I said, I'll write it up so we can all systematically discuss. > > So if it's "never", large folio is disabled. But why "madvise" enables large > folio unconditionly? Suppose it's only enabled for the VMA range which user > madvise large folio (or THP)? > > Specific for the hink setting, my understand is that we can't choose it only > by this testing. Other workloads may have different behavior with differnt > hink setting. > > > Regards > Yin, Fengwei >