From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8BD3C4167B for ; Tue, 5 Dec 2023 16:06:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34EBA6B0075; Tue, 5 Dec 2023 11:06:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D7B96B0078; Tue, 5 Dec 2023 11:06:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17BAC6B007B; Tue, 5 Dec 2023 11:06:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C36736B0075 for ; Tue, 5 Dec 2023 11:06:53 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 749B4C013B for ; Tue, 5 Dec 2023 16:06:53 +0000 (UTC) X-FDA: 81533243106.30.0D0794D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id 71192C00E7 for ; Tue, 5 Dec 2023 16:06:18 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JfAYiPlI; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701792379; a=rsa-sha256; cv=none; b=jeAyMZb3cFlWfPTzI2SoB204tXQPvals5pdVXT6XIg7C2LeJgmpfSnpLDlaKqSYmdetsuK 2EtcGFeXr46CTm4CoatFzcsSqu2L0OSwDStFDa77991nZ0NN3b7wLGHRxcB8SpMc+0HgRa ZEJJii2PczyCLHkoqmiYDNR3PEZ5JcM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JfAYiPlI; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701792379; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vZ9hGmOAU2FBZdAcWBT4EroA4dZyM8ao7E1RVyy5W1Q=; b=EtTteSvw/922x+Djm+VR9SFi2xttOaNgdlzk5HKlLhc8k7McJo7wAuMjVuU69rY726yQIA dblBC54t//HknRMBixbEue1tDs+23g1VFOZFqP1s7q9Un6eQSfGyv9kQpsHi1Oz76nsz54 t52xuwsFBdqM2dEsg3Tto6n9YJbSGTI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=vZ9hGmOAU2FBZdAcWBT4EroA4dZyM8ao7E1RVyy5W1Q=; b=JfAYiPlIjHY6yui/t2m/kE0wwI NQ+2jYzHxNa0mPbd0fmcYgXvhv1NLrJ5ufZAOKI5GeGjrSTknq7p2pKaPTDOxqhZGPcCk0hsGAvwN FQ9iDXDki6wTVSFIHpXWiZoF7qh/DD4rSm9SX9yTLrkn/1CjUcf+W1NHn4VSyiizp7h/kk4ZcImh1 60HHfxOOehBijlCbWAJsEoulSTiEIpMiFMX+0aftLr1+veiUBaBSdh82CB3/txeuAhQsjmreIAHe+ 24RZQgCsAM8qSjBcquM5jzjbUwQ1txDmwnq25RI28EikbuYdQhlRmba18Ls5HIJDDhUvx3N9zFvtl gnYjTGNg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rAXvi-0022Oo-57; Tue, 05 Dec 2023 16:05:50 +0000 Date: Tue, 5 Dec 2023 16:05:50 +0000 From: Matthew Wilcox To: Ryan Roberts Cc: John Hubbard , Andrew Morton , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , David Rientjes , Vlastimil Babka , Hugh Dickins , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v6 0/9] variable-order, large folios for anonymous memory Message-ID: References: <20230929114421.3761121-1-ryan.roberts@arm.com> <181a25c2-219e-4af9-9f8e-e5f514bbc4b6@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <181a25c2-219e-4af9-9f8e-e5f514bbc4b6@arm.com> X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 71192C00E7 X-Stat-Signature: d4uczruyajsa96ckiyy98irxufrjbjq7 X-HE-Tag: 1701792378-777059 X-HE-Meta: U2FsdGVkX1/cF6dbdxg0ErYmc+v4TRgw3a8C0JUhQtKjcuAwTIgpXVwPpL/tyYNLStishVGDrO+lMOnF71l1N0qFFYEOjd586TdVsvhKtgPzrCLHETbVZfK6oLK1kQAbEuRVtlJQPfFfZzj8O4+cnvWMdaP9K3u7lOuktO08gAFgm3eZkv7m0Ts4VCd6ZN/wX+hr3VPQJd3aKTfSjM14EHbiIpkP7iLMhVP68D2TcOZbXos5Q0S8YttXbUmDMOhTBS86tU0LkJaWwN7+Hn4cFj+oCmgWqidnJC7tuFezyf/oeiQOD1RR5QkTrsopdF/w6u6W7VVwMR0Jp2kGOH+aihhucmkAYuy+noH2l9cyUHZFNWv9rUG5tazCHLNLZF3PX6oXgopE0WZDPT7Zw2IfdB+Krfy1TGZfkqNFzN+HN4PdWs3sXk4Jc284w5yVOv2BLFPBSc+U0zD5Z/VOQOlO4RQmiS1c/11hRH5V3ZlKQK0BeahXF4qFi36lPg+7uLl8pdFwjPJVVL3cIzhHUOeKU/U3UCw0JAj/95/kLD/YGAhrqo6Ps9ZwFy/tFdcIBqdsOyEbE8/ocjCquavwwMePmKeHl7ie39mc54NsvN0gLW4H/bIdAYAug7g0g65p2WqLbts2ypuZVqB9Jj4av2LhfcK8Eymi1INO61nT1c64C64pxEaxRWtuBcZ45xMbTuk8rKo++hlgRHdkx0LEgOlfdkJ5JFaOXCFfvdcsr/2iTIfjiCjb1mLMel1EA8EKI7UpZEHr0XtlI1Inndrl9iWuXO73WjtUrtcKgEqHTpqcbk5lzRoIWPD2h/BfbTS/+VrXIMWpKThY/fyec6fVdKZQ8O4YCTPK2qmtcgJuS6lbcJe6IQuLLkDkincmdWfpAEKXmUJW2EHNMyaDa80qnHkIOjXhoXlAmZ8Xqwd5JkeVDeXAYLr/4Ijt4KbspRMQEugIJcPvwnvcHnZlXqEMCQ1 zjWZgDyr LJZ6JCc5d0BUIbMzlVTTGdUMD2H/F1KPnwyF/TRDN8n7vYNMtR0v1Nyyo2rIOsK+pmb1EYwnFBsec3gBpFGWjJuiZHc/Lc2yu5tqYR0alOxx1FLg3KwudhO+2w2HXf8xRTiOPrWEdnOmoSk9DkzBaD7CxQdRyeUSVmUeNnM6fjJxNaNfNRgeDSuXsKG5od9xdc0Wj+gorwWU9Eq4BrJcclG77Ew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Nov 14, 2023 at 10:57:07AM +0000, Ryan Roberts wrote: > On 13/11/2023 15:04, Matthew Wilcox wrote: > > On Mon, Nov 13, 2023 at 10:19:48AM +0000, Ryan Roberts wrote: > >> On 13/11/2023 05:18, Matthew Wilcox wrote: > >>> My hope is to abolish the 64kB page size configuration. ie instead of > >>> using the mixture of page sizes that you currently are -- 64k and > >>> 1M (right? Order-0, and order-4) > >> > >> Not quite; the contpte-size for a 64K page size is 2M/order-5. (and yes, it is > >> 64K/order-4 for a 4K page size, and 2M/order-7 for a 16K page size. I agree that > >> intuitively you would expect the order to remain constant, but it doesn't). > >> > >> The "recommend" setting above will actually enable order-3 as well even though > >> there is no HW benefit to this. So the full set of available memory sizes here is: > >> > >> 64K/order-0, 512K/order-3, 2M/order-5, 512M/order-13 > >> > >>> , that 4k, 64k and 2MB (order-0, > >>> order-4 and order-9) will provide better performance. > >>> > >>> Have you run any experiements with a 4kB page size? > >> > >> Agree that would be interesting with 64K small-sized THP enabled. And I'd love > >> to get to a world were we universally deal in variable sized chunks of memory, > >> aligned on 4K boundaries. > >> > >> In my experience though, there are still some performance benefits to 64K base > >> page vs 4K+contpte; the page tables are more cache efficient for the former case > >> - 64K of memory is described by 8 bytes in the former vs 8x16=128 bytes in the > >> latter. In practice the HW will still only read 8 bytes in the latter but that's > >> taking up a full cache line vs the former where a single cache line stores 8x > >> 64K entries. > > > > This is going to depend on your workload though -- if you're using more > > 2MB than 64kB, you get to elide a layer of page table with 4k base, > > rather than taking up 4 cache lines with a 64k base. > > True, but again depending on workload/config, you may have few levels of lookup > for the 64K native case in the first place because you consume more VA bits at > each level. Sorry, missed this email ... let's work it through. With 4k, and a 48-bit VA space, we get 12 bits at the lowest level, then 9 bits each layer, so 4 * 9 + 12 = 48. With a 2MB allocation, we eliminate the bottom layer and examine three cachelines to find it (PGD entry, PUD entry, PMD entry) With 64k, we get 16 bits at the lowest level, then 13 bits each layer, so 3 * 13 + 16 = 55. With a 2MB allocation, we can't eliminate the bottom layer, so we still have to examine three cachelines to find it (PGD, PMD, PTE). If you can fit into a 42-bit address space, you can reduce it by one cache miss, but my impression is that applications which use 21 bits of address space for a single allocation want more address space than your average application.