From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4025E94114 for ; Fri, 6 Oct 2023 20:06:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73B8280017; Fri, 6 Oct 2023 16:06:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EB9480008; Fri, 6 Oct 2023 16:06:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58C7380017; Fri, 6 Oct 2023 16:06:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 49D2E80008 for ; Fri, 6 Oct 2023 16:06:29 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 19FF71CA36F for ; Fri, 6 Oct 2023 20:06:29 +0000 (UTC) X-FDA: 81316118898.06.5BF7842 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id D419B80003 for ; Fri, 6 Oct 2023 20:06:26 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EUXd7fDZ; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2pNqmbK30SfljLBAy/RLPHu30QLqa3v+lOBaHfBrwSc=; b=AszRWHS4unL70vxIMmy8K2vq3Lrw6ubePxqg9sprsJt3sFYJBmsyMoI3NrioidOvl8XYaq wQAZvBuML5HtYEA07lEQPKWrxWj59btR+zLTsErA9FlAANrkHdIP0TboYyRIizkfLzBCiu hwX0Mpig+4EBqqNHfTt1tsQkLPNAyZc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622787; a=rsa-sha256; cv=none; b=PstEDqSoJ4ejCS4yM2ts+dhAr12Rr0LxJKQ2DxbYGwUA+WyW/oK0HDA4IloaOueVISj8Qb GJPj0XQAsPgvx3q/CMW+YUiyxZtaHkXMydWNsZ/84T7aczdHCZ48KHGXidOkqwkWKQc6Oo 6yI40NR/bfp/M7iTWE8tgnfTyr/b7js= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EUXd7fDZ; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696622786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2pNqmbK30SfljLBAy/RLPHu30QLqa3v+lOBaHfBrwSc=; b=EUXd7fDZsGhHjRAaS9XH+d8PMqC5A1Mom6G/wFhDXvvaDj6nEprcEb/Hjp2+7KC7RK7PZf ZJ5s8dNsJImxT/lTyUnKDkqxzjOpKn08ywDSKTW99GXT2TEKSaXFPBg1SysVnQMOAxNFio i6CRQ8HyJob8tp0GX+QHJUKfGUgzB1I= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-126-KYoUSfz3MWidBbwp3NUKWg-1; Fri, 06 Oct 2023 16:06:24 -0400 X-MC-Unique: KYoUSfz3MWidBbwp3NUKWg-1 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4055ce1e8c4so16414165e9.0 for ; Fri, 06 Oct 2023 13:06:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696622783; x=1697227583; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2pNqmbK30SfljLBAy/RLPHu30QLqa3v+lOBaHfBrwSc=; b=LOLr02eg55n6Edv5oFdSTpbv5kj8CFtnnbgcNuJglueZVnNL8wVMWzntWR5nmeKiKq Z/0L+65s5NjC2AkXY1B2eh1DU/ZdQhzpjpSCUwdPHT5Lp34NVMRSRvkZDUL1gywnAWUI 5SX1qbl7RKYCZ/X5Ux0x0SurCIu57kQit7IAHAatW7cm0p8xbgzoB50orVyComTOijw9 ywSugBXClpgGbjyMwEVfGMg2KOY1znA4u2Uf/d+oNUcXsjYz242xg1GRRsEZQEJobrls 5063J3dQNVsPehje/Tsplcy9CdFtp5QmemrLJd1VCi27Zq+MJuuaX1xTIDqloiOu2xkz sk0g== X-Gm-Message-State: AOJu0Yz2hFu/gbVPQpBTLL72p3JBerPzGZnUosh3o/CQhT9o47t4czc/ e/MUTkCMldVYS9WbMvTxtJuv1r7zsaw7b6zWIoDLEtB5zBT8y9is/KS1CaoYdugUGOJ5hvu8yKR jtCIw5iKmiY8= X-Received: by 2002:a05:600c:224d:b0:406:80b4:efd5 with SMTP id a13-20020a05600c224d00b0040680b4efd5mr5910081wmm.14.1696622783354; Fri, 06 Oct 2023 13:06:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFVtdtIXkB99HLEvT5OLCfRrKm1R27RRmahSWx3yYs3WD5IkfkHtlye486IeL3QRpVuJOugng== X-Received: by 2002:a05:600c:224d:b0:406:80b4:efd5 with SMTP id a13-20020a05600c224d00b0040680b4efd5mr5910057wmm.14.1696622782865; Fri, 06 Oct 2023 13:06:22 -0700 (PDT) Received: from ?IPV6:2003:cb:c715:ee00:4e24:cf8e:3de0:8819? (p200300cbc715ee004e24cf8e3de08819.dip0.t-ipconnect.de. [2003:cb:c715:ee00:4e24:cf8e:3de0:8819]) by smtp.gmail.com with ESMTPSA id m10-20020a7bce0a000000b00405953973c3sm6639233wmc.6.2023.10.06.13.06.21 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 Oct 2023 13:06:22 -0700 (PDT) Message-ID: <6d89fdc9-ef55-d44e-bf12-fafff318aef8@redhat.com> Date: Fri, 6 Oct 2023 22:06:21 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 To: Ryan Roberts , Andrew Morton , Matthew Wilcox , Yin Fengwei , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230929114421.3761121-1-ryan.roberts@arm.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v6 0/9] variable-order, large folios for anonymous memory In-Reply-To: <20230929114421.3761121-1-ryan.roberts@arm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D419B80003 X-Rspam-User: X-Stat-Signature: s4m6zr69oiy8wbc3rtaabmcsdrmuji8f X-Rspamd-Server: rspam03 X-HE-Tag: 1696622786-286982 X-HE-Meta: U2FsdGVkX1/zI+CVxERPUYmoXdLJetjHPlompWU6ph11v4CkEDL66xS+b2DkTPtxHRslkAXXWTDMEaQvuT1wdNzrLPgq0WDo/hrJ2heccetGxbDTwFwlDntaGTxpejuq6aGCVabE7q/+GJuL8tjL+YoDEGPdm03L8o/fScL//UNZ0JwjH5oWElSRIAi4IzMCGllRONTH3OerQ/Vrer1Pj2Qd766aa/DQ7qdq+/Qs50h1kVqSFlMQSZWNcH+r3om4/unp11PbNuzp4KPIqXpaQzAoMhcNm+k5MWqpvcztzdFZC9jBMQngYSkeXZzEl40c0k0FtB+sd2Py+dG0u+Jw8JmJIbE9HPBOtVbkbF+Oe2EL85O7E7hl4hQkB9nZAJ8EawIquUTBMCveOg2NaI2krWdTUb7xs/PWQT137wOGZ3fo/pKSkH38HwIs9T7vKkjvpViUSUbbpIjU/zpEarQvt6dH30sNl+upj77V1XnNWYrTwplFhdtfet/4koeEPkN+MBAVzXwb4Qvb3nJ62YSx0lWq49CnxlECKVAWvr8ec1t/CeUJ2h6l0796Hl/a9TdryHVDIokUlWqGL/rUP9azzBzs0Zm6nD6PrRr149moUEC9KT8tn2drX1iKWb4ttox2PxmmvGo8MfZluYZRYK+VKvNKMxiw+vdRsH3O8mijHVH4tHXsjJ2yF0BRMG7znDjxxwWlgpSrFoMSfrCXTGp9/IggD6eEUksl3n0hY/qvPldZkDLsZIH1UEmE34eUh5lB1EnK6SR8emOVjJ7RvRD/+il8SN8MCMKsv30CDz4wY/Oh8uCGIMNzsuApv5Q3qZ1i2Xg1HSNXKQhZ9oZT5G+Kp/TKS7ScctWwug5Y6K9ewSP5o+GCljVDnp9aM2SFS12HnBkcgQxchQEXHLppgUGljy4pvfzE/0aTWEGCEnNTMz0joQAphuuWWvQGJKEVR6O261vBXRH6GRbFYQqrWr3 swYEyJWQ NoFkRF03HzIEhnrIzJbJO529nIvbOPqz2weD8tPisCtt5RGDjOgzz3ySpIhk9776BNTBU/PWG2oNKeux90fSH9AZSCKJZskKH2pyQtrLiESDx0c1g4aDCBI+De3uoLxa3tO9AbzQcoYDgM0skZxAuonNKQuOboo3ff/YZFf0T+Io+5hWpaH0seSgbwVnH8DZLgMb3k0C9WMGEBR5/aPTM56Dz5ueYyffBpxg1Fj81AGe+Di0PKTSYZtPpql+tmVCSU0HxszUvnIh/LEEcJSs2K/QGM/D2TSjALeENK6AMNu/doKIlUapEOFXUCgdwlInI/PnJD6ST/in8xxoxLweju2hFve/d9l1Ex2Ilz7Rzvf0tso8HiJLSMa8voj1Mt8oRrbFAyOux3UWZGzsQqakvf5xZ6BMEGvSHAq03UP49DoclOYqZSTvM1Ul+6VjmLruL0oS95/nvpQgr1xo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 29.09.23 13:44, Ryan Roberts wrote: > Hi All, Let me highlight some core decisions on the things discussed in the previous alignment meetings, and comment on them. > > This is v6 of a series to implement variable order, large folios for anonymous > memory. (previously called "ANON_LARGE_FOLIO", "LARGE_ANON_FOLIO", > "FLEXIBLE_THP", but now exposed as an extension to THP; "small-order THP"). The > objective of this is to improve performance by allocating larger chunks of > memory during anonymous page faults: Change number 1: Let's call these things THP. Fine with me; I previously rooted for that but was told that end users could be confused. I think the important bit is that we don't mess up the stats, and when we talk about THP we default to "PMD-sized THP", unless we explicitly include the other ones. I dislike exposing "orders" to the users, I'm happy to be convinced why I am wrong and it is a good idea. So maybe "Small THP"/"Small-sized THP" is better. Or "Medium-sized THP" -- as said, I think FreeBSD tends to call it "Medium-sized superpages". But what's small/medium/large is debatable. "Small" implies at least that it's smaller than what we used to know, which is a fact. Can we also now use the terminology consistently? (e.g., "variable-order, large folios for anonymous memory" -> "Small-sized anonymous THP", you can just point at the previous patch set name in the cover letter) > > 1) Since SW (the kernel) is dealing with larger chunks of memory than base > pages, there are efficiency savings to be had; fewer page faults, batched PTE > and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel > overhead. This should benefit all architectures. > 2) Since we are now mapping physically contiguous chunks of memory, we can take > advantage of HW TLB compression techniques. A reduction in TLB pressure > speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce > TLB entries; "the contiguous bit" (architectural) and HPA (uarch). > > The major change in this revision is the addition of sysfs controls to allow > this "small-order THP" to be enabled/disabled/configured independently of > PMD-order THP. The approach I've taken differs a bit from previous discussions; > instead of creating a whole new interface ("large_folio"), I'm extending THP. I > personally think this makes things clearer and more extensible. See [6] for > detailed rationale. Change 2: sysfs interface. If we call it THP, it shall go under "/sys/kernel/mm/transparent_hugepage/", I agree. What we expose there and how, is TBD. Again, not a friend of "orders" and bitmaps at all. We can do better if we want to go down that path. Maybe we should take a look at hugetlb, and how they added support for multiple sizes. What *might* make sense could be (depending on which values we actually support!) /sys/kernel/mm/transparent_hugepage/hugepages-64kB/ /sys/kernel/mm/transparent_hugepage/hugepages-128kB/ /sys/kernel/mm/transparent_hugepage/hugepages-256kB/ /sys/kernel/mm/transparent_hugepage/hugepages-512kB/ /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/ /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/ Each one would contain an "enabled" and "defrag" file. We want something minimal first? Start with the "enabled" option. enabled: always [global] madvise never Initially, we would set it for PMD-sized THP to "global" and for everything else to "never". That sounds reasonable at least to me, and we would be using what we learned from THP (as John suggested). That still gives reasonable flexibility without going too wild, and a better IMHO interface. I understand Yu's point about ABI discussions and "0 knobs". I'm happy as long as we can have something that won't hurt us later and still be able to use this in distributions within a reasonable timeframe. Enabling/disabling individual sizes does not sound too restrictive to me. And we could always add an "auto" setting later and default to that with a new kconfig knob. If someone wants to configure it, why not. Let's just prepare a way to to handle this "better" automatically in the future (if ever ...). Change 3: Stats > /proc/meminfo: > Introduce new "AnonHugePteMap" field, which reports the amount of > memory (in KiB) mapped from large folios globally (similar to > AnonHugePages field). AnonHugePages is and remains "PMD-sized THP that is mapped using a PMD", I think we all agree on that. It should have been named "AnonPmdMapped" or "AnonHugePmdMapped", too bad, we can't change that. "AnonHugePteMap" better be "AnonHugePteMapped". But, I wonder if we want to expose this "PteMapped" to user space *at all*. Why should they care if it's PTE mapped? For PMD-sized THP it makes a bit of sense, because !PMD implied !performance, and one might have been able to troubleshoot that somehow. For PTE-mapped, it doesn't make much sense really, they are always PTE-mapped. That also raises the question how you would account a PTE-mapped THP. The hole thing? Only the parts that are mapped? Let's better not go down that path. That leaves the question why we would want to include them here at all in a special PTE-mapped way? Again, let's look at hugetlb: I prepared 1 GiB and one 2 MiB page. HugePages_Total: 1 HugePages_Free: 1 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 1050624 kB -> Only the last one gives the sum, the other stats don't even mention the other ones. [how do we get their stats, if at all?] So maybe, we only want a summary of how many anon huge pages of any size are allocated (independent of the PTE vs. PMD mapping), and some other source to eventually inspect how the different sizes behave. But note that for non-PMD-sized file THP we don't even have special counters! ... so maybe we should also defer any such stats and come up with something uniform for all types of non-PMD-sized THP. Sane discussion applies to all other stats. > > Because we now have runtime enable/disable control, I've removed the compile > time Kconfig switch. It still defaults to runtime-disabled. > > NOTE: These changes should not be merged until the prerequisites are complete. > These are in progress and tracked at [7]. We should probably list them here, and classify which one we see as strict a requirement, which ones might be an optimization. Now, these are just my thoughts, and I'm happy about other thoughts. -- Cheers, David / dhildenb