From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21FBDC41513 for ; Mon, 16 Oct 2023 10:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 370748D0053; Mon, 16 Oct 2023 06:13:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 320518D0001; Mon, 16 Oct 2023 06:13:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20F858D0053; Mon, 16 Oct 2023 06:13:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 12A3F8D0001 for ; Mon, 16 Oct 2023 06:13:45 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B955F160BA3 for ; Mon, 16 Oct 2023 10:13:44 +0000 (UTC) X-FDA: 81350913168.11.2AD1459 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 6CBD6C0012 for ; Mon, 16 Oct 2023 10:13:42 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iLrlh2Ja; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697451222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dxRH41+dnFDUBr5Humwo2lbbBZQfA7p5P+yHcjzrfH8=; b=Jr0dU3Fp9lHj2L6NRZKtYd70plgjayFFhu1yFh1zwLNY7vEXep1oePMIk1T6yEhPy3ZU4E O5lnP362ntsFCl+KGZy2gsyBNU+bQQy1EbUu8oCzlUn9WScoya+hIh0aqxu/4d32NB9qXn 9LWyKLhL/0TvrSllERrz4GLU39hS34w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iLrlh2Ja; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697451222; a=rsa-sha256; cv=none; b=YlOoXK12MnPDjsrep/O4XawFzMll0JyzLU+sdZxHUK2fW2Oy04V8HQ/4QTpDVqg2eoZjcZ +aAcyRCvQNfwkDO3jRXLBskEVABFQGGT+3Ir6J/K5SMs9wJC2au7FTaarVMpm10ngUGnwB hjBRuSAgw2NgqEERUU4qYbPzDfDfAw8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697451221; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dxRH41+dnFDUBr5Humwo2lbbBZQfA7p5P+yHcjzrfH8=; b=iLrlh2Ja9OQ153rbz5a9TFIMS4EyQNFeLjVj2EcANJFjx18PpjZ0auGLGA3ceIqdzPrByn z6svrt96hZYnLqeBDvD24c9mRqJgCLzmvgqTlGnM9ib89LfpOq+Hq2mHIV9O9TYVvAgn07 RbdrC4FKAHu8SJvdhM9NY+7sMmKPdcE= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-637-6pIEzPBuNea2wc0Q08NZTQ-1; Mon, 16 Oct 2023 06:13:40 -0400 X-MC-Unique: 6pIEzPBuNea2wc0Q08NZTQ-1 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4053d832d51so28475265e9.0 for ; Mon, 16 Oct 2023 03:13:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697451219; x=1698056019; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dxRH41+dnFDUBr5Humwo2lbbBZQfA7p5P+yHcjzrfH8=; b=l23SnZcxC8LFfU4xbF3Hi2SVoA69E2sIwhbR2lhnEUb9MuvSO7dc0NiNV6LZdwIiCG IsNrOn4sG6ntY3QcDlU7IPGCQqoir5DdNhFi7ZzQX1IrFVCiPhgy4vZ/F9D0jpOvEo4d huqy0QQ1GgdR0kCh6GrGPC91P9wC66HQSauG+DfuP+BT6gjV/WDoWxdUMv+jD2l9jLxF BkIw2ct6a/bYrZBJhXLGijwzhdHUOGWWAXHRrl473u58yE1W6j28hzhy3ui+IUEcMm7a EaET849zXZNKnk29A+cAI+qhckCeMJOiKXXXXBTB4BpTa/gEN0JEX54sQ2ZsNA02ANci qkTg== X-Gm-Message-State: AOJu0YyYJ1qCOAGdl5EGLEgHp51zus/rrk4U6uSAhu8X9xDwPlVx+qpL A+KQ6lQ0pazV51tZSrtMQA0TO3/uoPxRIFBxFwfVbsdEQhgGUwUpUnZAloHVnAnDVb5qrBFtnDa k651v8lbrS0U= X-Received: by 2002:a05:600c:3ba1:b0:406:872d:7725 with SMTP id n33-20020a05600c3ba100b00406872d7725mr28451048wms.1.1697451218765; Mon, 16 Oct 2023 03:13:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGqg4QfTqHsvV7pZkkXkam/S/csXbyJ+c0DTeYRzzFfCh03r3PQlZs+TCpWaY/auPpEJslJrg== X-Received: by 2002:a05:600c:3ba1:b0:406:872d:7725 with SMTP id n33-20020a05600c3ba100b00406872d7725mr28451028wms.1.1697451218295; Mon, 16 Oct 2023 03:13:38 -0700 (PDT) Received: from ?IPV6:2003:cb:c73c:9300:8903:bf2e:db72:6527? (p200300cbc73c93008903bf2edb726527.dip0.t-ipconnect.de. [2003:cb:c73c:9300:8903:bf2e:db72:6527]) by smtp.gmail.com with ESMTPSA id az23-20020a05600c601700b00406447b798bsm6791489wmb.37.2023.10.16.03.13.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Oct 2023 03:13:36 -0700 (PDT) Message-ID: <65dbdf2a-9281-a3c3-b7e3-a79c5b60b357@redhat.com> Date: Mon, 16 Oct 2023 12:13:35 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 To: Matthew Wilcox Cc: Naoya Horiguchi , linux-mm@kvack.org, Andrew Morton , "Kirill A. Shutemov" , Mike Kravetz , Miaohe Lin , Vlastimil Babka , Muchun Song , Naoya Horiguchi , linux-kernel@vger.kernel.org, Ryan Roberts , Hugh Dickins References: <20231010142801.3780917-1-naoya.horiguchi@linux.dev> <63d119f7-5adb-861a-00c2-69a92b19ef9b@redhat.com> <20231012150226.GA473412@u2004> <86170ebf-cbe3-1cda-dcb4-87e18695f9cd@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v1 0/5] mm, kpageflags: support folio and fix output for compound pages In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Stat-Signature: pxwcedusnfzfpt6skh4w3o8qo4qm5uqz X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6CBD6C0012 X-HE-Tag: 1697451222-215500 X-HE-Meta: U2FsdGVkX1/KmVdEWt7Hq6P1fnCOhg9cIHZjJ9MezufmY8CcfFJTX9qp14nWp6DbFK1XVtOiEdUjnqOsANclzmHJJxmR881Z1rJD1CGj4Om6L+H3z/285PpaOlJhWB0J5V5kee+3IPwthSZG0/PmZLDsFIbMOvP686QWWpdAE9o1U1tMpzrIaCInqYdwRj9mPCG/UqJF/AHTCIHf7Yt2QnZl/z5YB1LVLqlLw0lhnPWwRG/GyTZGyubjQZjX51+lQ3mGyM3e9bft5yDWx3QKjqIFXdQz3HRsvDhwPa9xbETeMxyydAeia19Rdcy2bvM9uUc7k3SZVy7zS4JPfENnXQUv/FyPkN2aEqPhQOydSh/DNzxMWWjtMAgI8kxl3r+K8F+9uO4C5wmv6Khf7VetonFwRW9sw9Fxqr3h+YU6tCZXEvBU38Gv31ITRUPie7PKnbG3pREpMcBUpNC4UPTDifHjtGxpLS+vWILmDFBhmYtcyEjML/dkox6jLWMqFsbveHFBbP/qp4bP8zJVQ7rnj/r0W4tVZlsS3Vz4/OY7X0tC/c8KxglMRJVgy8+d8JWUC7P4Ch+4CEWsW28JCbtnottSB31VepF4gpTKqLEhcmSLQDbl8StKrN0N6+A1GLfH0Oake1/daRWrLpvJ87NaOgtivbBC/NzvBeDqWzPs1La4lxjgfI+hGdAbZPpsMsWmaxctf2jhUznZR6oN2FNB9hQdOivKnsmfWCWILbv3tBY0zwllgLhnRLBLYs/QLA79gNhmHIPzls8CdkEqV1dPwuEdKjyuqctVRE8q4f8nLBNC4PFIpcQY+Vd6Tuf6syhj3HUAOd4TXsg1syYa9akm+oIWTaareUN+Q8+fThs1blxjT0mwhJE0iPyQRfIe4FGzpnMSJ6YgFdOJltklNC3RPxWNppGt2ecf8OQrV2NSnufbiSUYENFMaUZJDtAOfR8h6NfLqv7qnvLm4hvDnb5 s/j7UAXC eadMjg7QMCT16DcTP72jFmQZojHUVocILvwd6NIRq4t4co9f1ajeSvHnCK9dWy2BMe7YG5DdAgkBm+rWQPnNH6bdIbFjsCTahibCfy56kOKVIati3FCpZd/pnqYueiVq/QC8gKyF/Y0VQxXfzWcxe0hCjE8d0hJMhH1AemX41Trg1S8CIm/QhOEjkVP/zinZwS4Im9+8GGVrOOnZEQcHazLI0qsIYUHT0uC5DvKg4TyN43Is3sxim/dGNYDn88zTNmR4OtklsEAqJ4AbeuwmlNETzCfy+AQk73HrNfpF/yopCgCHTsZJMyJQROC8mO5CO3HQnudhqbGDhEmg+1y0Ox/KOZfZQrxoJ3Iq/ARBdVTYDdX/P5jAGsGDrZGQTLbiMsE49wQMIu3EWZKUf9Fd1KI/FRz9DYGDwFDCR5V6YeTpcAOn7qPQfvOlwb1yhXeG2KMtfttKJXOrfDTzdQ3MHPNYRR5rv7+peb8F+kd0iGclsIGiGaLNQPKIsuOXAXrajwpgB6kPufWgqP1+aa/yKJ6+shImCrUKgSxhWnmtOzDc8stc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >>>> It does sound inconsistent. What exactly do you want to tell user space with >>>> the new flag? >>> >>> The current most problematic behavior is to report folio as thp (order-2 >>> pagecache page is definitely a folio but not a thp), and this is what the >>> new flag is intended to tell. >> >> We are currently considering calling these sub-PMD sized THPs "small-sized >> THP". [1] Arguably, we're starting with the anon part where we won't get >> around exposing them to the user in sysfs. >> >> So I wouldn't immediately say that these things are not THPs. They are not >> PMD-sized THP. A slab/hugetlb is certainly not a thp but a folio. Whereby >> slabs can also be order-0 folios, but hugetlb can't. > > I think this is a mistake. Users expect THPs to be PMD sized. We already > have the term "large folio" in use for file-backed memory; why do we > need to invent a new term for anon large folios? I changed my opinion two times, but I stabilized at "these are just huge pages of different size" when it comes to user-visible features. Handling/calling them folios internally -- especially to abstract the page vs. compound page and how we manage/handle the metadata -- is a reasonable thing to do, because that's what we decided to pass around. For future reference, here is a writeup about my findings and the reason for my opinion: (1) OS-independent concept Ignoring how the OS manages metadata (e.g., "struct page", "struct folio", compound head/tail, memdesc, ...), the common term to describe a "the smallest fixed-length contiguous block of physical memory into which memory pages are mapped by the operating system.["[1] is a page frame -- people usually simplify by dropping the "frame" part, so do I. Larger pages (which we call "huge pages", FreeBSD "superpages", Windows "large pages") can come in different sizes and were traditionally based on architecture support, whereby architectures can support multiple ones [1]; I think what we see is that the OS might use intermediate sizes to manage memory more efficiently, abstracting/evolving that concept from the actual hardware page table mapping granularity. But the foundation is that we are dealing with "blocks of physical memory" in a unit that is larger than the smallest page sizes. Larger pages. [the comment about SGI IRIX on [1] is an interesting read; so are "scattered superpages"[3]] Users learned the difference between a "page" and a "huge page". I'm confident that they can learn the difference between a "traditional huge page" and a "small-sized huge page", just like they did with hugetlb (below). We just have to be careful with memory statistics and to default to the traditional huge pages for now. Slowly, the term "THP" will become more generic. Apart from that, I fail to see the big source of confusion. Note: FreeBSD currently similarly calls these things on arm64 "medium-sized superpages", and did not invent new terms for that so far [2]. (2) hugetlb Traditional huge pages started out to be PMD-sized. Before 2008, we only supported a single huge page size. Ever since, we added support for sizes larger (gigantic) and smaller than that (cont-pte / cont-pmd). So (a) users did not panic because we also supported huge pages that were not PMD-sized; (b) we managed to integrate it into the existing environment, defaulting to the old PMD-sized huge pages towards the user but still providing configuration knobs and (c) it is natural today to have multiple huge page sizes supported in hugetlb. Nowadays, when somebody says that they are using hugetlb huge pages, the first question frequently is "which huge page size?". The same will happen with transparent huge pages I believe. (3) THP preparation for multiple sizes With /sys/kernel/mm/transparent_hugepage/hpage_pmd_size added in 2016, we already provided a way for users to query the PMD size for THP, implying that there might be multiple sizes in the future. Therefore, in commit 49920d28781d, Hugh already envisioned " some transparent support for pud and pgd pages" and ended up calling it "_pmd_size". Turns out, we want smaller THPs first, not larger ones. (4) Metadata management How the OS manages metadata for its memory -- and how it calls the involved datastructures -- is IMHO an implementation detail (an important one regarding performance, robustness and metadata overhead as we learned, though ;) ). We were able to introduce folios without user-visible changes. We should be able to implement memdesc (or memory type hierarchies) without user-visible changes -- except for some interfaces that provide access to bare "struct page" information (classifies as debugging interfaces IMHO). Last but not least, we ended up consistently calling these "larger than a page" things that we map into user space "(transparent) huge page" towards the user in toggles, stats and documentation. Fortunately we didn't use the term "compound page" back then; it would have been a mistake. Regarding the pagecache, we managed to not expose any toggles towards the user, because memory waste can be better controlled. So the term "folio" does not pop up as a toggle in /sys and /proc. t14s: ~ $ find /sys -name "*folio*" 2> /dev/null t14s: ~ $ find /proc -name "*folio*" 2> /dev/null Once we want to remove the (sub)page mapcount, we'll likely have to remove _nr_pages_mapped. To make some workloads that are sensitive to memory consumption [4] play along when not accounting only the actually mapped parts, we might have to introduce other ways to control that, when "/sys/kernel/debug/fault_around_bytes" no longer does the trick. I'm hoping we can still find ways to avoid exposing any toggles for that; we'll see. [1] https://en.wikipedia.org/wiki/Page_(computer_memory) [2] https://www.freebsd.org/status/report-2022-04-2022-06/superpages/ [3] https://ieeexplore.ieee.org/document/6657040/similar#similar [4] https://www.suse.com/support/kb/doc/?id=000019017 > >> Looking at other interfaces, we do expose: >> >> include/uapi/linux/kernel-page-flags.h:#define KPF_COMPOUND_HEAD 15 >> include/uapi/linux/kernel-page-flags.h:#define KPF_COMPOUND_TAIL 16 >> >> So maybe we should just continue talking about compound pages or do we have >> to use both terms here in this interface? > > I don;t know how easy it's going to be to distinguish between a head > and tail page in the Glorious Future once pages and folios are separated. Probably a page-based interface would be the wrong interface for that; fortunately, this interface has a "debugging" smell to it, so we might be able to replace it. -- Cheers, David / dhildenb