From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B69C4167B for ; Tue, 5 Dec 2023 20:16:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9CE46B0098; Tue, 5 Dec 2023 15:16:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A26996B009B; Tue, 5 Dec 2023 15:16:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89FCF6B009C; Tue, 5 Dec 2023 15:16:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7523F6B0098 for ; Tue, 5 Dec 2023 15:16:40 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3A554120282 for ; Tue, 5 Dec 2023 20:16:40 +0000 (UTC) X-FDA: 81533872560.24.F460C32 Received: from mail-vk1-f170.google.com (mail-vk1-f170.google.com [209.85.221.170]) by imf12.hostedemail.com (Postfix) with ESMTP id 398AE40027 for ; Tue, 5 Dec 2023 20:16:38 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dk92mdFI; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.170 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701807398; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l9+Wq42pLVrA2d2mGEg2raSlh8pC5hY5dBJM43fq2Fc=; b=Cz1WplZ9p3p9jM8XgMgycGRoFknPHdJQHsu6m7ccBLp9wBLFz3TWorPlvOjxyZZ4lOAohY tVTHDNdEmPDW8RGm4I+hg6QjCG3Pc01b5HbhoctVucL6J7msTybYk1oxHaNjqlLwMnKfXj K0kzm/hDWX49vnVfrWPWJzrOMkq+c94= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701807398; a=rsa-sha256; cv=none; b=XoF0ZoCVMbtItanzFsrMOH49Nj0QIClmb5ifUqNvlykVhHS7IwrpE3AwggC8H2+YLKAhQn WkFrx2aPLGI5voZuLmS9jWQz8HrDibfXlzBeBFycrRk5xunGyEcOT6EVJ8v6m5AX5tP9oM xhSIufAORnOeCUYeJV2u11SN+kB5u5o= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dk92mdFI; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.170 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-vk1-f170.google.com with SMTP id 71dfb90a1353d-4b2d6c6d28dso829795e0c.2 for ; Tue, 05 Dec 2023 12:16:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701807397; x=1702412197; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=l9+Wq42pLVrA2d2mGEg2raSlh8pC5hY5dBJM43fq2Fc=; b=dk92mdFI4DoFUa4muG3CYXnkv0JeaG8pyMuO1VsVrL2UJ7KButRh3OmBWXB7qCFwM0 9NDs8pjqDWw7a3V/vjJ8Jj40iUi3iPT+SaoEgfZ76/2YurEVHk8oCm/AFIxdm+Q3icN+ x6qCoTICXADw/2lbyykZ8e2Esch/rqbQDZeaJZDeMWsjJRMm7lMOOsvqBadglT/nUqcd hwpd1ESz7aJZz+UhlYVf4QbgVtON/ll2RPLQvQpDSq/0FbpMfp1sQ3VijQIJPno9HtFD AF8yjSDLSI4vu4uCSyjYRx8PqBvIag1VSSJ770oCaQS21hGOED70xx4VgIvwC1wYLJU9 b8dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701807397; x=1702412197; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l9+Wq42pLVrA2d2mGEg2raSlh8pC5hY5dBJM43fq2Fc=; b=SFM93nwV2Ru9RsEqacG1lE+4b71bEFB0teb6OFfoMHLEnbzqbZfXHA3HBw3DmIHE58 O+czAMZgA4MMwlk8FKktAiB3oNyCyjgwRJPf9xMmHvuvXAM7R/LjKOxGMo7LxjpX0pJ+ oV6m25dtQEc5rN+YQR1ywnPFjWvXE7Kw9PVjd4VEGbAsmdDTq5cBYNa6ibDryMfV1T+L j5NPb0jXz0Q/PXcRvLDVEGHcOdZbw8LEUUZ6iM6OoEk1C7h7PizvzaJiQNpiTO548muX i5mVNLcoYOMWbVj+s0nMTfb2Yf62ye5/f9xH8gZzjFuPRNmjOci+YPsBfHJ4kjMmQSx9 yDfg== X-Gm-Message-State: AOJu0YzmQEejpGULMup7LjDXdAOFYD49HO7itQkYFerBsbZi7AWKDuOw F26WUY+RLnys8Uvg5pkXs4QSOUh5zHNtvwshMMM= X-Google-Smtp-Source: AGHT+IFsG1wrOHB4MUR7FM6YXg7XOf4tAZGac+r8z69FHrn5twCXmzStWUF4eY6A2G/3TIl1XlHkjQOQEk5gmtMnfbc= X-Received: by 2002:a1f:e301:0:b0:4b2:7749:aa96 with SMTP id a1-20020a1fe301000000b004b27749aa96mr3434041vkh.11.1701807397152; Tue, 05 Dec 2023 12:16:37 -0800 (PST) MIME-Version: 1.0 References: <20231204102027.57185-1-ryan.roberts@arm.com> <20231204102027.57185-5-ryan.roberts@arm.com> <5216caaf-1fcf-4715-99c3-521e2a1cc756@arm.com> In-Reply-To: <5216caaf-1fcf-4715-99c3-521e2a1cc756@arm.com> From: Barry Song <21cnbao@gmail.com> Date: Wed, 6 Dec 2023 09:16:26 +1300 Message-ID: Subject: Re: [PATCH v8 04/10] mm: thp: Support allocation of anonymous multi-size THP To: Ryan Roberts Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang , Alistair Popple , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: ya4gpgg3cue56mquopxow4hs3p7oucao X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 398AE40027 X-Rspam-User: X-HE-Tag: 1701807398-733922 X-HE-Meta: U2FsdGVkX18OmSgFFRpIOaFAeEDpME+PqjXMWIMdA6SSySpZZ/FHx8+lWHZGCFSMxZbSplcNAQfaKxOho0zn1Nnf8Vd5yf2aVihDvTWsoV8grf9qew7bY8O09v7Lhu5zfMwO6p06W4oFQfsRu9thP+RHr9ZuDePY4h5YX2hNUxSdrfVJsByocyq7vvX8CzTNlCZc3BcKAzz2Tsf/Aq62N1Y7BjInUJkruyiowWTawGcFl8686udQISsAKOXnIji7d0ZeSzCZ9bSScJRppweEKh6cH9Uv0cBtQo9i+3o3dqvooi9glbsoTb56fjR5N95cbZgPviIvNH7ihzjRh48+WsLX/+ISjFtjT70RseKc4GyXHWm4jTGwm/MFaQnLKHefPAE4J86WTbAlhJiGd18CMuNa7NxBJ5Rm+KfpnH2v61PveU0Q3RN11zi3zYYS29m1T6SeBj6Z7yhhsEKy+SMxQazlR02jh53ce0zLsnWshJvijmkV9IMQY4YThm/6wLOmNVioBeXT1D2leZFBx8JIEQUrOsj9AZdyE8zxmKpEItZEs0YNLsXfJXYzG38FNV5jmm5Yhkr3rfCn7SiePfxlmlsgjjkzHX5prDAgreNXrpIpp73ujMmHploIxkXS+UNPSSj5iMktlRiFPf42YFEgJUqYb4aUbxRHkvwEwCLavmt7xygf8j4WtabY3N4wNQDxefxm5fkQ0sDDXwLbtLApgbpX+WTAx4/URr3eJr0fwur3JJHL0/yQoPCGBHFv3hPx9AhA3fzsbNMmo0oQAE+V432TcuaPA+US5juLBX/4/I3bvEb8MxjhVscFQ/rct+yTe6oLscSoTkeVNqBm/JT3Ck7vzzp78d3xH6KVsAy17XNpfPP6728iaL2M0CmCUjXKTqI01TmieWiwMcjECFiQcLfJ0EJL/fBFkQ/6IrQ2kEsJ09qTrb5YgCZcWKtG93Ku1l8FXSbX6lE6+o5KYwR /jcaqW0k r9QSLVZzTCrn5nYBXekHyYeArONvhlwIlFn+4mynnzUP8YBbOmZMCAjQGd0QYpw91u3uDsXmjrR0CjHe/C9eHlQu+7iT44oqsW2tT6CcaSJFJzuynxMeaJmblZ6CFL09TtaV+M8ZxMUpHrtF3+3/yWYIdX4UXEm8CqZxxTyLzk2O+zSr0zP/rehHLsbdmDue4EsNbDm46WVeu59fh9KseZct2BNS3WjGeelXV9NUMPcm/mYuK0E8Slq1AOxZVKqTgcic3vDDfvVjkVB0YxXgsRkBZq1k7vSZ0Vh5yt2AA9OEy5YNHjIfb4gys6zWWbZYnUzSma0mGzR6SZCEWtlQFmBjr9MmKL7VVGhH4x7Q1DdMy4hQxDttivUvlT82MgFcArwDBbrCOKEoF6u457GQcADV5J883i1fhcnuI+aM3/CxQ2mTNCwXTgml3lw+f4lQQp6EbkywbF8nhLD4pWR8C8AVdNRHCXPWnrPKhcv4/9Ac7EkJ8XvoXQ+hPpPyXeKoffsdv+S7DHEM9qXvbg+g07ENzl8r7Dy1nUYTPSFL5EJWWmjG552Y7NXu2tqXKWf81isYRMv9YsGjJlfJPytKaDptPAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Dec 5, 2023 at 11:48=E2=80=AFPM Ryan Roberts = wrote: > > On 05/12/2023 01:24, Barry Song wrote: > > On Tue, Dec 5, 2023 at 9:15=E2=80=AFAM Barry Song <21cnbao@gmail.com> w= rote: > >> > >> On Mon, Dec 4, 2023 at 6:21=E2=80=AFPM Ryan Roberts wrote: > >>> > >>> Introduce the logic to allow THP to be configured (through the new sy= sfs > >>> interface we just added) to allocate large folios to back anonymous > >>> memory, which are larger than the base page size but smaller than > >>> PMD-size. We call this new THP extension "multi-size THP" (mTHP). > >>> > >>> mTHP continues to be PTE-mapped, but in many cases can still provide > >>> similar benefits to traditional PMD-sized THP: Page faults are > >>> significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending o= n > >>> the configured order), but latency spikes are much less prominent > >>> because the size of each page isn't as huge as the PMD-sized variant = and > >>> there is less memory to clear in each page fault. The number of per-p= age > >>> operations (e.g. ref counting, rmap management, lru list management) = are > >>> also significantly reduced since those ops now become per-folio. > >>> > >>> Some architectures also employ TLB compression mechanisms to squeeze > >>> more entries in when a set of PTEs are virtually and physically > >>> contiguous and approporiately aligned. In this case, TLB misses will > >>> occur less often. > >>> > >>> The new behaviour is disabled by default, but can be enabled at runti= me > >>> by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabl= ed > >>> (see documentation in previous commit). The long term aim is to chang= e > >>> the default to include suitable lower orders, but there are some risk= s > >>> around internal fragmentation that need to be better understood first= . > >>> > >>> Signed-off-by: Ryan Roberts > >>> --- > >>> include/linux/huge_mm.h | 6 ++- > >>> mm/memory.c | 106 ++++++++++++++++++++++++++++++++++++--= -- > >>> 2 files changed, 101 insertions(+), 11 deletions(-) > >>> > >>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > >>> index bd0eadd3befb..91a53b9835a4 100644 > >>> --- a/include/linux/huge_mm.h > >>> +++ b/include/linux/huge_mm.h > >>> @@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr; > >>> #define HPAGE_PMD_NR (1< >>> > >>> /* > >>> - * Mask of all large folio orders supported for anonymous THP. > >>> + * Mask of all large folio orders supported for anonymous THP; all o= rders up to > >>> + * and including PMD_ORDER, except order-0 (which is not "huge") and= order-1 > >>> + * (which is a limitation of the THP implementation). > >>> */ > >>> -#define THP_ORDERS_ALL_ANON BIT(PMD_ORDER) > >>> +#define THP_ORDERS_ALL_ANON ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) = | BIT(1))) > >>> > >>> /* > >>> * Mask of all large folio orders supported for file THP. > >>> diff --git a/mm/memory.c b/mm/memory.c > >>> index 3ceeb0f45bf5..bf7e93813018 100644 > >>> --- a/mm/memory.c > >>> +++ b/mm/memory.c > >>> @@ -4125,6 +4125,84 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > >>> return ret; > >>> } > >>> > >>> +static bool pte_range_none(pte_t *pte, int nr_pages) > >>> +{ > >>> + int i; > >>> + > >>> + for (i =3D 0; i < nr_pages; i++) { > >>> + if (!pte_none(ptep_get_lockless(pte + i))) > >>> + return false; > >>> + } > >>> + > >>> + return true; > >>> +} > >>> + > >>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > >>> +static struct folio *alloc_anon_folio(struct vm_fault *vmf) > >>> +{ > >>> + gfp_t gfp; > >>> + pte_t *pte; > >>> + unsigned long addr; > >>> + struct folio *folio; > >>> + struct vm_area_struct *vma =3D vmf->vma; > >>> + unsigned long orders; > >>> + int order; > >>> + > >>> + /* > >>> + * If uffd is active for the vma we need per-page fault fidel= ity to > >>> + * maintain the uffd semantics. > >>> + */ > >>> + if (userfaultfd_armed(vma)) > >>> + goto fallback; > >>> + > >>> + /* > >>> + * Get a list of all the (large) orders below PMD_ORDER that = are enabled > >>> + * for this vma. Then filter out the orders that can't be all= ocated over > >>> + * the faulting address and still be fully contained in the v= ma. > >>> + */ > >>> + orders =3D thp_vma_allowable_orders(vma, vma->vm_flags, false= , true, true, > >>> + BIT(PMD_ORDER) - 1); > >>> + orders =3D thp_vma_suitable_orders(vma, vmf->address, orders)= ; > >>> + > >>> + if (!orders) > >>> + goto fallback; > >>> + > >>> + pte =3D pte_offset_map(vmf->pmd, vmf->address & PMD_MASK); > >>> + if (!pte) > >>> + return ERR_PTR(-EAGAIN); > >>> + > >>> + order =3D first_order(orders); > >>> + while (orders) { > >>> + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order)= ; > >>> + vmf->pte =3D pte + pte_index(addr); > >>> + if (pte_range_none(vmf->pte, 1 << order)) > >>> + break; > >>> + order =3D next_order(&orders, order); > >>> + } > >>> + > >>> + vmf->pte =3D NULL; > >>> + pte_unmap(pte); > >>> + > >>> + gfp =3D vma_thp_gfp_mask(vma); > >>> + > >>> + while (orders) { > >>> + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order)= ; > >>> + folio =3D vma_alloc_folio(gfp, order, vma, addr, true= ); > >>> + if (folio) { > >>> + clear_huge_page(&folio->page, addr, 1 << orde= r); > >> > >> Minor. > >> > >> Do we have to constantly clear a huge page? Is it possible to let > >> post_alloc_hook() > >> finish this job by using __GFP_ZERO/__GFP_ZEROTAGS as > >> vma_alloc_zeroed_movable_folio() is doing? > > I'm currently following the same allocation pattern as is done for PMD-si= zed > THP. In earlier versions of this patch I was trying to be smarter and use= the > __GFP_ZERO/__GFP_ZEROTAGS as you suggest, but I was advised to keep it si= mple > and follow the existing pattern. > > I have a vague recollection __GFP_ZERO is not preferred for large folios = because > of some issue with virtually indexed caches? (Matthew: did I see you ment= ion > that in some other context?) > > That said, I wasn't aware that Android ships with > CONFIG_INIT_ON_ALLOC_DEFAULT_ON (I thought it was only used as a debug op= tion), > so I can see the potential for some overhead reduction here. > > Options: > > 1) leave it as is and accept the duplicated clearing > 2) Pass __GFP_ZERO and remove clear_huge_page() > 3) define __GFP_SKIP_ZERO even when kasan is not enabled and pass it dow= n so > clear_huge_page() is the only clear > 4) make clear_huge_page() conditional on !want_init_on_alloc() > > I prefer option 4. What do you think? either 1 and 4 is ok to me if we will finally remove this duplicated clear_huge_page on top. 4 is even better as it can at least temporarily resolve the problem. in Android gki_defconfig, https://android.googlesource.com/kernel/common/+/refs/heads/android14-6.1-l= ts/arch/arm64/configs/gki_defconfig Android always has the below, CONFIG_INIT_ON_ALLOC_DEFAULT_ON=3Dy here is some explanation for the reason, https://source.android.com/docs/security/test/memory-safety/zero-initialize= d-memory > > As an aside, I've also noticed that clear_huge_page() should take vmf->ad= dress > so that it clears the faulting page last to keep the cache hot. If we dec= ide on > an option that keeps clear_huge_page(), I'll also make that change. > > Thanks, > Ryan > > >> Thanks Barry