From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B9D0C25B06 for ; Thu, 11 Aug 2022 23:00:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D48B6B0073; Thu, 11 Aug 2022 19:00:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 184346B0075; Thu, 11 Aug 2022 19:00:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 024768E0001; Thu, 11 Aug 2022 19:00:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E0D826B0073 for ; Thu, 11 Aug 2022 19:00:05 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BB45541901 for ; Thu, 11 Aug 2022 23:00:05 +0000 (UTC) X-FDA: 79788831570.10.17A59D2 Received: from mail-vs1-f48.google.com (mail-vs1-f48.google.com [209.85.217.48]) by imf10.hostedemail.com (Postfix) with ESMTP id 5D8B9C0188 for ; Thu, 11 Aug 2022 23:00:05 +0000 (UTC) Received: by mail-vs1-f48.google.com with SMTP id k2so1006851vsk.8 for ; Thu, 11 Aug 2022 16:00:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc; bh=27PgIpqf/7IrKXemxCZ0QU4yOMIMu6eeW8/O/aowK08=; b=sbHc6PYTA1u3Vdj4XX/DprcMwSPQrCYVachJlRFTyu6wHb27JeSZlni2sXrwGwbsXw zuCHJolXIxsiCycfLERmcvUUPuSg27+fSFe6UIaFPaSChYUPJS8XNBtd+Yd4gWXUAL1f GrkvwwItrIQ2PBUPiWRJvY6GmnPh1OsP1LD0ZXdd4/tbaeU6XScVXpkzFt3Dw4pEYct+ 2zQtSA5zONpzWvS2AQCiXpAwFAzmOOm/xRlgSUEGQ/+uWCaPia+PgNFoNs6O5CGFur14 bKHMcavso3hScUezz0KZzPck1tG7KTHRVzwXULJxak7QFkBEzDxypTRGzGj3Gl5G8RHy Wbtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc; bh=27PgIpqf/7IrKXemxCZ0QU4yOMIMu6eeW8/O/aowK08=; b=RIENlAHAM7hfA8HLJLRuu0f19Nw6fWM7+7CBUNg5i8zc7WmwrFEGbBjVnjTW4NX7NB a6XpA4MM0R2ax0Efnta7yGPDDO2dfB8Ccfj+o76i+/WjxlnpvLt15ijBKfDKfJGpX8Hc kln49yszNj+gFWKDPvj23bcGc6CdoXsmkrCE9NlRLdE/JN0YSBVrNxcbX3pc9F85Oz9u hiWqOu+5eXtjlcVcZMzX8rLxH/y3Ix2MgNxVC6jga34P4fNf4IRdtc96kYJlwdgpJRwI B+/V7jFt9imUe8Zmmzrz/1a4Hr5xGBcSFMv+N4e7FErevOB9zImOA9p2AIaQfZS8J58+ tLWw== X-Gm-Message-State: ACgBeo3yU5JKjy5BhtFM1/LiD2Ro9yWSgSXS/sbJ95t1bA8lWsmma5KQ bfq/EHS3c53Aj51K2yen5jt3B7YJPLcS/GSDLzm/MA== X-Google-Smtp-Source: AA6agR7a2DO3sohlbvRF9brUBNPB25FG43m/e3hhC0YToCad8gXlqFDldrUSQF25PdAXWJI/pgaQP6Z6QM8buxbzwjA= X-Received: by 2002:a05:6102:5094:b0:388:6903:5f09 with SMTP id bl20-20020a056102509400b0038869035f09mr742614vsb.46.1660258804465; Thu, 11 Aug 2022 16:00:04 -0700 (PDT) MIME-Version: 1.0 References: <20220805184016.2926168-1-alexlzhu@fb.com> <0b16dbac6444bfcdfbeb4df4280354839bfe1a8f.camel@fb.com> <1F8B9D85-A735-4832-AD58-CA4BD474248D@fb.com> <868F0874-70E8-4416-B39B-DA74C9D76A40@fb.com> <3195C304-2140-4E5D-890D-AC55653193E5@fb.com> In-Reply-To: From: Yu Zhao Date: Thu, 11 Aug 2022 16:59:28 -0600 Message-ID: Subject: Re: [PATCH v3] mm: add thp_utilization metrics to /proc/thp_utilization To: Yang Shi Cc: "Alex Zhu (Kernel)" , Rik van Riel , Kernel Team , "linux-mm@kvack.org" , "willy@infradead.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" , Ning Zhang , Miaohe Lin Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660258805; a=rsa-sha256; cv=none; b=V+NcCx68qBai3fGFrkg5kbIh6qa0srKd93d8cNTXtLNd+17utPj0AO1tJjmwVgJMLcz6bR 9ex6k0zJKqZF1E7LSpqNZ7pdUJ6S0sFhVFpY/LK1mXPAxopSU/fEjY1TpjdwB7LvJ1tQga 2C/cscGwkFBPm6V8V3SKFax/D0quvoA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sbHc6PYT; spf=pass (imf10.hostedemail.com: domain of yuzhao@google.com designates 209.85.217.48 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660258805; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=27PgIpqf/7IrKXemxCZ0QU4yOMIMu6eeW8/O/aowK08=; b=VEets8Gn+CEWIoWhMS9tRBFzD2CFUhJcEbzwB30t299mOCqqmbdY6nr9p5DMWLWhqKMvDi u78ZIACAgfKGobXug+gvjjmnTLPBnOc780B+3UP4REs8hoAUns0MDqp7o7vsXtzU7AJMe/ GPAEe09oj8cRYxSD2+sOZ+E3gXHxlmI= X-Stat-Signature: dre9bd566yitr13gxoym8gf1qpzi6gnf X-Rspamd-Queue-Id: 5D8B9C0188 X-Rspam-User: X-Rspamd-Server: rspam03 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sbHc6PYT; spf=pass (imf10.hostedemail.com: domain of yuzhao@google.com designates 209.85.217.48 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1660258805-843735 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 11, 2022 at 4:12 PM Yang Shi wrote: > > On Thu, Aug 11, 2022 at 2:55 PM Yu Zhao wrote: > > > > On Thu, Aug 11, 2022 at 1:20 PM Alex Zhu (Kernel) wro= te: > > > > > > Hi Yu, > > > > > > I=E2=80=99ve updated your patch set from last year to work with folio= and am testing it now. The functionality in split_huge_page() is the same = as what I have. Was there any follow up work done later? > > > > Yes, but it won't change the landscape any time soon (see below). So > > please feel free to continue along your current direction. > > > > > If not, I would like to incorporate this into what I have, and then r= esubmit. Will reference the original patchset. We need this functionality f= or the shrinker, but even the changes to split_huge_page() by itself it sho= uld show some performance improvement when used by the existing deferred_sp= lit_huge_page(). > > > > SGTM. Thanks! > > > > A side note: > > > > I'm working on a new mode: THP=3Dauto, meaning the kernel will detect > > internal fragmentation of 2MB compound pages to decide whether to map > > them by PMDs or split them under memory pressure. The general workflow > > of this new mode is as follows. > > I tend to agree that avoiding allocating THP in the first place is the > preferred way to avoid internal fragmentation. But I got some > questions about your design/implementation: > > > > > In the page fault path: > > 1. Compound pages are allocated as usual. > > 2. Each is mapped by 512 consecutive PTEs rather than a PMD. > > 3. There will be more TLB misses but the same number of page faults. > > 4. TLB coalescing can mitigate the performance degradation. > > Why not just allocate base pages in the first place? Khugepaged has > max_pte_none tunable to detect internal fragmentation. If you worry > about zero page, you could add max_pte_zero tunable. > > Or did you investigate whether the new MADV_COLLAPSE may be helpful or > not? It leaves the decision to the userspace. There are two problems we have to workaround. 1. External fragmentation that prevents later compound page allocations. 2. The cost of taking mmap_lock for write. IIRC, the first reference I listed describes the first problem. (It uses a similar reservation technique.) From a very high level, smaller page allocations add more entropy than larger ones and accelerate the system toward equilibrium, in which state the system can't allocate more THPs without exerting additional force (compaction). Reserving compound pages delays the equilibrium whereas MADV_COLLAPSE tries to reverse the equilibrium. The latter has a higher cost. In addition, it needs to take mmap_lock for write. > > In khugepaged: > > 1. Check the dirty bit in the PTEs mapping a compound page, to > > determine its utilization. > > 2. Remap compound pages that meet a certain utilization threshold by > > PMDs in place, i.e., no migrations. > > > > In the reclaim path, e.g., MGLRU page table scanning: > > 1. Decide whether compound pages mapped by PTEs should be split based > > on their utilizations and memory pressure, e.g., reclaim priority. > > 2. Clean subpages should be freed directly after split, rather than swa= pped out. > > > > N.B. > > 1. This workflow relies on the dirty bit rather examining the content o= f a page. > > 2. Sampling can be done by periodically switching between a PMD and > > 512 consecutive PTEs. > > 3. It only needs to hold mmap_lock for read because this special mode > > (512 consecutive PTEs) is not considered the split mode. > > 4. Don't hold your breath :) > > > > Other references: > > 1. https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf > > 2. https://www.usenix.org/system/files/osdi21-hunter.pdf