From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 535E4C38145 for ; Tue, 6 Sep 2022 20:28:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC0198D0009; Tue, 6 Sep 2022 16:28:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6E488D0002; Tue, 6 Sep 2022 16:28:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A36078D0009; Tue, 6 Sep 2022 16:28:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 913DB8D0002 for ; Tue, 6 Sep 2022 16:28:03 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6053AA0A65 for ; Tue, 6 Sep 2022 20:28:03 +0000 (UTC) X-FDA: 79882797246.02.75EDE6A Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by imf22.hostedemail.com (Postfix) with ESMTP id 1929CC0084 for ; Tue, 6 Sep 2022 20:28:02 +0000 (UTC) Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1225219ee46so31149706fac.2 for ; Tue, 06 Sep 2022 13:28:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=9CzL4WHDGnelFWaIyimDlFqmrZG9xxxcC/rKl+Gm2Ms=; b=IXma1HNcKU7F9Jdd4eiegCsdsCFpbSSW56I2gWI1KGpM8c0XCbHQi/8JOoogLLJo5m esDS7BEUtm7V8ByhcV/4vdixWDsD6u3DcUgsUYChLF+R+lnWOmeBr66JURNBXrUXORCe l4TbYkLwxCoh670rZYzxF4xDMDhI4cHEa4VNOW3lHxmFWTaUryiMJzRFiP1++M4JhXdN sxIY3O41hg71D+pqGGwovQh5i5N/EAJR5WKgkmYN0OdAlUdOHC0JcgVSB8izfdhNVTz8 5B25LEhb+rWBTk7XxgNpuXx1hbrGBXrn/At06Q8nadupx5i7WM/sOQL7DKTo5wCVX+tI mH+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=9CzL4WHDGnelFWaIyimDlFqmrZG9xxxcC/rKl+Gm2Ms=; b=hPe2spxk/C8zNjm3oH41G7LmU9tpekTrSM/N01/Of8mK+3SDOkd+zWfIOjMo7XNmS1 oMXUePYwvUP5ZEQWEcYq5ThkJK2by+aHdKnr6lanUKqVfpcPwEEtmd2myaB752Go/bQe bahw136y0n6OygiKCQ2905hZ1R2MhS21TG6CgIwH1WSJdVE2ooaKRK82LAqFVN+ddaa6 hUsOHVwPWE9SxVxyWwleADsm2fQlPcwuD0l1l49toxPp3eJOU73WGcQcJ4bnD7p+z3KG P6n4VpAOt6OLTXgtkOGFwgUHhtWlLXV1DWBtJLrSGtSzF00v+dN/9/4NouKzervXFKsQ mvgg== X-Gm-Message-State: ACgBeo0nU85Ov81wuhlgwljSePGSyfPzqHu7rq+N9JHvln6ac3U1cZiY 6HnQUAATFqgvKbhCuLiYIwzx8Q+QzZjbloCd/MxA+Q== X-Google-Smtp-Source: AA6agR4u+YMdlebZPapS1COOH3iJGf533O/b8a8ZQ985Ia9TGclK0yXYrWTVynRNO+rTsDSd0eFN6bW5TW79mXqulFs= X-Received: by 2002:a05:6870:c1d4:b0:127:9423:22b8 with SMTP id i20-20020a056870c1d400b00127942322b8mr19982oad.123.1662496081923; Tue, 06 Sep 2022 13:28:01 -0700 (PDT) MIME-Version: 1.0 References: <20220903021221.1130021-1-zokeefe@google.com> In-Reply-To: From: "Zach O'Keefe" Date: Tue, 6 Sep 2022 13:27:25 -0700 Message-ID: Subject: Re: [PATCH mm-unstable] mm/khugepaged: fix struct collapse_control load_node definition To: Yang Shi Cc: Andrew Morton , linux-mm@kvack.org, kbuild-all@lists.01.org Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IXma1HNc; spf=pass (imf22.hostedemail.com: domain of zokeefe@google.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662496083; a=rsa-sha256; cv=none; b=SBB9RZZZNg3y70AcVlsMF7jFxJjylXw9051uAy3beAcQJMpL5wI467CMR8eXRhefe89l8J cIVXv0F4esDeNhqq0LL4hBzCig4LOGiMu8O5VD3G2lciUVLXpAzooZK3Xz6r7he03NiGt7 eBV55hTgD8m++1h0eOrZaMvGSJtVLTk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662496083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9CzL4WHDGnelFWaIyimDlFqmrZG9xxxcC/rKl+Gm2Ms=; b=zewomkNhjiVNXLQASeduGz+Zt9OReLiMrJqw55xGtG4PmQ6bj/C8FYdFx6M1MPjnJFVv3R Eh6Xg2nMdcdFjA0T9ArJzEoI7vbjKmlG8fgTnu4UwN3jsv4C6A0qxclBE7L4lIv+xKoN0o 3izEUCag48/EZHCfnPg5yfKHDdbZaFQ= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IXma1HNc; spf=pass (imf22.hostedemail.com: domain of zokeefe@google.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: q1mez6soksogwu5xnxepcgyyuawtxaem X-Rspamd-Queue-Id: 1929CC0084 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1662496082-442659 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 6, 2022 at 1:04 PM Yang Shi wrote: > > On Tue, Sep 6, 2022 at 12:14 PM Zach O'Keefe wrote: > > > > On Tue, Sep 6, 2022 at 11:46 AM Yang Shi wrote: > > > > > > On Fri, Sep 2, 2022 at 7:12 PM Zach O'Keefe wrote: > > > > > > > > HPAGE_PMD_ORDER is used to determine a minimal amount of bits needed to store > > > > per-node counters in struct collapse_control, defining the array to be either a > > > > u16[HPAGE_PMD_ORDER] or u32[HPAGE_PMD_ORDER]. > > > > > > > > For some architectures/configs, HPAGE_PMD_ORDER isn't resolved at preprocessor > > > > time (suitable for resolving preprocessor directives) but rather expands into a > > > > compiler built-in that is resolved at compile-time. E.g. for 32-bit MIPS (see > > > > link for full config): > > > > > > > > ---8<-- > > > > >> arch/mips/include/asm/pgtable.h:238:26: warning: "__builtin_ffs" is not defined, evaluates to 0 [-Wundef] > > > > 238 | #define PTE_T_LOG2 (__builtin_ffs(sizeof(pte_t)) - 1) > > > > | ^~~~~~~~~~~~~ > > > > arch/mips/include/asm/pgtable-32.h:65:43: note: in expansion of macro 'PTE_T_LOG2' > > > > 65 | # define PGDIR_SHIFT (2 * PAGE_SHIFT - PTE_T_LOG2 - 1) > > > > | ^~~~~~~~~~ > > > > include/asm-generic/pgtable-nop4d.h:11:33: note: in expansion of macro 'PGDIR_SHIFT' > > > > 11 | #define P4D_SHIFT PGDIR_SHIFT > > > > | ^~~~~~~~~~~ > > > > include/asm-generic/pgtable-nopud.h:18:25: note: in expansion of macro 'P4D_SHIFT' > > > > 18 | #define PUD_SHIFT P4D_SHIFT > > > > | ^~~~~~~~~ > > > > include/asm-generic/pgtable-nopmd.h:20:25: note: in expansion of macro 'PUD_SHIFT' > > > > 20 | #define PMD_SHIFT PUD_SHIFT > > > > | ^~~~~~~~~ > > > > include/linux/huge_mm.h:109:25: note: in expansion of macro 'PMD_SHIFT' > > > > 109 | #define HPAGE_PMD_SHIFT PMD_SHIFT > > > > | ^~~~~~~~~ > > > > include/linux/huge_mm.h:105:26: note: in expansion of macro 'HPAGE_PMD_SHIFT' > > > > 105 | #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) > > > > | ^~~~~~~~~~~~~~~ > > > > mm/khugepaged.c:90:5: note: in expansion of macro 'HPAGE_PMD_ORDER' > > > > 90 | #if HPAGE_PMD_ORDER < 16 > > > > | ^~~~~~~~~~~~~~~ > > > > >> arch/mips/include/asm/pgtable.h:238:39: error: missing binary operator before token "(" > > > > 238 | #define PTE_T_LOG2 (__builtin_ffs(sizeof(pte_t)) - 1) > > > > | ^ > > > > arch/mips/include/asm/pgtable-32.h:65:43: note: in expansion of macro 'PTE_T_LOG2' > > > > 65 | # define PGDIR_SHIFT (2 * PAGE_SHIFT - PTE_T_LOG2 - 1) > > > > | ^~~~~~~~~~ > > > > include/asm-generic/pgtable-nop4d.h:11:33: note: in expansion of macro 'PGDIR_SHIFT' > > > > 11 | #define P4D_SHIFT PGDIR_SHIFT > > > > | ^~~~~~~~~~~ > > > > include/asm-generic/pgtable-nopud.h:18:25: note: in expansion of macro 'P4D_SHIFT' > > > > 18 | #define PUD_SHIFT P4D_SHIFT > > > > | ^~~~~~~~~ > > > > include/asm-generic/pgtable-nopmd.h:20:25: note: in expansion of macro 'PUD_SHIFT' > > > > 20 | #define PMD_SHIFT PUD_SHIFT > > > > | ^~~~~~~~~ > > > > include/linux/huge_mm.h:109:25: note: in expansion of macro 'PMD_SHIFT' > > > > 109 | #define HPAGE_PMD_SHIFT PMD_SHIFT > > > > | ^~~~~~~~~ > > > > include/linux/huge_mm.h:105:26: note: in expansion of macro 'HPAGE_PMD_SHIFT' > > > > 105 | #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) > > > > | ^~~~~~~~~~~~~~~ > > > > mm/khugepaged.c:90:5: note: in expansion of macro 'HPAGE_PMD_ORDER' > > > > 90 | #if HPAGE_PMD_ORDER < 16 > > > > | ^~~~~~~~~~~~~~~ > > > > ---8<-- > > > > > > > > load_node is already special-cased for CONFIG_PPC64 in > > > > "mm-khugepaged-add-struct-collapse_control-fix-fix", and we could continue down > > > > that path by #define'ing away edge cases, but I think at this point it easier > > > > to just let load_node be an array of u32s. If it's ever an issue, we can > > > > revisit. > > > > > > > > This essentially rolls back both: > > > > > > > > "mm/khugepaged: use minimal bits to store num page < HPAGE_PMD_NR" > > > > "mm-khugepaged-add-struct-collapse_control-fix-fix" > > > > > > > > (though we'll leave it as a u32 vs an int). > > > > > > > > Link: https://lore.kernel.org/linux-mm/202209021349.F73i5d6X-lkp@intel.com/ > > > > Fixes: 5f7bb00bd9cd ("mm/khugepaged: use minimal bits to store num page < HPAGE_PMD_NR") > > > > Fixes: 03053c821af2 ("mm-khugepaged-add-struct-collapse_control-fix-fix") > > > > Signed-off-by: Zach O'Keefe > > > > --- > > > > mm/khugepaged.c | 7 +------ > > > > 1 file changed, 1 insertion(+), 6 deletions(-) > > > > > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > > index af3b07eb2389..29142a35d24c 100644 > > > > --- a/mm/khugepaged.c > > > > +++ b/mm/khugepaged.c > > > > @@ -93,13 +93,8 @@ struct collapse_control { > > > > bool is_khugepaged; > > > > > > > > /* Num pages scanned per node */ > > > > -#if defined(CONFIG_PPC64) > > > > u32 node_load[MAX_NUMNODES]; > > > > -#elif HPAGE_PMD_ORDER < 16 > > > > - u16 node_load[MAX_NUMNODES]; > > > > -#else > > > > - u32 node_load[MAX_NUMNODES]; > > > > -#endif > > > > > > I recall we need this to circumvent some build time warning about > > > stack usage? Or I misremember? > > > > > > > Hey Yang, thanks for keeping a look out here, and good memory! > > > > The stack usage issue was due to struct collapse_control being > > allocated on the stack when MAX_NUMNODES was massive[1]. Now that we > > kmalloc() them for MADV_COLLAPSE, idea is we'd want to limit required > > memory and the original change here was an optimization due to an > > observation that we really didn't need 32 bits to hold a maximum value > > of HPAGE_PMD_NR[2], but determining some safe upper bound for the > > number of requires bits per arch/config as been a bit of a pain, with > > special casing adding to support POWER64 already[3], and now this mips > > issue that would otherwise require something special. > > Aha, thanks for refreshing my memory. > > The patch is fine to me. I'm wondering whether it is worth introducing > a new macro, for example, ARCH_HAVE_HPAGE_PMD_ORDER_MACRO (maybe not a > good name), then we could make the code #ifdef the new macro. Likewise, I don't know if it's worth it or not. The config generating the original -Wframe-larger-than= had 1024 nodes. If we use that as an upper bound the number of nodes in a system then it's a difference of 2 KiB / MADV_COLLAPSE call in the extreme case. Now, this is the first time I've actually used kmalloc(), so I don't have a good sense of how frugal we should be with memory. 2 KiB does sound like a lot - but I suspect the common case will be < 32 bytes. Do we have a sense for the largest sane CONFIG_NODES_SHIFT? The only systems I deal with are 1,2 or 4 nodes, so I don't know what a sensible upper bound is here. > > > > > [1] https://lore.kernel.org/linux-mm/CAAa6QmQXn9odA-dsm-FFKqQmDvBuH0qDkmPFfXDS1so+Loe+4Q@mail.gmail.com/ > > [2] https://lore.kernel.org/linux-mm/Ys2CeIm%2FQmQwWh9a@google.com/ > > [3] https://lore.kernel.org/linux-next/Ytk0TQO9khUQgjxT@google.com/