From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DB79C38145 for ; Wed, 7 Sep 2022 18:47:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6889D6B0072; Wed, 7 Sep 2022 14:47:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 638DE6B0073; Wed, 7 Sep 2022 14:47:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5013F8D0001; Wed, 7 Sep 2022 14:47:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 425E36B0072 for ; Wed, 7 Sep 2022 14:47:48 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2574240F4E for ; Wed, 7 Sep 2022 18:47:48 +0000 (UTC) X-FDA: 79886173416.20.07A6ADE Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) by imf28.hostedemail.com (Postfix) with ESMTP id BF223C00A1 for ; Wed, 7 Sep 2022 18:47:47 +0000 (UTC) Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-1225219ee46so38369652fac.2 for ; Wed, 07 Sep 2022 11:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=6XVVFWjFywoe53RZ4WNVSWintgozVoDBpSLVXx1ImNI=; b=cBxuHmzIaMGGJ7s/uLioqfb0sQJcN6D6BszYQKmMFNPGB5nowkj+jt2SzbFwtDKYB6 frLZI3cpVLZh37xkL4gnp4yTqukMnNAIGwR07Gj6Xn1/tLvsilJ67dWuqSBCbR7up8/z +y+sgDjXLRBbvPWEuhlA0p5fAFgYyC+OuAQrchIDby1u7rzJ/8AYh7JbnIihkpF4KxG+ igMBNEZOKkDypysydBYPGeh4Rbix3TyXOW9xG+BvKABP5WeIDjG6b/YbN85RVjEmrUTF hh0sI2oWoRc9rRjickIIY8KnkDawTOHbCgDgi4YBCWppvkKrZLhbEEiBuNNoL3NzjBRU at1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=6XVVFWjFywoe53RZ4WNVSWintgozVoDBpSLVXx1ImNI=; b=ArsZHA/CFMDCCDW7bWjTakrMkiaZTZ2i/hw88qGosHerpyaf5kmOMn9wTpjfHD7xvQ aRss5roQcGML18YMveo8dOk8hEKl72yBR/9CbSlPG2sgyCwP3Mw/QDPJ3j9SOFFXK0ir am8qMy9s+VRUbATRtE55B7YOgoWInAZ62UjMWDwEyMrGeiouKHJ9qZlkf/cKkDdQ+3BA ytaCbK8spSg4oFtfmH1Aby72qMepC5Le1bXjb9+vxPwa3RM9NInen5tuuNPYm0kSYxWh itPo/0j15Vdn4xB8+8CMlHQBFLiqKZjwo42r+uaUnBYBWXWuCbdCo3VlBaNtOG7Ojb3V E6hA== X-Gm-Message-State: ACgBeo2pumNrC019wgubCUYq43F7I8Q+pKH1Bb6ccAawqPVeFyfhFg9U 9jpykUR5HHk0K/3+j4VgMSLp+eNOs36Gfty6L6H61Q== X-Google-Smtp-Source: AA6agR4jog6VmuD4Dj30GJcrpODi92rfqmZ6tpI5XK1G/BnmcizKzrVC+coLmFKGs6hq/kBTBIraYBEqaohn/PatP0Y= X-Received: by 2002:a05:6870:4308:b0:10b:8cbe:c945 with SMTP id w8-20020a056870430800b0010b8cbec945mr11237oah.220.1662576466601; Wed, 07 Sep 2022 11:47:46 -0700 (PDT) MIME-Version: 1.0 References: <20220903021221.1130021-1-zokeefe@google.com> In-Reply-To: From: "Zach O'Keefe" Date: Wed, 7 Sep 2022 11:47:10 -0700 Message-ID: Subject: Re: [PATCH mm-unstable] mm/khugepaged: fix struct collapse_control load_node definition To: Yang Shi Cc: Andrew Morton , linux-mm@kvack.org, kbuild-all@lists.01.org Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cBxuHmzI; spf=pass (imf28.hostedemail.com: domain of zokeefe@google.com designates 209.85.160.50 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662576467; a=rsa-sha256; cv=none; b=Ok5J8Ka+yHOPA5pe+u7kYPL1rI5Isnfk7qyg2098FjCQh1+TcB7++5CAGLo2tNy2jha9Yc 6S/GxhGateMaym+wLmNPd1YG2EUXTg5q8whjBGjP+CLDZ/jw9Rhi23drQEirxyIv910SaG BTc9F/glZ0UjBGnpur9hMA9i7YDSUm8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662576467; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6XVVFWjFywoe53RZ4WNVSWintgozVoDBpSLVXx1ImNI=; b=64GRj3Sym0GM+dBgilQzzYznR6/aeFUnXdYzrGanN/K1+CWkdZGvlxvvXACypiWVNsg5Uo iDevwNcTy0S/DQnp6eJhIEJ9Cncl/rr1vvWgOLNBkvZCzT/x+xBWYvU+UV4c+qrsErJgac hAPj+6+Gu+K8sK52Vx2msKWNVe/I/MI= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cBxuHmzI; spf=pass (imf28.hostedemail.com: domain of zokeefe@google.com designates 209.85.160.50 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Queue-Id: BF223C00A1 X-Rspamd-Server: rspam05 X-Stat-Signature: 4ijg9dg1ct863gzo3ykjb68nm6o7r8ap X-HE-Tag: 1662576467-473839 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Sep 7, 2022 at 10:07 AM Yang Shi wrote: > > On Tue, Sep 6, 2022 at 1:28 PM Zach O'Keefe wrote: > > > > On Tue, Sep 6, 2022 at 1:04 PM Yang Shi wrote: > > > > > > On Tue, Sep 6, 2022 at 12:14 PM Zach O'Keefe wrote: > > > > > > > > On Tue, Sep 6, 2022 at 11:46 AM Yang Shi wrote: > > > > > > > > > > On Fri, Sep 2, 2022 at 7:12 PM Zach O'Keefe wrote: > > > > > > > > > > > > HPAGE_PMD_ORDER is used to determine a minimal amount of bits needed to store > > > > > > per-node counters in struct collapse_control, defining the array to be either a > > > > > > u16[HPAGE_PMD_ORDER] or u32[HPAGE_PMD_ORDER]. > > > > > > > > > > > > For some architectures/configs, HPAGE_PMD_ORDER isn't resolved at preprocessor > > > > > > time (suitable for resolving preprocessor directives) but rather expands into a > > > > > > compiler built-in that is resolved at compile-time. E.g. for 32-bit MIPS (see > > > > > > link for full config): > > > > > > > > > > > > ---8<-- > > > > > > >> arch/mips/include/asm/pgtable.h:238:26: warning: "__builtin_ffs" is not defined, evaluates to 0 [-Wundef] > > > > > > 238 | #define PTE_T_LOG2 (__builtin_ffs(sizeof(pte_t)) - 1) > > > > > > | ^~~~~~~~~~~~~ > > > > > > arch/mips/include/asm/pgtable-32.h:65:43: note: in expansion of macro 'PTE_T_LOG2' > > > > > > 65 | # define PGDIR_SHIFT (2 * PAGE_SHIFT - PTE_T_LOG2 - 1) > > > > > > | ^~~~~~~~~~ > > > > > > include/asm-generic/pgtable-nop4d.h:11:33: note: in expansion of macro 'PGDIR_SHIFT' > > > > > > 11 | #define P4D_SHIFT PGDIR_SHIFT > > > > > > | ^~~~~~~~~~~ > > > > > > include/asm-generic/pgtable-nopud.h:18:25: note: in expansion of macro 'P4D_SHIFT' > > > > > > 18 | #define PUD_SHIFT P4D_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/asm-generic/pgtable-nopmd.h:20:25: note: in expansion of macro 'PUD_SHIFT' > > > > > > 20 | #define PMD_SHIFT PUD_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/linux/huge_mm.h:109:25: note: in expansion of macro 'PMD_SHIFT' > > > > > > 109 | #define HPAGE_PMD_SHIFT PMD_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/linux/huge_mm.h:105:26: note: in expansion of macro 'HPAGE_PMD_SHIFT' > > > > > > 105 | #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) > > > > > > | ^~~~~~~~~~~~~~~ > > > > > > mm/khugepaged.c:90:5: note: in expansion of macro 'HPAGE_PMD_ORDER' > > > > > > 90 | #if HPAGE_PMD_ORDER < 16 > > > > > > | ^~~~~~~~~~~~~~~ > > > > > > >> arch/mips/include/asm/pgtable.h:238:39: error: missing binary operator before token "(" > > > > > > 238 | #define PTE_T_LOG2 (__builtin_ffs(sizeof(pte_t)) - 1) > > > > > > | ^ > > > > > > arch/mips/include/asm/pgtable-32.h:65:43: note: in expansion of macro 'PTE_T_LOG2' > > > > > > 65 | # define PGDIR_SHIFT (2 * PAGE_SHIFT - PTE_T_LOG2 - 1) > > > > > > | ^~~~~~~~~~ > > > > > > include/asm-generic/pgtable-nop4d.h:11:33: note: in expansion of macro 'PGDIR_SHIFT' > > > > > > 11 | #define P4D_SHIFT PGDIR_SHIFT > > > > > > | ^~~~~~~~~~~ > > > > > > include/asm-generic/pgtable-nopud.h:18:25: note: in expansion of macro 'P4D_SHIFT' > > > > > > 18 | #define PUD_SHIFT P4D_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/asm-generic/pgtable-nopmd.h:20:25: note: in expansion of macro 'PUD_SHIFT' > > > > > > 20 | #define PMD_SHIFT PUD_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/linux/huge_mm.h:109:25: note: in expansion of macro 'PMD_SHIFT' > > > > > > 109 | #define HPAGE_PMD_SHIFT PMD_SHIFT > > > > > > | ^~~~~~~~~ > > > > > > include/linux/huge_mm.h:105:26: note: in expansion of macro 'HPAGE_PMD_SHIFT' > > > > > > 105 | #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) > > > > > > | ^~~~~~~~~~~~~~~ > > > > > > mm/khugepaged.c:90:5: note: in expansion of macro 'HPAGE_PMD_ORDER' > > > > > > 90 | #if HPAGE_PMD_ORDER < 16 > > > > > > | ^~~~~~~~~~~~~~~ > > > > > > ---8<-- > > > > > > > > > > > > load_node is already special-cased for CONFIG_PPC64 in > > > > > > "mm-khugepaged-add-struct-collapse_control-fix-fix", and we could continue down > > > > > > that path by #define'ing away edge cases, but I think at this point it easier > > > > > > to just let load_node be an array of u32s. If it's ever an issue, we can > > > > > > revisit. > > > > > > > > > > > > This essentially rolls back both: > > > > > > > > > > > > "mm/khugepaged: use minimal bits to store num page < HPAGE_PMD_NR" > > > > > > "mm-khugepaged-add-struct-collapse_control-fix-fix" > > > > > > > > > > > > (though we'll leave it as a u32 vs an int). > > > > > > > > > > > > Link: https://lore.kernel.org/linux-mm/202209021349.F73i5d6X-lkp@intel.com/ > > > > > > Fixes: 5f7bb00bd9cd ("mm/khugepaged: use minimal bits to store num page < HPAGE_PMD_NR") > > > > > > Fixes: 03053c821af2 ("mm-khugepaged-add-struct-collapse_control-fix-fix") > > > > > > Signed-off-by: Zach O'Keefe > > > > > > --- > > > > > > mm/khugepaged.c | 7 +------ > > > > > > 1 file changed, 1 insertion(+), 6 deletions(-) > > > > > > > > > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > > > > index af3b07eb2389..29142a35d24c 100644 > > > > > > --- a/mm/khugepaged.c > > > > > > +++ b/mm/khugepaged.c > > > > > > @@ -93,13 +93,8 @@ struct collapse_control { > > > > > > bool is_khugepaged; > > > > > > > > > > > > /* Num pages scanned per node */ > > > > > > -#if defined(CONFIG_PPC64) > > > > > > u32 node_load[MAX_NUMNODES]; > > > > > > -#elif HPAGE_PMD_ORDER < 16 > > > > > > - u16 node_load[MAX_NUMNODES]; > > > > > > -#else > > > > > > - u32 node_load[MAX_NUMNODES]; > > > > > > -#endif > > > > > > > > > > I recall we need this to circumvent some build time warning about > > > > > stack usage? Or I misremember? > > > > > > > > > > > > > Hey Yang, thanks for keeping a look out here, and good memory! > > > > > > > > The stack usage issue was due to struct collapse_control being > > > > allocated on the stack when MAX_NUMNODES was massive[1]. Now that we > > > > kmalloc() them for MADV_COLLAPSE, idea is we'd want to limit required > > > > memory and the original change here was an optimization due to an > > > > observation that we really didn't need 32 bits to hold a maximum value > > > > of HPAGE_PMD_NR[2], but determining some safe upper bound for the > > > > number of requires bits per arch/config as been a bit of a pain, with > > > > special casing adding to support POWER64 already[3], and now this mips > > > > issue that would otherwise require something special. > > > > > > Aha, thanks for refreshing my memory. > > > > > > The patch is fine to me. I'm wondering whether it is worth introducing > > > a new macro, for example, ARCH_HAVE_HPAGE_PMD_ORDER_MACRO (maybe not a > > > good name), then we could make the code #ifdef the new macro. > > > > Likewise, I don't know if it's worth it or not. The config generating > > the original -Wframe-larger-than= had 1024 nodes. If we use that as an > > upper bound the number of nodes in a system then it's a difference of > > 2 KiB / MADV_COLLAPSE call in the extreme case. Now, this is the first > > time I've actually used kmalloc(), so I don't have a good sense of how > > frugal we should be with memory. 2 KiB does sound like a lot - but I > > suspect the common case will be < 32 bytes. Do we have a sense for the > > largest sane CONFIG_NODES_SHIFT? The only systems I deal with are 1,2 > > or 4 nodes, so I don't know what a sensible upper bound is here. > > CONFIG_NODES_SHIFT is 10 in my default config. [..] Good to know! Thank you! > [..] So every kmalloc > actually allocates 4K (u32) or 2K (u16). But 2K difference for each > MADV_COLLAPSE call should be not a big deal on the machines that run > THP. So it may be overkilling. SG - can always change it later if it proves to be a lot / if we have many MADV_COLLAPSE calls in flight. > Anyway I'm fine to the patch, you could have my Reviewed-by: Yang Shi > Thanks as always! Zach > > > > > > > > > > > > > [1] https://lore.kernel.org/linux-mm/CAAa6QmQXn9odA-dsm-FFKqQmDvBuH0qDkmPFfXDS1so+Loe+4Q@mail.gmail.com/ > > > > [2] https://lore.kernel.org/linux-mm/Ys2CeIm%2FQmQwWh9a@google.com/ > > > > [3] https://lore.kernel.org/linux-next/Ytk0TQO9khUQgjxT@google.com/